The dust has settled from releasing version 24 of XP-Dev.com and I would like to thank each and everyone one of XP-Dev.com users for being extremely patient on last Sunday (8th Feb 2009). I am extremely grateful to have such wonderful users and really appreciate all the kind words of support during the release via twitter and our old blog page.

Having said that, I am going to attempt to dissect what exactly happened, and answer the question How in the world did I manage to make a 3 hour standard release into a 15 hour sit-around-and-wait ?

The Subversion 1.5 Upgrade

Subversion 1.5 is great. I have used the merge tracking features in the past and makes life working as a team with a complex code base a whole lot easier. Back in my previous job we had Subversion 1.4 installed and it was an outright pain to merge different sections of the codebase. Branching, though usually frowned upon, was a common practice and the act of manually merging files was as silly as trying to clean a football stadium with a toothbrush – it can be done, but you’ll be 105 years old by the time you’ve finished.

So, Subversion 1.5 is useful – it gets developers more productive, and in turn they are happy, which means they write good code, which means managers can deliver good software, which means clients are happy, which means managers are happy, which means developers are happy, which means ... you get the point!

Now, I’m going to let you on a secret – and you can’t tell anyone. The idea of XP-Dev.com has actually been around since March 2008. Oh yes. It has only evolved to a decent platform since November 18 2008 when I relaunched it under a new platform. So, back in March 2008 when I went live with the first version of XP-Dev.com, Subversion 1.5 was in “beta” - there was some merge tracking information but it was not ready to be put on an enterprise grade platform like XP-Dev.com :) caveat emptor!!.

I went ahead with Subversion 1.4, thinking "I’m sure I will be able to upgrade everything later on with such ease, I will make it look like taking a candy from a baby" (btw, I don’t usually do that. I only take humour away from babies. I don’t know what it is, but babies just hate me).

Fast forward to January 2009 when I was writing the migration scripts. There is a magic command that I could execute on all of the repositories that will upgrade them to support features in Subversion 1.5. In fact, a Subversion 1.4 repository could be served directly from Subversion 1.5, but that’s hardly a good setup. If I’m going to deploy Subversion 1.5, I will turn on all the features for Subversion 1.5. The upgrade needs to be justified.

Now, here comes the command. It’s a very simple one, and I could just about grasp the concept of it:

svnadmin update <repository path>

And that’s it.

One command, and it will do all the magic. You’ll be able to have all the merge tracking goodness from the upgraded repository.

But, that’s not the end of the story just yet. If you look closer at what svnadmin upgrade does, you’ll see that it just “patches” the repository:

$ svnadmin help upgrade
upgrade: usage: svnadmin upgrade REPOS_PATH

Upgrade the repository located at REPOS_PATH to the latest supported
schema version.

This functionality is provided as a convenience for repository
administrators who wish to make use of new Subversion functionality
without having to undertake a potentially costly full repository dump
and load operation.  As such, the upgrade performs only the minimum
amount of work needed to accomplish this while still maintaining the
integrity of the repository.  It does not guarantee the most optimized
repository state as a dump and subsequent load would.

Looking closer, the sentence that got me a little anxious was:

As such, the upgrade performs only the minimum amount of work needed to accomplish this while still maintaining the integrity of the repository. It does not guarantee the most optimized repository state as a dump and subsequent load would.

Now, I have a new motto for 2009: Take the freaking stairs.

That means that I am never going to take any shortcuts at all, especially on things that I am building for the long term, like XP-Dev.com.

The Upgrade Calculations

When I ran my tests, I did my timing calculations based on svnadmin upgrade and that was pretty darn fast. My estimate came to 1 hour and a bit.

However, I thought to myself "Is this what my users would want ? An upgraded repository that’s not really optimised? No. No. No. Do it the right way for crying out loud! Take the stairs and do a proper job. Do a full reload."

So, I did. The only thing was that I didn’t recalculate how long it was going to take as I was busy doing other development work on XP-Dev.com as part of the release. The tests were all perfect – reloads were fine, and all the repositories had the same UUIDs, so it was not a problem. In fact, looking back, there hasn’t been a single support ticket submitted about corrupted repositories.

On the morning of Feb 8, I took down the web server that serves the repositories as I did not want any stale checkins between doing a full svnadmin dump and svnadmin load. Kicked off the upgrade script and realised after 2 hours that it was still running, and there were thousands, upon thousands of repositories still left to be upgraded.

And, that’s how I managed to make a standard 3 hour release to 15 hours. I think it was worth the trouble.

Reflection

Would I have done it differently ? Would I have done what most competitors are doing ? Would I just install Subversion 1.5 and serve Subversion 1.4 repositories and upgrade on a case by case basis ?

NO.

I would have done the same thing again.

Does this come as a shock to you ? Maybe. But here’s my philosophy in running a business: be honest, be transparent, customers come first and no shortcuts. As soon as I knew that it was going to take much longer, I updated the blog and notified the current status on Twitter – notifying all users on what’s going on.

If there was one thing that I would do differently, it would be this:

• Recalculate the reload times correctly. I need a slap for this.
• Put up a notice on the blog and Twitter a few days in advance
• Do the whole upgrade and sing away into the sunset

So, there. A long explanation on what happened. I do apologise, and I will try not to make silly mistakes like this again.