a Tale of a May Release….

19 10 2008

Ok, as you can see the writing has stopped for a while, not that it was that prolific before that, but …. So here is what’s been consuming the time for a while:

There is this project that was called “Release Cinq”, it started sometime in 2007 (the real start date is a bit of a mystery  to me just yet, as the case was made to senior management and they approved it. Around September of 2007, after some re-shuffling, I came to be the Development Manager for this project.

The original developers in Europe, were all 10+ years of experience, and the description of the project was:

.- We need to refactor the application because of systemic issues that affect stability, in particluar the database has high levels of contention and locking, bringing the system down from time to time. (this application is used 24×7 in many of the installations).

.- We are well advanced on the refactoring, and we should be done with part1 by december, and then part 2 is much easier maybe an extra month or two.

Here is my first mistake:

I trusted the information that I was given, without digging into the code or the process, It all made sense, the system has a known issue, that requires some core refactoring, the issue has been identified, and the solution is known, so all good. Well it turns out that the “refactoring” meant “rewrite” !!!!, there was an actual separate source tree where functionality was being rewritten from scratch, using new libraries, new architecture, new everything !!!, there was little or non existent documentation for the code, there were no QA test plans, no unit test, no automated build, no nightly build etc…… OMG!!!!

Of course there was no part 1 and 2 !!!, it all needed to be done, as old code was not even compiled with the new one, so looking at the progress, and what needed to be done, We put together a plan that would take us to completion by the end of February, which of course took longer as parts that were deemed to be “left out of the refactoring” still needed to be “ported” to the new codeline which was substantially different.

So, by April all was done, ready for QA, ready for performance testing, but the risk was too high to make it a “customer release”, and the cost was “too high”, so we “moved on” on the next release, and everyone just assumed that the Cinq was ok.

And here is my mistake number two:

I went against my gut, got overriden for budgetary considerations, and I allowed it. knowing that it was a full rewrite, knowing the previous history, and knowing how critical this was. I should have stopped this and forced everyone to complete testing and all activities related to the release. I should have gotten to the point of actually quitting over this, but, as history wil have it, I did not.

[...]

So we moved on, added new functionality on top of this new platform that was never deplyed, and made customer wait even more for a release that actually had some new stuff. As expected the QA process for “Cinq dot one” was PAINFUL, the performance was horrible, and there were many customizations that would not work, so we had to spend 3 pretty much 10 weeks more than planned to get it to a Beta level, and then go over 7 weeks of Beta to fix the last kinks, and last week we were done, Beta has been successful, data migrated, services trained, documentation done, i.e. the normal stuff

The end of the story ?

Well the company was not doing so great (this is one of 10+ product lines), you know one of those where the stock was $12 a year ago, and now is limping at $1.50, and last Wed the company basically decided that anything that was not cashflow positive needed to be ASAP, or get stopped. So I (and my team) were on the later category, as it happens when working on leading products trying to capture market, so We were all laid off last Wed…