There is this project that was called “Release Cinq”, it started sometime in 2007 (the real start date is a bit of a mystery  to me just yet, as the case was made to senior management and they approved it. Around September of 2007, after some re-shuffling, I came to be the Development Manager for this project.

The original developers in Europe, were all 10+ years of experience, and the description of the project was:

.- We need to refactor the application because of systemic issues that affect stability, in particluar the database has high levels of contention and locking, bringing the system down from time to time. (this application is used 24×7 in many of the installations).

.- We are well advanced on the refactoring, and we should be done with part1 by december, and then part 2 is much easier maybe an extra month or two.

Here is my first mistake:

I trusted the information that I was given, without digging into the code or the process, It all made sense, the system has a known issue, that requires some core refactoring, the issue has been identified, and the solution is known, so all good. Well it turns out that the “refactoring” meant “rewrite” !!!!, there was an actual separate source tree where functionality was being rewritten from scratch, using new libraries, new architecture, new everything !!!, there was little or non existent documentation for the code, there were no QA test plans, no unit test, no automated build, no nightly build etc…… OMG!!!!

Of course there was no part 1 and 2 !!!, it all needed to be done, as old code was not even compiled with the new one, so looking at the progress, and what needed to be done, We put together a plan that would take us to completion by the end of February, which of course took longer as parts that were deemed to be “left out of the refactoring” still needed to be “ported” to the new codeline which was substantially different.

So, by April all was done, ready for QA, ready for performance testing, but the risk was too high to make it a “customer release”, and the cost was “too high”, so we “moved on” on the next release, and everyone just assumed that the Cinq was ok.

And here is my mistake number two:

I went against my gut, got overriden for budgetary considerations, and I allowed it. knowing that it was a full rewrite, knowing the previous history, and knowing how critical this was. I should have stopped this and forced everyone to complete testing and all activities related to the release. I should have gotten to the point of actually quitting over this, but, as history wil have it, I did not.

[...]

So we moved on, added new functionality on top of this new platform that was never deplyed, and made customer wait even more for a release that actually had some new stuff. As expected the QA process for “Cinq dot one” was PAINFUL, the performance was horrible, and there were many customizations that would not work, so we had to spend 3 pretty much 10 weeks more than planned to get it to a Beta level, and then go over 7 weeks of Beta to fix the last kinks, and last week we were done, Beta has been successful, data migrated, services trained, documentation done, i.e. the normal stuff

The end of the story ?

Well the company was not doing so great (this is one of 10+ product lines), you know one of those where the stock was $12 a year ago, and now is limping at $1.50, and last Wed the company basically decided that anything that was not cashflow positive needed to be ASAP, or get stopped. So I (and my team) were on the later category, as it happens when working on leading products trying to capture market, so We were all laid off last Wed…

In the management side of things the key is to instrument the process, i.e. be able to answer the question of : How are we doing ?.

In software development many processes have been established to help on this, and more likely than not successful teams will use a combination. I personally have found that a semi agile way works, having “short” iterations of about 3 weeks, with specific goals, and estimates that are counted down against a burndown chart on a daily basis.

Of course that is not enough, we need to know that the code is actually working and stable, so a nightly build, continuous integration, nightly unit test suite and nightly performance test suite, help ensure that the new features are not breaking the old stuff and that the code base is stable and performing in the expected range.

We even go a bit ahead, and run a nightly customer database upgrade, so any time that changes are made to the schema(s) we make sure that all data can be migrated without errors.

Finally a code coverage report over the unit testing lets us know what parts we are testing and which ones we are not, which is great input for QA too.

All of those give me what I think is a pretty good instrumented process, it is visible to everyone, and different players know their responsibilities, if the unit test is broken, then it needs to fixed, if the performance is slow, then it needs to be explained, and usually fixed, and after a while the team works on this almost automatically. I get emails explaining where the failure happened and how it was fixed

This instrumentation has made the project a success, and has made the management of the offshore team somehow easier. I’d be curious to hear your experiences on this.

This is how it works, I would send an email to one of the QA engineers assigned to the project, something along the lines of :

Hey Mr A. can you please send me an update on the progress of the QA cycle ? in particular on the Regression Test Plan execution as I am worry we might be running late.

As a reply I would get a a quick note: “I’ll send you the information by tomorrow” and a couple of people added to the cc: like his Manager, and the other QA engineer. Later on his Manager would send an update, and now the CC list would also include the Dev. director, and of course his comment will include now the VP of QA… so by now the innocent question has been made visible all the way to the top, Well just one step before the CTO or CEO of the company.

Don’t get me wrong, I am all about visibility and letting stake holders know what’s going on, but it needs to be done in a sensible fashion. I thought I was the only one thinking that this was a problem, but a few months later I was talking with the QA VP, and she happened to mention somthing that I connected to this practice: I usually don’t fully read the emails, as I get too many, unless I am on the To: field. Which I think was her way of saying If I am not on the To: I’ll go to the archive directly.

Sometimes I even remove people from the cc: as I think the topic is going into too much detail, just to find out on the next answer that they’ve been added again.

In the end I suspect is a self defense mechanism, maybe a way to keep a supervisor apraissed of how the time is spent ? maybe a way to “prove” that I was against that from the beginning, or just a I told you so (even if you were not interested at the time). If that’s the case, maybe is a “strange” work environment where people feel the need to self protect using these growing cc list.

So when to use the Cc ? My rule is to include someone in the Cc: when likely that person will become involved in the conversation, either on the email, or even later in a phone or face to face conversation. And I usually even include the reason towards the end of the email, like:

Mr S. I am Cc-ing you as I think we’ll need to include this one on the next release of Product Y

email at work is just one of the many communication tools available today, and even though it is so easy to include anyone on that innocent cc. field, always ask the question: Is this person really interested on this email ?

Of course I’d be curious to know your experiences on these growing Ccs, or is it just me going crazy ?

The project started by getting new developers on the team back in Shanghai, and of course a few training weeks, a few trips for face to face sessions, and another set of trips to move releases along when they start to run a bit late… I’ll most definitely write another post some time about the whole setting up an offshore team. Now 18 months later, we have had 3 succesful releases out, all of them on time, feature complete, delivered to the customers, so a success by any “external” measure, and I take pride on that .

With my past teams though, pride and ownership of the code has taken place, usually after a few months working together, maybe a succesulf release or two, maybe after working on a particularly hard problem, maybe after praise from the rest of the company, maybe just a combination of all.

This ownership and pride, usually comes in the way of:

• Taking care of the code, keep a clean code base. For example fixing those deprecated calls, and not just the one that you happen to find, but see if it’s used somewhere else.
• Checking out what other are doing, it is “our application” not just my lines in that application
• Refactoring as needed instead of copy/paste small pieces of code here and there
• Doing some “backburner” projects (Hey, what if I change this ? maybe I can get some more performance out of it)
• Enhancing the “misc” “util” kind of classes, so other benefit from commonly used tools.

With the current team, these actions usually take place propted by myself or Mr C. (another senior engineer based on the US). One of the common tasks for myseld and Mr C. is the code review, checking in what was checked into source control by the offshore team, and provide feedback and recommendations on the implementation, we have noticed how the team has grown, we’ve stopped sending the same feedback time and again, and usually later feedback goes more on the design and clarity of the code, so we know that they can take the feedback and make it part of the standard development, however the other part of the feedback goes into the ether, when we say, maybe that should be refactored to support the new xxxx or make a utility class thta can be used from X and Y… that’s the feedback that I believe is more about inspiring the team to make things better, not just make them work, and that’s the one where we have not been so succesful so far…

 Of course I’d be curious to hear your experience and how have you managed to inspire your remote teams, or if you are on the remote side, how have you been inspired to do so, to take ownership and pride on the product being developed.

So there you have it, just a brief intro, just a brief note for now, and more to come. yeah, yeah yeah, I know: blog regularly, pick a schedule and stick to it, just like cliché #9 so I, hereby commit to start at least once every couple of weeks…..

and yes, this one counts as the first one.

So what I’ll be writing about ? I suspect it might be a mix of tech stuff (Yes I am a software engineer), maybe some traveling (I do travel from time to time), and maybe some personal, maube some photo comments (as I do love flickr) and my takes on life (well who wouldn’t)…