February 5, 2006

...Learn TDD with Codemanship

CHAOS Revisited

A very discomforting article on cio.com has made me think again about the findings of the Standish Group's infamous CHAOS report.

I've used the report myself as ammunition at senior management level, but the mention of "predictability" in the cio.com article in the same breath as the CHAOS statistics has made me rethink a little (well, actually, a lot).

According to the report, in 1994 about 84% of software projects were either compromised (over time, over budget) or failed (got canned). The report cites all sorts of factors in project failure, some of which I've applied to project "health checks" in the recent past.

The problem with the report's findings is that they actually don't tell us anything of any real value. So what if we could complete 100% of projects on time and under budget? What value does that have? The money's been spent, and what did the customer get for it? In the eyes of CHAOS, we could deliver something of little or no use to anyone, and as long as we do it by the deadline, spend what we said we'd spend, and avoid getting canned early, our project has been a success.

When you read the movie reviews in your newspaper, do you think the ratings the critics give to each film have anything to do with whether or not the movie was completed on time and on budget? Do you think the top ten movies list is ordered by schedule or budget predictability, or by box office takings?

I think predictability is a red herring. Who knew that '1941' would turn out to be a flop? Who could have predicted that 'Hook' would be a turkey? And yet we see overall that Steven Spielberg's movies are generally far more successful than any other directors. I bet both those films were brought in on time and on budget. In his excellent book Ubiquity, Mark Buchanan discusses a power law in the distribution of size of box office takings and the number of films who achieve them. For every film that takes $300 million, there are thousands that don't break even. It points to strong evidence that movies are like earthquakes, or stock markets, or populations. The size and timing of their success is every bit as unpredictable.

I suspect software projects are the same. Whether we get them in on time and on budget, the value of the resulting software will vary according to the same power laws. The most valuable systems will be many orders of magnitude more rare than the majority of systems that barely break even.

There is a factor in this that makes success a little more predictable in movies than in software development: statistical gravity. I explained in an earlier post how success attracts more success. A director who, by blind luck, has a major hit, now has better actors, better technicians, bigger budgets, for his or her next film. They also have a major advantage when it comes to marketing their next film - "from the director who brought you..." So the chances of subsequent successes are massively increased.

Having said that, a sequel of a hit movie is much less of a risk than a completely new movie. When a software release turns out, by pure chance, to be very valuable, we can build on that code to create the sequel. So maybe they're more similar than I first thought. In fact, maybe they're VERY similar indeed. Maybe the trick is to know a winner when you see one, and to back the winner as soon as you know.

Anyway, my final thought is this: should we be more concerned about exploiting and building on random success when it occurs, or should we be totally focused on the budget and schedule of our initial gambles? Certainly, most businesses bet all their money on just one project to achieve a specific goal. And certainly, most software developers never stick around long enough to know if their application is a hit or a flop. The worry I have about the cio.com article, and the CHAOS report, is that it's central argument is that it's okay to put our eggs all in one basket while we focus on improving the chances of individual projects.

Perhaps the recommendation should be to hedge your bets, and back a winner when one emerges?
Posted 14 years, 11 months ago on February 5, 2006