September 25, 2006
The Waterfall DelusionToday is one of the days when I feel justified in acting like a child again (well, on purpose, at least). I've been looking forward to getting my hands on a copy of Richard Dawkins' new book, The God Delusion for weeks, and it hits the shop this morning. So for me Christmas - a secular Christmas, that is - has come early and I just can't wait to open my present.
This book as been a long time coming. Every few weeks some very respectable people dressed in black come to my door and do their level best to convert me to [insert namke of half-baked religious philosophy here]. Now it's the turn of our arch-atheist to turn the tables and do his level best to convert those of faith to the church of rationalism. (Whose membership has been dwindling rapidly in recent years.)
I feel like dressing up in my Sunday best and taking copies of this book from door to door, asking if people have "heard the bad news"...
Another entirely irrational belief is that there are circumstances where waterfall development is preferable to evolutionary design. Not only does the maths not stack up on that argument (for the same reason that we shouldn't bet good money on even the best golfer scoring a hole in one), but the evidence is overwhelmingly in favour of the evolutionary approach to solving complex problems. So much so that, like God, there's absolutely no rational reason to believe otherwise.
Surely if there's one single, solitary thing we in the software industry should be able to agree on it's that:
Evolutionary design is always better
The power of the evolutionary approach to solving complex problems can be neatly illustrated with a very simple example. Imaghine you are asked to guess a 4-digit number:
X X X X
One approach might be to guess the whole number in a single go. What are the odds of success? Well, there are 10,000 possible combinations of 4 digits, so our chances are 1/10,000 of getting it right first time. It might, therefore, take as many as 10,000 guesses to get it right.
A better approach might be to guess one digit at a time. What are the odds of guessing the first digit correctly in one go? 1/10. To guess all 4 might take as many as 40 guesses. That's a massive improvement on 10,000.
The power of the incremental approach is in breaking down a complex problem, finding the correct the solution for which is highly improbable, and solving it in more probable chunks. We must expect our first attempt at a solution to be wrong, and that we will need several attempts, and therefore it's utter folly to plan to deliver once at the end of the project - because we are effectively gambling our entire budget on getting a hole in one. And yet this is exactly what most projects do.
The reason why is probably the same as the reason why so many people still say they believe in God: it pays to be faithful in the company of people who share that delusion. The economic benefits of putting your faith in waterfall development are readily apparant. Most clients are so ready to believe in it that it should come as no suprise that software developers become members of the same church. Few clients want to do business with non-believers.
But the damage done by the waterfall mentality - while a drop in the ocean compared to the damage done by organised religion - is still immense. Billions upon billions of dollars are wasted every year on "big bang" projects.
Posted 3 weeks, 2 days ago on September 25, 2006