March 13, 2014

...Learn TDD with Codemanship

Waterfall, Reality Avoidance & People Who Say "No"

I have this theory.

One of the problems some managers have with iterative software development is that, when it's done well - seeking early and frequent feedback and acting on it, as opposed to just incrementally executing a waterfall plan - it reduces the scope for avoiding reality.

On a waterfall project, reality can be avoided for months or even years. The illusion of progress can be maintained, through form filling and the generation of reams of reports that nobody ever reads, right up until the point that software needs to be seen to be working.

If it were my money, this would scare the shit out of me - not knowing what my money's been spent on until the last moment.

But I can see the attraction for managers. It's not their money. And typically they get rewarded for this illusion of progress, which can go as far as pretending the software is ready the night before it's deployed into a live environment.

One of my early experiences as a freelancer in London was leading a team on the development of a web site that the company had been kicking around as an idea for 2-3 years. Naturally, after 2 years of business analysis, and 6 months of database design - based on what use cases, one can only imagine - the team were given six whole weeks to implement the site.

Our project manager was pretty canny, and understood how much of a squeeze this was going to be. So, back in 1999, I first tried my hand at a new thing called "Extreme Programming", because we felt what we were doing was extremely ambitious, and the right thing to do with such short timescales was to iterate extremely.

But the customer wouldn't play ball. We wanted to show him working software, but he literally refused to come downstairs and take a look at what the company's money was being spent on. He insisted, instead, that the project manager wrote reports. Lots of reports. Daily reports. Detailed reports.

And when the reports said "we are not as far as the plan says we should be", the reports were rejected. And new reports had to be written, saying that, actually, we were on plan.

For doing this, the project manager got promoted to Chief Technology Officer. The developers, who unanimously refused to play along and kicked up quite a fuss, got let go almost immediately afterwards.

A new project manager was appointed, who was more than happy to live with the illusion. I recall distinctly listening to a conversation with the business in which he told them that a piece of work that could take months and had not even been started was actually done, tested and ready to deploy. There's delusion, and then there's DELUSION.

Of course, it dodn't deploy. It couldn't. It didn't exist. But, again, months of him telling the business what they wanted to hear, regardless of the reality that unfolded, got him promoted to senior programme manager. I was long gone by this point, thankfully. (On to the next delusional dotcom boom shitstorm.)

This was an important lesson for me - most failure is. I learned that, in many organisations, people aren't rewarded for what they achieve. They're rewarded for what they're prepared to claim they've achieved. And even when it turns out to be an out-and-out lie, and the business is left with egg all over their faces, they may still get rewarded.

To their own detriment, too many businesses reward people for saying "Yes", even when the real answer - the answer they need to hear - is "No".

Software developers - people who actually make software happen - do not have that luxury, though many try. Software either is, or isn't. It either does, or it doesn't. When the chips are down, we can't fake it. We may say "it's ready", but when it ships reality will pull its pants down for all the world to see.

Among other reasons, I believe this one of the key reasons why managers don't like us. They need us to make things happen, but we have this annoying habit of saying "No" and telling them things they didn't want to hear.

Iterative development throws light in dark corners they'd rather we didn't look, and as such that's why I believe waterfall is a reality avoidance mechanism for many managers. Specifically, they want to avoid having to go back to their bosses or their customers and say "No", because in the game they're playing - which can be s distinctly different game with distinctly different aims to the game developers play - they lose points for doing that.

The only meaningful way to bring reality back into play is to realign the goals of managers and developers and get everyone playing the Let's Actually Deliver Something Of Value game. Managers need to be rewarded for testable achievements, and steered away from peddling illusions. The reason this doesn't happen more often, I suspect, is because the value of illusion increases the further up the ranks you go. If a PM gets a pat on the back for saying "we're on track", the CTO gets a trip to Disneyland, and the CEO gets a new Mercedes. Hence, the delusion gets stronger as we go higher. People running governments tend to be the most delusional of all, such is their power and influence. This effect is what produces the sometimes gargantuan IT failures only governments seem capable of creating.

The Emperor has no clothes. Bah humbug.

Posted 6 years, 9 months ago on March 13, 2014