December 10, 2013

...Learn TDD with Codemanship

Government Is The Wrong Customer For A Software Project

Hitting the news today is more questions in Parliament about the beleaguered new IT system for Universal Credit.

While deadlines slip and scope creeps, DWP ministers are facing tough questions about what - if indeed anything, should we believe the protestations of those at the DWP - appears to have gone wrong.

What sets this apart from the usual story of government waste is the focus on how this software was written. It's a surreal experience to watch MPs throwing out technical jargon - not least because it's immediately apparent to anyone in the know that none of them has quite grasped what these words really mean.

At the centre of the maelstrom is the age-old battle between "waterfall" and "agile" software development. DWP boss Iain Duncan Smith is being roundly whipped for not knowing exactly what he wanted at the start, and the project is coming under white-hot scrutiny because it appears now to be "late" and "over budget".

So, it's all very well and good being "agile", just as long as the requirements are fixed at the start and the plan doesn't change.

To be honest, I saw this coming. Government is the wrong customer for an Agile project. And by that I mean; government is the wrong customer for a software project.

Software development - let's stop calling it "Agile Software Development" - requires high levels of trust among the parties involved, and lots of open and honest communication.

That's not what governments do. Governments specialise in cultures driven by fear, blame, bureaucracy and secrecy. Couple that with a case of the Peter Principle on steroids, where the least qualified wield the most power and influence, and you've got the customer from hell.

Exhibit A is this ridiculously uninformed debate about "waterfall vs. agile" that this project has stirred up, like a swarm of angry bee managers who have never flown or collected pollen themselves, but know all the flying and pollen-collecting buzzwords.

If you're new to this software lark, let me spell it out for you: as an industry, we rejected the idea of "waterfall" software development decades ago. The idea itself grew out of people misreading a paper by Winston Royce in 1971 that specifically warned against trying to get it right first time. For longer than I've been alive, the software industry has known that waterfall doesn't work. I would go further; not only does waterfall not work, it doesn't even exist. Nobody has ever done it. Sure, many have tried...

Here's what "waterfall software development" really means in practice:

A team attempts to understand the requirements, come up with a complete design, implement the whole thing, tests it and then ship it to the customer. The customer says "no thanks, that's not what we need". Queue frantic "stabilisation" phase where customers report hundreds of "bugs" that, on closer examination, turn out to be change requests. Y'know, the change requests we didn't allow them to make because we fixed the requirements and the design too early. Along with a whole bunch of actual bugs, because - and you might find this hard to believe - teams that try to do waterfall also tend to be a bit slapdash in their programming practices. Shock!

Much of the real value in the software is added during this phase. It's a highly iterative, feedback-driven process. Arguably, it's the highly iterative, feedback-driven process they should have been using from the start. But hey, we live and learn.

Another very common outcome on "waterfall" projects is to create - through sheer force of personality sometimes - the illusion of "done" so they can claim to have delivered "on time and on budget". Goalposts get shifted, sometimes until they're out of town. And this is fine, as long as you're not still claiming to have delivered what you originally promised, and you're prepared for the big slowdown that will kick in as soon as that first version goes live and your entire development team's tied up fixing bugs. Future evolutions of the software will be far less affordable, and therefore far less likely. Expect to have to rewrite the entire thing again from scratch (maybe several times) if you need to accommodate significant changes.

You can't cheat reality. You may have convinced your customer - nay, they may have deluded themselves by this point - that the software's good enough and that the plan worked, but you'll still have to pay the piper. If it's not good enough, that cost will be in the chaos and lost productivity of the end users forced to adopt it, as well as the bug fixing and grindingly slow pace of on-going innovation that will rob you of opportunities down the road.

Yes, waterfall can be a comforting partner in the casino of software development, whispering reassurances that all is well into your ear. But then it will mug you in the car park on your way out, and you'll end up not just a loser, but a bloodied and battered one at that.

The Universal Credit debacle, of course, was clearly sign-posted from the beginning. Our first clue should have been the sheer size and scope of it. Whenever a customer says "this new system will replace EVERYTHING" we should turn and run in the opposite direction.

Statistics on IT project failure are pretty stark on the relationship between project cost and failure. Anything over about $1,000,000 is almost guaranteed to run into trouble. When the DWP claim that it's "normal" for our industry to write-down amounts as large as £40M on a single software application, they might like to judge that again, but this time perhaps omitting the banks. Outside of financial services, it is not normal to waste this kind of money.

The budget alone told me that this was a shitstorm in the making. Not as big a shitstorm as the NHS Connecting for Health programme, obviously. But a shitstorm nonetheless.

Governments should steer clear of undertaking large software projects.

They should focus instead on workable open data standards, and devolve the creation of software that captures and consumes this data. The NHS programme could have worked more simply by agreeing data and privacy/security standards for patient records and then allowing hospitals and healthcare trusts to commission or develop the software they need, implementing these standards for I/O.

UC, too, could work better as a devolved open program, driven by job centres and partners/employers where DWP sets the standard for sharing data and the people who use the data - or whose data it is - can develop applications targeted at their specific problems.

We've seen many times in the software industry how open standards scale far better than proprietary solutions: the World Wide Web being the most spectacular example. They also give individual organisations more choice over when and how they adopt such standards. If adopting the standard brings them benefits (no pun intended), then in the longer term adoption will spread and the growth of adoptees will drive yet more adoption, such is the nature of sharing. And if they don't benefit from it, then, well, failure is probably the best thing all round.

Posted 1 week, 6 days ago on December 10, 2013