July 12, 2011
As the Customer, You Can Make Agility HappenAgility, I've discovered after many years - okay, so I actually discovered it right at the start, but needed a good excuse to blog about something - that all the power lies with the customer.
If you want to force an "Agile transition", all you have to do is ask the right questions.
For example, if you ask to see working software at the end of every week, deployed on to your computers, then the developers are not going to have much choice but to deliver something on a regular basis. The weekly "show-and-tell" is one of the customer's most powerful weapons. And criminally underutilised. I've been amazed at how reluctant many customers are to even take 15 minutes out of their week to see where their money went. (And when I say "their money", I mean someone elses's money that they have responsibility for spending wisely, of course.)
And, as a customer, you're holding most of the planning cards, too. If you choose to change the plan based on feedback every week, identifying new requirements that are coming to light as the work unfolds, and reprioritising exciting requirements, then you can. All you have to do is accept that fixed-price R&D is a folly, and that your focus should be on what you're getting for your money, not on whether you're on plan and on budget according to some grand overall plan that was plucked out of the air at a time when you knew the least. Once your emphasis shifts from sticking to a plan and a budget to maximising the value you get from every weekly delivery of working software, you are very much in Agile territory, whether the developers like it or not.
By all means, have high-level "headline" business goals for the software. Indeed, be one of the 1% of software projects that even know what their business goals are. And measure your progress week-on-week towards those business goals, and then steer the software accordingly.
And what about the quality of the software? How can a non-technical person have any influence on that? For sure, it will come back to bite you on the backside if the software is of a low quality. The time it takes to fix 100 bugs discovered in a release is roughly 10-100x what it would cost to have weeded them out before release. That's time that could have been spent adding more value to the software. It's not uncommon for teams, after a few months, to wind up spending half their time just fixing bugs. Which is a terrific way to waste your money, don't you think?
Of course, we have to be careful what we mean by a "bug". Too many teams end up dealing with what are actually change requests - "the software does this, but what we wanted was that" - which have been reported as bugs. This is usually because the requirements were ambiguous and ill-formed. The developers interpreted them, and - not unsurprisingly - didn't quite deliver what you had in mind. The devil's in the detail, you see. Delivering working software's a great way to find out what you really needed - this is a learning process, after all. But it's one hell of an expensive way to found out what you wanted in the first place.
As a customer, you can at least ensure that the developers deliver what you ask for, even if it turns out you asked for the wrong thing. Insist on agreeing acceptance tests for every requirement. These should be clear, unambiguous and executable; that's to say, they should expressed in a form that you can actually perform as a real test. Use specific examples, ideally culled from the real world, to make it absolutely crystal clear what the software should do in key business scenarios. (E.g., "Given a bank account with a balance of $50 and no arranged overdraft facility, when the account holder tries to withdraw $50.01, the system tells them they have insufficient funds and the account is not debited")
Make your expectations absolutely clear. The software should pass all the acceptance tests you agree with the developers. If it doesn't, then they are not "done". Be clear, too, that you do not want the software to do any more than the tests require. If delivering features wrong is a waste of time and money, then delivering features nobody asked for is doubly so.
So now you have them caught in your vice-like grip. Every week, you sit down and plan with them the work for that week. Every requirement that gets scheduled is expressed as a suite of precise, executable tests using realistic examples. At the end of the week, there's a show and tell, at which the first thing you as the customer will do is perform the acceptance tests to see whether you got what you asked for.
The developers will have scored each of your feature requests for its relative complexity or difficulty (in much the same way that golfers score holes, estimating how many shots it might take to complete each one.) At the end of each week, tot up the total number of those estimated complexity points to see how much "stuff" the team got done. You can realistically schedule the same amount of "stuff" for the next week.
What we tend to find with the average software development team is that, working on the same software week after week, we see a visible decline in the amount of "stuff" they deliver over weeks, months and years. It's not uncommon to find, after even just a few months, that teams are achieving a small fraction of what they were at the start. Why is this, and what can you do, as a non-programmer, to slow that decline and get more "stuff" for your money?
The reason why productivity declines, often rapidly, in software development is that, over time, the code gets harder and harder to change. It gets complicated, difficult to understand, riddled with duplication, and congealed with dependencies that make it nigh-on impossible to change a line of code without breaking many other parts of the software. It becomes rigid and brittle.
On a twelve month project, we might expect - if unmanaged - that adding or changing a line of code in month twelve could cost 4-5 times as much as it did at the start. A feature request that would have cost $1,000 to implement in the first month could cost $5,000 at the end of that year.
A disciplined programmer will continually take steps to keep the code malleableand robust for as long as possible. She will work hard to ensure that her code is kept clean - simple, modular, easy to understand, low in duplication and with the dependencies carefully managed to limit the costly ripple effect when making changes. She was also ensure that she is able to retest the code comprehensively as quickly and cheaply as possible, which usually means she will write her own automated tests as she goes - since a suite of well-written automated developer tests can check if anything is broken in a matter of seconds. Waiting for your test team to tell her will introduce long feedback loops. By the time she hears that her change has introduced a bug, it may have been weeks or months since she made that mistake. Instead of just fixing it there and then, you will have to schedule a bug fix task, and she will have to go through the entire test and release cycle again before you'll see that bug fix in the finished product. That massively escalates the cost of quality, and the cost of making changes safely.
So, as a customer, you have to be very interested in code cleanliness and in automated developer testing (and, while we're about it, automating your acceptance tests - because there's nothing worse than discovering months after the fact that a feature that was tested and working is no longer working).
Now, naturally, you're probably not very interested in hearing about test coverage reports or about code quality metrics. And why should you be? But you should be interested in the end results of quality - or the lack thereof. I can tell you whether a development team has been taking sufficient care over the code, and putting enough effort into test automation, by a couple of very simple tests.
First of all, deliveries will be reliable. That is to say, when the developers say they're done, they really are done. As evidenced by the lack of bug reports post-release. If teams get hundreds of bug reports over the period of, say, a few months of software releases, I would wager good money that they're not automating their tests. What happens is a release goes out, a flood of bug reports come in, the developers work frantically to fix those bugs and then the software goes through another release. But their frantic bug-fixing has had side effects. Inadvertantly, they've introduced new bugs. And their inability to quickly retest the code means that those new bugs may well have gone undetected. So a whole new batch of bug reports go in, and they repeat the process. The feedback cycles are very long, and it can be very difficult to stabilise the software. If they're lucky, with each bug-fix release the overall number of known issues will decrease until the bug count's low enough for the software to be usable in the real world. But sometimes the overall bug count doesn't go down, and teams get caught in a vicious Groundhog Day of bug-fixing and more bug-fixing that can last months.
Think of the software as a boat that has leaks. It's taking on water, and the overall bug count is the level of that water. In come your developers with buckets to bail out the water. But, as they madly bail, the water is still coming in through the leaks. In software development, bugs "leak" into the code through the gaps between us writing or changing code and then testing the code. The more frequently we test the code, the smaller we make those leaks. Good programmers test and retest the code literally hundreds of times a day, and the only way to achieve such short code-test feedback cycles is by automation. A computer can execute 10,000 tests in under a minute. A test team, doing it all manually following the same test scripts, could take weeks or months.
So, as a customer, you should take a very keen interest in the water level in your boat and in how fast water appears to be leaking in. Monitor bug counts. Know how many bugs are still open. Know how many bugs are being reported each week. A good team will achieve very low bug counts - an average of < 1 bug per weekly release (that's about 1 bug every few weeks for an average project), and the number of open bugs will hover as close to zero as makes little difference. Of course, that's just a ballpark. But if you're seeing dozens of bugs reported each week (or hundreds over months), and the average count of open bugs is in the dozens or even hundreds, then you can be sure there's a problem.
You should also take a keen interest in how the pace of development's being sustained. Maybe you started with the developers delivering a dozen features in a weekly release, and by the sixth month that's down to 2-3. Many agile teams that use Burndown Charts, but who don't put enough effort into code quality, see an often alarmingly rapid decline in their ability to make progress. The backlog of your requirements gets, well, backed-up, and you can see often quite clearly that progress is slowing.
As a customer, you can demand low bug counts. Or, rather, you can refuse to accept software that has any known bugs (with the previous caveat about bugs not being change requests, of course).
This can get contentious on teams where the term "bug" is open to interpretation. Having unambiguous executable acceptance tests can help massively in this respect. The software must pass all of the acceptance tests. And it should do nothing other than what's required by the acceptance tests.
I tend to close the stable door thus:
1. If the software fails an acceptance test, that is a bug
2. If the software displays any undesired behaviour (defined by a set of what I call "invariant tests" - tests that it must always pass in any scenario, like, for example, "the system will never respond with an unhandled exception message"), that is a bug
3. If the software passes all of the agreed tests, and you require it to do something that is not defined by any test, that is a new requirement or change request
In any case, it's only a bug when a test is failed. And, since good programmers don't say they're done until they've passed all your tests, expect not to get software that has bugs in it.
Now, a reality a check: when we bandy about requirements like "it will never" or "it will always", we'd be making a pretty grand claim if we said that we'd proved that oour software never does things it's not supposed to, and always does things it is supposed to. Except on very, very simple software, the number of tests we'd have to write to genuinely prove it would be astronomical. Effectively infinite. But we tend to find a law of diminishing returns. 10 tests for a software function may make us 99% sure it works. 100 tests may make us 99.9% sure. 1000 may give us 99.99% assurance, and so on. It is often not possible to achieve 100% assurance, and therefore often impractical to even try. But it is entirely realistic to achieve a level of assurance that's as close to 100% as dammit through everyday good programming practices like developer testing, pair programming (when an extra pair of eyes scrupulously inspects the code as it's being written for potential problems) and the like. And I've learned from experience that we can ratchet up the level of test assurance really quite far for those critical bits of the software without enduring too much pain.
What I'm trying to say is that code that's 99.99% bug-free is entirely achievable, but there'll always be that one-in-a-million freak chance scenario, when ley lines converge and Jupiter is in Uranus etc etc and a bug that's been lurking hidden deep in the nested conditional logic of the application will finally surface. Which is why I leave a tiny bit of wiggle room for development teams; that one-bug-every-once-in-a-blue-moon type of scenario.
So, as a customer, if you demand to see working software every week, and demand that the software is only "working" if it passes your executable acceptance tests, and focus on delivering maximum value each week that makes progress towards your high-level business goals instead of obsessing over sticking to the original plan, and if you expect software to be delivered with almost no bugs (allowing for that once in a blue moon scenario), and if you expect the pace of development to be sustained for months, even years, then you can back a team of programmers into that corner we call "Agile", and they'll either deliver on that or they won't. And, of course, if they don't, then as the customer you can apply the ultimate sanction and then go find a team that can.
Posted 7 years, 6 months ago on July 12, 2011