March 16, 2011
Code Lab - TDD vs. Non-TDDToday I made a third pass at the Code Lab experiment I blogged about recently. I can now reveal my own personal results.
Over 3 iterations, average time taken to complete the kata without TDD was 28m 40s. Average time with TDD was 25m 27s. Without TDD, on average I made 5.7 passes (delivering into acceptance testing). With TDD, on average I made 1.3 passes (in two attempts, they passed first time, in one it took 2 passes.)
Now, this was a baby experiment, of course. And not exactly laborary conditions. But I note a couple of interesting things, all the same.
Firstly, in real life, the feedback loop from acceptance testing - which in this case was automated - would be longer. Much longer. In the experiment I'll be running at SC2011, a delay will be built in. Had I enforced a penalty of, say, 2 minutes, then I'd be adding an average of 11 minutes to the non-TDD times and 2.5 mins to the TDD times.
As has been rightly pointed out by my merry band of Twitter Chums (or "Twums", as I call them, because I'm right-on with all the social media speak) teams that don't do TDD are also unlikely to do automate their acceptance tests (or, for that matter, to even write them down beforehand).
What tends to happen is that the testing phase is done manually, and can be pretty patchy and ineffective compared to comprehensive automated system testing. So bugs don't get spotted and they make it through to a release.
This creates the illusion of speed that defect-prevention critics cite as justification for doing development "quick-and-dirty". Those teams tend not to track the very considerable expense of fixing bugs discovered in production. Nor do they track the expense of making changes without fast and effective regression testing.
This all boils down to the definition of "done". The wider you leave those goalposts, the easier it is to claim you are "done". Had I cut out 50% of the automated acceptance tests, the non-TDD'd code may have passed acceptance testing earlier and non-TDD would have looked faster.
But let's be clear about this, not knowing about the bugs does not mitigate their impact. Any more than not knowing where the mines are buried will make them safe to step on.
The second thing I've noted is that, even though I attempted to refactor in both styles (though, without tests, strictly speaking it's not "refactoring"), the TDD'd code ended up cleaner by a noticeable margin. this could be because TDD has a built-in reminder to refactor often. Alas, I didn't do a screen capture (doh!) so I can't go back and see how often I actually refactored in all of the attempts. It may also be that building up a suite of automated tests adds confidence in refactoring, so I might have approached it a little less gingerly when doing TDD (like walking in sturdy boots vs. bare feet).
These little experimental sessions stop at the first release, and with ineffective acceptance testing it may be possible to conclude that not taking steps to prevent defects seems faster. Even when you can look at the clock and see that it's actually slower, it feels faster.
To extend the experiment might be to simulate 3-4 releases over the course of a day, with patchy acceptance tests, and the missing tests included in a production test suite, with a penalty for production feedback that's an order of magnitude larger than for acceptance testing feedback (e.g., 20 minutes). New requirements would be added with each release, but bugs from previous releases would also have to be fixed when reported.
Posted 3 weeks, 6 days ago on March 16, 2011