November 13, 2007
How Good Are Your Unit Tests?One question that really bugs me is: what is the average effectiveness of unit tests?
Why do I want to know this? Well, because people are always asking me, that's why. I show them Test-driven Development as it is written (so mote it be), and demonstrate 100% code coverage, 100% path coverage, 100% constraint coverage and then use mutation testing to demonstrate just how really very bloody good my unit tests are. (And they are very bloody good indeed. Oh yes.)
And it's usually around this point that someone in the audience puts their hand up and innocently asks:
"So how far do people go on average projects?"
(Because that's the level of ambition I'm up against, godammit! 90% of developers just want to not be worse than average.)
And the truth is, I have no idea what the average level of unit test effectiveness is. None whatsoever. Diddly.
You see, I'm a bit like the Queen. She thinks that the world smells like wet paint, because wherever she goes that's what she can smell.
I think that all teams achieve pretty high levels of unit test effectiveness, because wherever I go that's what I expect from the teams I work with. And they hate to let me down. No, they really do.
So I have a fairly skewed perception of what goes on out there among the hordes of the great unwashed. I hear - like those tales of sea monsters and 9ft tall ape men that explorers brought back with them from exotic, far off lands - that there are teams out there who have no unit tests at all. None. Not one.
But such a team would surely not dare subject themselves to my piercing gaze, and therefore - for exactly the same reasons that Queen Victoria didn't believe that lesbians existed - I choose to dismiss these as the wild imaginings of a depraved lunatic.
So how about your unit tests? Do they cover every line of code? Do they cover every path through the code? Do they cover every combination of logical constraints in the code? If you randomly noodle with your code, do your tests always pick up these "mutations"? How often do you run your unit tests? How much change happens between each test run?
And do you even care?
Feel free to let me know?
Posted 2 weeks, 6 days ago on November 13, 2007