December 3, 2007
Mutation Testing Drives A Bus Through the Gaps Between Your Code And Your TestsWhat if my test assertions are too weak? It's pretty easy to achieve high coverage if you make test assertions like "X should not be null" and "Age should be greater than 0", or - and I've actually seen this, folks - Assert.IsTrue(true)
The gap between your test assertions and the actual rules in your code could be so wide that you could drive a proverbial bus through it. You want your tests to fit around your code like a wet suit. Nice and snug. No loose, flapping bits where eels and crabs and squid could sneak in and wreak havoc.
Imagine a unit test that checks that account transactions are added to the account's history collection. Again, I've seen this a lot, where developers assert that a collection's length has increased by one. But that's not the whole story, and you could drive a pretty big bus between that assertion and what the code is actually required to do.
Bugs can hide that gap and remain undetected for longer. The bigger the gap, the more likely it is that bugs will be lurking.
For me, "high coverage" would mean that I would want assurance that my tests fit around all my code nice and snug, on top of the levels of coverage we've already looked at. One way to assess if the gaps are bus-sized is to drive a bus through them and see what happens.
Mutation testing, using tools like Jester does pretty much exactly that. It randomly changes your code - so that it still compiles - and then runs your suite of unit tests to see if any of them pick up these deliberately introduced bugs.
Posted 13 years, 7 months ago on December 3, 2007