February 10, 2016
Two Good Reasons To Refactor To Parameterised Tests (Plus, How To Use Algorithmically Generated Test Data)For training workshops and other fun activities, I've been playing around with unit testing frameworks that generate test data algorithmically. For example, generating test inputs for JUnit parameterised tests using JCheck, and noodling with the built-in features of NUnit that do similar good stuff.
I continue to promote the value of refactoring our test code into parameterised tests when the opportunity arises (2 or more tests that are essentially different examples of the same behaviour or rule). Once you have parameterised tests, you can buy yourself quite mind-boggling levels of coverage very cheaply.
To illustrate, this morning I took a bit of date-driven logic that had unit tests, and using JCheck, wrote a custom test data generator that enabled me to run that test over a range of 2 million days (about 5,000 years). It was about 10 extra lines of code, and the tests cases took roughly 0.5 seconds to run. That's 5,000 years of testing for the price of 10 LOC and 0.5 seconds to execute. That seems like a small price to pay. Hence my enthusiasm for parameterised tests.
Parameterising tests also presents me with an opportunity to create test method names that read like a true specification.
Take this little example of a test fixture for a Fibonacci number calculator:
Each individual test in an example of a behaviour. I could refactor this code into parameterised tests to remove some of the duplication, but I can use the test method names to more clearly describe the rules of my Fibonacci calculator:
And now that I have some parameterised tests, I can get extra mileage from them using a tool like JCheck:
With a bit of extra code, I can squeeze out a million test cases (randomly generated), reusing the original test code. If - for some bizarre reason - the Fibonacci calculator was critical to our business, I believe that's a heck of a lot of extra assurance for very little extra effort. Those test cases took about 2.5 seconds to run.
Notice, too, that when we algorithmically generate test inputs, we have to algorithmically generate the expected outputs. This can cause us problems, of course. For that big dollop of extra assurance when we really need it, I'm willing to pay the price of having two implementations that calculate Fibonacci numbers. Redundancy is a tried-and-tested strategy for safety-critical systems.
Where the real danger lies would be if I used the same algorithm to generate the expected outputs. If the algorithm was wrong, the tests would still agree with the actual results. So I've used a different algorithm to generate the expected outputs. The odds of both giving exactly the same wrong answers by two sufficiently different routes are very remote.
Other examples might be using Quicksort to test the results of an implementation of Bubble Sort, or calculating 3D transformations in spherical polar coordinates for an implementation that calculates them in Cartesian coordinates. And so on. The basic idea is, don't test logic using the same logic. It's a similar idea to double-entry book keeping in accounting; if we come to the same answers by two different routes, they're very unlikely to be identically wrong.
So there you have it. Refactoring test code to parameterised tests can not only help to make the tests more self-explanatory, but they can give us a jumping off point to very high levels of test assurance relatively cheaply, when it's really needed.
Posted 2 years, 7 months ago on February 10, 2016