January 4, 2018
The Impact of Fast-Running Unit Tests Can Be ProfoundThe most common issue I find that holds dev teams back is the testing bottleneck. How long it takes to check that your software still works before rolling it out is a major factor in how often you can do releases.
Consider a rudimentary "maturity model" for checking that our code is fit for release: it's a spectrum of maturity, with the lowest level (let's call it Level 0) being that we don't test it at all and just release it for the users to "test" in the field, and the highest level being testing continuously to try to ensure bugs don't make it into the next 10 minutes, let alone into a production release (call that Level 5).
And there are all levels in between 0 and 5. You might be manually testing before a big release. You might be manually testing iteratively, every couple of weeks. You might be running automated GUI tests overnight. You might have a suite of, say, Cucumber tests that take an hour to run. And so on. Or we might have a mix of 50/50 GUI and unit tests. Or a bunch of "unit" tests that hit databases, making them integration tests. And so on.
There are 3 axes for our maturity model:
x. How effective our tests are at detecting bugs
y. How quickly they run
z. How often we run them
These factors all interrelate. Catching more bugs often means running more tests, which takes longer. And the longer the tests take to run, the less often we're likely to run them.
Together, they answer the question: how long before a bug is likely to be detected?
Teams have to climb the maturity model if they want to release more reliable code more often and reap the business benefits of Continuous Delivery.
They not only have to improve at writing fast-running automated tests, which is a non-trivial skillset that takes years to master, but also at test analysis and design, so the tests they write are asking more of the right questions. (Yes, it's not all about automation.)
Slow-running tests (manual or automated) is a very common bottleneck I find in dev teams, who wrestle with the much higher cost of removing bugs resulting from catching them much later. I've watched teams go round and round in circles trying to stabilise their product to make it acceptable for a major release, sometimes for many months and at a cost of millions. Such costs are typically dwarfed by the knock-on opportunity cost to the business waiting for critical updates to their software and systems.
I also come into contact with a lot of teams who've been writing automated tests for years, but have remained at a fairly low level of testing maturity. Their tests run slow (hours). Their tests miss a bunch of stuff. While these teams don't suffer from prolonged "stabilisation phases" before releases, they still feel like they're wading through treacle to get working code out of the door. High productivity at the birth of a new code base quickly drops to a trickle of new features and a great deal of bug fixing.
The aim for teams striving for sustainable Continuous Delivery is to be able to re-test their code every single time a change is made. Make one change. Run the tests. Fix the one thing you broke if you broke it. Then on to the next baby step.
This means that your tests must run in seconds, not hours, or days, or weeks. And you need high confidence that if you broke the code, a test would show that.
The effect of tightening up test execution can be profound for dev teams, and for the businesses relying on them. I've witnessed some miracles in my time where organisations that were on their knees trying to evolve their legacy systems eventually managed to stand up and walk, even run, as their testing cycles accelerated.
So, for a developer, writing effective fast-running automated tests is a key skill. It's something we should learn early, and continue to improve on throughout our careers.
If you or your team needs to work on your unit testing chops, I've designed a jam-packed 1-day training workshop that'll kickstart things. And this month, bookings are half-price.
Posted 2 years, 3 months ago on January 4, 2018