May 25, 2016
How Many Bugs In Your Code *Really*?A late night thought, courtesy of the very noisy foxes outside my window.
How many bugs are lurking in your code that you don't know about?
Your bug tracking database may suggest you have 1 defect per thousand lines of code (KLOC), but maybe that's because your tests aren't very thorough. Or maybe it's because you deter users from reporting bugs. I've seen it all, over the years.
But if you want to get a rough idea of how many bugs there are really, you can use a kind of mutation testing.
Create a branch of your code and deliberately introduce 10 bugs. Do your usual testing (manual, automated, whatever it entails), and keep an eye on bugs that get reported. Stop the clock at the point you'd normally be ready to ship it. (But if shipping it *is* your usual way of testing, then *start* the clock there and wait a while for users to report bugs.)
How many of those deliberate bugs get reported? If all 10 do, then the bug count in your database is probably an accurate reflection of the actual number of bugs in the code.
If 5 get reported, then double the bug count in your database. If your tracking says 1 bug/KLOC, you probably have about 2/KLOC.
If none get reported, then your code is probably riddled with bugs you don't know about (or have chosen to ignore.)
Posted 2 years, 7 months ago on May 25, 2016