January 31, 2007
Metrics Speak - So Be Careful What You Wish ForThe trouble with metrics is that they talk to the people they're measuring. When you're selecting or designing metrics, you should ask yourself "what is this metric telling people to do?"
And it's tricky - very tricky indeed - to design a metric that says what you really meant it to say. For example, a client of mine is, quite rightly, keen to find out how good a fit their product is for their customers' needs. They create reusable software components that their customers build into their own software products. In measuring their output, my client has proposed that they measure the percentage of the components that are used unmodified by their customers.
Their customers may take a component and, if it's not quite what they need, make changes to the component's code to fit their specific requirements. Our metric is telling me to make it harder for them to do that. So maybe we should stop shipping the code and insist on shipping binaries only.
Their customers may use component A, but leave component B on the shelf. The metric is telling me to redesign component A so that I have to use component B as well.
So, what the metric is really saying to me is "stop shipping source code and make every component dependent on as many other components as possible". Now is that what we meant it to say?
On first reading, the metric sounds very reasonable, but a few minutes of gaming and it quickly becomes clear that we've got the wrong metric design for the original goal of creating a product that is a better fit for the customers. I'm sure you can think of plenty of other gaming scenarios - e.g., what if we take all the useful functionality in one component and spread it across the whole set of components?
I'll have a nice, quiet lunch first, and then I'll tell them...
Posted 13 years, 10 months ago on January 31, 2007