June 29, 2007

...Learn TDD with Codemanship

Gambling On Outsourcing - Metrics Can Help

The lack of transparency in software development makes it very hard to objectively assess the capability of software development organisations.

If I'm hiring a lawyer, I can find out what her track record is - how many cases has she won? How many has she lost? But if I'm hiring a development partner - perhaps based in a foreign country - there's nowhere I can go to find out what an organisation's previous form really is.

As a consultant, I'm often called upon to pass judgement upon the technical abilities of prospective employees. I'm also asked, on occasion, to provide an assessment of a potential development partner.

This is actually a much tougher task than you might think. First of all, most outsourcing suppliers present a very different face in pre-sales than the one that you'll actually end up working with after you've signed on the dotted line. It's no secret that these "skin shops", as they're sometimes called, put their best foot forward, using their most technically - and personally - impressive people to help clinch the deal.

It's also no secret that many of these same companies will try to use the cheapest developers they can get away with when staffing the team who'll actually be writing the code. So you think you're getting The A Team for your business-critical project, but you end up in reality with The Chuckle Brothers - fresh out of University and stuffed full of the terrible nonsense that computer science degrees seem to fill these green-behind-the-ears engenues' heads with (I mean, what are they teaching these people?!)

This makes life hard for the informed gambler. You want to know which horse to bet on, but you don't get to examine the horse who'll actually be running the race, so you have no hard information to base your decision on.

When betting on horses, it's advisable to study the results of their previous races

Of course, you could speak to some of their clients. And that works about as well...

How many managers have you spoken to who will freely admit that they hired the wrong development partner or that their project was somehow anything less than a roaring success? That's why I take case studies with a bucket-full of salt. Everybody papers over the gaping cracks in those situations. And when sales people focus on building close personal relationships, they and their clients tend to be as thick as thieves.

The problem is that we have such a terrible lack of transparency in this industry that we're all free to screw up over and over again, and the next potential victim (sorry, "client") is still none the wiser that, behind the sharp suits and the pre-sales A Team, we're really a serial project-wrecker. Even companies who do have that reputation somehow manage to carry on winning business. And more fools us, I say.

How about asking for evidence of their capability? Each potential vendor would be asked to go away and write a proposal, and included in that must be a set of metrics - performance indicators - drawn from a project built using similar technology and with a similar scope. Their proposal implies that they have the capability to deliver. These metrics are designed to test that assertion and provide a picture of the ballpark they're in in terms of price, performance and quality.

This suite of metrics could include measures of code complexity, defect density, test coverage, coupling and cohesion of classes and packages, and - most tellingly - cost per thousand lines of code.

And what jolly fine fun this wheeze is, too. Not only might it be great fun to see the look on the sales guys' faces as you explain what you want them to do, but it's also a lovely little ethical conundrum for them to lose sleep over.

Of course, they could cheat. That much is apparant. But, and this is the fun part, the figures they give us will influence the service level agreement they'll have to commit to, so they would actually be making a rod for their own backs if they were anything less than realistic about their capability.

What the metrics could reveal is that vendors who look cheap when measured in $/Person/Hour might actually turn out to be very expensive when measured by the cost and quality of what they ultimately deliver. I don't have the proof yet, but I strongly suspect that off-shoring could turn out to be the more expensive option when measured in those terms.

I'm rather hoping this will catch on...

* Of course, for some clients, their only goal is to cut down their monthly outgoings - and good luck to them!
Posted 1 week, 6 days ago on June 29, 2007