August 29, 2006

...Learn TDD with Codemanship

Comparing Project Plans

A few posts ago I proposed a partial solution to the problem of how we can compare two project plans and distinguish "good" from "bad". I used a simple dice game to illustrate the concept of value forecasting, whereby the points that would be scored when throwing a particular number using the sum of two dice are multiplied by the probability of throwing that number.

These value forecasts are not, in of themselves, particularly interesting. It's only when we compare them to each other that we get some useful numbers.

Take for example these two moves in the game:

A. Throw 7 to earn 3 points
B. Throw 11 to earn 5 points

If you were a betting man, which would you put your money on? Move A has a value forecast of (1/6) * 3 = 0.5. Move B has a value forecast of (1/18) * 5 = 0.28. A's forecast is almost twice as high as B's.

These value forecasts represent the balance of risk and reward. The aim of a good plan is to minimise risk and maximise reward. So the better plan would have the higher value forecast.
But it's a little more complicated than that. Consider two plans that are identical in the moves they contain, but differ in the sequence of those moves. For example, using the convention { dice throw required / value, [more moves] } to describe a plan in sequence:

Plan A = { 7/3, 3/4, 6/6, 9/6, 4/9 }

Plan B = { 6/6, 4/9, 7/3, 9/6, 3/4 }

Which of these is the better plan? They contain exactly the same items, but are they equally balanced for risk and reward? Obviously not. Order is very important. We may have scheduled a move to throw a 7 and earn 9 points (which has the highest value forecast), but if we leave it until last, don't we risk never getting around to it within a limited number of dice throws?

This is where it could start to get very complicated indeed if we tried to figure out the probability of throwing a 7 as part of a sequence of dice throws within a fixed number of throws. Luckily, we don't need to. Again, we're only interested in the relative picture. Moves that are scheduled earlier have a higher probability of being completed. It makes sense, really. The earlier I start a move, the more throws of the dice I have to get the number I need.

So the better plan is the one with the higher value forecasts scheduled earlier. Can we propose a simple metric that will indicate the extent to which value is "front-loaded" in a plan? Well, as a matter of fact we can. Essentially, we're measuring the sortedness of the plan in terms of descending value forecast. So item 1 should be greater than or equal to item 2. And item 2 should be greater than or equal to item 3. And item N should be greater than or equal to N + 1 (if N isn't the last in the sequence). If the difference between N and N + 1 is zero or more, then they are ordered correctly. But if N + 1 > N, then the difference will be a negative number. So all we have to do is tot up the total of the differences for each N and N + 1, and then we can normalise that number by divding by the total of the absolute values of the differences.

Phew! Still with me? Let's illustrate for Plan A and Plan B:

Plan A's value forecast = { (1/6) * 3, (1/18) * 4, (5/36) * 6, (1/9) * 9, (1/12) * 9 } = { 0.5, 0.22, 0.83, 1, 0.75 }

Plan B's value forecast = { 0.83, 0.75, 0.5, 1, 0.22 }

Sortedness of Plan A = ((0.5 - 0.22) + (0.22 - 0.83) + (0.83 - 1) + (1 - 0.75)) / ( |0.5 - 0.22| + |0.22 - 0.83| + |0.83 - 1| + |1 - 0.75| ) = -0.19

Sortedness of Plan B = ((0.83 - 0.75) + (0.75 + 0.5) + (0.5 - 1) + (1 - 0.22)) / ( |0.83 - 0.75| + |0.75 + 0.5| + |0.5 - 1| + |1 - 0.22| ) = 0.38

Plan B is significantly more sorted than Plan A, and is therefore the better of the two plans - relatively speaking.

So how do I apply this to real project plans? It's easy to figure out the odds of throwing 7 with two dice. But how does that translate into Function Points, or Ideal Engineering Days, or EASY, MEDIUM, HARD estimates? The trick is to think back to the estimates we gave in the dice game. We actually estimated how many throws of the dice it might take to get the desired outcome. So we figure that when we estimate that it could take 10 throws to get a 9, and 7 throws to get a 7, what we're actually saying is that we believe throwing 9 with two dice is 70% as probable as throwing 7, which is why we think we need those extra throws.

And if a developer estimates that user story A might take them 3 days, and user story B might take 2 days, what she's really saying is that she believes story A is 67% as probable as story B. She could potentially complete story A in 2 days, but it's less likely. We're interested in relative probability here, because we're only really interested in the sortedness of each plan.

Taking the example of user stories, I just use the inverse of the estimate to create a measure of relative probability. So 3 Ideal Engineering Days becomes 1/3, and 2 Ideal Engineering Days becomes 1/2. It works just as well for other kinds of estimates.

You do, of course, need an estimate of relative value, too. This should come from the customer on your project, since we developers have some very strange ideas about what constitutes real value. This could be a number between 1 and 10, or 1 and 5, or even 1 and 3. (1 and 2 would be pushing it!). Or you could assign values to the old MOSCOW formula (e.g., MUST HAVE = 4 points, SHOULD HAVE = 3, COULD HAVE = 2, WILL NOT HAVE = 1).

The important thing to remember is that we are interested in the balance of risk and reward (maximise reward, minimise risk) and especially in the sortedness of each plan. Anecdotally, I can tell you that this the sortedness of project plans is a major factor in success when playing the dice game. I suspect it's also a major factor in real projects. I aim to build statistical proof for the game - using a computer simulation of a very large number of randomly generated plans. I would also be interested in applying it to real projects plans - and not just for software projects. If it's as key to success as I believe, then that places huge emphasis on the risk-reward balance, and on the planning process. In Agile planning - and that's the way to do it, folks - it elevates the customer into a pivotal role that can make all the difference to the project's chances of success, since identifying value is an essential part of the equation.

But what about dependencies? What if feature A requires us to complete feature B first? What if A and B are being done by different project teams, with different - but dependent - plans? How do we scale this model up? Well, that's a whole other can of worms that I'll come back to at a later date.
Posted 14 years, 11 months ago on August 29, 2006