April 29, 2006

...Learn TDD with Codemanship

Complexity Heads vs. Clockwork Heads

We're a funny lot, aren't we? You see a horse race. You see which horse wins. You note all the "relevant" details, including the time the race started, the day of the year, the weather, the alignments of the planets, and so on.

A year later all the key details are to be reproduced precisely. Same day of the year and time of the day, same horses, same riders, same race course, same weather, same alignment of the planets (okay, after just a year that's unlikely - but let's go with it for the sake of argument...) You know which horse won last time. And, according to your notes, every detail is exactly as it was the year before.

So here's the question: would you bet everything you have on last year's winner? Is it a forgone conclusion?

I wouldn't. I suspect you wouldn't, either. You might bet more with your knowledge, but only a madman would bet everything.

A horse race is conceptually simple. You get some horses and they run around in a circle and the first one to cross a line in that circle is the winner. Okay, so the conditions might change from race to race, and a horse might have a bad day, but on the whole horse racing is one of the simpler things in life.

I can think of lots of things far, far more complicated than horse racing. Like developing software, for example. Or running a business. Or transforming a nation's health system. These all sound really rather more complicated to me.

Science has shown us time and again that complexity is a key factor in unpredictability - maybe the key factor. The more complex something gets, the harder it becomes to predict it's future. To the point where most things in life - like horse races - are inherently unpredictable. Not necessarily truly random, mind you: just too complicated to predict accurately. It doesn't matter how many variables we can pin down - the horse, the rider, the course, the date and time, the conditions, the planets, the position, momentum and quantum state of every single particle within a 1 billion mile radius - there are always more variables we can't pin down. Indeed, physics itself has conspired through quantum uncertainty to make the future unknowable, even at the smallest and simplest level.

And so most things in life are things we can never be 100% certain of. Most things in life we're lucky to be 50% certain of. A lot of life is like flipping a coin. And we learn to cope with this uncertainty, just like we learn not to bet everything we own on a single horse. We have evolved to cope with complexity and uncertainty. We have evolved to trust our instincts, and to guard ourselves against the risk of putting all our eggs in one basket. Being omnivores, for example, is an example of dietary spread betting. If food stuff X runs out, we can always survive on food stuff Y. The Giant Panda is not so lucky.

But there's another aspect to our nature that works against us in complex and unpredictable situations: our desire to understand. Specifically, our desire to understand why things happen, and to assign a cause to every phenomenon we see. Our brains have a unique ability - well, we like to think it's unique, anyway - to construct the mental equivalent of clockwork models that explain how the things around us work, and allow us to predict how these things will behave. We have a very accurate clockwork model that tells us why the sun comes up in the morning, and why it sets in the evening. We literally set our clocks by it. The massive success of Newtonian mechanics created a world of steam engines, motor cars, supersonic jet planes and trips to the Moon. It ushered in the industrial revolution and transformed the world and the lives of many people living on it. The management gurus of the industrial age could hardly be blamed for thinking that the clockwork approach could transform organisations (and societies) too.

But, as scientists discovered decades ago, clockwork has its limitations. The limiting factor in clockwork is complexity. Make a machine complicated enough and all of a sudden we can't set our watches by it anymore. It becomes unstable, erratic, dangerously sensitive to to the tiniest perturbations, and ultimately unpredictable.

So we seem to have these two minds: one has evolved to deal with complexity and uncertainty, and one has evolved to build clockwork machines and to expect and demand stability and predictability. Like Wurzel Gummidge, we have our clockwork head which we're supposed to wear when the clockwork approach should be applied, and we have our complexity head which we're supposed to wear when the clockwork model breaks down and things become unstable.

Only we're not so good at changing heads at the right time. Often we try to tackle complex problems with our clockwork head on. It should be noted that it's rarely a problem the other way around, though. Since most things in life are complex and unpredictable, we should keep our complexity heads screwed on most of the time. But we don't. We're kind of comfortable in our clockwork heads. We like predictability. Most people would rather have a guaranteed $1 than a 1:1,000 chance of $1,000. Bizarrely, most managers would rather have guaranteed failure than a small chance of success. They'd rather deliver useless rubbish on time and under budget than deliver something genuinely useful late and over budget.

So we continue to apply clockwork strategies far beyond the boundaries where clockwork strategies apply. And it hurts when we do it. It hurts our pockets when we bet everything on a single horse. It hurts our societies when we try to run complex organisations like they were Swiss watches. It hurts our environment when we try to second-guess what another tonne of CO2 might do to the sea levels around Florida. While we all sit around prevaricating about what we think's going to happen tomorrow, or the day after, or the day after that, we're wasting valuable time and resources trying to know the unknowable and control the uncontrollable.

What we ought to be doing is asking, given that nobody knows what's going to happen for sure, what are we going to do about that? Since we need to expect the unexpected, shouldn't we be building our capability to adapt and to improvise? Shouldn't we be formulating strategies with our complexity heads on?

But what kinds of strategies would our complexity heads come up with? How can we test those strategies? Where is the boundary where clockwork ends and these strategies should begin? How do we know when to change heads? And what happens at the boundary? What happens when things aren't complicated or unpredictable enough, but aren't simple or predictable, either?

And what is it in our nature that helps us to manage with complexity and uncertainty? Does research into human intelligence and the nature of consciousness point to an ability to at least "feel" the unknowable? Is instinct better at predicting the behaviour of complex systems than clockwork ever could be?
Posted 14 years, 8 months ago on April 29, 2006