December 30, 2005

...Learn TDD with Codemanship

Evolution vs. Eureka Moments

New research into how companies develop products, and the level of their creativity and innovation, suggests that process-oriented methods (TQM, Six Sigma, ISO 9000, CMMi etc) belong on the factory floor and not in the software development workspace.

Putting aside the minor gripe that we don't seem very clear on what "agility" really means, I want to explore this a little further, and see how it might apply to process improvement.

The dichotomy seems to center around order/disorder and complexity, and the role they play in the creative process. (There's that "p" word again!) Creativity seems to stem from the variety of choices we have available from which to synthesise new ideas. People are creatures of habit, and - just as we like our daily routines - we have a natural tendency to slip into comfortable and familiar ways of doing things. We have a seemingly in-built fear of change, especially when the outcome is not certain. Process improvement & TQM are the least frightening paths to a different way of doing things.

They resemble most closely the process of evolution. From reading Richard Dawkins' marvellous book The Blind Watchmaker, I gleaned a useful analogy that - coincidentally - centres around a computer program Dawkins created in (for his sins) BASIC.

Dawkins' program involved the generation of what anthropologist and broadcaster Desmond Morris called "biomorphs". Each biomorph has a small number of "genes", in the form of a single digit. Each "gene" gave the program instructions on the form the biomorph would take in any given generation. Each form was either successful, and therefore allowed to reproduce to create a new generation of biomorphs, or failed and effectively died. The process of biomorph reproduction was basically to make a copy of the successful biomorph's genes, and then a random variation of just a single gene was introduced, making the child slightly different in some way to its parent - but not falling too far from the tree, so to speak.

Over successive generations, the biomorphs became increasingly more complex, just as life did on Earth over hundreds of millions of years. This is much like incremental process improvement (or exploitative innovation - a term coined by a study conducted in Georgia, USA meaning innovation that builds largely on existing knowledge).

If you consider biomorphs with a fixed number of genes, each of which can have a value from 0-9, then what you have is a kind of "biomporh space" - a matrix of all the possible biomorphs dictated by all the available gene combinations. The process of evolution is therefore a walk through biomorph space, moving just one space at a time because we're only varying one gene at a time.

There is an element of risk in evolution - the risk that a random variation will produce an unsuccessful form. In statistical terms, there are safe and unsafe areas of biomorph space, dictated by the constraints used to decide on natural selection. These safe areas tend to be made up of clusters of similar forms, and therefore the closer the apple falls to the tree - the more like its parent the child is - the less risk of the child being unsuccessful. The further away from the parent the child is, the greater the risk of it being an evolutionary failure.

Incremental improvement produces successive generations of processes or products that are mostly like their parents, with small variations in each cycle of improvement. Therefore, if process X is successful in cycle N, then - provided it's near enough to X - process X+1 will be more likely to work in cycle N+1. This is good, yes?

Yes, but it relies on something not changing - or at least, not changing too quickly. The constraints of natural selection in Dawkins' model were fixed. The safe areas remained safe, and the unsafe areas remained unsafe. The problem starts when the rules change, and areas of safety and danger shift rapidly in comparison to the cycle of evolution. Thus, giant pandas got on very nicely, thank you, when their environment was relatively stable and bamboo shoots were usually plentiful. Introduce rapid changes in their environment - like human beings, for example - and, oh dear....

So, in a stable environment it could be argued that incremental improvement is safest. But in an unstable, rapidly changing environment, change may happen too slowly for the biomorphs children to escape the rapidly encroaching danger zones.

The giant panda is a fine example of what might prevent a slowly evolving form from suddendly leaping out of danger to avoid extinction. The key element is choice. If the bamboo shoots run out, I can choose to eat something else. The giant panda cannot. It has evolved slowly, incrementally to be perfectly suited to an environment that is fast disappearing - far faster than it can possibly hope to adapt.

When forms become too highly specialised, too good at a narrow spectrum of things, they risk becoming victims of discontinuous change.

My curiosity is aroused. Do continuous, incremental approaches to performance improvement lead to specialisation? Were Kodak just so damn good at chemical photography that digital photography threatened to pass them by? Was it Kodak's longstanding - and up to then, hugely successful - commitment to continuous incremental improvement that put them in the danger zone when the whole market got turned upside down? Were the likes of Casio, HP and Compaq better placed to compete because they had more choices available to them?

In my search for a definition of "agility", this might be one place worth starting. Is "agility" what humans displayed when we adapted to the cold of the last ice age? Is "agility" the ability to adapt faster than our environment is changing? To be "agile", do we need to be capable of falling further from the tree?

The obvious question arises: why are we more "agile" than giant pandas? We evolved through the same slow, incremental process of cumulative selection. I suggest the answer might lie in what we have that giant pandas don't (asides from a more varied diet), and that's imagination. Through various trials and tribulations, we have evolved the ability to adapt faster than evolution allows, using our creative minds. It is through rapid innovation ("blue sky thinking") that humanity has been able to adapt to discontinuous change.
Posted 15 years, 1 month ago on December 30, 2005