September 30, 2016
Software Development Doesn't Scale. Dev Culture DoesFor a couple of decades now, the Standish Group have published an annual "CHAOS" reported, detailing the results of surveys taken by IT managers about the outcomes of IT projects.
One clear trend that emerged - and remains as true today as in 1995 - is that the bigger they are, the harder they fall. The risk of an IT project failing outright rises rapidly with project size and cost. When they reach a certain size - and it's much smaller than you may think - failure is almost guaranteed.
The reality of software development is that, once we get above a dozen or so people working for a year or two on the same product or system, the prognosis does not look good at all.
This is chiefly because - and how many times do we need to say this, folks? - software development does not scale.
If that's true, though, how do big software products come into existence?
The answer lies in city planning. A city is made up of hundreds of thousands of buildings, on thousands of streets, with miles of sewers and underground railways and electrical cabling and lawns and trees and shops and traffic lights and etc etc.
How do such massively complex structures happen? Is a city planned and constructed by a single massive team of architects and builders as a single project with a single set of goals?
No, obviously not. Rome was not built in a day. By the same guys. Reporting to one boss. With a single plan.
Cities appear over many, many decades. The suburbs of London were once, not all that long ago, villages outside London. An organic process of development, undertaken by hundreds of thousands of people and organisations all working towards their own unique goals, and co-operating or compromising when goals aligned or conflicted, produced the sprawling metropolis that is now London.
Trillions of pounds has been spent creating the London of today. Most of that investment is nowhere to be seen any more, having been knocked down (or bombed) and built over many times. You could probably create a "London" for a fraction of the cost in a fraction of the time, if it were possible to coordinate such a feat.
And that's my point: it simply isn't possible to coordinate such a feat, not on that scale. An office complex? Sure. A housing estate? Why not? A new rail line with new train stations running across North London? With a few tens of billions and a few decades, it's do-able.
But those big projects exist right the edge of what is manageable. They invariably go way over budget, and are completed late. If they were much bigger, they'd fail altogether.
Cities are a product of many lifetimes, working towards many goals, with no single clear end goal, and with massive inefficiency.
And yet, somehow, London mostly looks like London. Toronto mostly looks like Toronto. European cities mostly look like European cities. Russian cities mostly look like Russian cities. It all just sort of, kind of, works. A weird conceptual cohesion emerges from the near-chaos.
This is the product of culture. Yes, London has hundreds of thousands of buildings, designed by thousands of people. But those people didn't work in bubbles, completely oblivious to each others' work. They could look at other buildings. Read about their design and their designers. Learn a thousand and one lessons about what worked and what didn't without having to repeat the mistakes that earned that knowledge.
And knowledge is weightless. It travels fast and travels cheaply. Hence, St Petersburg looks like the palaces of Versailles, and that area above Leicester Square looks like 19th century Hong Kong.
Tens of thousands of architects and builders, guided by organising principles plucked from the experience of others who came before.
Likewise, with big software products. Many teams, with many goals, building on top of each other, cooperating when it makes sense, compromising when there are conflicts. But, essentially, each team is doing their own thing for their own reasons. Any attempt to standardise, or impose order from above, fails. Every. Single. Time.
Better to focus on scaling up developer culture, which - those of us who participate in the global dev community can attest - scales beautifully. We have no common goal, no shared boss; but, somehow, I find myself working with the same tools, applying the same practices and principles, as thousands of developers around the world, most of whom I've never met.
Instead of having an overriding architecture for your large system, try to spread shared organising principles, like Simple Design and S.O.L.I.D. It's not a coincidence that hundreds of thousands developers use dependency injection to make external dependencies swappable. We visit the same websites, watch the same screencasts, read the same books. On a 10,000-person programme, your architect isn't the one who sits in the Big Chair at head office drawing UMLL diagrams. Your architect is Uncle Bob. Or Michael Feathers. Or Rebecca Whirfs-Brock. Or Barbara Liskov. Or Steve Freeman. Or even me (a shocking thought!)
But it's true. I probably have more influence over the design of some systems than the people getting paid to design it. And all I did was blog, or record a screencast, or speak at a conference. Culture - in this web age - spreads fast, and scales rapidly. You, too, can use these tools to build bridges between teams, share ideas, and exert tacit influence. You just have to let go of having explicit top-down control.
And that's how you scale software development.
April 18, 2016
SC2016 Mini-Project: Code Risk Heat Map
Software Craftsmanship 2016 - Mini-Project
Code Risk Heat Map
Estimated Duration: 2-4 hours
Author: Jason Gorman, Codemanship
Create a tool that produces a "heat map" of your code, highlighting the parts (methods, classes, packages) that present the highest risk of failure.
Risk should be classified along 3 variables:
1. Potential cost of failure
2. Potential risk of failure
3. Potential system impact of failure (based on dependencies)
Use colour coding/gradation to visually highlight the hotspots (e.g., red = highest risk, green = lowest risk)
Also, produce a prioritised list of individual methods/functions in the code with the same information.
PLEASE INSTALL ANY METRICS OR CODE ANALYSIS TOOLS YOU MIGHT WANT TO USSE *BEFORE* THE EVENT
August 17, 2015
The Small Teams ManifestoSo.... grrr.... arrrgh... etc.
Another week, another software project set up to fail right from the start.
What angers me most is that the key factors we know can severely damage our chances of succeeding in delivering software of value are well understood.
The main one is size: big projects usually fail.
We know pretty empirically that the development effort required to deliver software grows exponentially as the software grows. If it takes a team 2 hours to deliver 10 lines of working software, it might take them 4 hours to deliver 20 lines of working software, and 8 hours to deliver 30 lines, and so on.
There's no economy of scale in software development, and we know for a fact that throwing more bodies at the problem just makes things worse. A team of 2 might deliver it in six months. A team of 10 might take a year, because so much of their time will be taken up by being in a team of 10.
The evidence strongly suggests that the probability of project failure grows rapidly as project size increases, and that projects costing more than $1 million are almost certain to run into severe difficulties.
In an evidence-loaded article from 1997 called Less Is More, Steve McConnell neatly sums it all up. For very good reasons, backed up with very good evidence, we should seek to keep projects as small as possible.
Not only does that tend to give us exponentially more software for the same price, but it also tends to give us better, more reliable software, too.
But small teams have other big advantages over large teams; not least is their ability to interact more effectively with the customer and/or end users. Two developers can have a close working relationship with a customer. Four will find it harder to get time with him or her. Ten developers will inevitably end with someone having that working relationship and then becoming a bottleneck within the team, because they're just a proxy customer - seen it so many times.
A close working relationship with our customers is routinely cited as the biggest factor in software development success. The more people competing for the customer's time, the less time everyone gets with the customer. Imagine an XP team picking up user stories at the beginning of an iteration; if there's only one or two pairs of developers, they can luxuriate in time spent with the customer agreeing acceptance tests. If there are a dozen pairs, then the customer has to ration that time quite harshly, and pairs walk away with a less complete understanding of what they're being asked to build (or have to wait a day or two to get that understanding.)
Another really good reason why small teams tend to be better is that small teams are easier to build. Finding one good software developer takes time. Finding a dozen is a full-time job for several months. I know, I've tried and I've watched many others try.
So, if you want more reliable software designed with a close collaboration with the customer at an exponentially cheaper price (and quite possibly in less time, too), you go with small teams. Right?
So why do so many managers still opt for BIG projects staffed by BIG teams?
I suspect the reasons are largely commercial. First of all, you don't see many managers boasting about how small their last project was, and it's the trend that the more people you have reporting to you, the more you get paid. Managers are incentivised to go big, even though it goes against their employer's interests.
Also, a lot of software development is outsourced these days, and there's in obvious incentive for the sales people running those accounts to go as big as possible for as long as possible. Hence massively overstaffed Waterfall projects are still the norm in the outsourcing sector - even when they call it "Agile". (All sorts of euphemisms like "enterprise Agile", "scaling up Agile" etc etc, which we tend to see in this sector more often.)
So there are people who do very well by perpetuating the myth of an economy of scale in software development.
But in the meantime, eye-popping amounts of time and money are being wasted on projects that have the proverbial snowball's chance in hell of delivering real value. I suspect it's so much money - tens of billions of pounds a year in Britain alone, I'd wager - and so much time wasted that it's creating a drag effect on the economy we're supposed to be serving.
Which is why I believe - even though, like many developers, I might have a vested interest in perpetuating The Mythical Man-Month - it's got to stop.
I'm pledging myself to a sort of Small Team Manifesto, that goes something like this:
We, the undersigned, believe that software teams should be small, highly skilled and working closely with customers
Yep. That's it. Just that.
I will use the hashtag #smallteams whenever I mention it on social media, and I will be looking to create a sort of "Small Teams avatar" to use on my online profiles to remind me, and others, that I believe software development teams should be small, highly-skilled and working closely with customers.
You, of course, can join in if you wish. Together, we can beat BIG teams!
August 7, 2015
Taking Baby Steps Helps Us Go FasterMuch has been written about this topic, but it comes up so often in pairing that I feel it's worth repeating.
The trick to going faster in software development is to take smaller steps.
I'll illustrate why with an example from a different domain: recording music. As an amateur guitar player, I attempt to make recorded music. Typically, what I do is throw together a skeleton for a song - the basic structure, the chord progressions, melody and so on - using a single sequenced instrument, like nice synth patch. That might take me an afternoon for a 5 minute piece of music.
Then I start working out guitar parts - if it's going to be that style of arrangement - and begin recording them (muso's usually call this "tracking".)
Take a fiddly guitar solo, for example; a 16-bar solo might last 30 seconds at ~120 beats per minute. Easy, you might think to record it in one take. Well, not so much. I'm trying to get the best take possible, because it's metal and standards are high.
I might record the whole solo as one take, but it will take me several takes to get one I'm happy with. And even then, I might really like the performance on take #3 in the first 4 bars, and really like the last 4 bars of take #6, and be happy with the middle 8 from take #1. I can edit them together, it's a doddle these days, to make one "super take" that's a keeper.
Every take costs time: at least 30 seconds if I let my audio workstation software loop over those 16 bars writing a new take each time.
To get the takes I'm happy with, it cost me 6 x 30 seconds (3 minutes).
Now, imagine I recorded those takes in 4-bar sections. Each take would last 7.5 seconds. To get the first 4 bars so I'm happy with them, I would need 3 x 7.5 seconds (22.5 seconds). To get the last 4 bars, 6 x 7.5 seconds (45 seconds), and to get the middle 8, just 15 seconds.
So, recording it in 4 bar sections would cost me 1m 22.5 seconds.
Of course, there would be a bit of an overhead to doing smaller takes, but what I tend to find is that - overall - I get the performances I want sooner if I bite off smaller chunks.
A performance purist, of course, would insist that I record the whole thing in one take for every guitar part. And that's essentially what playing live is. But playing live comes with its own overhead: rehearsal time. When I'm recording takes of guitar parts, I'm essentially also rehearsing them. The line between rehearsal and performance has been blurred by modern digital recording technology. Having a multitrack studio in my home that I can spend as much time recording in as I want means that I don't need to be rehearsed to within an inch of my life, like we had to be back in the old days when studio time cost real money.
Indeed, the lines between composing, rehearsing, performing and recording have been completely blurred. And this is much the same as in programming today.
Remember when compilers took ages? Some of us will even remember when compilers ran on big central computers, and you might have to wait 15-30 minutes to find out if your code was syntactically correct (let alone if it worked.)
Those bad old days go some way to explaining the need for much up-front effort in "getting it right", and fuelled the artificial divide between "designing" and "coding" and "testing" that sadly persists in dev culture today.
The reality now is that I don't have to go to some computer lab somewhere to book time on a central mainframe, any more than I have to go to a recording studio to book time with their sound engineer. I have unfettered access to the tools, and it costs me very little. So I can experiment. And that's what programming (and recording music) essentially is, when all's said and done: an experiment.
Everything we do is an experiment. And experiments can go wrong, so we may have to run them again. And again. And again. Until we get a result we're happy with.
So biting off small chunks is vital if we're to make an experimental approach - an iterative approach - work. Because bigger chunks mean longer cycles, and longer cycles mean we either have to settle for less - okay, the first four bars aren't that great, but it's the least worst take of the 6 we had time for - or we have to spend more time to get enough iterations (movie directors call it "coverage") to better ensure that we end up with enough of the good stuff.
This is why live performances generally don't sound as polished as studio performances, and why software built in big chunks tends to take longer and/or not be as good.
In guitar, the more complex and challenging the music, the smaller the steps we should take. I could probably record a blues-rock number in much bigger takes, because there's less to get wrong. Likewise in software, the more there is that can go wrong, the better it is to take baby steps.
It's basic probability, really. Guessing a 4-digit number is an order of magnitude easier if we guess one digit at a time.
August 1, 2015
My First, Last & Only Blog Post About #NoEstimatesI've been keeping one eye on the whole #NoEstimates debate on Twitter, and folk have asked me my opinion quite a few times. So here it is.
I believe, very firmly, that the problem with estimation stems from us asking the wrong question.
In fact, this is where many big problems in software development arise; by asking the customer "What software would you like us to build?"
This naturally leads to a shopping list of features, and then a request to know "How much will all that cost and how long will it take?"
If we asked instead "What problem are we trying to solve, and how will we know when we've solved it?" - together with accompanying questions like "When do you need this solution?", "What is a solution worth to you?" and "How much money do you have to invest in solving it?" - we can set out on a different journey.
I believe software development needs to be firmly grounded in reality, and the reality is that it's R&D. At the start, the honest answer to questions like "What features are needed?", "How much will it cost?" and "How long will it take?" is I Don't Know.
Pretending to know the unknowable is what lands us in hot water in the first place. We don't know if we can solve the problem with the budget and the time available.
In the management quest for accounting certainties, though, nobody wants to hear that, and no developer with a mortgage to pay wants to admit it. So we go with the fairy tale instead.
Once we're in the fairy tale - where we know if we deliver this list of features, it will solve the customer's problem, and we can predict how long and how much it will take - it's almost impossible to get out of it. Budgets are committed. Deadlines are agreed. Necks are on chopping blocks.
So, what we do instead, is we wait for the reality to unfold, and then when it no longer matches the fairy tale, there's a major shitstorm of blame and recrimination. Typically, the finger is pointed at everyone and everything except that first mistake; the original sin of software projects: pretending to know the future.
After getting their fingers burned once, the customer's and manager's instinct is to "fix" the problem by "improving" estimates next time around. This is fixing the fairy tale by inventing an even more elaborate fairy tale, to try and disguise the fact that's it's fantasy. This is the management equivalent of sacrificing virgins to make it rain.
The only way out of the estimating nightmare is to call "bullshit" on it, and publicly accept - indeed, embrace - the uncertainty that's inherent in what we're doing.
Yes, you might lose the business if you start out saying "I don't know", but consider that the business you're losing is the same old Death March teams have been suffering for decades. That's not work. That's just passing the time for money.
By all means offer a guess, so the customer can budget realistically. But you must be absolutely 100% crystal clear with them that, at the end of the day, we don't know. We just don't know. It's a punt.
Sell yourself on what you do know. What's your track record as a team? What have you delivered in the past? How much did that cost? How long did that take? And - most importantly, but regrettably least asked - did it work?
When a movie studio hires a director, the director makes no guarantees that this new film will be a commercial success, or that it will cost no more than budgeted, or be completed dead on time. The history of cinema is littered with amazingly good, and often very successful, movies that cost more and took longer than planned. But somehow, James Cameron seems to have no trouble getting movies off the ground. That's because of his track record, not his ability to accurately predict production costs and schedules.
Studios gamble with huge sums of money, and - yes - they do ask for estimates, and things do get hairy when schedules slip and costs overrun, but fundamentally they know what game they're in.
It's time we did, too.
April 25, 2015
Non-Functional Tests Can Help Avoid Over-Engineering (And Under-Engineering)Building on the topic of how we tackle non-functional requirements like code quality, I'm reminded of those times where my team has evolved an architecture that developers taking over from us didn't understand the reasons or rationale for.
More than once, I've seen software and systems scrapped and new teams start again from scratch because they felt the existing solution was "over-engineered".
Then, months later, someone on the new team reports back to me that, over time, their design has had to necessarily evolve into something similar to what they scrapped.
In these situations it can be tricky: a lot of software really is over-engineered and a simpler solution would be possible (and desirable in the long term).
But how do we tell? How can we know that the design is the simplest thing that a team could have done?
For that, I think, we need to look at how we'd know that software was functionally over-complicated and see if we can project any lessons we leearn on to non-functional complexity.
A good indicator of whether code is really needed is to remove it and see if any acceptannce tests fail. You'd be surprised how many features and branches in code find their way in there without the customer asking for them. This is especially true when teams don't practice test-driven development. Developers make stuff up.
Surely the same goes for the non-functional stuff? If I could simplify the design, and my non-functional tests still pass, then it's probable that the current design is over-engineered. But in order to do that, we'd need a set of explicit non-functional tests. And most teams don't have those. Which is why designs can so easily get over-engineered.
Just a thought.
March 1, 2015
Continuous Inspection at NorDevConOn Friday, I spent a very enjoyable day at the Norfolk developer's conference NorDevCon (do you see what they did there?) It was my second time at the conference, having given the opening keynote last year, and it's great to see it going from strength to strength (attendance up 50% on 2014), and to see Norwich and Norfolk being recognised as an emerging tech hub that's worthy of inward investment.
I was there to run a workshop on Continuous Inspection, and it was a good lark. You can check out the slides, which probably won't make a lot of sense without me there to explain them - but come along to CraftConf in Budapest this April or SwanseaCon 2015 in September and I'll answer your questions.
You can also take a squint at (or have a play with) some code I knocked up in C# to illustrate a custom FxCop code rule (Feature Envy) to see how I implemented the example from the slides in a test-driven way.
I'm new to automating FxCop (and an infrequent visitor to .NET Land), so please forgive any naivity. Hopefully you get the idea. The key things to take away are: you need a model of the code (thanks Microsoft.Cci.dll), you need a language to express rules against that model (thanks C#), and you need a way to drive the implementation of rules by writing executable tests that fail (thanks NUnit). The fun part is turning the rule implementation on its own code - eating your own dog food, so to speak. Throws up all sorts of test cases you didn't think of. It's a work in progress!
I now plan, before CraftConf, to flesh the project out a bit with 2-3 more example custom rules.
Having enjoyed a catch-up with someone who just happens to be managing the group at Microsoft who are working on code analysis tools, I think 2015-2016 is going to see some considerable ramp-up in interest as the tools improve and integration across the dev lifecycle gets tighter. If Continuous Inspection isn't on your radar today, you may want to put it on your radar for tomorrow. It's going to be a thing.
Right now, though, Continuous Inspection is very much a niche pastime. An unscientific straw poll on social media, plus a trawl of a couple of UK job sites, suggests that less than 1% of teams might even be doing automated code analysis at all.
I predicted a few years ago that, as computers get faster and code gets more complex, frequent testing of code quality using automated tools is likely to become more desirable and more do-able. I think we're just on the cusp of that new era today. Today, code quality is an ad hoc concern relying on hit-and-miss practices like pair programming, where many code quality issues often get overlooked by pair who have 101 other things to think about, and code reviews, where issues - if they get spotted at all in the to-and-fro - are flagged up long after anybody is likely to do anything about them.
In related news, after much discussion and braincell-wrangling, I've chosen the name for the conference that will be superceding Software Craftsmanship 20xx later this year (because craftsmanship is kind of done now as a meme). Watch this space.
February 13, 2015
Intensive TDD, Continuous Inspection Recipes & Crappy Remote Collaboration ToolsA mixed bag for today's post, while I'm at my desk.
First up, after the Intensive TDD workshop on March 14th sold out (with a growing waiting list), I've scheduled a second workshop on Saturday April 11th, with places available at the insanely low price of £30. Get 'em while they're hot!
Secondly, I'm busy working on a practical example for a talk I'm giving at NorDevCon on Feb 27th about Continuous Inspection.
What I'm hoping to do is work through a simple example based on my Dependable Dependencies Principle, where I'll rig up an automated code analysis wotsit to find the most complex, most depended upon and least tested parts of some code to give early warning about where it might be most likely to be broken and might need better testing and simplifying.
To run this metric, you need 3 pieces of information:
* Cyclomatic Complexity of methods
* Afferent couplings per method
* Test coverage per method
Now, test coverage could mean different things. But for a short demonstration, I should probably keeep it simple and fairly brute force - e.g., % LOC reached by the tests. Not ideal, but in a short session, I don't want to get dragged into a discussion about coverage metrics. It's also a readily-available measure of coverage, using off-the-shelf tools, so it will save me time in preparing and allow viewers to try it for themselves without too much fuss and bother.
What's more important is to demonstrate the process going from identifying a non-functional requirement (e.g., "As the Architect, I want early warning about code that presents a highr risk of being unreliable so that I can work with the developers to get better assurance for it"), to implementing an executable quality gate using available tools in a test-driven manner (everybody forgets to agree tests for their metrics!), to managing the development process when the gate is in place. All the stuff that constitutes effective Continuous Inspection.
At time of writing, tool choice is split between a commercial code analysis tool called JArchitect, and SonarQube. It's a doddle to rig up in JArchitect, but the tool costs £££. It's harder to rig up in SonarQube, but the tools are available for free. (Except, of course, nothing's ever really free. Extra time taken to get what you want out of a tool also adds up to £££.) We'll see how it goes.
Finally, after a fairly frustrating remote pairing session on Wednesday where we were ultimately defeated by a combination of Wi-Fi, Skype, TeamViewer and generally bad mojo, it's occured to me that we really should be looking into remote collaboration more seriously. If you know of more reliable tools for collaboration, please tweet me at @jasongorman.
February 4, 2015
Why Distribution & Concurrency Can Be A Lethal Cocktail For The Unwitting Dev TeamPicture the scene: it's Dec 31st 1990, a small town in upper state New York. I'm at a New Year's Eve party, young, stupid and eager to impress. The host mixes me my first ever Long Island Iced Tea. It tastes nice. I drink three large ones, sitting at their kitchen table, waxing eloquent about life, the universe and everything in my adorable English accent, and feeling absolutely fine. Better than fine.
And then, after about an hour, I get up to go to the bathroom. I'm not fine. Not fine at all. I appear to have lost the use of my legs, and developed an inner-ear problem that's affecting my normally balletically graceful poise and balance.
I proceed to be not-fine-at-all into the bathroom sink and several other receptacles, arguably none of which were designed for the purpose I'm now putting them to.
Long Island Iced Tea is a pretty lethal cocktail. Mixed properly, it tastes like a mildy alcoholic punch with a taste not dissimilar to real iced tea (hence the name), but one look at the ingredients puts pay to that misunderstanding: rum, gin, vodka, tequila, triple sec - ingredients that have no business being in the same glass together. It is a very alcoholic drink. Variants on the name, like "Three Mile Island" and "Adios Motherf***er", provide further clues that this is not something you serve at a child's birthday party.
I end the evening comatose on a water bed in a very hot room. This completes the effect, and Jan 1st 1991 is a day I have no memory of.
Vowing never to be suckered into a false sense of security by something that tastes nice and makes me feel better-than-fine for a small while, I should have known better than to get drawn like a lamb to the slaughter into the distributed components craze that swept software development in the late 1990's.
It went something like this:
Back in the late 1990's, aside from the let's-make-everything-a-web-site gold rush that was reaching a peak, there was also the let's-carve-up-applications-that-we-can't-even-get-working-properly-when-everything's-in-one-memory-address-space-and-there's-no-concurrency-and-distribute-the-bits-willy-nilly-adding-network-deficiencies-distributed-transactions-and-message-queues fad.
This was enabled by friendly technology that allowed us to componentise our software without the need to understand how all the undrelying plumbing worked. Nice in theory. You carve it, apply the right interfaces, deploy to your application server and everything's taken care of.
Except that it wasn't. It's very easy to get and up and running with these technologies, but we found ourselves continually having to dig down into the underlying detail to figure out why stuff wasn't working the way it was supposed to. "It just works" was a myth easily dispelled by looking at how many books on how this invisible glue worked were lying open on people's desktops.
To me, with the benefit of hindsight, object request brokers, remote procedure calls, message queues, application servers, distributed transactions, web services... these are the hard liquor of software development. The exponential increase in complexity - the software equivalent of alcohol units - can easily put unwitting development teams under the table.
I've watched so many teams merrily downing pints of lethal-but-nice-tasting cocktails of distribution and concurrency, feeling absolutely fine - better than fine - and then when it's time for the software to get up and walk any kind of distance... oh dear.
It turns out, this stuff is hard to get right, and the tools don't help much in that respect. They make it easy to mix these cocktails and easy to drink as much as you think you want, but they don't hold your hand when you need to go to the bathroom.
These tools are not your friends. They are the host mixing super-strength Long Island Iced Teas and ruining your New Year with a hangover that will never go away.
Know what's in your drink, and drink in moderation.
January 21, 2015
My Solution To The Dev Skills Crisis: Much Smaller TeamsPutting my Iconoclast hat on temporarily, I just wanted to share a thought that I've harboured almost my entire career: why aren't very small teams (1-2 developers) the default model in our industry?
I think back to products I've used that were written and maintained by a single person, like the guy who writes the guitar amp and cabinet simulator Recabinet, or my brother, who wrote a 100,000 line XBox game by himself in a year, as well as doing all the sound, music and graphic design for it.
I've seen teams of 4-6 developers achieve less with more time, and teams of 10-20 and more achieve a lot less in the same timeframe.
We can even measure it somewhat objectively: my Team Dojo, for example, when run as a one day exercise seems to be do-able for an individual but almost impossible for a team. I can do it in about 4 hours alone, but I've watched teams of very technically strong developers fail to get even half-way in 6 hours.
People may well counter: "Ah, but what about very large software products, with millions of lines of code?" But when we look closer, large software products tend to be interconnected networks of smaller software products presenting a unified user interface.
The trick to a team completing the Team Dojo, for example, is to break the problem down at the start and do a high-level design where interfaces and contracts between key functional components are agreed and then people go off and get their bit to fulfil its contracts.
hence, we don't need to know how the spellcheck in our word processor works, we just need to know what the inputs and expected outputs will be. We could sketch it out on paper (e.g., with CRC cards), or we could sketch it out in code with high-level interfaces, using mock objects to defer the implementation design.
There'll still be much need for collaboration, though. It's especially important to integrate your code frequently in these situations, because there's many a slip 'twixt cup and microservice.
As with multithreading (see previous blog post), we can aim to limit the "touch points" in component-based/service-oriented/microservice architectures so that - as much as possible - each component is self-contained, presents a simple interface and can be treated as a black box by everyone who isn't working on its implementation.
Here's the thing, though: what we tend to find with teams who are trying to be all hifalutin and service-oriented and enterprisey-wisey is that, in reality, what they're working on is a small application that would probably be finished quicker and better by 1-2 developers (1 on her own, or 2 pair programming).
You only get an economy of scale with hiding details behind clean interfaces when the detail is sufficiently complex that it makes sense to have people working on it in parallel.
Do you remember from school biology class (or physics, if you covered this under thermodynamics) the lesson about why small mammals lose heat faster than large mammals?
It's all about the surface area-to-volume ratio: a teeny tiny mouse presents a large surface area proportional the volume of its little body, so more of its insides are close to the surface and therefore it loses heat through its skin faster than, say, an elephant who has a massive internal volume proportional to its surface area, and so most of its insides are away from the surface.
It may be stretching the metaphor to breaking point, but think of interfaces as the surface of a component, and the code behind the interfaces as the internal volume. When a component is teeny-tiny, like a wee mouse, the overhead in management, communication, testing and all that jazz in splitting off developers to try to work on it in parallel makes it counterproductive to do that. Not enough of the internals are hidden to justify it. And so much development effort is lost through that interface as "heat" (wasted energy).
Conversely, if designed right, a much larger component can still hide all the detail behind relatively simple interfaces. The "black box-iness" of such components is much higher, in so much as the overhead for the team in terms of communication and management isn't much larger than for the teeny-tiny component, but you get a lot more bang for your buck hidden behind the interfaces (e.g., a clever spelling and grammar checker vs. a component that formats dates).
And this, I think, is why trying to parallelise development on the majority of projects (average size of business code base is ~100,000 lines of code) is on a hiding to nowhere. Sure, if you're creating on OS, with a kernel, and a graphics subsystem, and a networking subsystem, etc etc, it makes sense to a point. But when we look at OS architectures, like Linux for example, we see networks of "black-boxy", weakly-interacting components hidden behind simple interfaces, each of which does rather a lot.
For probably 9 our of 10 projects I've come into contact with, it would in practice have been quicker and cheaper to put 1 or 2 strong developers on it.
And this is my solution to the software development skills crisis.