June 27, 2018
Team CraftWe're a funny old lot, software developers.
90% of us are working on legacy code 90% of the time, and yet I can only think of one book about working with legacy code that's been published in the last 20 years.
We spend between 50%-80% of our time reading code, and yet I can only think of a couple of books about writing code that's easier to understand that have ever been published.
We have a problem with our priorities, it would seem. And maybe none more so than in the tiny amount of focus we place on how we work together as teams to get shit done.
Our ability to work together, to communicate, to coordinate, to build shared undersanding and reach shared decisions and to make stuff happen - I call it Team Craft - rarely gets an airing in books, training courses and conferences.
In my TDD workshop, we play a little game called Evil FizzBuzz. If you've applied for a developer job in recent years, you may well have been asked to do the FizzBuzz coding exercise. It's a trivial problem - output a list of integers from 1 to 100, replace any that are divisible by 3 with "Fizz", any that are divisible by 5 with "Buzz", and any that are divisible by 3 and 5 with "FizzBuzz". Simple as peas.
I made it "evil" by splitting the rules up and requiring that individual pairs only work on code for their rule. (e.g., they can only work on generating a sequence from 1..100, or only on replacing numbers with Fizz, or Buzz etc).
They must coordinate their efforts to produce a single unified solution that passes my customer acceptance test - a complete comma-delimited sequence of the required length, with the numbers, the Fizzes, the Buzzes and FizzBuzzes in the right place. This is an exercise - superficially - in Continuous Integration. But, it turns out, it exercises far more than that.
An average developer can complete FizzBuzz in less than 30 minutes. An average team can't complete it in under an hour. No, seriously. 9 out of 10 teams who attempt it don't complete it. Go figure!
Watching teams attempt Evil FizzBuzz is fascinating. The first observation I've made - from dozens of teams who've tried it - is that the individual technical skills of the developers on the team appears to have little bearing on how they'll fare.
FizzBuzz is easy. It doesn't require strong Code Fu. And yet, somehow, it defeats 90% of teams. There must be something else at play here; some other skillset outside of coding and unit testing and refactoring and Git and wotnot that determines how a team will perform.
Over the years since it was introduced, I've developed an instinct for which teams will crack it. I can usually tell within the first 10 minutes if they're going to complete Evil FizzBuzz within the hour, just by looking at the way they interact.
Here are the most typical kinds of rocks I've seen teams' ships dashed on trying to complete Evil FizzBuzz.
2. Priorities - the team spends 30 minutes discussing the design, and then someone starts to think about setting up the GitHub repository and a CI server.
3. Forgetting They're In a Team - I see this one a lot. For example, someone sets up a repository, then forgets to invite the rest of the team to contribute to it. Or - and this is my favourite - someone writes their code in a totally different set of project files, only realising too late that their bit isn't included in the end product. To coordinate efforts in such a small solution space, developers need to be hyper-aware of what the rest of the team are doing.
4. Trying To Win The Argument Instead Of The Game - as with 1-3, this is also very common on development teams. We get bogged down in trying to "win" the debate about what language we should use or whether we should use the Chain of Responsibility design pattern or go for tabs or spaces, and completely lose sight of what we're setting to achieve in the first place. This effect seems to escalate the more technically strong individuals on the team are. Teams of very senior developers or software architects tend to crash and burn more frequently than teams of average developers. We've kind of made this rod for our own backs, as a profession. Career advancement tends to rely more on winning arguments than achieving business goals. Sadly, life's like that. Just look at the people who end up in boardrooms or in government: prepared for leadership in the debating societies of our top schools and colleges. Organisations where that isn't part of the culture tend to do much better at Evil FizzBuzz.
5. All Talk, No Code, No Pictures - the more successful teams get around a whiteboard and visualise what they're going to do. They build a better shared understanding, sooner. The teams who stand around in a circle talking about it invariably end up with every pair walking away with a different understanding, leading to the inevitable car crash at the end. It's especially important for each pair to understand how their part fits in with the whole. The teams that do best tend to agree quickly on how the parts will interact. I've known this for years: the key to scaling up development is figuring out the contracts early. Use of stubs and mocks can help turn this into an explicit executable understanding. Also, plugging their laptops into the projector and demonstrating what they intend is always an option - but one that few teams take up. To date, no team has figured out that Mob Programming is allowed by the rules of the exercise, but a couple of teams came close in their use of the available technology in the room.
6. Focus On Plans, Not Goals - It all seems to be on track; with 5 minutes to go the team are merging their respective parts, only to discover at the very last minute that they haven't solved the problem I set them. Because they weren't setting out to. They came up with a plan, and focused on executing that plan. The teams that crack it tend to revisit the goals continually throughout the exercise. Does this work? Does this work? Does this work? Equally, teams who get 30 minutes in and don't realise they've used 50% of their time show a lack of focus on getting the job done. I announce the time throughout, to try and make them aware. But I suspect often - when they've got their heads down coding and are buried in the plan - they don't hear me. The teams who set themselves milestones - e.g. by 20 minutes we should have a GitHub repository with everyone contributing and a CI server showing a green build so we can start pushing - tend to do especially well.
From long experience on real teams, I've observed relationships between these elements of Team Craft. Teams that lack clear objectives tend to consume themselves with internal debate and "pissing contests". It also tends to make prioritising nigh-on impossible. Tabs vs spaces matters a lot more when you think you have infinite time to debate it. Lack of visualisation of what we're going to do - or attempt to do - tends to lead to less awareness of the team, and less effective coordination. And all of these factors combined tend to lead to an inability to make shared decisions when they're needed.
But before you conclude from this that the individual technical skills don't matter, I need to tell you about the final rule of Evil FizzBuzz: once the build goes green for the first time, it must not go red again. Breaking the build means disqualification. (Hey, it's an exercise in Continuous Integration...)
A few teams get dashed on those rocks, and the lesson from that is that technical discipline does matter. How we work together as teams is crucial, but potentially all for nought if we don't take good care of the fundamentals.
January 21, 2018
Delegating "Junior" Development Tasks. (SPOILER ALERT: It doesn't work)When I first took on a leadership role on a software development team 20 years ago, from the reading I did, I learned that the key to managing successfully was apparently delegation.
I would break the work down - GUI, core logic, persistence, etc - and assign it to the people I believed had the necessary skills. The hard stuff I delegated to the most experienced and knowledgeable developers, The "easy" stuff, I left to the juniors.
It only took me a few months to realise that this model of team management simply doesn't work for software development. In code, the devil is in the detail. To delegate a task, I had to explain precisely what I wanted that code to do, and how I wanted it to be (in terms of coding standards, our architecture, and so on).
If the task was trivial enough to give to a "junior" dev, it was usually quicker to do it myself. I spent a lot more time cleaning up after them than I thought I was saving by delegating.
So I changed my focus. I delegated work in big enough chucks to make it worthwhile, which meant it was no longer "junior" work.
Looking back with the benefit of 20 years of hindsight, I realise now that delegating "junior" dev tasks is absurd. It's like a lead screenwriter delegating the easy words to a junior screenwriter. It would also probably be a very frustrating learning experience for them. I'm very glad I never went through a phase in my early career of doing "junior" work (although I probably wrote plenty of "junior" code!)
The value in bringing inexperienced developers in to a team is to give them the opportunity to learn from more seasoned developers. I got that chance, and it was invaluable. Now, I recommend to managers that their noobs pair up with the old hands on proper actual software development, and allow for the fact that it will take them longer.
This necessitates - if you want the team to be productive as a whole - that the experienced developers outnumber the juniors. Actually, let's not call them that. The trainees.
Over time - months and years - the level of mentoring required will fall, until eventually they can be left to get on with it. And to mentor new developers coming in.
But I still see and hear from many, many people who are stuck in the hell of a Thousand Junior Programmers, where senior people - often called "architects" - are greatly outnumbered by people still wet behind the ears, to whom all the "painting by numbers" is delegated. This mindset is deeply embedded in the cultures of some major software companies. The result is invariably software that's much worse, and costs much more.
It also leads to some pretty demoralised developers. This is not the movie industry. We don't need runners to fetch our coffee.
ADDENDUM: It also just occurred to me, while I'm recalling, that whenever I examined those "junior" dev tasks more closely, their entire existence was caused by a problem in the way we were doing things (e.g., bugginess, lack of separation of presentation and logic, duplication in data access code, etc). These days, when it feels like "grunt" work - repetitive grind - I stop and ask myself why.
November 21, 2017
What Can We Learn About Dev Team Performance from Distributed System Design?There are all sorts of analogies for software development teams ("teams are like a box of chocolates" etc), and one I find very useful is to picture them as distributed information processing systems.
Each worker process (person) has a job to do. Each job has information inputs and outputs. Each job requires data (knowledge). And the biggest overhead is typically not the time or effort required to process the information in each process, but the communication overhead between processes. This is why, as we add more people (worker processes), performance starts to degrade dramatically.
But, ironically, most of the available thinking about dev team performance focuses on optimising the processes, and not the communication between the processes.
Following this analogy, if we apply performance patterns for distributed computing to dev teams, we can arrive at some basic principles. In particular, if we seek to minimise the communication overhead without harming outcomes, we can significantly improve team performance.
Processes communicate to share data. The less data they need to share, the lower the communication overhead. And this is where we make our classic mistake; the original sin of software development, if you like.
Imagine our processes can act on data at a low level, but all the conditional logic is executed by external management processes that coordinate the workflow. So every time a worker process needs a decision to be made, it must communicate with a management process and wait for a response. Yes, this would be a terrible design for a distributed system. And yet, this is exactly how most dev teams operate. Teamwork is coordinated at the task level - the level of details. A more performant design would be to give individual worker processes a goal, and then let them make any decisions required to achieve that goal. Tell them the what and then let them figure out the how for themselves.
And I can attest from personal experience that dev teams that empower their developers to make the technical decisions perform much better.
But, as any developer who's worked on a team will tell you, there still needs to be coordination beween developers to reach consensus on the how. A classic example is how many teams fail to reach a consensus on how they implement model-view-controller, each one coming up with their own architecture.
Often, the amount of coordination and consensus needed can be front-loaded. Most of the key technical decisions will need to be made in the first few weeks of development. So maybe just take the hit and have a single worker process (the whole team) work together to establish baseline data, a skeleton of technical and logical architecture, of technical standards and common protocols (e.g., check-in etiquette) on which everyone can build mostly autonomously later. I've been doing this with teams since the days affordable portable data projectors became available. These days they call it "mob programming".
And, of course, there's unavoidably one shared piece of mutable data all processes have no choice but to act on in parallel: the code.
Much has been said on the subject of distributed version control of source code, most of focusing on entirely the wrong problem. Feature Branching, for example, tries to achieve more autonomy between developers by isolating their code changes from the rest of the team for longer. If every check-in is a database transaction (which it is - don't say it isn't), then this is entirely the wrong lever to be pulling on to speed things up. When we have many processes committing transactions to shared database, making the transactions bigger and longer won't speed the system up, usually. We're aiming not to break the data. The only way to be sure of that is to lock the data while the transaction's being written to the database. (Or to partition the data so that it's effectively no longer shared - more on that in a moment.)
To avoid blocking the rest of the worker processes, we need transactions to be over as soon as possible. So our check-ins need to be smaller and more frequent. In software development, we call this Continuous Integration.
It also helps if we split the shared data up, so each blob of data is accessed by fewer worker processes. More simply, the smaller the shared codebase, the less of a CI overhead. Partition systems into smaller work products.
But, just as partitioning software systems into - say - microservices - can increase the communication overhead (what were once method calls are now remote procedure calls), partitioning shared codebases creates a much greater overhead of communication between teams. So it's also vitally important that the various codebases are as decoupled as possible.
I rail against developers who add third-party dependencies to their software for very simple pieces of work. I call it "buying the Mercedes to use the cigarette lighter". In the world of microservices, system component needs to be largely responsible for doing their own work. Only add a dependency when the development cost of writing the code to do that bit of work is significantly greater than the potential ongoing communication overhead. You have to be merciless about minimising external dependencies. Right now, developers tend to add dependencies far too lightly, giving the additional costs little or no thought. And our tools make it far too easy to add dependencies. I'm looking at you, Maven, NuGet, Docker etc.
So, to summarise, here are my tips for optimising the performance of development teams:
1. Give them clear goals, not detailed tasks
2. Make developers as autonomous as possible. They have the technical data, let them make the technical decisions.
3. Accept that, initially, parallelism of work will be very difficult and risky. Start with mob programming to establish the technical approach going forward.
4. Small and frequent merging of code speeds up team performance. Long-lived code branches tend to have the reverse effect to that intended.
5. Partition your architectures so you can partition the code.
6. Manage dependencies between codebases ruthlessly. Duplicated logic can be cheaper to live with than inter-team communication.
November 7, 2017
Why Agile's Not For MeThere's a growing consensus among people who've been involved with Agile Software Development since the early (pre-Snowbird) days that something is rotten in the state of Agile.
Having slowly backed out of the Agile movement over the last decade or more (see my semi-jocular posts on Post-Agilism from 2007), I approach the movement as a fairly skeptical observer.
Talking with folk both inside and outside the Agile movement - and many with one foot in and one foot out - has highlighted for me where the wheels came off, so to speak. And it's a story that's by no means unique to Agile Software Development. Like all good ideas in software, it's never long before the money starts taking an interest and the pure ideas that it was founded on get corrupted.
1. Too Much Emphasis On Working Software
But, arguably, Agile Software Development was fundamentally flawed straight out of the gate (or straight out of the ski resort, more accurately). If I look for a foundation for Agile, it clearly has its roots in the concept of evolutionary software development. Evolution is a goal-seeking algorithm that searches for an optimum solution by iterating designs rapidly - the more rapidly the better - and feeding back in what we learn with each iteration to improve our solution.
There are two key words in that description: iterating and goal-seeking. There is no mention of goals in the original Agile Manifesto. The manifesto stipulates that the measure of progress is "working software". It does not address the question of why we should build that software in the first place.
And so, many Agile teams - back in the days when Extreme Programming was still a thing - focused on iterating software designs to solve poorly-defined - or not defined at all, let's face it - business problems. This is pretty much guaranteed to fail. But, bless our little cotton socks, because we set ourselves the goal of delivering "working software", we tended to walk away thinking we'd succeeded. Our customers... not so much.
This was the crack in Agile through which the project office snuck back in. (More about them later.)
2. Not Enough Emphasis On Working Software
As Agile evolved as a brand, more and more of us tried to paint ourselves in the colours of management consultants. Because, let's be frank, that's where the big bucks are. People who would once have been helping you to fix your build script were now suddenly self-professed McKinsey-style business gurus telling you how to "maximise the flow of value" in your enterprise, often to comic effect because nobody outside of the IT department took us seriously.
And then, one day - to everyone's horror - somebody outside the IT department did start taking us seriously, and suddenly it wasn't funny any more. Agile "crossed the chasm", and now people were talking about "going Agile" in the boardroom. Management and business magazines now routinely run articles about Agile, typically seeking input from people I've certainly never heard of who are now apparently world-leading experts. None of these people has heard of Kent Beck or Ward Cunningham or Brian Marick or any other signatory of the original Agile Manifesto. Agile today is very much in the hands of the McKinseys of this world. A classic "be careful what you wish for" moment for those from the IT department who aspired to be dining at the top table of consulting.
Agile's now Big Business. And the business of Agile is going BIG. Like every good and pure thing that falls into the hands of management consultants, Agile has mutated from a small, beautiful bird singing a twinkly tune to a bloated enterprise albatross with a foghorn.
3. We Didn't Nuke The Project Office From Orbit To Be Sure
I'm often found hanging around on street corners muttering to myself incoherently about the leadership class. Well, it's good to have a hobby.
Across the world - and especially in the UK - we have a class of people who have no actual practical skills or specific expertise to speak of, but a compelling sense of entitlement that they should be in charge, often of things they barely understand.
In the pre-Agile Manifesto world, IT was ruled by the leadership class. There was huge emphasis on processes, driven by the creation of documents, for the benefit of people who were neither using the software or writing it. This was a non-programmer's idea of what programming should be. In the late 1990's, the project office was the Alpha and the Omega of software and systems development. People who'd never written a line of code in their lives telling people who do it day-in and day-out how it should be done.
Because, if they let programmers make the decisions, they'll do it wrong!!! And, to be fair, we often did do it wrong. We built the wrong thing, and we built it wrong. It was our fault. We let the project office in by frequently disappointing our customers. But their solution just meant that we still did it wrong, only now we did it wrong on a much grander scale.
And just as we developers kidded ourselves that, because we delivered working software, that meant we had succeeded, managers deluded themselves that - because the team followed the prescribed processes - the customer's needs had been met.
Well, nope. We ticked the boxes while the customer got ticked off.
It turns out that the working relationship between software developers and their customers is, and always has been, the crux of the problem. Teams that work closely and communicate effectively with customers tend to build the right thing, at least. There's no process, standard or boxes-and-arrows diagram that can fix a dysfunctional developer-customer relationship. CMMi all you like. It doesn't help in the end. And, as someone who specialised on software process engineering and wore the robes and pointy hat of a Chief Architect, I would know.
The Agile Manifesto was a reaction to the Big Process top-heavy approach that had failed us so badly in the previous decades. Self-organising teams should work directly with customers and do the simplest things to deliver value. Why write a big requirements specification when we can have a face-to-face conversation with the customer? Why create a 200-page architecture document when developers can just gather round a whiteboard when they need to talk about design?
XP in particular seemed to be a welcome death knell for value-sucking Plan-Driven, Big Architecture, Big Process roles. It was the end for those projects like the one where I was the only developer but for some reason reported to three project managers, spending a full day every week travelling the country helping them to revise their constantly out-of-date Gantt charts.
And, for a while, it was working. The early noughties was a Golden Age for me of working on small teams, communicating directly with customers, making the technical decisions that needed to be made, and doing it our way.
But the project office wasn't going to just slink away and die in a corner. People with power rarely relinquish it voluntarily. And they have the power to make sure they don't need to.
Just as before, we let them back in by disappointing our customers. A lack of focus on end business goals - real customer needs - and too much focus initially on the mechanics of delivering working software created the opportunity for people who don't write code to proclaim "Look, the people writing the code are doing Agile wrong!"
And, again, their solution is more processes, more management, more control. And, hey presto, our 6-person XP projects transformed into beautiful multi-team Enterprise Agile butterflies. Money. That's what I want.
Back To Basics
Agile today is completely dominated by management. It's no longer about software development, or about helping customers achieve real goals. It's just as top-heavy, process-oriented and box-ticky as it ever was in the 1990s. And it's therefore not for me.
Working closely with customers to solve real problems by rapidly iterating working software on small self-organising teams very much is, still. But I fear the word for that has had its meaning so deeply corrupted that I need to start calling it something else.
How about "software development"?
December 8, 2016
What Do I Think of "Scaled Agile"?People are increasingly asking me for my thoughts on "scaled agile", so I though I'd take a quiet moment to collect my thoughts in one place.
Ever since that fateful meeting in Snowbird, Utah in 2001, some commercially-minded folk have sought to "scale up" the Agile brand so it can be applied to large organisations.
I'll give you an example of the kind of organisation we're talking about: a couple of years ago I was invited into a business that had about 150 teams of developers all effectively working on the same system (or different versions of the same system). I was asked to put together a report and some recommendations on how TDD could be adopted across the organisation.
The business in question was peppered throughout with Agile consultants, Scrum Masters, Lean experts, Kanban experts, and all manner of Agile flora and fauna.
Teams all used user stories, all had Scrum or Kanban boards, all did daily stand-ups, and all the paraphernalia we associate with Agile Software Development.
But if there was one thing they most definitely were not, it was agile. Change was slow and expensive. There was absolutely no sense of overall direction or control, or of an overall picture of progress. And the layers of "scaled Agile" the managers had piled on top of all that mess was just making things worse.
It's just a fact of life. Software development doesn't scale. Once software projects go above a certain size (~$1 million), chaos is inevitable, and the best you can hope for is an illusion of control.
And that, in my considerably wide experience of organisations of all sizes attempting to apply agile principles and practices, is all that the "scaled agile" methods can offer.
I've seen some quite well-known case studies of organisations that claim to be doing agile at scale with my own eyes, and they just aren't. 150 teams doing scaled agile, it turns out, is just 150 teams doing their own thing, and on a surface level making it look like they're all following a common process. But they're still dogged by all the same problems that any organisation trying to do software development at scale is dogged by. You can't fix nature.
Instead, you have to acknowledge the true nature of development at scale; that these are highly complex systems, not conducive to overall top-down control, from which outcomes organically emerge, planned or not.
Insect colonies do not follow top-down processes. A beehive isn't command and control, even if it might look to the casual observer that there's a co-ordinated plan they're all following. What's actually happening is that individual bees are responding to broadcast messages ("goals") about where the pollen, or the threat, can be found, and then they respond according to a set of simple internalised rules to that message, co-operating at a local level so as not to bump into each other.
In software development teams, the internalised rules - often unspoken and frequently at odds with the spoken or written rules - and the interactions at a local level determine what the outcomes the system will produce. We call these internalised rules "culture". Culture is often simple, but buried so deep that changing the culture can take a long time.
In particular, culture round the way we communicate and collaborate tends to steer the ship in particular directions, regardless of which direction you point the rudder. This is a property of complex adaptive systems called "strange attractors".
Complex systems have a property called "homeostatis" - a tendency, when disturbed, to iteratively revert back to their original dynamic state, as determined by their strange attractors. Hence, a heartbeat can rise to more than 150 bpm, but will eventually return to a resting heart rate of about 70-80 bpm.
We can apply external stimuli to a system to try and change the way it performs, but the intrinsic properties of the agents within that system, and particularly their interactions, will ultimately determine the outcome.
Methods like SAFe, LeSS and DAD are attempts to exert top-down control on highly complex adaptive organisations. As such, in my opinion and in the examples I've witnessed, they - at best - create the illusion of control. And illusions of control aren't to be sniffed at. They've been keeping the management consulting industry in clover for decades.
The promise of scaled agile lies in telling managers what they want to hear: you can have greater control. You can have greater predictability. You can achieve economies of scale. Acknowledging the real risks puts you at a disadvantage when you're bidding for business.
But if you really want to make a practical difference in a large software development organisation, the best results I've seen have come from focusing on the culture: what do people really value? What do people really believe? What are people's real habits? What do they really do under pressure?
You build big, complex products out of small, simple parts. The key is not in trying to exert control over the internal workings of each part, but to focus on how the parts - and the small, simple teams who make them - interact. Each part does a job. Each part will depend on some of the other parts. An overall architecture can emerge by instilling a set of good, practical organising principles across the teams - a design culture, like we have in the architecture of buildings, for example. The teams negotiate with each other to resolve potential conflicts, like motorists on our complex road systems trying to get where they need to go without bumping into each other.
Another word for this is "anarchy". I advice you not to use it in client meetings. But that is what it is.
I think it's very telling that so many of the original signatories of the Agile Manifesto have voiced scepticism - indeed, in some cases been very scathing - of "scaled agile". The way I see it, it's the precise opposite of what they were trying to tell us at Snowbird.
This is why, as a professional, I've invested so much time in training and coaching developers and teams, rather than in management consulting. I certainly engage with bosses, but when they ask about "scaled agile" I tell them what I personally think, which is that it's a mirage.
September 30, 2016
Software Development Doesn't Scale. Dev Culture DoesFor a couple of decades now, the Standish Group have published an annual "CHAOS" reported, detailing the results of surveys taken by IT managers about the outcomes of IT projects.
One clear trend that emerged - and remains as true today as in 1995 - is that the bigger they are, the harder they fall. The risk of an IT project failing outright rises rapidly with project size and cost. When they reach a certain size - and it's much smaller than you may think - failure is almost guaranteed.
The reality of software development is that, once we get above a dozen or so people working for a year or two on the same product or system, the prognosis does not look good at all.
This is chiefly because - and how many times do we need to say this, folks? - software development does not scale.
If that's true, though, how do big software products come into existence?
The answer lies in city planning. A city is made up of hundreds of thousands of buildings, on thousands of streets, with miles of sewers and underground railways and electrical cabling and lawns and trees and shops and traffic lights and etc etc.
How do such massively complex structures happen? Is a city planned and constructed by a single massive team of architects and builders as a single project with a single set of goals?
No, obviously not. Rome was not built in a day. By the same guys. Reporting to one boss. With a single plan.
Cities appear over many, many decades. The suburbs of London were once, not all that long ago, villages outside London. An organic process of development, undertaken by hundreds of thousands of people and organisations all working towards their own unique goals, and co-operating or compromising when goals aligned or conflicted, produced the sprawling metropolis that is now London.
Trillions of pounds has been spent creating the London of today. Most of that investment is nowhere to be seen any more, having been knocked down (or bombed) and built over many times. You could probably create a "London" for a fraction of the cost in a fraction of the time, if it were possible to coordinate such a feat.
And that's my point: it simply isn't possible to coordinate such a feat, not on that scale. An office complex? Sure. A housing estate? Why not? A new rail line with new train stations running across North London? With a few tens of billions and a few decades, it's do-able.
But those big projects exist right the edge of what is manageable. They invariably go way over budget, and are completed late. If they were much bigger, they'd fail altogether.
Cities are a product of many lifetimes, working towards many goals, with no single clear end goal, and with massive inefficiency.
And yet, somehow, London mostly looks like London. Toronto mostly looks like Toronto. European cities mostly look like European cities. Russian cities mostly look like Russian cities. It all just sort of, kind of, works. A weird conceptual cohesion emerges from the near-chaos.
This is the product of culture. Yes, London has hundreds of thousands of buildings, designed by thousands of people. But those people didn't work in bubbles, completely oblivious to each others' work. They could look at other buildings. Read about their design and their designers. Learn a thousand and one lessons about what worked and what didn't without having to repeat the mistakes that earned that knowledge.
And knowledge is weightless. It travels fast and travels cheaply. Hence, St Petersburg looks like the palaces of Versailles, and that area above Leicester Square looks like 19th century Hong Kong.
Tens of thousands of architects and builders, guided by organising principles plucked from the experience of others who came before.
Likewise, with big software products. Many teams, with many goals, building on top of each other, cooperating when it makes sense, compromising when there are conflicts. But, essentially, each team is doing their own thing for their own reasons. Any attempt to standardise, or impose order from above, fails. Every. Single. Time.
Better to focus on scaling up developer culture, which - those of us who participate in the global dev community can attest - scales beautifully. We have no common goal, no shared boss; but, somehow, I find myself working with the same tools, applying the same practices and principles, as thousands of developers around the world, most of whom I've never met.
Instead of having an overriding architecture for your large system, try to spread shared organising principles, like Simple Design and S.O.L.I.D. It's not a coincidence that hundreds of thousands developers use dependency injection to make external dependencies swappable. We visit the same websites, watch the same screencasts, read the same books. On a 10,000-person programme, your architect isn't the one who sits in the Big Chair at head office drawing UMLL diagrams. Your architect is Uncle Bob. Or Michael Feathers. Or Rebecca Whirfs-Brock. Or Barbara Liskov. Or Steve Freeman. Or even me (a shocking thought!)
But it's true. I probably have more influence over the design of some systems than the people getting paid to design it. And all I did was blog, or record a screencast, or speak at a conference. Culture - in this web age - spreads fast, and scales rapidly. You, too, can use these tools to build bridges between teams, share ideas, and exert tacit influence. You just have to let go of having explicit top-down control.
And that's how you scale software development.
September 13, 2016
4 Things You SHOULDN'T Do When The Schedule's SlippingIt takes real nerve to do the right thing when your delivery date's looming and you're behind on your plan.
Here are four things you should really probably avoid when the schedule's slipping:
1. Hire more developers
It's been over 40 years since the publication of Fred L. Brooks' 'The Mythical Man-Month'. This means that our industry has known for almost my entire life that adding developers to a late project makes it later.
Not only is this born out by data on team size vs. productivity, but we also have a pretty good idea what the causal mechanism is.
Like climate change, people who reject this advice should not be called "skeptics" any more. In the face of the overwhelming evidence, they're Small Team Deniers.
Hiring more devs when the schedule's slipping is like prescribing cigarettes, boxed sets and bacon for a patient with high blood pressure.
2. Cut corners
Still counterintuitively, for most software managers, the relationship between software quality and the time and cost of delivery is not what most of us think it is.
Common sense might lead us to believe that more reliable software takes longer, but the mountain of industry data on this clearly shows the opposite in the vast majority of cases.
To a point - and it's a point 99% of teams are in no danger of crossing - it actually takes less effort to deliver more reliable software.
Again, the causal mechanism for this is well understood. And, again, anyone who rejects the evidence is not a "skeptic"; they're a Defect Prevention Denier.
The way to go faster on 99% of projects is to slow down, and take more care.
3. Work longer hours
Another management myth that's been roundly debunked by the evidence is that, when a software delivery schedule's slipping significantly, teams can get back on track by working longer hours.
The data very clearly shows that - for most kinds of work - longer hours is a false economy. But it's especially true for writing software, which requires a level of concentration and focus that most jobs don't.
Short spurts of extra effort - maybe the odd weekend or late night - can make a small difference in the short term, but day after day, week after week overtime will burn your developers out faster than you can say "get a life". They'll make stupid, easily avoidable mistakes. And, as we've seen, mistakes cost exponentially more to fix than to avoid. This is why teams who routinely work overtime tend to have lower overall productivity: they're too busy fighting their own self-inflicted fires.
You can't "cram" software development. Like your physics final exams, if you're nowhere near ready a week before, then you're not gong to be ready, and no amount of midnight oil and caffeine is going to fix that.
You'll get more done with teams who are rested, energised, feeling positive, and focused.
4. Bribe the team to hit the deadline
Given the first three points we've covered here, promising to shower the team with money and other rewards to hit a deadline is just going to encourage them to make those mistakes for you.
Rewarding teams for hitting deadlines fosters a very 1-dimensional view of software development success. It places extra pressure on developers to do the wrong things: to grow the size of their teams, to cut corners, and to work silly hours. It therefore has a tendency to make things worse.
The standard wheeze, of course, is for teams to pretend that they hit the deadline by delivering something that looks like finished software. The rot under the bonnet quickly becomes apparent when the business then expects a second release. Now the team are bogged down in all the technical debt they took on for the first release, often to the extent that new features and change requests become out of the question.
Yes, we hit the deadline. No, we can't make it any better. You want changes? Then you'll have to pay us to do it all over again.
Granted, it takes real nerve, when the schedule's slipping and the customer is baying for blood, to keep the team small, to slow down and take more care, and to leave the office at 5pm.
Ultimately, the fate of teams rests with the company cultures that encourage and reward doing the wrong thing. Managers get rewarded for managing bigger teams. Developers get rewarded for being at their desk after everyone else has gone home, and appearing to hit deadlines. Perversely, as an industry, it's easier to rise to the top by doing the wrong thing in these situations. Until we stop rewarding that behaviour, little will change.
September 6, 2016
Empowered Teams Can Make Decisions The Boss Disagrees WithComing into contact, as I do, with many software development teams across a wide range of industries, you begin to recognise patterns.
One such pattern that - once I noticed it - I realised is very prevalent in Agile Software Development is what I call the Empowered Straightjacket. Teams are "empowered" to make their own decisions, but when the boss doesn't like a decision they've made, he or she overrules it.
Those who remember their set theory will know that if the set of all possible decisions a team is allowed to make can only include decisions the boss agrees with, then they are effectively working in the same set (or a subset) of the boss's rules.
That is not empowerment. Just in case you were wondering.
To have truly empowered development teams, bosses have to recognise that just being the boss doesn't necessarily make them right, and disagreeing with a decision doesn't necessarily make it a bad decision.
Unfortunately, the notion that decisions are made from above by more enlightened individuals has an iron grip on corporate culture.
Moving away from that means that managers have to reshape their role from someone who makes decisions to someone who facilitates the decision-making process (and then accepts the outcome graciously and with energy and enthusiasm.)
Once we recognise that there are other - more democratic and more objective - ways of making decisions, and that the decisions are just as likely to be right (if not more likely) than our own, then we have a golden opportunity to scale decision-making far beyond what traditional command-and-control hierarchies are capable of.
To scale Agile, you must learn to let go and let teams be the masters of their own destinies. You have no superhuman ability to make better decisions than a team full of highly-educated professionals.
The flipside of this is that developers and teams have to allow themselves to be empowered. With great empowerment comes great responsibility. And developers who've been cushioned from the responsibility of making decisions for a long time can run a mile when it comes a'knocking. Like prisoners who can't cope on the outside after a long stretch of regimented days doing exactly what they tell exactly when they tell you, devs who are used to management "taking care of all that" can panic when someone hands them a company credit card and says "if you need anything, use this". It reminds me of how my grandmother's hands used to shake when she had to write a cheque. Granddad took care of all that sort of thing.
This can lead to developers lacking confidence, which leads to them being afraid to take initiative. They may have learned their craft in environments where failure is not tolerated, and learned a survival strategy of not being responsible for anything that matters.
In this situation, developers rise up through the ranks - usually by length of service - and perpetuate the cycle by micromanaging their teams.
Based on my own subjective experiences leading dev teams (and being led): ultimately, developers empower themselves. The maxim "it's easier to ask for forgiveness than permission" applies here.
Now, t'was ever thus. But the rise of Agile Software Development has forced many managers to at least pretend to empower their teams. (And, let's face it, the majority are just pretending. Scrum Masters are not project managers, BTW.)
That's your cue to seize the day. They may not like it. But they'll have to pretend they do.
February 26, 2016
75% Of Us Think Software Developers Would Do Better Work In Small PartnershipsA conversation that's been ongoing between myself and many other developers I know over the last few years has been around the question of how developers should organise ourselves to do our best work.
Anecdotally, there's a feeling that the way we do things now is dysfunctional. Someone who knows little about writing software hires a bunch of developers, sticks them in a noisy environment that they cannot control, and gives them a boss who is incentivised to do all the wrong things.
The end result is predictably poor. Buggy and unusable systems, dissatisfied users, missed windows of business opportunity, huge cost overruns, and outright failures are commonplace.
And, to add insult to injury, it usually sucks for the teams who have to put up with doing poor work.
So, why haven't we fixed this? Two reasons come up repeatedly:
1. The people doing the work usually aren't in charge
2. The people who are in charge have a vested interest in doing it wrong
Take team size, for example; when was the last time you heard a manager boasting about how small his last project was? When did you last hear a sales person try to persuade their customer it could be done with less people? When did you last hear a shareholder say that they shouldn't expand their development/outsourcing business any more because it will compromise quality? When did you last hear a manager say "Actually, I don't know anything about writing software. I'm just going to get in the way. I'll see myself out."?
There's a lot of money to be gained by making teams bigger than they need to be. I'm reminded of a meeting I once had with the project managers of a development shop in East Sussex, trying to persuade them of the benefits of defect prevention. "But why would we want to ship sooner and with less bugs? We get paid by the person-hour for development and maintenance." End of meeting.
And excellence doesn't scale easily in our line of work. Every person you add to the team is something of an outlier - if they're good software developers. About 1 in 10 of us is actually any good at the job. Building a team of good developers - who work well together - is very difficult. The larger you make that team, the more difficult it gets. Eventually, to keep expanding, businesses lower the bar and start drawing from that 90% that - let's be frank - aren't quite so good. By the time we get to teams of hundreds of developers, you find them scraping the bottom of the barrel.
These people aren't fools. So much of software development is not dysfunctional because we're stupid. Far from it, in fact. It's dysfunctional because that's how some of the people involved make more money. Overstaffing teams sucks for the people on those teams, but it's more revenue for the business doing the overstaffing. Sales people earn more commission for doing bigger deals. Shareholders see their stock prices go up when these businesses expand, arguably taking on more work than they can effectively handle.
The big consultancies don't make their billions by consistently delivering quality working software. They make their money by overstaffing. Massively. And the managers who bring them in view that as a good thing, because - like I said - who boasts about how small their last project was?
Even when it's internal IT - where you'd think value-for-money would be a bigger driver - we still see the same dysfunctions. The people in charge tend not to be incentivised to succeed. Like the journeyman CEOs who carve out long and lucrative careers, earning millions of pounds, running public companies into the ground. If it all goes belly-up, a CTO or IT Director can just move on to feed on the next dead whale. Failure is big business these days. I've met many tech managers who were serial failures, and it never seemed to impact on their careers adversely. Indeed, does it even come up in the interview for their next job? "So, you managed a department of 1,000 people with a budget of £200 million a year? Might I ask what you actually achieved with that?"
And they're welcome to it, of course. But, as a software developer, I always found that sort of work soul-destroying. Believe it or not, many developers like to do good work, and are very disappointed when the software they poured blood, sweat and tears made of Red Bull into over months and years gets throw in the dumpster because it just wasn't good enough or just wasn't what the users needed.
And we have little to no incentive in expansion for the sake of expansion. We're usually not given shares. We don't get paid more for doing the work, necessarily, just because there's twice as much work being done. We just get shittier work. And our end users get shittier software. If they get any software at all.
Increasingly, I and others are thinking that the business model is the problem. The profit motive, in these situations, doesn't produce better software or more satisfied users. Because the extra profits go to people who aren't doing the work and aren't using the software. To fix that, we need to remove these people from the equation. And the way to do that might just be for software developers to work in partnerships, like lawyers and GPs and architects tend to.
Sure, there are some giant law firms. But not very many. Most law is practised by people working in small firms, controlled by the lawyers themselves. There are no shareholders demanding growth and maximisation of profits. There are no bosses who don't understand the practice of law. And, at any size, they stand or fall on the results they get for their clients. For all their faults - and they have a few - lawyers at the very least don't carve out lucrative careers from successive failures.
So, yesterday, I put this to the Twitters with a little unscientific poll:
Would software developers work better in small partnerships than as employees of large (& expanding) businesses?— Codemanship (@codemanship) February 25, 2016
As you can see, the majority of respondents think we would work better in small partnerships.
So, let's get practical for a moment, and think about how this might work. You and a handful of colleagues, who you know are good at their job and you can work with, get together and form, say, a Limited Liability Partnership. As equal partners, you each get the same share of the profits. Maybe you pay yourselves base salaries, and then the rest depends on how the partnership performs. Maybe you take on junior partners who take a smaller share and bring them up to an equal share after they've proved themselves. Maybe you even take on trainee developers who just get paid a salary, but - according to the rules of your partnership - after they meet certain criteria, they can become partners, too.
So you add your LLP to a database of development firms, specifying your specialities, just like lawyers would. There may well be other information available about your track record, too. Like we can get for plumbers and electricians and other tradespeople these days, so we can better judge who might do the best job for us.
Then, when Johnny CTO needs new software, he picks up the phone and calls your LLP. Just like someone looking for a builder.
As an individual partner, it won't help you to overstaff the team. You'll probably end up earning less, as everyone apart from the trainees get a share of the profits. You earn more by doing better work that delivers more value. There. How do you like them onions?
Of course, this would require a complete overhaul of the way software and IT projects are run. But, to be fair, more than half of developers working today are technically self-employed. So we're sort of half way there anyway, aren't we?
What would be different would be how we engage with the work. No more would we be faceless cogs in giant unfeeling machines, just doing as we're told even when we know it's the wrong thing to do.
We'd be more like building contractors, getting involved from the start and making the technical decisions while we talk to the customers directly, and connect with real end users.
Like architects, we'd work in spaces designed by us for the work we do. Like GPs, we'd apply our equivalent of clinical standards and professional ethics. And, like lawyers, we'd get paid handsomely by building a track record of verifiable successes.
Well, it's a thought, anyway....
September 12, 2015
TDD Katas Too Easy For You? Try the Codemanship Team DojoOne thing I hear regularly is how the kinds of practical exercises we do in training workshops and pair programming interviews are "too trivial" to test real developers.
Curiously, and without any exceptions, it turns out that the people who make such claims are unable to complete the exercises to a high standard in the allotted time; leading me to think that just maybe we overestimate ourselves sometimes. And it's not escaped my attention that those who brag the loudest tend to do them least well.
But if you really want a bigger challenge - one that's more befitting of your programming genius - then there's always my Team Dojo
There are a number of user stories, with executable acceptance tests, for a social network for programmers from which we can explore and build teams based on a number of criteria.
The exercise is undertaken from a standing start. All you get is your computers and a network connection. You'll have to decide what language(s) and platforms you'll be developing for, set up version control if you think you need it (which, of course, you do), build automation, CI, all of that good stuff - none of it is provided.
Once you've got the sausage machine up and running, you then need to work through the user stories, designing and implementing working code in order to have someone outside your team verify that it does indeed pass each acceptance test on a machine that isn't yours.
Give yourself a maximum of a standard working day (8 hours) to complete it. Afterwards, assess the quality of the implementation code for readability, simplicity, lack of duplication etc. Give yourself a percebtage score for Code Cleanliness, and then multiply the points you picked up from passing acceptance tests by that.
Most good developers can do it in under a day. Curiously, teams of 3 or more tend to struggle to compete it in 8 hours. The rare great teams can do it in under 4 hours. Go figure!
You will learn LOTS, though you may well wish for the naïve simplicity of FizzBuzz by the time you get half-way through...