November 21, 2017

Learn TDD with Codemanship

What Can We Learn About Dev Team Performance from Distributed System Design?

There are all sorts of analogies for software development teams ("teams are like a box of chocolates" etc), and one I find very useful is to picture them as distributed information processing systems.

Each worker process (person) has a job to do. Each job has information inputs and outputs. Each job requires data (knowledge). And the biggest overhead is typically not the time or effort required to process the information in each process, but the communication overhead between processes. This is why, as we add more people (worker processes), performance starts to degrade dramatically.

But, ironically, most of the available thinking about dev team performance focuses on optimising the processes, and not the communication between the processes.

Following this analogy, if we apply performance patterns for distributed computing to dev teams, we can arrive at some basic principles. In particular, if we seek to minimise the communication overhead without harming outcomes, we can significantly improve team performance.

Processes communicate to share data. The less data they need to share, the lower the communication overhead. And this is where we make our classic mistake; the original sin of software development, if you like.

Imagine our processes can act on data at a low level, but all the conditional logic is executed by external management processes that coordinate the workflow. So every time a worker process needs a decision to be made, it must communicate with a management process and wait for a response. Yes, this would be a terrible design for a distributed system. And yet, this is exactly how most dev teams operate. Teamwork is coordinated at the task level - the level of details. A more performant design would be to give individual worker processes a goal, and then let them make any decisions required to achieve that goal. Tell them the what and then let them figure out the how for themselves.

And I can attest from personal experience that dev teams that empower their developers to make the technical decisions perform much better.

But, as any developer who's worked on a team will tell you, there still needs to be coordination beween developers to reach consensus on the how. A classic example is how many teams fail to reach a consensus on how they implement model-view-controller, each one coming up with their own architecture.

Often, the amount of coordination and consensus needed can be front-loaded. Most of the key technical decisions will need to be made in the first few weeks of development. So maybe just take the hit and have a single worker process (the whole team) work together to establish baseline data, a skeleton of technical and logical architecture, of technical standards and common protocols (e.g., check-in etiquette) on which everyone can build mostly autonomously later. I've been doing this with teams since the days affordable portable data projectors became available. These days they call it "mob programming".

And, of course, there's unavoidably one shared piece of mutable data all processes have no choice but to act on in parallel: the code.

Much has been said on the subject of distributed version control of source code, most of focusing on entirely the wrong problem. Feature Branching, for example, tries to achieve more autonomy between developers by isolating their code changes from the rest of the team for longer. If every check-in is a database transaction (which it is - don't say it isn't), then this is entirely the wrong lever to be pulling on to speed things up. When we have many processes committing transactions to shared database, making the transactions bigger and longer won't speed the system up, usually. We're aiming not to break the data. The only way to be sure of that is to lock the data while the transaction's being written to the database. (Or to partition the data so that it's effectively no longer shared - more on that in a moment.)

To avoid blocking the rest of the worker processes, we need transactions to be over as soon as possible. So our check-ins need to be smaller and more frequent. In software development, we call this Continuous Integration.

It also helps if we split the shared data up, so each blob of data is accessed by fewer worker processes. More simply, the smaller the shared codebase, the less of a CI overhead. Partition systems into smaller work products.

But, just as partitioning software systems into - say - microservices - can increase the communication overhead (what were once method calls are now remote procedure calls), partitioning shared codebases creates a much greater overhead of communication between teams. So it's also vitally important that the various codebases are as decoupled as possible.

I rail against developers who add third-party dependencies to their software for very simple pieces of work. I call it "buying the Mercedes to use the cigarette lighter". In the world of microservices, system component needs to be largely responsible for doing their own work. Only add a dependency when the development cost of writing the code to do that bit of work is significantly greater than the potential ongoing communication overhead. You have to be merciless about minimising external dependencies. Right now, developers tend to add dependencies far too lightly, giving the additional costs little or no thought. And our tools make it far too easy to add dependencies. I'm looking at you, Maven, NuGet, Docker etc.

So, to summarise, here are my tips for optimising the performance of development teams:

1. Give them clear goals, not detailed tasks

2. Make developers as autonomous as possible. They have the technical data, let them make the technical decisions.

3. Accept that, initially, parallelism of work will be very difficult and risky. Start with mob programming to establish the technical approach going forward.

4. Small and frequent merging of code speeds up team performance. Long-lived code branches tend to have the reverse effect to that intended.

5. Partition your architectures so you can partition the code.

6. Manage dependencies between codebases ruthlessly. Duplicated logic can be cheaper to live with than inter-team communication.




November 20, 2017

Learn TDD with Codemanship

10 Days Left to Book Half-Price TDD Training

A quick reminder about the special offer I'm running this month to help teams whose training budgets have been squeezed by Brexit uncertainty.



If you confirm your booking for a 1, 2 or 3-day TDD training workshop this month (for delivery before end of Feb 2018), you'll get a whopping 50% off.

This is our flagship course - refined through years delivering TDD training to thousands of developers - and is probably the most hands-on and comprehensive TDD and code craft training workshop you can get... well, pretty much anywhere. There are no PowerPoint presentations, just live demonstrations and practical exercises to get your teeth into.

As well as the basics, we cover BDD and Specification by Example, refactoring, software design principles, Continuous Integration and Continuous Delivery, end-to-end test-driven design, mocking, stubbing, data-driven and property-based unit testing, mutation testing and heap more besides. It's so much more than a TDD course!

And every attendee gets a copy of our exclusive 200-page TDD course book, rated 5 stars on goodreads.com, which goes into even more detail, with oodles of extra practical exercises to continue your journey with.

If you want to know more about the course, visit http://www.codemanship.com/tdd.html, or drop me a line.


November 7, 2017

Learn TDD with Codemanship

Why Agile's Not For Me

There's a growing consensus among people who've been involved with Agile Software Development since the early (pre-Snowbird) days that something is rotten in the state of Agile.

Having slowly backed out of the Agile movement over the last decade or more (see my semi-jocular posts on Post-Agilism from 2007), I approach the movement as a fairly skeptical observer.

Talking with folk both inside and outside the Agile movement - and many with one foot in and one foot out - has highlighted for me where the wheels came off, so to speak. And it's a story that's by no means unique to Agile Software Development. Like all good ideas in software, it's never long before the money starts taking an interest and the pure ideas that it was founded on get corrupted.

1. Too Much Emphasis On Working Software

But, arguably, Agile Software Development was fundamentally flawed straight out of the gate (or straight out of the ski resort, more accurately). If I look for a foundation for Agile, it clearly has its roots in the concept of evolutionary software development. Evolution is a goal-seeking algorithm that searches for an optimum solution by iterating designs rapidly - the more rapidly the better - and feeding back in what we learn with each iteration to improve our solution.

There are two key words in that description: iterating and goal-seeking. There is no mention of goals in the original Agile Manifesto. The manifesto stipulates that the measure of progress is "working software". It does not address the question of why we should build that software in the first place.

And so, many Agile teams - back in the days when Extreme Programming was still a thing - focused on iterating software designs to solve poorly-defined - or not defined at all, let's face it - business problems. This is pretty much guaranteed to fail. But, bless our little cotton socks, because we set ourselves the goal of delivering "working software", we tended to walk away thinking we'd succeeded. Our customers... not so much.

This was the crack in Agile through which the project office snuck back in. (More about them later.)

2. Not Enough Emphasis On Working Software

As Agile evolved as a brand, more and more of us tried to paint ourselves in the colours of management consultants. Because, let's be frank, that's where the big bucks are. People who would once have been helping you to fix your build script were now suddenly self-professed McKinsey-style business gurus telling you how to "maximise the flow of value" in your enterprise, often to comic effect because nobody outside of the IT department took us seriously.

And then, one day - to everyone's horror - somebody outside the IT department did start taking us seriously, and suddenly it wasn't funny any more. Agile "crossed the chasm", and now people were talking about "going Agile" in the boardroom. Management and business magazines now routinely run articles about Agile, typically seeking input from people I've certainly never heard of who are now apparently world-leading experts. None of these people has heard of Kent Beck or Ward Cunningham or Brian Marick or any other signatory of the original Agile Manifesto. Agile today is very much in the hands of the McKinseys of this world. A classic "be careful what you wish for" moment for those from the IT department who aspired to be dining at the top table of consulting.

Agile's now Big Business. And the business of Agile is going BIG. Like every good and pure thing that falls into the hands of management consultants, Agile has mutated from a small, beautiful bird singing a twinkly tune to a bloated enterprise albatross with a foghorn.

3. We Didn't Nuke The Project Office From Orbit To Be Sure

I'm often found hanging around on street corners muttering to myself incoherently about the leadership class. Well, it's good to have a hobby.

Across the world - and especially in the UK - we have a class of people who have no actual practical skills or specific expertise to speak of, but a compelling sense of entitlement that they should be in charge, often of things they barely understand.

In the pre-Agile Manifesto world, IT was ruled by the leadership class. There was huge emphasis on processes, driven by the creation of documents, for the benefit of people who were neither using the software or writing it. This was a non-programmer's idea of what programming should be. In the late 1990's, the project office was the Alpha and the Omega of software and systems development. People who'd never written a line of code in their lives telling people who do it day-in and day-out how it should be done.

Because, if they let programmers make the decisions, they'll do it wrong!!! And, to be fair, we often did do it wrong. We built the wrong thing, and we built it wrong. It was our fault. We let the project office in by frequently disappointing our customers. But their solution just meant that we still did it wrong, only now we did it wrong on a much grander scale.

And just as we developers kidded ourselves that, because we delivered working software, that meant we had succeeded, managers deluded themselves that - because the team followed the prescribed processes - the customer's needs had been met.

Well, nope. We ticked the boxes while the customer got ticked off.

It turns out that the working relationship between software developers and their customers is, and always has been, the crux of the problem. Teams that work closely and communicate effectively with customers tend to build the right thing, at least. There's no process, standard or boxes-and-arrows diagram that can fix a dysfunctional developer-customer relationship. CMMi all you like. It doesn't help in the end. And, as someone who specialised on software process engineering and wore the robes and pointy hat of a Chief Architect, I would know.

The Agile Manifesto was a reaction to the Big Process top-heavy approach that had failed us so badly in the previous decades. Self-organising teams should work directly with customers and do the simplest things to deliver value. Why write a big requirements specification when we can have a face-to-face conversation with the customer? Why create a 200-page architecture document when developers can just gather round a whiteboard when they need to talk about design?

XP in particular seemed to be a welcome death knell for value-sucking Plan-Driven, Big Architecture, Big Process roles. It was the end for those projects like the one where I was the only developer but for some reason reported to three project managers, spending a full day every week travelling the country helping them to revise their constantly out-of-date Gantt charts.

And, for a while, it was working. The early noughties was a Golden Age for me of working on small teams, communicating directly with customers, making the technical decisions that needed to be made, and doing it our way.

But the project office wasn't going to just slink away and die in a corner. People with power rarely relinquish it voluntarily. And they have the power to make sure they don't need to.

Just as before, we let them back in by disappointing our customers. A lack of focus on end business goals - real customer needs - and too much focus initially on the mechanics of delivering working software created the opportunity for people who don't write code to proclaim "Look, the people writing the code are doing Agile wrong!"

And, again, their solution is more processes, more management, more control. And, hey presto, our 6-person XP projects transformed into beautiful multi-team Enterprise Agile butterflies. Money. That's what I want.

Back To Basics

Agile today is completely dominated by management. It's no longer about software development, or about helping customers achieve real goals. It's just as top-heavy, process-oriented and box-ticky as it ever was in the 1990s. And it's therefore not for me.

Working closely with customers to solve real problems by rapidly iterating working software on small self-organising teams very much is, still. But I fear the word for that has had its meaning so deeply corrupted that I need to start calling it something else.

How about "software development"?





November 6, 2017

Learn TDD with Codemanship

Half-Price TDD Training - Special Offer



Just a quick plug for a special offer Codemanship are doing for November. To help companies whose training budgets are being squeezed by Brexit uncertainty, if you confirm your booking for a 1, 2 or 3-day TDD training workshop this month (for delivery before end of Feb 2018), you'll get a whopping 50% off.

This is our flagship course - refined through years delivering TDD training to thousands of developers - and is probably the most hands-on and comprehensive TDD and code craft training workshop you can get... well, pretty much anywhere. There are no PowerPoint presentations, just live demonstrations and practical exercises to get your teeth into.

As well as the basics, we cover BDD and Specification by Example, refactoring, software design principles, Continuous Integration and Continuous Delivery, end-to-end test-driven design, mocking, stubbing, data-driven and property-based unit testing, mutation testing and heap more besides. It's so much more than a TDD course!

And every attendee gets a copy of our exclusive 200-page TDD course book, rated 5 stars on goodreads.com, which goes into even more detail, with oodles of extra practical exercises to continue your journey with.

If you want to know more about the course, visit http://www.codemanship.com/tdd.html, or drop me a line.


October 24, 2017

Learn TDD with Codemanship

Don't Want to TDD? Don't TDD. Just Be Honest About Your Reasons


A growing body of evidence strongly suggests that Test-Driven Development produces code that is - on average - more reliable (less bugs per KLOC) and more maintainable (simpler, less duplication, more modular, and far faster and cheaper to re-test). And it can do this with little or no extra effort.

Hooray for our side!

But, of course, I'm always at pains to be clear that TDD is not compulsory. If you don't want to do it, then don't do it.

But if not TDD, then what else are you doing to ensure your code is reliable and maintainable? Perhaps you're writing automated tests after you've written the code, for example. Perhaps you're writing assertions inside the code and using a tool like QuickCheck to drive test cases. Perhaps you're doing Design by Contract. Perhaps you're using a model checker. What matters is the end result. TDD is optional.

You'll have to forgive my skepticism when people tell me they choose not to do TDD, though. Usually - it transpires upon seeing the resulting software - what they really mean is "we choose not to write reliable and maintainable code" and "we choose not to worry about the release after this one." The world is full of legacy production code that wasn't intended to last, much of it decades old.

So, by all means, do it some other way if that's what floats your boat. But be honest with yourself about your reasons. Eventually, whatever your stated justification, your code will tell the real story.






October 13, 2017

Learn TDD with Codemanship

Time for the Chief Software Developer?

A blog post by Extreme Programming co-creator Kent Beck about ageism in software development has set me thinking again about this whole issue.

As 50 looms on the horizon, I'm becoming only too aware of this predilection employers have for younger - cheaper - developers. "Do you think they might be a little old?" is a question I routinely hear from hiring managers in start-ups in particular.

The net effect of this - aside from throwing great talent and experience on the scrapheap - is we're a profession of perpetual beginners. Young people entering software development are very lucky if they're exposed to industry veterans in any practical way.

So insights about refactoring, say, that date back to the 1980s have to be rediscovered and reinvented year after year. The upshot is that most developers can't refactor code. So most code suffers from a severe lack of maintainability. By the time developers have learned to write better code, our industry dictates that they move away from the code-face. And around we go.

My solution to the "How come you're not a CTO yet?" question that inevitably comes up with someone of my vintage has been to sidestep it. As a developer, I'm getting too old and too expensive for most employers. But as a trainer and coach, 25 years of professional experience is a distinct advantage. So I code every day, and work with other developers on actual software, instead of sitting in meetings or being a PowerPoint jockey.

But sidestepping the job shouldn't be necessary. There should be a viable career path for people who want to stay hands-on. They're both necessary and valuable, especially to technology companies.

I maintain that there should be such a thing as a Chief Software Developer, who sits on the board like a CTO or CIO, but writes code on a day-to-day basis. They would be there to defend the reality of writing software from the uninformed "optimism" of bosses who think it's easy, and who just want everything yesterday, at half the price, and hang the consequences for the business.

I can think of many, many start-ups who could have avoided The Hell of A Thousand Complete Rewrites had such a person had one hand on the wheel. Indeed, I can think of a few very large companies who'd still be trading today if someone with genuine authority had said "No, let's do this right".

I'm a realist, though. I appreciate that history teaches us such authority is rarely given freely. Development teams need to take it. Perhaps one way teams could pave the way would be to identify who in their team would be a Chief Software Developer, and then just start calling them that. After a few years of many teams in many businesses doing it, the drip-drip-drip effect might just make it an officially acknowledged reality.

Who would be your Chief Software Developer?




September 22, 2017

Learn TDD with Codemanship

Evil FizzBuzz (or "So you think you're a team?")

Among the advanced topics we cover on the third day of the Codemanship TDD course, Continuous Integration presents some of the biggest challenges for dev teams.

Participants are subjected to an exercise called Evil FizzBuzz, which you might like to try on your team (or at your coding event).

Based on the drinking game, FizzBuzz is a very simple algorithm.

1. Generate a sequence of integers from 1 to 100.
2. Any integers that are divisible by 3, replace with "Fizz".
3. Any that are divisible by 5, replace with "Buzz".
4. Any that are divisible by 3 and 5, "FizzBuzz".
5. And, for a bit of extra spice, any that are prime numbers, replace with - or concatenate if already "Fizz" or "Buzz" - "Wizz".
6. Output as a comma-delimited string.

So the first 15 in the sequence would go:

1, Wizz, FizzWizz, 4, BuzzWizz, Fizz, Wizz, 8, Fizz, Buzz, Wizz, Fizz, Wizz, 14, FizzBuzz

This is pretty straightforward for a programmer to code a solution to, and makes a spiffy exercise for learning about triangulation in TDD.

Now to make it evil...

* Split the group up into 6 pairs (or threes, or ones, depending on how many people you've got).

* Assign each part of FizzBuzz above (1-6) to a pair. They can only work on code for that part of the whole.

* Task them to work together - but only coding/TDD-ing their individual parts - to deliver a complete solution that produce the desired output.

Give them about an hour. And stand back and enjoy the train wreck.

To achieve this, they need distributed version control. So someone in the group needs to create, say, a GitHub repository that they can all contribute to. Then someone needs to put the skeleton of a source code project in that repository for pairs to work in. Then someone needs to set up Continuous Integration for that source code project so that merges can be built and tested.

All of this typically takes up more than half the time allotted. And until they have a green build to merge into, everybody's blocked from pushing. The yak shaving's what trips up half the groups I've seen attempt Evil FizzBuzz. DevOps is not commonly our strong suit.

And, of course, they have to agree on stuff. They have to agree on what language they're going to use. They have to agree on a basic design for how all the parts will fit together. They have to agree on how the process of collaboration's going to work if they're not going to end up tripping over each other's feet.

This is where the other half usually come unstuck. Most developers and dev orgs just aren't used to this level of collaboration. It's intense. Really intense. What usually happens is they either spend 50 minutes out of their hour arguing and getting nowhere, or they just go off into their separate corners and do their own thing. Both lead to failure.

And then there's the whole lesson behind the exercise: if the group isn't disciplined about CI, they will fail to deliver Evil FizzBuzz. Guaranteed.

What I mean by that is that the protocols of CI need to be keenly observed to prevent pairs merging conflicting changes on top of each other. And the feedback CI gives us about where the code's going must not be ignored. Every pair should be keeping one eye on the build. When they see a new build succeed, it's time to get the latest changes and see how it fits in with what you're doing.

Agreeing on things. DevOps. Constant communication. Situational awareness. Coordinating. All things dev teams tend to suck at.

And that's why I love this exercise. Especially on the rare occasions that you see a group ace it, like my training client this week. It speaks volumes about them as a team, and it's a joy to watch it unfold as each build goes green and you see the solution taking shape in front of your eyes.

The purpose of Evil FizzBuzz is to (hopefully) open dev teams' eyes to CI as a means of communication in collaborative design, and in particular to just how intense that collaboration often needs to be, and how disciplined about CI they need to be for it to work.

I'll bet you a shiny penny your team can't do it. Most can't.

Now prove me wrong.






August 21, 2017

Learn TDD with Codemanship

Codemanship Code Craft FxCop Rules

So, here they are. Hot from the oven, my FxCop code rules for the upcoming Codemanship Code Craft "Driving Test".


Some rubbish code, yesterday.

If you're signed up to be one of our valiant guinea pigs for the trail driving test on Sept 16th, I heartily recommend you download them and get a bit of practice. Try writing code that breaks each of the 11 rules, and then refactoring that code to make the nasty messages go away.

There's versions for Visual Studio 2013, 2015 and 2017, plus instructions on installing and suing the rules with your own projects.

And even if you're not doing the driving test on Sept 16th, have a go anyway. Your code may not be as clean as you think ;)

Any bugs or false positives, drop me a line.



August 19, 2017

Learn TDD with Codemanship

Time For Learning - An Inconvenient Truth

I've watched many tweet debates ("twebates"?) recently on the subject of finding time for learning in software development.

In the culture of the code craft movement, the consensus has been that you have to put in the hours. And by that, they tend to mean your own hours, outside of the day job. I've seen many job ads stipulating that candidates would need to show evidence of this extra-curricular commitment: blogs, speaking at conferences, OSS contributions, personal projects and all that.

The counter argument comes chiefly from people advocating greater diversity in software. Single parents, for example, have a lot on their plate that makes popping along to the Extreme Tuesday Club or speaking at a conference in, say, Norway, logistically difficult. Where's the time in their day/week/year to read all three volumes of the Art of Computer Programming?

My perspective on all this, I'm afraid, is cold and sobering. It takes a lot of reading and talking and sharing and experimentation (also known as "trying new stuff") to get good at writing software - and to stay good at it.

That's an inescapable reality. It's an inconvenient truth about software development. Everyone wants those skills, but nobody's willing to pay to develop them. Cui bono? Demonstrably, the employer benefits from more skilled developers. So they should make a contribution to bulding those skills. Simples.

What we're really debating is where does that time come from? Most employers aren't willing to support learning out of their own budgets. They expect developers to arrive fully formed, and that means that anything aside from direct on-the-job experience is down to us to learn in our own time. It's wholly inadequate to the task because we can only learn things that have immediate relevance to what we're doing.

Imagine if doctors had to learn everything that way. "Well, Mr Gorman, I'm afraid you have a burst appendix. This hasn't come up before, so I'm going on a course to learn how to treat it. See you in 2 weeks."

This also excludes people whose backgrounds and situations make finding those extra hours every week very difficult. This is why I believe offering developers "10% time" or "20% time" is really very necessary if we want a more diverse profession. This is another inconvenient truth about software development. Job ads that demand large amounts of extra hours of "elective" work are effectively restricting applications to people with not a lot else going on in their lives.

In practice, the code crafter landscape is still pretty homogenous. When I run public events, we still get about 85% men, and most of those are white. Very occasionally, someone with a disability comes along - and I always try to make sure the event's accessible, and advertise that fact.

But the fact remains that there are a lot of potentially great developers out there who, much as they'd like to, can't get along to a Saturday workshop, and whose employers won't let them take time out for learning during the working week.

Those are the people we in the dev community rarely see. But we shouldn't assume that they're not there because they don't want to learn.

If your job ads say you're committed to increasing diversity, and then demand a large portfolio of extra-curricular activities, you have a cognitive dissonance.

So, my question is this: how do we square this circle? My current belief is that we must adapt the very nature of our jobs so that time for learning and deliberate practice is built into the working week. I believe that this should become the norm, whereas today it's very much the exception.

I come from the school of "if this needs to happen, then let's just do it". We made the mistake, as professionals, of letting other people manage our time. If we're to move forward then that needs to stop. As "prima donna" as this sounds, we should take that time, and not ask for permission.

Because if we ask for permission, we know what the answer will be.




August 17, 2017

Learn TDD with Codemanship

Your House, Your (Code Quality) Rules


Picking up where I left off on the custom FxCop rules for the Codemanship Code Craft "Driving Test" has reminded me of something that's vitally important.

This morning I wrote a class that enumerates a type's collaborators. The code currently looks like this:



Codified in this class is an understanding of what I mean by a "collaborator", for the purposes of the driving test.

First of all, I'm not including non-project types. This is a judgement call to keep design rules realistic. My rule will limit the number of collaborating types to 3. If that includes core library types etc, it's going to be really tough.

I'm including fields, parameters and local variables. I'm also including the declaring types of any bound members. So if I call a method that returns an object and then I call a method on that, it'll be included.

I'm also not counting base classes as collaborators. Again, it's a judgement call.

I'm working alone, so I get to make the rules. But in a team setting that absolutely should not happen. Don't send the "tools dev" away to work in isolation on quality gates for Continuous Inspection. Because what will happen is, when they return and unleash their rules on the rest of the team, there'll be tears before bedtime.

The whole team needs to be involved. This is a great candidate for mob programming, in my experience. While you're waiting for business requirements in the early stages of a project/product, here's what the team could be doing to get the delivery engine up and running.

It will require the team to have discussions about code quality with a level of precision they probably never have before. I think this is a good thing.