December 7, 2017

Learn TDD with Codemanship

"This would never have happened if we'd written it in Haskell" - Bah Humbug!

Spurred on by a spate of social media activity of the "We replaced a system written in X with one written in Y, and it was way better" kind, I just wanted to throw my hat into the ring on this topic.

As someone with practical insights into high-integrity software development, I can confidently attest that this is bunk of the highest order. There is no programming language that assures reliability.

Sure, there are languages with built-in features that can help, but you actually have to do stuff to make sure your code works 99.999999999% of the time. Y'know? Like testing and wotnot.

For example, you can inflict all kinds of damage in C, thanks to direct manipulation of memory, but you don't have to abuse those features of the language. A Bugatti Veyron has a top speed of 254 mph, but you don't have to drive it at 254 mph.

"We would never have had that crash if we'd been driving a Volvo" really means "We'd never have had that crash if we'd been driving slower".

If you want to avoid dangling pointers in a C program, you can. It just takes a bit of know-how and a bit of discipline. Don't blame the language for any shortcomings you might have in either. The difference the language makes is small compared to the difference you make.


ADDENDUM: Just to clarify, I'm not saying better languages and tools don't help. What I'm saying is that the difference they make can be very small compared to other factors. How do I know this? Well, I've been programming for 35 years and have worked in a dozen or more languages on real projects. So there's that. But also, I've worked with a lot of teams, and noticed how the same team using different tools gets similar results, while different teams using identical tools can get strikingly different results. So I conclude that the team makes the bigger difference, by orders of magnitude. So I choose to focus more on teams and how they work than on the tools, by orders of magnitude. And it's not as if tools and technology don't get enough focus within the industry :)



December 5, 2017

Learn TDD with Codemanship

Automating Code Reviews

This comes up quite often these days, so I thought I'd scribble my thoughts down, for posterity if nothing else.

I increasingly come across dev teams who have adopted a policy where every check-in needs to be reviewed before it can be accepted. In many cases, this has created a bottleneck while developers waiting to get a green build are stuck on the availability of their peers to do the reviewing.

Imagine, every time you want to check your code in, you have to wait for a tester to put your code through its paces. We knew that was a major bottleneck, so we started automating our tests. If the tester would normally check to see what happens if a customer cancels an order, we would write a unit test for the cancel() function of an order.

It's really not much different for code inspections. If a reviewer would normally check that no classes are too big (say, having more than 200 lines of code), we could write a bit of code to inspect every class and report any that exceed our limit.

A pretty comprehensive code inspection could cover a large amount of code, checking for a whole range of issues, in a tiny fraction of the time it takes a human. More importantly, those checks could be run any time. No need to wait for Jenny to get off the phone, or Rajesh to come back from lunch. You'd no longer be blocked.

This, of course, takes some considerable investment early on to develop the right suite of automated quality checks. But I see more and more teams struggling to maintain the pace of development and high code quality, and such an investment really pays for itself many times over, even on relatively short timescales.

It's for this reason that I'm going to be giving Continuous Inspection a big push in 2018. I think most teams should seriously consider it.


December 1, 2017

Learn TDD with Codemanship

Don't Succumb To "Facebook Envy". Solve The Problem In Front Of You

A trend that's been growing for some time now is what I call "Facebook envy". Dev teams working on bread-and-butter problems seem almost embarrassed not to be solving problems on the scale Facebook have to.

99.9% of developers are not working at this scale, and are never likely to. And yet I see a strange obsession with scale that too often distracts teams from more pressing problems.

I use the analogy of a rock band obsessing over how their songs can be arranged for a 90-piece orchestra for a performance at the massive O2 Arena in London, and failing to prepare for their upcoming gig in the bowling alley at the back of the local pub.

Of course, we hear the stories about tech start-ups who didn't prepare for greatness and discovered that their architecture didn't scale. We hear those stories precisely because of the pervasiveness of those businesses in our lives after the fact. Just like all the stories we read about how bands became mega-successful, because who wants to read about bands that didn't? History is written by the winners.

What we don't hear about is the other 999/1000 tech start-ups who did prepare for greatness and wasted their time and their money doing it.

Before a tech start-up needs to scale, it needs to survive. Surviving means solving the problems that are in front of you. By all means, keep one eye on the future - a sense of direction's important. But not both eyes.

It's a similar thing to cash flow. Sure, your product may be super-profitable eventually. But if you can't pay your staff and keep the lights on in the meantime, you won't be there to collect.

The best way to scale-proof your start-up is to solve today's problems in a way that doesn't block you from adapting to tomorrow's. This is why I work so hard to persuade teams to focus on the factors that make software and systems hard to change, instead of on trying to anticipate those changes at the start. It tends to make the end product far more complicated than it needed to be, and things rarely turn out the way you planned.

Some common technologies are inherently scalable, too. Indeed, these days, most technology stacks are, even if it takes a bit more imagination to achieve it. Facebook are the best example of this. Who'd have thought, 12 years ago, that PHP and MySQL would scale to a billion users? Facebook solved the problems that were in front of them. They didn't adapt those technologies speculatively... just in case they ended up with a billion users.

If you use scalable technologies, design your architectures in such a way that they would be easy to partition if needed (i.e., separate concerns cleanly from the get-go), and - most importantly - deliver code that can be changed when new times require it, then you'll be able to solve today's tangible problems and keep the door open to tomorrow's intangible possibilities.



November 21, 2017

Learn TDD with Codemanship

What Can We Learn About Dev Team Performance from Distributed System Design?

There are all sorts of analogies for software development teams ("teams are like a box of chocolates" etc), and one I find very useful is to picture them as distributed information processing systems.

Each worker process (person) has a job to do. Each job has information inputs and outputs. Each job requires data (knowledge). And the biggest overhead is typically not the time or effort required to process the information in each process, but the communication overhead between processes. This is why, as we add more people (worker processes), performance starts to degrade dramatically.

But, ironically, most of the available thinking about dev team performance focuses on optimising the processes, and not the communication between the processes.

Following this analogy, if we apply performance patterns for distributed computing to dev teams, we can arrive at some basic principles. In particular, if we seek to minimise the communication overhead without harming outcomes, we can significantly improve team performance.

Processes communicate to share data. The less data they need to share, the lower the communication overhead. And this is where we make our classic mistake; the original sin of software development, if you like.

Imagine our processes can act on data at a low level, but all the conditional logic is executed by external management processes that coordinate the workflow. So every time a worker process needs a decision to be made, it must communicate with a management process and wait for a response. Yes, this would be a terrible design for a distributed system. And yet, this is exactly how most dev teams operate. Teamwork is coordinated at the task level - the level of details. A more performant design would be to give individual worker processes a goal, and then let them make any decisions required to achieve that goal. Tell them the what and then let them figure out the how for themselves.

And I can attest from personal experience that dev teams that empower their developers to make the technical decisions perform much better.

But, as any developer who's worked on a team will tell you, there still needs to be coordination beween developers to reach consensus on the how. A classic example is how many teams fail to reach a consensus on how they implement model-view-controller, each one coming up with their own architecture.

Often, the amount of coordination and consensus needed can be front-loaded. Most of the key technical decisions will need to be made in the first few weeks of development. So maybe just take the hit and have a single worker process (the whole team) work together to establish baseline data, a skeleton of technical and logical architecture, of technical standards and common protocols (e.g., check-in etiquette) on which everyone can build mostly autonomously later. I've been doing this with teams since the days affordable portable data projectors became available. These days they call it "mob programming".

And, of course, there's unavoidably one shared piece of mutable data all processes have no choice but to act on in parallel: the code.

Much has been said on the subject of distributed version control of source code, most of focusing on entirely the wrong problem. Feature Branching, for example, tries to achieve more autonomy between developers by isolating their code changes from the rest of the team for longer. If every check-in is a database transaction (which it is - don't say it isn't), then this is entirely the wrong lever to be pulling on to speed things up. When we have many processes committing transactions to shared database, making the transactions bigger and longer won't speed the system up, usually. We're aiming not to break the data. The only way to be sure of that is to lock the data while the transaction's being written to the database. (Or to partition the data so that it's effectively no longer shared - more on that in a moment.)

To avoid blocking the rest of the worker processes, we need transactions to be over as soon as possible. So our check-ins need to be smaller and more frequent. In software development, we call this Continuous Integration.

It also helps if we split the shared data up, so each blob of data is accessed by fewer worker processes. More simply, the smaller the shared codebase, the less of a CI overhead. Partition systems into smaller work products.

But, just as partitioning software systems into - say - microservices - can increase the communication overhead (what were once method calls are now remote procedure calls), partitioning shared codebases creates a much greater overhead of communication between teams. So it's also vitally important that the various codebases are as decoupled as possible.

I rail against developers who add third-party dependencies to their software for very simple pieces of work. I call it "buying the Mercedes to use the cigarette lighter". In the world of microservices, system component needs to be largely responsible for doing their own work. Only add a dependency when the development cost of writing the code to do that bit of work is significantly greater than the potential ongoing communication overhead. You have to be merciless about minimising external dependencies. Right now, developers tend to add dependencies far too lightly, giving the additional costs little or no thought. And our tools make it far too easy to add dependencies. I'm looking at you, Maven, NuGet, Docker etc.

So, to summarise, here are my tips for optimising the performance of development teams:

1. Give them clear goals, not detailed tasks

2. Make developers as autonomous as possible. They have the technical data, let them make the technical decisions.

3. Accept that, initially, parallelism of work will be very difficult and risky. Start with mob programming to establish the technical approach going forward.

4. Small and frequent merging of code speeds up team performance. Long-lived code branches tend to have the reverse effect to that intended.

5. Partition your architectures so you can partition the code.

6. Manage dependencies between codebases ruthlessly. Duplicated logic can be cheaper to live with than inter-team communication.




November 20, 2017

Learn TDD with Codemanship

10 Days Left to Book Half-Price TDD Training

A quick reminder about the special offer I'm running this month to help teams whose training budgets have been squeezed by Brexit uncertainty.



If you confirm your booking for a 1, 2 or 3-day TDD training workshop this month (for delivery before end of Feb 2018), you'll get a whopping 50% off.

This is our flagship course - refined through years delivering TDD training to thousands of developers - and is probably the most hands-on and comprehensive TDD and code craft training workshop you can get... well, pretty much anywhere. There are no PowerPoint presentations, just live demonstrations and practical exercises to get your teeth into.

As well as the basics, we cover BDD and Specification by Example, refactoring, software design principles, Continuous Integration and Continuous Delivery, end-to-end test-driven design, mocking, stubbing, data-driven and property-based unit testing, mutation testing and heap more besides. It's so much more than a TDD course!

And every attendee gets a copy of our exclusive 200-page TDD course book, rated 5 stars on goodreads.com, which goes into even more detail, with oodles of extra practical exercises to continue your journey with.

If you want to know more about the course, visit http://www.codemanship.com/tdd.html, or drop me a line.


November 19, 2017

Learn TDD with Codemanship

Everything Else Is Details

For pretty much all my freelancing and consulting career, I've strongly advocated driving software development directly from testable end user goals. I'm not talking here about use cases, or the "so that..." art of a user story. I'm talking actual goals. Not "reasons to use the software", but "reasons to build it in the first place".

Although the Agile movement has started to catch up, with ideas like "business stories" and "impact mapping", it's still very much the exception not the rule that teams set out on their journey with a clear destination in mind.

Once goals have been established, the next step is to explore and understand the current model. How do users currently do things? And this is where a see another classic mistake being made by dev teams. They build an understanding of the existing processes, and then just reproduce those as they currently are in code. This can bake in the status quo, making it doubly hard for businesses to adapt and improve.

The rubber meets the road when we work with our customers to build a shared vision of how things will work when our software has been delivered. And, most importantly, how that will help us achieve our goals.

The trick to this - a skills that's sadly still vanishingly rare in our industry - is to paint a clear picture of how the world will look with our software in it, without describing the software itself. A true requirements specification does not commit in any way to the implementation design of a solution. It merely defines the edges of the solution-shaped hole into which anything we create will need to fit.

I think we're getting better at this. But we're still very naïve about it. Goals are still very one-dimensional - typically just focusing on financial objectives - and fail to balance multiple stakeholder perspectives. The Balanced Scorecard has yet to arrive in software development. Goals are usually woolly and vague, too, with no tests we could use to measure how we're doing. And - arguably our biggest crime as an industry - goals are often confused with strategies and solutions. 90% of the requirements specs I read are, in fact, solution designs masquerading as business requirements.

This ought to be the job of a business analyst. Not to tell us what software to build, but instead to describe what problem we need to solve. What we need from them is a clear, testable vision of how the world will be different because of our software. What needs to change? Then our job is to figure out how - if possible - software could help change it. Does your team have this vision?

I continue to strongly recommend that dev teams ditch the backlogs (and any other forms of long-term plans or blueprints), sit down with their customers and other key stakeholders, and work to define a handful of clear, testable business goals.

Everything else is details.




November 7, 2017

Learn TDD with Codemanship

Why Agile's Not For Me

There's a growing consensus among people who've been involved with Agile Software Development since the early (pre-Snowbird) days that something is rotten in the state of Agile.

Having slowly backed out of the Agile movement over the last decade or more (see my semi-jocular posts on Post-Agilism from 2007), I approach the movement as a fairly skeptical observer.

Talking with folk both inside and outside the Agile movement - and many with one foot in and one foot out - has highlighted for me where the wheels came off, so to speak. And it's a story that's by no means unique to Agile Software Development. Like all good ideas in software, it's never long before the money starts taking an interest and the pure ideas that it was founded on get corrupted.

1. Too Much Emphasis On Working Software

But, arguably, Agile Software Development was fundamentally flawed straight out of the gate (or straight out of the ski resort, more accurately). If I look for a foundation for Agile, it clearly has its roots in the concept of evolutionary software development. Evolution is a goal-seeking algorithm that searches for an optimum solution by iterating designs rapidly - the more rapidly the better - and feeding back in what we learn with each iteration to improve our solution.

There are two key words in that description: iterating and goal-seeking. There is no mention of goals in the original Agile Manifesto. The manifesto stipulates that the measure of progress is "working software". It does not address the question of why we should build that software in the first place.

And so, many Agile teams - back in the days when Extreme Programming was still a thing - focused on iterating software designs to solve poorly-defined - or not defined at all, let's face it - business problems. This is pretty much guaranteed to fail. But, bless our little cotton socks, because we set ourselves the goal of delivering "working software", we tended to walk away thinking we'd succeeded. Our customers... not so much.

This was the crack in Agile through which the project office snuck back in. (More about them later.)

2. Not Enough Emphasis On Working Software

As Agile evolved as a brand, more and more of us tried to paint ourselves in the colours of management consultants. Because, let's be frank, that's where the big bucks are. People who would once have been helping you to fix your build script were now suddenly self-professed McKinsey-style business gurus telling you how to "maximise the flow of value" in your enterprise, often to comic effect because nobody outside of the IT department took us seriously.

And then, one day - to everyone's horror - somebody outside the IT department did start taking us seriously, and suddenly it wasn't funny any more. Agile "crossed the chasm", and now people were talking about "going Agile" in the boardroom. Management and business magazines now routinely run articles about Agile, typically seeking input from people I've certainly never heard of who are now apparently world-leading experts. None of these people has heard of Kent Beck or Ward Cunningham or Brian Marick or any other signatory of the original Agile Manifesto. Agile today is very much in the hands of the McKinseys of this world. A classic "be careful what you wish for" moment for those from the IT department who aspired to be dining at the top table of consulting.

Agile's now Big Business. And the business of Agile is going BIG. Like every good and pure thing that falls into the hands of management consultants, Agile has mutated from a small, beautiful bird singing a twinkly tune to a bloated enterprise albatross with a foghorn.

3. We Didn't Nuke The Project Office From Orbit To Be Sure

I'm often found hanging around on street corners muttering to myself incoherently about the leadership class. Well, it's good to have a hobby.

Across the world - and especially in the UK - we have a class of people who have no actual practical skills or specific expertise to speak of, but a compelling sense of entitlement that they should be in charge, often of things they barely understand.

In the pre-Agile Manifesto world, IT was ruled by the leadership class. There was huge emphasis on processes, driven by the creation of documents, for the benefit of people who were neither using the software or writing it. This was a non-programmer's idea of what programming should be. In the late 1990's, the project office was the Alpha and the Omega of software and systems development. People who'd never written a line of code in their lives telling people who do it day-in and day-out how it should be done.

Because, if they let programmers make the decisions, they'll do it wrong!!! And, to be fair, we often did do it wrong. We built the wrong thing, and we built it wrong. It was our fault. We let the project office in by frequently disappointing our customers. But their solution just meant that we still did it wrong, only now we did it wrong on a much grander scale.

And just as we developers kidded ourselves that, because we delivered working software, that meant we had succeeded, managers deluded themselves that - because the team followed the prescribed processes - the customer's needs had been met.

Well, nope. We ticked the boxes while the customer got ticked off.

It turns out that the working relationship between software developers and their customers is, and always has been, the crux of the problem. Teams that work closely and communicate effectively with customers tend to build the right thing, at least. There's no process, standard or boxes-and-arrows diagram that can fix a dysfunctional developer-customer relationship. CMMi all you like. It doesn't help in the end. And, as someone who specialised on software process engineering and wore the robes and pointy hat of a Chief Architect, I would know.

The Agile Manifesto was a reaction to the Big Process top-heavy approach that had failed us so badly in the previous decades. Self-organising teams should work directly with customers and do the simplest things to deliver value. Why write a big requirements specification when we can have a face-to-face conversation with the customer? Why create a 200-page architecture document when developers can just gather round a whiteboard when they need to talk about design?

XP in particular seemed to be a welcome death knell for value-sucking Plan-Driven, Big Architecture, Big Process roles. It was the end for those projects like the one where I was the only developer but for some reason reported to three project managers, spending a full day every week travelling the country helping them to revise their constantly out-of-date Gantt charts.

And, for a while, it was working. The early noughties was a Golden Age for me of working on small teams, communicating directly with customers, making the technical decisions that needed to be made, and doing it our way.

But the project office wasn't going to just slink away and die in a corner. People with power rarely relinquish it voluntarily. And they have the power to make sure they don't need to.

Just as before, we let them back in by disappointing our customers. A lack of focus on end business goals - real customer needs - and too much focus initially on the mechanics of delivering working software created the opportunity for people who don't write code to proclaim "Look, the people writing the code are doing Agile wrong!"

And, again, their solution is more processes, more management, more control. And, hey presto, our 6-person XP projects transformed into beautiful multi-team Enterprise Agile butterflies. Money. That's what I want.

Back To Basics

Agile today is completely dominated by management. It's no longer about software development, or about helping customers achieve real goals. It's just as top-heavy, process-oriented and box-ticky as it ever was in the 1990s. And it's therefore not for me.

Working closely with customers to solve real problems by rapidly iterating working software on small self-organising teams very much is, still. But I fear the word for that has had its meaning so deeply corrupted that I need to start calling it something else.

How about "software development"?





November 6, 2017

Learn TDD with Codemanship

Half-Price TDD Training - Special Offer



Just a quick plug for a special offer Codemanship are doing for November. To help companies whose training budgets are being squeezed by Brexit uncertainty, if you confirm your booking for a 1, 2 or 3-day TDD training workshop this month (for delivery before end of Feb 2018), you'll get a whopping 50% off.

This is our flagship course - refined through years delivering TDD training to thousands of developers - and is probably the most hands-on and comprehensive TDD and code craft training workshop you can get... well, pretty much anywhere. There are no PowerPoint presentations, just live demonstrations and practical exercises to get your teeth into.

As well as the basics, we cover BDD and Specification by Example, refactoring, software design principles, Continuous Integration and Continuous Delivery, end-to-end test-driven design, mocking, stubbing, data-driven and property-based unit testing, mutation testing and heap more besides. It's so much more than a TDD course!

And every attendee gets a copy of our exclusive 200-page TDD course book, rated 5 stars on goodreads.com, which goes into even more detail, with oodles of extra practical exercises to continue your journey with.

If you want to know more about the course, visit http://www.codemanship.com/tdd.html, or drop me a line.


October 31, 2017

Learn TDD with Codemanship

Safer Duck Typing: Could We Have Our Cake & Eat It?

I've been thinking a lot lately about how we might reconcile the flexibility of duck typing, like in Python and Ruby, with the safety of static type checking like we have in Java and C#.

For many, the swappable-by-default you get with duck typing is a boon for an evolving domain model. And for me... well, there's a reason why people invented static type checking, and I would prefer not to go back to the bad old days of having to keep that model in my head.

Is there a "having our cake and eating it" arrangement possible here? I think there just might be.

In programming, "type safety" means there's a low risk of trying to invoke methods that the target object doesn't support. Static type systems like in Java enforce this as part of the design. All objects have a type, and that type describes what methods they support. Part of the "drag" of programming in Java is having to qualify this for every single object we want to use. Want to pass an object as a parameter in a method call? You'll need to specify what type that object is. So it's not possible to invoke a method that an object doesn't support - well, not easily, anyway.

It seems to me that, in languages like Python, we have that information in our code. We just need to apply it somehow.



In this example, two methods are invoked on a parameter of the calculate_net_salary() function. Whatever is passed in as that parameter value must implement those two methods with the exact signatures. But the Python compiler doesn't see that. If I passed in an instance of a class that didn't implement those methods, it would be perfectly happy.

Scala introduced a concept called structural typing, where we can specify with function parameters what methods any object passed in as that parameter value must support. e.g.,


def quacker(duck: {def quack(value: String): String}) {
println (duck.quack("Quack"))
}


I like how the required interface is defined by the client - I think this is how it ought to be. But I hate that I need to define this at all. It's duplicated information, and extra work that we have to remember to do. It, too, requires me to carry the model in my head. And when these contracts change, it's more code we have to change.

As far as I can see, the information's already in the function's implementation. We could examine the body of a function or method and build a table of the methods it calls. Then we trace back the source of the target instance. (This may require a bit of fancy shooting, but it's doable, surely?)

In my calculate_net_salary() I can trace the target of the invocations of income_tax() and national_insurance() to the parameter tax_calc. Following that back up the call stack to code that invokes calculate_net_salary(), I see that the class of the target instance is TaxCalculator.



Now I can examine this class to see if it does indeed support those methods.

This is something I've been saying all along about duck typing: the type is there, it's just implied.

I think what's needed is a belt-and-braces prototype to see if this could work in practice. If it really could work, then I - personally - would find something like this very useful. Would you? Do you know of something that already does this that my Googling has overlooked?





October 24, 2017

Learn TDD with Codemanship

Don't Want to TDD? Don't TDD. Just Be Honest About Your Reasons


A growing body of evidence strongly suggests that Test-Driven Development produces code that is - on average - more reliable (less bugs per KLOC) and more maintainable (simpler, less duplication, more modular, and far faster and cheaper to re-test). And it can do this with little or no extra effort.

Hooray for our side!

But, of course, I'm always at pains to be clear that TDD is not compulsory. If you don't want to do it, then don't do it.

But if not TDD, then what else are you doing to ensure your code is reliable and maintainable? Perhaps you're writing automated tests after you've written the code, for example. Perhaps you're writing assertions inside the code and using a tool like QuickCheck to drive test cases. Perhaps you're doing Design by Contract. Perhaps you're using a model checker. What matters is the end result. TDD is optional.

You'll have to forgive my skepticism when people tell me they choose not to do TDD, though. Usually - it transpires upon seeing the resulting software - what they really mean is "we choose not to write reliable and maintainable code" and "we choose not to worry about the release after this one." The world is full of legacy production code that wasn't intended to last, much of it decades old.

So, by all means, do it some other way if that's what floats your boat. But be honest with yourself about your reasons. Eventually, whatever your stated justification, your code will tell the real story.