December 10, 2018
The Gaps Between The ToolsOn a training course I ran for a client last week, I issued my usual warning about immediately upgrading to the latest versions of dev frameworks. Wait for the ecosystem to catch up, I always say.
This, naturally, drew some skepticism from developers who like to keep bang up-to-date with their toolsets. On this occasion, I was vindicated the very next day when one participant realised they couldn't get Cucumber to work with JUnit 5.
I abandoned an attempt to be bang up-to-date using JUnit 5 for a training workshop in Sweden the previous month. The workshop was all about "Third-Generation Testing", which required integration between unit testing and a number of other tools for generating test cases, tracking critical path code coverage, and parallelising test execution. I couldn't get any of them to work with the new JUnit Jupiter model.
So I reverted back to safe old JUnit 4. And it all worked just spiffy.
No doubt, at some point, the tools ecosystem will catch up with JUnit 5. But we're not there today. So I'm sticking with JUnit 4. And NUnit 2.x. And .NET Framework 3.5. And the list goes on and on. Basically, take the latest version, subtract 1 from the major release number, and that's where you'll find me.
For sure, the newer versions have newer features, which may or may not prove useful to me. But I'm more concerned about the overall development workflow, and that's why compatibility and interoperability mean more to me than new features.
We're notoriously bad at building dev tools and frameworks that "play nice" with each other. Couple that with a gung ho attitude to backwards compatibility, and you end up with a very heterogenous landscape of tools that can barely co-exist within much wider development practices and processes. It's our very own Tower of Babel.
In other technical design disciplines, like electronic engineering, tool developers worked hard to make sure things work together. Simulation tools plug seamlessly into ECAD tools, which talk effortlessly with manufacturing tools and even accounting solutions to provide a relatively frictionless workflow from initial concept to finished product. The latest release of your ASIC design tool may have some spiffy new features, but if it won't work with that expensive simulator you invested in, then upgrading will just have to wait.
Given that many of us are engaged professionally in integrating software to provide our customers with end-to-end processes, it's surprising that we ourselves invest so little in getting our own house in order.
Looking at the average software build pipeline, it tends to be a Heath Robinson affair of clunky adaptors, fudges and workarounds to compensate for the fact that - for example - every test automation tool produces a different output for what is essentially the exact same information. And it boggles the mind why we need 1,001 different adaptors to run the tests in the first place; every combination of build tool + test framework imaginable.
If test automation tools all supported the same basic command interface, and produced their outputs in the same standard formats, we could focus on the task in hand instead of wasting time reinventing the same plumbing over and over again. JUnit 5 would already work with Cucumber. No need to wait for someone to patch them back together again.
And if you're a tool or framework developer protesting "But how will the tools evolve if nobody upgrades?", my advice is to stop breaking my workflows when you release new versions. They're more important than your point solution.
I vote we start focusing more on the gaps between the tools.
December 9, 2018
Big Dependency Problems Lie In The Small DetailsJust a quick thought about dependencies. Quite often, when we talk about dependencies in software, we mean dependencies between modules, or between components or services. But I think perhaps that can blinker us to a wider and more useful understanding.
A dependency is a relationship between two pieces of code where a change to one could break the other.
If we consider these two lines of code, deleting the first line would break the second line. The expression x + 2 depends on the declaration of x.
Dependencies increase the cost of changing code by requiring us, when we change one thing in our code, to then change all the code that depends on it. (Which, in turn can force us to have to change all the code that depends on that. And so on.)
If our goal is to keep the cost of changing code low, one of the ways we can achieve that is to try to localise these ripples so that - as much as possible - they're contained within the same module. We do this by packaging dependent code in the same module (cohesion), and minimising the dependencies between code in different modules (coupling). The general term for this is encapsulating.
If I move x and y into different classes, we necessarily have to live with a dependency between them.
Now, deleting x in Foo will break y in Bar. Our ripple spreads across the class boundary.
Of course, in order to organise our code into manageable modules, we can't avoid having some dependencies that cross the boundaries. This is often what people mean when they say that splitting up a class "introduces dependencies". It doesn't, though. It redistributes them. The dependency was always there. It just crosses a boundary now.
And this is important to remember. We've got to write the code we've got to write. And that code must have dependencies - unless you're smart enough to write lines of code that in no way refer to other lines of code, of course.
Remember: dependencies between classes are actually dependencies between the code in those classes.
As we scale up from modules to components or services, the same applies. Dependencies beween components are actually dependencies beteween modules in those components, which are actually dependencies between code inside those modules. If I package Foo in one microservice and Bar in another: hey presto! Microservice dependencies!
I say all of this because I want to encourage developers, when faced with dependency issues in large-scale architecture, to consider looking at the code itself to see if the solution might actually lie at that level. You'd be surprised how often it does.
December 8, 2018
True Agile Requirements: Get It Wrong Quickly, Then IterateI'm going to be arguing in this post that our emphasis in the software design process tends to be wrong. To explain this, I'm going to show you some code. Bear with me.
This is a simple algorithm for calculating square roots. It's iterative. It starts with a very rough guess for the square root - half the input - and then refines that guess over multiple feedback cycles, getting it progressively less wrong with each pass, until it converges on a solution.
I use this algorithm when I demonstrate mutation testing, deliberately introducing errors to check if our test suite catches them. When I introduce an error into the line that makes the initial guess:
e.g., changing it to:
The tests still pass. In fact, I can change the initial guess wildly:
And the tests still pass. They take a little longer to run is all. This is because, even with an initial guess 2 million times bigger, it just requires an extra few iterations to converge on the right answer.
What I take from this is that, in an iterative problem solving process, the feedback loops can matter far more than the initial input. It's the iterations that solve the problem.
When I see teams, including the majority of agile teams, focusing on the initial inputs and not on the feedback cycles, I can't help feeling they're focusing on the wrong thing. I believe we could actually start out with a set of requirements that are way off the mark, but with rapid iterating of the design, arrive at workable solution anyway. It would maybe take an extra couple of iterations.
For me, the more effective requirements discpline is testable goals + rapid iterations. You could start with a design for a word processor, but if your goal is to save on heating bills, and you rapidly iterate the design based on customer feedback from real world testing (i.e., "Nice spellchecker, but our gas bill isn't going down!"), you'll end up with a workable smart meter.
This is why I so firmly believe that the key to giving customers what they need is to focus on the factors that affect the speed of iterating and how long we can sustain the pace of evolution. The cost of changing software is a big factor in that. To me, iterating is the key requirements discipline, and therefore the cost of changing software is a requirements issue.
Time spent trying to get the spec right, for me, is time wasted. I'd rather get it wrong quickly and start iterating.
December 2, 2018
Architecture: The Belated Return of Big Picture ThinkingA question that's been stalking me is "When does architecture happen in TDD?"
I see a lot of code (a LOT of code) and if there's a trend I've noticed in recent years it's an increasing lack of - what's the word I'm looking for? - rationality in software designs as they grow.
When I watch dev teams produce working software (well, the ones who do produce software that works, at least), I find myself focusing more and more on when the design decisions get made.
In TDD, we can make design decisions during four distinct phases of the red-green-refactor cycle:
1. Planning - decisions we make before we write any code (e.g., a rough sequence diagram that realises a customer test scenario)
2. Specifying- decisions we make while we're writing a failing test (e.g., calling a function to do what you need done for the test, and then declaring it in the solution code)
3. Implementing - decisions we make when we're writing the code to pass the test (e.g., using a loop to search through a list)
4. Refactoring - decisions we make after we've passed the test according to our set of organising principles (e.g., consolidating duplicate code into a reusable method)
If you're a fan of Continuous Delivery like me, then a central goal of the way you write software is that it should be (almost) always shippable. Since 2 and 3 imply not-working code, that suggests we'd spend as little time as possible thinking about design while we're specifying and implementing. While the tests are green (1 and 4), we can consider design at our leisure.
I can break down refactoring even further, into:
4a. Thinking about refactoring
4b. Performing refactorings
Again, if your goal is always-shippable code, you'd spend as little time as possible executing each refactoring.
Put more bluntly, we should be applying the least thought into design while we're editing code.
(In my training workshops, I talk about Little Red Riding Hood and the advice her mother gave her to stay on the path and not wander off into the deep dark forest, where dangers like Big Bad Wolves lurk. Think of working code as the path, and not-working code as the deep dark forest. I encourage developers to always keep at least one foot on the path. When they step off to edit code, they need to step straight back on as quickly as possible.)
Personally - and I've roughly measured this - I make about two-thirds of design decisions during refactoring. That is, roughly 60-70% of the "things" in my code - classes, methods, fields, variables, interfaces etc - appear during refactoring:
* Extracting methods, constants and such to more clearly document what code does
* Extracting methods and classes to consolidate duplicate code
* Extracting classes to eliminate Primitive Obsession (e.g., IF statements that hinge on what is obviously an object identity represented by a literal vaue)
* Extracting and moving methods to eliminate Feature Envy in blocks of code and expressions
* Extracting methods and classes to split up units of code that have > 1 reason to change
* Exctracting methods to decompose complex conditionals
* Extracting client-specific interfaces
* Introducing parameters to make dependencies swappable
And so on and so on.
By this process, my code tends to grow and divide like cells with each new test. A complex order emerges from simple organising principles about readabililty, complexity, duplication and dependencies being applied iteratively over and over again. (This is perfectly illustrated in Joshua Kerievky's Refactoring to Patterns.)
I think of red-green-refactor as the inner loop of software architecture. And lots of developers do this. (Although, let's be honest, too many devs skimp on the refactoring.)
But there's architecture at higher levels of code organisation, too: components, services, systems, systems of systems. And they, too, have their organising principles and patterns, and need their outer feedback loops.
This is where I see a lot of teams falling short. Too little attention is paid to the emerging bigger picture. Few teams, for example, routinely visualise their components and the dependencies between them. Few teams regularly collaborate with other teams on managing the overall architecture. Few devs have a clear perspective on where their work fits in the grand scheme of things.
Buildings need carpentry and plumbing. Roads need tarmaccing. Sewers need digging. Power lines need routing.
But towns need planning. Someone needs to keep an eye on how the buildings and the roads and the sewers and the power lines fit together into a coherent whole that serves the people who live and work there.
Now, I come from a Big ArchitectureTM background. And, for all the badness that we wrought in the pre-XP days, one upside is that I'm a bit more Big Picture-aware than a lot of younger developers seem to be these days.
After focusing almost exclusively on the inner loop of software architecture for the last decade, starting in 2019 I'm going to be trying to help teams build a bit of Big Picture awareness and bring more emphasis on the outer feedback loops and associated principles, patterns and techniques.
The goal here is not to bring back the bad old days, or to ressurect the role of the Big Architect. And it's definitely not to try to reanimate the corpse of Big Design Up-Front.
This is simply about nurturing some Big Picture awareness among developers and hopefully reincorporating the outer feedback loops into today's methodologies, which we misguidedly threw out with the bathwater during the Agile Purges.
And, yes, there may even be a bit of UML. But just enough, mind you.
October 19, 2018
How Not To Use An ORM?An anti-pattern I see often is applications - often referred to as "enterprise" applications - that have database transactions baked into their core logic via a "data access layer".
It typically goes something like this:
"When the order page loads, we fetch the order via an Order repository. Then we take the ID of that order and use that to fetch the list of order items via an Order Item repository. Then we load the order item product descriptions via a Product repository. We load the customer information for the order, using the customer ID field of the order, via a Customer repository. And then the customer's address via an Address repository.
"It's all nicely abstracted. We have proper separation of concerns between business logic and data access because we're using repositories, so we can stub out all the data access for testing.
"Yes, it does run a little slow, now that you ask. I wonder why that is?"
Then, behind the repositories, there's usually a query that's constructed using the object key or foreign keys - to retrieve the result of what ought to be a simple object navigation: order.items is implemented as orderItemRepository.items(orderId). You may believe that that you've abstracted the database because you're going through a repository interface, and possibly/probably using an object-relational mapping tool to fetch the entities, but if you're writing code that stitches object graphs together using keys and foreign keys, then you are writing the ORM tool. You're just using the off-the-shelf ORM as an xDBC substitute. It's the old "we used an X tool to build an X tool" problem. (See also "MVC frameworks built using MVC frameworks".)
The goal of an ORM is to make the mapping from tables and joins to object graphs Somebody Else's ProblemTM. That's a simpler way of defining true separation of concerns. As such, we should aim to write our core logic in the simplest object-oriented way we can, so that - ideally - the whole thing could run in memory with no database at all. Saving and fetching stored objects just happens. Not a foreign key or object repository in sight. It can vastly simplify the code (including test code).
The most powerful and flexible ORMs - like Hibernate - make this possible. I've written entire "enterprise" applications that could be run in memory, with the mapping and persistence happening entirely outside the core logic. In terms of hexagonal architecture, I treat data access as an external dependency and try to minimise it as much as possible. I don't write data access "layers".
Teams that go down the "layered" route tend to end up with heaps of code that depends directly on the ORM they're using (to write an ORM). It's a similar - well, these days, identical - problem to Java teams who do dependency injection using Spring and end up massively dependent on Spring - to the extent that their code can only be run in a Spring context.
At best, they end up with thousands of tests that have to stub and mock the data access layer so they can test ther core logic. At worst, they end up only being able to test their core logic with a database attached.
The ORM's magic doesn't come for free, of course. Yes, there's a heap of tweaking you need to do to make a completely seperated persistence/mapping component work. Many decisions have to be made (e.g., lazy loading vs. pre-emptive vs. SQL views vs. second level caching etc etc) to make it performant, but you were making those decisions anyway. You just weren't using the ORM to handle them, because you were too busy writing your own.
October 12, 2018
TDD Training - Part I (Classic TDD), London, Sat Dec 1stMy flagship Codemanship TDD training course returns in a series of 3 standalone Saturday workshops aimed at self-funding learners.
It's the exact same highly popular training we've delivered to more than 2,000 developers since 2009, with 100% hands-on learning reinforced by our jam-packed 200-page TDD course book.
Part 1 is on Saturday Dec 1st in central London, and it's amazingly good value at just £99.
Part I goes in-depth on "classic" TDD, the super-important refactoring discipline, and software design principles that you can apply to your code as it grows and evolves to keep it easy to change so you can maintain the pace of development.
- Why do TDD?
- An introduction to TDD
- Red, Green, Refactor
- The Golden Rule
- Working backwards from assertions
- Testing your tests
- One reason to fail
- Writing self-explanatory tests
- Speaking the customer's language
- Triangulating designs
- The Refactoring discipline
- Software Design Principles
- Simple Design
- Tell, Don’t Ask
The average price of a public 1-day dev training course, per person, is around £600-800. This is fine if your company is picking up the tab.
But we've learned over the years that many devs get no training paid for by their employer, so we appreciate that many of you are self-funding your professional development. Our Saturday workshops are priced to be accessible to professional developers.
In return, developers who've attended our weekend workshops have recommended us to employers and colleagues, and most of the full-price client-site training and coaching we do comes via these referrals.
Please be advised that we do not allow corporate bookings on our workshops for self-funders. Group bookings are limited to a maximum of 4 people. If you would like TDD training for your team(s), please contact me at firstname.lastname@example.org to discuss on-site training.
Find out more at the Eventbrite course page
October 6, 2018
Be The Code You Want To See In The WorldIt's no big secret that I'm very much from the "Just Do It" school of thought on how to apply good practices to software development. I meet teams all the time who complain that they've been forbidden to do, say, TDD by their managers. My answer is always "Next time, don't ask".
After 25 years doing this for a living, much of that devoted to mentoring teams in the developer arts , I've learned two important lessons:
1. It's very difficult to change someone's mind once it's made up. I wasted a lot of time "selling" the benefits of technical practices like unit testing and refactoring to people for whom no amount of evidence or logic was ever going to make them try it. It's one of the reasons I don't do much conference speaking these days.
2. The best strategies rely on things within our control. Indeed, strategies that rely on things beyond our control aren't really strategies at all. They're just wishful thinking.
The upshot of all this is an approach to working that has two core tenets:
1. Don't seek permission
2. Do what you can do
Easy to say, right? It does imply that, as a professional, you have control over how you work.
Here's the thing: as a professional, you have control over how you work. It's not so much a matter of getting that control, as recognising that - in reality - because you're the one writing the code, you already have that control. Your boss is very welcome to write the code themselves if they want it done their way
Of course, with great power comes great responsibility. You want control? Take control. But be sure to be acting in the best interests of your customer and other stakeholders, including the other developers on your team. Code is something you inflict on people. Do it with kindness.
And so there you have it. A mini philosophy. Don't rant and rave about how code should be done. Just do it. Be the code you want to see in the world.
Plenty of developers talk a good game, but their software tells a different story. It's often the case that the great and worthy and noble ideas you see presented in books and at conferences bear little resemblence to how their proponents really work. I've been learning, through Codemanship, that it's more effective to show teams what you do. Talk is cheap. That's why my flagship TDD workshop doesn't have any slides. Every idea is illustrated with real code, every practice is demonstrated right in front of you.
And there isn't a single practice in any Codemanship course I haven't applied many times on real software for real businesses. It's all real, and it all really works in the real world.
What typically prevents teams from applying them isn't their practicality, or how difficult they are to learn. (Although don't underestimate the learning curves.) The obstacles are normally whether they have the will to give it a proper try, and tied up in that, whether they're allowed to try it.
My advice is simple: learn to do it under the radar, in the background, under the bedsheets with a torch, and then the decision to apply it on real software in real teams for real customers will be entirely yours.
October 1, 2018
50% Off Codemanship Training for Start-ups and CharitiesOne of the most fun aspects of running a dev training company is watching start-ups I helped a few years ago go from strength to strength.
The best part is seeing how some customers are transforming their markets (I don't use the "d" word), and reaping the long-term benefits of being able to better sustain the pace of innovation through good code craft.
I want to do more to help new businesses, so I've decided that - as of today - start-ups less than 5 years old, with less than 50 employees, will be able to buy Codemanship code craft training half-price.
I'm also extending that offer to non-profits. Registered charities will also be able to buy Codemanship training for just 50% of the normal price.
September 28, 2018
Micro-cycles & Developing Your Inner Egg TimerWhen I'm coaching developers in TDD and refactoring, I find it important to stress the benefits of keeping one foot on the path of working code at all times.
I talk about Little Red Riding Hood, and how she was warned not to stray off the path into the deep dark forest. Bad things happen in the deep dark forest. Similarly, I warn devs to stay on that path of code that works - code that's shippable - and not go wandering off into the deep dark forest of code that's broken.
Of course, in practice, we can't change code without breaking it. So the real skill is in learning how to make the changes we need to make by briefly stepping off the path and stepping straight back on again.
This requires developers to build a kind of internal egg timer that nudges them when they haven't seen their tests pass for too long.
An exercise I've used to develop my internal egg timer uses a real egg timer (or the timer on my smartphone). When I'm mindfully practicing refactoring, for example, I'll set a timer to countdown for 60 seconds, and start it the moment I edit any code.
The moment a source file goes "dirty" - no longer compiles or no longer passes the tests - the countdown starts. I have to get back to passing tests before the sands run out (or the alarm goes off).
I'll do that for maybe 10-15 minutes, then I'll drop the countdown to 50 seconds and do another 10-15 minutes. Then 40 seconds. Then 30. Always trying, as best I can, to get what I need to do done and get back to passing tests before the countdown ends.
I did this every day for about 45-60 minutes for several months, and what I found at the end was that I'd grown a sort of internal countdown. Now, when I haven't seen the tests pass for a few minutes, I get a little knot in my stomach. It makes me genuinely uncomfortable.
I do a similar exercise with TDD, but the countdowns apply the moment I have a failing test. I have 60 seconds to make the test pass. Then 50. Then 40. Then 30. This encourages me to take smaller steps, in tighter micro-cycles.
If my test requires me to take too big a leap, I have to scale back or break it down to simpler steps to get me where I want to go.
The skill is in making progress with one foot firmly on the path of working code at all times. Your inner egg timer is the key.
September 25, 2018
Third-Generation Testing - Øredev 2018, Malmö, November 22ndIf you're planning on coming to Øredev in Sweden this November, I'm running a brand new training workshop on the final day about Third-Generation Software Testing.
First-generation testing was manual: running the program and clicking the buttons ourselves. We quickly learned that this was slow and often patchy, creating a severe bottleneck in development cycles.
Second-generation testing removed that bottleneck by writing code to test our code.
But what about the tests we didn't think of?
Exploratory testing brought us back to a manual process of exploring what else might be possible - what combinations of inputs, user actions and pathways - using the code we delivered, outside of the behaviours encoded in our automated tests.
Manual exploratory testing suffers from the same setbacks as any kind of manual testing, though. It's slow, and can miss heaps of cases in complex logic.
Third-generation testing automates the generation of the test cases themselves, enabling us to explore much wider state spaces than a manual process could ever hope to achieve. With a little extra test code, and a bit of ingenuity, you can explore thousands, tens of thousands, hundreds of thousands and even millions of extra test cases - combinations, paths, random inputs and ranges - using tools you already know.
In this workshop, we'll explore some simple techniques for adapting and reusing our existing unit tests to exhaustively test our critical code. We'll also look at techniques for identifying what code might need us to go further, and how we can use Cloud technology to execute millions of extra tests in minutes.
You can find out more and book your place at http://oredev.org/2018/sessions/third-generation-software-testing