September 17, 2007

...Learn TDD with Codemanship

Architect as Tester - using xUnit to Test Design Quality

A while back, I caused upset and outrage among the literally several architects who read this blog by daring to propose that in this post-Agile world we find ourselves in, an architect is essentially a kind of tester.

Today I want to fan the flames of controversy once again by providing a practical illustration that I hope will cement in your mind at least the beginning of a vague notion that they might indeed be the same thing.

Let's imagine that we are working on a project where the team have just agreed that no method should contain more than 20 lines of code. It's a very simple example, I know - but bear with me.

And let's imagine that we have agreed that code that breaks this design rule is broken code - and should therefore not get integrated into our shared code repository. (Just as we've agreed that code that fails any of its unit tests should not get integrated.)

Now let's stand back and think about this problem from a different angle. Where do many of our unit tests come from? A lot of them are based on business rules. For example, we might have a rule that states that every hotel room must have a working telephone.

Fans of domain-driven development (or "formal specification", as it used to be know back in the bad old days when it wasn't "Agile" enough) will tell you that you should try to describe such rules in terms of a well-defined model of the problem domain. So our rule about hotel rooms having telephones might be more formally described with a bit of pseudo-code against an imaginary domain model as:

foreach ( Room room in hotel.getRooms())
Assert.IsTrue(room.getPhone() != null &&;

And yes, that's a sort of unit test assertion I've slotted into it there - because, if you're doing test-driven development, then your tests are your specifications.

Just as an aside, in the Object Constraint Language, ir might look a bit like this:

context Hotel inv: rooms->forAll(phone.oclExists and phone.isWorking)

Or something like that. Anyway, I digress. The point I was trying to make is that we express our tests in terms of a well-defined domain model.

The domain of architecture is the structure of software. With a well-defined domain model of our code we can just as readily express rules that apply to the structure of the software we deliver. Rules like:

foreach (Method method in type.getMethods() )
Assert.IsTrue(method.LinesOfCode() <= 20 );

This is more than just a theoretical analogy, though. Given some way of turning software into objects that can be queried - e.g., a parse tree created by a parser, or an IL model created by a disassembler, or a type model exposed by reflection, for example, we could expose the structure of our software and ask meaningful questions about it - and even construct executable tests using tools like xUnit.

We already have tools that do this - like FxCop and NDepend. But we don't really think of them as test automation tools. But the point I wanted to make is that - looked at from a certain, admittedly jaunty angle - that's exactly what they are.

If methods with more than 20 lines of code are caught by a test like this, that provides immediate and concrete feedback to the developer. In evolutionary design, software is shaped mostly by feedback, and so you can use tests like these - perhaps a little more sophisticated - to influence the evolution of the system and to gradually steer it towards a level fo quality that the majority of teams - Agile or otherwise - currently fall woefully short of.

Piling speculation upon speculation, I wonder whether the lack of such immediate and concrete feedback is the real reason why.

Just for jolly, as a sort of postcript to this discussion. I've knocked up an NUnit test that interrogates a model of a .NET assembly that has been disassembled by Cecil, the open source IL disassembler from the folks that brought us Mono. I think Cecil is also used by NDepend. I can't get actual Lines of Code from this model, but I can get the number of IL instructions in a method. Usually there's about 4-5 IL instructions for a line of code - based on numbers I've seen generated by tools like NDepend, so I've substituted a limit of 100 IL instructions instead of 20 LOC.

The assembly I'm testing is nunit.framework.dll - the same one being used to define the unit test - so we're really in introspection hell here :-)

Microsoft Research are also working on a framework for analysing code, aimed chiefly at academics, called Phoenix. The models it creates are much more powerful, but that comes at a price in terms of complexity and learning curve - so much so that I think it's not a viable choice at the moment for this sort of thing. Maybe someone will write a simplified wrapper for it to make it more accessible to you average John Q. Programmer like me.

There is also a question of scale. One of the burning questions that gets asked in the parlours and boudoirs of architects and Agilistas alike is "how do we scale up architectural governance and still reap the benefits of emergent design and empowered developers?" How can we flex our architectural muscles on the scale of dozens of projetcs and millions of lines of code?

I suspect automation might hold the key to solving this thorny problem. Tests like the one above can be quickly and economically applied to very large systems. The only drawback I can see at the moment is that we don't understand software design well enough to create the kind of "archibots" that would be able to spot more subtle and complex design flaws. I also suspect, because we're dealing with the analysis of very, very large networks here, that the number-crunching overhead could quickly run away with us. At which point practical solutions like grid computing and server clouds might start rearing their ugly heads.

But in the medium term, I can see us getting considerable value from automated design quality testing in a less ambitiuous vein, and will continue to explore and refine the theory and statr putting it into useful practice.
Posted 13 years, 2 months ago on September 17, 2007