December 21, 2006

...Learn TDD with Codemanship

Object-Relational Test Coverage - No Free Lunch

Duncan Pierce has blogged about the need to test object-relational persistence mappings. I think I know the project he's referring to, as I worked with him on it. My own credo is that we test objects and object persistence separately, since they are separate concerns. That your bank transfer works correctly is one problem. That it saves to the database correctly is entirely another. I have followed this rule semi-religiously for several years, from testing hand-rolled persistence code using the Data Access Object and Unit Of Work design patterns to more sophisticated layers using Hibernate or equivalent.

There are major performance gains to be made from keeping the two kinds of tests apart, as well. 10,000 unit tests on plain objects all running in the same memory space will usually execute many times faster than 1000 tests that involve database transactions. So Duncan's is great advice that I highly recommend you heed.

It also has another key advantage - you can change your business logic (and business logic tests) without becoming dependent on persistence interfaces. Of course, if your persistence code is nicely abstracted this is less of a problem, but you still tend to get a lot of transactional code mixed in with the logic.

One topic that Duncan didn't cover was persistence test coverage - how much testing is enough? On the project Duncan alluded to, we (and that includes me) didn't really think of that. As a result, I'm sure our coverage was less than good enough, and looking back I can see where some of the problems we encountered might have been exacerbated by that.

Most importantly, we didn't really view our Hibernate mappings as testworthy code, and we weren't particularly test-driven about how we fleshed out those mapping files. We were kind of, sort of test-driven, but nowhere near as strictly as we were with other parts of the system. I see this as a purely perceptual issue - it just didn't occur to us!

Even without attributes, it could require a dozen or so unit tests to cover a simple object graph like this

Any part of a mapping can break our system if it's done wrong, and the rules of XP tell us to test anything that might possibly break. Following the rules of test-driven development, we should probably have written a unit test for every persistent operation, and had test fixtures for every persistent class. Our persistence tests weren't very modular, and the one fixture grew very big very quickly - crating a module gravity well, which is a classic refuctoring.

In a refactored set of persistence tests, we would have modular tests that cover every breakable aspect of a persistent object graph. We would test that every persistent class can be saved and retrieved correctly. That would include tests for every persistent attribute. It would also include tests for every relationship, included tests for insertions into collections (and ordered collections etc), cascaded updates and cascaded deletes.

And, if at all possible, we should have implemented our mappings one test at a time. If it ain't tested, it shouldn't be mapped. In all honesty, and purely because we weren't keeping score, I suspect our mapping test coverage was lower than 60%, expressed in terms of what could go wrong with a mapping, and I also suspect many of the problems we experienced were bugs that slipped through those cracks in our coverage.

Finally, I'd like you to consider this truism: there's no such thing as a free lunch. While tools like Hibernate can save you a lot of back-breaking work hand-coding your persistence layers, you still end up creating code of some kind - even though you may not recognise it as "code" - that can go wrong, and you still, therefore, end up with the need for comprehensive tests.

And while you're thinking about that, you might also ask yourself how - using the available tools - we might be able to measure our O-R mapping test coverage automatically. That's something I'd find useful!
Posted 14 years, 1 month ago on December 21, 2006