July 8, 2012
Testing The Testers - A Vague Hiring Process.Over sunday lunch with a tester friend today, I got to thinking about testing interviews.
There's been quite a lot of good ideas floating around recently about interviewing developers (e.g., Hibri Marzook's Pair Programming Interviews workshop at SC2012), and I've seen testers put through their paces with what are essentially developer interviews, too.
But testing is not programming - though programming may well be involved. On the principle that if you want to see if a juggler can juggle, ask to see them juggle, what kind of practical techniques could we use to put a tester through his or her paces?
What occured to me over lunch is that there'd be three distinct areas I'd look into.
The most obvious is the tester's ability to find bugs. Bring them in (after some basic vetting to weed out the testers who, let's face it, just aren't - still too many of those about, sadly) and sit them down with a copy of some software in which there are known bugs. Then give them a fixed amount of time to find those bugs, and document them in a useful way (i.e., how to reproduce them.)
This is sort of a human variant of mutation testing. We test the tester by introducing known defects into the code and then see if they can find them.
We could make it more meaningful by introducing the bugs in places where bugs would be more likely to lurk (long/complex methods, multithreaded code accessing global variables etc) so that they could use their understanding of the relationship between code and quality to make educated guesses. You could also include an incomplete automated test suite so they could look for parts of the software that aren't being tested, where bugs are more likely to lurk. You could even be really cheeky and leave a test failing, to see if they even bother to check. You might also like to leave them a pile of user stories with points assigned to them by the customer for relative value, or feature usage statistics, to test their ability to not only find bugs, but find the most important ones first.
There's more to being a tester than finding bugs, of course. So the second thing I'd want to look into is the tester's ability to drive out the details of what a customer wants and "bridge the communication gap", as Gojko Adzic puts it.
One way I thought of might be to get a "customer" - a non-technical domain/application expert - to describe features of an existing piece of software to our candidate. The candidate can ask questions and use examples and test cases to firm up their understanding of what it should actually be like, eventually agreeing a set of acceptance test scripts for each feature with the "customer". Because this software actually exists, we can execute this tests against a running version of it, and test the tests, effectively.
Finally, these days, a tester often needs to be a programmer - and a pretty handy one at that. So my third focus would be on programming skills, probably with an emphasis on automating tests. I might ask them to write Selenium scripts for the acceptance tests they agreed for this existing piece of software, looking not only for test automation abilities, but also clean code and generally good dev instincts.
Realistically, you might be looking at a whole day to put a tester through their paces, but this could be a progression. If they can't find bugs, probably not much point moving on to the next stage, so it might only be a whole day if you're actually any good.
And then there's the whole question of team fit. Sure, they may have the technical chops, but can this person actually work well with us? Maybe round the day off, if they get through all the previous stages, with a Team Dojo with the candidate fulfilling tester duties.
So, in practice, how might I do it? I think I might run it as elimination rounds. Invite a sixteen of the best candidates in to do the bug-finding exercise, and select the best eight at that to do the "customer" exercise, and the best four from that to do a pair programming interview to check their dev skills, and the two remaining after that participate in a Team Dojo to determine which one will be a better fit. (Those numbers are pretty arbitrary - you may be looking for several testers, for example - but that's the general idea. Whittle them down over the course of a day.)
Of course, I'm just thinking out loud. Again.
Posted 6 days, 12 hours ago on July 8, 2012