I spoke at Boston SPIN last night, and facilitated the hiring initiative roundtable before the main presentation. One of the roundtable participants explained that he was looking for entry-level testers. And, since no one (okay, not quite no one, but almost no one) teaches testing in college, he wanted to develop an audition to see if people would like testing. He doesn’t want to hire people as testers who don’t want to be testers.
I suggested he ask candidates to sit down with the product and test. Give the candidates a pencil and paper and blank defect reports. Let the candidate test for 20-30 minutes. At the end, debrief the candidate.Some debriefing questions:
What did you notice about the testing?
Some (or all) of these questions:
What did you find challenging about the testing?
What was easy?
What did you enjoy the most?
What did you enjoy the least?
What issues did your testing raise?
Can you see yourself testing each day? In what circumstances?
These questions follow the The Art of Focused Conversation: objective questions, reflective questions, interpretive questions, and decisional questions.In this case, it’s not the audition that’s tricky to develop, but debriefing the audition. But, especially if you’re considering hiring entry-level people for roles they haven’t considered, debriefing could be a critical part of the audition.
I’ve also applied the Blog Housekeeping changes to this blog. Let me know if you have trouble.
I was speaking with a colleague the other day, and he told me about an audition he’s been using for developers for years: asking them to implement a String-Copy function in the language in which they’ll be developing. Some of their reactions are telling:
Some have said, “Why? The language has it?”
Some have worked for 20 minutes, and figured it out.
Some have worked for over one hour and not figure it out.
My colleague’s reply to the “Why?” people is to explain he’s looking for an example of their language skills. Only one person walked out of the interview.
For the people who finish in under an hour, they discuss how this version works. In fact, some candidates realize they have performance problems with their code during the walkthrough, and then want to change it.
Some people freeze, or just can’t figure it out, so for the people who work for an hour but can’t get it — and want to keep working — he says, “Go home, figure it out, and email it to me.”
What my colleague is looking for is simple: does the person know the language well enough to write something that he or she is willing to discuss with something else. It doesn’t have to be perfect and it doesn’t even have to be right. My colleague has to see just a little evidence that the candidate has thought through the problem, has tried to implement something, and has checked his or her work. Brilliant!
Over the years, my colleague has kept the implementations on these pieces of paper, and he was surprised by how many techniques a person could use to implement a String-Copy. (I’m not
For whatever jobs you’re hiring, think about how you’ve asked the candidates to show that they have thought through the problem, tried to implement something, and has checked their work. That’s an audition, and it’s especially good if most people can solve the problem enough to discuss their implementation in about 20-30 minutes. That’s because many people become nervous or resigned if the problem seems overwhelming or impossible.
When I write columns for a site or magazine, I discuss the areas to write about with the editor. What the editors originally wanted was a here’s-how-to-use-your-intuition article about hiring. I explained that was a bad idea, and I would write a how-not-to-use-your-intuition article
Intuition is neither good nor bad when hiring — it just is. What’s critically important is to use your intuition to generate behavior-description questions and auditions. That way you gather data about the candidate and your intuition. Win-win all the way around.
I had dinner recently with someone who said, “We put our candidates through the wringer. First we test them, then we interview them, then we give them another test.” I spoke with someone else whose manager wants an extensive role play to pit candidates against each other (sounds a little like The Apprentice to me). And Keith Ray pointed me towards Permutation Algorithms and Job Interview Questions (scroll down a bit to get to this part).
In addition to the puzzle questions, all of these are techniques to trip candidates, not true auditions. OK, if you are having trouble finding people who know your specific language and you can’t take the time or money to train them (why the heck not?), then maybe some sort of test is acceptable. But not a whiteboard test. People program on a computer. If you want to test their knowledge of a programming language, make the test something they do on a computer. And don’t kid yourself, the test is not an audition. Solving puzzles, particpating in role plays, developing any kind of software that doesn’t reflect your product — all of those are tests — are not auditions. They tell you little or nothing about how the person will work at work.I liked what Kaplan said in his essay
I don’t think quickly on my feet and I get “stage fright”. Even for algorithms that I’ve written many times, like in-order binary tree traversal, I’ve frozen and been unable to recall the algorithm.
Most of us are nervous in an interview and can’t always quickly recall our accomplishments. Rapid-fire interrogations, puzzles, programming algorithms you can look up — all of those are intellectually interesting and not suitable for auditions.If you want to see how a person will work at work, develop a real audition — a technique to see the person’s work behaviors. If you want a programming problem solved, give the person a computer and access to whatever tools that person needs. If you want to extend a design, ok, that’s something that can be done at a whiteboard. (Don’t you find design generally a discussion among people? Wouldn’t you participate in that too?) But whatever you do, give the candidate time to think. People don’t have to solve problems on their feet at work — they have time to think. The faster you want to move, the more time you need to think. When you remove the access to generally available software libraries, time to think, or even discussions on the web, you’re removing the very tools a fine candidate may use daily at work. Why would you do that?
When I test my students in my classes, I use essay questions, because I want to know what they’ve learned. The essay questions take longer and are harder to grade, but they give me insight into what students learned (and what I didn’t teach). Behavior-description questions in an interview are similar. And, auditions, are a form of behavior-description question, except that you can watch the behavior or discuss the results of some work. Don’t make your interviews like multiple-choice tests. You’ll get people who can talk the talk, but may or may not be able to do the work.
In Laurent’s Hiring and Testing post, he wondered why we spend so much time in the up-front stages of hiring. Why not use the probation period that seems to be the law in France, and is part of many companies’ stated HR procedures?A bunch of reasons: as managers, we need to give substantive and useful feedback during probation; many professional people are uncomfortable with the notion of being on probation, and you still have to invest the time and money on interviews and reference checks (although you might be able to spend a little less). I’ll address each of these issues briefly.
How many of you are comfortable with giving substantive and useful feedback? I suspect that you haven’t been trained on how to provide feedback, and if you haven’t, it’s dang hard. Esther and I have developed our Five Steps to Effective Feedback, see Collaboration and Teamwork for some of the content.
Probation means testing of a person’s fitness for work, but too often we think of it as something we do after we’ve made a mistake.
It’s hard to take an agile approach to hiring if firing is time-consuming and difficult. If firing is difficult, then you still have to define the job and interview carefully, audition, and check references. If you don’t, you could be stuck with someone who’s not working well for an extremely long time.
Probation is a good idea, and has only been pro forma for many companies. The necessary preconditions are the hiring manager’s ability to give useful feedback and the ability to fire easily.
