Take a look at HBS’s Working Knowledge,Hiring for Executive Intelligence. Some quotes that rocked my world:
IQ test questions don’t assess the practical, on-your-feet thinking skills needed in business. What’s more, these tests have been repeatedly accused of racial and gender bias. Yet, despite these very real shortcomings, IQ tests are still a better predictor of managerial success than any other assessment tool. [HUH??? --JR]Because each question in the behavioral interview essentially assesses the same qualities, there’s no need for the grueling three-to-four-hour sessions favored by hiring managers today. They need only ask enough questions to get a reliable appraisal of the candidate’s work experience, job knowledge, and social skills. [Excuse me, but that's not been my experience --JR]Despite their advantages, behavioral interviews really only establish a candidate’s minimum qualifications; they don’t identify star talent. [Of course behavior-descriptions are not enough by themselves. --JR]
The article goes on to say …tests should focus on the particular cognitive subjects associated with executive work: accomplishing tasks, working with and through others, and judging oneself.
Ok, so now the good folks who publish in HBR agree that auditions are key to evaluating a executive’s potential for a position. The example they use is actually a good example.
Auditions for senior management are harder to develop than auditions for technical staff and first- and mid-level managers, but they are certainly not impossible. Any audition that requires the exec to disclose the questions he/she has about the circumstances and leading to a decision and the ability to make a decision is useful. Executive auditions do look more like case studies and need to be evaluated that way.
December 6th, 2005
Because the cost of a bad hire is so high, some organizations are resorting to a variety of psychometric tests to determine if it’s worth interviewing and possibly hiring a candidate. If you, like me, are suspicious of tests, take a look at The Einstein-Clavin Effect by Wendell Williams. My favorite quote:
… controlled research studies confirm the fact that people who “fake good” on self-reported tests can outscore folks who give honest responses. Burn that into memory: People who “fake good” on self-reported tests can outscore folks who give honest responses.
So, if you’re using some kind of test (tests are not auditions, which are useful), see if you can correlate any previous hiring with the results on the test. I bet you can’t.
Tests don’t predict behavior on the job. Behavior-description questions and auditions do. I’d much rather spend the time interviewing well than pay for a test that doesn’t predict work behaviors.
April 6th, 2005
I had dinner recently with someone who said, “We put our candidates through the wringer. First we test them, then we interview them, then we give them another test.” I spoke with someone else whose manager wants an extensive role play to pit candidates against each other (sounds a little like The Apprentice to me). And Keith Ray pointed me towards Permutation Algorithms and Job Interview Questions (scroll down a bit to get to this part).
In addition to the puzzle questions, all of these are techniques to trip candidates, not true auditions. OK, if you are having trouble finding people who know your specific language and you can’t take the time or money to train them (why the heck not?), then maybe some sort of test is acceptable. But not a whiteboard test. People program on a computer. If you want to test their knowledge of a programming language, make the test something they do on a computer. And don’t kid yourself, the test is not an audition. Solving puzzles, particpating in role plays, developing any kind of software that doesn’t reflect your product — all of those are tests — are not auditions. They tell you little or nothing about how the person will work at work.I liked what Kaplan said in his essay
I don’t think quickly on my feet and I get “stage fright”. Even for algorithms that I’ve written many times, like in-order binary tree traversal, I’ve frozen and been unable to recall the algorithm.
Most of us are nervous in an interview and can’t always quickly recall our accomplishments. Rapid-fire interrogations, puzzles, programming algorithms you can look up — all of those are intellectually interesting and not suitable for auditions.If you want to see how a person will work at work, develop a real audition — a technique to see the person’s work behaviors. If you want a programming problem solved, give the person a computer and access to whatever tools that person needs. If you want to extend a design, ok, that’s something that can be done at a whiteboard. (Don’t you find design generally a discussion among people? Wouldn’t you participate in that too?) But whatever you do, give the candidate time to think. People don’t have to solve problems on their feet at work — they have time to think. The faster you want to move, the more time you need to think. When you remove the access to generally available software libraries, time to think, or even discussions on the web, you’re removing the very tools a fine candidate may use daily at work. Why would you do that?
When I test my students in my classes, I use essay questions, because I want to know what they’ve learned. The essay questions take longer and are harder to grade, but they give me insight into what students learned (and what I didn’t teach). Behavior-description questions in an interview are similar. And, auditions, are a form of behavior-description question, except that you can watch the behavior or discuss the results of some work. Don’t make your interviews like multiple-choice tests. You’ll get people who can talk the talk, but may or may not be able to do the work.
December 16th, 2004