The pros and cons of structured job interviews and competency-based questions

I’ve recruited a lot of people — mostly but not exclusively UX/UI designers and researchers. Over time, unsurprisingly, I’ve evolved my approach and whilst I didn’t start with them, did adopt competency-based questions as the mainstay of my interviews.

Let’s make sure we’re all on the same page — when I talk about ‘competency-based’ I mean questions along the lines of ‘Can you think of a time when…’, and then you ask for a time when the candidate dealt with an uncooperative colleague, or did their best work etc.

The rationale behind such questions is that evidence of past behaviour is the best predictor of future behaviour, and that asking people about what they have actually done and getting them to be specific about it is better than asking what someone hypothetically might do in a given situation. It’s a good rationale.

Since I’ve been job hunting and have now been on the receiving end of such questions I’ve gained a more rounded view of the pros and cons of this approach. What it boils down to is this — interviewers can be more focused on following interview protocol than getting the best information from the candidate. It’s also tied in with having a structured interview. HR advisers quite rightly point out that if you want to be able to properly compare candidates then you need to be consistent in your approach and the questions you ask. It’s a scientific experiment.

When I first recruited people I had a chat and decided on not very clear criteria whether I thought they’d be any good. I actually got good feedback from my boss on the quality of the people that I took on so it wasn’t a disaster — and I’m still in touch with most of those people.

When I was subsequently presented with a structured discussion guide, where I had to score the candidate on each section, I was sceptical at first. However, I did very quickly find it to be useful, especially where I was interviewing a number of people. As an aside I’ll point out that I would always interview with a colleague. This co-interviewer would usually be one of my team — it gave them the experience and development which they enjoyed, and helped to validate my own impressions of the candidate. Having done the scoring on each candidate I was surprised how much it helped to clarify our thoughts.

Evolution

The thing is, it’s not possible to control for all the variables of an interview in a way that may be possible in a scientific experiment. People react differently to questions and to context. I started off asking all the questions in the guide as they were written, and in that order. I quickly realised that caused some problems. The questions that were confusing or irrelevant could be re-written or dropped, which was done. But sometimes in answering a question a candidate would end up covering some or all of a later question without knowing it, so when it came to that subsequent question it wouldn’t make much sense just to ask it straight. So we might say something like ‘Apart from the thing you just told us about, what else did you do about x?’

The other thing was that people get nervous in interviews. When they do they get tunnel vision and their thinking closes down. We did what we could to try to make them feel comfortable but that only works so far. So sometimes we’d ask ‘Tell us about a time when…’ and the candidate would answer a completely different question, or rabbit on for ages getting mired in detail. The strict version of the protocol says let them talk, say thanks, and move on to the next question. That’s what I did at first, but very quickly realised that it wasn’t helping me or the candidate.

I started to give some nudges, and sometimes just stopped the candidate outright and tried to re-focus them. I was interested in an answer to the question, not how well they coped with an artificial interview. I would prompt the candidate to ‘tell me a bit more about that’, or ‘that’s not quite what I’m getting at, is there a different example you can think of’.

On the receiving end

It’s been quite instructive being on the receiving end of this process. There’s a risk that the application process (see my post on that Internal and recruitment applications need as much UX as ecommerce) and interview style become major filters in their own right, rather than the candidate’s ability to do the job.

I’ve found some of the competency-based questions quite hard. To suddenly come up with an example of experience from a long work history that meets my understanding of what is actually being asked has been challenging at times. On one occasion I couldn’t really understand what the difference was between the questions I was being asked, and there was no guidance. So it’s not surprising that I waffled, and the interviewers didn’t really get to hear what I was capable of.

I have also been asked to ‘Think of a time when…’, when the straight answer is ‘I can’t, because it never happened’. What I do is try to be honest about it but think of an analagous situation that might still score me a point or two. When I was interviewing people and that happened I would then fall back on ‘Ok, I understand that’s a situation you’ve never been in, so what do you think you would do if it occurred?’ At least then I find out if the candidate understood the context and appropriate actions rather than just giving him or her a low score and moving on.

Sometimes I’ll think of the ‘right’ answer to a question just as I walk out of the interview room. You could argue that the ability to think on your feet is an essential attribute of the job, and it would be a fair point, but the context and nature of what you’re being asked about is different.

Introverts are typically reflective. If you’re having a meeting at work you’ll get the best input from the introverts if you let them know what you want from them in advance. Otherwise they’ll let you know after (or, often, not) that they’ve thought of something they should have said in the meeting. Interviews are no different. The format of ‘give me an answer now to an important question’ discriminates against introverts. I’m an introvert.

Top tips for interviewers

So here’s where I’ve got to in my thinking around job interviews.

• Firstly and most important, as an interviewer never lose sight of why you’re there. You are trying to find the best candidate for the job, not the person who is best at interviews.
• Use an interview guide as a guide, not a script. The topics covered are what you need to find out about, and your job is to ask questions, prompt and direct the candidate so that you do find out. Don’t just stand back while someone digs their own grave.
• Consider letting the candidate know in advance what the questions are — or at least the subject areas that are going to be covered.
• Overall be flexible, subtle and nuanced in your questions. Try to understand the person in front of you rather than blindly following process.

Originally published at nickgassman.com on September 27, 2018.