Triplebyte Blog

We help engineers join great companies
Try our coding quiz

Share this article

A Humility Training Exercise for Technical Interviewers

By Ammon Bartram on Feb 4, 2019

A Humility Training Exercise for Technical Interviewers

tl;dr Humility is an important quality in technical interviewers. Our data shows that interviewers who are strongly confident in their own abilities give less consistent interview scores. Interviewers who are aware of their own weaknesses (and of how noisy interviews can be) in contrast, give more consistent scores. We've developed an exercise to help train interviewers in this area.

Programming interviews are noisy. Two interviewers judging the same candidate will often reach decidedly different conclusions about the candidate's skill, even in the same specific area. This noise is a significant obstacle to interview accuracy. Reducing this noise is one of the primary goals when training technical interviewers.

Training Interviewers

We train a lot of interviewers at Triplebyte. We employ a team of 40 experienced engineers to conduct interviews with candidates as they go through our platform. When we train new members of this team, we focus on several things. We make sure that interviewers are strong and up-to-date in the areas they will be measuring (it's surprisingly hard, sometimes, to distinguish a candidate who gives an unusual answer because they are an expert in an area from someone who gives an unusual answer because they don't know what they are talking about). We make sure that interviewers have clear guidelines for what skills they are assessing (this is the best defense against pattern matching bias in interviewers). However, I now think it's equally important to train interviewers in humility [1].

Lack of recognition of your own weaknesses is a major source of interview noise. This is true because overconfident interviewers judge candidates too harshly. The field of software engineering is broad enough that no single engineer can master it all. However, we all convince ourselves that the areas that we have mastered are the most important. And we don't fully respect another engineer if they are weak in an area where we are strong (even if they are very strong in other areas). In interviews, this manifests as a bias against candidates whose technical strengths are dissimilar to those of their interviewers. We measure this at Triplebyte by having multiple interviewers observe and grade the same interview. The effect persists even when interviewers grade areas unrelated to their own strength, and even when they use structured grading rubrics. Interviewers just give lower scores to candidates who are not like them. This is noise. It makes interviews less accurate, and we need to reduce it.

The solution, we've found, is to train interviewers in humility. Interviewers who are aware of their own weaknesses (and aware of how noisy interviews can be) are less influenced by areas other than the ones they are supposed to be evaluating, and give more consistent scores.

An Exercise to Build Humility

So, how can you train interviewers to be humble? How can you make yourself more humble? The answer, I think, is to experience what a candidate goes through. Interviewing for a job is humbling. You get grilled. You have to remember things you've not thought about in years. Smart people point out embarrassing flaws in your logic and code. You never know quite as much as you thought you did. And almost everyone fails a good percentage of their interviews.

We've developed an exercise that we use to let our interviewers experience being a candidate. At first we tried simply asking interviewers to interview each other. This did not work, however, because they were not able to give honest feedback. If you interviewed your co-worker and ended up thinking that they were kind of bad, would you tell them this honestly? Most people in this situation wouldn't. We needed to get around this somehow.

The solution came unexpectedly. The exercise actually started while I was trying to hire for our interview team. Part of the evaluation process that I used was asking candidates to interview me (it got really meta), and, in order to make these interviews more interesting, I gave a mix of (my attempt at) good and bad answers. I immediately noticed how this got them to give honest (and humbling) feedback--even on my “good” answers. From this experience, we developed the following exercise to train all our interviewers in humility:

  • Pair up with a co-worker, and have them ask you some of their favorite interview questions.
  • Tell them in advance that you are going to intentionally answer some of the questions poorly (role-playing answers that a weak candidate might give).
  • Then, as the interview progresses, do exactly this. About half the time give your best answer. The other half of the time give an intentionally poor answer.
  • After the interview is over, ask your co-worker to critique your answers.

What this does is free your co-worker to be 100% honest. They don't know which parts of the interview were really you trying to perform well. Moreover, they are on the hook to notice the bad answers you gave. If you gave an intentionally poor answer and they don't “catch” it, they look a little bad. So, they will give an honest, detailed account of their perceptions.

Be careful with this exercise! I've done this a bunch, and it's deeply humbling. It almost always results in someone you respect pointing out things you're bad at. And it has some potential to create conflict. I think it should probably only be done inside teams with a good degree of internal trust (the danger is convincing team members that other team members are not very good). But the result is powerful. It highlights clearly both the extent to which strong engineers are weak in certain areas, and the extent to which interviewers jump to conclusions about what a candidate means. I think everyone who conducts interviewers should put themselves through this exercise.

Conclusion

It's important for technical interviewers to be humble. This creates a better experience for the candidate, and it also makes interviews more accurate. The best interviewers are aware of their own limitations, and have a healthy appreciation of how capricious the process can be. To get better at these things, interviewers need to spend more time as candidates, being interviewed themselves. It's hard to create this experience among co-workers, but we've come up with a (dangerous!) exercise that does a pretty good job.

I'd love it if people tried this exercise more broadly. I think it might be something that should become standard for interview teams at most companies. If you give it a try, email me at ammon@triplebyte.com and let me know how it went.


[1] I've interviewed over 1000 people since starting Triplebyte. Some of the them probably don't feel that I was humble when I spoke with them. All I can say to this is... I'm sorry if I did a bad job interviewing you. Everything I write about here I apply to myself.

Get offers from top tech companies

Take our coding quiz

Liked what you read? Here are some of our other popular posts…

How to Interview Engineers

By Ammon Bartram on Jun 26, 2017

We do a lot of interviewing at Triplebyte. Indeed, over the last 2 years, I've interviewed just over 900 engineers. Whether this was a good use of my time can be debated! (I sometimes wake up in a cold sweat and doubt it.) But regardless, our goal is to improve how engineers are hired. To that end, we run background-blind interviews, looking at coding skills, not credentials or resumes. After an engineer passes our process, they go straight to the final interview at companies we work with (including Apple, Facebook, Dropbox and Stripe). We interview engineers without knowing their backgrounds, and then get to see how they do across multiple top tech companies. This gives us, I think, some of the best available data on interviewing.

Read More

Bootcamps vs. College

By Ammon Bartram on May 19, 2016

Programming bootcamps seem to make an impossible claim. Instead of spending four years in university, they say, you can learn how to be a software engineer in a three month program. On the face of it, this sounds more like an ad for Trump University than a plausible educational model.

But this is not what we’ve found at Triplebyte. We do interviews with engineers, and match them with startups where they’ll be a good fit. Companies vary widely in what skills they look for, and by mapping these differences, we’re able to help engineers pass more interviews and find jobs they would not have found on their own. Over the last year, we’ve worked with about 100 bootcamp grads, and many have gone on to get jobs at great companies. We do our interviews blind, without knowing a candidate's background, and we regularly get through an interview and give a candidate very positive scores, only to be surprised at the end when we learn that the candidate has only been programming for 6 months.

Read More

How to pass a programming interview

By Ammon Bartram on Mar 8, 2016

Being a good programmer has a surprisingly small role in passing programming interviews. To be a productive programmer, you need to be able to solve large, sprawling problems over weeks and months. Each question in an interview, in contrast, lasts less than one hour. To do well in an interview, then, you need to be able to solve small problems quickly, under duress, while explaining your thoughts clearly. This is a different skill. On top of this, interviewers are often poorly trained and inattentive (they would rather be programming), and ask questions far removed from actual work. They bring bias, pattern matching, and a lack of standardization.

Read More
Get the Triplebyte newsletter Subscribe

Get the Triplebyte newsletter

Be the first to get the latest Triplebyte content straight to your inbox.