Icon blog
Triplebyte Hiring Blog

Hire the best engineers with Triplebyte
Start hiring

Share this article

Programming Interview Questions Are Too Hard and Too Short

By Charles Treichler on Feb 18, 2019

Programming Interview Questions Are Too Hard and Too Short

tl;dr Programming interview questions can feel unnecessarily difficult. Sometimes they actually are. And this isn't just because they make interviews excessively stressful. Our data shows that harder programming questions actually do a worse job of predicting final outcomes than easier ones.


Programming under time pressure is difficult. This is especially true during interviews. A coding exercise that would seem simple under normal circumstances somehow becomes a formidable challenge under the bright lights of an interview room. Stress hormones cloud your thinking during interviews (even though, sadly, neither fight nor flight is an effective response to a menacing programming problem). And it can almost feel like the questions are designed to be perversely difficult. I actually think this is more than just a feeling.

Interview questions are designed to be hard. Because the cost of hiring a bad engineer is so much higher than the cost of rejecting a good engineer, companies are incentivized to set a high bar. And for most companies that means asking hard questions. Intuitively this makes sense because harder questions seem like they should result in a more rigorous screening process. But intuition turns out to be a poor guide here. Our data shows that harder questions are actually less predictive than relatively easy ones.

Interview Questions Are Too Hard

Hard questions do filter out bad engineers, but they also filter out good engineers (that is, they have a high false-negative rate). Easy questions, in contrast, produce fewer false-negatives but more false-positives (since more engineers get them right, including some bad ones). Balancing these two signals is the core problem you face when selecting the optimal difficulty level for interview questions. Companies—seeking to avoid false positives at all cost—tend to skew towards the harder end of this spectrum.

However, whether or not a candidate answers a question correctly is not the only source of signal during an interview. You can also evaluate their process by, for example, observing how long it takes them to finish, how clean their code is, and how much they struggle while finding a solution. Our analysis shows that this second source of signal (process) is almost as predictive as the first (correctness).

But there is an additional trade-off here. The questions that carry the most process signal are significantly easier than the questions that carry the most correctness signal. The reason for this becomes clear when you distill process down to how much a candidate struggles while finding a solution (the aspect of process most directly related to question difficulty). If a question is hard enough to carry a strong correctness signal, then all candidates will struggle with it (even those who eventually answer it correctly). So struggle will carry no signal.

Conversely, questions that carry a strong process signal will be easy enough for most candidates to answer them correctly, thus having little to offer in terms of correctness signal. An optimally difficult question is one that balances process and correctness to extract the maximum signal from the combination of these two factors (which may not be the peak signal for either).

We interviewed thousands of engineers and scored their answers along multiple dimensions, including process and correctness, and compared these scores with later performance. And, after regression analysis (looking at both process and correctness signal) our data showed that the most predictive questions were actually much easier than we expected (and easier than the questions that many companies ask).

Harder questions ultimately filter out too many qualified candidates to be optimal. So, if you want to make your hiring process more accurate, you should probably ask easier questions.

I want to state clearly, however, that this doesn't mean you should lower the bar and pass more people. Asking easier questions doesn't necessarily mean making your interviews easier. The difficulty level of the questions you ask and where you set your decision-threshold are independent decisions. You can still implement an extremely rigorous hiring process by asking relatively easy questions and evaluating them demandingly. Our finding is simply that easier questions provide more signal. But what you do with that signal is up to you.

Easier interview questions are also less stressful, which is an important advantage. Stress causes candidates to under-perform. But, on the other hand, when candidates are more comfortable, they perform their true best, which actually makes interviews more predictive. I think that interviewers tend to under-estimate the effects of stress on candidates while over-estimating their own abilities. When you're the one asking the questions, it's easy to forget how hard it is to get much real programming done in 30-60 minutes. To counter this bias, we've adopted a rule at Triplebyte stating that interviewers have to give candidates 3X as much time to answer a question as they think it would take to solve it themselves. Usually this works out to be the right amount of time.

Interview Questions Are Too Short

Easier questions are beneficial for another important reason. They allow you to fit more content into your interviews. This means you can use longer, multi-part problems which have compounding benefits in terms of prediction. You can ask questions that ramp up in difficulty over time, and these longer, real-world questions are more predictive than their shorter and harder cousins.[1]

This is partly because longer questions serve as better proxies for real-life programming. Production programming involves working with a relatively large codebase over a long period of time, and longer questions just do a better job of approximating this reality.

Additionally, longer questions allow you to offer hints when a candidate gets stuck. I think this is vital because even strong engineers can get tripped up at some point during a coding problem. Asking longer questions gives candidates the chance to recover from errors and show off their skills later on during an exercise. A single misstep shouldn't ruin an entire interview. Finally, being able to offer help makes interviews less stressful which, again, results in more accurate outcomes.

Conclusion

Companies can get stuck in a vicious cycle when it comes to question difficulty. They start out asking questions that are already too hard and too short, which leads them to make sub-optimal hiring decisions based off noisy signals. When they experience poor outcomes, they may attempt to correct by making their process even more rigorous (which generally means asking even harder questions). But this only makes their interviews less accurate. And so on. Both companies and candidates are hurt by this cycle. I see companies missing out on talent and candidates missing out on jobs—and everyone getting more stressed than they need to be.

I hope interviewers will embrace this finding. Not only can it make their work more accurate, but easier as well. It's much less time intensive to come up with straightforward, multi-step problems than perversely difficult short challenges.

So, here's our advice, if you actually want to make your interviews more accurate, you probably need to start asking easier programming questions. This doesn't mean lowering the bar. It just means getting a better signal so you can hire the right people.

If you decide to ask easier questions during programming interviews, or find this post helpful in some other way, I'd love to hear from you! Email me at charlie@triplebyte.com


[1]“The best predictor of how someone will perform in a job is a work sample test (29 percent). This entails giving candidates a sample piece of work, similar to that which they would do in the job, and assessing their performance at it.” - (Wired Magazine)

Stop wasting time on bad candidates. Start hiring great engineers.

Start Hiring

Liked what you read? Here are some of our other popular posts…

How to Interview Engineers

By Ammon Bartram on Jun 26, 2017

We do a lot of interviewing at Triplebyte. Indeed, over the last 2 years, I've interviewed just over 900 engineers. Whether this was a good use of my time can be debated! (I sometimes wake up in a cold sweat and doubt it.) But regardless, our goal is to improve how engineers are hired. To that end, we run background-blind interviews, looking at coding skills, not credentials or resumes. After an engineer passes our process, they go straight to the final interview at companies we work with (including Apple, Facebook, Dropbox and Stripe). We interview engineers without knowing their backgrounds, and then get to see how they do across multiple top tech companies. This gives us, I think, some of the best available data on interviewing.

Read More

Bootcamps vs. College

By Ammon Bartram on May 19, 2016

Programming bootcamps seem to make an impossible claim. Instead of spending four years in university, they say, you can learn how to be a software engineer in a three month program. On the face of it, this sounds more like an ad for Trump University than a plausible educational model.

But this is not what we’ve found at Triplebyte. We do interviews with engineers, and match them with startups where they’ll be a good fit. Companies vary widely in what skills they look for, and by mapping these differences, we’re able to help engineers pass more interviews and find jobs they would not have found on their own. Over the last year, we’ve worked with about 100 bootcamp grads, and many have gone on to get jobs at great companies. We do our interviews blind, without knowing a candidate's background, and we regularly get through an interview and give a candidate very positive scores, only to be surprised at the end when we learn that the candidate has only been programming for 6 months.

Read More

How to pass a programming interview

By Ammon Bartram on Mar 8, 2016

Being a good programmer has a surprisingly small role in passing programming interviews. To be a productive programmer, you need to be able to solve large, sprawling problems over weeks and months. Each question in an interview, in contrast, lasts less than one hour. To do well in an interview, then, you need to be able to solve small problems quickly, under duress, while explaining your thoughts clearly. This is a different skill. On top of this, interviewers are often poorly trained and inattentive (they would rather be programming), and ask questions far removed from actual work. They bring bias, pattern matching, and a lack of standardization.

Read More
Get the Triplebyte newsletter Subscribe

Get the Triplebyte newsletter

Be the first to get the latest Triplebyte content straight to your inbox.