Icon blog
Triplebyte Blog

We help engineers join great companies
Try our coding quiz

Back to Blog

Technical Interview Performance by Editor/OS/Language

By Mike Robbins on Sep 19, 2017

Technical Interview Performance by Editor/OS/Language

My co-worker Daniel is really into Emacs. It’s his primary editor, his grocery list (via org-mode), and #4 of the 6 things he can’t live without on his OkCupid profile. Other engineers in the office, however, prefer Vim (or Sublime Text, or even RubyMine). Naturally, this leads to war. But after months of salvos (VimGolf one-upmanship, gratuitous references to this Stack Overflow question) no clear victor has emerged.

So last week, I set out to see what the Triplebyte data has to say on the issue. Take this with a grain of salt (correlation does not imply causation), but there is a correlation between what editor an engineer uses, their language and OS, and their ability to pass programming interviews. And, according to our data, Vim users pass interviews at a rate ever-so-slightly higher than Emacs users! [1]

About the Data

At Triplebyte we evaluate hundreds of engineers every week. Each interview involves talking to one of our interviewers (like me!) over Google Hangouts. We record what language and editor each engineer uses. After our interview, we introduce the engineers to other companies (they skip the phone screens).

To gather the data I wanted, I pulled the editor / language data for every Triplebyte interview over the last year, as well as outcomes (pass/fail) for both the Triplebyte interviews and the interviews with companies the same engineers did through our platform. Here is the breakdown of editor use:

Editors and IDEs

It’s not just an Emacs and Vi(m) world anymore (was it ever?):

Editor - Breakdown.png

The #1 most common editor we see used in interviews is Sublime Text, with Vim close behind.

Emacs represents a fairly small market share today at just about a quarter the userbase of Vim in our interviews. This nicely matches the 4:1 ratio of Google Search Trends for the two editors.

However, I’m really looking for signals of quality, not quantity. Which engineers are most likely to successfully pass our interview?

Editor - Tech interview pass rate.png

Vim takes the prize here, but PyCharm and Emacs are close behind. We’ve found that users of these editors tend to pass our interview at an above-average rate.

On the other end of the spectrum is Eclipse: it appears that someone using either Vim or Emacs is more than twice as likely to pass our technical interview as an Eclipse user.

I was curious to see what happened after our interview, so I looked at how likely each group was to pass their final onsite interviews and get a job offer at the companies we connected them with:

Editor - Onsite success rate.png

For the most part, this looks fairly even across editors, which tells us that even if Vim users are stronger as a population compared to Sublime users when it comes to Triplebyte’s interview, we’re still identifying the correct subset of Sublime users that will do well at the companies we work with.

I was surprised to find one huge outlier here: Eclipse! Oddly enough, it appears that even after we pre-screen Eclipse users for their programming skills, they still get offers from only half as many of the onsite interviews we send them to compared with the rest of our population. I don’t have a good explanation for this correlation, and it’s something I’ll be looking into. (In any case, don’t worry, we still happily accept and find jobs for many great Eclipse users too!)

Programming Languages

You can use whatever language you like in the first section of our technical interview. Today, a plurality of engineers are choosing to interview in Python:

Language - Breakdown.png

Hey, what happened to PHP? As a former PHP developer myself, I found it interesting that it’s now just a small drop in the “other” bucket. As one of the interviewers here, I’ve personally never seen anyone interview in PHP yet (although others have).

Python is also known for being beginner-friendly, so will an Eclipse-like trend play out here? Well, not quite:

Language - Tech interview pass rate.png

In this case, we find that the average Ruby, Swift, and C# users tend to be stronger, with Python and Javascript in the middle of the pack.

As we dig into these numbers, it's worth noting that there's an unavoidable strong pairing between certain editors and certain languages: 90% of Eclipse or IntelliJ users also used Java, and 99% of PyCharm users used Python. One fascinating and statistically significant result I found is that a Java developer who used anything other than Eclipse was 47% more likely to pass our interview compared to those who did use Eclipse.

Here’s what happens after we select engineers to work with and send them to onsites:

Language - Onsite success rate.png

There are no wild outliers here, but let’s look at the C++ segment. While C++ programmers have the most challenging time passing Triplebyte’s technical interview on average, the ones we choose to work with tend to have a relatively easier time getting offers at each onsite.

I’m not entirely sure how to explain this, but one hypothesis might just be to look at the overall supply and demand on each side of our platform. We see strong demand for talented systems and low-level programmers from the companies we work with. On the other side, we’re constrained by a relatively small fraction of people applying using C++, multiplied by a relatively smaller fraction of those who we find talented enough to work with. Another thought is that perhaps more niche specializations might give more signal to our matching process. In any case, this tells me we should be looking for more talented C++ developers!

Operating Systems

If you’ve frequented a coffee shop in San Francisco in the past few years, it should be no surprise that the majority of engineers are interviewing on macOS today:

OS - Breakdown.png

In this case, the most popular choice is also the most successful:

OS - Tech interview pass rate.png

We found that Windows users struggle a bit more on average. In this category, I was surprised not to see Linux in the lead. As a former Linux desktop user myself, I’d have guessed that Linux users might have a wide range of technical knowledge related to their daily computing.

The trend continues in onsite interviews:

OS - Onsite success rate.png

Curiously, the Apple fans we work with also seem to pass about 20% more of their onsite interviews than Windows or Linux users. (This reminded me of an old OkCupid blog post comparing data from their Apple and Android users.)

It’s hard to know why exactly this happens, but there may be a hidden variable here. For example, while 60% of the people we interviewed used macOS, if you walk into any tech company in San Francisco you’re inevitably going to count far more than 60% MacBooks. While our interviews are background-blind, having a few years of professional programming experience usually helps!

Conclusion

As an interviewer and engineer here, I thought it’d be cool to see what kind of data we had on the tools engineers are using today.

In our 2017 dataset, we found that an engineer writing Ruby in Vim on macOS has a better-than-average chance of passing our technical interview, while one writing C++ in Eclipse on Windows is on the other end of the scale. We don’t explicitly use these indicators to decide who we work with, but this is a side product of data collected in our interview process.

At a higher level, we believe that the hiring process should be focused on discovering strengths, not uncovering weaknesses, and we’ve written extensively on How to Interview Engineers. Our best suggestion for companies is to allow candidates to use the tools that let them best show their personal strengths. In our case, this means we allow engineers to choose any language and environment of their choice, and do convenient interviews from home using screen sharing.

If you’re an engineer and want to give our process a try, the first step is completing our multiple-choice programming quiz here.

If you’re a company looking for engineers and would like to learn more about working with Triplebyte, you can get started here.


[1] To be clear, I don't think this has any predictive value for hiring, and I don’t think engineers should change their editor based on our analysis. Emacs (and Eclipse) are great editors!

Back to Blog

Liked what you read? Here are some of our other popular posts…

How to Interview Engineers

By Ammon Bartram on Jun 26, 2017

How to Interview Engineers

We do a lot of interviewing at Triplebyte. Indeed, over the last 2 years, I've interviewed just over 900 engineers. Whether this was a good use of my time can be debated! (I sometimes wake up in a cold sweat and doubt it.) But regardless, our goal is to improve how engineers are hired. To that end, we run background-blind interviews, looking at coding skills, not credentials or resumes. After an engineer passes our process, they go straight to the final interview at companies we work with (including Apple, Facebook, Dropbox and Stripe). We interview engineers without knowing their backgrounds, and then get to see how they do across multiple top tech companies. This gives us, I think, some of the best available data on interviewing.

Read More

Does it Make Sense for Programmers to Move to the Bay Area?

By Mark Lane on Dec 14, 2016

Does it Make Sense for Programmers to Move to the Bay Area?

If you’re a programmer considering a move to the Bay Area, you probably know at least two basic facts: 1) tech salaries are higher here than elsewhere, and 2) living here is really expensive. Both facts have been true for a long time, but they have become especially true in the past four years. Since 2012 home prices have risen by about 60% and rents by about 70% in both the San Francisco and San Jose metro areas. The absence of any apparent upper limit to these increases has given rise to a new journalistic subgenre, the Bay Area Housing Horror Story. Maybe you’ve heard about the cheapest house in San Francisco, a $350,000 “decomposing wooden shack” whose interior is “unlivable in its current condition”? Or the tent next to Google X that was renting for $895 a month? Or the guy on Reddit who calculated that it would be cheaper to commute daily to the Bay Area from Las Vegas by plane than to rent an apartment in San Francisco?

Read More

Bootcamps vs. College

By Ammon Bartram on May 19, 2016

Bootcamps vs. College

Programming bootcamps seem to make an impossible claim. Instead of spending four years in university, they say, you can learn how to be a software engineer in a three month program. On the face of it, this sounds more like an ad for Trump University than a plausible educational model.

But this is not what we’ve found at Triplebyte. We do interviews with engineers, and match them with startups where they’ll be a good fit. Companies vary widely in what skills they look for, and by mapping these differences, we’re able to help engineers pass more interviews and find jobs they would not have found on their own. Over the last year, we’ve worked with about 100 bootcamp grads, and many have gone on to get jobs at great companies. We do our interviews blind, without knowing a candidate's background, and we regularly get through an interview and give a candidate very positive scores, only to be surprised at the end when we learn that the candidate has only been programming for 6 months.

Read More