Triplebyte Blog

We help engineers join great companies
Try our coding quiz

Share this article

The Rise of Microsoft Visual Studio Code

By Lyn Levenick and Ammon Bartram on Dec 4, 2018

The Rise of Microsoft Visual Studio Code

tl;dr Visual Studio Code usage is rising rapidly! VS Code is now the editor chosen by the majority of engineers during programming interviews, and it appears to be rapidly taking market share from other top editors.

Triplebyte interviews hundreds of engineers every week. For each interview, we record the editor, language and operating system used. We don't use this information to decide who passes our interview (I don't think that would be fair). However, it is fascinating data! It gives us insight into which tools different cohorts of engineers prefer, and how these preferences change over time. It also allows us to identify correlations between the tools engineers choose and their performance during programming interviews.

My co-worker Mike wrote a post about this data a year ago. But Triplebyte has grown a LOT since then, and we now have enough data to dive much deeper. That's my goal for this blog post.

Editor Usage Rates

To get started, I pulled data on the editors used during all interviews conducted over the last year:

editor-popularity-margin.png

The first thing that jumps out from this graph is the prominence of Visual Studio Code. With 17% of the pie, VS Code was the editor used by the plurality of Triplebyte candidates last year. This was a surprise to me for two reasons. First, VS Code is a relatively new product from Microsoft, and, second, last year it didn't even appear on our charts—its share was small enough to fall into the “other” bucket.

To get a better view of this, I graphed the same data over time:

editor-popularity-over-time-margin.png

Yikes! VS Code is eating everyone else's lunch! The story here is pretty clear. Over the past year, VS Code usage has gone from 5% to 22%. Over the same time, Sublime Text usage has fallen from 17% to 11%, and Atom usage has fallen from 11% to 6%. Even Eclipse is falling. And VS Code is accelerating every month. Google Trends shows the same thing:

editor-popularity-google-trends-margin.png

VS Code is on the rise and is poised to become the most dominant editor we've ever seen. Maybe I should give it a try!

Interview Pass Rates

Editor usage is only one part of the story, however. Not all usage is equal. I wanted to see which editors are used by the best programmers. To look at that, I pulled data on how Triplebyte candidates performed during our interview, grouped by the editor they used:

editor-performance-margin.png This chart shows the rates at which each editor's users pass our interview compared to the mean pass rate for all candidates. First, notice the preeminence of Emacs and Vim! Engineers who use these editors pass our interview at significantly higher rates than other engineers. And the effect size is not small. Emacs users pass our interview at a rate 50% higher than other engineers. What could explain this phenomenon? One possible explanation is that Vim and Emacs are old school. You might expect their users to have more experience and, thus, to do better. However, notice that VS Code is the third best editor—and it is brand new. This undercuts that narrative a bit (and makes VS Code look even more dominant).

Do Emacs and Vim users have some other characteristic that makes them more likely to succeed during interviews? Perhaps they tend to be more willing to invest time and effort customizing a complex editor in the short-term in order to get returns from a more powerful tool in the long-term?

On the negative end, engineers who use Eclipse, intelliJ and Visual Studio pass our interview at lower rates. What do Eclipse, IntelliJ and Visual Studio have in common? Well, they are all IDEs. However, PyCharm is also a full-featured IDE, and it shows a high pass rate. What else could be going on? Another distinguishing feature of Eclipes, IntelliJ, and Visual Studio is their strong association with Java and C#, so might we merely be seeing the result of a correlation between specific editors and specific languages?

To investigate this, I looked at interview pass rates by language as well:

language-performance-margin.png Java and C# do have relatively low pass rates, although notice that Eclipse has a lower pass rate than Java (-21.4% vs. -16.7), so we cannot fully explain its poor performance as Java dragging it down.

Also, what's going on with Go? Go programmers are great! To dig deeper into these questions, I looked at editor usage by language:

editor-by-language-margin.png The percentages on this graph are per editor. So we can see, for example, that 97% of engineers using PyCharm program in Python (which makes sense — it's in the name). Eclipse is dominated by Java (94%) and Visual Studio is mostly C# and C++ (88%). I can't really say which way the causality goes, but it seems that both the languages (Java, C#) and the IDEs (Eclipse, Visual Studio) are associated with lower pass rates in interviews. This data comes from our internal interviews, but the same result holds for interviews conducted by the outside companies who use our platform.

Triplebyte does not take language or editor selection into account when making interview decisions. And there are tons of great C# and Java programmers out there (and we've helped many of them get jobs with companies on our platform). However, it seems that the average C# or Java engineer who goes through our process does less well than the average Ruby or Go engineer. I have no idea why.

Another finding from this chart is the difference between VS Code and Sublime. VS Code is primarily used for JavaScript development (61%) but less frequently for Python development (22%). With Sublime, the numbers are basically reversed (51% Python and 30% JavaScript). It's interesting that VS Code users pass interviews at a higher rate than Sublime engineers, even though they predominately use a language with a lower success rate (JavaSript).

Experience / Location

To wrap things up, I sliced the data by experience level and location. Here you can see language usage by experience level:

language-by-experience-margin.png Again, rows sum to 100%, so the chart shows what percentage of people with a given level of experience use each language. Most notable here is how popular Python and Java are among candidates with only internships or part time experience. Are we seeing folks who recently graduated from college using the languages in which they were taught?

Then there's editor usage by experience level:

editor-by-experience-margin.png First off, you can see VS Code usage dropping off as experience level increases. It's definitely more popular among junior engineers. You can also see that Vim and Emacs are more popular among more experienced engineers. It seems plausible that this is indeed the main reason why Vim and Emacs users have such a high pass rate in our interviews.

Finally, we looked at the relationship between location and language used:

language-by-location.png I like this chart for what it says about Bay Area geography. On the peninsula, where larger companies tend to be located, you see a lot of Java developers. In San Francisco, where startups dominate, you see more JavaScript.

Conclusions

Here's a summary of the trends in tool usage we saw over the past year along with some intriguing relationships between the tools engineers choose and their programming ability:

Visual Studio Code is on the rise. Over the past year, it has become the most popular editor across the board, and it's gaining ground every month. I wonder if the editor landscape is experiencing a tectonic shift, from a relatively fractured market to a world in which a single editor has over 50% of the market?

Engineers who use old school editors like Vim or Emacs tend to be strong programmers. That said, this finding may be best explained by the fact that these editors are especially popular among more experienced engineers.

Engineers who use Go are also especially strong. If you know why, please let me know.

Engineers who use Eclipse or Visual Studio do less well (on average) in our interviews. This correlation still stands for interviews with external companies who use our platform.

Take all of this with a grain of salt. I want to end by saying that we don't think any of this is causative. That is, I don't recommend that you start using Emacs and Go (or stop using Eclipse and Java) on the basis of this data. There are strong programmers who use every set of tools. The most important thing in any interview is using the tool that you are most comfortable with.

I would really appreciate your feedback. I have a lot of questions about this data, and I would love to know what other programmers think about what we've seen over the past year. Email Ammon at ammon@triplebyte.com or Lyn at lyn@triplebyte.com!

No matter which editor you choose, if you're an engineer interested in being matched with top tech companies in San Francisco, New York, Los Angeles, and/or Seattle, check out our process.

If you're a company interested in hiring great engineers, explore our platform!

Get offers from top tech companies

Take our coding quiz

Liked what you read? Here are some of our other popular posts…

How to Interview Engineers

By Ammon Bartram on Jun 26, 2017

We do a lot of interviewing at Triplebyte. Indeed, over the last 2 years, I've interviewed just over 900 engineers. Whether this was a good use of my time can be debated! (I sometimes wake up in a cold sweat and doubt it.) But regardless, our goal is to improve how engineers are hired. To that end, we run background-blind interviews, looking at coding skills, not credentials or resumes. After an engineer passes our process, they go straight to the final interview at companies we work with (including Apple, Facebook, Dropbox and Stripe). We interview engineers without knowing their backgrounds, and then get to see how they do across multiple top tech companies. This gives us, I think, some of the best available data on interviewing.

Read More

Bootcamps vs. College

By Ammon Bartram on May 19, 2016

Programming bootcamps seem to make an impossible claim. Instead of spending four years in university, they say, you can learn how to be a software engineer in a three month program. On the face of it, this sounds more like an ad for Trump University than a plausible educational model.

But this is not what we’ve found at Triplebyte. We do interviews with engineers, and match them with startups where they’ll be a good fit. Companies vary widely in what skills they look for, and by mapping these differences, we’re able to help engineers pass more interviews and find jobs they would not have found on their own. Over the last year, we’ve worked with about 100 bootcamp grads, and many have gone on to get jobs at great companies. We do our interviews blind, without knowing a candidate's background, and we regularly get through an interview and give a candidate very positive scores, only to be surprised at the end when we learn that the candidate has only been programming for 6 months.

Read More

How to pass a programming interview

By Ammon Bartram on Mar 8, 2016

Being a good programmer has a surprisingly small role in passing programming interviews. To be a productive programmer, you need to be able to solve large, sprawling problems over weeks and months. Each question in an interview, in contrast, lasts less than one hour. To do well in an interview, then, you need to be able to solve small problems quickly, under duress, while explaining your thoughts clearly. This is a different skill. On top of this, interviewers are often poorly trained and inattentive (they would rather be programming), and ask questions far removed from actual work. They bring bias, pattern matching, and a lack of standardization.

Read More
Get the Triplebyte newsletter Subscribe

Get the Triplebyte newsletter

Be the first to get the latest Triplebyte content straight to your inbox.