Over 100,000 programmers took Triplebyte’s quiz last year. The data below reveals how five multiple-choice questions easily separate the real software engineers from the rest.

Like the infamous decade-old Fizz Buzz Test, these questions are trivial for anyone who can build software at a professional level, but are likely to stump anyone who can’t hack it. The questions below are the first five on our quiz, and 98% of successful engineers answer at least 4 of 5 correctly. I’m confident that if you’re an engineering manager running an interview, you wouldn’t give an offer to someone who performed below that line.

Results

Before I show the questions, here’s the data on how successful engineers are based on how many questions they answer correctly:

Success versus Questions Answered

I’ve defined “successful” as someone who receives at least one inbound message from a company that matched their personal preferences, and I normalized to the 5/5 group. Engineers who correctly answered 4/5 or 5/5 accounted for 98% of all successful users on our platform. In contrast, 3/5 or below indicated near-certain failure.

If you’re concerned about circular reasoning, note that we train our quiz scoring ML model against downstream outcomes. The Triplebyte network only works because we help good programmers get filtered inbound opportunities that they actually like, based on skills, regardless of resume. And if you read the questions below, I’m sure you’ll agree that these all fit solidly into the FizzBuzz category!

Questions

Here are the five questions. (Note: these are the first five questions from our most popular “Generalist” quiz. If you take one of our specialized quizzes for ML, Data Science, DevOps, Front-End, iOS, or Android, you’ll see different questions, but the same concept applies.)

Question 1 of 5: What kind of SQL statement retrieves data from a table?

This question basically asks, “Have you ever seen a SQL query before?” 80% answered correctly.

Question 2 of 5: Fill in the missing line of code

Basic imperative logic fill-in-the-blank. 76% answered correctly.

If you’re concerned that we’re asking in Python syntax, I’d argue that this is just a convenient pseudocode language for this example, and any programmer should be able to select the correct answer even if they’ve never touched Python before.

Question 3 of 5: Why is caching used to increase read performance?

Caching comes up in so many contexts that it’s basically a universal CS concept, and an incredibly practical one in day-to-day software engineering. 89% answered correctly. Wow! That’s the highest of the five.

Question 4 of 5: Which of the following is used to maintain a user's logged-in state as they browse multiple pages on a website?

“Do you know how websites work?” 80% answered correctly.

Question 5 of 5: What is the value of z after the following code runs?

“Can you read code?” While presented here in JavaScript syntax, the dictionary/hash/associative array/map concept will be familiar to anyone who ever touched a language more powerful than BASIC. (Nothing against BASIC, my own first programming language!)

Only 62% answered correctly. This is, by far, the lowest of the five questions. Personally, I’m very surprised that this one is harder than Question #2, but that’s the data, and it’s statistically significant (p<0.001). (If you have any ideas on why we’re losing so many people on this one in particular, let me know -- email below.)

Data

Out of over 100,000 software engineers who took this quiz last year, here’s the distribution of actual quiz scores:

Distribution of actual quiz scores

Surprisingly, only 42% of people got 5/5 correct! The top 67% correctly answered at least 4 of 5. (And that 67% accounted for 98% of success on our platform.)

Performance on the five questions is not independent. Multiplying together the individual question probabilities, we’d expect only 27% to get 5/5 correct, but the real quiz data shows 42% achieving a perfect score. This is 55% higher than expected if each question was an independent random variable. In fact, a simple model treating all questions as independent implies a peak at 4/5, as well as more weight in the 3/5 bin than the real data:

chart_model.png

The correlation between questions present in the real quiz data suggests that we’re measuring an underlying factor like, “Are you a software engineer?”

Grade yourself

If you’d have correctly answered at least 4 out of 5 of the questions above without any help, you passed “FizzBuzz 2.0” and should probably take the Triplebyte quiz. (If 3/5 or lower, I’ll be honest: it probably won’t help you much.)

After taking the quiz, you’ll see your scores and have the option to create a profile that lets tech companies contact you when they have opportunities that match your preferences, like “company size 500+,” “remote only,” “inclusive workplace,” “min salary $X,” or even “pet friendly offices.” Whatever you like. We work with hundreds of companies, from public FAANG giants and private unicorns down through stealth-mode startups, and it’s easy to block your current employer from seeing your Triplebyte profile.

Discussion

Categories
Share Article

Continue Reading