Normally, we use our blog to talk about insights we’ve gathered from our giant mountain of interview data. This one’s gonna be a little different. We’ve just launched two big new features: our data science and DevOps tracks. For this post, we thought we’d share a look into our internal process and talk about how we ship a big project like this on a tight schedule – in other words, let’s talk about the glamorous, tumultuous, life-or-death world of project management!
Back in September, we released our machine learning interview track to meet the growing demand for job offerings in ML and AI. The occasion marked our first release of a specialized interview track since 2017, when we launched our front-end and mobile offerings. But we quickly realized that we still wanted to launch two new ones: data science, because our ML track (which emphasizes coding) wasn’t serving data scientists very well; and DevOps, a valuable role we were not meaningfully testing for at all.
And we decided that we wanted to launch both of them in about four weeks.
October 24: T-minus 28 days
OPERATION CHEEZ-IT [CURRENT OBJECTIVE: IDENTIFY PRODUCT NEEDS]
At this point, we’re about two sprints into our fourth financial quarter. (Like a lot of tech startups, we follow an Agile methodology and a two-week sprint schedule here at Triplebyte.) Our team likes to give each sprint a unique name; we’ve found that having a defined enemy to rally against is good for morale. We recently ran out of alphabetized Sam Jackson movies (godspeed, Zambezia), so now we’re naming them after objects that our product manager Alex can break with his bare hands.
After consulting with C-suite, Aaron (the head of our Product Team) announces our release plan at the beginning of Sprint Number Three: Cheez-It Cracker. We need to move fast here, because the winter holidays are descending upon us. Applicants and recruiters are about to take breaks from the grind and spend time with their families. We want to get our products in front of them before they descend into the month-long food coma that marks the U.S. holiday season.
The first thing we need for our new interview track is content – quiz questions, useful coding challenges, and predictive test material. This will be handled primarily by our Interview Team Lead, Elliott – with a bit of assistance from our in-house developers and the veritable army of technical interviewers we have on contract.
Before we decide what skills to test, however, we have to learn what companies are looking for. We want to show them engineers who would pass their on-sites. And we want to set ourselves up for repeat business, so making sure we achieve a reliable, lasting fit between company and applicant really matters.
Fortunately, companies are happy to say what they’re looking for in the job listings they add to our website. Even though our DevOps track release is still a month out, for example, there are plenty of open DevOps roles posted to our platform already. (The same was true of ML roles before we launched its dedicated track: many of our ‘generalist’ applicants have also picked up valuable expertise in these specialized areas, which our client companies are eager to tap into.) Elliott’s actually been collecting a folder of DevOps listings and identifying their common threads for a while now – we knew this day was coming!
Of course, job listings only tell part of the story. The ones we’ve looked at usually emphasize years of experience in particular roles or with certain technologies, even though our own evaluation has found only a very small – and extremely noisy – correlation between an engineer’s demonstrable skills and the length of their resume. So the next step we take is to book some calls with the managers, engineers, and founders in charge of filling these positions to learn more what skills they're looking for.
October 31: T-minus 21 days
OPERATION CHEEZ-IT [CURRENT OBJECTIVE: CREATE EVALUATION]
In the meantime, another team is getting ready for action: our Product department is starting to spec out the necessary product and process changes to support the new tracks. To protect against industrial espionage, they rendezvous at a top-secret location wearing impenetrable disguises.
There’s a lot we have to get done to make the new interviews ready for primetime – after all, each new track is essentially two products in one. To applicants, we’re offering a job-hunting service; to companies, we’re presenting a pre-vetted smörgåsbord of potential recruits. Each side has their own set of tools and navigates our website in a different way. Our Product Team is actually split to reflect this: we have dedicated product managers for each side of the platform.
Tressia leads the applicant-facing side of our product, so she figures out what we’ll have to add to support our new DevOps and data scientist friends. We’re going to need a whole new candidate flow, for one thing. (Actually two things, as we need one for each new track.) There are also menus and text fields all over the site we have to look at. For example, we need to make sure newly accepted engineers can select DevOps- and data science-specific tools from the drop-down lists in our sign-up questionnaire. (Another case that’s particularly easy to overlook – the text on the “You Passed Our Interview!” certificate!) While Tressia lays out everything that needs to be changed, our designers are drawing up mockups for the new icons and page layouts. For example, they’re going to add DevOps and data science quiz links to the Triplebyte homepage, because it currently looks like this:
Alex, breaker of Cheez-Its, leads the company side. He’s scoping out our landing page, FAQ section, and filtering features to make sure that recruiters and hiring managers will find the new pool of candidates intuitive and convenient to navigate.
Back on the Interview Team, Elliott is hammering out the ideal criteria for our new evaluations. For example, it turns out that precisely identifying what a “DevOps position” encompasses is actually surprisingly difficult. We know that this track in particular will require the most new material from us, so we want to make sure we have a clear goal in mind when we’re making it.
Over the next week or so, Elliott gets in touch with eight different companies on our platform (ranging from seed-stage startups to giants like Palantir) to ask them each a series of questions, such as:
What will your eventual hire be working on from day to day? What tools and tech will they use? What skills should this person have, and how will you evaluate them?
The answers we get are always informative, though not always in the most straightforward way. Many of our client companies have a very clear-cut profile in mind – but it’s often a very different profile from one company to the next. A few of them are... a lot more equivocal. (“We're not quite sure what they'll do, because we don't actually have anyone working in this role right now – we'll define that based on who we hire, we're looking for a senior person who can answer a lot of things.”) When Elliott's not scribbling down their responses, he's canvassing articles and videos to get a clearer picture of the industry standard. We take note of common threads and fundamentals until we have a fleshed-out profile of what skills are called for in each role.
Once we know what we’re looking for, we figure out the best way to test for it. Thankfully, we don’t always have to re-invent the wheel. We have hundreds of questions in our internal test bank already, from the quiz and interview tracks we’ve developed in the past; they’re even categorized conveniently by subject area for the benefit of our matching algorithm. Data science in particular shares a lot of its core concepts with machine learning, so some of our ML resources are also appropriate for that track. We aren’t going to rely on recycled test material all that much (if that was all we needed, the new tracks would be redundant), but the bits we do incorporate offer another advantage – our ML model and interviewing team are trained on those topics already.
November 7: T-minus 14 days
OPERATION d̶o̶n̶g̶l̶e̶ DONUT [CURRENT OBJECTIVE: IMPLEMENTATION]
It’s the start of a new sprint! After briefly reflecting on the trials and triumphs of Operation Cheez-It, we set out on the next phase of product launch: the part where we actually build it. This is where the Engineering Team steps in, going over the specs we’ve established and making the necessary changes. Our Product and Engineering teams convene every Thursday to synchronize and swap notes; the sprint officially kicks off the next morning with a stand-up meeting to establish tasks and brainstorm new items for Alex to break. (Another, shorter, smaller meeting is held afterward to narrow down the suggested set of items to one suitably flimsy and expendable.) The meeting is adjourned (and the new sprint announced) with a public display of athletic prowess.
Our in-house engineers are afforded quite a bit of autonomy. We collaborate and work together a lot too, of course, but we focus on upfront communication to get on the same page, and then giving each other space and time to get in the zone and settle into a comfortable workflow. As long as you’re proactive in driving your projects to completion (which includes staying on top of your own code reviews and feature testing, and promptly reaching out to others if a blocker appears), you can mostly arrange your own agenda.
While chasing down every little segment of code that needs to be tweaked could prove tedious, we have a tool in our belt to help streamline the process. Remember, the last time we launched a track was less than two months ago, and we had an inkling at the time that we’d want to be able to re-trace our steps afterward. So this time around, we can refer to the great Index of Code Changes: an internal document that enumerates (and links to) every commit and pull request we made for the ML track launch. For every new quiz question, job filter, and template we’re about to add, we can just jump straight to the relevant portion of our codebase from The Index.
With the official release now in sight, we’re also beginning to look at announcement and marketing options. While the Interview Team and Product Team (of which ‘Product Engineering’ is a subset) have handled almost all of the work up to this point, “getting the word out” is a weighty task in itself that will pull in the rest of the company. One of the first teams to get sucked into its gravity well is our Writing department (of which your humble author is a subset). Our in-house writers are primarily responsible for writing company-facing profile summaries – succinct articles that encapsulate what each candidate on our platform has to offer in terms of skills and experience. (Imagine a LinkedIn blurb crossed with a critic review, and you’ve got the gist.) Every time an engineer passes our interview, their interviewer’s notes are passed to the brilliant, talented, and heartbreakingly gorgeous members of the Writing Team – a crack squad of silver-tongued wordsmiths selected for our ability to synthesize interview scores, off-the-cuff interviewer commentary, and oft-laconic resumes into columns of scintillating prose. Nonetheless, we are not trained DevOps or data science experts ourselves, so now that the interview content for the new tracks has mostly stabilized, we meet with the Interview Team (i.e., the resident experts) in order to acquire a high-level overview of each field and learn which concepts and skills are important to talk about. Even though the first DevOps and data science candidates have yet to be admitted, the writers are going to apply our newfound knowledge quite heavily over the next couple of weeks – using it to compose high-quality content pieces like the one you’re reading now. Not only will this draw more eyeballs when we launch – the research we do as writers for our respective blog posts is a good way to supplement the new material we’re learning.
November 14: T-minus 7 days
OPERATION d̶o̶n̶g̶l̶e̶ DONUT [CURRENT OBJECTIVE: GENERATE DEMAND]
We’re in the home stretch now! Fortunately, we’ve finally laid down just about all of the engineering work we need for the launch – all that’s left is to make sure that the rest of the company (collectively, the Business Team) is prepared to support our new candidates and customers. Like the Product Team, the Business Team is split into two parts to handle each “half” of our product. On the company-facing side, we have a Marketing & Sales department (to bring new companies to our platform) and a Customer Success team (who serve as liaisons for the companies that we already have). Their equivalents on the candidate-facing side are known internally as the Growth Team (which helps us reach new engineers) and the Talent Managers (who support the applicants going through our process), respectively.
(Some applicants are puzzled about how, and why, using Triplebyte is free for them. The secret is that we’re paid by our client companies: qualified engineers command a high premium on the market, and headhunting for them tends to be an expensive chore rife with false signals, so employers are happy to compensate us for making their lives easier.)
For the Talent Managers, life won’t change much – their job is to guide applicants through the stormy sea of job searching, equip them with the resources and information they need to pass on-sites, and advise them on how to negotiate when the offers roll in. It’s an important part of our business model that we’re proud to offer, but fortunately it isn’t one that’s going to need too much tweaking before the new DevOps and data science candidates arrive.
For the rest of the business team, though, it’s a different story. The Growth Team, for instance, is brainstorming ways to court a subset of engineers that our quiz and interview haven’t historically catered to. Referencing silly memes in our ads is great for maxing out our exposure and notoriety – and for accessorizing the office (we still have the unicycle from the Dat Boi photoshoot!) But it's an approach that's not so apt for targeting a specific skillset or showing what we have to offer. Fortunately, we still have plenty of options here – for example, presenting a short question or two from the new quiz in a sidebar ad (with a hyperlink to our website) is a good way to show off the new content while appealing directly to our audience.
On the company side, our Customer Success operatives are putting the word out to our clients that our specialized pipeline for DevOps engineers and data scientists is going to be available soon. This is welcome news to the companies who’ve been looking for them already, and encourages the rest of our clientele to add their own open roles to our site – when we finally admit dedicated candidates for those tracks, they’ll have plenty of opportunities ready to go. Meanwhile, Marketing and Sales are developing and rehearsing the messages they’ll use to announce the release and pitch it to specific customers. We also send out a reminder email blast to every company on our platform, just in case – not all of them are assigned to one of our Customer Success agents, which is a perk companies unlock through our subscription model. (Other subscription perks include ATS integration, access to our full network of previously-vetted engineers, advanced filtering options, and heated seats.)
November 21: T-minus 3... 2... 1...
It’s zero hour! After two exhausting sprints have passed – to say nothing of the innocent snack foods torn asunder along the way – the quiz content is finalized, our interviewers are primed, the specs are implemented, and the bugs are ironed out. The writers’ blog posts are proofed, the biz team is briefed, the banner ads are about to unfurl. The atmosphere is one of breathless anticipation. We’ve all been juggling a lot the last few weeks.
At last, the decisive moment is upon us. The office gathers in silence behind Shade 14 welding glass as Elliott removes two small keys from a reinforced lockbox. He presents one to each of our product leads, both clad head to toe in shiny reflective hazmat suits. A murmur ripples through the crowd as they step up to the console, purpose-built for the occasion. After exchanging a delicate nod, moving in perfect synchrony, they slot their keys into the mechanism and rotate them in unison. There is no sound but the absence of sound, the cacophony of HVAC systems and Howard Street traffic suddenly hushed to a standstill. The overhead lights sparkle briefly from the power surge before they blue-shift into the ultraviolet. The office plants, drained of life-force, wilt and wither into husks.
Just a moment later the world fades back into normality, the bustling hubbub and watery morning sunlight of downtown SF restored. A happy sigh passes over the office – and then chuckles, hugs, and high-fives. Another successful sprint! We made it.
Next one starts tomorrow.
Triplebyte helps engineers find great jobs by assessing their abilities, not by relying on the prestige of their resume credentials. Take our 30 minute multiple-choice coding quiz to connect with your next big opportunity and join our community of 200,000+ engineers.