Two months ago, my husband, Kyle, a startup founder, decided to exit his startup and began searching for a product manager position. Dreading the dreaded technical interview, Kyle studied harder than a high school senior studying for the SATs.
In the world of hiring software engineers, product managers, and other technical talent, the technical interview is a candidate's first step: a coding test conducted in person or at home using a shared screen, designed to test a candidate's foundational knowledge.
Kyle borrowed “Cracking the Coding Interview” from a friend, a must-read for people moving between tech jobs. He spent hours each day trawling through test prep websites like LeetCode and solving sample questions from ChatGPT. The chatbot acted as a tutor, helping him get stuck on complex algorithms.
I wondered if candidates could use chatbots in technical interviews, as Kyle did. Software developers are already using these tools to ship better products faster. Is it really that taboo for candidates to get the help of ChatGPT or GitHub Copilot to get through the tests faster?
As a tech reporter, I asked around. I found that a fierce debate was brewing in Silicon Valley over the use of artificial intelligence in technical interviews. A survey of 24 founders, executives, recruiters, and venture capitalists revealed that companies have vastly different approaches, with a widening gulf between early adopters and status quo defenders.
“While AI adoption is on the rise within companies, awareness of using AI as a tool in technical interviews is still elusive,” said Tammy Hung, head of talent at Emergence Capital.
Proponents say banning chatbots from technical interviews is like banning calculators from math tests: if employees are allowed to use these tools to write and review code, then chatbots should be allowed in interviews and candidates' interactions with them should be tested.
“You want to see how a candidate would perform if they were actually in your organization,” says Santosh Sankar, managing partner at supply chain and mobility investment firm Dynamo Ventures. “You want to give them all the tools within your organization to test their ability to solve problems efficiently and creatively?”
It depends on who you ask.
During a technical interview, you may be asked to use a calculator. San Francisco Chronicle/Hearst Newspapers via Getty Images/Getty Images
Linear, a startup that makes tools for software developers, gives employees free use of the tools but asks candidates to disable them during interviews. “We want to understand a candidate's skills and capabilities without the aid of the tools,” said Christina Cordova, Linear's chief operating officer.
Ram Sriharsha, chief technology officer at Pinecone, an early leader in the vector database race, says he doesn't explicitly ban chatbots, but he wants to make sure candidates can solve a problem from start to finish on their own. Zeta, a finance app for couples and families, is decidedly ambivalent. “I don't actually care if they're using Copilot,” said Zeta co-founder Kevin Hopkins. “The reason is, I use the same technology every day to do the job I'm paid to do.”
And yet, nearly everyone I spoke to agreed that the technical interview process is broken.
Typically, interviewers ask candidates to write code to solve a puzzle, such as “find the median of two sorted arrays” or “given a phone number, return all combinations of letters on a keypad that the number could represent.”
The problem with these questions is that they're easy to fudge, says Aline Lerner of Interviewing.io, a company that offers mock interviews with engineers from major companies. “They test how much you remember, not how much you know,” she says.
The internet is full of forums for frequently asked questions, and engineers use websites like LeetCode and HackerRank to learn the data structures and algorithms they will be asked to answer.
The question now is: will chatbots make technical interviews better or worse?
The technical interview is open book
People familiar with the interview process say companies are already modifying tests to prevent cheating: tech-screening and testing app Codility has created a question bank of chatbot-resistant tasks, while testing company CoderPad has a feature that allows interviewers to disable copy-paste or set timers for tasks.
“If you decide to make your technical phone interview open book, the burden of coming up with great questions is greater,” Learner said.
Yossi Kalon, a software engineer at Google who conducted hundreds of technical interviews, now hires for his startup Squidcloud, and prefers using brain teasers and open-ended questions to test candidates' skills in designing real-world software systems. Kalon said he doesn't mind using a chatbot, as long as they're upfront about that.
He also warns candidates that he plans to dig deeper to improve their understanding: A question asking you to design a software subsystem might ask how to balance network traffic across connected servers around the world, or what steps to take when a system fails. He looks to see if candidates are reading from another screen, and he expects quick responses.
Founders are moving in Mr. Calon's direction: Anvil, an early-stage startup that makes software to streamline paperwork, offers applicants a take-home test to design a software system.
“It's great to be able to use AI to augment your development capabilities,” said Anvil CEO Mangit Ng, “but you have to be able to explain what the code is doing, so you can weed out people who just rely on ChatGPT to write the code.”
Big Tech's Reluctance
So far, major tech companies have opted not to use chatbots in technical interviews.
Technical interviews have long been a way for companies like Meta and Google, which get millions of applications a year, to screen candidates early in the hiring process. Learner said the method became popular because of its scale: Employers with hundreds or even tens of thousands of engineers can train their engineers on the same set of questions and then let anyone conduct the interviews.
It also made it easy for companies to benchmark their results against past applicants, says CoderPad CEO Amanda Richardson, who says that in the past 12 months, the company has conducted 1.5 million technical interviews on its platform for companies like Spotify, LinkedIn, Snowflake and Shopify.
Founders often take cues from tech giants like Google when it comes to the interview process. Tolga Akmen/Getty Images
Richardson said large companies are less likely to make big changes to their recruiting processes because it's harder to implement change across an entire organization. “Another concern companies have is that any time you change your recruiting process, you lose predictability in the results,” he added.
CoderPad, which creates virtual environments where interviewers and candidates can code together in real time, added a chatbot to its interface a year ago, but Richardson says only 20% of its enterprise customers have enabled the feature, compared with half of its self-service customers, which include individuals and small startups.
Founders are taking cues from big tech companies when it comes to designing their interview processes, so we're unlikely to see startups switch to this space en masse until companies like Meta and Google do the same.
For that to happen, big companies first need to update their job descriptions, said Tigran Sloyan, CEO of Codesignal, a testing company that has a built-in chatbot for writing code and debugging solutions.“Hiring is a highly regulated field, and you can’t expect someone to demonstrate a skill unless it’s part of the job requirements,” Sloyan said.
Fear of missing out
Natan Fisher, who runs Singlesprout, a recruiting firm that specializes in recruiting tech talent, has seen applicants drop out over the past year after learning that companies ban chatbots from interviewing.
These anecdotes highlight a growing concern for tech companies: If interviews are any indication of corporate culture, candidates will assume, rightly or wrongly, that a ban on chatbots means the company will be slow to adopt these tools into daily operations.
Rahul Vohra's Superhuman uses artificial intelligence to speed up emails, but eschews chatbots for technical interviews. Stephen McCarthy/Getty Images
Speaking to startups, this doesn't seem to be true. Email app Superhuman doesn't allow chatbots in technical interviews. “Of course, we expect to use AI to help us move faster and more productively once the job interview begins,” the company's CEO Rahul Vora said in an email. Pinecone and Linear are also deep in the AI space, and are looking for applicants who can solve problems without the help of a chatbot.
But stereotypes remain and some companies will change their stance on chatbots in interviews to show candidates they are on the cutting edge.
“I think companies that don't adapt will not only be hurt in their ability to vet candidates, but companies that allow it will be able to attract better talent,” Fisher said.
