AI coding tools have reduced productivity for some engineers: research

AI code editors are the mainstay of software development adopted by high-tech giants such as Amazon, Microsoft, and Google.

With an interesting twist, new research suggests that AI tools can reduce productivity for some developers.

According to a new study in Model Evaluation & Threat Research (METR), experienced developers using AI coding tools took 19% longer to complete the problem than problems that did not use generative AI assistance.

Even after completing the task, participants were unable to accurately measure their productivity, according to the study, and the average AI-assisted developer thought that their productivity had increased by 20%.

How the research was set up

The Metr survey recruited 16 developers with a large open source repository they've worked on over the years. Developers were randomly assigned to two groups, groups that were allowed to use AI coding aid and those that were not.

The AI Assist Coder allows you to select the vibe coding tool you used. Most people selected the cursor on the Claude 3.5/3.7 sonnet. Business Insider contacted Cursor for comment.

Developers without AI actively spent more than 10% of their coding, the study says. AI Assist Coders spent more than 20% of the time reviewing AI output, encouraging AI, waiting for AI or idle.

“Really Amazing” Results – But it's important to remember how fast your AI tools are progressing

Metr Researcher Nate Rush told BI that he uses an AI code editor every day. He had no formal predictions about the findings of the study, but Rush said he wrote down positive productivity figures he expected to reach the study. He remains surprised by the negative end result and notes that he removes it from the context.

“What we're looking at is the specificity of our setting,” Rush said, explaining that developers who don't have five to ten years of expertise in participants are more likely to see different outcomes. “But the fact that we found a slowdown was absolutely amazing.”

Steve Newman, a serial entrepreneur and co-founder of Google Docs, described the findings of the Substack Post as “bad as it's not true,” but after a more careful analysis of the study and its methodology, the study felt reliable.

Related Stories

Business Insider tells the innovative stories you want to know

Business Insider tells the innovative stories you want to know

“The study does not publish AI coding tools as fraud, but it reminds us that there are important limitations (at least for now),” Newman writes.

METR researchers said they found evidence of multiple contributors to slowing productivity. Excessive optimism was one factor. Before completing the task, developers predicted that AI would reduce implementation time by 24%.

For a skilled developer, it may still be quick to do what you know well. A METR study found that AI-assist participants slowed down on issues they were better accustomed to. They also reported that their level of experience made it more difficult for AI to help them.

Also, AI may not yet be reliable enough to write clean and accurate code. The AI-assisted developers in this study accepted less than 44% of the generated code and spent 9% cleaning the AI output.

Ruben Bloom, one of the research developers, posted a response thread to X. Coding assistants have developed quite a bit since joining in February.

“If the results are valid at this point, I think that's one thing. If people are quoting in another three months, I think they're making a mistake,” Bloom wrote.

Metr's rush acknowledges that a 19% slowdown is a “point-in-time measurement” and wants to study the numbers over time. Rush is by the takeout of research that increased AI productivity could be more personalized than expected.

“A lot of developers have said this very interesting anecdote: “I feel the desire to know this information and use AI more wisely,” Rush said. “At a personal level, these developers know the real impact of productivity. They can make more informed decisions.”