It's like a new version of “The Tortoise and the Hare.” A group of experienced software engineers took part in an experiment where they were tasked with completing part of a task with the help of an AI tool. Thinking like fast rabbits, developers hoped that AI would help them work faster and be more productive. If anything, technology has slowed it down. Given the circumstances of the experiment, the turtle approach without AI would have been faster.
The results of this experiment, part of a recent study, were a surprise to the software developers responsible for using AI, as well as the study's authors, Joel Becker and Nate Rush, technical staff members at the nonprofit technology research organization Model Evaluation and Threat Research (METR).
The researchers had 16 software developers with an average of five years of experience perform 246 tasks. Each task is part of a project you were already working on. For half of the tasks, developers were allowed to use AI tools, and most chose code editors Cursor Pro or Claude 3.5/3.7 Sonnet. For the other half, the developers performed the tasks themselves.
Software developers who believed that AI tools could increase productivity predicted that the technology would reduce task completion time by an average of 24%. Instead, thanks to AI, their task time increased by 19% compared to when they were not using the technology.
“While we would like to believe that using AI in our work didn't reduce productivity, it's possible that AI wasn't as helpful as we had hoped, or even that it hindered our efforts,” study participant Philipp Burkhardt wrote in a blog post about his experience.
Why AI slows down some workers
So where did the rabbit go astray? The study found that experienced developers were more likely to be working in the middle of their own projects with plenty of additional context that the AI assistant didn't have, meaning they had to retrofit their own agendas and problem-solving strategies to the AI's output, and they also spent plenty of time debugging it.
“The majority of developers in the study talked about the fact that even though they generally get AI output that is useful to them, and AI often does very impressive work or some type of very impressive work, these developers said that they still have to spend a lot of time cleaning up the resulting code in order to actually make it fit their projects,” said study author Rush. luck.
Other developers were wasting time writing chatbot prompts and waiting for the AI to generate results.
The findings contradict lofty promises about AI's ability to transform the economy and workforce, including boosting U.S. GDP by 15% by 2035 and ultimately increasing productivity by 25%. In fact, many companies are yet to realize a return on their AI investments. According to an MIT report published in August, only 5% of 300 AI implementations achieved rapid revenue acceleration. According to a Harvard Business Review and Analysis Services research report published last month, only 6% of companies fully trust AI to perform core business practices.
But Rush and Becker avoid making sweeping claims about what their findings mean for the future of AI.
First, the sample in this study was small and not generalizable, and included only specialized groups who were completely new to these AI tools. The study also measured technology at a specific point in time, and the authors say it does not exclude the possibility that AI tools will be developed in the future that actually help enhance developer workflows.
Broadly speaking, the aim of this study was to put the brakes on the heavy adoption of AI in the workplace and elsewhere, recognizing that more data on the real-world effects of AI needs to be known and accessible before further decisions are made about its application.
“Some of the decisions we are making now regarding the development and deployment of these systems can have very significant consequences,” Rush said. “If you're going to do it, make high-quality measurements instead of just taking the obvious answer.”
AI’s far-reaching impact on productivity
Economists are already arguing that METR's research is consistent with the broader narrative on AI and productivity. Aneesh Raman, LinkedIn's chief economic opportunity officer, said AI is starting to chip away at entry-level jobs, but benefits for skilled workers such as experienced software developers could decline.
“For people who already have 20 years of experience, five years in this specific example, if their existing way of working is already working well for them, it may not be our primary job to look for them and force them to use these tools,” said Anders Humram, assistant professor of economics at the University of Chicago Booth School of Business. luck.
Humram similarly conducted research on the impact of AI on productivity. In a May labor study, he found that among 25,000 workers in 7,000 workplaces in Denmark, a country that has adopted AI as well as the United States, productivity increased by just 3% for employees using the tool.
Humram's research supports the argument of MIT economist and Nobel laureate Daron Acemoglu that markets are overestimating the productivity gains from AI. Acemoglu argues that only 4.6% of operations within the U.S. economy will be made more efficient by AI.
“If companies rush to automate everything, even processes that shouldn't be automated, they will waste time and energy and miss out on any of the promised productivity benefits,” Acemoglu previously wrote. luck. “The hard truth is that increasing productivity through any technology requires organizational alignment, a variety of complementary investments, and improving employee skills through training and on-the-job learning.”
Examples of decreased productivity among software developers demonstrate the need to think critically when implementing AI tools, Humram said. While previous research on AI productivity has focused on self-reported data or specific, narrow tasks, data on the challenges of skilled workers using the technology complicates the picture.
“In the real world, many tasks are not as simple as just typing them into ChatGPT,” says Humlum. “Many experts have extensive experience [they’ve] The accumulated valuable expertise is very useful and one should not ignore it and abandon the accumulated valuable expertise.
“I would like to take this as a good reminder to be very careful about when you use these tools,” he added.
The first version of this story was Fortune.com July 20, 2025.
