It's like a new narrative of “Turtle and Hare.” A group of experienced software engineers took part in an experiment where they were tasked with completing some of the work with the help of AI tools. Thinking it like a speedy rabbit, developers hoped that AI would promote their work and increase productivity. Instead, technology has made them slower. The AI-free turtle approach in the experimental context would be faster.
The results of this experiment, published in this month's survey, were amazed by the software developers responsible for using AI, as well as the research authors Joel Becker and Naitrash, and the technical staff at the Nonprofit Technology Research Organization Model Evaluation and Threat Research (METR).
The researchers performed 246 tasks to 16 software developers with an average of five years of experience, each carrying out some of the projects they already worked for. For half of the tasks, developers were allowed to use AI tools. Most of them chose the selected code editor Cursor Pro or Claude 3.5/3.7 Sonnet.
Believing AI tools would make them more productive, software developers predicted that the technology would reduce task completion times by an average of 24%. Instead, AI has inflated task time 19% more than when it was not using technology.
“I want to believe that productivity hasn't been struggling while using AI for my tasks, but it's unlikely that it wasn't as useful or hindering my efforts as I expected.”
Why AI slows down some workers
So where did Knoll go down the road? An experienced developer may have approached his work in the middle of his project in many additional contexts that the AI assistant didn't have. So, research shows that AI assistants didn't have one and had to remodel their agenda and problem-solving strategies into AI output.
“The vast majority of developers who participated in the study speak to the fact that even when they obtain generally useful AI outputs, AI often can do very impressive or very impressive jobs. luck.
Other developers either lost time writing chatbot prompts or waited for the AI to produce results.
The findings of this study contradict a noble promise about AI's ability to transform the economy and workforce, including a 15% increase in GDP by 2035 and ultimately a 25% increase in productivity.
However, Rush and Becker are moving away from making drastic claims about what the findings of their research mean for the future of AI.
For one, the study samples are small and not common, and include only a group of specialized people where these AI tools are new. The study also measured the technology at a particular moment, the authors said that it did not rule out the possibility that AI tools could be developed in the future that would help developers to enhance their workflows.
The aim of this study was broadly to pump the brakes on the passionate implementation of AI in the workplace and elsewhere, recognizing that we need to recognize more data on the actual effectiveness of AI, and that we should make it accessible before making more decisions about the application.
“Some of the decisions we are making now, centering on the development and deployment of these systems, are potentially very high,” Rush said. “If you're trying to do that, you don't just get the obvious answer. Take high quality measurements.”
The broader impact of AI on productivity
Economists have already argued that Metr's research is consistent with the broader narrative of AI and productivity. According to Anesh Raman, Chief Economic Opportunity Officer at LinkedIn, AI is beginning to chip at entry-level positions, but could offer reduced returns to skilled workers, such as experienced software developers.
“For those who already have 20 years of experience or five years in this particular example, it may not be the main job that they need to force them to use these tools if they already work at work with existing methods of work.” luck.
Humlum has similar research into the impact of AI on productivity. He found in a study from May that out of 25,000 workers in 7,000 workplaces in Denmark, countries that consume similar AI as the US have improved a modest 3% among employees using tools.
Humlum's research supports the claim that the market for MIT economist and Nobel Prize winner Daron Acemoglu overestimates productivity gains from AI. Acemoglu claims that only 4.6% of tasks within the US economy will become more efficient with AI.
“Even processes that should not be automated, rush to automate everything, companies waste time and energy and don't get the promised productivity benefits,” Acemoglu wrote previously luck. “The difficult truth is that to gain productivity from any technology requires organizational coordination, a complementary range of investments, and improving worker skills through training and hands-on learning.”
It shows the need for critical thinking about when AI tools will be implemented if software developers are hampered, Humlum said. Previous studies on AI productivity have considered self-reported data or tasks that contain specific tasks, but data on challenges from skilled workers using this technology complicates the photographs.
“In the real world, many tasks are not as easy as entering them into ChatGpt,” says Humlum. “Many experts have a lot of experience [they’ve] It is accumulated, so beneficial, we should not ignore it and give up on the valuable accumulated expertise. ”
“I see this as a good reminder to be extremely cautious about when to use these tools,” he added.
