“The 19% slowdown observed among experienced developers reflects the real-world friction that integrates stochastic proposals into deterministic workflows rather than the prosecution of AI as a whole,” explains Gogia, with measurements “not just downstream rework, code tunes, and peer review cycles.”
Evidence from a wider industry
The METR findings are consistent with regard to trends identified in Google's 2024 Devops Research and Assessment (DORA) report, based on responses from over 39,000 experts. 75% of developers report using AI tools to be more productive, but the data tells a different story. With every 25% increase in AI adoption, DIP with 1.5% delivery rates and system stability reduced by 7.2%. Additionally, 39% of respondents reported having little or no trust in AI-generated code.
These results contradict previous optimistic research. Research from MIT, Princeton and the University of Pennsylvania analyzed data from over 4,800 developers from Microsoft, Accenture, and Anoter Fortune 100 companies, and found that on average developers using GitHub Copilot completed 26% of tasks. Another controlled experiment led the developer to complete the coding tasks 55.8% faster with GitHub Copilot. However, these studies typically used simpler and more isolated tasks compared to the complex, real-world scenarios examined in the METR study.
