AI tools slow down experienced open source developers
Surprising findings from a 2025 study
A new study by the METR organization reveals surprising findings about the impact of AI tools on the productivity of experienced developers. The results starkly contrast to widespread expectations about the efficiency gains that artificial intelligence will bring.
Study design and methodology
The randomized controlled study was conducted between February and June 2025 and included 16 experienced open-source developers who worked on 246 tasks in mature projects in which they already had extensive prior experience. The tasks were assigned randomly, and developers were either allowed or prohibited from using early 2025-generation AI tools. Cursor Pro and Claude 3.5/3.7 Sonnet were mainly used.
The researchers collected extensive data through 143 hours of screen recordings, source code management systems, interviews, and surveys. This multi-layered approach allowed them to analyze the results from different angles.
The surprising main result
Contrary to all expectations, using AI tools led to a 19% increase in task completion time. This result not only surprised the researchers but also contradicted the predictions of all those involved:
Developer forecasts: 24% time savings expected
Developer post-estimates: 20% time savings assumed
Expert forecasts: Economists expected 39% time savings, ML experts 38%
Root cause analysis: Why AI slowed things down
The researchers identified 20 possible factors for the slowdown and categorized them into four groups. Five factors were found to contribute to the slowdown:
Primary factors of the slowdown
Excessive optimism regarding the usefulness of AI
High developer familiarity with the repositories
Large and complex repository structures
Low AI reliability
Implicit repository context that AI could not capture
Factors with unclear effects
Nine other factors showed mixed or unclear effects, including experimentally induced AI overuse, unrepresentative task distribution, AI-induced problem expansion, and suboptimal AI usage.
Discrepancy between perception and reality
The study reveals a significant gap between AI's perceived and actual impact on developer productivity. This suggests that AI capabilities in real-world work environments may be lower than benchmarks suggest.
Limitations and restrictions
The researchers caution against overgeneralizing the results. The observed slowdown may be specific to the environment studied, in particular:
The high level of developer familiarity with the repositories
The size and maturity of the repositories considered
The AI systems used and their reliability
Conclusion and implications
The study suggests that AI systems with higher reliability, lower latency, and better user guidance can improve developer productivity in similar environments. However, the results illustrate that the path to productivity-enhancing AI support is more complex than initially assumed.
In practice, this means that companies and developers should realistically assess their expectations of AI tools and carefully evaluate the introduction of such systems, rather than assuming automatic productivity gains.