AI coding tools may not speed up every developer, study shows

The landscape of software engineering has seen a significant shift with the rise of AI coding tools like Cursor and GitHub Copilot. These tools are designed to boost productivity by automating code writing, debugging, and testing tasks. Powered by advanced AI models from leading organizations such as OpenAI, Google DeepMind, Anthropic, and xAI, these platforms have demonstrated impressive performance in various software engineering assessments. However, a recent study from the non-profit research group METR raises doubts about the actual productivity enhancements provided by these tools, particularly for seasoned developers. The research involved a randomized controlled trial with 16 experienced open-source developers who tackled 246 real tasks within large code repositories they frequently contribute to. The developers were split into two groups: one that could utilize AI tools like Cursor Pro and another that had to rely solely on traditional coding methods. Before starting, the developers estimated that the use of AI tools would likely cut their task completion time by 24%. Contrary to these expectations, the results revealed that using AI tools increased completion time by 19%. The researchers noted, "Surprisingly, we find that allowing AI actually increases completion time—developers are slower when using AI tooling." Interestingly, only 56% of participants had prior experience with Cursor, the primary AI tool tested, and while the majority (94%) had used some web-based language learning models (LLMs) in their workflows, for many, this was their first interaction with Cursor. Despite being trained on the tool beforehand, the findings suggest that developers should approach the expected productivity boosts from AI coding tools with caution. The METR team posits several reasons for this unexpected slowdown. Developers often find themselves spending excessive time crafting prompts and waiting for AI responses rather than actively coding. Additionally, AI tools can struggle with complex codebases, which were a significant focus of this study. While the authors are careful not to dismiss the potential of AI coding tools entirely, they emphasize that their findings warrant skepticism regarding the universal effectiveness of these technologies. Previous studies have shown that AI tools can indeed enhance developer efficiency, and the researchers acknowledge that advancements in AI are rapid, suggesting that results could differ significantly in just a few months. Ultimately, this research serves as a reminder of the complexities involved in integrating AI into software development, highlighting that while AI tools have made strides in recent years, they may not be the panacea for productivity that many had hoped for.