AI coding tools could make experienced software engineers less productive, a new study suggests

Recent research has unveiled a surprising twist in the realm of software development, suggesting that the introduction of AI coding tools might actually hinder the productivity of seasoned developers. Conducted by Model Evaluation & Threat Research (METR), the study highlights that experienced programmers utilizing AI assistance took 19% longer to resolve coding issues than their counterparts who relied on traditional methods. The study involved 16 developers working on extensive open-source projects they had been engaged with for years. Participants were randomly divided into two groups: one allowed to use AI coding tools and another that wasn’t. Notably, the AI-assisted developers spent a significant amount of their time—over 20%—on tasks related to reviewing AI outputs, waiting for AI responses, or being idle, while those without AI reported more than a 10% increase in active coding time. Interestingly, the developers using AI tools believed their productivity had improved by 20%, despite the study's findings to the contrary. METR researcher Nate Rush expressed his astonishment at the results, noting that the slowdown was unexpected, particularly considering the experienced developers involved in the study. He emphasized that the context of the study was crucial, suggesting that less experienced developers might find AI tools more beneficial. The findings have sparked discussion among industry experts. Steve Newman, co-founder of Google Docs, acknowledged the credibility of the study after thorough examination, remarking that while AI coding tools are not fraudulent, they do have significant limitations at present. One of the contributing factors to the slowdown could be over-optimism regarding the capabilities of AI. Developers initially predicted a 24% reduction in implementation time due to AI assistance. However, the study revealed that skilled developers often perform better when relying on their own knowledge and experience. In fact, AI-assisted participants faced challenges with tasks they were familiar with, indicating that the technology may still be developing its reliability in producing accurate code. With less than 44% of AI-generated code being accepted by the developers, and a notable 9% of their time spent refining these outputs, the research paints a complex picture of AI's role in software development. As coding assistants continue to evolve, some experts believe that the results of this study may change in the future, prompting Rush to suggest that productivity gains from AI could be more individualized than previously thought. This realization encourages developers to approach AI tools with greater discernment, making decisions informed by their unique productivity experiences.