Study finds AI tools made open source software developers 19 percent slower

A recent randomized controlled trial has unveiled surprising results regarding the impact of AI tools on the productivity of experienced open source developers. Contrary to the common belief that these technologies enhance coding efficiency, the study indicated that developers became 19% slower when utilizing current AI applications. Conducted by researchers at METR (Model Evaluation and Threat Research), the study involved 16 seasoned software developers, each with extensive experience in specific open source projects. The participants were monitored across 246 distinct tasks essential to maintaining their repositories, including bug fixes, feature implementations, and code refactoring. For half of the tasks, developers employed AI tools such as Cursor Pro and Anthropic's Claude, while for the other half, they worked without any AI assistance. Prior to the experiment, developers anticipated that the AI tools would reduce task completion times by 24%. After completing the tasks, they estimated that the AI had made them 20% faster on average. However, the findings revealed a stark reality: tasks aided by AI tools took significantly longer than those completed without them. The METR researchers analyzed screen recordings from a portion of the developers and discovered that, although AI tools initially seemed to save time in coding, testing, and searching for information, these gains were overshadowed by the time spent reviewing AI outputs, prompting the systems, and waiting for AI-generated responses. Furthermore, developers accepted less than 44% of the AI-generated code without modifications, with 9% of the total task duration in AI-assisted segments dedicated to reviewing this code. This study raises important questions about the efficiency of current AI tools in real-world coding scenarios, challenging the narrative that they are always beneficial for software development.