-
AI coding assistants reduced the productivity of experienced software developers by 19%, suggests a new METR study.
-
The study showed that developers were reckless in the AI tools, which expected a 20% productivity boost, even after its use.
-
Critics warn that AI code editors have improved since the study period of February and that the results are location specific.
AI -Code -Editors have quickly become a mainstay of software development, employed by tech giants such as Amazon, Microsoft and Google.
In an interesting turn, a new study suggests that AI tools have made some developers less productive.
Experienced developers who used AI coding tools took 19% longer to complete problems than those who do not use generative AI assistance, according to a new study of model Evaluation & Threat Research (METR).
Even after completing the tasks, participants could not accurately gauge their own productivity, the study said: the average AI-assisted developers still thought that their productivity had won by 20%.
How the study was set up
The study by METR recruited 16 developers with large, open-source repositories on which they have been working for years. The developers were randomly assigned in two groups: they were allowed to use AI coding aid and those who were not.
The AI-assisted codingers could choose which atmospheric coding tool they used. Most chose Cursor with Claude 3.5/3.7 Sonnet. Business Insider reached for cursor for comment.
Developers without AI spent more than 10% coding more time, according to the study. The AI-supported codingers spent more than 20% more time assessing AI outputs, which encouraged AI, waiting for AI or being inactive.
While participants actively cod more time without AI, AI-assisted participants spent more time playing and waiting for AI, evaluating the output and stationaryMeter
A ‘really surprising’ result – but it is important to remember how quickly AI -tools progress
Metr -researcher Nate Rush told BI that he uses an AI code -editor every day. Although he did not make a formal prediction about the results of the study, Rush said that he brought down positive productivity figures that he expected to reach the study. He remains surprised by the negative end result – and warns to get it out of the context.
“Much of what we see is the specificity of our setting,” said Rush, in which he explains that developers without the 5-10 years of expertise of the participants would probably see different results. “But the fact that we found a delay at all was really surprising.”
Steve Newman, serial entrepreneur and co -founder of Google Docs, described the findings in a substitue post as “a pity to be true”, but after a more careful analysis of the study and the methodology he found the study credible.
“This study does not expose AI coding aids as fraud, but it reminds us that they have important limitations (at least for the time being),” Newman wrote.
The METR researchers said they found evidence for several contributors to the productivity delay. Over-optimism was a likely factor: before the tasks were completed, developers predicted that AI would reduce the implementation time by 24%.
For skilled developers it might still be faster to do what you know well. The METR study showed that AI-assisted participants delayed the problems with which they were more known. They also reported that their level of experience made it harder for AI to help them.
AI may not be reliable enough to produce clean and accurate code. AI-assisted developers In the study, less than 44% of the generated code accepted and spent 9% of their time cleaning AI outputs.
Ruben Bloom, one of the developers of the study, placed a responseread on X. Codeer assistants have developed considerably since he participated in February.
“I think that if the result is valid at the moment, that’s one thing, I think if people quote in another 3 months, they will make a mistake,” Bloom wrote.
The hurry from METR acknowledges that the 19% delay is a “point-in-time measurement” and that he would like to study the figure over time. Rush states at the collection meal of the study that the productivity of AI can be more individualized than expected.
“A number of developers told me this really interesting anecdote, who is:” Knowing this information, I feel this desire to use AI more judiciary, “said Rush.” On an individual level, these developers know their actual productivity impact. They can make better informed decisions. “
Read the original article about Business Insider
