Using AI might actually slow down experienced devs

News
By published

Experienced developers aren't getting any benefit from AI, report claims

ChatGPT coding
(Image credit: Shutterstock)
  • Experienced developers actually spent 19% more time on tasks when using AI, report claims
  • Only 44% of AI-generated code was accepted by experienced devs
  • Developers still feel that work is easier when using AI

Artificial intelligence might not be as beneficial to experienced developers as it can be to new starters and those who are yet to develop the right skills, new research has claimed.

A new study conducted by Model Evaluation & Threat Research (METR) has suggested not only were the developers less optimistic following the study, but the real results suggest that artificial intelligence actually ended up costing them time.

The study found that, with AI, devs spent less time coding and searching, and more time prompting, waiting for and more important, reviewing, AI output. An estimated 9% of the time went on reviewing and cleaning up AI-generated code, with AI suggestions generally on the right tracks but lacking in detail.

AI doesn't actually same these developers any time

Observing 16 experienced developers across 246 real tasks on mature open-source projects that they were already familiar with, the researchers analyzed how the developers interacted with popular tools from Cursor Pro and Claude 3.5/3.7.

Before the study, the 16 experienced developers in question expected to reduce task time by 24% when combining their expertise with artificial intelligence. After the study, they reduced their expectations to just 20%, however post-study analysis reveals that AI actually increased task completion time by 19%.

Fewer than 44% of the suggestions were accepted, with a lack of contextual knowledge and large, complex repositories highlighted as contributing factors to developer slowdown. The study also noted that the experienced developers already had high familiarity with the codebases, leaving little room for AI to add any meaningful value.

However, despite the slowdown, many developers continued to use AI tools because the work felt less effortful, making work feel more pleasant even if it wasn't faster.

"AI capabilities in the wild may be lower than results on commonly used benchmarks may suggest," the research paper concludes.

You might also like

Craig Hale
Craig Hale

With several years’ experience freelancing in tech and automotive circles, Craig’s specific interests lie in technology that is designed to better our lives, including AI and ML, productivity aids, and smart fitness. He is also passionate about cars and the decarbonisation of personal transportation. As an avid bargain-hunter, you can be sure that any deal Craig finds is top value!

You must confirm your public display name before commenting

Please logout and then login again, you will then be prompted to enter your display name.