AI isn't really helping coders write better or more secure code

JavaScript code on a computer screen
(Image credit: Shutterstock / BEST-BACKGROUNDS)

A paper by researchers at Stanford University has found that coders who employed AI assistants such as GitHub Copilot and Facebook InCoder actually ended up writing less secure code. 

What's more, such tools also lull developers into a false sense of security, with many believing that they produce better code using the help.

Nearly 50 subjects, each with varying levels of expertise, were given five coding tasks, using various languages, with some aided by an AI tool, and others without any help at all.

Language games

The authors of the paper - Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh - stated that there were "particularly significant results for string encryption and SQL injection".

They also referenced previous research which found that around 40% of programs created with assistance from GitHub Copilot contained vulnerable code, although a follow-up study found that coders using Large Language Models (LLM), such as OpenAI's code-cushman-001 codex - on which GitHub Copilot is based - only resulted in 10% more critical security bugs.

However, the Stanford researchers explained that their own study looked at OpenAI's codex-davinci-002 model, a more recent model than cushman, which is also used by GitHub Copilot.

They also looked at multiple programming languages, including Python, Javascript and C, whereas the other paper only focused on the latter, which the authors attribute to its inconclusive findings. In fact, in the Stanford paper, those using AI to code in C didn't result in significantly more errors either.

One of the five tasks involved writing a code in Python, and here code was more likely to be erroneous and insecure when using an AI helper. What's more, they were also "significantly more likely to use trivial ciphers, such as substitution ciphers (p < 0.01), and not conduct an authenticity check on the final returned value."

The authors hope that their study leads to further improvements in AI rather than dismissing the technology altogether, due to the potential productivity improvements such tools can offer. They just maintain that they should be used cautiously since they can mislead programmers into thinking they are infallible.

They also think AI assistants can encourage more people to get involved with coding regardless of their experience, who may also be put off by the air of gatekeeping around the discipline.

Via The Register

TOPICS
Lewis Maddison
Reviews Writer

Lewis Maddison is a Reviews Writer for TechRadar. He previously worked as a Staff Writer for our business section, TechRadar Pro, where he had experience with productivity-enhancing hardware, ranging from keyboards to standing desks. His area of expertise lies in computer peripherals and audio hardware, having spent over a decade exploring the murky depths of both PC building and music production. He also revels in picking up on the finest details and niggles that ultimately make a big difference to the user experience.

Read more
A profile of a human brain against a digital background.
Securely working with AI-generated code
AI writer
Coding AI tells developer to write it himself
AI Education
AI in 2025: Moving beyond code generation to intelligent development platforms
Bored frustrated business people working in the office with an efficient robot.
Shut it all down? Microsoft research suggests AI usage is making us feel dumber – but you don't need to panic yet
ChatGPT app on an iPhone
ChatGPT and Google Gemini are terrible at summarizing news, according to a new study
hacker.jpeg
Thousands of GitHub repositories exposed via Microsoft Copilot
Latest in Security
Microsoft
"Another pair of eyes" - Microsoft launches all-new Security Copilot Agents to give security teams the upper hand
Lock on Laptop Screen
Medusa ransomware is able to disable anti-malware tools, so be on your guard
An abstract image of digital security.
Fake file converters are stealing info, pushing ransomware, FBI warns
Insecure network with several red platforms connected through glowing data lines and a black hat hacker symbol
Coinbase targeted after recent Github attacks
hacker.jpeg
Key trusted Microsoft platform exploited to enable malware, experts warn
IBM office logo
IBM to provide platform for flagship cyber skills programme for girls
Latest in News
Apple Watch Ultra 2 timer
The Apple Watch is getting a sleep alarm upgrade it probably should have had 10 years ago
Nikon Z5
The Nikon Z5 II could land soon – here's what to expect from Nikon's rumored entry-level full-frame camera
Google Pixel Watch 3
Google Pixel Watches hit with delayed notifications, crashing, and performance issues following Wear OS 5.1 update
Zendesk Relate 2025
Zendesk Relate 2025 - everything you need to know as the event unfolds
Disney Plus logo with popcorn
You can finally tell Disney+ to stop bugging you about that terrible Marvel show you regret starting
Google Gemini AI
Gemini can now see your screen and judge your tabs