ChatGPT and Google Gemini are terrible at summarizing news, according to a new study

ChatGPT app on an iPhone
(Image credit: Shutterstock / Primakov)

  • A new study from the BBC says AI chatbots are unable to accurately summarize news
  • The study asked ChatGPT, Gemini, Copilot, and Perplexity to summarize BBC news articles
  • 51% of responses had 'significant issues' and 19% introduced factual errors

A new study from the BBC has found that four of the world's most popular AI chatbots including ChatGPT are inaccurately summarizing news stories.

The BBC asked ChatGPT, Copilot, Gemini, and Perplexity to summarize 100 news stories from the news outlet and then rated each answer to determine just how accurate the AI responses were.

The study found that "51% of all AI answers to questions about the news were judged to have significant issues of some form." and "19% of AI answers which cited BBC content introduced factual errors, such as incorrect factual statements, numbers and dates."

The study showcases multiple examples of inaccuracies that showcased differing information to the news it was summarizing. The examples note that "Gemini incorrectly said the NHS did not recommend vaping as an aid to quit smoking" and "ChatGPT and Copilot said Rishi Sunak and Nicola Sturgeon were still in office even after they had left."

Inaccuracies aside, there's another crucial finding. The report found that AI "struggled to differentiate between opinion and fact, editorialised, and often failed to include essential context."

While these results are unsurprising considering how often we see issues with news summarization tools at the moment, including Apple Intelligence's mix-ups that have led Apple to temporarily remove the feature in iOS 18.3, it's a good reminder not to believe everything you read from AI.

Are you surprised?

From the study, the BBC concludes that "Microsoft's Copilot and Google's Gemini had more significant issues than OpenAI's ChatGPT and Perplexity,"

While this research doesn't necessarily give us much more info, it validates the skepticism towards AI summary tools and emphasizes just how important it is to take information from AI chatbots with a pinch of salt. AI is developing rapidly and large language models (LLMs) are released almost weekly at the moment, so it's to be expected that mistakes will happen. That said, from my personal testing I've found inaccuracies and hallucinations to be less frequent now in software like ChatGPT than it was just a few months ago.

Sam Altman said in a blog post yesterday that AI is progressing faster than Moore’s law and that means we'll continue to see constant improvements to software and how it interacts with the world around it. For now, however, it's probably best not to trust AI for your daily news, and if it's tech-based you may as well stick with TechRadar instead.

You may also like

John-Anthony Disotto
Senior Writer AI

John-Anthony Disotto is TechRadar's Senior Writer, AI, bringing you the latest news on, and comprehensive coverage of, tech's biggest buzzword. An expert on all things Apple, he was previously iMore's How To Editor, and has a monthly column in MacFormat. He's based in Edinburgh, Scotland, where he worked for Apple as a technician focused on iOS and iPhone repairs at the Genius Bar. John-Anthony has used the Apple ecosystem for over a decade, and is an award-winning journalist with years of experience in editorial.

You must confirm your public display name before commenting

Please logout and then login again, you will then be prompted to enter your display name.

Read more
A hand reaching out to touch a futuristic rendering of an AI processor.
What are AI Hallucinations? When AI goes wrong
Google TV smart interface showing movies page
Google TV is testing AI news summaries on the home page, and I can't think of anything I want less
ChatGPT vs Gemini comparison
I compared GPT-4.5 to Gemini 2.0 Flash and the results surprised me
Bored frustrated business people working in the office with an efficient robot.
Shut it all down? Microsoft research suggests AI usage is making us feel dumber – but you don't need to panic yet
Apple Intelligence
Apple's AI headlines are more of a break from reality than breaking news
A laptop screen showing the Google and ChatGPT logos
Goodbye Google? People are increasingly switching to the likes of ChatGPT, according to major survey – here’s why
Latest in Artificial Intelligence
An iPhone showing the ChatGPT logo on its screen
5 better prompts to use with ChatGPT
AI fashion
I asked ChatGPT 4o, Gemini Live, and Siri what to wear, and only one could really help me look my best
Gemini on a mobile phone.
Worryingly, Google Gemini’s new AI image generation features can be used to remove watermarks from images and I'm concerned
Apple Intelligence
It’s crunch time for Apple Intelligence as Apple execs look for a solution to the company’s AI woes
A super close up image of the Google Gemini app in the Play Store
It's official: Google Assistant will be retired for phones this year, with Gemini taking over
Super Mario Odyssey
ChatGPT is the ultimate gaming tool - here's 4 ways you can use AI to help with your next playthrough
Latest in News
Metroid Prime 4
I reckon the Nintendo Switch 2 could launch with Metroid Prime 4 – here’s why
Samsung Galaxy Z Fold 6
New rumors predict a foldable iPhone will launch next year – and cost almost twice as much as the iPhone 16 Pro Max
Pebble smartwatch countdown
Pebble confirms its smartwatch announcement is just hours away
Logo of YouTube Shorts
Is YouTube auto-playing Shorts when you open the app? Well, you’re not alone - here’s how to fix it
Google DeepMind panel discussion
“More sovereignty and protection” - Google goes all-in on UK AI with data residency, upskilling projects, and startup investments
Nintendo Switch 2
Nintendo Switch 2 expected to have AI upscaling and I can't wait to finally play Tears of the Kingdom with upgraded graphics