A spy in your pocket? How the UK’s proposed on-device nude image blocking could work in reality

This year, London Tech Week is set to be remembered for an 'innovation' in how we use our devices — but not one that tech fans would expect or, perhaps, want.

On its opening day on Monday, British Prime Minister Keir Starmer took the stage at the Olympia event and gave Big Tech an ultimatum: Apple and Google have three months to implement on-device scanning technology to prevent children from accessing explicit images.

"If we are serious about unlocking the opportunities that tech can bring, then we must also be serious about protecting our children from those who look to abuse it," said Starmer in his speech.

For the government, the move is non-negotiable to stop the rise in online child sexual abuse material (CSAM) and grooming incidents. It's also a feasible task, according to Starmer.

"This is not an impossible challenge," he said, threatening to "change the law" if tech companies don't comply.

A wave of privacy backlash was quick to follow online, especially among privacy tech providers and advocates.

Talking to Bloomberg, President of the encrypted messaging app Signal, Meredith Whittaker, went as far as to say that her company "would rather exit a market than undermine the technical guarantees that people trust for their privacy".

So, is on-device scanning the solution to children's safety that we were waiting for, or a disproportionate response set to erode privacy and security and expose confidential communications? Let's lay out all we know.

How on-device scanning could work

Right now, we know very little about how these scanning technologies will work in practice. But we do know that Apple, Google, and other companies have been making progress with on-device detection functionalities and children's safety.

After introducing broader child safety features and age checks for UK iPhone users, this week Apple unveiled some new tools to help parents protect kids and teens online. These include a simpler way for guardians to manage child accounts and age ratings in the App Store, alongside a SensitiveContentAnalysis framework for developers to build apps that can check for and blur nudity.

Communication Safety is another tool that Apple says will enhance child safety while also safeguarding privacy. Turned on by default for accounts under 18, the feature automatically blurs explicit images received via iMessage, AirDrop, FaceTime video messages, and shared photo albums.

Google has also been making progress. Parents using Android can already manage screen time, track the location of their children's devices, and block camera access. The Big Tech giant also includes a Sensitive Content Warning feature on Messages to detect and blur explicit images.

These are certainly a good start, but none address the more regularly used methods of messaging communications among teens, like Snapchat and Instagram.

Equally, blurring explicit harmful content is one thing, but blocking it altogether is a step further that requires a very high bar for precision to avoid unintended consequences and trigger false positives

That balance, however, is something another company promises to have found.

UK cybersecurity firm SafeToNet — and the linked children's online safety advocate group SafeToNet Foundation — have developed HarmBlock: a technology that works at the operating system level to detect and block explicit images in the camera app, messages, and even encrypted environments like WhatsApp.

Talking to TechRadar, SafeToNet's Co-Founder, Sharon Pursey, explains that HarmBlock works by scanning for image pixels on three different levels: at the screen, on storage, and at the point of broadcast.

Pursey is assured that the rate of accuracy is 98% plus and says that HarmBlock can easily distinguish between an explicit sexual image and an innocent picture of someone on a beach, for example.

So, are SafeToNet, Google, and Apple working together to build what the government's asking for? "I can't deny or confirm," Pursey told TechRadar.

A Home Office representative, however, confirmed to us that the "technology is already there," but did not specify which ones. The Home Office also told us that what the government is now asking is "to apply it differently".

Can scanning ever be private and secure?

Experts have long warned against the idea of so-called client-side scanning. Back in 2023, experts' criticisms led UK lawmakers to decide to halt any scanning requirements within the Online Safety Act until it was technically feasible to do so privately and securely.

Now, the government appears to be convinced that that day has arrived, but technologists and privacy advocates aren't buying it.

From a technological point of view, Mullvad VPN's Founder, Daniel Berntsson, believes that operating system-level scanning will inevitably harm the trust that users have in their devices. And this, he explains, will consequently impact people's privacy rights.

"No one will be able to trust their ultimately government-controlled device for anything requiring privacy, including, among lots of other things, many uses of VPN services," Berntsson told TechRadar.

Robin Wilton, Internet Society's Senior Director for Internet Trust, echoes these concerns. He argues that these functionalities will introduce security flaws into the OS that could later be exploited by criminals and hostile governments alike.

"Whether it’s done by weakening encryption, keeping copies of users’ keys, or reading their messages before they are encrypted, the bottom line is that users can no longer have confidence in the security or privacy of what they send and receive," Wilton told TechRadar.

In a post on X, the UK's Home Office Secretary, Shabana Mahmood, ensured that "there is no reporting, no data collection, no monitoring, and no images leaving the device".

But, Policy Manager at the Open Rights Group, James Baker, isn't convinced.

"I don't know how the Home Secretary can claim that no data will be lost when they're telling companies to develop this software," Baker told TechRadar. "And if she knows that the software exists, then why isn't she being open about saying what it is they're going to do?"

From her side, Pursey from SafeToNet, assures that HarmBlock has been built in a totally privacy-preserving way. Images, she explains, are never stored, but deleted as soon as they are blocked. Everything happens on the device.

Security experts, however, are also all worried about the risk of mission creep. Once these features are on the device, they argue, it would be easy for either the current or future governments to expand on these scanning abilities.

In a fierce open letter, Signal raised this exact problem (among others) by noting: "We know that mass surveillance and censorship capabilities, however sincere-sounding the promises of those who initiate them are, never remain narrowly scoped."

The age verification paradox

While we need to wait and see how the UK government’s proposed final detection and blocking tools will work in practice, there is another intrinsic requirement that's making privacy advocates worried — mandatory age verification.

To determine whether a device is owned by an adult or a child, all users will have to go through ID checks. This, according to Silkie Carlo, director of Big Brother Watch, "will only result in population-wide ID checks for all of us to use our phones, tablets, and laptops".

Beyond whether that's an ethically proportionate request, there may also be a serious privacy and security aspect. Current age verification methods have so far proven to be inadequate in protecting people's sensitive data. The exposure of 70,000 Discord users’ government-issued ID photos because of a third-party vendor should be a stark reminder.

This was among the reasons that led to a coalition of over 400 scientists calling for a halt on age checks until a "scientific consensus" is reached on the balance of benefits versus harm to the wider population.

This is exactly why, according to Baker from Open Right Group, on-device scanning cannot be done in a privacy-friendly way.

"It is going to harm everyone's privacy because everyone is going to have to go through a digital ID checkpoint just to access their device and to access the internet," he told TechRadar.

What's next?

It's hard to say at this stage whether the UK plans to force locked devices on children is the right way to address online harms, nor if it's really possible to do so without breaking devices' privacy and security for all Brits.

What's certain, however, is that this is only the first step in the current commitment to making UK kids safer online.

Drawing on the results of its safety consultation, the government is expected to announce a social media ban for teens next week. According to indiscretions reported by Politico, restrictions will target specific features like livestreaming and disappearing messages.

For digital rights experts, however, a ban isn't a real fix. Commenting on this point, Wilton from the Internet Society told us: "I think secure and privacy-preserving age verification technology can be developed — but that doesn’t solve the human or societal parts of the problem".

It’s a position shared by child safety group, 5RightsFoundation, which has urged the government to start challenging the harmful tech companies' business model rather than policing kids.

Baker, from Open Rights Group, remains optimistic that it isn't too late to reverse this global direction, though. He told TechRadar: "I think it's going to spawn at some point a new global movement of people who are fighting for their free expression rights and open access to the internet. So, watch that space."

With more threats to come for digital freedom expected next week, we may not have to wait too long.

Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews, and opinion in your feeds. Make sure to click the Follow button!