‘Vanity metrics’ are jeopardizing AI ROI

From boardroom meetings to LinkedIn posts, it’s hard to escape people shouting about their latest achievements with AI. And it’s true that AI tools are unlocking remarkable new frontiers for enterprise organizations.

But for all the noise, many of the claims shared proudly on social media ultimately amount to hot air; celebrating results that don’t mean much to the company or its audience.

These so-called “vanity metrics” cloak AI projects in positive affirmations, but fail to deliver true insight – and as a result, they hold back AI adoption and prevent enterprises from accessing its true value.

Vanity metrics might look impressive in a slide deck, but they fail to measure real business outcomes. This is a familiar trap. Many will remember the early days of social media, when companies boasted about follower counts rather than tracking if the right audiences are engaging and influencing sales revenue.

These surface-level metrics miss the ‘so what’ and highlight that organizations are struggling to measure and communicate the true value of AI.

Without a clear view of value, many organizations are embarking on AI projects without a defined path to ROI. In 2024, we commissioned a report that found only 32% of companies understood how to measure ROI on AI projects.

Since then, a recent report has found that nearly half of businesses still lack structured ROI frameworks. Clearly, organizations are unsure about the route to value and afraid to share that they’re flying blind.

Under pressure to appear on the front foot, enterprises adopt metrics that look good on paper, but are really more hype than help. And when we’re vague about value, we risk sacrificing AI projects on the altar of false confidence.

More work, no pay-off

Measuring the wrong things as proof of success can lead to a false sense of confidence in AI projects. A reliance on vanity metrics gives organizations a skewed perspective on project success, while obscuring how an AI project is performing in a wider context. Surface-level results untethered to business objectives give no true indication of value.

Say you’re onboarding a new AI agent to take over customer service processes, like Klarna did last year. A bot might emulate the work of more than 853 full-time human agents, handle queries at an unprecedented pace, and reportedly save the firm $60m in costs. These metrics sound impressive on the surface.

But they may only capture half the picture. Klarna was just months into this new initiative before it made a swift U-turn towards rehiring human agents, having cut too many humans out of the loop far too quickly. Despite the impressive results shared in public, things clearly weren’t going to plan behind closed doors.

This cautionary tale is a reminder that, however tempting it is to tout shiny metrics, the wider context across the business may tell a different story. The reality on the ground could be more work for remaining staff, hidden costs, and an increase in disgruntled customers.

This kind of unconditional affirmation can provide false assurances to the boardroom that a pilot or roll-out is a success, when in reality it could benefit from a strategic reset or course correction.

Measuring what matters to you

In the face of hype-fueled AI declarations, I’m a firm believer that success can’t be boiled down to a single metric.

Tracking the granular detail of how AI speeds up the process of monitoring, reviewing, replying, and filing data subject access requests, for example, might not trend on LinkedIn or get you featured in the broadsheets, but it will tell you precisely whether AI is addressing the problem it was deployed to solve.

These insights are layered, accrue over time, and can’t be boiled down to a topline figure.

Like any other enterprise tech solution, AI projects need clear, developed objectives and solid KPIs tied to business outcomes as part of their foundations. Measures of impact will be specific to an organization and its goals, and should track how projects perform in the context of the wider business.

Getting this right from the start ensures that organizations can see the return on investment in real time. Being able to observe, measure, and tweak projects once launched not only ensures sustained value, but also gives a solid justification for further investment into new AI solutions elsewhere in the business.

Following the golden thread of value

Understanding what to measure and tracking the success of AI projects doesn’t have to be complicated. But it does require an understanding of the golden thread of value that should run through every initiative.

The first step for tracking an AI project should always be to define an AI use case as a solution to a specific business problem, rather than thinking of AI as a generic solution to any problem. The best use cases address well-documented pain points, often repeatable tasks that can be most readily automated.

KPIs should be identified upfront and aligned to business objectives. If we take a customer service chatbot as an example, KPIs could consider how fast queries are responded to, but should also track customer satisfaction scores, repeat contacts, and escalation rates to human agents.

Anchoring these to bigger business objectives, like customer retention and regulatory compliance, is what reveals the full picture of the solution’s success. KPIs should be quantified upfront and tracked early, with reporting continuing over the course of months, not just days or weeks.

The impact of AI solutions will evolve over time, so ongoing reporting is vital for keeping projects on track.

Assigning ownership is one of the most overlooked aspects of tracking success. When AI projects sit entirely within AI and data teams, they may be optimized for technical performance rather than business outcomes. Who ‘owns’ the solution should be clearly identified, accountable for the results, and have a seat at the decision-making table.

This helps ensure the project aligns with broader business value and, critically, helps move AI projects away from isolated tech solutions to an integral part of business strategy.

AI is increasingly embedded into everyday business operations, but widening adoption does not necessarily reflect widespread understanding of how this technology can generate value.

The organizations that will get lasting ROI from AI initiatives won’t always be the ones sharing the most eye-catching stats on LinkedIn. Instead, they’ll be the ones celebrating real impact on their business objectives and resisting the temptation to mistake noise for progress.

We've featured the best AI website builder.

This article was produced as part of TechRadarPro's Expert Insights channel where we feature the best and brightest minds in the technology industry today. The views expressed here are those of the author and are not necessarily those of TechRadarPro or Future plc. If you are interested in contributing find out more here: https://www.techradar.com/news/submit-your-story-to-techradar-pro