Why traditional metrics are giving CISOs a false sense of security

The rising threat of cyberattacks has cranked up the pressure for CISOs right at the heart of business resilience. But their job has become all the more difficult.

Our research found that 50% of organizations now carry critical security debt, meaning they have software vulnerabilities that have been left unresolved for longer than a year.

That’s an open invitation for cyber criminals and requires a comprehensive, long-term application risk management strategy to fix it. Yet most organizations still equate more scans with better security. 

This assumption is creating serious security gaps, especially across software supply chains and CI/CD pipelines.

The fact is, not only do traditional security KPIs not measure real security efficacy—they also create a false sense of progress. Recent pipeline and dependency compromises, like the Shai-Hulud supply chain wormware campaign, are a good example of why high scan volume alone does little to prevent breaches.

CISOs need to refocus. The most important metrics measure vulnerability backlogs, undetected attacker dwell time, and existing security controls with proven ability to mitigate real-world threat, not just theoretical risk. Ultimately, depth and validation matter far more than breadth.

Why volume-based security KPIs fail CISOs and boards alike 

Measuring against volume-based KPIs, like the number of scans run, vulnerabilities found and alerts generated, only tracks the effort taken to increase security—not the actual outcome. These traditional KPIs tell you how needed security measures are, but not whether they are stopping anything meaningful.

For example, a scan finding 10,000 low impact issues might look productive on a dashboard, but at the same time a single exploitable dependency might have been untouched for months, presenting a critical, unresolved security risk.

Board members and the C-suite see rising KPI numbers and automatically assume strengthened protection when, in fact, it could be quite the opposite. This blurred measurement line skews the reality of how security teams are tackling security risk.

These industry wide tropes are inadvertently rewarding security teams for generating noise but not reducing actual risk. And with the average fix time for security flaws rising from 171 days to 252 days over the past five years, the delay to remediation quietly backlogs security risks.

Those vulnerabilities hidden in the depths of the supply chain and pipeline are a ticking time bomb.

With security teams already stretched and struggling to find the capacity for finding and fixing vulnerabilities, these outdated metrics encourage a culture where security teams and CISOs look “on top of it”, right up until an old, known flaw gets exploited – at which point, it could be too late.

Pipeline compromise and dependency risk have made point-in-time scanning obsolete 

With the rapid pace of technological advancement and the apparent rise in successful cyberattacks, point-in-time scanning is now inadequate. It overlooks critical time factors—such as the mean time to remediate or the duration an attacker can operate undetected—which are precisely what attackers exploit.

Modern attacks happen in the gap between scans, with security snapshots unable to catch moving targets. For CI/CD pipelines, they are obsolete. Code changes multiple times a day and dependencies update automatically.

And nowadays, an attacker doesn’t even need to evade a scan. They just wait for the next build, commit, or dependency pull and, by the time the scan report is read, the environment it assessed no longer exists.

Scanners traditionally inspect source or binaries, but not the inner workings of the build process, meaning a malicious build step can inject code after a scan has passed.

This happened with the infamous SolarWinds Orion attack, which compromised thousands of organizations (including US government agencies) back in 2020, injecting malicious code into software updates that were then distributed to the unsuspecting customers.

If the build is already poisoned, then the scan is irrelevant.

What CISOs need to prioritize this year 

As cyber risk increases and hackers become more sophisticated, balancing the challenges associated with assessing risk and proving the value of application security is becoming more of a minefield for CISOs. They need metrics and that security teams can prioritize to better reflect real application and supply-chain security risk.

These include the backlog reduction of exploitable flaws, the time it takes to fix critical issues in production, and evidence that the fixes actually work, rather than just a scan. The shift isn’t from less measurement to more measurement. It’s from counting security activity to measuring true exposure and business resilience.

Ultimately, security metrics should tell leadership how much risk has been removed and how quickly systems are back to normal—not how hard the security team worked to find it. This change in positioning will help us all become better equipped to properly defend against risk.

We've featured the best online cybersecurity course.

This article was produced as part of TechRadarPro's Expert Insights channel where we feature the best and brightest minds in the technology industry today. The views expressed here are those of the author and are not necessarily those of TechRadarPro or Future plc. If you are interested in contributing find out more here: https://www.techradar.com/news/submit-your-story-to-techradar-pro