SPEC invalidates thousands of benchmark results — multiple Intel Xeon CPUs affected, are they not as fast as they seem?

(Image credit: Intel)

The Standard Performance Evaluation Corporation (SPEC), which tasks itself with maintaining standardized benchmarks for computing components, has invalidated more than 2,600 results for Intel Xeon chips.

The non-profit consortium revealed that it would no longer publish SPEC CPU 2017 results for Intel PCs running a specific version of the Intel compiler.

The decision stems from concerns over what the consortium describes as a targeted optimization for a particular workload. In other words, SPEC is accusing Intel of tweaking its products to obtain better benchmark results.

2,600 Intel benchmarks invalidated

SPEC CPU 2017 is a widely used benchmark for high-end servers, data centers and workstations, testing performance across a total of 43 benchmarks. The controversy relates to the Intel oneAPI DPC++/C++ Compiler, which has been allegedly optimized for the 523.xalancbmk_r / 623.xalancbmk_s benchmarks.

A disclaimer is now attached to more than 2,600 results: “The compiler used for this result was performing a compilation that specifically improves the performance of the 523.xalancbmk_r / 623.xalancbmk_s benchmarks using a priori knowledge,” thus invalidating the results.

Although benchmark figures are generally non-reflective of real-world conditions, testing is designed to be as life-like as possible, so optimizations designed to cater to specific benchmarks can result in skewed (and unrealistic) results.

According to a Phoronix report, 2022.0 to 2023.0 models are affected, leaving models either side of this date range unaffected by the disclaimer notices. The article continues: “The SPEC CPU specific optimization could result in a 9% overall SPECint speed uplift and around 4% for the SPECint rate.”

Intel told us in a statement: "We respect the SPEC committee decision and are focused on delivering real world workload performance for our customers."

TechRadar Pro has reached out to SPEC for any further information, but we did not receive an immediate response.

More from TechRadar Pro

Craig Hale

With several years’ experience freelancing in tech and automotive circles, Craig’s specific interests lie in technology that is designed to better our lives, including AI and ML, productivity aids, and smart fitness. He is also passionate about cars and the decarbonisation of personal transportation. As an avid bargain-hunter, you can be sure that any deal Craig finds is top value!