Benchmark results are all very well, but can you feel the numbers? When it comes to PC performance and especially gaming grunt, that's the key question.
Of course, everybody knows high-end PC components are piddle-poor value for money. You don't need us to tell you Intel's latest £800 six-core monster, the Core i7 3960X, isn't eight times as good as a £100 quad-core AMD Phenom II.
The same goes for every other component class. Whether it's graphics cards, motherboards or especially storage, your bang for buck plummets horribly at the top of the price scales. None of that, however, is the same as saying high-end clobber doesn't deliver any benefits for the cost.
There's little doubt extreme edition processors and the latest multi-billion transistor GPUs will spew out unholy benchmark numbers. The thing is, we're less sure how much impact that has on subjective gaming pleasure. Putting it another way, if we plonked you down in front of two PCs, one trimmed out with the finest kit known to man, one built down to a modest price point, and fired up your favourite games, would you actually be able to see or feel the difference?
So let's be clear about this. Our task this month isn't merely to prove that spending a shedload on the finest PC components isn't cost effective. It's to find out whether it makes the slightest difference to the way games look and feel.
To do that, we've built two very different systems, focussing on the three core components that influence performance. So that'll be the processor, graphics card and hard drive.
Our money-no-object rig weighs in at nearly £3,000 for those components alone. And our budget-oriented alternative? It's less than £300. Can a system costing just one tenth the price of another really deliver an indistinguishable gaming experience? Time to find out.
First up, let's lay out some ground rules for this intriguing contest. Critically, our main focus is gaming. That doesn't mean we'll be excluding overall system performance entirely. We'll chuck that into the mix for context. But it won't influence the overall result.
However, what will effect the outcome, and something we're contriving to be quite specific about, is the display. The first thing you'll have noticed is that we're not including it in the core set of components. The reasons for this are the frankly enormous array of possible PC monitors and the related issue of user preference.
For most gamers, if money was no object they'd probably run the most powerful possible PC. When it comes to screens, the choice isn't so obvious. Would you go for the largest possible display? Perhaps, but only if the system in question doesn't double as a multi-purpose desktop.
Then again, maybe it's maximum resolution you should be aiming for? In which case, you'll have to compromise on screen size, since the highest resolution monitors are not the largest.
Controlled viewing
With all that in mind, we've opted for a single control monitor. While there is a niche of gamers who demand the highest possible resolutions, it's debatable how much difference it makes to the visuals when you extend beyond 1080p Full HD.
Moreover, 1080p has become the de facto standard for many games. It's also a resolution that dominates regardless of screen size. The vast majority of PC monitors from 22- to 30-inch and beyond now sport a 1,920 x 1,080 pixel grid. The control screen we've gone for is a 24-inch example.
The make and model are not important. What does matter is that 1080p pixel grid. It puts a limit on the number of pixels any gaming PC is likely to have to pump out.
Immediately, that plays into the hands of the cheaper system. There's little doubt that life would be much harder for it at a resolution like 2,560 x 1,600. But our argument is that even at the high end and regardless of screen size, the most likely resolution for gaming is going to be 1,920 x 1,080. So, that's what we're sticking to.
Taking that logic and running with it a little further, we've removed another image-related stipulation. The two systems are not required to run at the same image quality settings in-game. If you think that tilts things even further in favour of the cheapo rig, you'd be absolutely right.
But we think our reasoning makes sense and that is this comparison isn't about the numbers. It's about the gaming experience - nothing more, nothing less. Our plan is to set the two systems up and put gamers in front of them in a blind comparison test.
So, what matters isn't making sure that the two systems are running exactly the same level of anti-aliasing, anisotropic filtering or shader complexity. What matters is whether gamers can tell the difference. If dialling down the eye candy doesn't make a difference that's noticeable, then that's exactly what we'll do.
Hardware hits
However, one thing you should be very clear about is that we won't be compromising the settings on the high-end rig. For each and every game we'll tune it to look as spectacular as physically possible. The challenge will be getting the low-end system right.
As ever, it's a question of playing off image quality and frame rates with the latter being particularly critical. Knock the anti-aliasing down from 8x to 4x and few, if anyone, will notice. Chop the frame rate in half and that's a different matter, especially if the result ever dips below 30 frames per second.
With all that in mind, it's clear that this entire experiment is heavy on the subjectivity. But that, frankly, is the whole point. Like we said, we all know how a benchmark fisticuffs would turn out. And it wouldn't be pretty. What we didn't know going in was exactly how the real world gaming experience compared.
So what are the hardware variables, how did we set the systems up and what games did we go for?
Component wise, our focus is on the three components that have the biggest influence on gaming performance: the processor, the graphics card and the hard drive. Obviously, there's a little more to it than that.
But beyond those three, not only do the performance implications drop off, the price delta shrinks enormously, too. Our reasoning was to allow roughly £100 each for the low end rig. We didn't put a price limit on the top-end monster, but in the end the average wasn't a millions away from a nicely symmetrical £1,000 per component.
With our systems specified, built and saddled up with a fresh copy of Windows 7, the next task was game title selection.
This is a subject ripe for hang-ups, so we elected to go with some very straightforward criteria. Elder Scrolls V: Skyrim made the cut because it's a super hot title, it looks great, it's fairly demanding and it ticks the large-levels and long-draw distance boxes.
Next up is Crysis 2. Okay, as a game to play it's pretty crappy. And its execution reeks of console-port compromises. But in many ways its graphical fidelity can be regarded as the benchmark. In parts it's absolutely stunning.
Our final candidate is DiRT 3. We wanted a driving game in the mix and there's no doubting DiRT 3 is a looker. Granted, this trio is hardly comprehensive. But obvious omissions such as, perhaps an MMORPG such as WOW don't tend to be hugely performance intensive. They're coded to support very broad user bases. So, while there will always be a few exceptions, we reckon our chosen trio gives a pretty decent picture of overall gaming performance.
Tuning the tests
Of course, choosing games is only half the solution. We next had to select image quality settings. From the outset we'd decided that running at 1080p native resolution was essential, so that bit was easy.
Configuring the high-end system was very straight forward, too. The system has performance to burn so it was simply a case of maxing everything out and then jumping in-game to make sure everything was running smoothly.
Like we said, we didn't want our budget rig getting an easy ride. It would have to compete with the very highest possible image quality settings currently available. Indeed, tuning up the cheapo machine was intriguing.
Going in, we'd expected that it would be a finely balanced process of playing off image quality settings and frame rates. One thing we were certain about was that smooth frame rates were essential. Any keen gamer is going to pick up on jerky, unresponsive performance.
Image quality, on the other hand, is a different matter. How many people, for instance, can tell the difference between 16 times anisotropic filter and eight times filtering? Likewise, for any given game, does knocking settings such as shader or shadow quality down make a big difference to perceived image quality?
We were in for a long haul. Or so we thought. It transpired, it was much easier for the simple reason that it handled the highest image quality settings much better than we expected. One reason for this is the undeniable influence that console games have on game engines. The Xbox 360 and PS3 are positively ancient but retain an iron grip over game development. That's lead to stagnation.
The bottom line is that we needed to do relatively little to get our affordable PC running smoothly. Mostly it was a case of dropping the anti-aliasing from 8x to 4x (though Crysis 2 doesn't actually offer proper AA, which is probably a big help) and tweaking just a handful of further settings. For the most part, both rigs ran with ultra settings across the board.
Blind test
The final part of the puzzle is the actual blind comparison. The key here was to keep the identities of the systems and the components included completely concealed.
In fact, we didn't even want our testers knowing what an enormous gulf in hardware specification they were experiencing. Instead, all our testers knew was that they were comparing the gaming experience on two different systems.
We also wanted to get impressions of different aspects of gaming performance including in-game frame rates, graphical quality and game load and level load times. With that in mind, the test went something like this.
Both machines were up and running at the same time on the same desk and via the same 24-inch high quality 1080p monitor. Switching between the two was a matter of the few seconds required to move the DVI cable across, allowing an almost instant comparison.
Our first test involved playing each title from an identical pre-loaded and paused game state without observing level loads or desktop performance. We wanted an unbiased assessment of the in-game experience, something that can be coloured by sitting through a lengthy level load.
Our testers were allowed around ten minutes per title to jump back and forth between machines, as often as they liked. For this comparison we didn't make any specific demands, we simply asked the testers how they thought the two systems compared.
Next was a test starting from the desktop and involving loading, first, the main game interface and then a game level, after which further game play and general impressions were taken. as well as adding storage performance into the mix, we also asked our testers more specific questions regarding performance and image quality.
Which, if either, of the two systems had better graphics? Did one or the other deliver a noticeably smoother frame rate or superior response to mouse and keyboard inputs?
Desktop test
Finally, we allowed our testers a little time to try out basic PC usage to get a feel for desktop performance: firing up a few apps, surfing the web and watching some high definition video.
Among all this there's a commonly encountered and gaming-specific issue that isn't covered and that's installation. There's absolutely no question the high-end machine with its ultra-fast SSD storage was much quicker during the installations. Without a doubt, this is something our testers would have noticed.
All of the above, of course, concerns subjective experiences. When you get right down to it, that's all that actually matters in the real world. If it feels the same, who cares if it runs 10 times faster?
That said, we also ran a suite of benchmarks to provide context and to underline the significance or insignificance of the comparative numbers. If our testers couldn't feel the performance difference, we looked at whether that reflected in the numbers. Likewise, if the gap in system performance was big enough to be picked up subjectively, was that also reflected in the benchmark numbers?
So there you have it. Two systems. one for those with money to burn. Another for those on a tight budget. Tested by keen gamers and as you'll see, the results are frankly astonishing.
