Benchmarks don't lie! Is Windows 7, XP or Vista better for gaming?
How we tested
Vista produced strong results in testing, particularly in Far Cry 2, where it managed the best frame rates of any system
Updated: read our Windows 7 review, plus, now it's been released, our full comparison: Windows 7 vs Vista vs XP.
Even now, it's hard to think of a single killer app that makes DX10 look any better than the huge swathe of DX9 games. This is why there are still plenty of people out there who are running their games under Windows XP. Add Windows 7 into the mix and you now have a three-way fight for your gaming attentions. But which one should you have running on your main gaming rig?
The reliable Windows XP, the pretty but oft-maligned Vista or the new kid on the block who hasn't even earned his racing wheels yet? We think it's time to put these operating systems to the test on a suite of benchmarks and see which one impresses us the most.
How we tested
In order to assess how these operating systems perform on a mainstream gaming PC, we've elected to torture a Dell for your delectation. The Studio XPS M435T we've picked boasts a Core i7 920 at its heart and 6GB of DDR3 1066MHz triple channel memory, making it a serious number cruncher when given room to strut its stuff.
Things aren't quite so carefree in the graphics department though, and in order to hit a decent price point (the machine can be picked up for around £700) it features a single 512MB GeForce 9800GT. Not a gaming powerhouse, but a capable enough performer for our testing.
For the benchmarks we've focused on gaming, putting Codemasters' seminal GRID, Ubisoft's brilliant Far Cry 2 and the splendid RTS-'em-up World in Conflict to task separating the OS wheat from the chaff.
GRID boasts a brilliantly optimised rendering engine, which shouldn't push the hardware too much. Far Cry 2 is taxing with all the features turned on and requires just as much rendering muscle as processing power. World in Conflict shows off the differences between DX9 and DX10 rendering and is one of the most reliable benchmarks we've used in the office.
To accompany these results, we've also benchmarked the operating systems using that perennial favourite, 3DMark06. Futuremark may have released 3DMark Vantage a while ago now, but we're still not convinced about its usefulness for assessing a system's gaming performance. For overall system performance we've used PCMark05 (although this only works in 32-bit versions of Windows) and Cinebench R10 for more serious work (there are separate 32-bit and 64-bit executables).
Finally, we've recorded boot times for the operating systems and for GRID and Far Cry 2. The results for these timed tests were a little surprising as well.
Meeting expectations
We'll freely admit that we expected Windows 7 to shine like a particularly shiny thing in these gruelling tests. One thing that came to light straight away though, is that it isn't as honed under the hood as it feels. The interface might be smooth and everything might work, but the code isn't optimised, there's debug code swilling around and, from what we can see, Nvidia's drivers still have room for improvement.
There is some good news with Windows 7 though. Firstly, raw processing performance is stronger than in Windows XP and Vista. The Maxon Cinebench test can be used on machines boasting up to 16 cores, so it quite happily handles the eight that show up here (that's HyperThreading across four real cores).
The first two sets of results focus purely on CPU rendering and put Windows 7 in the lead, particularly in the multiple-core test. Vista isn't far behind, but if you want a serious rendering platform there are good things coming. This message was backed up when we configured the graphics settings as low as possible in World in Conflict.
By doing this, we push the graphics card out of the rendering loop as far as we can and focus on straight computation. The 64-bit version of Windows 7 managed 245fps, while 32-bit Vista brought up the rear with 210fps. Microsoft has heavily optimised the code paths in its soon-to-be -released OS, which will benefit gamers as well as more serious number crunchers.
The key reason Windows 7 isn't topping the benchmarks is due to the state of Nvidia's drivers. This isn't a dig at the GPU giant, merely a statement of fact – after all, the OS hasn't been released yet. Cast your mind back to the release of Vista and the shaky start that had and we'd hazard a guess that Nvidia will be pushing up the framerates as the launch date nears.
We saw a hint of this as we went from the 179.11 drivers that were installed by default, up to the latest available at the time of testing: 181.71. Framerates didn't rocket, but a few extra frames are always welcome.
Vista is at the other end of this development cycle now and, as such, has a solid driver base. Indeed, Vista produced strong results in testing, particularly in Far Cry 2, where it managed the best frame rates of any system. Strangely, Ubisoft's game put in comparable performance in DX10 and DX9, although it's good looking whichever code path you run it on.
DX10 vs the rest of the world
As far as graphics drivers are concerned, Windows XP is in the strongest position. The performance garnered from this operating system was great everywhere except for Far Cry 2 – and judging by the minor differences in framerates between DX10 and DX9 and Vista, it appears that this engine was developed with DX10 in mind.
If you're planning to play World in Conflict in DX9 mode for instance, you'll get the best framerates in Windows XP. GRID is the same – you'll get a couple more frames out of this game in XP than any other OS. Of course, that begs the question: what about DX10?
It's true that you're going to need Vista or Windows 7 if you want to run DX10 games. The real question is, do you actually want to run games in DX10 mode? World in Conflict certainly looks a lot better in DX10, but it's still the same game in DX9 and you suffer a major penalty for turning on all that graphical loveliness, dropping from 42fps to 27fps in Vista.
To be fair to Massive Entertainment, this was one of the first DX10 titles and possibly isn't as optimised as it could be. Even Microsoft isn't pushing DX10 for its Games for Windows titles as much as it did initially.
The official Games for Windows site lists a total of nine games and some of those are rather suspect – Hellgate: London suffered an unfortunate early death, while Age of Conan's DX10 implementation still isn't live (there's a cut-down client being tested right now, but many of the effects that were shown off nearly a year ago aren't present).
Crysis is still one of the platform's best looking titles, although it's hardly cutting-edge now. For many gamers, the lack of any real benefit to DX10 is still the problem – it just doesn't do enough to warrant going out of your way to upgrade to. Sure, there are some nice effects, but clever shader programming can knock out incredible effects in DX9 too. Cast your eyes over GRID's loveliness for simple proof of this.
Game developers don't want to limit themselves to a niche of the existing market just for the odd effect either – it's simply not worth it. You only need look at the popularity of the likes of WoW to see that games don't need hi-tech visuals to be successful.
You also shouldn't forget that a fair chunk of games are console ports and that Xbox 360 doesn't have a pile of DX10 cleverness hiding behind the red ring of death – it's DX9 in nature (although admittedly with some clever DX10-like tricks up its wizard's sleeve).
So we're not convinced about DX10 as a selling point for an OS although, to be fair, it's a great way of showing off the capabilities of your graphics card. It's telling that DirectX 11 isn't going to have quite the same marketing push that its predecessor did (who can forget the ridiculous Flight Simulator images Microsoft pushed out to show how much better DX10 was than DX9). In other words don't expect any DirectX 11 only games to appear any time soon.
32-bit or 64-bit?
A more pertinent question these days is whether to install a 32-bit or 64-bit OS. Windows XP started the ball rolling with x64, but in truth the driver support was too ropey to make this a serious consideration – indeed, we couldn't get our test machine stable enough for testing in Windows XP x64. Vista and 7 are a different story though, and it's the 64-bit versions that show the best performance in Far Cry 2.
Part of the reason that the 64-bit versions do so well is because the operating system and game/ benchmark has access to the full gamut of memory. Our test rig has 6GB of DDR3 and the difference this makes to the benchmarks is obvious. Memory is one of the main driving factors in a move to a 64-bit OS and as prices of RAM continue to drop, so that option becomes more and more appealing.
In Far Cry 2, for instance, we saw an increase of two or three extra frames per second in-game, plus faster loading times for the actual games themselves, both for initial loading and for levels. Factor in much smoother alt-tabbing and it's easy to see that the move to 64-bit brings many benefits and no obvious downsides.
Speaking of loading times, our testing highlighted some interesting numbers and some confusing ones as well. Far Cry 2, for instance, may run relatively slowly in Windows XP, but it loads incredibly quickly in it – twice as quickly as it does in Vista 64.
GRID loaded quickest on the 64-bit operating systems, which points to efficient use of memory once again. The best loading time for the OS goes to Windows 7 though, rolling in at just over a minute. Note that these times were recorded from the moment the power button was pressed, and so include the POST as well.
The choice of which OS you should go for isn't just about loading times or even frame rates. It's also about usability. It's about how comfortable you are with using it and it's about compatibility. Windows 7 is commendable at this stage for just working with pretty much everything we threw at it (although for some reason on this machine it wouldn't play ball with Fraps). Once again, this gives us high hopes for its future, particularly when compared to the ill-tempered Vista launch.
Only once you've used an operating system can you really know if it's for you. Try as we might, we're still not comfortable with Vista, while Windows 7 doesn't irk us anywhere near is much. It's a personal thing. Indeed, every time we mention that Windows Vista is less than perfect we get plenty of readers complaining about the fact. Maybe all our messing around with operating systems on a daily basis has spoiled us, but given the choice we'd still go for Windows XP for pure DX9 speed and Windows 7 64-bit for the good all-round experience.
Conclusion: which Windows is best for gaming?
Conclusion
Should you change your operating system just to get a few more frames out of your hardware? That really depends on what you use your machine for. If, for instance, you overclock your rig and are out to get the fastest frame rates around, then sticking with Vista when you could be getting more frames out of XP seems churlish.
We're not talking double figure differences here, but then again, your overclocking generally only amounts to a few extra frames as well. If, however, you want to run DX10 games, then Vista is in surprisingly good shape – it's by far the fastest way to enjoy the likes of Far Cry 2 and there's no question that your games will run.
Ideally you'll have 4GB of RAM or more and be able to run Vista 64 too. And what of Windows 7? It's fast, particularly for number crunching, and can perform well with the latest games, but if you're looking for speed alone it doesn't quite beat Vista yet.