Our Graphics Card Testing Suite

Rob Williams

Editor-in-Chief
Staff member
Moderator
Hi all:

When we posted our HD 4890 vs. GTX 275 article last month, we found an obvious issue after-the-fact. While many other sites out there showed the two cards to be quite comparable, our results favored NVIDIA overall... and this is a problem. One explanation is that the titles we're currently using have been used since last December, and many of them favor NVIDIA today.

In December, that wasn't the case. As seen in our HD 4870 vs. GTX 260/216, both ATI's and NVIDIA's equivalent offerings fared well throughout all of our tests, and in the end, the performance was pretty-much equaled. Certainly not the case now though, and NVIDIA has obviously released driver enhancements to improve many of the titles we still use. So... we need a change.

Due to the sheer amount of testing involved, I usually try to put off our testing suite for as long as possible, but we clearly are due for a change. Once we have a new game suite, I don't expect to be able to re-benchmark every card we have that quickly, but I will get as many done as quickly as possible. I'll mainly focus on current cards though, not last-gen, if I can help it. We manually benchmark each and every card / game configuration, and that's obviously extremely time-consuming.

So my question is, how about some recommendations of what we should use? What would YOU like to see us use, that would give both companies an equal chance at success? Here's how our suite looks now:

Crysis Warhead
Call of Duty: World at War
Far Cry 2
Left 4 Dead
Mirror's Edge
Need for Speed: Undercover

We also include 3DMark Vantage results, but I personally consider those to be rather useless. There's still a demand for such results though, so I don't see us ceasing use anytime soon.

The obvious games to rid would be Far Cry 2 (mostly because I can't stand it anymore), and Mirror's Edge. I'd also be removing Left 4 Dead, because, well, I can't stand that one anymore either (;)) and due to the game's mechanics, it takes far more than two or three playthroughs to achieve an accurate result. Also, that game happens to run well on <em>any</em> current GPU (even low-end) all the way up to 2560x1600 4xAA, so including it is a little needless.

Since the last time we revised our game suite, there have been a few good high-end titles to come out that might be worth including going forward. Of these, F.E.A.R. 2: Project Origin and Tom Clancy's H.A.W.X immediately come to mind, although the latter I'm a bit skeptical about (achieving accurate and repeatable scores in a flight game seems a little sketchy, but I could be wrong). F.E.A.R. 2 is "best played" on NVIDIA, while H.A.W.X is on ATI's side.

So, without making this into a novel, do you guys have any recommendations for games you'd like to see us use for testing in our next suite revision? Specific reasons of why would be great too!
 

Merlin

The Tech Wizard
I think at one time you considered GRID racing driver as one of the games to benchmak.
I know it's intensive on CPU and GPU
So maybe take another look at that game?
 

Doomsday

Tech Junkie
Crysis and COD remain good. World in Conflict is a heavy game, really tests the GPU, including Empire: Total War. i agree with the dude who looks like Dumbledore,lol!
I think the games which r usually made exclusive to PC favor Nvidia more and the Console ports favor ATi more, just a thought!
The games which u can consider in the future can be:
Call of Juarez: Bound in Blood, FUEL and OverLord II....
 

Rob Williams

Editor-in-Chief
Staff member
Moderator
Hmm, I swore I responded to this. Sorry for the late response folks...

I considered GRID, but the main issue is that it's a difficult game, and each run is different than the last. That means that each run varies too heavily, and results cannot be compared. I might genuinely suck at racing games, but GRID is difficult, no doubt about it. I can't seem to get around the first corner in the first race without spinning out, so it really can't be included, sadly.

That's why I've been including Need for Speed: Undercover. I like to include at least <em>one</em> racing title, and this one in particular still looks good, and is easy to play. I'm not sure what other sites do for GRID, whether it be timedemos or what, but I really can't get a hang of it in order to have accurate results with each run. But on the other side of the coin, NFS: Undercover seems to heavily favor NVIDIA, so it's a tough one. I'll sit down with GRID again soon and give it another honest shot, because I'd like to include it. We'll see where that goes.

World in Conflict is one particular title I've been meaning to give a try for a while, but it's been a matter of time. That also tends to be CPU bound from what I understand, so I could really kill two birds with one stone with a single game... sounds good to me. I'll take a look. As for CoJ: Bound in Blood... I hope that's a good one to use for testing. The original definitely was.
 

Ben

Site Developer
Don't any games offer the ability to record a certain path and then play it back? I swear Counter-Strike had something like this. It was rendered on your PC however, so it would be useful for benchmarking.
 

Rob Williams

Editor-in-Chief
Staff member
Moderator
That's what's called a "timedemo", and we pride ourselves on not using them. Actually, the only time did we use them was for our first-ever GPU review, so we've been "clean" for a good four years ;-)

Timedemos have been debated for years, but I've come to the conclusion that actually playing the game is the best way to measure the performance. Timedemo's goals are to run through a set run as fast as possible, which means that it's not realistic. Plus, because it's a timedemo, it becomes a lot more CPU-bound than if you were actually playing the game, and to me, that's going to alter the results.

Manual runthroughs are still not perfect, because for one, I'm not a robot, and two, gameplay can vary from one run to another (which is why we run the test more than once and make sure the scores keep stable). Because we manually play through these games, rather than have the computer do it for us, we can remain completely confident in our results. It's that simple, really.
 

Rob Williams

Editor-in-Chief
Staff member
Moderator
So... I'm about to test out a few different games to see what's suitable for benchmarking. As I mentioned before, CoD: World at War and Crysis Warhead will remain. The former because it's a perfect game to benchmark (is stressful to all GPUs), and the latter for kind of the same reason, but the added fact that it's become de facto.

The games I'm going to test out are FEAR 2 and also HAWX. FEAR carries an NVIDIA logo, while HAWX carries ATI... so things should even out. I'd love to include games that don't cater to one side at all, but that seems difficult. If the game is worthy of benchmarking, it has one of the two... it's that simple. I plan to benchmark both tomorrow (Mon) with both an HD 4870 1GB and GTX 260/216 and see how fair things are.

I'm still looking for other recommendations though. I'd love to include an RTS game, but from what I've seen, none of those titles are worth much for GPU benchmarking... and most that I've touched cap their FPS. Those games are heavily CPU bound, and it's for that reason that we'll be looking over a few for our CPU suite revision.

Where racing is concerned, I'm still stumped there, but I'm going to give GRID another go and see if I can somehow get a hang of it so I can use it for benchmarking. If not, I'm not sure what I'll do. NFS: Undercover might be the lone option, seeing as the PC market is slim on racing titles suitable for benchmarking.

Edit: Just realized Burnout Paradise is on the PC, so I'm going to give it a spin. Prototype also looks like it might be worth testing out... it's getting some great ratings.
 

Rob Williams

Editor-in-Chief
Staff member
Moderator
World In Conflict for RTS?!?

Well that's one I just downloaded (Steam), so I'll be testing it out. That is likely to be the game we'll use for CPU reviews... I don't think it has the graphical requirements to become a proper stature for our GPU test suites. I'll test regardless though.
 

Rob Williams

Editor-in-Chief
Staff member
Moderator
Whew. /wipes sweat from brow. This is exactly the hassle I hope it'd be! ;-)

I had high hopes for being able to include Burnout Paradise, but it's a no-go. The developers, for whatever reason, decided to cap the frame rate to 60 FPS, and at the same time, omitted the ability to disable Vsync. When comparing one graphics card to another to see which is better, a game with a capped frame rate makes absolutely no sense. It's too bad... this game has great graphics (and is fun, too).

As it stands, the updated selection for our games is:

Call of Duty: World at War
Crysis Warhead
FEAR 2
Grand Theft Auto IV
GRID
Tom Clancy's HAWX

The first two are chosen for what I hope are obvious reasons, while our additions include FEAR 2, Grand Theft Auto IV (I never thought I'd include this, but it IS strenuous), GRID and also Tom Clancy's HAWX. Note that HAWX is sure to be removed... I'm just looking for a replacement. That title suffers from the same issue as Left 4 Dead... the frame rates are very high on any GPU. Great for the consumer, but pointless for our goals. Plus, without Vsync, it's actually hard on the eyes. No one will want to run the game that way.

Below are our results so far. Please note that each game was tested twice at a particular resolution and then averaged out. For our real results (as in, the results that will be posted on the site), each game will be tested three times and averaged (for even more fine-tuning). Also, the first number in these results is the Min FPS, the second is the Avg FPS. For 3DMark, it's 3DMark/GPU/CPU.

ATI Radeon HD 4890 (Catalyst 9.6)

3D Mark Vantage
Performance: 11929 / 10381 / 21583
Extreme: 4900 / 4709 / 21555

CoD: World at War
1680x1050: 37 / 61.969
2560x1600: 17 / 34.786

Crysis Warhead
1680x1050: 24 / 41.115
2560x1600: 12 / 22.546

FEAR 2
1680x1050: 76 / 120.692
2560x1600: 41 / 58.464

Grand Theft Auto IV
1680x1050: 37 / 58.277
2560x1600: 21 / 38.029

GRID
1680x1050: 85 / 112.478
2560x1600: 57 / 73.362

Tom Clancy's HAWX
1680x1050: 141 / 176.087
2560x1600: 62 / 108.480

NVIDIA GeForce GTX 275 (GeForce 186.08)

3D Mark Vantage
Performance: 12620 / 11095 / 21477
Extreme: 5561 / 5351 / 21658

CoD: World at War
1680x1050: 35 / 64.409
2560x1600: 21 / 39.069

Crysis Warhead
1680x1050: 25 / 45.527
2560x1600: 11 / 24.867

FEAR 2
1680x1050: 72 / 113.035
2560x1600: 38 / 56.747

Grand Theft Auto IV
1680x1050: 41 / 56.946
2560x1600: 33 / 45.422

GRID
1680x1050: 74 / 101.378
2560x1600: 51 / 61.134

Tom Clancy's HAWX
1680x1050: 190 / 237.519
2560x1600: 105 / 135.429

Basing things off of 3DMark Vantage, NVIDIA is the one with the slightly faster card here, which also proves true with our initial results here. NVIDIA proved faster with CoD: WoW, Crysis Warhead, GTA IV and also HAWX. The favor went to ATI in FEAR 2 and also GRID.

Would love your thoughts on these results guys, and also suggestions. I've been recommended to check out the latest Riddick game, since it has great graphics and is benchmarker-friendly, so I think I'm going to go grab that off of Steam as well and give it a test later. I really have no interest in leaving a as our sixth title, for the reasons above. But as it stands in today's PC game landscape, I simply cannot find any other games worth including... but if I'm somehow missing one that you know of, please speak up.
 

Rob Williams

Editor-in-Chief
Staff member
Moderator
World In Conflict for RTS?!?

Alright, I just gave that game a test now (at least with its built-in benchmark), and I had no idea it was so graphically-demanding (yes, I'm embarassed). So I'm going to give it a good test soon, and if all goes well, I have a feeling IT will replace HAWX, rather than Riddick. It would be great to get an RTS in there for once... been far too long since we've last had one in our GPU content.
 

Rob Williams

Editor-in-Chief
Staff member
Moderator
Alright, I took World in Conflict for a spin, and I like it. Here are the results:

ATI Radeon HD 4890

World in Conflict
1680x1050: 46 / 56.737
2560x1600: 28 / 35.701

NVIDIA GeForce GTX 275

World in Conflict

1680x1050: 47 / 62.220
2560x1600: 29 / 34.945

There are a few variances here, but they are minor. It's hard to exact a runthrough with a title like this, where circumstances can change on a per-run basis. Regardless, thanks for the recommendation Doomsday!

So as it stands, our game line-up will be:

Call of Duty: World at War
Crysis Warhead
FEAR 2
Grand Theft Auto IV
GRID
World in Conflict

I'm of course up for suggestions/questions/recommendations before it's set in stone, though!

It's going to be at least a week before I can get on actual benchmarking... waiting for a fresh motherboard to arrive (which will become the GPU testing motherboard). That's fine though, because it will probably take me that long to iron out the details for our CPU and motherboard methodologies!

Feels good to finally get that out of the way, though!
 

Doomsday

Tech Junkie
no problemo! :)
i played World In Conflict and it really sucker punched my system even at 1440*900! it lagged a lot at high details with 2 x AA......Dragon Age: Origins is also looking to be a rather heavy game for the future!

great line up, just hope some new GPUs, like the HD4770, start coming as i am looking forward to the 40nm line up for my GPU upgrade! :D
 
Last edited:

Rob Williams

Editor-in-Chief
Staff member
Moderator
Yeah, I had to have not published the HD 4770 article yet, but I simply couldn't have done it with the same test suite we've been using. Would have been a disservice to our readers. Can't wait for the motherboard I need to arrive though, then I can get down to testing!
 
Top