Techgage's Testing Methodologies

[Rob Williams]

You can use this thread to discuss our article which takes a look at our testing methodologies. So far, we explain how we install and optimize Windows Vista, and also delve into the specifics regarding our graphics card testing. As we complete the revisions for our other methodologies (CPU and motherboards in particular), we'll update the article. We'll also update soon regarding our CPU and chassis cooling.

Discuss!

[2Tired2Tango]

Originally Posted by Rob Williams

You can use this thread to discuss our article which takes a look at our testing methodologies. So far, we explain how we install and optimize Windows Vista, and also delve into the specifics regarding our graphics card testing. As we complete the revisions for our other methodologies (CPU and motherboards in particular), we'll update the article. We'll also update soon regarding our CPU and chassis cooling.

Discuss!

I'm glad to see that your are accounting for many of the influences upon foreground performance in a multitasking environment. However, I might suggest that booting the system into a single tasking environment (eg. FreeDOS) and running contrived exercises will give you a far better indication of comparative performance. The more systems you test in exactly the same series of exercises the better your performance ratings will become.

Along with the stuff you are disabling, you should also consider disabling system file checking, the computer browser, network location awareness, upnp discovery, net bios, net DDE, print spooler, shell hardware detection, SSDP discovery and the bits service. All of these can steal processor time without notifying you.

[Rob Williams]

Originally Posted by 2Tired2Tango

I'm glad to see that your are accounting for many of the influences upon foreground performance in a multitasking environment. However, I might suggest that booting the system into a single tasking environment (eg. FreeDOS) and running contrived exercises will give you a far better indication of comparative performance. The more systems you test in exactly the same series of exercises the better your performance ratings will become.

I'm a little confused... you're talking about benchmarking in a non-Windows environment? While I'm all for benchmarking in such a way that will deliver us the most accurate results, no one uses an OS like Freedos... they use Windows. As long as we cut down whatever might cause inconsistancies, we're good.

We don't run a benchmark just once, but twice, or more if the first two runs show too much of a delta. If we can continually run our benchmarks and, in the end, see stable results, we're confident of our setup.

There are a couple of things I forgot to mention in the article, but will add soon.

First, before each run is conducted, the networking device is disabled (there's never a cable plugged into it anyway, unless I specifically need the Internet for some reason). Second, before we finalize any of our results, we first run each benchmark twice over (or more) as previously mentioned, and then restore the entire machine once again to re-test each graphics card, processor or what-have-you once again, just as an added measure.

That latter step is the reason this initial graphics card review is taking me so long. I've already completed benchmarking every card here, but I'm in the process now of re-testing each card once again in order to increase the confidence in our results even further. I'm glad I take this extra step, because about 5 - 8% of our tests will show different slightly results after the restore.

Originally Posted by 2Tired2Tango

Along with the stuff you are disabling, you should also consider disabling system file checking, the computer browser, network location awareness, upnp discovery, net bios, net DDE, print spooler, shell hardware detection, SSDP discovery and the bits service. All of these can steal processor time without notifying you.

I appreciate you listing all these. As I mentioned in the article, I had no intention of going to extreme lengths in disabling system services. I used to do that all the time with our Windows XP benchmarking machines, and it could end up causing more harm than good. None of these services you mention are as beefy as the ones we disable, and most of them should never run at all (network-related, printing).

When Vista first launched, I spent a lot of time figuring out which services crept up during our benchmarking, and the ones I didn't disable, were far too minor to even matter. Things like Firewall, Defender, Update and especially Search had to be disabled. I've caught Search using upwards of 100MB of RAM before. That's understandable for a real machine, but ludicrous for a benchmarking rig ;-)

The goal was to not produce such a highly optimized machine that it becomes unrealistic, but to at least get rid of all the potential effected results by certain rogue services.

Thanks for the input.

[2Tired2Tango]

Originally Posted by Rob Williams

I'm a little confused... you're talking about benchmarking in a non-Windows environment?

Wellll... sort of.

My concern is that in some cases you wouldn't actually be testing the device (video card, for example) but rather the environment it's in. The only way to truly test a device without interference from other tasks is to use hardware diagnostics in a heads up environment.

One example I can give you is my own benchmarking of hard disks done a couple of years back, when the 7200 rpm drives were coming out. I wrote code that would block multitasking during the tests and compared 5400 and 7200 drives, I also compared IDE and SATA. Without fail I got far better performance with the multitasker blocked than I did in even the most optimized standard windows environment.

However I do take your point that nobody actually does this so you're kinda stuck "going with the flow" in a sense. You're looking for "real use" style testing and that's not a bad thing... it's just not a pure test of hardware. It's not an invalid test so long as it's done in a way that is comparable to the way others are doing the same tests...



(Or it might be that, with the heat and humidity these last couple of days, I'm just being extra cranky...)

[Rob Williams]

I don't entirely disagree with similar methods, but it all comes down to what makes most sense for what we're trying to accomplish. Almost 90% of our visitors run Windows, so to benchmark anything else would send most people running. Plus, we're still looking to create realistic environments, and nobody is going to optimize the heck out of their machine realistically, except the hardcore enthusiast.

As you said, we're happy as long as our results are accurate from run to run. So far on the application side, we've had few issues with realiable results, and if there is a benchmark that sways a little bit from one run to the next, it's normally caused by storage. With our PCs equipped with Intel's latest SSDs though, we're even trying to rid that issue as much as humanly possible.