Latest Forum Posts

Latest News Posts
Coming Soon!
Social
Go Back   Techgage.com > Site HQ > Site and Forum Comments & Issues

Site and Forum Comments & Issues Comments? Suggestions? Out-right flames? Found a bug? Feel free to post about it here!

Reply
 
Thread Tools
Old 08-17-2009, 08:12 PM   #1
Rob Williams
Editor-in-Chief
 
Rob Williams's Avatar
 
Join Date: Jan 2005
Location: Atlantic Canada
Posts: 13,351
Default Techgage's Testing Methodologies

You can use this thread to discuss our article which takes a look at our testing methodologies. So far, we explain how we install and optimize Windows Vista, and also delve into the specifics regarding our graphics card testing. As we complete the revisions for our other methodologies (CPU and motherboards in particular), we'll update the article. We'll also update soon regarding our CPU and chassis cooling.

Discuss!
__________________
Intel Core i7-3960X, GIGABYTE G1.Assassin 2, Kingston 16GB DDR3-2133, NVIDIA GeForce GTX 770 2GB
Kingston HyperX 3K 240GB SSD (OS, Apps), WD VR 1TB (Games), Corsair 1000HX, Corsair H70 Cooler
Corsair 800D, Dell 2408WFP 24", ASUS Xonar Essence STX, Gentoo (KDE 4.11. 3.12 Kernel)

"Take care to get what you like, or you will be forced to like what you get!" - H.P. Baxxter
<Toad772> I don't always drink alcohol, but when I do, I take it too far.


Rob Williams is offline   Reply With Quote
Old 08-18-2009, 07:50 AM   #2
2Tired2Tango
Tech Monkey
 
Join Date: Mar 2009
Location: Ontario Canada
Posts: 648
Default

Quote:
Originally Posted by Rob Williams View Post
You can use this thread to discuss our article which takes a look at our testing methodologies. So far, we explain how we install and optimize Windows Vista, and also delve into the specifics regarding our graphics card testing. As we complete the revisions for our other methodologies (CPU and motherboards in particular), we'll update the article. We'll also update soon regarding our CPU and chassis cooling.

Discuss!
I'm glad to see that your are accounting for many of the influences upon foreground performance in a multitasking environment. However, I might suggest that booting the system into a single tasking environment (eg. FreeDOS) and running contrived exercises will give you a far better indication of comparative performance. The more systems you test in exactly the same series of exercises the better your performance ratings will become.

Along with the stuff you are disabling, you should also consider disabling system file checking, the computer browser, network location awareness, upnp discovery, net bios, net DDE, print spooler, shell hardware detection, SSDP discovery and the bits service. All of these can steal processor time without notifying you.
__________________
Tango

"An unexamined life is not worth living" ... Socrates

Last edited by 2Tired2Tango; 08-18-2009 at 07:53 AM.
2Tired2Tango is offline   Reply With Quote
Old 08-18-2009, 12:20 PM   #3
Rob Williams
Editor-in-Chief
 
Rob Williams's Avatar
 
Join Date: Jan 2005
Location: Atlantic Canada
Posts: 13,351
Default

Quote:
Originally Posted by 2Tired2Tango
I'm glad to see that your are accounting for many of the influences upon foreground performance in a multitasking environment. However, I might suggest that booting the system into a single tasking environment (eg. FreeDOS) and running contrived exercises will give you a far better indication of comparative performance. The more systems you test in exactly the same series of exercises the better your performance ratings will become.
I'm a little confused... you're talking about benchmarking in a non-Windows environment? While I'm all for benchmarking in such a way that will deliver us the most accurate results, no one uses an OS like Freedos... they use Windows. As long as we cut down whatever might cause inconsistancies, we're good.

We don't run a benchmark just once, but twice, or more if the first two runs show too much of a delta. If we can continually run our benchmarks and, in the end, see stable results, we're confident of our setup.

There are a couple of things I forgot to mention in the article, but will add soon.

First, before each run is conducted, the networking device is disabled (there's never a cable plugged into it anyway, unless I specifically need the Internet for some reason). Second, before we finalize any of our results, we first run each benchmark twice over (or more) as previously mentioned, and then restore the entire machine once again to re-test each graphics card, processor or what-have-you once again, just as an added measure.

That latter step is the reason this initial graphics card review is taking me so long. I've already completed benchmarking every card here, but I'm in the process now of re-testing each card once again in order to increase the confidence in our results even further. I'm glad I take this extra step, because about 5 - 8% of our tests will show different slightly results after the restore.

Quote:
Originally Posted by 2Tired2Tango
Along with the stuff you are disabling, you should also consider disabling system file checking, the computer browser, network location awareness, upnp discovery, net bios, net DDE, print spooler, shell hardware detection, SSDP discovery and the bits service. All of these can steal processor time without notifying you.
I appreciate you listing all these. As I mentioned in the article, I had no intention of going to extreme lengths in disabling system services. I used to do that all the time with our Windows XP benchmarking machines, and it could end up causing more harm than good. None of these services you mention are as beefy as the ones we disable, and most of them should never run at all (network-related, printing).

When Vista first launched, I spent a lot of time figuring out which services crept up during our benchmarking, and the ones I didn't disable, were far too minor to even matter. Things like Firewall, Defender, Update and especially Search had to be disabled. I've caught Search using upwards of 100MB of RAM before. That's understandable for a real machine, but ludicrous for a benchmarking rig ;-)

The goal was to not produce such a highly optimized machine that it becomes unrealistic, but to at least get rid of all the potential effected results by certain rogue services.

Thanks for the input.
__________________
Intel Core i7-3960X, GIGABYTE G1.Assassin 2, Kingston 16GB DDR3-2133, NVIDIA GeForce GTX 770 2GB
Kingston HyperX 3K 240GB SSD (OS, Apps), WD VR 1TB (Games), Corsair 1000HX, Corsair H70 Cooler
Corsair 800D, Dell 2408WFP 24", ASUS Xonar Essence STX, Gentoo (KDE 4.11. 3.12 Kernel)

"Take care to get what you like, or you will be forced to like what you get!" - H.P. Baxxter
<Toad772> I don't always drink alcohol, but when I do, I take it too far.


Rob Williams is offline   Reply With Quote
Old 08-18-2009, 12:39 PM   #4
2Tired2Tango
Tech Monkey
 
Join Date: Mar 2009
Location: Ontario Canada
Posts: 648
Default

Quote:
Originally Posted by Rob Williams View Post
I'm a little confused... you're talking about benchmarking in a non-Windows environment?
Wellll... sort of.

My concern is that in some cases you wouldn't actually be testing the device (video card, for example) but rather the environment it's in. The only way to truly test a device without interference from other tasks is to use hardware diagnostics in a heads up environment.

One example I can give you is my own benchmarking of hard disks done a couple of years back, when the 7200 rpm drives were coming out. I wrote code that would block multitasking during the tests and compared 5400 and 7200 drives, I also compared IDE and SATA. Without fail I got far better performance with the multitasker blocked than I did in even the most optimized standard windows environment.

However I do take your point that nobody actually does this so you're kinda stuck "going with the flow" in a sense. You're looking for "real use" style testing and that's not a bad thing... it's just not a pure test of hardware. It's not an invalid test so long as it's done in a way that is comparable to the way others are doing the same tests...



(Or it might be that, with the heat and humidity these last couple of days, I'm just being extra cranky...)
__________________
Tango

"An unexamined life is not worth living" ... Socrates

Last edited by 2Tired2Tango; 08-18-2009 at 12:42 PM.
2Tired2Tango is offline   Reply With Quote
Old 08-18-2009, 11:32 PM   #5
Rob Williams
Editor-in-Chief
 
Rob Williams's Avatar
 
Join Date: Jan 2005
Location: Atlantic Canada
Posts: 13,351
Default

I don't entirely disagree with similar methods, but it all comes down to what makes most sense for what we're trying to accomplish. Almost 90% of our visitors run Windows, so to benchmark anything else would send most people running. Plus, we're still looking to create realistic environments, and nobody is going to optimize the heck out of their machine realistically, except the hardcore enthusiast.

As you said, we're happy as long as our results are accurate from run to run. So far on the application side, we've had few issues with realiable results, and if there is a benchmark that sways a little bit from one run to the next, it's normally caused by storage. With our PCs equipped with Intel's latest SSDs though, we're even trying to rid that issue as much as humanly possible.
__________________
Intel Core i7-3960X, GIGABYTE G1.Assassin 2, Kingston 16GB DDR3-2133, NVIDIA GeForce GTX 770 2GB
Kingston HyperX 3K 240GB SSD (OS, Apps), WD VR 1TB (Games), Corsair 1000HX, Corsair H70 Cooler
Corsair 800D, Dell 2408WFP 24", ASUS Xonar Essence STX, Gentoo (KDE 4.11. 3.12 Kernel)

"Take care to get what you like, or you will be forced to like what you get!" - H.P. Baxxter
<Toad772> I don't always drink alcohol, but when I do, I take it too far.


Rob Williams is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may post new threads
You may post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is On

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
fan testing!!!! Psi* Overclocking and Cooling 3 07-04-2010 09:19 AM
Heatsink testing methodology b1lk1 Overclocking and Cooling 18 05-29-2010 07:19 PM
A Testing Hypothesis Rob Williams Reviews and Articles 10 07-17-2009 10:01 AM
Testing Motherboard tanusgreystar Motherboards 3 09-02-2008 02:49 AM


All times are GMT -4. The time now is 03:35 AM.