Heatsink testing methodology

b1lk1 · Apr 5, 2010

Hey guys, I figured I post this up there so we can continue the discussion about Techgage's heatsink testing methodology. I promise I will not close this thread or moderate posts. Let's keep the discussion civil and we can all learn something (including me).

My views on a synthetic bench for testing these things is that it is totally unrealistic for predicting how it will act inside a case in real world use. I do realize that all chassis are not the same as is the fact for CPU's and motherboards, etc... But, it is not that hard to see how it performs inside our test PC and equate those results to how well your PC will perform. If you are competent enough to install a heatsink then you have the requisite knowledge to formulate the effectiveness it will have on your particular PC.

While there are no plans on setting up a synthetic test bed in the near future one never knows what the future could hold.

Thoughts?

Optix · Apr 5, 2010

I like having the testing done in a case. Not many of us are running a tech station or leave our gear sitting on a box out in the open so as I said before the results are indicative of what most of us will be running.

What I would like to see is some sort of standard across the board though. A mid tower case with a single intake fan at X RPM/X CFM and a single exhaust fan at X RPM/X CFM. Also the room should be at a constant temperature for all tests, not just the current one.

I'm pretty new here so that may be the way it is done but this would ensure that you know what you're getting into with, what I would say is the bare minimum in the say and age for case cooling. Plus it would mean you could start listing past results so you could have an idea about how the heatsink of for review stacks up against past ones.

b1lk1 · Apr 5, 2010

Lucky for me I do all my testing in my basement. Near constant 68F year round, only varies by 2 degrees +/-. Also, we have a dehumidified/humidifier system that keeps the humidity constant around 60%.

I am using the Zalman GS 1000 chassis for all tests. It has a single intake fan on the bottom and I install one exhaust fan in the roof out of the 2 slots available. I also install one rear exhaust fan behind the CPU. When I test coolers like the ECO, I just remove the rear fan where the cooler mounts its own fan.

We also just built a nice i5 750 system for all testing from the ECO on forward. We chose the S1156 platform since it is the most mainstream. Same for the i750, it is one of the more affordable choices that is likely to be chosen by a majority of people.

If you read the CoolIT ECO review, you will see our current setup and methodology used. I will be posting new reviews with heatsinks as they come in so keep a watch for them to start appearing very soon.

Psi* · Apr 6, 2010

I thought I responded to this, but apparently I must have hit the wrong key to post ... 1st time for everything.

Reviews need to be factual. There is room for opinion in areas that are not tested. The reviewer noting personal views are interesting and important.

To dis a method without testing tho is just possibly wishful thinking & believing that one's logic is superior to empirical analysis. This is recognizing that personal experience ...aka credentials ... are worth while although not necessarily the last word.

Per the topic, it takes little effort to find competing reviews evaluating HSFs in the box (aka "real world testing" but with synthetic test software :-\) versus the so called synthetic test fixture. There are reviewers that test both ways and present the results as such. In doing so there is considerable correlation between both. An advantage of the synthetic fixture is that precise heat loadings can be tested at every 30 watts or so. This has a nice side effect of understanding what the thermal load of the CPU is with various stress software.

It also appears that reviewers that use synthetic test fixtures also look at other features of the HSF. Surface roughness of the base of the HSF for instance. Cooling efficiency of the heatsink with various fan speeds as well as dBA noise is also interesting.

Just to pick at fan noise, the Radio Shack noise meter (I have 1 of course, somewhere in my office but don't remember the scientific name) has a frequency bandwidth ... more sensitive at some range of the audible range than others. I *know* that a fan in the open will sound different that in any box or PC cases specifically. Also, different PC cases will "let" out different frequencies than other cases ... different sound filtering in other words. I have a few different Lian Li cases. One is optimized for cooling (lots of holes), a couple of others are optimized for noise (noise absorbing foam in the side panels). These are all mid towers but will have quite different results depending what is considered "best". Not to mention air flow efficiency. :-\

I recognize and appreciate the "in the box" testing and that it is necessary for marketing credibility in the world (industry?) of reviewers for enthusiasts. However, it introduces so many variables that a personal understanding needs to had of the reviewer to gain an understanding of his point of view. Ugh! More variables ... ya' think? Does this reviewer understand the "scientific method"? This is challenging for a written review. If complemented with a video review, then much more is revealed mostly unintentionally I suspect ... but the identical information is more credible.

b1lk1 · Apr 7, 2010

Our market is not in scientifically testing heatsinks. Our aim is to show the reader how well it performs on setup X and give that data so the reader can make a judgement decision as to how said headsink will work on their personal setup.

While I am sorry you feel we are not testing properly, we are doing the same type of testing that the majority of websites employs. As I have said before, we are not paid to do this nor are we given free access to any equipment to use for testing outside a basic PC configuration. All this extra test gear would be coming out of my pocket and seeing how I am married with 2 young kids with a single income to support my household, this is just not on my budget list.

I do realize we cannot make everyone happy, but you seem to be looking for information and testing that 99% of the websites reviewing heatsinks and cooling devices will not offer. I do welcome your critique and as Techgage grows who knows how technical our coverage will grow.

If I were testing units that cost hundreds or thousands of dollars I would completely agree that we need to give more scientific data. But the parts we test rarely exceed $75, never mind $100. They are not complex devices requiring a high level of knowledge to operate and install. To me, expecting the reader to know enough about them to form an opinion of their own is not un-realistic. Even if they do have questions, this is why we open discussion threads and people can ask our opinions on how it will perform in their system.

Anyhow, before I turn this into a diatribe, I just wanted to give some more background as to how and why we test as we do. I hope this helps people understand that we do very much care about giving accurate repeatable results with the equipment we have available on hand.

Psi* · Apr 7, 2010

My comments are solely based on what can be found for enthusiast PC HSFs ... and, I think I pretty clearly state that.

Rob Williams · Apr 9, 2010

Optix said:
I like having the testing done in a case. Not many of us are running a tech station or leave our gear sitting on a box out in the open so as I said before the results are indicative of what most of us will be running.

I had a discussion about this with NVIDIA the other day, and the person I was talking to actually said "Good good." when I said that we tested everything inside of a chassis. I can understand the convenience of a tech station, but as far as I'm concerned, that's NOT realistic. I don't know of a single person who runs their PC like that, and if there is someone, I'd like to know the reason why.

Some people think that a tech station improves airflow to components (given there is no frame), but that's not always the "case". You still need a good airflow scheme to rid hot air quickly and effectively... which is exactly the goal of many chassis out there. I know people even from other sites who test with a tech station but don't have an idea airflow scheme, so the heat that a component creates gets pushed nowhere... it just hovers around and slowly rises.

Psi* said:
There are reviewers that test both ways and present the results as such. In doing so there is considerable correlation between both.

If someone test drove a Ferrari in space, would that have any relevance to someone looking to purchase one? No, and the same thing applies here. There's no sense in testing outside of a chassis when that scenario applies to 0% of our readers.

Psi* said:
I recognize and appreciate the "in the box" testing and that it is necessary for marketing credibility in the world (industry?) of reviewers for enthusiasts. However, it introduces so many variables that a personal understanding needs to had of the reviewer to gain an understanding of his point of view. Ugh! More variables ... ya' think?

I have to agree with Bill in that going hardcore on testing heatsinks is not our goal, because it's not what we want, nor what our readers want. There is a significant drop in ROI when we focus on aspects and spend a lot of time on things that very few people will ever care about. The vast majority of our readers are enthusiasts and casual consumers, not people who've earned a major in thermal dynamics (well, I'm sure there are a few).

As much as I'd like to go all out on all areas of our content, it's just not feasible given our overall lack of manpower, money and desire (again, intricate testing isn't something that many of our readers care about). Despite all this, I wholeheartedly feel that we put more passion into our testing than most other sites out there, so I believe regardless of how we do things, our results are going to be accurate.

Optix · Apr 9, 2010

The people I know who are running a tech station or leave their gear out in the open account for about 1% and I spend a good amount of time on a lot of forums but even they usually have one or two fans aimed at the components instead of allowing things to run their course.

If you can start to build a temperature library by testing on the same platform I'll be one happy camper. That is the only thing missing from the Coolit ECO review. Most people like something to compare the results to so they can determine if they should shell out the money. $66 is a lot of change for a heatsink in my opinion if I were to use the ECO as an example.

Is there one member of your staff that performs all of your heatsink reviews? b1lk1? If so it sounds like he is all geared up to do this.

b1lk1 · Apr 10, 2010

We will be building from here out with the i5 platform for the foreseeable future.

Psi* · Apr 10, 2010

b1lk1 said:
Lucky for me I do all my testing in my basement. Near constant 68F year round, only varies by 2 degrees +/-. Also, we have a dehumidified/humidifier system that keeps the humidity constant around 60%.

I am using the Zalman GS 1000 chassis for all tests. It has a single intake fan on the bottom and I install one exhaust fan in the roof out of the 2 slots available. I also install one rear exhaust fan behind the CPU. When I test coolers like the ECO, I just remove the rear fan where the cooler mounts its own fan.

We also just built a nice i5 750 system for all testing from the ECO on forward. We chose the S1156 platform since it is the most mainstream. Same for the i750, it is one of the more affordable choices that is likely to be chosen by a majority of people.

If you read the CoolIT ECO review, you will see our current setup and methodology used. I will be posting new reviews with heatsinks as they come in so keep a watch for them to start appearing very soon.

It is terribly easy to go off topic & I apologize ... personal agendas & all that.

I finally read the CoolIT ECO review & noted the case ... I have some questions & will post at the appropriate thread.

Per this topic;
Do you replace the m/b back plate to that supplied with each heatsink?
Is there a test of an Intel stock HSF available for comparison?
I like that you are using OCCT Linpack. Not to raise a debate, but it kills stock Dell systems. LOL "Real world" == Dell fail or at least thermal throttling ... even the workstatiions.
I have seen comments made about concave/convex heatsink bases ... of the same heatsink. Maybe a close up of the razor blade test of the base?

Rob Williams · Apr 12, 2010

Optix said:
If you can start to build a temperature library by testing on the same platform I'll be one happy camper. That is the only thing missing from the Coolit ECO review. Most people like something to compare the results to so they can determine if they should shell out the money. $66 is a lot of change for a heatsink in my opinion if I were to use the ECO as an example.

Up to this point this has proven difficult, because situations and hardware keeps changing. For the past little while, we've merely been testing the waters, and because Bill takes care of more than just CPU coolers, we've had to be creative with how we are utilizing all of the hardware, and where we should dedicate certain pieces of hardware to. For our upcoming CPU suite, we should have a platform that will last a good while. For our upcoming one, if Bill hasn't mentioned it already, we'll be sticking to a LGA1156 socket, since it's Intel's mainstream offering and will have the greatest relation to the highest number of consumes.

Psi* said:
I like that you are using OCCT Linpack. Not to raise a debate, but it kills stock Dell systems. LOL "Real world" == Dell fail or at least thermal throttling ... even the workstatiions.

What do you mean, "kills"? If Dell's machines are unstable when running LINPACK, that's no one's problem... it's Dell's problem, and it's obviously doing something wrong. LINPACK-based stress-test tools are designed to be as hardcore on the machine as possible, which means a LOT of heat. If this affects Dell's machines in any way, then they have a fault worth looking at.

As for Intel's stock heatsink, we will include that information once we have the methodology and platform sorted out.

Psi* · Apr 12, 2010

as requested ...

I have avoided being contentious, but sometimes my will power is not what it could be.

@rob @bill ... I have seen many claims and nothing substantiating the claims.

Where is the data that indicates the LGA1156 socket is Intel's mainstream offering? Per a very unscientific "survey" at Newegg, check out the number of reviewers for the 1366 socket (& only the I7-920) versus 1156 socket (all of the CPU variations) .. to save you the time, it is 2,126 vs <1,000.

As far as the number of techgage's membership and if the "Site Contest: Win an Intel Core i7-980X Extreme Edition Gaming PC! " is some indication of the following, you have 201 (at the moment) after 2 weeks. Slim pickin's. Just to beat the previous point home, you can some how get enough data to support any of your claims when only a couple of hundred people visit your site in 2 weeks? Right.

Isn't thermal throttling a "fail" as far as Dell computers are concerned? And I stated this? yes? This is not so much a swipe at Dell as it is at using OCCT Linpack. Isn't this program considered synthetic? YES, it is! So how is it any more valid than a synthetic test fixture which can actually be calibrated to the real software (game) du joir?

I am not arguing for synthetic testing ... er ... synthetic fixture ... I am confused now. And the difference is what?

Am I the last one on techgage that ever googles anything? Does not anyone google "synthetic pc heatsink testing" for instance, much less "pc heatsink reviews". There are already dozens of reviews for the same products that b1lk1 seems confused about. More reviews of the same old stuff .. yawn ... and this is as creative as he can get? Then he deletes posts and locks the thread. I am sorry, but he is a jerk. The ability to do this on a forum with no rules ... could this be a reason why there are only 201 entries in your current contest. Your reviewers are lost in their own little world and no one really cares what they say because it has already been said elsewhere ... and many times.

Last, what was it with the ridiculous analogy of a Ferrari in space? Are you under too much pressure or something? Just to take this Ferrari thing a bit. Maybe you meant an analogy of sitting in a Ferrari in rush hour traffic versus the German Autobahn? Or versus a closed oval track as a synthetic test analogy? Although truly synthetic would be just on a dyno.

I'll not publish this as this is site is your dream. If you want to do things your way, then just say that it is being done this way because "this is my site". Don't come up with a bunch of made up bull shit. I am sure that I am not the only one that smells it.

Psi* · Apr 12, 2010

http://forums.anandtech.com/showthread.php?t=2050379&highlight=w5580

The end of post #32 are the comments about this $10K workstation failing with Linpack. For more Dell so called failures, just google for them. Is it a failure when only the synthetic test program fails?

Rob Williams · Apr 12, 2010

Psi*'s #12 post was originally sent as a private message, and I asked him to post it here. You can read it, and Techgage's staff posts in this and other threads, and draw your own conclusions.

Psi* said:
Where is the data that indicates the LGA1156 socket is Intel's mainstream offering? Per a very unscientific "survey" at Newegg, check out the number of reviewers for the 1366 socket (& only the I7-920) versus 1156 socket (all of the CPU variations) .. to save you the time, it is 2,126 vs <1,000.

LGA1156 is Intel's mainstream solution, it's not something we're making up. It's fact. LGA1156 covers the entire current-gen playing field, from the under $100 segment up to the enthusiast $600 segment. LGA1366 on the other hand is 100% part of the enthusiast segment, beginning with $300 and currently going up to $999.

Given that the LGA1366 Core i7-920 was available for a full 10.5 months before even the first LGA1156 hit the scene, it makes sense that it has a staggering number of reviews and the later chips do not. For most people, it was the only true option up until Lynnfield.

Psi* said:
As far as the number of techgage's membership and if the "Site Contest: Win an Intel Core i7-980X Extreme Edition Gaming PC! " is some indication of the following, you have 201 (at the moment) after 2 weeks. Slim pickin's. Just to beat the previous point home, you can some how get enough data to support any of your claims when only a couple of hundred people visit your site in 2 weeks? Right.

It's hugely ironic that you question our methods of calculating various figures, yet at the same time equate the number of entrants to our latest contest as the total number of people who visit our site. Our "10 Things Linux Does Better Than Windows" article, posted on Friday, has to date had 9,171 unique reads. Your numbers are a bit off.

Psi* said:
Isn't thermal throttling a "fail" as far as Dell computers are concerned?

No, I don't consider it a fail, because it's STILL stable. LINPACK-based tests are designed to be more hardcore than most people need it to be, because it stresses a PC in what could be considered a worst-case scenario. This to me is a good thing, because it tells us without a reasonable doubt that if the PC passes a LINPACK test, then it's stable. Just because the PC throttles with LINPACK, it doesn't mean it will throttle with typical high workloads.

Why do I like running with LINPACK? Because I like confidence, and LINPACK gives me that confidence (cue thumbs-up and super-white teeth smile).

Psi* said:
More reviews of the same old stuff .. yawn ... and this is as creative as he can get?

As I have mentioned before, we put far more of a thought process into our methodologies than most other websites, and we're pleased with what we come up with. There's a point where going too hardcore on certain types of testing is no longer going to interest most people. The majority of our visitors are not hardcore enthusiasts, and they are not going to enjoy exhaustive reviews (nor are those the kinds of heatsink reviews we want to write... again, ROI).

Also, it's not only Bill who comes up with a CPU cooler methodology, the entire team does. The entire Techgage staff talk on a regular basis and get input from each other before a methodology is ever finalized.

Psi* said:
Then he deletes posts and locks the thread.

Bill absolutely did NOT delete any posts, but he did lock two threads (and it's not meant to be a secret). I have attached the most recent part of our moderator log, stretching back to before Bill even created his original thread. If you can find proof within it that he deleted posts, please feel free to speak up.

Psi* said:
Last, what was it with the ridiculous analogy of a Ferrari in space?

As a reminder, here is a list of two things that our readers don't do:

1) Drive Ferrari's in space.
2) Use their PC outside of a chassis.

Psi* said:
Don't come up with a bunch of made up bull shit. I am sure that I am not the only one that smells it.

I can't exactly apologize for preferring to test via real-world methods in lieu of synthetics.

b1lk1 · Apr 13, 2010

OK, after having a night to sleep on this, here is my one and only reply to what this thread has degenerated into.

Personal attacks on internet forums are just no way to go about getting your point across. While I do not let such methods truly bother me, it is just distasteful that these tactics have once again resurfaced after we respectfully requested they end. Threads were closed and then new threads re-opened in the hopes this would not continue, but I see it is not going to happen.

I would like to reiterate that we are one of the very few sites that even open up our testing methodologies to public viewing and scrutiny. We ask for input and then we thoughtfully and respectfully reply to all input we get. We do not rashly dismiss ideas people have as that would stifle innovation and change.

Psi, you have the right to your opinions, but please refrain from your current trend of outright disrespect. I have not once disrespected you and have continually listened and researched what you asked. Let's get this topic back to civility and make it productive once again.

Psi* · Apr 13, 2010

b1lk1 said:
Psi, you have the right to your opinions, but please refrain from your current trend of outright disrespect. I have not once disrespected you and have continually listened and researched what you asked. Let's get this topic back to civility and make it productive once again.

For obvious reasons I originally sent that as a private message to Rob. He responded with "<!--[if gte mso 9]><xml> <u1:WordDocument> <u1:View>Normal</u1:View> <u1:Zoom>0</u1:Zoom> <u1:TrackMoves/> <u1:TrackFormatting/> <u1

unctuationKerning/> <u1:ValidateAgainstSchemas/> <u1:SaveIfXMLInvalid>false</u1:SaveIfXMLInvalid> <u1:IgnoreMixedContent>false</u1:IgnoreMixedContent> <u1:AlwaysShowPlaceholderText>false</u1:AlwaysShowPlaceholderText> <w

oNotPromoteQF/> <u1:LidThemeOther>EN-US</u1:LidThemeOther> <u1:LidThemeAsian>X-NONE</u1:LidThemeAsian> <u1:LidThemeComplexScript>X-NONE</u1:LidThemeComplexScript> <u1:Compatibility> <u1:BreakWrappedTables/> <u1:SnapToGridInCell/> <u1:WrapTextWithPunct/> <u1:UseAsianBreakRules/> <w

ontGrowAutofit/> <u1:SplitPgBreakAndParaMark/> <w

ontVertAlignCellWithSp/> <w

ontBreakConstrainedForcedTables/> <w

ontVertAlignInTxbx/> <u1:Word11KerningPairs/> <u1:CachedColBalance/> </u1:Compatibility> <w

oNotOptimizeForBrowser/> <u2:mathPr> <u2:mathFont u2:val="Cambria Math"/> <u2:brkBin u2:val="before"/> <u2:brkBinSub u2:val="--"/> <u2:smallFrac u2:val="off"/> <u2:dispDef/> <u2:lMargin u2:val="0"/> <u2:rMargin u2:val="0"/> <u2:defJc u2:val="centerGroup"/> <u2:wrapIndent u2:val="1440"/> <u2:intLim u2:val="subSup"/> <u2:naryLim u2:val="undOvr"/> </u2:mathPr> </u1:WordDocument> </xml><![endif]--> [FONT=&quot]Please do me an enormous favor and post this entire PM verbatim in our forums.[/FONT]"

FWIW I lost sleep after the post

Brett Thomas · Apr 13, 2010

For what it's worth -

I worked in the industry for several years before coming to Techgage, at what was (at the time) the premier modding and OC site in Europe (and for modding, arguably worldwide) - bit-tech.net. I have brought a lot of my own testing knowledge to bear with TG, and I still stand behind Bill's choice. Are there some things I'd change? Sure, we'd make sure the room is adjusted to be universally 68F instead of +-2. But in almost all other ways, Bill's testing is entirely repeatable, which is the hallmark to any scientific method.

You ask that we tell you it's because we want it, so here it is:
Techgage, as a team, has decided that synthetic heat-plate benchmarks do not provide IN OUR OPINION a beneficial return to our readers over a repeatable, clearly described real-world testing. THE END. It IS because WE think so.

Could a synthetic benchmarking be a nice addition? ABSOLUTELY, assuming that we took the time to explain with each article just as much about the synthetic bench (and why it doesn't mix with real-world) as we do with why we chose the real world.

HOWEVER: A synthetic bench will do one of two things - parrot the real-word (because the HSF is working almost or as efficiently in the case), or contradict the real-world drastically because different airflow designs greatly affected its efficiency. If the first part is accurate, it's a lot of extra work for something not many people (IN OUR OPINION) will use to supplement their buying decisions. If it's the opposite, well, then people will junk the synthetic bench in favor of the real world because they run their computer in a case to begin with.

Call it cost-benefit if you will - we can either give good, repeatable results on a variety of HSFs in a timely manner, or we can spend twice as long with each $50 HSF to turn our confidence intervals from 90% to 95%, while taking that time out of other products you could be aware of.

I'm not saying it's because other sites don't. I'm not saying it's because synthetic benching is useless. I'm saying that it's because our opinion is that the additional cost and time for producing this additional test does not warrant our readers enough of an added benefit.

To be perfectly honest, if you find an HSF that you are interested in here, we know you will probably go to SPCR or something to get the absolute scientific dB test and specific deltas. It's our job to do a GOOD job of providing you with an overview (that is repeatable and trustable) of as many options as we can so that you can see if it meets your needs first, so you can spend your specific research time wisely.

I hope that helps put our position in a little bit clearer light!

Psychosomatic · Apr 13, 2010

Psi* said:
Where is the data that indicates the LGA1156 socket is Intel's mainstream offering?

Besides Intel's own marketing, I'd look at Valve's Steam Hardware Survey. While it doesn't explicitly differentiate between sockets, it does give a unique perspective into the "average gamer" and their rig.

Here are some excerpts:

Processor Vendor
Intel - 69.49%
AMD - 30.51%

Intel CPU Speeds
1.7 Ghz to 1.99 Ghz - 8.03%
2.0 Ghz to 2.29 Ghz - 8.45%
2.3 Ghz to 2.69 Ghz - 26.84%
2.7 Ghz to 2.99 Ghz - 10.79%
3.0 Ghz to 3.29 Ghz - 8.62%

Physical CPUs (Intel and AMD)
2 cpus - 55.16%
3 cpus - 0.95%
4 cpus - 26.54%

Multi-GPU systems
NVIDIA SLI (2 GPUs) - 1.91%
ATI Crossfire (2 GPUs) - 0.12%
NVIDIA SLI (4 GPUs) - 0.04%
NVIDIA SLI (3 GPUs) - 0.03%

Given that Steam has something around 2.7 million simultaneous users during daily peak time, I'd say that it's a large enough sample.

GFreeman · May 29, 2010

I like these reviews written here. They are excellent quality reviews and factial if you ask me. When writing something you always have to take in consideration that beginners might not get what you are on about. I find the articles interesting to read, informative and easy to understand. Good comparisons too between the different products too! Thank you for sharing this information, keep up the good work

The hard work is appreciated!

Heatsink testing methodology

b1lk1

Tech Monkey

Optix

Basket Chassis

b1lk1

Tech Monkey

Psi*

Tech Monkey

b1lk1

Tech Monkey

Psi*

Tech Monkey

Rob Williams

Editor-in-Chief

Optix

Basket Chassis

b1lk1

Tech Monkey

Psi*

Tech Monkey

Rob Williams

Editor-in-Chief

Psi*

Tech Monkey

Psi*

Tech Monkey

Rob Williams

Editor-in-Chief

Attachments

b1lk1

Tech Monkey

Psi*

Tech Monkey

Brett Thomas

Senior Editor

Psychosomatic

Obliviot

GFreeman

Coastermaker