Grr, motherboard benchmarks!

[Rob Williams]

Whew, apologies in advance for such a long post.

I am in the process of re-evaluating how we review and benchmark motherboards, and would love some input from the community, since you are the reason we are here doing what we do

What matters most to you in a motherboard review? Is it performance? Features? Overclocking-ability? When you read a motherboard review, what do you hope to walk away with?

The reason I ask, is that I'm considering lessening the amount of benchmarks we use, and focus more on other areas, such as overclocking, features and et cetera. As it stands, I'm finding it difficult to find benchmarks (real-world or not) that enable repeatable, accurate results.

Take for example PCMark Vantage. Earlier, I benchmarked the main suite five times in a row on the Intel DX48BT2 motherboard. Here's how the scores went (from memory):

5,400 / 5,002 / 4,700 / 4,700 / 5,015

Results like that don't exactly make me feel that confident when making up a graph. In reality, if I took the first score I saw, it would put this board on top of all that I have here, when in truth, that might not accurately portray real performance (exactly why would a motherboard perform so much better than another, anyway?).

SYSmark 2007 Preview is another that tends to vary, but usually not as much. Bapco has just released a new tool that "optimizes" the PC in various ways (especially helpful in the bloated Vista), so that repeatable results can be seen. I'm in the process of testing that out now, so I am hoping it will turn out good so that we can rightly continue using it in our CPU and motherboard articles.

As far as other benchmarks go, VirtualDub/DivX seems to be extremely reliable (1 or 2 second variances), as does 3DS Max 9 and Adobe Lightroom. 3DS Max 9 is always within the second (not saying much since each run lasts under a minute).

SANDRA's Memory Bandwidth varies a little bit, but that's to be expected. That one will just be a run of five times and averaged off. The key factor there will just be seeing if one motherboard is far worse than it should be. No one is going to care if a motherboard is 50MB/s slower when dealing with 7,000MB/s figures. HD Tach, for testing the I/O, is another one that can vary a little bit, but the goal again is to make sure the board doesn't suffer in some key area.

3DMark Vantage also proved to be pretty reliable overall, but it's use in a motherboard review is pretty debatable. So I'm curious to hear from you guys. Do you have benchmark recommendations, or have suggestions on what you'd like to see in our motherboard reviews? I'm kind of in a bind here... I've have four motherboards here that need reviewing, but I've been spending so much time on figuring out a perfect methodology, I've been unable to get one up :-/

Let's hear your thoughts!

(PS, reviews in progress are Intel DX48BT2, ASUS Rampage Formula, ASUS P5E3 Premium & Gigabyte X48T-DQ6)

 

[Kougar]

Tough call. And you put it pretty well, take one motherboard and it's going to perform the same as most others. Major brands all perform the same, econo-brands that focus on price foremost tend to perform a little less, but otherwise it all depends on the chipset to see any difference.

I'd say an emphasis on BIOS options, features, and overclocking are what I personally look for, the benchmarks are great for showing that no board has some strange deficiency, but otherwise I don't pay attention to them beyond that.

 

[Merlin]

Tough call. And you put it pretty well, take one motherboard and it's going to perform the same as most others. Major brands all perform the same, econo-brands that focus on price foremost tend to perform a little less, but otherwise it all depends on the chipset to see any difference.

I'd say an emphasis on BIOS options, features, and overclocking are what I personally look for, the benchmarks are great for showing that no board has some strange deficiency, but otherwise I don't pay attention to them beyond that.

That sounds good.

May I add that some of the new boards like certain Ram, even though it would just state that ( for instance ) this board takes 1200Mhz Ram....then you buy that ram and it doesn't work.
That, in a review would be great.
And any other little quirks

:techgage::techgage: Merlin :techgage::techgage:

 

[Kougar]

Oh yes, I always mention quirks when I can. I run into them frequently, and they are more of an issue than some people realize.

P31 chipsets for example... no end to the RAM compatibility issues. Might as well be a 975X MCH, as they share the same limitations with RAM. Some Gigabyte boards don't boot 2.2v, even with Crucial Ballistix kit. Their 965P boards had issues with D9 chips. I think it was Nvidia's reference 680i boards that didn't work with 65nm Quads + 4 RAM module combos.

As of late, another thing I am now paying big attention to is component choices, and power circuitry. # of CPU phases, but also # of phases on the memory and northbridge... even video cards. GPUs used to have just two phases and one more for the RAM, but considering how much power they require.... Quite a few manufactureres have been offering 3-phase 8800GTs, for example.

Then there is Gigabyte... X38 has a single phase for each of the RAM+MCH power, but X48 finally fixed this and comes with two phases for each. When using four RAM modules (Especially at high frequencies), two-phase power should be critical.

Of course... things will be flipped on their head once Nehalem arrives. It's going to be so fun...

 

[Rob Williams]

Sorry for the late response guys, I've been caught up with work.

First and foremost, thanks a lot for the input, it's always appreciated. The fact is, I do need to change a few things in the reviews, and I like what's been said here. From now on, I'll run a few select benchmarks, but focus more on overclocking and features unique to that board.

May I add that some of the new boards like certain Ram, even though it would just state that ( for instance ) this board takes 1200Mhz Ram....then you buy that ram and it doesn't work.
That, in a review would be great.

This is not a bad idea at all. Although I'm not going to realistically test a wide-range of memory to make sure it works, I could stick to Google and see if there are any consistent issues, and mention them in the review.

Regarding power phases, I will begin including a little mention of that as well. As for GPUs, Palit is one of the originators in that, and it's something they like to brag about. Personally, I have no clue how many phases a GPU or motherboard has just by looking at it, so I'll have to do a little research.

Whew, my mind is just in a huge spin lately, because I have no idea how I'll proceed with things. It's just so hard to settle on a methodology, when there is a hundred ways to do it.

I might also be changing our GPU reviews a little bit also. Right now, I benchmark eight games on all of our high-end cards... I am heavily thinking of decreasing that to six or even five, and our "budget" cards from six to four or three. Since we manually benchmark every single game, it makes sense to take the route that's still effective, but also frees up a lot more time.

In the end, I have a feeling a lot of people could care less if there was six or eight games used. If one card is faster than another, then it's faster than another.

/me slams head off of desk.

 

[Merlin]

In the end, I have a feeling a lot of people could care less if there was six or eight games used. If one card is faster than another, then it's faster than another.

I would say too many games, maybe just the latest games.
And rendering in Cinema is for the video render cards , that's another ballgame.
Let's see....
I look at power consumption and power connection, shaders, ram and type of video ram ( DDR3, DDR4 ).
I do look at the Crysis specs, seems most compare that to what resolution they can run it at.
That's a popular one...
Any game below Crysis you know would run well.

The comparision chart, with other cards as to where that cards falls to the others.

I look at final thoughts...pros and cons

You're right on the mark at testing..... for myself, I skim over the rest.

:techgage::techgage: Merlin :techgage::techgage:

Post note.........oooops....I wrote this for video cards ...my bad

 

[Rob Williams]

After some contemplation, I've decided on these benchmarks for the motherboard reviews:

SYSmark 2007 Preview

DivX + VirtualDub
Adobe Lightroom
Autodesk 3DS Max 9

HD Tach RW/3
Sandra Memory Bandwidth

3DMark Vantage
Half-Life 2: Episode Two

SYSmark 2007 is a lot more reliable than PCMark Vantage, so I'm sticking with it and dropping the latter. The suite includes a mode to loop it three full times, while averaging off the scores, and each time I've run it, the scores have never varied that much. The overall score would sway one or two points, but that's still a lot better than the massive differences PCMark can show.

DivX, Lightroom and 3DS Max are there because they each tackle a unique multi-media task: Video Encoding, Picture Editing and 3D Rendering. Plus, they have all proven to be SPOT ON with each run, so they are all good to use.

HD Tach will make sure that the motherboard excels at delivering solid hard drive performance, while Sandra's Memory Bandwidth tool will be used primarily to make sure a board is in line with where it should be. So far, no board has really swayed far, but that's a good thing.

3DMark Vantage and HL2 are the gaming benchmarks used. In my tests, 3DMark Vantage also proves quite reliable in delivering repeatable results, so it will be used going forward. Half-Life 2 is the game of choice, as it's a title that thrives on both the CPU and GPU, unlike some games, such as Crysis, which could almost care less about the processor.

That takes care of the benchmarks. Past that, I'll try to focus on other features of the board. I can't really say what, though, as I've had no time this weekend to tackle it. I am hoping to have the P5E3 Premium review up on Wednesday at the latest, however, so I guess you'll know by then. Once posted, I'll definitely be looking for more suggestions

As for GPU reviews, I think I'll use Crysis, Call of Duty 4, Half-Life 2: Episode Two, Call of Juarez and Unreal Tournament III. Going from eight to five would make a huge difference in the time it takes to benchmark each card, and as I've mentioned already, I can't see it bothering too many people. The fact is, we manually benchmark each and every card that passes through here. I hope people appreciate that more than ten games that are run through a timedemo.

 

[Rob Williams]

Apparently I don't know what I'm doing, because on a whim, I decided to install the E8400 and suddenly could achieve nice FSB overclocks, as seen in the ASUS P5E3 Premium review.

Oddly enough, I did try Dual-Cores in the past, but they must not have been very FSB friendly. The E8400 opened up huge potential here.

 

[Kougar]

Congrats Rob, about bloody time!!

Now if you can just raise that CPU multi now... Is this the same chip you had overclocking issues with before? There's really nothing to it, but there are plenty of superfluous BIOS settings that'll only limit clocks instead of helping with them, from my limited experience. Can ya think of any usual settings you might've done differently this time?

 

[Rob Williams]

The only difference is the CPU. The max FSB with the QX9650, which I've been using for a while, is 470FSB. On the other motherboards where I wasn't able to hit 500MHz, I was using the Quad-Core, which is why I never hit the goal, I am assuming.

It's kind of foolish, actually. With a Quad-Core, the FSB is much harder to raise, but it could all depend on the CPU itself. Another QX9650 might allow 500MHz FSB on the same board, I don't know. I do recall switching to an E6600 on one board and was still unable to hit 500MHz FSB, so I have a feeling it was the CPU that was the problem at that time.

This E8400 just blew the doors open, though. 500MHz FSB is possible on both the P5E3 Premium and also the Gigabyte X48T-DQ6. Tomorrow I might install the Rampage Formula and see how that goes, but I assume I'll have no issue reaching the same. It's the Intel board I'm not looking forward to overclocking... those are not fun at all, going by my experience with Bad Axe 2 and the D5400XS.

As for increasing the CPU multi... I'm not looking to overclock this CPU. I'm just using it for the sake of reaching a high FSB on the board Definitely looks like I should get some DDR3 in though, but I might wait until after Computex next month. So much work here to do already

 

[Kougar]

Getting anything over 425FSB on a Quadcore that is perfectly stable on the full multiplier is extremely rare... even with 45nm Quads 500FSB is NOT something that is common, infact I'd call it a fluke chance if someone was able to do so without using the lowest multipliers... or maybe even WITH using the lowest multipliers.

You should be able to hit 500FSB+ without touching the multiplier on the Gigabyte board, and I beleive you will not even need to touch the MCH voltage until you go above that 500 mark level.

What's the fun of a high FSB without an actual increase in the CPU frequency? I never was much of a fan for pure FSB overclocks, I prefer to see the maximum final stable clockspeed! Of course I tend to usually be practical, so that'd be why...

 

[Rob Williams]

Like I mentioned, I am not looking to overclock the E8400, so I'm not worried about changing a multi. Keeping it 500x6 means I can leave the CPU voltage alone, since it's still at stock speeds.

"What's the fun of a high FSB without an actual increase in the CPU frequency?"

If it were a CPU review, I'd overclock the CPU. This is a motherboard review, so all I care about is the FSB overclock

 

[Merlin]

Like I mentioned, I am not looking to overclock the E8400, so I'm not worried about changing a multi. Keeping it 500x6 means I can leave the CPU voltage alone, since it's still at stock speeds.

"What's the fun of a high FSB without an actual increase in the CPU frequency?"

If it were a CPU review, I'd overclock the CPU. This is a motherboard review, so all I care about is the FSB overclock

What board did you use?

I tried mine and could only get uo 475 FSB on the EVGA 780i board

:techgage::techgage: Merlin :techgage::techgage:

 

[Kougar]

If it were a CPU review, I'd overclock the CPU. This is a motherboard review, so all I care about is the FSB overclock

Ah, but a purely FSB only overclock tells nothing about how the board will overclock on a full CPU multiplier. When Quads first were around most boards seemed to have serious issues with the Q6600's 9 multiplier, and most users had to lower it just to overclock because the boards weren't tuned right.

It's just my personal preference speaking, but I would compare a pure FSB overclock to having as much relevance as say gaming benchmarks done at 800x600 resolution.

 

[Rob Williams]

It's just my personal preference speaking, but I would compare a pure FSB overclock to having as much relevance as say gaming benchmarks done at 800x600 resolution.

Ok, I see what you are getting at now... I'm a little slow. I do have to disagree though. The FSB is a FSB, it has nothing to do with the multiplier except the obvious. If an FSB hits 500MHz without issue at 500x6, then 500x9 shouldn't prove a problem either, unless the CPU isn't going to handle it (at 4.5GHz, it's likely not to).

This particular motherboard review was the first one where I didn't bother overclocking the CPU in order to find the max FSB... because it is easier to just not have to worry about the CPU at all. In the past, I did keep the CPU multi at stock as long as I could, lowering it only when I needed to.

I have never experienced an occasion where lowering the multiplier availed me a higher FSB overclock. In my testing, I usually did lower the multiplier in order to lessen the stress on the CPU, but I've never had that help me achieve a higher FSB.

In your scenario with the Q6600, I don't think using a 9x would lessen the chance of a successful overclock. Even at an extraordinarily modest 400MHz, that would put the CPU at 3.6GHz. That in itself is not stable for everyone (including me). If someone had an ES chip and could either hit the same clock stable with a lower multi but higher FSB, or vice versa, then I would like to investigate further.

But as I said, I've never experienced this personally, and really don't believe the multiplier itself has much say in the stability of a system. So while I'm not saying you are wrong, I just have never experienced it. It would be difficult to test nowadays as well, because while 500MHz is achievable, it's hard to test that theory when you don't have a Dual-Core that can handle 4.5GHz without issue. I know I sure don't have one of those lying around, haha.

Another thing to bear in mind, is something that I don't think many people realize. If an FSB at 500MHz is stable at 1.55v, when using an E8400 for example, with a 6x multiplier, it does not mean that the same FSB voltage will be stable when the CPU is overclocked, using an 8x or 9x. Though the FSB itself didn't change, the CPU did, adding to the stress on the system. To counteract this, even more voltage on the NB will be required. This is something I've have to do numerous times in the past, including with the picture below. While 470x6 was fine and stable with 1.55v, I had to bump it to 1.69v for 470x9.

 

[Kougar]

Ok, I see what you are getting at now... I'm a little slow. I do have to disagree though. The FSB is a FSB, it has nothing to do with the multiplier except the obvious. If an FSB hits 500MHz without issue at 500x6, then 500x9 shouldn't prove a problem either, unless the CPU isn't going to handle it (at 4.5GHz, it's likely not to).

I would have thought the same, but going by what I was reading on the XS forums awhile back people were claiming higher overclocks with lowered multipliers. Best I could tell it was an issue with BIOS tuning, it took awhile for 965P to get up to par with overclocking Quads and some of the Extreme dualcore chips. I understand what you are saying though.

From what I've seen post Conroe, ES chips are not as good a bin as launch silicon... Nothing against your screenie but just in general I tend to ignore ES sample overclocks unless the overclocker/website doing the OCing specifically mentions the ES chip's known capabilities. Even then often the ES chip typically gets maxed out early on.

I wouldn't know why raising the CPU multi would affect the NB voltage, since as you said the FSB stays the same. If the FSB stays the same, then so does the RAM... hmm, on second thought it would directly affect the tRD setting, so I take that back. Good point

 

[Rob Williams]

Well for the time being, we can just agree to disagree, since I still don't believe that the multiplier has anything to do with the stability of an overclock. The fact is, with a 9x multiplier, the clock speed is high, so it's hard to blame the multiplier for anything. If the same overclock could be accomplished with a much higher FSB but lower multi or vice versa, then I'd begin to rethink it.

I am confused with the ES comment. If the launch silicon is better, wouldn't that make the screenshot more impressive? Or am I completely misunderstanding this? I personally don't believe ES chips are different than retail offerings. I don't doubt that they are cherry picked, but I've seen many people achieve far better overclocks with retail chips than I have with ES, so I consider them to be representative of what most people would receive.

Aside from that, I use the E8400 Dual-Core for testing out max FSB, which is not an ES chip, but still manages to handle a combination of high FSB and high multiplier.

I wouldn't know why raising the CPU multi would affect the NB voltage, since as you said the FSB stays the same. If the FSB stays the same, then so does the RAM... hmm, on second thought it would directly affect the tRD setting, so I take that back. Good point

"Though the FSB itself didn't change, the CPU did, adding to the stress on the system. To counteract this, even more voltage on the NB will be required."

The FSB did not change, but the added stress to the system as a whole requires more voltage for stability. Why exactly is this the case? I have no idea, but I've experienced it enough to believe it.

I don't think we'll agree on this anytime soon, but I'm still confident that it doesn't matter what multiplier is used. If an FSB overclock can be hit, an FSB overclock should be feasible at any multiplier, as long as the CPU can handle it. Plus, you don't seem to trust overclocks with ES chips anyway ;-)

As it stands, I like testing with low multipliers because it eases off the CPU and means I don't have to push 1.5v all the time into it (that QX9650 was 1.575v, which is why I was antsy to stop it after an hour).

 

[Kougar]

Well for the time being, we can just agree to disagree, since I still don't believe that the multiplier has anything to do with the stability of an overclock. The fact is, with a 9x multiplier, the clock speed is high, so it's hard to blame the multiplier for anything. If the same overclock could be accomplished with a much higher FSB but lower multi or vice versa, then I'd begin to rethink it.

Exactly! However I quickly gave up trying to dig up some XS forum threads from late 2006 to prove it, so that's fine. Half the time I had just attributed it to new overclockers not knowing exactly what they were doing.

I am confused with the ES comment. If the launch silicon is better, wouldn't that make the screenshot more impressive? Or am I completely misunderstanding this? I personally don't believe ES chips are different than retail offerings. I don't doubt that they are cherry picked, but I've seen many people achieve far better overclocks with retail chips than I have with ES, so I consider them to be representative of what most people would receive.

I'd actually made that comment on ES chips before I noticed your screenie WAS an ES chip! I kinda get what you are saying about it being more impressive, probably some truth to that. I didn't mean anything specifically except that a retail QX9650 should hit another 200-300 MHZ though IMHO, at least near 4500MHz?

"Though the FSB itself didn't change, the CPU did, adding to the stress on the system. To counteract this, even more voltage on the NB will be required."

The FSB did not change, but the added stress to the system as a whole requires more voltage for stability. Why exactly is this the case? I have no idea, but I've experienced it enough to believe it.

Yeah, I wasn't seeing it until I realized that is exactly what affects tRD. tRD by itself is the a CPU multiplier of sorts that basically controls the NB's internal clocks. The chipset has it's own multiplier with various FSB straps, and the tRD affects that multiplier. Is why a lower tRD yields higher performance, but requires a good amount of NB voltage when trying to hold it at 6 with a 400FSB overclock... the chipset is basically giving it at all its got. Anything near 500FSB, better off setting a 11 or 12 tRD and tightening it again after hitting the max FSB to take that out of the OC picture.

Agreeing to disagree is fine, I certainly don't expect to sway anyone without proof, and I'm not going to attempt to dig up those XS threads! ES sample overclocks are fine, and I have nothing against them. But when using an ES chip to explore how well a new motherboard overclocks for review purposes, I think a better clocking retail chip gives a better picture.

 

[Rob Williams]

Sadly, I don't have much time to conjure up a nice response, but I appreciate all of your opinions. I do personally think an ES chip is worthy of being used, as I don't believe it to differ that much from a retail chip. I am incredibly confident that anyone with a retail setup should be able to hit the same clocks, and if I found continuous evidence of that not to be the case, then I'd change my tune.

That aside, you still care about high overclocks along with the FSB, while I don't. So it would make little sense to even consider picking up a retail chip, since I still strongly believe there is no difference. That also said, overclocking is not that important to me, to go through such lengths. Being able to go over 400MHz might matter for some people, but for them majority of our readers and those who are planning to pick up a new board, overclocking is not everything.

 

[Kougar]

I am incredibly confident that anyone with a retail setup should be able to hit the same clocks, and if I found continuous evidence of that not to be the case, then I'd change my tune.

I have no issue with this, I would fully agree with you here. My point was a retail chip would overclock farther, therefore it naturally is a better test of a motherboard's capabilities. Basically I'm quibbling over pennies of difference.

If it was a spectacular ES chip that was obviously above average, then by all means use the ES chip. I just prefer to see launch chips because they are slightly better... however I also prefer to see the reviews state what the maximum results were with the CPU from past experience, as to provide a guideline. Some ES chips have some brutal FSB walls where they just won't function above.