AMD Rejects BAPCo's SYSmark 2012 - Should We?

[Rob Williams]

In a rare move, AMD has publicly ousted a leading industry benchmark, BAPCo's SYSmark 2012, released mere weeks ago. The company believes both that the benchmark is unfair in its weighting of scores, and also that it's irrelevant for the average consumer. We delve deep into these claims, and offer up our own opinions.

Read through our in-depth look into the matter of AMD's rejecting SYSmark 2012, and once done, discuss it here!

 

[Unregistered]

bapco Not for profit?

It may not make a profit but its entire reason for being is to help the member companies make more profit.

Why don't they let the users select the weighting factors? After all I'm the best person to know what I use my computer for.

 

[Rob Williams]

Why don't they let the users select the weighting factors? After all I'm the best person to know what I use my computer for.

Or, it could at least base the testing off of current and on-going trends. There are numerous reports out there that show what people are doing with their PCs, so it's not an unknown. But aside from that, SM2012 runs some tests with outdated software, so there's more than just one factor plaguing it.

AMD wants to see a new consortium form to right the wrongs of BAPCo, but I think it's easier said than done. Even Intel itself has created a couple of in-house test suites that largely went unused and were forgotten about.

We've even considered building a similar suite using all of our real-world benchmarks, and then build a score of some sort out of the aggregate. That in itself was challenging enough that we had to give up due to it being a huge time-sink.

 

[Unregistered]

history of skewed benchmarks

Somewhere in a closet I have a publication Motorola put out about 80286/68020 comparisons that it said Intel was fudging; I wish I could dig it out.

And of course we all remember the Intel compiler that checks for the availability of certain instructions in a way that makes sure they won't be used for AMD processors that in fact support them, even though they set the appropriate capability bits to indicate their presence in a proper test.

It's getting close to July 4th, so the phrase "a long train of abuses and usurpations" is perhaps appropriate.

 

[Kayden]

Given the application list above, we can't jump to conclusions and state that SM2012 does not make use of the GPU, because some of the applications listed can. We won't be able to prove it to ourselves until we receive our copy and can monitor the GPU's usage.

Until you can test it in house NO. There is a difference between Faith and Blind Faith. With out a specific reason and real data to back it up with a 3rd party to back it up, there is no reason for me to take this on Blind Faith from AMD. They are a good company for what they do but they do not deal in the truth and I do not believe any one company with out proof.

 

[Kayden]

Found this TG article on Slashdot btw.

http://tech.slashdot.org/story/11/06/21/1958226/AMD-Rejects-SYSmark-Benchmark

 

[Unregistered]

In a rare move, AMD has publicly ousted a leading industry benchmark, BAPCo's SYSmark 2012, released mere weeks ago. The company believes both that the benchmark is unfair in its weighting of scores, and also that it's irrelevant for the average consumer. We delve deep into these claims, and offer up our own opinions.

Read through our in-depth look into the matter of AMD's rejecting SYSmark 2012, and once done, discuss it here!

First off, thanks for letting us reply without registering.

I'm going to side with AMD here to a point. My biggest argument is what do they suggest? I do think since this is a test for businesses that this test should only be published with a required disclaimer saying it's for business applications and the scores do not reflect what average consumers would experience. Because AMD is right, consumers should not look at these results and use it to compare two CPUs to decide which is better. I have not used FineReader OCR since 2001 (FYI it worked great back then!) and I would not care how well a CPU does OCR.

As a consumer I am concerned about internet functionality. Different versions of flash, silverlight and JavaScript should be tested. Google Chrome, Opera and different versions of IE should be on the list. iTunes is missing too. Transcoding is probably included in After effects, if not it should be on this list too

 

[Unregistered]

Rubbish

Somewhere in a closet I have a publication Motorola put out about 80286/68020 comparisons that it said Intel was fudging; I wish I could dig it out.

Motorola and Intel were testing apples vs oranges for years. A lack of compilers compatible across both platforms made this entirely useless... never once did you see a test run for example on a System V based 68000 and a System V based x86. Fact of the matter is, in the very rare cases where you did, you'd see multi processor 68000 based workstation beasts up against IBM ATs. Clock for clock, the 68020 and the 80286 were so damn close in performance that it made little difference. We were talking about 1-2% in favor of one or the other. Intel fudged these tests and Motorola fudged these tests... sales people who didn't have a clue paid for tests to be run on both sides which would show them in favor.

And of course we all remember the Intel compiler that checks for the availability of certain instructions in a way that makes sure they won't be used for AMD processors that in fact support them, even though they set the appropriate capability bits to indicate their presence in a proper test.

The Intel compiler checked for processor capabilities based on the processor model and generation. This is how many programs handle this type of testing. You don't check for individual instructions... if you did, you'd have to recompile for every combination etc... Instead, you optimize one or two different ways and if all the instructions for the faster path is supported, then you execute that one. AMD was being a whiner baby then as they're being a whiner baby now. All they really had to do was get in contact with Intel (which oddly enough, they DO know how to work together with) and ask them to add their CPU IDs to the list or offer them an alternative piece of code for processor detection for their compilers which would make it possible to identify AMD processors capable of running those pipelines. Instead, some whiner baby ran to the press and cried about how Intel was being so mean because they didn't bother testing to see if their competitor supported certain instructions.

It's getting close to July 4th, so the phrase "a long train of abuses and usurpations" is perhaps appropriate.

One of the classic blunders from hell. Claiming George III was our enemy, an abuser and a usurper. Oddly, after writing our declaration of independence, we learned it was actually a greedy parliament and George III was the closest thing we had to a friend as he fought in our favor against the actual abusers. He just failed to get very far with it. So... maybe again you're singling out the wrong enemy because some idiot with a journal told you he didn't like someone? Oh... sorry, you haven't read this far in the posting, you stopped at the headline and you'll go from there like with everything else won't you?

 

[Rob Williams]

Somewhere in a closet I have a publication Motorola put out about 80286/68020 comparisons that it said Intel was fudging; I wish I could dig it out.

You have to give credit to Intel for one thing... few other semiconductor companies have been accused of cheating for nearly 30 years ;-)

And of course we all remember the Intel compiler that checks for the availability of certain instructions in a way that makes sure they won't be used for AMD processors that in fact support them, even though they set the appropriate capability bits to indicate their presence in a proper test.

That's one reason I've come to appreciate SPEC's CPU benchmark, since it's architecture-agnostic. Nowadays, though, Intel has the biggest advantage by being the first to market with certain instruction sets, while it seems that those that AMD introduce don't get used in commercial applications. SSE4 is a big example of this, where huge gains could be seen with video encoders that could take advantage of it.

Until you can test it in house NO. There is a difference between Faith and Blind Faith. With out a specific reason and real data to back it up with a 3rd party to back it up, there is no reason for me to take this on Blind Faith from AMD. They are a good company for what they do but they do not deal in the truth and I do not believe any one company with out proof.

Our copy will be here before the weekend; I'm not sure what's taken it so long. Due to the AMD Lynx launch right around the corner, I am not sure how much time I'll have to dedicate to testing it out, but I hope to tackle it soon. BAPCo told us that a whitepaper would be released soon, so I'm looking forward to looking through that.

First off, thanks for letting us reply without registering.

No problem; thanks for taking the time to post!

I'm going to side with AMD here to a point. My biggest argument is what do they suggest? I do think since this is a test for businesses that this test should only be published with a required disclaimer saying it's for business applications and the scores do not reflect what average consumers would experience. Because AMD is right, consumers should not look at these results and use it to compare two CPUs to decide which is better. I have not used FineReader OCR since 2001 (FYI it worked great back then!) and I would not care how well a CPU does OCR.

I couldn't agree more on the latter point. While it does have its niche market, most regular consumers don't even know what OCR is, and if they did, they likely wouldn't have a need to use it. So to include it in a benchmarking suite that aims to give consumers a good idea of what their PC is capable of, it's a bit strange. It'd be like Toyota giving us stats on how well its Corolla handles driving on molten rock... it's just not that relevant for most people.

As a consumer I am concerned about internet functionality. Different versions of flash, silverlight and JavaScript should be tested. Google Chrome, Opera and different versions of IE should be on the list. iTunes is missing too. Transcoding is probably included in After effects, if not it should be on this list too

To be honest, I think some of the things mentioned here are a little more specific than this benchmark intends to be. If SYSmark had the goal of dividing up the scores at the end further to explain which browser performed the best, that might be a little different, but this is a benchmark designed to last for a couple of years. That's where things get tough, because SM2012 releases with IE8, when IE9 is available... and when this benchmark is retired, we might well see IE11. At the same time, the IE choice in general is a little questionable.

I agree on the general idea though. Lots of people use things like iTunes, so that test (or one similar) should be included. The same goes for transcoding and the like, though SM2012 might touch up on that a little. With what you've mentioned, I think PCMark 7 is the better benchmark for that kind of consumer-level use. It tests out some Web technologies in a custom browser, and also tackles video conversion, image manipulation, et cetera.

That all said, outside of suite usage we use individual benchmarks in our own testing for all the things mentioned above. We don't use iTunes, though we might implement dBpoweramp for the sake of encoding later (unless we can see that iTunes proves in itself to be a reliable, repeatable benchmark).

All they really had to do was get in contact with Intel (which oddly enough, they DO know how to work together with) and ask them to add their CPU IDs to the list or offer them an alternative piece of code for processor detection for their compilers which would make it possible to identify AMD processors capable of running those pipelines. Instead, some whiner baby ran to the press and cried about how Intel was being so mean because they didn't bother testing to see if their competitor supported certain instructions.

Add their CPU IDs to what list? Intel's compiler? I am still not sure this is the biggest issue here, as Intel has instruction sets AMD does not, and vice versa (it's just that no one seems to take AMD's seriously from what I can tell, or Intel's are generally more important / built better). AVX in particular is one of the first times I recall where both Intel and AMD architectures will include the same instruction set within the span of a year.

Thanks for the comments folks. I wish I could comment on more, but I am not familiar with all of it.

 

[DarkStarr]

AFAIK unregistered is wrong, (the second one) from what I read about the intel compiler issue was if it saw genuine amd instead of genuine intel it automatically picked the slowest instructions for the AMD cpu but of course then it ran the fastest instructions for Intel CPUs.

 

[Kayden]

Our copy will be here before the weekend; I'm not sure what's taken it so long. Due to the AMD Lynx launch right around the corner, I am not sure how much time I'll have to dedicate to testing it out, but I hope to tackle it soon. BAPCo told us that a whitepaper would be released soon, so I'm looking forward to looking through that.

As long as TG test's it completely, now if it says it is or it isn't I don't care but no matter which way it goes I trust you guys here and I will take you at your professional word. As long as you show the data to back up the claim of course. (c;

 

[Kougar]

Why don't they let the users select the weighting factors? After all I'm the best person to know what I use my computer for.

Then how would you be able to compare scores generated from one PC to another, if the weighting factors were different? I understand your meaning, but nobody designs a program that analyzes a computer but doesn't publish any results. It's a novel idea but it would drastically change how 99% of benchmarking programs function, and the people that use them. Might be a nifty idea for a startup though!

There were a few valid points to AMD's issue with Sysmark. At first I thought it was an issue about AMD vs Intel processor performance... but it's more than that. AMD's core focus is APUs now, their CPU+GPU combo chips.

Sysmark 2012, despite the name, uses IE8 and Firefox 3.6.... but it is IE9 and Firefox 4 that both introduce GPU-acceleration. That is to say IE9 and FF4 are rendered by the GPU instead of the processor. Not to mention both updated versions offer other important browser bits like full HTML5 support too.

I also would like to note something I read: NVIDIA already left the BAPCo consortium as well... and what do they have most in common with AMD?

On the flipside, Sysmark is just a system-wide benchmark. It doesn't focus on any single part of the system like say, PCMark 7. It just gives several large, broad-ranging scores with an almost meaningless name, like productivity or communications. I think some of the stuff is just getting overblown.. the simple matter is the benchmark no longer serves any use for AMD or NVIDIA as both are GPU-centric companies now. They need a program that focuses on parts of the system, not a generalized system-level score, so they left.