Latest Forum Posts

Latest News Posts
Coming Soon!
Social
Go Back   Techgage.com > Archives > Reviews and Articles

Reviews and Articles Discussion for Techgage content is located here. Only staff can create topics, but everyone is welcome to post.

Reply
 
Thread Tools
Old 06-21-2011, 03:53 PM   #1
Rob Williams
Editor-in-Chief
 
Rob Williams's Avatar
 
Join Date: Jan 2005
Location: Atlantic Canada
Posts: 13,351
Default AMD Rejects BAPCo's SYSmark 2012 - Should We?

In a rare move, AMD has publicly ousted a leading industry benchmark, BAPCo's SYSmark 2012, released mere weeks ago. The company believes both that the benchmark is unfair in its weighting of scores, and also that it's irrelevant for the average consumer. We delve deep into these claims, and offer up our own opinions.

Read through our in-depth look into the matter of AMD's rejecting SYSmark 2012, and once done, discuss it here!
__________________
Intel Core i7-3960X, GIGABYTE G1.Assassin 2, Kingston 16GB DDR3-2133, NVIDIA GeForce GTX 770 2GB
Kingston HyperX 3K 240GB SSD (OS, Apps), WD VR 1TB (Games), Corsair 1000HX, Corsair H70 Cooler
Corsair 800D, Dell 2408WFP 24", ASUS Xonar Essence STX, Gentoo (KDE 4.11. 3.12 Kernel)

"Take care to get what you like, or you will be forced to like what you get!" - H.P. Baxxter
<Toad772> I don't always drink alcohol, but when I do, I take it too far.


Rob Williams is offline   Reply With Quote
Old 06-21-2011, 06:55 PM   #2
Unregistered
Guest Poster
 
Posts: n/a
Default bapco Not for profit?

It may not make a profit but its entire reason for being is to help the member companies make more profit.

Why don't they let the users select the weighting factors? After all I'm the best person to know what I use my computer for.
  Reply With Quote
Old 06-21-2011, 07:35 PM   #3
Rob Williams
Editor-in-Chief
 
Rob Williams's Avatar
 
Join Date: Jan 2005
Location: Atlantic Canada
Posts: 13,351
Default

Quote:
Originally Posted by Unregistered View Post
Why don't they let the users select the weighting factors? After all I'm the best person to know what I use my computer for.
Or, it could at least base the testing off of current and on-going trends. There are numerous reports out there that show what people are doing with their PCs, so it's not an unknown. But aside from that, SM2012 runs some tests with outdated software, so there's more than just one factor plaguing it.

AMD wants to see a new consortium form to right the wrongs of BAPCo, but I think it's easier said than done. Even Intel itself has created a couple of in-house test suites that largely went unused and were forgotten about.

We've even considered building a similar suite using all of our real-world benchmarks, and then build a score of some sort out of the aggregate. That in itself was challenging enough that we had to give up due to it being a huge time-sink.
__________________
Intel Core i7-3960X, GIGABYTE G1.Assassin 2, Kingston 16GB DDR3-2133, NVIDIA GeForce GTX 770 2GB
Kingston HyperX 3K 240GB SSD (OS, Apps), WD VR 1TB (Games), Corsair 1000HX, Corsair H70 Cooler
Corsair 800D, Dell 2408WFP 24", ASUS Xonar Essence STX, Gentoo (KDE 4.11. 3.12 Kernel)

"Take care to get what you like, or you will be forced to like what you get!" - H.P. Baxxter
<Toad772> I don't always drink alcohol, but when I do, I take it too far.


Rob Williams is offline   Reply With Quote
Old 06-21-2011, 09:09 PM   #4
Unregistered
Guest Poster
 
Posts: n/a
Default history of skewed benchmarks

Somewhere in a closet I have a publication Motorola put out about 80286/68020 comparisons that it said Intel was fudging; I wish I could dig it out.

And of course we all remember the Intel compiler that checks for the availability of certain instructions in a way that makes sure they won't be used for AMD processors that in fact support them, even though they set the appropriate capability bits to indicate their presence in a proper test.

It's getting close to July 4th, so the phrase "a long train of abuses and usurpations" is perhaps appropriate.
  Reply With Quote
Old 06-21-2011, 09:53 PM   #5
Kayden
Techgage Staff
 
Kayden's Avatar
 
Join Date: Sep 2010
Location: CA
Posts: 710
Default

Quote:
Given the application list above, we can't jump to conclusions and state that SM2012 does not make use of the GPU, because some of the applications listed can. We won't be able to prove it to ourselves until we receive our copy and can monitor the GPU's usage.
Until you can test it in house NO. There is a difference between Faith and Blind Faith. With out a specific reason and real data to back it up with a 3rd party to back it up, there is no reason for me to take this on Blind Faith from AMD. They are a good company for what they do but they do not deal in the truth and I do not believe any one company with out proof.
__________________
Koolance PC4-1036BK Liquid Cooled Aluminum Full Tower Case (Upgraded to displace 1320watt of heat) - Koolance 360 ver 1.2 CPU Water Block - PrimoChill ICE Non-Conductive Fluid - Asus P6X58D Premium - Intel I7 950 - Evga GTX580 FTW Hydro Copper 2 x2 in SLI - Tt SMART M1200W PSU - Kingston HyperX 12GB 1600 XMP - Klipsch 5.1 THX Cert Speakers - Sata II OCZ Vertex 4 128GB, Kingston 120GB SV300, WD 2TB, & 320GB - Seagate 1TB - Cyborg RAT 7 MMO Mouse - Logitech G19 - Logitech G13 Gameboard - 1-over-3 Viewsonic's VA2448M-LED 24-Inch for Nvidia Surround - Ergotech Triple Horizontal LCD Monitor Arm Desk Stand


Kayden is offline   Reply With Quote
Old 06-21-2011, 10:47 PM   #6
Kayden
Techgage Staff
 
Kayden's Avatar
 
Join Date: Sep 2010
Location: CA
Posts: 710
Default

Found this TG article on Slashdot btw.

http://tech.slashdot.org/story/11/06...mark-Benchmark
__________________
Koolance PC4-1036BK Liquid Cooled Aluminum Full Tower Case (Upgraded to displace 1320watt of heat) - Koolance 360 ver 1.2 CPU Water Block - PrimoChill ICE Non-Conductive Fluid - Asus P6X58D Premium - Intel I7 950 - Evga GTX580 FTW Hydro Copper 2 x2 in SLI - Tt SMART M1200W PSU - Kingston HyperX 12GB 1600 XMP - Klipsch 5.1 THX Cert Speakers - Sata II OCZ Vertex 4 128GB, Kingston 120GB SV300, WD 2TB, & 320GB - Seagate 1TB - Cyborg RAT 7 MMO Mouse - Logitech G19 - Logitech G13 Gameboard - 1-over-3 Viewsonic's VA2448M-LED 24-Inch for Nvidia Surround - Ergotech Triple Horizontal LCD Monitor Arm Desk Stand


Kayden is offline   Reply With Quote
Old 06-22-2011, 03:55 AM   #7
Unregistered
Guest Poster
 
Posts: n/a
Default

Quote:
Originally Posted by Rob Williams View Post
In a rare move, AMD has publicly ousted a leading industry benchmark, BAPCo's SYSmark 2012, released mere weeks ago. The company believes both that the benchmark is unfair in its weighting of scores, and also that it's irrelevant for the average consumer. We delve deep into these claims, and offer up our own opinions.

Read through our in-depth look into the matter of AMD's rejecting SYSmark 2012, and once done, discuss it here!
First off, thanks for letting us reply without registering.

I'm going to side with AMD here to a point. My biggest argument is what do they suggest? I do think since this is a test for businesses that this test should only be published with a required disclaimer saying it's for business applications and the scores do not reflect what average consumers would experience. Because AMD is right, consumers should not look at these results and use it to compare two CPUs to decide which is better. I have not used FineReader OCR since 2001 (FYI it worked great back then!) and I would not care how well a CPU does OCR.

As a consumer I am concerned about internet functionality. Different versions of flash, silverlight and JavaScript should be tested. Google Chrome, Opera and different versions of IE should be on the list. iTunes is missing too. Transcoding is probably included in After effects, if not it should be on this list too
  Reply With Quote
Old 06-22-2011, 04:43 AM   #8
Unregistered
Guest Poster
 
Posts: n/a
Thumbs down Rubbish

Quote:
Originally Posted by Unregistered View Post
Somewhere in a closet I have a publication Motorola put out about 80286/68020 comparisons that it said Intel was fudging; I wish I could dig it out.
Motorola and Intel were testing apples vs oranges for years. A lack of compilers compatible across both platforms made this entirely useless... never once did you see a test run for example on a System V based 68000 and a System V based x86. Fact of the matter is, in the very rare cases where you did, you'd see multi processor 68000 based workstation beasts up against IBM ATs. Clock for clock, the 68020 and the 80286 were so damn close in performance that it made little difference. We were talking about 1-2% in favor of one or the other. Intel fudged these tests and Motorola fudged these tests... sales people who didn't have a clue paid for tests to be run on both sides which would show them in favor.

Quote:
And of course we all remember the Intel compiler that checks for the availability of certain instructions in a way that makes sure they won't be used for AMD processors that in fact support them, even though they set the appropriate capability bits to indicate their presence in a proper test.
The Intel compiler checked for processor capabilities based on the processor model and generation. This is how many programs handle this type of testing. You don't check for individual instructions... if you did, you'd have to recompile for every combination etc... Instead, you optimize one or two different ways and if all the instructions for the faster path is supported, then you execute that one. AMD was being a whiner baby then as they're being a whiner baby now. All they really had to do was get in contact with Intel (which oddly enough, they DO know how to work together with) and ask them to add their CPU IDs to the list or offer them an alternative piece of code for processor detection for their compilers which would make it possible to identify AMD processors capable of running those pipelines. Instead, some whiner baby ran to the press and cried about how Intel was being so mean because they didn't bother testing to see if their competitor supported certain instructions.

Quote:
It's getting close to July 4th, so the phrase "a long train of abuses and usurpations" is perhaps appropriate.
One of the classic blunders from hell. Claiming George III was our enemy, an abuser and a usurper. Oddly, after writing our declaration of independence, we learned it was actually a greedy parliament and George III was the closest thing we had to a friend as he fought in our favor against the actual abusers. He just failed to get very far with it. So... maybe again you're singling out the wrong enemy because some idiot with a journal told you he didn't like someone? Oh... sorry, you haven't read this far in the posting, you stopped at the headline and you'll go from there like with everything else won't you?
  Reply With Quote
Old 06-22-2011, 12:46 PM   #9
Rob Williams
Editor-in-Chief
 
Rob Williams's Avatar
 
Join Date: Jan 2005
Location: Atlantic Canada
Posts: 13,351
Default

Quote:
Originally Posted by Unregistered View Post
Somewhere in a closet I have a publication Motorola put out about 80286/68020 comparisons that it said Intel was fudging; I wish I could dig it out.
You have to give credit to Intel for one thing... few other semiconductor companies have been accused of cheating for nearly 30 years ;-)

Quote:
Originally Posted by Unregistered View Post
And of course we all remember the Intel compiler that checks for the availability of certain instructions in a way that makes sure they won't be used for AMD processors that in fact support them, even though they set the appropriate capability bits to indicate their presence in a proper test.
That's one reason I've come to appreciate SPEC's CPU benchmark, since it's architecture-agnostic. Nowadays, though, Intel has the biggest advantage by being the first to market with certain instruction sets, while it seems that those that AMD introduce don't get used in commercial applications. SSE4 is a big example of this, where huge gains could be seen with video encoders that could take advantage of it.

Quote:
Originally Posted by Kayden View Post
Until you can test it in house NO. There is a difference between Faith and Blind Faith. With out a specific reason and real data to back it up with a 3rd party to back it up, there is no reason for me to take this on Blind Faith from AMD. They are a good company for what they do but they do not deal in the truth and I do not believe any one company with out proof.
Our copy will be here before the weekend; I'm not sure what's taken it so long. Due to the AMD Lynx launch right around the corner, I am not sure how much time I'll have to dedicate to testing it out, but I hope to tackle it soon. BAPCo told us that a whitepaper would be released soon, so I'm looking forward to looking through that.

Quote:
Originally Posted by Unregistered View Post
First off, thanks for letting us reply without registering.
No problem; thanks for taking the time to post!

Quote:
Originally Posted by Unregistered View Post
I'm going to side with AMD here to a point. My biggest argument is what do they suggest? I do think since this is a test for businesses that this test should only be published with a required disclaimer saying it's for business applications and the scores do not reflect what average consumers would experience. Because AMD is right, consumers should not look at these results and use it to compare two CPUs to decide which is better. I have not used FineReader OCR since 2001 (FYI it worked great back then!) and I would not care how well a CPU does OCR.
I couldn't agree more on the latter point. While it does have its niche market, most regular consumers don't even know what OCR is, and if they did, they likely wouldn't have a need to use it. So to include it in a benchmarking suite that aims to give consumers a good idea of what their PC is capable of, it's a bit strange. It'd be like Toyota giving us stats on how well its Corolla handles driving on molten rock... it's just not that relevant for most people.

Quote:
Originally Posted by Unregistered View Post
As a consumer I am concerned about internet functionality. Different versions of flash, silverlight and JavaScript should be tested. Google Chrome, Opera and different versions of IE should be on the list. iTunes is missing too. Transcoding is probably included in After effects, if not it should be on this list too
To be honest, I think some of the things mentioned here are a little more specific than this benchmark intends to be. If SYSmark had the goal of dividing up the scores at the end further to explain which browser performed the best, that might be a little different, but this is a benchmark designed to last for a couple of years. That's where things get tough, because SM2012 releases with IE8, when IE9 is available... and when this benchmark is retired, we might well see IE11. At the same time, the IE choice in general is a little questionable.

I agree on the general idea though. Lots of people use things like iTunes, so that test (or one similar) should be included. The same goes for transcoding and the like, though SM2012 might touch up on that a little. With what you've mentioned, I think PCMark 7 is the better benchmark for that kind of consumer-level use. It tests out some Web technologies in a custom browser, and also tackles video conversion, image manipulation, et cetera.

That all said, outside of suite usage we use individual benchmarks in our own testing for all the things mentioned above. We don't use iTunes, though we might implement dBpoweramp for the sake of encoding later (unless we can see that iTunes proves in itself to be a reliable, repeatable benchmark).

Quote:
Originally Posted by Unregistered View Post
All they really had to do was get in contact with Intel (which oddly enough, they DO know how to work together with) and ask them to add their CPU IDs to the list or offer them an alternative piece of code for processor detection for their compilers which would make it possible to identify AMD processors capable of running those pipelines. Instead, some whiner baby ran to the press and cried about how Intel was being so mean because they didn't bother testing to see if their competitor supported certain instructions.
Add their CPU IDs to what list? Intel's compiler? I am still not sure this is the biggest issue here, as Intel has instruction sets AMD does not, and vice versa (it's just that no one seems to take AMD's seriously from what I can tell, or Intel's are generally more important / built better). AVX in particular is one of the first times I recall where both Intel and AMD architectures will include the same instruction set within the span of a year.

Thanks for the comments folks. I wish I could comment on more, but I am not familiar with all of it.
__________________
Intel Core i7-3960X, GIGABYTE G1.Assassin 2, Kingston 16GB DDR3-2133, NVIDIA GeForce GTX 770 2GB
Kingston HyperX 3K 240GB SSD (OS, Apps), WD VR 1TB (Games), Corsair 1000HX, Corsair H70 Cooler
Corsair 800D, Dell 2408WFP 24", ASUS Xonar Essence STX, Gentoo (KDE 4.11. 3.12 Kernel)

"Take care to get what you like, or you will be forced to like what you get!" - H.P. Baxxter
<Toad772> I don't always drink alcohol, but when I do, I take it too far.


Rob Williams is offline   Reply With Quote
Old 06-22-2011, 04:34 PM   #10
DarkStarr
Tech Monkey
 
DarkStarr's Avatar
 
Join Date: Apr 2010
Posts: 634
Default

AFAIK unregistered is wrong, (the second one) from what I read about the intel compiler issue was if it saw genuine amd instead of genuine intel it automatically picked the slowest instructions for the AMD cpu but of course then it ran the fastest instructions for Intel CPUs.
__________________
Intel Core i7 2700k (4.8 @ 100x48) Watercooled - 16GB Crucial Ballistix @ 1600 Mhz 9.9.9.24 2T
8GB
Corsair Vengeance @ 1600 Mhz 9.9.9.24 2T - Asus Sabertooth P67
- Asus Radeon 7970 Ref. (Non Ghz)
Heatkiller 79xx Ni-Bl (Soon) @ 1050/1500
- 64Gb Crucial M4 SSD - 3x Hitachi 1Tb - Corsair TX950W
Azza Genesis 9000B - 2x Samsung SyncMaster S27A550H - Vizio 32" LCD

DarkStarr is offline   Reply With Quote
Old 06-22-2011, 08:55 PM   #11
Kayden
Techgage Staff
 
Kayden's Avatar
 
Join Date: Sep 2010
Location: CA
Posts: 710
Default

Quote:
Originally Posted by Rob Williams View Post
Our copy will be here before the weekend; I'm not sure what's taken it so long. Due to the AMD Lynx launch right around the corner, I am not sure how much time I'll have to dedicate to testing it out, but I hope to tackle it soon. BAPCo told us that a whitepaper would be released soon, so I'm looking forward to looking through that.
As long as TG test's it completely, now if it says it is or it isn't I don't care but no matter which way it goes I trust you guys here and I will take you at your professional word. As long as you show the data to back up the claim of course. (c;
__________________
Koolance PC4-1036BK Liquid Cooled Aluminum Full Tower Case (Upgraded to displace 1320watt of heat) - Koolance 360 ver 1.2 CPU Water Block - PrimoChill ICE Non-Conductive Fluid - Asus P6X58D Premium - Intel I7 950 - Evga GTX580 FTW Hydro Copper 2 x2 in SLI - Tt SMART M1200W PSU - Kingston HyperX 12GB 1600 XMP - Klipsch 5.1 THX Cert Speakers - Sata II OCZ Vertex 4 128GB, Kingston 120GB SV300, WD 2TB, & 320GB - Seagate 1TB - Cyborg RAT 7 MMO Mouse - Logitech G19 - Logitech G13 Gameboard - 1-over-3 Viewsonic's VA2448M-LED 24-Inch for Nvidia Surround - Ergotech Triple Horizontal LCD Monitor Arm Desk Stand


Kayden is offline   Reply With Quote
Old 06-23-2011, 09:44 PM   #12
Kougar
Techgage Staff
 
Kougar's Avatar
 
Join Date: Mar 2008
Location: Texas
Posts: 2,653
Default

Quote:
Originally Posted by Unregistered
Why don't they let the users select the weighting factors? After all I'm the best person to know what I use my computer for.
Then how would you be able to compare scores generated from one PC to another, if the weighting factors were different? I understand your meaning, but nobody designs a program that analyzes a computer but doesn't publish any results. It's a novel idea but it would drastically change how 99% of benchmarking programs function, and the people that use them. Might be a nifty idea for a startup though!

There were a few valid points to AMD's issue with Sysmark. At first I thought it was an issue about AMD vs Intel processor performance... but it's more than that. AMD's core focus is APUs now, their CPU+GPU combo chips.

Sysmark 2012, despite the name, uses IE8 and Firefox 3.6.... but it is IE9 and Firefox 4 that both introduce GPU-acceleration. That is to say IE9 and FF4 are rendered by the GPU instead of the processor. Not to mention both updated versions offer other important browser bits like full HTML5 support too.

I also would like to note something I read: NVIDIA already left the BAPCo consortium as well... and what do they have most in common with AMD?

On the flipside, Sysmark is just a system-wide benchmark. It doesn't focus on any single part of the system like say, PCMark 7. It just gives several large, broad-ranging scores with an almost meaningless name, like productivity or communications. I think some of the stuff is just getting overblown.. the simple matter is the benchmark no longer serves any use for AMD or NVIDIA as both are GPU-centric companies now. They need a program that focuses on parts of the system, not a generalized system-level score, so they left.
__________________
Core i7 4770k 4.2Ghz
Gigabyte Z87X-UD5H
Crucial Ballistix Sport LP 1600MHz 32GB
EVGA GTX 480 HydroCopper FTW
ASUS Xonar DX
Corsair Neutron GTX 240GB | Windows 7 64-bit
Apogee XT + MCP655 & Thermochill Triple 140mm Radiator
Corsair AX1200 PSU | Cooler Master HAF-X

Kougar is offline   Reply With Quote
Reply

Tags
None

Thread Tools

Posting Rules
You may not post new threads
You may post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is On

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Top music of 2012? Rob Williams Off Topic 0 12-30-2012 04:37 PM
Euro 2012 Rob Williams Off Topic 2 06-12-2012 10:53 AM
Happy 2012! Rob Williams Off Topic 9 01-01-2012 10:22 PM
CES 2012 Will Be Microsoft's Last Rob Williams General Technology 1 12-22-2011 06:48 PM
Yahoo! Rejects Microsoft, Contemplates Merger With... AOL? Rob Williams Off Topic 0 02-11-2008 12:42 PM


All times are GMT -4. The time now is 09:32 AM.