We’re approaching the end of summer here in the US, and that means that the semiannual push for new CPUs and GPUs is imminent. This year, it’s looking like we’ll have some significant new hardware on tap: new capabilities, new price points, better performance or all three. That can only mean one thing:
Benchmarking. Lots of Benchmarking
The good news is that I’ll be covering some of these products for other publications, which generally means using the suite of performance tests they’ve developed over time. It also means I get to run performance tests I want to run.
When I was writing for Ziff-Davis, the folks at PCMag labs and I tacitly agreed to run a common set of benchmarks. That way, they could always refer to my tests, and easily compare them to their system benchmarks. It was useful on my end, too, because I could easily point to their system reviews and use them in my own articles.
Now, though, I get to do my own thing. For example, I really don’t care what Sysmark 2007 Preview tells me. It’s not useful to me for the kinds of things I do. I care about digital photography and video encoding performance. I care about games performance. I’m not too concerned about 3D Studio performance, though I may go ahead and run a few tests, like the Cinebench 10 benchmark, for my own use.
Photography and Video Encoding
But I will be expanding the photographic tests. The PCMag Photoshop filter benchmark is okay as far as it goes, but it seems pretty random in how it goes about what it does. I’ll try to build a similar test using filters that I’ve used, including noise reduction filters. I’m expanding the noise reduction benchmark, too, using both Noiseware and Noise Ninja.
Another test will be raw image conversion, which I’ll replicate using Adobe Photoshop CS4 RAW, but also Bibble Pro, a highly regarded photographic workflow tool. I may also include Adobe Lightroom, but I’m still mulling how to best use that particular tool as a performance test.
While we’re talking about Adobe, I’m hoping to include a Premiere Pro CS4 render to a DVD image. I don’t want to actually burn a DVD – including optical drive performance would add too many variables.
I’ll also take a look at some transcoding only benchmarks, like DiVX and MainConcept’s encoders.
While single-threaded testing might be important for all the reason you mentioned, I have a hard time imagining a true single-threaded usage in everyday life. No matter what else I might be doing, there is always something else. Ventrilo during the gaming session, or Pigdin while watching a movie. When giving the computer a hours-long task like transcribing video (and walking away) you are virtually guaranteed to have the system come along and do something under the covers. Defragging the disk, a home server backup, the iPod deciding that NOW is a good type to sync.
Right at this moment I have 88 processes running on my Windows 7 box. I have complete control over maybe 20 of them. Is a single-thread test even meaningful?
I’ll bet if you looked at your performance meter in task manager, you’ll see that all the cores of your CPU is about 98% unused. Those tiny OS tasks take up very little CPU usage, and are often idel.
When you launch something CPU intensive, that also happens to be single-threaded, Nehalem will run that task in one of the cores at a higher frequency than the other cores. That’s when you’ll see at least one of the CPU meters peg.
Finally, Windows 7 is smarter than Vista and much smarter than Windows XP at assigning and maintaining a single, intensive task on one of the cores, rather than doing a lot of core swapping, which slows things down and wastes resources.
I get all that. Just making the point that single-threaded performance is getting less meaningful. Since it is hardly possible to buy a single-threaded processor, nor to configure a machine to not have multiple uncontrollable processes it seems an academic discussion.
We’ll care that a task takes 60 minutes on the new box and 73 minutes on the old box. But until the software makers re-write a bunch of their core code, upgrading from a dual-core 9300 (for example) to an effective 8-core i920 possible just means that I will have a lot more idle time. Yes, identical tasks will be somewhat faster, but not 4 times as fast.
Ah, the good old days when upgrading from 25Mhz to 90Mhz actually meant almost 3x in performance boost.
“I care about digital photography and video encoding performance. I care about games performance.”
Finally! No more irrelevant stuff. I wanna see things like… encoding time in Vegas and FPS in ArmA2. Something I can do on my OWN machine to compare.
I can’t wait to see SSD Performance. This is definitely the way of the future and will be very valuable.
I do audio engineering and I am starting to dabble in video, and my hard ware is everything to me.
I like your approach. An approach shared by a UK magazine called Custompc, who have designed their own (I think) similar battery of tests, which you can download and use. Take a look at http://bit.ly/oolIs for the explanation and download, you may find it useful.
I used to pay attention to benchmarks.
Nowadays all I need is a rough number. 10% faster or 15% faster. And that’s it.
Your article could really just be a number. Take after videogame reviews and just assign the hardware a speedscore A 10 means it’s 10% faster. 15 means 15% faster. etc.
This would leads to lots of debate because assigning one overall number is judgement call. And imagine if all the various hardware sites started doing this. I can see the flames and discussions about why this or that site gave this or that piece of hardware such a high or low (speed)score.
Anyway benchmarks could use a little bit of fun and a judgement call like that. They’ve become stale.
Faster hardware is overkill for many tasks.
Pcgaming isn’t what it once was. It is not like new pcgaming hardware is even needed. DX11? It’s as if a million people yawned simultaneously.
Sure i7 gave video encoding a nice boost. Still takes too long and still can’t do video encoding very well at the same time you’re doing other tasks at least not that well on my 2.66ghz intel c2d cpu.
Software really has to come full front and take advantage of this hardware. Intel should just bury themselves in the lab for 5 years before announcing another cpu. By that time they’ll have a truly new and useful cpu for everyone.
Anyway applaud the effort to try and make benchmarks more relevant. HOpefully you’re not swimming upstream.
The other thing I’ll be doing will be percentile comparisons to a reference system. Right now, it’s a very high end system (Core i7 975). But I plan on keeping that around, and we’ll see systems that exceed that over time.
Sorry to post this here — delete after reading:
Loyd, I tried to send you an email this morning to the address listed on your contact page (loydcase@improbableinsights.com). A few mintues ago I got a ‘failed permanently’ message kicked back to me:
Hope that helps.
Thanks, Mark. I’ve fixed the contact page. The email is actually loyd (at) improbableinsights (DOT) com.