loonyblog.

Benchmark BS

There's an interesting editorial over at [H]ard|OCP that talks about the future of benchmarking, and how benchmarks like 3DMark03 are becoming increasingly unreliable. As a hardcore gamer, and more importantly, one who's going to be looking for a new video card in the near future, I found this to be pretty interesting, but what really caught my attention is the editorial's conclusion, in which Kyle says that in the very near future frame rate will no longer be the touchstone for gaming benchmarks.

That last bit is both exciting and a little frightening for hardcore gamers like myself. For years now, bragging rights have hinged on your FPS. If you've never looked with pride at a Quake timedemo number, there's no way to really convey the feeling, but I have no problem saying that every time I upgrade hardware I run Quake and 3DMark benchmarks to see how high I've managed to get up there. Just last week I overclocked my existing GeForce 4200 Ti, and was thrilled to see a minor improvement in my 3DMark scores, which translated noticeably into my Morrowind framerate.

The one real problem with benchmarks of any kind, or at least those seen at the major hardware sites, is that I don't think I've ever owned a system comparable to their benchmarking hardware (at least not at the same time they were still using it anyway). And I'm already in a high-end group, so I can't imagine how few people really own that kind of hardware. That's not necessarily their fault, of course. Hardware makers are notoriously anal about the kind of test beds they let their cards be benchmarked on, and while I don't know for certain what kind of pressure the major hardware sites are under, I imagine neither NVIDIA nor ATI would be in any rush to send a pre-release board to a site that intended to do benchmarks on an "average" user's PC.

Now, I don't expect to actually see any "real world" benchmarks from major hardware sites in the future, but it would certainly be nice for a change. Even though I'm likely to buy a really high-end video card sometime soon (certainly before Half-Life 2 comes out in September) there's no way I'm going to get the kind of framerates seen in these reviews. It might be nice to see something that would let me know how much of an improvement I'll see on my decidedly mid-range system.

Posted by jason at July 1, 2003 05:36 PM | TrackBack | Read more: