Simulated benchmarks have the benefit of providing a reliable and replicable test to compare between-group variability. Like a good scholastic standardized test, it allows a user to compare two systems with the reliability of a lab experiment. Unfortunately, lab experiments rarely simulate real life well enough to be applicable to the real world. Often, the lab experiment is used to isolate one or a few variables to reduce the likelihood that other, usually unmeasured, variables have confounded the relationship between the variables under study.
Gamers do not play games in a lab. In the real world, there is no ideal situation. Computer configurations vary so much between players that within-computer benchmarks provide a much better picture of the experience a gamer will actually perceive. For example, a gamer may want to know what performance his/her computer will put out when the resolution is changed, different video settings are used, more memory is added, or a video card is upgraded. These questions are real-world, requiring a real-world benchmark.
In a recent article, a review of Fraps from Beepa software mentioned that the utility was capable of performing benchmarking. Unlike simulated benchmarks, Fraps allows gamers to test their computer’s hardware under any conditions the gamers desire. For example, suppose that a particularly graphic-laden part of a video game is slowing down a gamer’s system. In other words, the frames-per-second (fps) experienced by the gamer drops below a playable level. The gamer is interested to know under what settings he/she may play the game to experience smooth transition throughout the map. Fraps allows the user to capture a variety of data concerning the frames-per-second, the latency between frames, and other statistics such as the minimum, average, and maximum fps. Capturing this data is as simple as pushing a button and playing the game while Fraps is running the benchmarking portion of the utility in the background.
Later, when the gamer is sure that Fraps has recorded enough data, the information may be analyzed. Analyzing the data is as simple as using a spreadsheet to do a few pertinent calculations.
The next article in this series will look at a sample benchmarking procedure. The basic benchmark will measure the fps experienced during a particularly video-intensive game and the advanced benchmark will show how powerful a gamer-defined benchmark can be in determining the capabilities of a computer.
