Frame Rating Dissected: Full Details on Capture-based Graphics Performance Testing
Battlefield 3 – HD 7970 versus GTX 680
Our first graph shows the results of the average frame rates per second of the single HD 7970, HD 7970 CrossFire, a single GTX 680 and GTX 680s in SLI. You can see that the single GTX 680 is running just a bit slower than the performance of the HD 7970 but when we add in that second card, the CrossFire configuration is actually noticeably faster! But what does the Frame Rating system actually see?
Here we can see how all of the raw frame times are shown on the screen via our capture system and the FCAT extraction tools. The frame time variance of the single card configurations, both HD 7970 and GTX 680 are pretty minimal, as indicated by a tighter, thinner line across the time window. When we add in the second card for SLI on the GTX 680s you can see that there is a less consistent frame time with a wider, more dense blue line. The orange line though is a much bigger story with a frame times cycling between near 0ms (the runts we see) and 15ms. Indicating not only some potential animation stuttering but also that the small frames are so insignificant as to be useless in our observed frame rates.
Remember that the Observed FPS takes out runts and drops and is the result and benefit of measuring performance with our new Frame Rating system. Clearly the advantages that CrossFire appeared to have over SLI in the first result nearly completely negated and in many cases the observed performance of a two card HD 7970 configuration is no better than that of a single card.
The same cannot be said for NVIDIA’s SLI though – the frames are presented to the gamer in a consistent pattern that indicates good scaling and that looks nearly identical to that of the first FRAPS-based graph.
Our minimum frame rate percentile graph shows a similar story to our observed frame rate and shows the frame rate of the HD 7970s in CrossFire to be very similar to that of a single HD 7970; in fact the two frame rates / frame times get CLOSER as we progress down the percentile curve. The GXT 680s in SLI are the only configuration to see any appreciable difference.
Finally, let’s look at our custom algorithm stutter graphic. Without a doubt the CrossFire configuration is the worst offender here and will have the most potential for animation stutter of any other GPUs compared here. Keeping mind that this graph has very little to do with absolute frame time (thus absolute frame rates), both the HD 7970 and GTX 680 have nearly identical, low variance levels.
Let’s dive into the next resolution, 2560x1440.
The single card comparisons again shine a positive light on the Radeon HD 7970 as it is able to maintain performance higher than the GTX 680 throughout or test runs of Battlefield 3. This information is also reporting a similar result for CrossFire though…
A quick glance at this graph tells you all you need to know – the CrossFire runt issue remains a problem at 2560x1440 in addition to 1920x1080. Without a doubt the average frame rates you see above are NOT indicative of real world performance for AMD’s multi-GPU solutions. Interestingly, the width of the blue line, for the GXT 680 SLI setup, are not much different than those of the HD 7970 and GTX 680 single cards. It seems that with less CPU bottleneck on the system the SLI configuration can actually improve frame time consistency compared to lower resolutions.
Based on the frame times above it should not surprise you to once again see our Observed FPS for CrossFire is very different than the FRAPS measured average frame rate. Both of the single GPU configurations as well as the GTX 680s in SLI show nearly identical results.
This is a RUN plot graph for the HD 7970 CrossFire configuration at 2560x1440. We aren't going to be showing these for most of the cards and games, only in a select few instances where we can illustrate a point. This time, the frame times plot above and this RUN graph can show the two sides of the same story. Notice that the FRAPS frame rate is seen at the top in the black line with the observed frame rate represented by the blue line. The seperation between them is eithe runts (orange) or dropped frames completely (red). Interestingly, there are two points in our 60 second run at the 43 second and 47 second mark where it looks like the CrossFire setup is performing correctly - where the FRAPS and observed frame rate lines meetup, though they quickly diverge again.
For comparison, here is the RUN plot from the GTX 680s in SLI. The data here shows us that there are no runt frames and no dropped frames from the GTX 680s. That is a stark comparison to what the HD 7970s show above.
Even though the GTX 680 is at the bottom of the pack in our percentile chart of minimum frame rates, the HD 7970 and HD 7970 CrossFire again show the lack of scalability of the CrossFire configuration. And it should give GeForce owners some pride that their single GTX 680 barely slower than a pair of Radeon HD 7970s in real-world performance.
Our ISU ratings show the HD 7970 CrossFire is again the most variant of configurations and we do see improvement with the GTX 680s in SLI – now matching the level of even a single Radeon HD 7970.
As I mentioned before, there are some cases where the 5760x1080 results for AMD CrossFire + Eyefinity were so bad, with so many dropped frames, that the Perl code in FCAT couldn’t produce a solid result. With Battlefield 3, this is one of those cases so you will not get any HD 7970s in CrossFire in our graphs here. What I can tell you though is that our video footage of the 57x10 EF+CF testing did show a dropped frame on every other color, telling me that the work of one of the GPUs was 100% thrown away and never shown to the gamer.
Without the HD 7970 CF config in our plots we see that while the GTX 680s in SLI is scaling quite well (going from somewhere in the 30s range to somewhere in the 50s) the HD 7970 single card remains the better single GPU option.
Frame variance looks pretty bad for the GTX 680s in SLI at 5760x1080 jumping from under 10ms to over 40ms in some cases. Meanwhile, both single cards have pretty tight data plots some noticeable spikes on the single GTX 680.
The observed FPS is identical to that of the FRAPS FPS since we did not see any runts or drops from our data.
Our percentile graph of minimum frame times shows us something interesting. Looking at the 50th percentile is CLOSE to looking at the average frame rate over the entire run of the test and the GTX 680s scale from 32 FPS to 52 FPS with the HD 7970 coming in around 38 FPS. But as we scale down the line to higher percentiles the minimum FPS of the GXT 680 SLI setup is coming down more steeply than the single cards and at the 95% mark are much closer together with the HD 7970 actually splitting the difference.
You can ignore the HD 7970 CF line here along the bottom – residual data and graph cleanup is needed. What is important though is that the frame time variance of SLI is clearly seen here and the “spikes” we saw in the frame time graph are affecting the overall variance poorly for NVIDIA’s GTX 680. While these results are interesting, without the plots from the HD 7970 CrossFire, I would leave the Battlefield 3 5760x1080 results open ended.
What is the total takeaway from our first set of results from Frame Rating, on Battlefield 3? First, if we look at only single GPU solutions, the GTX 680 falls behind the Radeon HD 7970 in average frame rate while exhibiting no more frame time variance or stutter. For SLI and CrossFire though, there isn’t even a debate and the real-world value of adding a second HD 7970 to your system is near zero. That’s a tough pill to swallow for any gamers out there with two cards already in their system and I expect we will get some hate-fueled email and comments on the subject but the data is true and matches our game play testing experiences.