Feedback

Frame Rating Part 3: First Results from the New GPU Performance Tools

Author:
Manufacturer: PC Perspective

In case you missed it...

UPDATE: We have now published full details on our Frame Rating capture and analysis system as well as an entire host of benchmark results.  Please check it out!!

In one of the last pages of our recent NVIDIA GeForce GTX TITAN graphics card review we included an update to our Frame Rating graphics performance metric that details the testing method in more detail and showed results for the first time.  Because it was buried so far into the article, I thought it was worth posting this information here as a separate article to solict feedback from readers and help guide the discussion forward without getting lost in the TITAN shuffle.  If you already read that page of our TITAN review, nothing new is included below. 

I am still planning a full article based on these results sooner rather than later; for now, please leave me your thoughts, comments, ideas and criticisms in the comments below!


Why are you not testing CrossFire??

If you haven't been following our sequence of stories that investigates a completely new testing methodology we are calling "frame rating", then you are really missing out.  (Part 1 is here, part 2 is here.)  The basic premise of Frame Rating is that the performance metrics that the industry is gathering using FRAPS are inaccurate in many cases and do not properly reflect the real-world gaming experience the user has.

Because of that, we are working on another method that uses high-end dual-link DVI capture equipment to directly record the raw output from the graphics card with an overlay technology that allows us to measure frame rates as they are presented on the screen, not as they are presented to the FRAPS software sub-system.  With these tools we can measure average frame rates, frame times and stutter, all in a way that reflects exactly what the viewer sees from the game.

We aren't ready to show our full sets of results yet (soon!) but the problems lie in that AMD's CrossFire technology shows severe performance degradations when viewed under the Frame Rating microscope that do not show up nearly as dramatically under FRAPS.  As such, I decided that it was simply irresponsible of me to present data to readers that I would then immediately refute on the final pages of this review (Editor: referencing the GTX TITAN article linked above.) - it would be a waste of time for the reader and people that skip only to the performance graphs wouldn't know our theory on why the results displayed were invalid.

Many other sites will use FRAPS, will use CrossFire, and there is nothing wrong with that at all.  They are simply presenting data that they believe to be true based on the tools at their disposal.  More data is always better. 

Here are these results and our discussion.  I decided to use the most popular game out today, Battlefield 3 and please keep in mind this is NOT the worst case scenario for AMD CrossFire in any way.  I tested the Radeon HD 7970 GHz Edition in single and CrossFire configurations as well as the GeForce GTX 680 and SLI.  To gather results I used two processes:

  1. Run FRAPS while running through a repeatable section and record frame rates and frame times for 60 seconds
  2. Run our Frame Rating capture system with a special overlay that allows us to measure frame rates and frame times with post processing.

Here is an example of what the overlay looks like in Battlefield 3.

View Full Size

Frame Rating capture on GeForce GTX 680s in SLI - Click to Enlarge

The column on the left is actually the visuals of an overlay that is applied to each and every frame of the game early in the rendering process.  A solid color is added to the PRESENT call (more details to come later) for each individual frame.  As you know, when you are playing a game, multiple frames will make it on any single 60 Hz cycle of your monitor and because of that you get a succession of colors on the left hand side.

By measuring the pixel height of those colored columns, and knowing the order in which they should appear beforehand, we can gather the same data that FRAPS does but our results are seen AFTER any driver optimizations and DX changes the game might make.

View Full Size

Frame Rating capture on Radeon HD 7970 CrossFire - Click to Enlarge

Here you see a very similar screenshot running on CrossFire.  Notice the thin silver band between the maroon and purple?  That is a complete frame according to FRAPS and most reviews.  Not to us - we think that frame rendered is almost useless. 

Continue reading our 3rd part in a series of Frame Rating and to see our first performance results!!

 

View Full Size

Data gathered from FRAPS

Here is a typical frame rate over time graph as generated by FRAPS.  Looks good right?  CrossFire and SLI are competitive with the advantage to the HD 7970s.

View Full Size

Data gathered from Frame Rating Capture

This is the same graph with data gathered from our method that omits RUNT frames that only represent pixels under a certain threshold (to be discussed later).  Removing the tiny slivers gives us a "perceived frame rate" that differs quite a bit - CrossFire doesn't look faster than a single card.

 

View Full Size

Data gathered from FRAPS - Click to Enlarge

This is the raw frame times as captured by FRAPS - again we are looking for a narrow band of frametimes to represent a smooth experience.  Both single cards do pretty well, but SLI sees a bit more variance and CrossFire sees a bigger one.  Quite a bit bigger.

View Full Size

Data gathered from Frame Rating Capture - Click to Enlarge

Here is the same data gathered by our new capture system - the CrossFire configuration looks MUCH worse with many frames hitting near 0ms of screen time.  That would be great if they were ALL like that but unfortunately they also scale up to 20ms and higher quite often.  Also notice NVIDIA's is actually MORE uniform indicating that there is some kind of smoothing going on after the frame leaves the game engine's hands. 

 

View Full Size

Data gathered from FRAPS - Click to Enlarge

Let's zoom in a bit - here is 100 frames of the FRAPS frametimes from above.  Notice the see-saw effect that the CrossFire output has...

View Full Size

Data gathered from Frame Rating Capture - Click to Enlarge

And see how much worse it is here in our Frame Rating Capture configuration.  The pattern is actually exaggerated on the CrossFire solution while the SLI configuration is smoother. 

 

Here are a couple more screenshots from our captures.

View Full Size

Frame Rating capture on Radeon HD 7970 CrossFire - Click to Enlarge

 

View Full Size

Frame Rating capture on GeForce GTX 680s in SLI - Click to Enlarge

 

View Full Size

Frame Rating capture on Radeon HD 7970 CrossFire - Click to Enlarge

 

View Full Size

Frame Rating capture on GeForce GTX 680s in SLI - Click to Enlarge

 

This is just a preview of what we have planned for our new Frame Rating Capture performance testing method.  We have gone through many games with this and the results can vary from looking better than FRAPS to looking much, much worse. 

I am eager to get your feedback - please feel free to leave comments below and the follow on to the conclusion of our GeForce GTX TITAN review!

 

February 26, 2013 | 04:34 AM - Posted by miahallen (not verified)

Excellent work PCPer :D
You'll be my new go-to resource with this type of investigative work going on ;)

February 26, 2013 | 04:48 AM - Posted by akaronin (not verified)

Did I missunderstood or did Ryan Shrout said that sometimes a Radeon HD 7970 CrossFire renders frames at near 0 ms !?

February 26, 2013 | 02:09 PM - Posted by Ryan Shrout

Its more like they are displayed on the screen for near 0ms of time.

February 26, 2013 | 01:50 PM - Posted by Anonymous (not verified)

Great article, looking forward to the full article.

February 26, 2013 | 02:24 PM - Posted by ToiT (not verified)

Awesome to hear about your initiative and innovation behind this guys, great work - I am very excited to see what comes of it in the future.

This seems to explain a lot of the reports at [H] about SLI just 'feeling' better than x-fire during game-play experience.

February 27, 2013 | 03:24 AM - Posted by Pengo (not verified)

A testing revolution! Keep it going!

March 5, 2013 | 04:21 PM - Posted by SilverSurfer755 (not verified)

This seems to effect multi-gpu more correct?

Should i still go ahead with buying a 7970 ghz or go for a gtx 680 instead?

Good job on the article, interesting stuff.

March 8, 2013 | 08:28 AM - Posted by Goofus Maximus (not verified)

I think we need a better name for those slivers than RUNT frames. ;)

How about "Framelets"? It makes one wonder if this is another round in the ongoing dance between nvidia and amd/ati to inflate the performance of their products compared to the competition...

March 10, 2013 | 08:13 AM - Posted by UltimateBou (not verified)

90% of players, plays with vsync on, triple buffer on.

these settings removes 90-100 % of stutter.

using RadeonPro dynamic vsync removes the remaing 10% of stutter

useless test for most players. Stutter exist without vsync and frame limiter... who plays with tearing? -.-'

March 10, 2013 | 10:01 AM - Posted by Mechromancer (not verified)

I run crossfired HD 4870 512MB gpus. I had horrible, jittery, performance until I ran across this fix:

1) create a 'user.cfg' file in the root of the Battlefield 3 game folder (by renaming a .txt to .cfg).

2) add these two lines-
RenderDevice.TripleBufferingEnable 1
RenderDevice.ForceRenderAheadLimit 1

This DRASTICALLY improved my frame smoothness. It worked even better when I enabled vsync (we all know crossfire likes vsync).

I've found that some games require these kinds of manual configuration file tweaks for AMD GPUs, and some for Nvidia GPUS (Borderlands 2 manual needed tweaks for my 550Ti setup). I'm starting to think games need to be optimized for AMD/Nvidia on both the driver side AND the game side, and not just the driver side nowadays.

March 10, 2013 | 01:37 PM - Posted by UltimateBou (not verified)

It's possible to avoid microstuttering but here they test only in the same way... very different results from user test with same methodologies.. it seems that they want to show only the "dark side" of amd card.. and green fan are really happy.

March 12, 2013 | 07:28 AM - Posted by UltimateBou (not verified)

http://img715.imageshack.us/img715/6445/crysis3exe2013031214463.jpg

frame latency locked to 16ms.

dynamic vsync (radeonpro), crossfire 7970ghz edition on 980x@4.2ghz.

1080p, very high all, fxaa, no blur, af16x.

don't try this at home... lol

March 14, 2013 | 08:08 AM - Posted by Trey Long (not verified)

Here's another article which gives frame latency details which seem to support Pc Per's contention.
http://www.hardwarecanucks.com/forum/hardware-canucks-reviews/60166-nvid...

March 14, 2013 | 10:03 AM - Posted by chefbenito (not verified)

Wow- This is some amazing work. Lots of sites benchmark, but guys you are creating a superior universal bench and that's huge!

So in case I missed it, when can we get our hands on this utility, I'd love to see real FPS instead of FRAPS avgs.

Also. I see no bias towards NV or AMD on this site at all. I think Josh is sort of an AMD fanboy as per all of his reviews and comments on the podcasts but I don't see any bias at all in any of the articles or editorial on this site.

April 21, 2013 | 01:49 PM - Posted by Anonymous (not verified)

This is an interesting benchmarking program, I must say... I myself use 2x 4gb 680s in SLI. I'm actually quite surprised at this, but I think that dropped (unrendered) frame would be an interesting spot to examine for AMD's source of driver issues and stuttering... However it isn't a large problem, it would definitely need fixing. I myself like AMD, but they do have issues that need fixing.

April 21, 2013 | 01:49 PM - Posted by Anonymous (not verified)

This is an interesting benchmarking program, I must say... I myself use 2x 4gb 680s in SLI. I'm actually quite surprised at this, but I think that dropped (unrendered) frame would be an interesting spot to examine for AMD's source of driver issues and stuttering... However it isn't a large problem, it would definitely need fixing. I myself like AMD, but they do have issues that need fixing.

January 10, 2014 | 05:44 PM - Posted by Anonymous (not verified)

Yeah, interesting theory...how about we not let one data set decide all things. mmmkay.

January 13, 2014 | 03:50 AM - Posted by bl4C (not verified)

what about testing Lucid Virtu MVP ?
the results might be interesting !

or at least we will finally know what the influence of this technology can be, instead of only speculation

oddly enough, i have not found any tests including Lucid technilogy anywhere ...

(if there are, please provide the link :) )

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Lines and paragraphs break automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockquote><p><br>
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options

By submitting this form, you accept the Mollom privacy policy.