Review Index:
Feedback

Frame Rating Dissected: Full Details on Capture-based Graphics Performance Testing

Visualizing Runts and Lower Frame Rates – Video Examples

I know that with the publication of this article finalizing our testing methods for Frame Rating, we are going to get lots of comments from gamers about how they have these configurations and that they don’t see the issues that we are discussing and penalizing AMD for.  Whether or not they are telling the truth or there is a bit self-preservation / fan-boyism at work, we wanted to do our best and try to provide example of these phenomenon and we made a handful of videos to share with our readers.

Keep in mind that these videos are using content captures through our Frame Rating system from Radeon HD 7950s in CrossFire and GeForce GTX 660 Ti cards in SLI – but the effect on visuals is identical to what we see with the HD 7970s and GTX 680s.  We used records from our 1920x1080 testing.  Also, because we are doing manual run throughs of all of these games now, no two runs are exactly alike; there will be differences in the overall gameplay sequences but we have done our best to make sure they are as close as possible.

If you would prefer to download this video file, you can do so here.

This is the longest our example videos that shows 50% speed and 30% speed.  Keep an eye on the smoothness of the animations on the screen including while walking straight ahead as well as while turn and rotating.  Because we have Vsync disabled on these systems you will see some horizontal tear on the screen on BOTH configurations and that is fully expected.  In order to show the examples as exactly as possible we had to leave that feature turned off but you should try to ignore those large “tears” and focus on the “micro” stutter on the grass on the ground, etc.

Did you see it?  We debating making this video a blind A/B test but instead decided it was better to just be up front with the results. 

If you would prefer to download this video file, you can do so here.

This video captured from Sleeping Dogs shows the same kind of animation smoothness advantage that NVIDIA SLI has over the Radeon HD 7950s in CrossFire.  This animation difference is not due strictly to stutter but that the fact that AMD configuration is seeing every other frame a runt, thereby cutting the number of frames in each time period in half compared to the 660 Tis in SLI. 

If you would prefer to download this video file, you can do so here.

Our next video is of Far Cry 3 and while I admit that neither experience is a smooth as we would like, the HD 7950s in CrossFire are showing a lot more stuttering than the 660 Ti cards.

If you would prefer to download this video file, you can do so here.

The visual animation issues are bit harder to see in our Battlefield 3 video due to some darker than normal scenes, but if you keep an eye on the sprint section down the alley way closely you will clearly see the slower animation fresh rate of the AMD CrossFire system.

If you would prefer to download this video file, you can do so here.

Our capture of Crysis 3 is another case where neither configuration is running as smooth as we should expect but it is very apparent that the AMD HD 7950s are running at a much lower observed frame rate than the 660 Ti SLI solution. 

If you would prefer to download this video file, you can do so here.

DiRT 3 is a bit different - this compares single card performance and smoothness on SINGLE cards, showing you that without taking multi-GPU issues into account, both vendors can provide high quality real-world experiences.  Both sides of the video look equally smooth and look to be providing a solid game play experience, which more or less backs up the results we found in our 1920x1080 data in today’s article.   

 

Even though we are showing videos slowed down to 50% and even 30% of their full frame rate, these animation differences are very apparent to me in real time, while playing the games.  Without a doubt there will be some variability in what kind of annoyance thresholds for each gamer but even if you can’t see the difference when looking at ONLY your video, you should be able to see the differences when looking at the two options side by side. 

One of the advantages of our capture system is that we keep literally every video we record, though obviously in a compressed format.  (Keeping 25GB, 60 second benchmark runs would hilarious.)  This allows us to verify our data analysis results through the extractor and Perl scripts with the original recorded video to see if the animation stutters, variances, runts, etc are all there. 

I realize we need some more videos and demonstrations of these animation issues and we are working on producing some more that run at various percent speeds including 100%.  If you have any more ideas for how to use standard video edit tools to create comparison that would help our readers understand the experience we are having with our hardware, please email me or let us know in the comments below!

March 27, 2013 | 07:34 PM - Posted by Ryan Shrout

I'll see if we can take a look at that.

March 29, 2013 | 03:31 PM - Posted by jgstew

The whole time I was reading this article, I was more and more curious how Virtu's technology would effect things. I'm curious about more than just their Virtual V-sync, but their other options as well for both single and multiple GPUs. Virtu has not had the scaling that SLI & Crossfire have had, but perhaps their technology would show well in other areas with this analysis.

I do feel that Frame Rating & the input to display latency are much more interesting metrics.

Great work on the article.

March 27, 2013 | 05:32 PM - Posted by serpinati of the wussu (not verified)

I've read somewhere that this will not be the norm for pcper to do this type of testing with all video card reviews (too labor intensive). Is this true?

If it is true, will pcper at least record the frame data and simply give it a quick look to make sure video cards aren't doing something absolutely crazy (for example, if you were reviewing those 7970's crossfire you might not plan to actually to analyze the frame times, but you would at least look over the recorded frames anyways and catch the crazy runt frames and mention it in your reviews.)

March 27, 2013 | 07:34 PM - Posted by Ryan Shrout

No, my plan is to take this route going forward.

March 27, 2013 | 05:57 PM - Posted by Anonymous (not verified)

Greatly appreciate the work behind this! And opening up the tools / scripts to everyone, pushing it to other hardware magazines. Huge kudos! Oh, and you should slap Anandtech over the head for not even mentioning your involvment into the new metering method in their introductory article, that only mentions nVidia...

Is there any way to determine latency between t_present and t_display? If not, it should be - maybe some kind of timestamp could be worked into the overlay? Because that would be interesting not only in the context of VSync, but also regarding nVidias frame metering, which must take some time analyzing the frametimes. Supposedly, AMD has some smoothing algorithm coming up for Crossfire as well, so there it would also be valuable information.

Regarding Adaptive / Smooth VSync: They clearly come off too good in this article. They're not the focus of the article of course, still a little more thought should have been put behind this, considering the amount of time taken to create this awe inspiring effort of an article and the tools behind it. Adaptive VSync turns VSync off when tearing is most annoyingly visible, i.e. at framerates below monitor refresh rate - true triple buffering (instead of the queue nowadays called triple buffering) would be the good solution here. Smooth Vsync does not seem to do anything particularly positive judging from what little measurement is available in the article - only time (further tests that is) will tell what it actually does...

March 27, 2013 | 07:36 PM - Posted by Ryan Shrout

We have more research into the different NVIDIA Vsync options coming up, stay tuned.

As for the timestamp different to check for the gaps between t_present and t_display...we are on that same page as well.  :)

March 27, 2013 | 05:57 PM - Posted by Mawrtin (not verified)

Have you tried CF using radeonpro? Apparently it offers some kinda of Dynamic V-sync similar to nVidias adaptive if I'm not mistaken.

March 27, 2013 | 08:03 PM - Posted by Marty (not verified)

No, it does not. It's a frame limiter, which eliminates some of the stutter, but causes more lag (increases latency).

March 27, 2013 | 06:16 PM - Posted by Mangix

Hmmm. I wonder if there is a difference between Double and Triple Buffered VSync. Newer Valve games support both. Would appreciate testing there.

Also, are there driver settings that help/harm frame rating? Nvidia's Maximum Pre-rendered frames setting sounds like something that can have an effect.

March 27, 2013 | 07:25 PM - Posted by xbeaTX (not verified)

AMD has complained on Anandtech for this type of test using fraps ... now anyone know the real situation and it's shocking
this article sets the new stantard of excellence ... congratulations and thanks for the enormous work done... keep it up! :)

March 27, 2013 | 07:37 PM - Posted by Ryan Shrout

Thank you!  Help spread the word!

March 28, 2013 | 01:55 AM - Posted by TinkerToyTech

Posted to my facebook page

March 27, 2013 | 07:40 PM - Posted by Marty (not verified)

Ryan, I've a suspicion that you wanted to select Adaptive VSync on NVidia, but have selected Adaptive (half refresh rate) instead in the control panel. Would you please check it out.

March 28, 2013 | 05:20 AM - Posted by showb1z (not verified)

²

Other than that great article. Would also be interested in results with frame limiters.

March 27, 2013 | 08:23 PM - Posted by Dan (not verified)

This site is so awesome that it is one of th eonly ones that I disable Adblock in Chrome for. You deserve the ad $'s.

Rock on, Ryan and crew!

March 29, 2013 | 10:37 AM - Posted by Ryan Shrout

Thank you we appreciate it!

March 27, 2013 | 08:45 PM - Posted by Mike G (not verified)

Thanks for the all of the time taken to accurately test and then explain your frame rating methods to us. I wonder if AMD would be willing to have a representative come on and speak on how they will be addressing this issue. I for one will be holding off purchasing an additional 7970 at this time.

March 27, 2013 | 09:43 PM - Posted by bystander (not verified)

They have, with AnandTech, though the person they spoke to was the single GPU driver guy, they did mention they have a plan to offer fixes in July.
http://www.anandtech.com/tag/gpus

March 27, 2013 | 09:51 PM - Posted by Soulwager (not verified)

Is your capture hardware capable of capturing 1080p @ 120hz? The data rate should be less than 1440p@60hz.

Also, I would like to see some starcraft 2 results. It's frequently CPU limited, and I'm wondering how that impacts the final experience when compared to a gpu limited situation. I'd recommend the "unit preloader" map as a benchmark run, once to pre-load all the assets and again for the capture.

March 29, 2013 | 10:38 AM - Posted by Ryan Shrout

Actually, I think 1080p@120 is a higher pixel clock than 25x14@60; we tried to do basic 120 Hz testing right before publication without luck but we'll be trying again soon.

As for SC2, we'll take a look.

March 29, 2013 | 03:00 PM - Posted by Soulwager (not verified)

You're right, the pixel clock is higher. I guess I was only thinking about total number of pixels that need to be recorded, but the smaller resolution is more heavily impacted by overscan.

March 28, 2013 | 12:28 AM - Posted by SPBHM

great stuff.

I would be interested in seeing some results with some framerate limit, not from vsync, but another limit, something like FPS max 45 for Crysis 3 (higher than a single card, around the average for the CF), you can easily do that with dxtory

March 28, 2013 | 12:49 AM - Posted by Trey Long (not verified)

Its nice to see the truth come out. Save the excuses and fix it AMD.

March 28, 2013 | 06:41 AM - Posted by Carol Smith (not verified)

Why did you completely ignore AMD's RadeonPro freeware ?
Which offers and has offered Dynamic V-Sync that's superior to Nvidia's Adaptive V-Sync for years now.
Tom's Hardware seems to be the only site that actually used this utility to make a fair comparison between SLI and Crossfire.

RadeonPro's Dynamic V-Sync was shown to be clearly superior to Nvidia's Adapative V-Sync Implementation .

Here is the Tom's link
http://www.tomshardware.com/reviews/radeon-hd-7990-devil13-7970-x2,3329-...

I hope that you learn from your fellow journalist's to include this fantastic solution in your upcoming article.
Thanks for your hard work and good luck.

March 28, 2013 | 09:27 AM - Posted by Marty (not verified)

A frame limiter improves stuttering, but increases lag. You loose some Fps too. So you have to decide between the devil and lucifer in the case of Crossfire.

March 29, 2013 | 10:01 AM - Posted by Carol Smith (not verified)

Exact same thing with Nvidia SLI and Adaptive V-Sync, if you don't want stuttering you have to sacrifice framrates.

March 29, 2013 | 10:39 AM - Posted by Ryan Shrout

I'm going to take a look, but this is NOT "AMD's" software.

March 28, 2013 | 06:50 AM - Posted by Andre3000 (not verified)

Thanks for the eye opener! Which AMD drivers have been used for this review? I have read the article.. but i might have missed it?

March 29, 2013 | 10:39 AM - Posted by Ryan Shrout

For AMD we used 13.2 beta 7 and for NVIDIA we used 314.07.

March 29, 2013 | 11:27 AM - Posted by Steve (not verified)

Ryan: Is it possible to test AMD's CF with older drivers to see if this problem has been around for a long time or if it is a more recent problem with AMD's continuous driver upgrades to improve speeds?

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Lines and paragraphs break automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockquote><p><br>
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options

By submitting this form, you accept the Mollom privacy policy.