Review Index:
Feedback

Frame Rating Dissected: Full Details on Capture-based Graphics Performance Testing

Capture Hardware and Extracting the Data

Capture Hardware

Once we have the overlay working, the next step in our Frame Rating process is to actually capture the video for later analysis.  While that sounds easy, it is far from the case.  Capturing the video on the testing system is out of the question. Instead we are using an external capture system built around an Ivy Bridge platform and a high-end capture card from Datapath, the VisionDVI-DL.

View Full Size

This card is capable of capturing video at 2560x1600 at 60 Hz with throughput as high as 650 MB/s.  We were actually the first customer for this card in the US and have spent quite a few days on the phone with driver engineers to make sure this kind of testing process was successful.  Meant mainly for single frame captures, getting the VisionDVI-DL to really meet the specifications turned out to be a more complicated process than we’d hoped.  The good news is we have it all figured out now and can reliably capture the necessary video.

The VisionDVI-DL essentially acts like and shows up as a monitor to the system.  It reports an EDID (extended display identification data) to the graphics card that is set through the driver on the capture system.  I am able to emulate a 1920x1080 monitor at 60 Hz refresh and up to a 2560x1440 @ 60 Hz as well.  Using a Gefen dual-link DVI splitter we can connect a single display output to the NVIDIA or AMD GPU (to avoid the issues of power consumption and GPU utilization with multiple monitors attached) and split the signal to our capture card and to a monitor for us to actually play the games.

With a data rate of nearly 500 MB/s when capturing video at 2560x1440 @ 60 Hz, some serious storage speed was needed for this capture system.  I decided to use our Pegasus R4 Thunderbolt array coupled with a set of four Corsair Force GS 240GB SSDs running in an RAID 0 array, giving us about 900 MB/s of write capability.

View Full Size

View Full Size

I cannot overestimate the importance of a completely correct capture in this step of the Frame Rating process.  If you deviate from the 60 Hz record rate even a single frame the data becomes basically useless.  Dropping a frame on the capture side would show up as dropped frames from the graphics card so avoiding that is paramount.  The software used for this capture will likely be VirtualDub on other sites that you see, and that was the program we used originally too.  However, the inability to automate the settings and having to re-adjust the options each time you load the program became a hassle so we developed our own that uses DirectShow filters and the Windows Platform SDK. 

 

Extracting the Data

Once you have the recorded raw AVI file, (that is about 25GB – yes GB – from a 60 second capture at 2560x1440) the next step is to actually read the colored bars and generate frame time data, runt data and drop data.  Using a basic DSP application, the extractor reads in each frame of the video sequentially and determines the number of scanlines in height that each color on the overlay bar has and records that information in an XLS file.

View Full Size

This is probably the most fool-proof part of the process.  I have uploaded a few example XLS files right here for you to see the output.

March 29, 2013 | 01:16 PM - Posted by John Doe (not verified)

Horse shit:

http://hardforum.com/showthread.php?t=1703048

March 27, 2013 | 11:36 AM - Posted by John Doe (not verified)

Moral of the story, get this before it goes out of stock:

http://www.newegg.com/Product/Product.aspx?Item=N82E16814187196

It's a card of very limited availability and is one hell of a damn good card.

Best cooler, only Tantalum caps and no cylindrical caps over VRM area, backplated, best looking. Such a boss. Reminds me of those black 68 Mustang Bosses.

March 27, 2013 | 09:30 AM - Posted by Mountainlifter (not verified)

Great Job, Ryan and team. No, Amazing job. I've been following your frame rating articles from the first one and I've been waiting for this full reveal for a long time.

I do hope you show some frame rating benches for the GTX TITAN too (cause I own one now).

I am finding it tough not to conclude that AMD's two GPU system is completely broken although it baffles me that a company of their scale could not detect/think of such issues.

They have been known to take shortcuts on drivers before. http://forum.notebookreview.com/sager-clevo/567681-should-i-switch-out-m... (I'm only getting this from a reliable NBR forumer).

March 27, 2013 | 10:38 AM - Posted by John Doe (not verified)

AMD drivers are put out by a single guy called Catalyst Maker... he has a little team to help him out, but after all, he's just one guy.

That's why they suck.

It's been this way for AGES. The last cards that didn't in one way or another suck, were the X1950's which were made by ATi and not AMD.

Also, dual 7900 setups aren't "completely broken". That's just your pointless assumption.

If there's anything that's "completely broken", it's your wallet for shelling out $1000 for a dumbass card.

March 27, 2013 | 10:51 AM - Posted by Josh Walrath

AMD/ATI used to have a smaller driver team than NV, but actually when AMD bought ATI and started down the APU path... they hired a lot more folks to support these products.  Catalyst Maker was originaly Terry Makedon, but he has moved onto marketing more than software development.  There are (iirc) several groups now working on drivers, but their workload is arguably greater than it was before.  Supporting several different architectures spanning from GPUs to APUs is certainly no easy task, and I think they continue to hire more people.  Reviews like this provide a lot more impetus inside companies to improve their products, and the software drivers certainly are the quick fix are for AMD now.

I think that NV still has a larger driver team, and that massive server room that pretty much only works on nightly builds of drivers from NV is still much bigger than what AMD has.  However, AMD is not as far behind as many think.  But obviously from what we have seen here, and read about from other sources, their focus was not exactly where it needed to be.

March 29, 2013 | 12:12 AM - Posted by Anonymous (not verified)

That's nice and all to amd but it looks like half a decade of total screw job right now, and outright fraud.

March 27, 2013 | 07:27 PM - Posted by Mountainlifter (not verified)

just leave this website man.

March 27, 2013 | 07:29 PM - Posted by Mountainlifter (not verified)

Would you kindly... leave this website man. Thanks.

March 27, 2013 | 07:29 PM - Posted by Mountainlifter (not verified)

Would you kindly... leave this website man. Thanks.

March 27, 2013 | 07:31 PM - Posted by Mountainlifter (not verified)

Was addressing john doe, not walrath.

March 27, 2013 | 09:16 PM - Posted by Josh Walrath

Heh, you can address me.  It won't offend.  As long as I don't offend Ryan, I could be good.

March 29, 2013 | 02:43 AM - Posted by John Doe (not verified)

Hey mountainlifter, would you lift my T-Bird if it gets stuck on a mountain?

Lift my cock while you're there as well.

GG.

March 29, 2013 | 09:14 AM - Posted by Justin Anderson (not verified)

I agree John Doe is retarded and needs to leave. I think he's is trying to pick up dudes on this site he keeps telling people to suck his dick. Why don't you just go to chat roulette if you want to get your rocks off man.

March 29, 2013 | 09:14 AM - Posted by Justin Anderson (not verified)

I agree John Doe is retarded and needs to leave. I think he's is trying to pick up dudes on this site he keeps telling people to suck his dick. Why don't you just go to chat roulette if you want to get your rocks off man.

March 29, 2013 | 09:15 AM - Posted by Justin Anderson (not verified)

I agree, John Doe is retarded and needs to leave. I think he's is trying to pick up dudes on this site he keeps telling people to suck his dick. Why don't you just go to chat roulette if you want to get your rocks off man.

March 29, 2013 | 12:43 PM - Posted by John Doe (not verified)

LOL that's hilarious.

That's all.

Hahaha.

April 2, 2013 | 07:51 PM - Posted by Anonymous (not verified)

Don't tell the truth, the haterz then come forth...

I mean how many times can we give AMD the most gigantic pass ?

Not on this - this new tech with the single card to suck up the actual frames delivered to the end gamer screen with the quad SSD's to take all that in on the fly - this is called PROGRESS, and it has exposed a huge, huge amd flaw.

I feel sorry for all those foolish amd fanboys who screamed at me to buy two amd and crossfire it's a better deal and OC and blah blah blah blah - it's now a big joke.

All those "top crossfire amd gamerz" denying what was right on their screens before them - now the scientific facts show what was before them runts and ghost frames...

I haven't seen a bigger emerging scandal in the video card wars ever.

I am very interested in seeing how it plays out. Of course I will accept NO EXCUSES from amd or their fans. I never have before, but this is an even bigger mess up than even I considered AMD capable of pulling, and that is saying an aweful lot.

You just keep doing your thing and laughing at the haterz, they should be ashamed of themselves honestly.

I want to see if they went with runt and ghost frames after the 69xx series was outed as junk for crossfire by HardOcp with Terry Makedon there in the forums- AMD "became aware" according to that head Cat Maker that there "was a problem" and my guess is the solution was runts and ghosts.
That would keep the DAMAGE to the last 2 major series - but I want data all the way back to 3870/3870X2 to see how deep the rabbit hole goes.

March 27, 2013 | 10:52 AM - Posted by Anonymous (not verified)

so basically what i'm getting from this story and the one at anand is that amd has been reviewed over the years with inflated fraps #s due to more runt and drop frames that are no good to gamers?

March 27, 2013 | 11:30 AM - Posted by Noah Taylor (not verified)

You are absolutely right, AMD's setups have fraudulently inflated their benchmark numbers for years which have helped them sell products and grow their % of discrete market share. This really is the most groundbreaking reveal I've ever come across in the gaming industry.

March 27, 2013 | 11:42 AM - Posted by John Doe (not verified)

It's not like nVidia hasn't done it either:

http://tpucdn.com/reviews/NVIDIA/7950GX2/images/singlebenchlow.jpg

March 27, 2013 | 11:43 AM - Posted by Anonymous (not verified)

Yeah, I'm sure AMD's share of the discrete market has grown thanks to inflated Xfire numbers. I mean, so many people have multi-GPU setups in the first place, amirite?

Plus, jumping to the conclusion that there's deliberate "fraud" going on can only be the result of deluded fanboy thought processes. Considering FCAT is the first tool that can really show this behavior, it's rather more reasonable to believe that AMD was simply unaware of it.

March 27, 2013 | 01:04 PM - Posted by John Doe (not verified)

Alright, let me get it the easily way for you.

What the hell does the inflated CF performance figures have to do with the frame latencies? NOTHING.

The frame latency is a problem only because AMD has yet to be able to completely man up the GCN architecture.

April 3, 2013 | 08:27 PM - Posted by Anonymous (not verified)

No, fraps showed the problem, and the extended FCAT has only highlighted it even more.
The other thing that showed the problem - countless end user complaints, many times raging radeon fanboys screaming you cannot see beyond 30fps because "the human eye is incapable of it", something we call fanboy fantasy, apologetics, excuses, denial, brain washed psycho hardcore amd total verdetrol absolutist.
There are a lot of them, and that unfortunately has allowed amd to scam the entire industry for this long.

You see we still have a boatload of deniers here. Total amd fanboys, nothing will ever change their minds, no facts are facts they can accept if the facts prove amd sucks.

March 29, 2013 | 09:15 AM - Posted by Anonymous (not verified)

Yes it is, but the river in Egypt will start to grow to cover the entire earth for the amd fanboys, oh wait it already has.

BURNED in the extreme, the failure of crossfire, the scam of the decade.
Ghosts and runts, but boy oh boy our fps looks frappalicious !

LOL - so, so pathetic.

nVidia's SLI monsters ? ZERO runts, ZERO ghost frames.
PERFECTION incarnate.

Let's review:
nVidia - perfect execution, no cheating no runts, no ghost frames
amd - lying failure, HALF OF THE FRAMERATE IS A LIE. ghost runt every other frame !

ROFL amd you truly suck as bad as I have been saying for half a decade, wait no, you are worse.

Now we will get treated to endless excuses, denials, it doesn't matter, who cares, and all the rest of the radeon ragers standard operating apologetic procedures they have gang banged on the net for years.

Watch for this development: amd fanboys new spewing propaganda
" using two cards is a hassle, you should buy a single card, and just upgrade when the next best single card comes out like the 7970 with how great it is. Who wants to waste money on two cards, it's like <1% of gamers, so no one really does this at any end user level except fanatics, it's just not needed and is a budget buster. "

LOL - A thousand excuses already coming

March 27, 2013 | 12:21 PM - Posted by Wingless (not verified)

How does changing Render Frame Ahead/Pre-Rendering affect Crossfire and SLI? In Battlefield 3, people are saying it gets smoother if you change it to 0 or 1 vs the default value of 3.

Thanks for your great analysis and extremely informative video.

March 28, 2013 | 06:56 AM - Posted by Wingless (not verified)

Battlefield 3
Guide: How to Fix Low FPS - Stuttering - Lag
http://battlelog.battlefield.com/bf3/forum/threadview/2832654347723598222/

There is a well documented stuttering fix for both Nvidia and AMD users on multiple forums. I've tried this for my HD 4870 Crossfire setup and it works. This particular user from the above link has a NVIDIA GTX 470.

5.Open notepad and paste this into it and save it as "user.cfg" inside your "program files/origingames/battlefield3" folder:

RenderDevice.TripleBufferingEnable 0
RenderDevice.ForceRenderAheadLimit 1
WorldRender.SpotLightShadowmapResolution 256
WorldRender.SpotlightShadowmapEnable 0
Render.DrawFps 1

With this applied to the game, are there any differences? Render Ahead seems to really affect these results and it would be nice if it were tested with FCAT.

Thanks

March 29, 2013 | 01:40 PM - Posted by John Doe (not verified)

Do you like Winger?

They're really a crappy band but they have some decent stuff LOL.

http://www.youtube.com/watch?v=rWSVGcivuGs

March 27, 2013 | 12:43 PM - Posted by Randy (not verified)

Absolutely fantastic work Ryan. This article really blew me away and gave me a lot of tools to better understand what I've been dealing with at home. One question though: you point out that vsync can solve most of amds problems provided the frame rate doesn't have to drop from 60 to 30, in which case the drastic stutter caused by such a change adversely impacts smoothness. What if instead of using vsync you used a frame rate limit that corresponded with the apps average framerate? This way when the frame rate dips below that threshold it isn't cutting its' self in half as it would with vsync and in theory not introducing nearly as much stutter.

I advised my friend to do this with his dual 6950 setup in far cry 3 and he said it gave excellent subjective performance. I used the same method for my single 7970 and also noticed a nice bump to the smoothness of the game, but again this is all subjective. It would be awesome if you could test this as if proven correct it would give a lot of amd owners the tools to make the best of a bad situation.

Again thanks for the tremendous work you've done here, this is truly awesome work.

March 27, 2013 | 01:43 PM - Posted by grommet

Quick question- Could Lucidlogix's Virtual Vsync technology help out with the CrossFire issues?
http://www.lucidlogix.com/technology-virtual-v-sync.html
Not that this would let AMD off the hook as this needs to be taken care of in-house, but could it help?

March 27, 2013 | 02:02 PM - Posted by Anon (not verified)

That's a nice question.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Lines and paragraphs break automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockquote><p><br>
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options

By submitting this form, you accept the Mollom privacy policy.