Review Index:
Feedback

Frame Rating Dissected: Full Details on Capture-based Graphics Performance Testing

How Games Work

 

Because of the complexity and sheer amount of data we have gathered using our Frame Rating performance methodology, we are breaking it up into several articles that each feature different GPU comparisons.  Here is the schedule:

 

Introduction

The process of testing games and graphics has been evolving even longer than I have been a part of the industry: 14+ years at this point. That transformation in benchmarking has been accelerating for the last 12 months. Typical benchmarks test some hardware against some software and look at the average frame rate which can be achieved. While access to frame time has been around for nearly the full life of FRAPS, it took an article from Scott Wasson at the Tech Report to really get the ball moving and investigate how each frame contributes to the actual user experience. I immediately began research into testing actual performance perceived by the user, including the "microstutter" reported by many in PC gaming, and pondered how we might be able to test for this criteria even more accurately.

The result of that research is being fully unveiled today in what we are calling Frame Rating – a completely new way of measuring and validating gaming performance.

The release of this story for me is like the final stop on a journey that has lasted nearly a complete calendar year.  I began to release bits and pieces of this methodology starting on January 3rd with a video and short article that described our capture hardware and the benefits that directly capturing the output from a graphics card would bring to GPU evaluation.  After returning from CES later in January, I posted another short video and article that showcased some of the captured video and stepping through a recorded file frame by frame to show readers how capture could help us detect and measure stutter and frame time variance. 

View Full Size

Finally, during the launch of the NVIDIA GeForce GTX Titan graphics card, I released the first results from our Frame Rating system and discussed how certain card combinations, in this case CrossFire against SLI, could drastically differ in perceived frame rates and performance while giving very similar average frame rates.  This article got a lot more attention than the previous entries and that was expected – this method doesn’t attempt to dismiss other testing options but it is going to be pretty disruptive.  I think the remainder of this article will prove that. 

Today we are finally giving you all the details on Frame Rating; how we do it, what we learned and how you should interpret the results that we are providing.  I warn you up front though that this is not an easy discussion and while I am doing my best to explain things completely, there are going to be more questions going forward and I want to see them all!  There is still much to do regarding graphics performance testing, even after Frame Rating becomes more common. We feel that the continued dialogue with readers, game developers and hardware designers is necessary to get it right.

Below is our full video that features the Frame Rating process, some example results and some discussion on what it all means going forward.  I encourage everyone to watch it but you will definitely need the written portion here to fully understand this transition in testing methods.  Subscribe to your YouTube channel if you haven't already!

Continue reading our analysis of the new Frame Rating performance testing methodology!!

How Games Work

Before we dive into why I feel that our new Frame Rating testing method is the best for the gamer, there is some necessary background information that you must understand.  While we all play games on a near daily basis, most of us don’t fully grasp the complexity and detail that goes into producing an image that makes its way from code a developer writes to the pixels on your monitor.  The below diagram attempts to simplify the entire process from the game engine to the display.

View Full Size

In the image above we have defined a few important variables based on time that will help us explain graphics anomalies.  The first is t_game and it refers the internal time that the game engine is using to keep track of its internal simulations and interactions.  This is where processes like the physics simulations, user interface, artificial intelligence and more are handled.  Different game engines keep time in different ways, but they usually fall into two categories: fixed or variable time steps.  In a fixed time method the game engine cycles the environment in its internal simulation on a regular, fixed iteration.  This is more predictable but it also forces other systems to sync up to it (drivers, GPU rendering) which may cause some issues.  The variable time step method allows a lot more flexibility, but it can be more complicated for developers to maintain a fluid feeling simulation because of simply not knowing when the next simulation update will take place. 

Following that is t_present, the point at which the game engine and graphics card communicate to say that they are ready to pass information for the next frame to be rendered and displayed.  What is important about this time location is that this is where FRAPS gets its time stamps and data and also where the overlay that we use for our Frame Rating method is inserted.  What you should notice right away though is that there is quite a bit more work that occurs AFTER the t_present command is sent and before the user actually sees any result.  This in particular is where capture method’s advantages stem from.  

After DirectX calls the game engine to recieve its time, the graphics driver gets its hands on it for the first time.  The driver maps the DX calls to its own specific hardware and then starts the rendering process on a GPU.  Once the frame is rendered, we will define another variable, t_render, which is reported when the image is ready to be sent to a display.  Finally, t_display will be defined as the time in which data from the frame is on the display, whether that be a complete frame (Vsync enabled for example) or a partial frame. 

You can likely already see where the differences between FRAPS data measurement and Frame Rating measurement start to develop.  Using capture hardware and analysis tools that we’ll detail later, we are recording the output from the graphics card directly as if it were a monitor.  We are essentially measuring frames at the t_display level rather than the t_present point, giving us data that has been filtered through the entire game engine, DirectX, driver, rendering and display process. 

It is also useful to discuss stutter and how it relates to these time definitions.  Despite some readers opinions, stutter in game play is related to the smoothness of animation and not hard wired to low frame rate.  If you have a steady frame rate of 25 FPS you can still have an enjoyable experience (as evident by the 24 FPS movies we all watch at the theater).  Instead, we should view stutter as a variance level between t_game and t_display; if the total display time runs at 50ms (20 FPS) you won’t have stuttering in animation (in most cases) but if total display time shifts between 20ms and 50ms you definitely will.

In our second video on Frame Rating, we looked at a capture from Dishonored in a frame by frame analysis and saw a rather large stutter.  I think a better term for that is a hitch, a large single frame rate issue that isn’t indicative of a microstutter smoothness problem.  Using the above variables, a hitch would be a single large instance of t_game – t_display.  As you’ll see in our results analysis pages (including games like Skyrim) this happens fairly often, even in some games that are smooth otherwise. 

Our 2nd video on Frame Rating from a couple months ago...

There is also a completely different discussion on the advantages and differences of capturing data from FRAPS versus capturing data with our Frame Rating methodology.  I believe that the advantages of hardware capture outweigh the concerns currently, but in reality the data that FRAPS generates isn’t that important, it just happens to be the closest data point to another metric we would love to know more about: game time. 

Game time is the internal game clock that the software engine uses to keep track of the physical world.  This clock could be based on a fixed time span for each tick or it could be variable or timed off of another source (like the OS or GPU driver).  An Intel GPU engineer, Andrew Lauritzen, recently made a great post over on the Beyond3D.com forums about game time, back pressure on the game pipeline and much more.  Here is a short portion of that post, but I would encourage everyone to read the entirety completely:

1) Smooth motion is achieved by having a consistent throughput of frames all the way from the game to the display.

2) Games measure the throughput of the pipeline via timing the back-pressure on the submission queue. The number they use to update their simulations is effectively what FRAPS measures as well.

3) A spike anywhere in the pipeline will cause the game to adjust the simulation time, which is pretty much guaranteed to produce jittery output. This is true even if frame delivery to the display (i.e. rendering pipeline output) remains buffered and consistent. i.e. it is never okay to see spikey output in frame latency graphs.

4) The converse is actually not true: seeing smooth FRAPS numbers does not guarantee you will see smooth display, as the pipeline could be producing output to the display at jittery intervals even if the input is consistent. This is far less likely though since GPUs typically do relatively simple, predictable work.

Clearly the best case for evaluating overall gaming performance will be to have access to the internal game and measure it in comparison the output from Frame Rating, the actual frames on your display.  Differences there could be analyzed to find exact bottlenecks in the pipeline from game code to display.  The problem is no game engine developers allow access to the information currently and the number of different engines in use today makes it difficult for even the likes of NVIDIA and AMD to gather data reliably.  There is opportunity for change here if an API were to exist (in DirectX for example) that would give all game engines reliable time iterations that we would then have access to.

 

NVIDIA's Involvement

You may notice that there is a lot of “my” and “our” in this story while also seeing similar results from other websites being released today.  While we have done more than a year’s worth of the testing and development on our own tools to help expedite a lot of this time consuming testing, some of the code base and applications were developed with NVIDIA and thus were distributed to other editors recently.

NVIDIA was responsible for developing the color overlay that sits between the game and DirectX (in the same location of the pipeline as FRAPS essentially) as well as the software extractor that reads the captured video file to generate raw information about the lengths of those bars in an XLS file.  Obviously, NVIDIA has a lot to gain from this particular testing methodology: its SLI technology looks much better than AMD’s CrossFire when viewed in this light, highlighting the advantages that SLI’s hardware frame metering bring to the table. 

The next question from our readers should then be: are there questions about the programs used for this purpose?  After having access to the source code and applications for more than 12 months I can only say that I have parsed through it all innumerable times and I have found nothing that NVIDIA has done that is disingenuous.  Even better, we are going to be sharing all of our code from the Perl-based parsing scripts (that generate the data in the graphs you’ll see later from the source XLS file) as well as a couple of examples of the output XLS files. 

Not only do we NEED to have these tools vetted by other editors, but we also depend on the community to keep us on our toes as well.  When we originally talked with NVIDIA about this project the mindset from the beginning was merely to get the ball rolling and let the open source community and enthusiast gamers look at every aspect of the performance measurement.  That is still the goal – with only one minor exception: NVIDIA doesn’t want the source code of the overlay to leak out simply because of some potential patent/liability concerns.  Instead, we are hoping to have ANOTHER application built to act as the overlay; it may be something that Beepa and the FRAPS team can help us with.

March 29, 2013 | 01:16 PM - Posted by John Doe (not verified)

Horse shit:

http://hardforum.com/showthread.php?t=1703048

March 27, 2013 | 11:36 AM - Posted by John Doe (not verified)

Moral of the story, get this before it goes out of stock:

http://www.newegg.com/Product/Product.aspx?Item=N82E16814187196

It's a card of very limited availability and is one hell of a damn good card.

Best cooler, only Tantalum caps and no cylindrical caps over VRM area, backplated, best looking. Such a boss. Reminds me of those black 68 Mustang Bosses.

March 27, 2013 | 09:30 AM - Posted by Mountainlifter (not verified)

Great Job, Ryan and team. No, Amazing job. I've been following your frame rating articles from the first one and I've been waiting for this full reveal for a long time.

I do hope you show some frame rating benches for the GTX TITAN too (cause I own one now).

I am finding it tough not to conclude that AMD's two GPU system is completely broken although it baffles me that a company of their scale could not detect/think of such issues.

They have been known to take shortcuts on drivers before. http://forum.notebookreview.com/sager-clevo/567681-should-i-switch-out-m... (I'm only getting this from a reliable NBR forumer).

March 27, 2013 | 10:38 AM - Posted by John Doe (not verified)

AMD drivers are put out by a single guy called Catalyst Maker... he has a little team to help him out, but after all, he's just one guy.

That's why they suck.

It's been this way for AGES. The last cards that didn't in one way or another suck, were the X1950's which were made by ATi and not AMD.

Also, dual 7900 setups aren't "completely broken". That's just your pointless assumption.

If there's anything that's "completely broken", it's your wallet for shelling out $1000 for a dumbass card.

March 27, 2013 | 10:51 AM - Posted by Josh Walrath

AMD/ATI used to have a smaller driver team than NV, but actually when AMD bought ATI and started down the APU path... they hired a lot more folks to support these products.  Catalyst Maker was originaly Terry Makedon, but he has moved onto marketing more than software development.  There are (iirc) several groups now working on drivers, but their workload is arguably greater than it was before.  Supporting several different architectures spanning from GPUs to APUs is certainly no easy task, and I think they continue to hire more people.  Reviews like this provide a lot more impetus inside companies to improve their products, and the software drivers certainly are the quick fix are for AMD now.

I think that NV still has a larger driver team, and that massive server room that pretty much only works on nightly builds of drivers from NV is still much bigger than what AMD has.  However, AMD is not as far behind as many think.  But obviously from what we have seen here, and read about from other sources, their focus was not exactly where it needed to be.

March 29, 2013 | 12:12 AM - Posted by Anonymous (not verified)

That's nice and all to amd but it looks like half a decade of total screw job right now, and outright fraud.

March 27, 2013 | 07:27 PM - Posted by Mountainlifter (not verified)

just leave this website man.

March 27, 2013 | 07:29 PM - Posted by Mountainlifter (not verified)

Would you kindly... leave this website man. Thanks.

March 27, 2013 | 07:29 PM - Posted by Mountainlifter (not verified)

Would you kindly... leave this website man. Thanks.

March 27, 2013 | 07:31 PM - Posted by Mountainlifter (not verified)

Was addressing john doe, not walrath.

March 27, 2013 | 09:16 PM - Posted by Josh Walrath

Heh, you can address me.  It won't offend.  As long as I don't offend Ryan, I could be good.

March 29, 2013 | 02:43 AM - Posted by John Doe (not verified)

Hey mountainlifter, would you lift my T-Bird if it gets stuck on a mountain?

Lift my cock while you're there as well.

GG.

March 29, 2013 | 09:14 AM - Posted by Justin Anderson (not verified)

I agree John Doe is retarded and needs to leave. I think he's is trying to pick up dudes on this site he keeps telling people to suck his dick. Why don't you just go to chat roulette if you want to get your rocks off man.

March 29, 2013 | 09:14 AM - Posted by Justin Anderson (not verified)

I agree John Doe is retarded and needs to leave. I think he's is trying to pick up dudes on this site he keeps telling people to suck his dick. Why don't you just go to chat roulette if you want to get your rocks off man.

March 29, 2013 | 09:15 AM - Posted by Justin Anderson (not verified)

I agree, John Doe is retarded and needs to leave. I think he's is trying to pick up dudes on this site he keeps telling people to suck his dick. Why don't you just go to chat roulette if you want to get your rocks off man.

March 29, 2013 | 12:43 PM - Posted by John Doe (not verified)

LOL that's hilarious.

That's all.

Hahaha.

April 2, 2013 | 07:51 PM - Posted by Anonymous (not verified)

Don't tell the truth, the haterz then come forth...

I mean how many times can we give AMD the most gigantic pass ?

Not on this - this new tech with the single card to suck up the actual frames delivered to the end gamer screen with the quad SSD's to take all that in on the fly - this is called PROGRESS, and it has exposed a huge, huge amd flaw.

I feel sorry for all those foolish amd fanboys who screamed at me to buy two amd and crossfire it's a better deal and OC and blah blah blah blah - it's now a big joke.

All those "top crossfire amd gamerz" denying what was right on their screens before them - now the scientific facts show what was before them runts and ghost frames...

I haven't seen a bigger emerging scandal in the video card wars ever.

I am very interested in seeing how it plays out. Of course I will accept NO EXCUSES from amd or their fans. I never have before, but this is an even bigger mess up than even I considered AMD capable of pulling, and that is saying an aweful lot.

You just keep doing your thing and laughing at the haterz, they should be ashamed of themselves honestly.

I want to see if they went with runt and ghost frames after the 69xx series was outed as junk for crossfire by HardOcp with Terry Makedon there in the forums- AMD "became aware" according to that head Cat Maker that there "was a problem" and my guess is the solution was runts and ghosts.
That would keep the DAMAGE to the last 2 major series - but I want data all the way back to 3870/3870X2 to see how deep the rabbit hole goes.

March 27, 2013 | 10:52 AM - Posted by Anonymous (not verified)

so basically what i'm getting from this story and the one at anand is that amd has been reviewed over the years with inflated fraps #s due to more runt and drop frames that are no good to gamers?

March 27, 2013 | 11:30 AM - Posted by Noah Taylor (not verified)

You are absolutely right, AMD's setups have fraudulently inflated their benchmark numbers for years which have helped them sell products and grow their % of discrete market share. This really is the most groundbreaking reveal I've ever come across in the gaming industry.

March 27, 2013 | 11:42 AM - Posted by John Doe (not verified)

It's not like nVidia hasn't done it either:

http://tpucdn.com/reviews/NVIDIA/7950GX2/images/singlebenchlow.jpg

March 27, 2013 | 11:43 AM - Posted by Anonymous (not verified)

Yeah, I'm sure AMD's share of the discrete market has grown thanks to inflated Xfire numbers. I mean, so many people have multi-GPU setups in the first place, amirite?

Plus, jumping to the conclusion that there's deliberate "fraud" going on can only be the result of deluded fanboy thought processes. Considering FCAT is the first tool that can really show this behavior, it's rather more reasonable to believe that AMD was simply unaware of it.

March 27, 2013 | 01:04 PM - Posted by John Doe (not verified)

Alright, let me get it the easily way for you.

What the hell does the inflated CF performance figures have to do with the frame latencies? NOTHING.

The frame latency is a problem only because AMD has yet to be able to completely man up the GCN architecture.

April 3, 2013 | 08:27 PM - Posted by Anonymous (not verified)

No, fraps showed the problem, and the extended FCAT has only highlighted it even more.
The other thing that showed the problem - countless end user complaints, many times raging radeon fanboys screaming you cannot see beyond 30fps because "the human eye is incapable of it", something we call fanboy fantasy, apologetics, excuses, denial, brain washed psycho hardcore amd total verdetrol absolutist.
There are a lot of them, and that unfortunately has allowed amd to scam the entire industry for this long.

You see we still have a boatload of deniers here. Total amd fanboys, nothing will ever change their minds, no facts are facts they can accept if the facts prove amd sucks.

March 29, 2013 | 09:15 AM - Posted by Anonymous (not verified)

Yes it is, but the river in Egypt will start to grow to cover the entire earth for the amd fanboys, oh wait it already has.

BURNED in the extreme, the failure of crossfire, the scam of the decade.
Ghosts and runts, but boy oh boy our fps looks frappalicious !

LOL - so, so pathetic.

nVidia's SLI monsters ? ZERO runts, ZERO ghost frames.
PERFECTION incarnate.

Let's review:
nVidia - perfect execution, no cheating no runts, no ghost frames
amd - lying failure, HALF OF THE FRAMERATE IS A LIE. ghost runt every other frame !

ROFL amd you truly suck as bad as I have been saying for half a decade, wait no, you are worse.

Now we will get treated to endless excuses, denials, it doesn't matter, who cares, and all the rest of the radeon ragers standard operating apologetic procedures they have gang banged on the net for years.

Watch for this development: amd fanboys new spewing propaganda
" using two cards is a hassle, you should buy a single card, and just upgrade when the next best single card comes out like the 7970 with how great it is. Who wants to waste money on two cards, it's like <1% of gamers, so no one really does this at any end user level except fanatics, it's just not needed and is a budget buster. "

LOL - A thousand excuses already coming

March 27, 2013 | 12:21 PM - Posted by Wingless (not verified)

How does changing Render Frame Ahead/Pre-Rendering affect Crossfire and SLI? In Battlefield 3, people are saying it gets smoother if you change it to 0 or 1 vs the default value of 3.

Thanks for your great analysis and extremely informative video.

March 28, 2013 | 06:56 AM - Posted by Wingless (not verified)

Battlefield 3
Guide: How to Fix Low FPS - Stuttering - Lag
http://battlelog.battlefield.com/bf3/forum/threadview/2832654347723598222/

There is a well documented stuttering fix for both Nvidia and AMD users on multiple forums. I've tried this for my HD 4870 Crossfire setup and it works. This particular user from the above link has a NVIDIA GTX 470.

5.Open notepad and paste this into it and save it as "user.cfg" inside your "program files/origingames/battlefield3" folder:

RenderDevice.TripleBufferingEnable 0
RenderDevice.ForceRenderAheadLimit 1
WorldRender.SpotLightShadowmapResolution 256
WorldRender.SpotlightShadowmapEnable 0
Render.DrawFps 1

With this applied to the game, are there any differences? Render Ahead seems to really affect these results and it would be nice if it were tested with FCAT.

Thanks

March 29, 2013 | 01:40 PM - Posted by John Doe (not verified)

Do you like Winger?

They're really a crappy band but they have some decent stuff LOL.

http://www.youtube.com/watch?v=rWSVGcivuGs

March 27, 2013 | 12:43 PM - Posted by Randy (not verified)

Absolutely fantastic work Ryan. This article really blew me away and gave me a lot of tools to better understand what I've been dealing with at home. One question though: you point out that vsync can solve most of amds problems provided the frame rate doesn't have to drop from 60 to 30, in which case the drastic stutter caused by such a change adversely impacts smoothness. What if instead of using vsync you used a frame rate limit that corresponded with the apps average framerate? This way when the frame rate dips below that threshold it isn't cutting its' self in half as it would with vsync and in theory not introducing nearly as much stutter.

I advised my friend to do this with his dual 6950 setup in far cry 3 and he said it gave excellent subjective performance. I used the same method for my single 7970 and also noticed a nice bump to the smoothness of the game, but again this is all subjective. It would be awesome if you could test this as if proven correct it would give a lot of amd owners the tools to make the best of a bad situation.

Again thanks for the tremendous work you've done here, this is truly awesome work.

March 27, 2013 | 01:43 PM - Posted by grommet

Quick question- Could Lucidlogix's Virtual Vsync technology help out with the CrossFire issues?
http://www.lucidlogix.com/technology-virtual-v-sync.html
Not that this would let AMD off the hook as this needs to be taken care of in-house, but could it help?

March 27, 2013 | 02:02 PM - Posted by Anon (not verified)

That's a nice question.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Lines and paragraphs break automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockquote><p><br>
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options

By submitting this form, you accept the Mollom privacy policy.