Feedback

Frame Rating: A New Graphics Performance Metric

Author:
Manufacturer: PC Perspective

A change is coming in 2013

If the new year will bring us anything, it looks like it might be the end of using "FPS" as the primary measuring tool for graphics performance on PCs.  A long, long time ago we started with simple "time demos" that recorded rendered frames in a game like Quake and then played them back as quickly as possible on a test system.  The lone result was given as time, in seconds, and was then converted to an average frame rate having known the total number of frames recorded to start with.

More recently we saw a transition to frame rates over time and the advent frame time graphs like the ones we have been using in our graphics reviews on PC Perspective. This expanded the amount of data required to get an accurate picture of graphics and gaming performance but it was indeed more accurate, giving us a more clear image of how GPUs (and CPUs and systems for that matter) performed in games.

And even though the idea of frame times have been around just a long, not many people were interested in getting into that detail level until this past year.  A frame time is the amount of time each frame takes to render, usually listed in milliseconds, and could range from 5ms to 50ms depending on performance.  For a reference, 120 FPS equates to an average of 8.3ms, 60 FPS is 16.6ms and 30 FPS is 33.3ms.  But rather than average those out by each second of time, what if you looked at each frame individually?

Video Loading...

Scott over at Tech Report started doing that this past year and found some interesting results.  I encourage all of our readers to follow up on what he has been doing as I think you'll find it incredibly educational and interesting. 

Through emails and tweets many PC Perspective readers have been asking for our take on it, why we weren't testing graphics cards in the same fashion yet, etc.  I've stayed quiet about it simply because we were working on quite a few different angles on our side and I wasn't ready to share results.  I am still not ready to share the glut of our information yet but I am ready to start the discussion and I hope our community find its compelling and offers some feedback.

View Full Size

At the heart of our unique GPU testing method is this card, a high-end dual-link DVI capture card capable of handling 2560x1600 resolutions at 60 Hz.  Essentially this card will act as a monitor to our GPU test bed and allow us to capture the actual display output that reaches the gamer's eyes.  This method is the best possible way to measure frame rates, frame times, stutter, runts, smoothness, and any other graphics-related metrics.

Using that recorded footage, sometimes reaching 400 MB/s of consistent writes at high resolutions, we can then analyze the frames one by one, though with the help of some additional software.  There are a lot of details that I am glossing over including the need for perfectly synced frame rates, having absolutely zero dropped frames in the recording, analyzing, etc, but trust me when I say we have been spending a lot of time on this. 

Continue reading our editorial on Frame Rating: A New Graphics Performance Metric.

The result are multi-GB files (60 seconds of game play produces a 16GB file) that have each frame presented to the monitor of the gamer in perfect fashion. 

View Full Size

View Full Size

The second image here was made in Photoshop to show you the three different "frames" of Unigine that make up the single frame that the gamer would actually see.  With a 60 Hz display, this equates to three frames being shown in 16ms, though each frame has a variable amount of real estate on the screen. 

We are still finalizing what we can do with all of this data, but it definitely allows us to find some unique cases:

View Full Size

This shot from Sleeping Dogs shows three frames on our single 16ms instance to the monitor, but notice how little screen spaces the green frame takes up - is measuring this frame even useful for gamers?  Should it be dropped from the performance metrics all together?

I put together this short video below with a bit more on my thoughts on this topic but I can assure you that January and February are going to bring some major change to the way graphics cards are tested.  Please, we want your thoughts!  This is an important dialogue for not only us but for all PC gamers going forward.  Scott has done some great work "Inside the Second" and we hope that we can offer additional information and clarification on this problem.  Rest assured we have not been sitting on our hands here! 

Video Loading...

Is this kind of testing methodology going to be useful?  Is it overkill?  Leave us your thoughts below!

January 3, 2013 | 08:58 PM - Posted by brisa117

Finally! This looks to be the best real-world analysis I've ever seen from a reviewer! What would be interesting would be to somehow compare what you're capturing and what monitor X is displaying with monitor reviews! No idea how that would work though! Best of luck, you guys rock!

January 3, 2013 | 09:02 PM - Posted by Ryan Shrout

I wouldn't expect to see differences there as the capture card is actually working as our "monitor".

January 3, 2013 | 09:38 PM - Posted by Anonymous (not verified)

This is where you will see the DEFINITIVE superiority of Nvidia products. At least you better, or Tom is going to have some explaining to do.

If we cross paths at CES I will personally shake your hands and kiss your feet.

support@bossrigs.com

January 3, 2013 | 11:22 PM - Posted by Anonymous (not verified)

I think it will appear that nvidia is missing frames to smooth out game play whilst AMD brute forcing all of them.

I also think people saying AMD has better picture quality will be finally proven against nvidia`s washed out/colorless frames.

At the end of the day, their is only soo much you can do with a gimped GPU vs a beefed up one.

April 8, 2013 | 03:14 PM - Posted by Anonymous (not verified)

The gimped GPU being AMD and the good one being Nvidia, correct ?

January 4, 2013 | 12:27 AM - Posted by nummakayne (not verified)

This is phenomenal stuff - a total game-changer. Kudos to you, and your shoutout to Scott Wasson. I have always had great respect for you and PC Perspective, especially since that time so many years ago (I think it was in 2004?) when I e-mailed you in response to a motherboard review and you actually took time out to respond to it. You, Mr. Shrout, are awesome and far too kind.

I wouldn't necessarily say I'm particularly knowledgeable at this kind of stuff but I think I understand enough to know that this is definitely the right way to truly paint an honest picture of GPU performance and the end-user experience.

It reminds of the time when Crysis came out and people were divided into two camps: one that argued that any shooter running at less than 60fps is not 'smooth' while reviewers and many others insisted Crysis was 'smooth enough' even at 25fps average with dips to the high teens.

Would this kind of testing reveal any superiority in the way CryEngine handled frame output? Again, maybe I'm not making sense but this new wave of 'high average frame rate doesn't always relate to a smooth user experience' testing reminded me of that time.

January 4, 2013 | 12:00 PM - Posted by Ryan Shrout

It very well could but I don't know if we'd go back that far since this testing is so time consuming.

January 4, 2013 | 12:53 AM - Posted by dragosmp (not verified)

A question about this part:
"At the heart of our unique GPU testing method is this card, a high-end dual-link DVI capture card capable of handling 2560x1600 resolutions at 60 Hz."

You haven't mentioned what is the frequency of the capture, is it 60Hz? It would seem so since you talk about 16ms frame times. If this is true, then you lack the means to measure frames faster than 16ms, which do exist (of course they do, we have over 60FPS games). Also to follow up on that idea, the capture card may lack the means to measure anything but multiples of 16ms frame times. Is it the case, or if I missed something please explain.

Nice start to 2013

January 4, 2013 | 11:33 AM - Posted by arbiter

It might be a limit on the card but 60Hz is pretty standard for most monitors and people that play games. It could be possible for 120FPS capture but thing less you dumb down the graphic options or spend a lot of SLI setup just smaller market. You could see how inflated the numbers are as like image above where in 1 16ms instance there was was counted as 3 frames. 60Hz is just a good point to work from since majority of people that use it and don't need SLI setup to get it for most games.

January 4, 2013 | 12:03 PM - Posted by Ryan Shrout

Keep in mind in this testing there are two "frame rates".  The first is the rate at which the game engine is being rendered.  The second is the frame rate with which our hardware is capturing on the other side. 

The second only matters in that it must be 60 Hz (or match the refresh rate of the display) EXACTLY.  We need that to be perfect so that we know we are getting each and every frame from the graphics card as it would be display on a 60 Hz monitor.  We can still measure frame rates much higher than 60 Hz with Vsync disabled as we showed you in that Photoshoped screenshot by measuring the "rendered frames".

January 4, 2013 | 02:00 AM - Posted by Supreeth (not verified)

Hi Ryan,

Good job with the new performance testing metric..
My question is with such a thing like the stuttering and latency observed in these tests of graphics cards..will the already existing cards be able to remove such effects which the consumers observe while gaming? If so how?

Once again nice job!!

January 4, 2013 | 12:03 PM - Posted by Ryan Shrout

Drivers drivers drivers.

January 4, 2013 | 05:47 PM - Posted by raffriff (not verified)

driver developers! driver developers! driver developers! *throws chair*

January 4, 2013 | 05:51 PM - Posted by raffriff (not verified)

Sorry for the multi post. I blame NoScript.

January 4, 2013 | 03:52 AM - Posted by Dave (not verified)

I'm really glad to see more review sites looking past FPS as the end-all be-all metric for performance.

It's the reason I read sites like Tech Report (who sorta pioneered the frame time metric) and [H]ardOCP who, while reporting FPS min, max, avg, etc., concentrate their conclusions on the "experience" rather than the raw numbers.

This is the most important "metric" - the experience. I think frame times and your expansion on that provide an objective way of measuring the "experience."

As others have alluded to, I'd rather have a game play smoothly at 45 FPS with very little latency and low frame times (giving a smooth experience) than a game that averages higher FPS but produces more latency and more higher frame times resulting in a less smooth experience.

Good work on this Ryan, and I hope to see more soon.

January 4, 2013 | 04:08 AM - Posted by Torvik (not verified)

Certainly Ryan I think you need to show any frames that are dropped and whether or not they did contain any meaningful data. Hardware that is consistently dropping meaningful data isn't working correctly in my opinion. If these were network cards constantly dropping data packets you'd swap it out straight away. Any dropped frames would be interesting to see especially if gpu vendors are using driver optimisations to specifically drop frames to increase perceived average frame rates. It's not hard to imagine GPU vendors have long gotten wise to press review techniques and given rise to the "review press kits" which test their hardware in the best light. Gamers don't want to see the best light we want to see the reality however horrific that might be. Only then can you give good independent advice and I think that's what you should be doing.

So the more data the better though it needs to be presented in a meaningful way to the average user. Perhaps starting with an article or part article that explains what all these terms are and how they would effect your gaming experience.

Also interested to see what triple buffering is doing here in terms of frame latency and render ahead frame limits in DX11. How switching on AA or AF effects frame latency is probably important too allowing a user to understand fully the consequence of switching graphics options on or off outside of a FPS number.

Ultimately myself as a gamer I am looking for a fluid artifact free gaming experience that would be like watching a movie. That's every gamers goal I think.

I think it's also important to set a benchmark by testing current generation hardware to get a baseline. Also being clear what driver versions and settings are being used as well.

I think moving in this direction will drive a new development in graphics hardware/software and will really probably be the biggest breakthrough for some time.

Well done.

January 4, 2013 | 12:06 PM - Posted by Ryan Shrout

Thanks!

What I meant by the question of "removing" small frames from our ratings is basically trying to decide if it should count in the real "performance frame rate."

For example, if over a 1 second time period there are 80 frames displayed but 30 of them are only shown for 20 scanlines or less, then essentially your "smooth" frame rate is more like 50.

January 5, 2013 | 07:14 AM - Posted by Torvik (not verified)

Yes I think you probably should be presenting a "smooth" frame rate of 50 in the case you described but you have to clearly state the cut off you are using and at what point you consider a frame to not contain enough data to justify being counted is that 5% of the screen or more or less?

Throughout reading all of this though I think you have trashed the idea of a single metric of frame rate as a valid performance measurement.

Perhaps a better way to look at this is to judge all cards by a set acceptable value of smooth motion framerate lets for arguments sake say 60fps standard vsync monitor rate. Then present a % of frame delivery outside of this baseline or inside this baseline

e.g 60 fps - 1 new frame drawn every 16ms then report the percentage of frames delivered slower than 16ms effectively creating a fluid/smoothness rating.

It's about consitency now not about min and max though there has to be a baseline.

The other way to present it might be to calculate the avg frame delivery time and again present the percentage of frames falling outside the cards average frame delivery time?

January 4, 2013 | 07:58 AM - Posted by gamerk2 (not verified)

What I think would be helpful would be to determine, on average, how many UNIQUE frames actually are drawn to the display over a second, rather then how many frames are created. This is VERY important for those of us who use Vsync, since frames are either all or nothing.

January 4, 2013 | 06:24 PM - Posted by Ryan Shrout

We should be able to determine that as well.

January 8, 2013 | 07:16 AM - Posted by gamerk2 (not verified)

Good to hear. In my mind, I don't care how many frames the GPU puts out, I care about how many unique frames make it to my screen. That, at the end of the day, is what really matters in my mind.

In any case, I'm glad FPS is finally being put to the wayside. This method especially will have a LOT of impact when looking at SLI/CF configs (including the proof that microstutter=latency, which I've suspected for some time now).

January 8, 2013 | 07:16 AM - Posted by gamerk2 (not verified)

Good to hear. In my mind, I don't care how many frames the GPU puts out, I care about how many unique frames make it to my screen. That, at the end of the day, is what really matters in my mind.

In any case, I'm glad FPS is finally being put to the wayside. This method especially will have a LOT of impact when looking at SLI/CF configs (including the proof that microstutter=latency, which I've suspected for some time now).

January 4, 2013 | 08:31 AM - Posted by Brokenstorm (not verified)

Would it be possible use this to test how a 60Hz monitor compares to a 120Hz one as far as dropped/partially displayed frames go?

I assume that as long as the output doesn't exceed the max data rate of the capture card (650MB/s) it shouldn't be a problem. Obviously this would not be possible at 1440p/1600p put perhaps it could work at 1080p/1200p.

January 4, 2013 | 12:07 PM - Posted by Ryan Shrout

That is the bottleneck - capture hardware.  We haven't tried it on a 120 Hz monitor yet but we plan to right after CES.

January 4, 2013 | 11:01 AM - Posted by Mike D (not verified)

These results will be very useful in crossfire/sli testing involving micro-stuttering to give the end user an idea of actual scaling. Keep up the good work Ryan & PCPer!

January 4, 2013 | 11:17 AM - Posted by YTech2 (not verified)

This is a great topic!

This may be something that GPU-Software engineers have been ignoring ("users won't notice").
This reminds me of concept testing for stereoscopic images, animation, and model rendering to provide fast rendering at best quality on highest resolutions.

Thank you for sharing. I understand about being quiet. Keep up the good work and I am looking forward to hear more about your findings through your podcast!

January 4, 2013 | 12:07 PM - Posted by Ryan Shrout

Thanks for listening!

January 4, 2013 | 12:48 PM - Posted by Fulgurant

I registered just to say that you're doing great work. Thanks for articulating (and attempting to measure) what many of us have wrestled with for years.

And good luck to those Bengals on Saturday. :)

January 4, 2013 | 01:32 PM - Posted by Ryan Shrout

Who Dey!

January 4, 2013 | 02:10 PM - Posted by xbeaTX (not verified)

my most sincere congratulations for your idea ... I think we will show very interesting results and closer to the real gaming experience.
it's time to say stop to the reviews that look like subliminal advertising.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Lines and paragraphs break automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockquote><p><br>
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options

By submitting this form, you accept the Mollom privacy policy.