Review Index:
Feedback

FCAT VR: Measuring VR Performance on the GTX 1060 and RX 480

Author:
Manufacturer: Various

Background and setup

A couple of weeks back, during the excitement surrounding the announcement of the GeForce GTX 1080 Ti graphics card, NVIDIA announced an update to its performance reporting project known as FCAT to support VR gaming. The updated iteration, FCAT VR as it is now called, gives us the first true ability to not only capture the performance of VR games and experiences, but the tools with which to measure and compare.

Watch ths video walk through of FCAT VR with me and NVIDIA's Tom Petersen

I already wrote an extensive preview of the tool and how it works during the announcement. I think it’s likely that many of you overlooked it with the noise from a new GPU, so I’m going to reproduce some of it here, with additions and updates. Everyone that attempts to understand the data we will be presenting in this story and all VR-based tests going forward should have a baseline understanding of the complexity of measuring VR games. Previous tools don’t tell the whole story, and even the part they do tell is often incomplete.

If you already know how FCAT VR works from reading the previous article, you can jump right to the beginning of our results here.

Measuring and validating those claims has proven to be a difficult task. Tools that we used in the era of standard PC gaming just don’t apply. Fraps is a well-known and well-understood tool for measuring frame rates and frame times utilized by countless reviewers and enthusiasts, but Fraps lacked the ability to tell the complete story of gaming performance and experience. NVIDIA introduced FCAT and we introduced Frame Rating back in 2013 to expand the capabilities that reviewers and consumers had access to. Using more sophisticated technique that includes direct capture of the graphics card output in uncompressed form, a software-based overlay applied to each frame being rendered, and post-process analyzation of that data, we could communicate the smoothness of a gaming experience, better articulating it to help gamers make purchasing decisions.

View Full Size

For VR though, those same tools just don’t cut it. Fraps is a non-starter as it measures frame rendering from the GPU point of view and completely misses the interaction between the graphics system and the VR runtime environment (OpenVR for Steam/Vive and OVR for Oculus). Because the rendering pipeline is drastically changed in the current VR integrations, what Fraps measures is completely different than the experience the user actually gets in the headset. Previous FCAT and Frame Rating methods were still viable but the tools and capture technology needed to be updated. The hardware capture products we used since 2013 were limited in their maximum bandwidth and the overlay software did not have the ability to “latch in” to VR-based games. Not only that but measuring frame drops, time warps, space warps and reprojections would be a significant hurdle without further development. 

View Full Size

View Full Size

NVIDIA decided to undertake the task of rebuilding FCAT to work with VR. And while obviously the company is hoping that it will prove its claims of performance benefits for VR gaming, it should not be overlooked the investment in time and money spent on a project that is to be open sourced and free available to the media and the public.

View Full Size

NVIDIA FCAT VR is comprised of two different applications. The FCAT VR Capture tool runs on the PC being evaluated and has a similar appearance to other performance and timing capture utilities. It uses data from Oculus Event Tracing as a part of the Windows ETW and SteamVR’s performance API, along with NVIDIA driver stats when used on NVIDIA hardware to generate performance data. It will and does work perfectly well on any GPU vendor’s hardware though with the access to the VR vendor specific timing results.

View Full Size

Continue reading our first look at VR performance testing with FCAT VR!!

The second part of the tool is in the processing of the results. NVIDIA built the FCAT VR Analyzer tool to import data that is output from the capture software and show frametime, dropped frames, warped frames, synthesized frames (space warp), reprojection, etc. By creating charts based on these data points a technical analysis of the performance can be created and reasonable and accurate assertions about the user experience can be compiled.

View Full Size

Compared to the previous FCAT software, which required a dedicated hardware capture system, the new FCAT VR can be run in a software-only environment. This should help it gain considerably more traction with the online media as well as prove to be useful for end users that thrive on performance comparisons and online discussions. (Be prepared for the graphs to show up a lot on your favorite forum!)

I’ve been helping test and improve FCAT VR for a number of months now, and one of the key areas that I pushed NVIDIA was on hardware capture validation of the software-based performance metrics. It was a surprisingly difficult task – splitting and capturing the video going to a VR headset was an unknown, creating the overlays required for post-processing analyzing was complex due the API variances, and accurately portraying minute performance differences in an always-on-VSync environment is tricky. But we were able to do it,  and in doing so independently validate the results that FCAT VR shows and match it with end-user experiences.

Though I don’t imagine many people will be looking to duplicate the methods, the process is complex and sensitive to specific hardware.

1.The HDMI output from the graphics card to the VR headset is divided with a high bandwidth HDMI splitter with one output going to the headset itself and the other to a Datapath capture card.

2.The capture card duplicates the EDID of the headset in order to trick the system into giving us full resolution, full refresh rate video.

3.We use VirtualDub (or another compliant capture tool with low overhead) to capture either full resolution video (2160x1200) or cropped video.

4.An overlay runs on the system being tested that shows two varying color segments – one that is inserted during game engine render, and one that is inserted after final warp and shifts from the VR API.

5.That captured video is post-process analyzed by a custom extractor tool to look for skipped colors in the patterns, multiple frames with the same color, etc. (This process is basically identical to previous FCAT.)

6.A data file is generated that can be imported and used in the same FCAT VR Analyzer software to be compared.

That process is significantly more time consuming than the software-only methods, and in truth, it provides less information due to the Vsync-enabled nature of VR gaming.

FCAT VR Analyzer allows us to easily shift and crop the data sets in the tool to better align captures. This is useful for GPU to GPU comparisons but is necessary to compare two captures from the same run. I was able to use both the software and hardware capture at the same time and look at the results from nearly 100 different benchmark tests.

View Full Size

In the top graph of this image, the brighter green line represents the hardware captured data results while the darker green line is the software based data. By properly aligning the peaks of the data sets, we can determine that the two results match up. The key here is that when the frame render time crosses ~11ms (or in this case dips below), we go from running at 90 FPS to running at 45 FPS which shows a very dramatic spike on the hardware captured data. While accurate and useful, the hardware capture doesn’t tell us the whole story and makes it hard to see how the performance of the GTX hardware is behaving. From what we see in the software-based data, the render time is well maintained at under 20ms, but above 11ms level for smooth 90 FPS gaming. When the frame time drops to sub-11ms, we see the frame rate hit 90 FPS and then eventually cycle back to 45 FPS as we move around in the game area.

Which brings us to the bottom two graphs presented by the FCAT VR Analyzer tool. These are individual and more detailed looks at each data set, attempting to show the gaming experience of each by quantifying the impact on dropped frames and warped frames. The graph looks at within the previous second, how many of the frames shown in the headset were “real” frames as rendered and expected by the game, how many were synthesized (Oculus ASW) and how many were dropped. The scale on the left is 0-90, representing the 90 frames available to be rendered each second. The amount of yellow or red in the graph at each time period tells you how many of those frames were dropped, or had to be warped by the VR APIs.

View Full Size

Beyond just the graphs, the tool allows us get access to a host of information like number of frames rendered, the delivered frame rate and my favorite, the unconstrained frame rate. While the delivered FPS generated metric tells you the frame rate as presented to the user with the 90 FPS Vsync lock in place, unconstrained tells us what the frame rate WOULD BE if the limits of vertical sync were not in place. While this is not directly attributable to any performance on current VR systems, it does tell us what the headroom of a GPU is for higher quality settings, higher refresh screens or future VR hardware and allows us to make interesting comparisons between hardware configurations.

The tool has grown quite a bit over the last few months and now has the ability to import large amounts of source data files and filter, helping users with massive amounts of benchmarks and comparisons to build (like us). You can adjust colors, set regions, zoom and export the graphs all from within the tool. We still have some requests in for further customization but the root tool works amazingly well for the timeline it was developed on.

Now that the embargo from NVIDIA on these tools is over, I can show some of the data we have gathered to give you a better idea of how they work, what they can demonstrate and why I think they are going to be the standard for VR performance evaluation going forward.


March 15, 2017 | 10:20 AM - Posted by Kairu43 (not verified)

Great write-up! Your GeForce.com link for FCATVR in the last sentence of the article is pointing to a file location on your C drive instead of Nvidia's site though. This tool looks really interesting and I look forward to playing with it at some point myself :)

March 15, 2017 | 10:21 AM - Posted by Ryan Shrout

What, you can access my hard drive? Huh... ;)

Thanks!

March 15, 2017 | 10:31 AM - Posted by Kairu43 (not verified)

Better than it was, but now it takes me to the first page of the article instead of the download page on Nvidia's site.

March 15, 2017 | 03:54 PM - Posted by Ken Addison

Fixed... Again!

March 15, 2017 | 03:58 PM - Posted by Kairu43 (not verified)

"Gentlemen, we can correct this link; we have the technology..." ;)
Thanks Ken!

March 15, 2017 | 10:23 AM - Posted by Zach Davis (not verified)

Thanks for all your hard work guys!

March 15, 2017 | 10:59 AM - Posted by Anonymous (not verified)

Tests I'd like to see:

Higher performing GPUs (Fury, 1070)

SLI/CF

7700K-based build (better for games than 5960X)

March 15, 2017 | 11:20 AM - Posted by Anonymous (not verified)

Interesting that the RX 480 loses in all Unreal Engine 4 games (and Dirt Rally, don't know about Codemasters Engine), an engine that is heavily tailored to Nvidia cards. Look at the only game using Unity, Edge Of Nowhere and, surprise, surprise, the RX 480 magically catches up to the GTX 1060. *rolleyes*

I wish more VR devs would use Unity or the UE devs build in better support for AMD Hardware.

March 15, 2017 | 11:23 AM - Posted by Anonymous (not verified)

It's clear that amd is in just stay alive mode in the gpu space, similar to where they were against Intel for the last four years with Bulldozer. I really don't expect Vega to be competitive for more than a month or two as Volta is right around the corner. It's a bad time to a gpu enthusiast from a competition perspective.

March 15, 2017 | 08:51 PM - Posted by Anonymous (not verified)

You could ignore AMD and just buy nVidia for GPU and Intel for CPU like any other normal person. You're not poor are you?

March 18, 2017 | 02:00 AM - Posted by anubis44

By definition, 'normal' people are not 'rich', as they are considered 'average' in wealth, hence the concept of the 'rich', ie. above average in wealth, having more money than the average person.

Just a tip. You're not going to win any friends on here spouting elitist cr@p like that. Nobody here cares how much money you think you have or actually have. It's utterly irrelevant. What matters is whether you are a person of substance, rather than a shallow, materialistic looser.

March 15, 2017 | 11:56 AM - Posted by Jann5s

How can the delivered fps be higher than the unconstrained fps? See the "Dirt Rally - High Preset" result on the "GTX 1060 vs RX 480" page

March 15, 2017 | 12:13 PM - Posted by Kairu43 (not verified)

Check the last paragraph on the "GPU Performance Scaling" page. It explains that phenomenon.

March 15, 2017 | 12:31 PM - Posted by Jann5s

Thx for pointing that out, here's the quote for the even lazier than me:

"Because the frametimes reported by the Oculus and SteamVR runtimes combine the CPU and GPU, but the unconstrained time is based on GPU alone. In situations where the frametime is very close to the 11ms/90FPS mark we see overlaps of CPU and GPU time (rather than it being purely sequential). The result is that unconstrained FPS will sometimes be lower than delivered FPS."

However, I still don't get it, I'm sorry. Perhaps a few more words are in order?

March 15, 2017 | 12:40 PM - Posted by Kairu43 (not verified)

My interpretation of this is that in this scenario, the CPU is getting some work done while the GPU is working; aka the system is performing better/faster simultaneously instead of work having to wait for one to finish before moving to the next compute device for the current operation.

Unconstrained FPS doesn't take the extra performance-gain into account, so when only looking at the GPU and expectation of frame delivery, it's presenting a slower/incomplete analysis of performance. It doesn't pick up that the system is doing some of the work at the same time instead of sequentially, so it's analysis is that it's taking longer than reality shows.

The internet will correct me if I'm wrong, but that was my interpretation of the situation.

March 15, 2017 | 12:46 PM - Posted by Anonymous (not verified)

So, Nvidia wants us to wait for AMD Vega for VR? OK, I'll wait.

March 15, 2017 | 02:05 PM - Posted by Anonymous (not verified)

What is the shader:ROP ratios for both the GTX 1060 and the RX 480? And it again looks like time will tell as to what if any gaming software and driver/API tweaks will do for AMD's polaris based SKUs! The RX 480 and Polaris SKUs are getting some tweaks in the form of the RX 580/other Polaris updates so maybe some retesting should happen then.

There is also maybe some driver/games/other software improvments coming along with any of the Polaris/Refresh RX 500 series SKUs so this very same bemchmark run on the RX 500 series SKUs and maybe some feature support that the RX 480 lacks compared to the GTX 1060.

I also want to see any current CF/SLI VR benchmarking along with any DX12/Vulkan explicit multi-GPU adaptor benchmarking where both CF/SLI are not used and any Dual GPU usage is done through DX12's/Vulkan"s API multi-GPU load balancing methods.

Vega is still a little ways off so some good looks at the RX 500 series Polaris refresh SKUs like any RX 480/RX 580 to GTX 1060 benchmarking, with some added multi-GPU VR/Non VR setups are in order. The RX 480's are dropping in price below the $200 dollar price point and if the RX 580s retail at around $199 then the RX 480 pricing will most likely hit the 150 price point at some point in time on any stock of RX 480s that may be remaining in the retail channels! And that bodes well for some affordable RX 480 CF gaming uaseg for VR/Non VR gaming usage with the new Ryzen/AM4 options that are now available and without.

March 15, 2017 | 08:27 PM - Posted by Mr.Gold (not verified)

The Rx 480 is $179, correct ? Isn't the GTX 1060 40% more ?
They seem to be different class of GPU based on price.

The Radeon Fury seem to be the direct competitor to the 1060, no ?
Exact same price as the 1060.

Would be cool when comparing cards, that the same class or card (based on price) get compared.

Because I'm pretty sure the GTX 1080 ti crush the 1060 at VR...
but then its also pointless the compare them both when they dont cost the same.

March 15, 2017 | 11:39 PM - Posted by renz (not verified)

depends where you live. in my country the two still pretty much about the same price. sometimes a bit cheaper than 1060. also RX480 with such price still depending on deal. AFAIK AMD has not officially cut RX400 series price.

the thing about geforce is they tend to sell well even without price cut so retailer have no reason to offer big deal on them unlike AMD cards that is much harder to sell.

March 18, 2017 | 02:01 AM - Posted by anubis44

"the thing about geforce is they tend to sell well even without price cut so retailer have no reason to offer big deal on them unlike AMD cards that is much harder to sell."

Sigh. Yes, the legions of the stupid strike again. I'm going to really enjoy watching AMD do to nVidia what they just did to Intel. :)

March 16, 2017 | 07:29 PM - Posted by rock1m1

Damn the GTX 1060 is stompin RX 480. Really want to get into performance testing myself.

March 17, 2017 | 03:06 AM - Posted by StephanS

Curious what model of the rx 480 and 1060 was used?

The gap in unconstrained FPS make little sense,
unless the rx480 may have been running at like 1ghz ?

I also found anything below 90fps in VR gives a very poor experience. its not like a game on a monitor going from 60 to 40fps, its noticeable but ok.. in VR dropping below 90fps is headache inducing.

So here the "hardocp" method would be good to use.
in short. what is the game setting to use to reach near rock solid 90fps.

can the gtx 1060 even reach that, and be used for solid VR ?
From this it seem it falls short with the tested settings.

March 17, 2017 | 05:04 PM - Posted by Allyn Malventano

We're using the standard RX480 and GTX 1060.

March 17, 2017 | 09:24 AM - Posted by moose (not verified)

Very interesting paper, but you should give more technical data on the videocards you tested. parameters like memory or GPU frequency, Oced or not, driver version, etc. are important. Not all 1080, 1060, 480 are born equal.

March 17, 2017 | 05:06 PM - Posted by Allyn Malventano

Standard cards were used, and drivers were current as of time of testing for the review.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Lines and paragraphs break automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockquote><p><br>
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options

By submitting this form, you accept the Mollom privacy policy.