Review Index:
Feedback

NVIDIA GeForce GTX TITAN Performance Review and Frame Rating Update

Author: Ryan Shrout
Manufacturer: NVIDIA

Frame Rating Update - Why CrossFire was Omitted

Why are you not testing CrossFire??

One question I know will be asked about this review is that in our benchmarks today you will not see results from AMD CrossFire configurations in 2-Way or 3-Way combinations.  AMD is only represented by a single Radeon HD 7970 GHz Edition card while we are using both the GTX 690 dual-GPU card, GTX 680s in SLI and GTX TITANs in SLI. 

Why the bias?

If you haven't been following our sequence of stories that investigates a completely new testing methodology we are calling "frame rating", then you are really missing out.  (Part 1 is here, part 2 is here.)  The basic premise of Frame Rating is that the performance metrics that the industry is gathering using FRAPS are inaccurate in many cases and do not properly reflect the real-world gaming experience the user has.

Because of that, we are working on another method that uses high-end dual-link DVI capture equipment to directly record the raw output from the graphics card with an overlay technology that allows us to measure frame rates as they are presented on the screen, not as they are presented to the FRAPS software sub-system.  With these tools we can measure average frame rates, frame times and stutter, all in a way that reflects exactly what the viewer sees from the game.

We aren't ready to show our full sets of results yet (soon!) but the problems lie in that AMD's CrossFire technology shows severe performance degradations when viewed under the Frame Rating microscope that do not show up nearly as dramatically under FRAPS.  As such, I decided that it was simply irresponsible of me to present data to readers that I would then immediately refute on the final pages of this review - it would be a waste of time for the reader and people that skip only to the performance graphs wouldn't know our theory on why the results displayed were invalid.

Many other sites will use FRAPS, will use CrossFire, and there is nothing wrong with that at all.  They are simply presenting data that they believe to be true based on the tools at their disposal.  More data is always better. 

Here are these results and our discussion.  I decided to use the most popular game out today, Battlefield 3 and please keep in mind this is NOT the worst case scenario for AMD CrossFire in any way.  I tested the Radeon HD 7970 GHz Edition in single and CrossFire configurations as well as the GeForce GTX 680 and SLI.  To gather results I used two processes:

  1. Run FRAPS while running through a repeatable section and record frame rates and frame times for 60 seconds
  2. Run our Frame Rating capture system with a special overlay that allows us to measure frame rates and frame times with post processing.

Here is an example of what the overlay looks like in Battlefield 3.

View Full Size

Frame Rating capture on GeForce GTX 680s in SLI - Click to Enlarge

The column on the left is actually the visuals of an overlay that is applied to each and every frame of the game early in the rendering process.  A solid color is added to the PRESENT call (more details to come later) for each individual frame.  As you know, when you are playing a game, multiple frames will make it on any single 60 Hz cycle of your monitor and because of that you get a succession of colors on the left hand side.

By measuring the pixel height of those colored columns, and knowing the order in which they should appear beforehand, we can gather the same data that FRAPS does but our results are seen AFTER any driver optimizations and DX changes the game might make.

View Full Size

Frame Rating capture on Radeon HD 7970 CrossFire - Click to Enlarge

Here you see a very similar screenshot running on CrossFire.  Notice the thin silver band between the maroon and purple?  That is a complete frame according to FRAPS and most reviews.  Not to us - we think that frame rendered is almost useless. 

 

View Full Size

Data gathered from FRAPS

Here is a typical frame rate over time graph as generated by FRAPS.  Looks good right?  CrossFire and SLI are competitive with the advantage to the HD 7970s.

View Full Size

Data gathered from Frame Rating Capture

This is the same graph with data gathered from our method that omits RUNT frames that only represent pixels under a certain threshold (to be discussed later).  Removing the tiny slivers gives us a "perceived frame rate" that differs quite a bit - CrossFire doesn't look faster than a single card.

 

View Full Size

Data gathered from FRAPS - Click to Enlarge

This is the raw frame times as captured by FRAPS - again we are looking for a narrow band of frametimes to represent a smooth experience.  Both single cards do pretty well, but SLI sees a bit more variance and CrossFire sees a bigger one.  Quite a bit bigger.

View Full Size

Data gathered from Frame Rating Capture - Click to Enlarge

Here is the same data gathered by our new capture system - the CrossFire configuration looks MUCH worse with many frames hitting near 0ms of screen time.  That would be great if they were ALL like that but unfortunately they also scale up to 20ms and higher quite often.  Also notice NVIDIA's is actually MORE uniform indicating that there is some kind of smoothing going on after the frame leaves the game engine's hands. 

 

View Full Size

Data gathered from FRAPS - Click to Enlarge

Let's zoom in a bit - here is 100 frames of the FRAPS frametimes from above.  Notice the see-saw effect that the CrossFire output has...

View Full Size

Data gathered from Frame Rating Capture - Click to Enlarge

And see how much worse it is here in our Frame Rating Capture configuration.  The pattern is actually exaggerated on the CrossFire solution while the SLI configuration is smoother. 

 

Here are a couple more screenshots from our captures.

View Full Size

Frame Rating capture on Radeon HD 7970 CrossFire - Click to Enlarge

 

View Full Size

Frame Rating capture on GeForce GTX 680s in SLI - Click to Enlarge

 

View Full Size

Frame Rating capture on Radeon HD 7970 CrossFire - Click to Enlarge

 

View Full Size

Frame Rating capture on GeForce GTX 680s in SLI - Click to Enlarge

 

This is just a preview of what we have planned for our new Frame Rating Capture performance testing method.  We have gone through many games with this and the results can vary from looking better than FRAPS to looking much, much worse. 

I am eager to get your feedback - please feel free to leave comments below and the follow on to the conclusion of our GeForce GTX TITAN review!

February 21, 2013 | 06:21 AM - Posted by DeadOfKnight (not verified)

I see Dirt 3 Graphs on the Crysis 3 page.

February 21, 2013 | 07:00 AM - Posted by Ryan Shrout

Fixed!

February 21, 2013 | 06:22 AM - Posted by j0hndoe

Man, that card is a BEAST. Look forward to reading more thoughts on it as you have more time with it! Great write up, Ryan. Thanks

February 21, 2013 | 07:03 AM - Posted by YTech2 (not verified)

TITAN is back for more! - Clarification - From photo provided, card offers 1 dual-link DVI-I and 1 dual-link DVI-D. For those using a DVI to VGA (Analogue) adapter would use the DVI-I.

Yes, various Analogue displays are still being used.

I do wonder about the performance of the TITAN in Stereoscopic Systems.

February 21, 2013 | 07:21 AM - Posted by Anonymous (not verified)

Well from what I'm seeing three gtx 680sc are better (faster) than two titans and I'm running 6000×1200 I could only afford two and with the limited quality of titan cards I think I'll pass...

February 21, 2013 | 07:22 AM - Posted by Anonymous (not verified)

meant to say limited quantity I'll be passing......

February 26, 2013 | 11:18 AM - Posted by John Doe (not verified)

Indeed, while the fuckers are eVGA are eating their hamburgers over those shitty "SuperClocked" cards they ripped you off $20 each.

"Superclocking" is flashing the cards BIOS 50 Mhz over the stock clock and HOSING BLIND FUCKERS.

sigh...

April 9, 2013 | 06:22 AM - Posted by Cito (not verified)

OMG!!!!!!!!!!!! 20$ WOW!!! they are robing people blind.....

Dude chill its just 20 bux i did it might as well.

But if you were to talk about the signature edition now then i would agree with you.

February 21, 2013 | 07:28 AM - Posted by D1RTYD1Z619

Finally a BF3 2560 x 1440 gtx 680 2gb sli benchmark. Do you think you'll bench the Titan against the 680 4gb versions?

February 21, 2013 | 08:05 AM - Posted by tackle70 (not verified)

BTW the temporary fix for CF stuttering is to either use Radeon Pro or MSI Afterburner to limit the framerate. If you are in a setting where you can get fairly consistent framerates at or above your monitor's refresh rate, you just limit the framerate to your refreshrate (60 in most cases). Otherwise, you limit it to around your average FPS.

It's not a perfect solution, but it does reliably deal with stutter on crossfire 7970s.

March 28, 2013 | 01:24 PM - Posted by Anonymous (not verified)

I'm so vindicated, and so disappointed...

" We aren't ready to show our full sets of results yet (soon!) but the problems lie in that AMD's CrossFire technology shows severe performance degradations when viewed under the Frame Rating microscope that do not show up nearly as dramatically under FRAPS. As such, I decided that it was simply irresponsible of me to present data to readers that I would then immediately refute on the final pages of this review - it would be a waste of time for the reader and people that skip only to the performance graphs wouldn't know our theory on why the results displayed were invalid. "

AMD sucks so badly, standard apologist mantra is issued.
This is the sub par life of AMD video cards.

They really suck, but our test that is more accurate than fraps shows that, so we will refute our own test and say they don't really suck, so we won't show you the pathetic cheaty runt missed frame data and PROVE AMD HAS BEEN CHEATING LIKE HECK FOR YEARS ON END!

Some of us always knew it, and always said it, and we were attacked relentlessly.
Well, let the attacks continue, as the cover up is still ongoing.

Maybe by now AMD has fixed their YEARS LONG ISSUES so a just cleaned up totally new result can be shown soon --- FORGETTING THE YEARS AMD USERS SUFFERED WITH THE AMD CRAP THAT WON'T BE SHOWN UNTIL IT IS "FIXED".

Then of course the bank robbers get off scott free.
Good job AMD, the Mob wishes it had that kind of pull, as does every politician in the entire world.

February 21, 2013 | 08:54 AM - Posted by Dan (not verified)

So, if you have $1k to power a 5760x1080 gaming setup, do you personally go with the Titan, 2x7970, 2x680 or a 690?

February 21, 2013 | 10:35 PM - Posted by Eric B (not verified)

3x7970

February 21, 2013 | 08:58 AM - Posted by grommet

Where are the promised "TITAN up the graphics" quotes?
I am disappointed.

February 21, 2013 | 09:18 AM - Posted by luciano (not verified)

Your framerate rating method looks like the best.
Could you guysreview, as other user comment here suggests, the frame cap "fix" results?
Thanks for the great review

February 21, 2013 | 09:28 AM - Posted by Tommy (not verified)

No 2560x1600?

No 690 sli in the sli results?

February 23, 2013 | 08:12 AM - Posted by btdog

No answer to your first question, but to your second question, a 690 is technically already in SLI (it's two 680s SLI'd on one card)

February 24, 2013 | 05:03 PM - Posted by Tommy (not verified)

I should have been more specific. By 690 SLI I mean quad-sli 690s.

February 21, 2013 | 10:19 AM - Posted by Anonymous (not verified)

I take it when they bring out a dual-GPU TITAN to compare against the 690s, it will be far better? Then you can test SLI for both dual-GPU TITAN and 690s. Will watch this space

February 21, 2013 | 10:36 AM - Posted by MarkT (not verified)

Feels like the drivers for titan are premature, Ryan is this the kind of perforance Nvidia is expecting or are there upcoming driver enhancements for this card?

February 21, 2013 | 10:53 AM - Posted by Zorkwiz

I'm not sure why this review justified a Gold Award at the $1000 price point. Sure, it's the best single GPU card you can get and the power/thermals look impressive, but the conclusion basically says that it's beaten in single screen setups by both the 690 and 680 SLI, which are the same price or cheaper.

I'm not convinced that the promise of better multi-monitor gaming performance gives it the Gold, especially since you guys have such a limited set of benchmarks for those setups thus far.

I'd still love to own one, but I feel like $899 would have made it a viable option, not $999.

February 21, 2013 | 03:31 PM - Posted by Ryan Shrout

How could $100 really matter in a card this pricey?

Your other point is fair, but I HAVE seen the 5760x1080 numbers and the potential perf advantage there is realized.

February 25, 2013 | 12:59 PM - Posted by HeavyG (not verified)

The people willing to pay for this card are going to buy it regardless if it is $999, $899, or $1099. There is a specific market for it, and it surely isn't the "best bang for your buck" target.

Heck, I bought a 690 just for the cool factor. I have the space on my board, and the 680s would have squeaked by a bit more in terms of performance. The 690 looked way cooler and I liked the idea of one day adding a 2nd 690. This card is really no different.

And by all means, you can still buy 680 SLIs if you want. Nobody is stopping you and nobody is saying that it isn't the right choice for you.

February 21, 2013 | 11:32 AM - Posted by Kitten Masher (not verified)

It's great to see some concrete frame rating data, but I feel like your presentation and analysis of it is a bit off.

Specifically, when you're looking at the distribution of rates, you're using a bar graph to try to help us infer the variance. Why don't you just show us a graph of the distribution of the frame ratings instead? I think visually it would be much easier to compare looking for a "skinnier" distribution with a smaller variance, and you can still mark up the distribution percentiles on it. I'm really curious to see how that might look with the crossfire data you showed, I would almost expect to see two 'mounds' on either side of the mean (although I could be wrong).

Also, your analysis feels a bit hollow, sometimes just reiterating what we saw in the graph without telling us what truly matters: do we care? Was there 'stuttering'? This is especially true since you can't just base it off of a % difference in the times across the percentiles. The actual time taken is also important, and it is also relative to the recorded FPS. All in all, what frame rate times actually mean to the end user isn't as black and white as "lower is better".

OR I'm completely clueless as to what's going on.

Anyways, it's still awesome to see some frame rating data and I look forward to seeing more in the future!

February 21, 2013 | 04:05 PM - Posted by Ryan Shrout

Appreciate the feedback, I'll look into that presentation option for data this week.

As for the analysis, I was kind of purposefully vague as I have a lot more data and compile to present the "whole" picture of CrossFire versus SLI.  

February 22, 2013 | 08:30 AM - Posted by Lord Binky (not verified)

Some games like Mechwarrior Online do not support SLI, so single GPU performance is still relevant.

February 25, 2013 | 01:07 PM - Posted by HeavyG (not verified)

It is relevant, but MOST games support SLI. Games that don't support it on release typically don't do very well. Take Rage, for example.

As for Mech Warrior... it is still in beta, right? It is using CryENGINE 3, right? I don't see it being a problem for too long, but I have been wrong before. After all, I pre-ordered Aliens Colonial Marines.

February 21, 2013 | 12:29 PM - Posted by serpico (not verified)

thank you, I appreciate the frametime graphs.

February 21, 2013 | 03:21 PM - Posted by 6GB Guy (not verified)

Could you test the TITAN against the GTX 680 4GB versions and the Sapphire HD 7970 TOXIC Edtion? It would be interesting to see how well a 6gb 7970 does against the Titan.

February 21, 2013 | 05:40 PM - Posted by ocre (not verified)

man you really went the extra mile on your frame time method. I second that it could be the best method so far. Great work. cant wait till your finally the point you can reveal all your findings. I know you put so much work into it. It seems very promising.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Lines and paragraphs break automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockquote><p><br>
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options

By submitting this form, you accept the Mollom privacy policy.