A quick look at performance results

We spent a bit of time with the new BF4 Mantle patch and the AMD Catalyst 14.1 beta driver.

Late last week, EA and Dice released the long awaited patch for Battlefield 4 that enables support for the Mantle renderer.  This new API technology was introduced by AMD back in September. Unfortunately, AMD wasn't quite ready for its release with their Catalyst 14.1 beta driver.  I wrote a short article that previewed the new driver's features, its expected performance with the Mantle version of BF4, and commentary about the current state of Mantle.  You should definite read that as a primer before continuing if you haven't yet.  

Today, after really just a few short hours with a useable driver, I have only limited results.  Still, I know that you, our readers, clamor for ANY information on the topic.  I thought I would share what we have thus far.

Initial Considerations

As I mentioned in the previous story, the Mantle version of Battlefield 4 has the biggest potential to show advantages in times where the game is more CPU limited.  AMD calls this the "low hanging fruit" for this early release of Mantle and claim that further optimizations will come, especially for GPU-bound scenarios.  Because of that dependency on CPU limitations, that puts some non-standard requirements on our ability to showcase Mantle's performance capabilities.

For example, the level of the game and even the section of that level, in the BF4 single player campaign, can show drastic swings in Mantle's capabilities.  Multiplayer matches will also show more consistent CPU utilization (and thus could be improved by Mantle) though testing those levels in a repeatable, semi-scientific method is much more difficult.  And, as you'll see in our early results, I even found a couple instances in which the Mantle API version of BF4 ran a smidge slower than the DX11 instance.  

For our testing, we compiled two systems that differed in CPU performance in order to simulate the range of processors installed within consumers' PCs.  Our standard GPU test bed includes a Core i7-3960X Sandy Bridge-E processor specifically to remove the CPU as a bottleneck and that has been included here today.  We added in a system based on the AMD A10-7850K Kaveri APU which presents a more processor-limited (especially per-thread) system, overall, and should help showcase Mantle benefits more easily.

We have several other platforms setup for testing, with various graphics cards and processors (including a Core i3-4330), but testing will take quite a bit more time to get all of the results we want to present.

Test System 1

  • Intel Core i7-3960X
  • 16GB DDR3-1866 memory
  • ASUS R9 290X DirectCU II

Test System 2

  • AMD A10-7850K
  • 16GB DDR3-2133 memory
  • ASUS R9 290X DirectCU II

 

Testing Methodology

As you might be aware, we use a graphics performance testing methodology called Frame Rating for nearly all of our reviews.  This process uses capture hardware to record the output from the test PC that gets post-processed to evaluate real-world end user experience.  Unfortunately, for Mantle, our tools don't quite work yet. Dice helped us out though by integrating a new feature with the latest BF4 patch to enable frame time logging directly through the in-game console.

PerfOverlay.FrameFileLogEnable 1 starts recording frame times to a CSV file.  Setting that variable to 0 will stop the recording.  All log files will find their way into the My Documents/Battlefield 4 folder.  Even better, because this is done at the engine level, we can get some interesting information from it.  Not only is the frame time for each display frame recorded,  we also get the GPU and CPU time for each frame as well.  This is interesting as it allows us to see how much time each individual component spent on each frame to check for bottlenecks on both ends.

THIS. IS. BETA!!

I'm sure you have all used beta Catalyst drivers before.  These are much different.  I might go as far as to consider these drivers, and the Mantle libraries themselves, in an "alpha" state.  There is a pretty sizeable list of known issues which AMD is being very upfront about.  Its includes the following:

  • Intermittent stuttering or stability issues may occur when utilizing Mantle with AMD CrossFire technology in BattleField 4
  • Mantle performance for the AMD Radeon HD 7000/HD 8000 Series GPUs and AMD Radeon R9 280X and R9 270X GPUs will be optimized for BattleField 4 in future AMD Catalyst releases. These products will see limited gains in BattleField 4 and AMD is currently investigating optimizations for them.
  • Notebooks based on AMD Enduro or PowerXpress technologies are currently not supported by the Mantle codepath in Battlefield 4.
  • AMD Eyefinity configurations utilizing portrait display orientations are currently not supported by the Mantle codepath in Battlefield 4.
  • Graphics hardware in the AMD A10-7850K and A10-7700K may override the presence of a discrete GPU under the Mantle codepath in Battlefield 4.
  • AMD testing for the AMD Catalyst 14.1 Beta has been concentrated on the following products: AMD Radeon R9 290X, R9 290, R9 280, R9 270, R7 260X, R7 260, HD 7000 Series, HD 8000 Series, A10-7850K and A10-7700K. Future AMD Catalyst releases will include full test coverage for all AMD products supported by Mantle.

The CrossFire stability issue, at least in my experience, is not a minor thing as I was never able to get it working properly on either an X79 platform or the A88X + Kaveri system.

In several talks with AMD this week I did express concern over this very early beta status of Mantle.  My fear is that with AMD having difficulty getting Mantle working with a single gaming title (Battlefield 4 today) after having a full month of extra time (remember the initial promised December release) does not bode well for Mantle in the long term.  AMD tells me that Mantle has been in the works for more than two years which should be enough to perfect the platform for these limited variables. It is possible that these growing pains of stability and performance are a one time occurrence to go along with a new graphics API.  But, once several or a dozen games are available using Mantle, how will AMD's support and development teams be able to handle it?

A Quick Note on Image Quality

I know that some of you are curious about the image quality comparisons between DX11 and Mantle codepaths.  Though I didn't do any testing in this story I did go to the man himself to ask the question.  Johan Andersson, Technical Director on Frostbite at Electronic Arts, told me quite clearly that "no, there shouldn't be any difference" in image quality between the two options in Battlefield 4.  I don't usually just take people at their word, but he is an exception!

Worst Case Scenario

Our standard level and section used for testing Battlefield 4 is the opening portions of the single player SP_Dam stage.  As it turns out, this section is quite GPU limited (as we would normally want for our GPU testing…) but this doesn't fair well for AMD's Mantle out of the gate.  Here are results with the Core i7-3960X and R9 290X at 2560×1600 with the Ultra preset.

Click to Enlarge

Click to Enlarge – Note the incorrect labeling here, it is NOT 1920×1080 but in fact 2560×1600

Results from both the Mantle version and the DX11 version are essentially the same here, averaging about 43 FPS.  That's a bit disappointing but, after the preparation from AMD about setting expectations, it more or less makes sense.  This is a high-end processor in a section of the game that is very GPU limited and thus we didn't expect to see a dramatic advantage for the Mantle API.  We did however expect to see SOME KIND of advantage…

Click to Enlarge

Click to Enlarge

If we look at the CPU time graph though from these two runs, it is clear that BF4's Mantle implementation is having an effect.  Notice how much more consistent the CPU time is on the Mantle test run without any kind of spikes of CPU usage.  Our Core i7-3960X is definitely being utilized more efficiently although, in this particular combination of hardware and level, it just doesn't result in any performance advantages.

« PreviousNext »