Battle of the Sixes, they call it

Comparing the gaming performance of the Apple iPhone 6 and the Samsung Galaxy S6 with a completely new methodology.

GameBench is a low-level application released in 2014 that attempts to bring the technical analysis and benchmarking capability of the PC to the mobile device. You might remember that I showed you some early results and discussed our use of the GameBench testing capability in my Dell Venue 8 7000 review a few months back; my understanding and practice of using the software was just beginning at that time and continues to grow as I spend time with the software.

The idea is simple yet powerful: GameBench allows Android users, and soon iOS users, the ability to monitor frame rates of nearly any game or 3D application that you can run on your phone or tablet to accurately measure real-world performance. This is similar to what we have done for years on the PC with FRAPS and allows us to gather average frames per second data over time. This is something that was previously unavailable to consumers or press for that matter and could be a very powerful tool for device to device comparisons going forward. The ability to utilize actual games and applications and gather benchmark data that is accurate to consumer experiences, rather than simply synthetic graphics tests that we have been forced to use in the past, will fundamentally change how we test and compare mobile hardware.

Image source: GameBench.net

Today, GameBench itself released a small report meant to showcase some of the kinds of data the software can gather while also revealing early support for Apple’s iPhone and iPad devices. Primary competitors for the comparison include the Apple iPhone 6, the Samsung Galaxy S6, HTC One M9 and Motorola Nexus 6.  I was able to get an early look at the report and offer some feedback, while sharing with our readers my views on the results.

GameBench tested those four devices in a total of 10 games:

  • Asphalt 8: Airborne
  • Real Racing 3
  • Dead Trigger 2
  • Kill Shot
  • Modern Combat 5: Blackout
  • Boom Beach
  • XCOM: Enemy Unknown
  • GTA: San Andreas
  • Marvel: Contest of Champions
  • Monument Valley

These games all vary in price and in play style, but they all are in the top 50 games lists for each platform and are known for their graphically intense settings and look.

Reported results from GameBench include a handful of interesting metrics starting with the median frame rate:

Our principle measurements are based on “frames per second” (fps). This includes the median fps, which represents the commonest frame rate experienced in the game and broadly correlates to what the player observes as graphical smoothness.

A strength of using the median is that it naturally overlooks fringe results, such as the 0fps or 60fps that commonly occur during a game’s menu or loading screens.  For this reason, the median fps is the primary metric upon which our rankings are based.

A weakness of the median, however, is that the commonest frame rate may not be very common in a game with erratic performance. So, we supplement the median fps with two secondary measures: the minimum fps, which might be observed as a temporary stutter during gameplay (even if this only occurs once and is barely noticeable); and also fps stability, which reflects how much of a game session was played at or around the median fps (within a 20 percent range of that median, to be precise).

These particular data points aren’t perfect and anyone that has followed our development of the Frame Rating system for PC gaming will know that average frame rates, medians and minimums, even combined, aren’t enough to give us the FULL breakdown of the user experience of a game. But they are a good start coming from nothing on Android and iOS previously.

Of the four initial devices tested, the iPhone 6 reports the highest median frame rate at 40 FPS using a geometric median across the entire bucket of 10 games listed above. It also has the highest minimum frame rate at 32 FPS. The Samsung Galaxy S6 follows with a median frame rate of 36 FPS and a minimum of 29 FPS. Frame rate stability is measured at 92% for the iPhone 6 and 84% for the S6, another indication that the iPhone 6 is providing the better overall mobile gaming experience.

Obviously, this test isn’t taking into account the dramatic specification differences between the devices. For example and likely most importantly, the iPhone 6 has a screen resolution of 1334×750 (1.0M pixels) while the Samsung Galaxy S6’s screen runs at 2560×1440 (3.68M pixels). For games that decide to run at the native resolution of the screen, the GPU portion of the SoC on the S6 has 3.68x more work to do for each frame, thus requiring more horsepower and more actual power to complete the task. In that light then, the 40 FPS vs. 36 FPS median frame rate difference shown in GameBench’s results above might indicate that the hardware of the S6 (Exynos 7420) is substantially more powerful than the Apple A8 found in the iPhone 6.

There are complications to the data of course though: some games don’t run at the native resolution on ANY phone or device and instead will render at a fixed resolution like 720p, then scale to the native screen size. Those games allow for a more direct and comparable benchmark result.

I’m not going to rehash each individual game benchmark that GameBench provides, so you should definitely head over to their story on the matter if you want all of the gory details. The author goes through all 10 tested games to reveal the individual cases of which phone performs better and on which platform.

But a couple of interesting points arise from the data. First, there is a huge concern over frame rate caps. Occurring in both Android and iOS versions of the game, but sometimes only on one operating system or the other, or even one phone or the other, frame rate caps inherently make it more difficult to compare performance of the games and platforms. For example, in Modern Combat 5 and Dead Trigger 2, the iPhone 6 is able to reach frame rates as high as 60 FPS while all three Android devices are limited and locked to a 30 FPS maximum. Even though the FPS stability metric aiming to tell us the overall smoothness of the experience shows the Galaxy S6 as the leader, the average frame rate of the S6 is less than half that of the iPhone 6. Frame rate limiting is likely due to one or more of several factors: lack of development effort, complications of a multi-device market and power conservation. Developers might be overwhelmed by the number of Android devices and SoC to tune for or it might be a desire to extend battery life on power hungry phones that are otherwise delivering a smooth user experience. Either way, it makes direct comparisons much more difficult.

Grand Theft Auto: San Andreas also shows us that when done right, developers are able to build a flexible and usable gaming experience for mobile users. GTA: SA allows for customization of graphical settings on the Android version of the game, setting a render resolution, draw distance, shadows and more. The iOS version used for the iPhone 6 does not offer these options. This closely mirrors the PC vs. console debates of gaming today as well – is it better to offer more flexibility to the gamer (PC and Android) to allow for specific settings while running the risk of a poor experience for those uninitiated to the task of setting them or is it best to dumb it down so that you know (consoles and iOS) that the frame rate will be smooth at the expensive of some image quality increases? Both scenarios have their advantages though it should be clear with my heavy PC background which direction I would like to see pursued.

Closing Thoughts

Image source: GameBench.net

Though we are early on with our own testing, GameBench is promising a revolution in the way we test and compare the gaming and graphics performance of mobile devices like smartphones and tablets. It is a welcome change – synthetic tests are fun to look at and to compare numbers but they were never very representative of what real-world experiences users see on their own devices. Synthetics still have a place to set expectations for hardware and to allow us to compare processor capabilities, but I am looking forward to adding GameBench to our standard suite of benchmarks for upcoming phone and tablet reviews. And for any readers that know gaming will be an important part of their mobile device experience, look for GameBench results to finally show you the truth about which hardware is right for you.