Basemark X Creator Comments on Benchmark Cheating

Subject: Mobile | October 4, 2013 - 12:27 AM |
Tagged: Android, cheating, basemark x

Even if you haven't been paying attention to the world of mobile benchmarking over the past week you have likely heard about the now rampant cheating that is going on with Android testing.  Device makers are doing simple detection for benchmark applications and unrealistically changing the performance attributes of the SoC (CPU and GPU) to improve benchmark scores.  This does not represent the behavior that an end user would see in real-world usage but is intended only to move the device up to the top of benchmark graphs to gain attention and drive sales.

Long time PC enthusiasts will recognize this problem though thanks to the openness of the PC ecosystem that issue is largely removed as there are independent press and researchers keeping all parties honest. 

Anandtech (and many other outlets) are again discussing the issue of cheating in mobile testing, even going as far as creating a chart titled "I Can't Believe I Have to Make This Table" that shows which benchmarks are being compromised by which devices and OS configurations.  I highly suggest you check out the story by Anand and Brian to get more details on the state of cheating in mobile benchmarks. 

The creator of one of the affected benchmarks, Basemark X, contacted the media with some interesting comments I wanted share. 

It has come to our attention that Galaxy Note 3 may be targeting our benchmark, Basemark X.

Rightware’s mission is to provide trusted performance evaluation tools you can depend on. Therefore, we have produced an updated version of Basemark X that solves this issue.

basemarkx.png

I asked Tero Sarkkinen, founder of Rightware, what could be done to prevent this type of unfair performance skewing going forward. 

Basically every benchmark and application out there can be targeted by a new handset or tablet and no one can really prevent it. What makes a difference is will the benchmark vendor do something about it when this is recognized.

At Rightware, we take our mission seriously and we monitor day in and day out what is going on. As in this case, we noticed that Note 3 is targeting Basemark X, we immediately provided the press with a version that the handset is not able to detect.

We get thousands of benchmark results in every day to our Power Board http://results.rightware.com and therefore we have a pretty good idea of what's going on.

In other words, we are not sticking our heads into the sand.

While the sentiment that "no one can really prevent it" is disappointing to hear, it is what we expected and what we are planning for.  Sarkkinen is confident that Rightware is able to stay up on the situation and is going to keep pace with online media and analysts to make sure these hardware vendors are staying honest. 

It's the best news we have seen in a sea of disappointing information on mobility benchmarking this week.