A philosophical look at SSD benchmarking

Subject: General Tech | April 14, 2015 - 01:42 PM |
Tagged: ssd, benchmarking, synthetic

[H]ard|OCP will be resuming their benchmarking of SSDs in the near future and wanted to introduce both their new contributor and his thoughts on benchmarking SSDs.  These drives offer several challenges when comparing performance that are not present when benchmarking spinning rust.  For instance some controllers use compression to increase IOPS whenever possible but slow down when incompressible data is passed through the drive, providing a challenge to properly show performance comparisons to similar drives with difference or no compression whatsoever.  Read through the article to see which synthetic benchmarks will remain as well as Chris' thoughts on new tests to accurately contrast the performance of SSDs.

View Full Size

"Many of our readers embrace our "real world" approach with hardware reviews. We have not published an SSD review for almost 2 years while we have been looking to revamp our SSD evaluation program. Today we wanted to give you some insight as to how we learned to stop worrying and love the real world SSD benchmark."

Here are some more Storage reviews from around the web:

Storage

Source: [H]ard|OCP

Video News


April 14, 2015 | 05:12 PM - Posted by jessterman21 (not verified)

Totally respect this approach. Benching SATA3 SSDs lately has become completely academic. However if you take these results at face-value and grab the cheapest SSD thinking it will perform the same as an 850 Pro, you'll be disappointed in the low random read/writes and high latency.

April 14, 2015 | 09:22 PM - Posted by Mark D (not verified)

I hope PCPer adopts a similar methodology. If GPUs were tested like SSDs are now, we'd get a bunch of synthetic tests that tell us things like pixel fillrate, memory bandwidth, triangles/sec, etc...and have no way whatsoever to translate that into results that matter. No one tests GPUs like that anymore because it's pointless, nowadays everyone has settled on avg/min FPS and frame consistency tests because that much better reflects the user experience.

Ideally we'd have some measure of how the SSD performs in real world situations, not even just crazy I/O bound situations. I want to know whether game load times decrease, whether importing a bunch of pictures, compiling code, etc...are actually faster with one SSD vs. another. Whether I can run a full drive virus scan, backup or game/app install faster. And in the terms that matter - not MB/sec or IOPS, but how many seconds it takes to complete the test/trace. Not access times in microseconds, but actual system responsiveness (measured in ms I suppose) under heavy load.

If the truth is that despite vast difference in specs, SSDs are indistinguishable in real world usage - we should be honest about that. If these stupid fast nvme drives offer nothing outside of the datacenter, then they should be cast aside like the quadros and xeons and keep the focus on parts that really matter to consumers.

April 14, 2015 | 10:30 PM - Posted by Anonymous (not verified)

I'm still on spinning rust, and will wait until laptops start coming with PCIe based slots for laptop SSDs, no more SATA for flash drives. And forget about getting some SSD testing done on laptops, it's always going to be on a specialized rig with plenty of PCIe lanes, and dedicated controllers slotted into ample PCIe slots. Get a few standard laptop builds based on different laptop form factors from ultrabooks to gaming laptops, and run some SSD benchmarks, if even to find out if most laptops are even capable of providing the bandwidth to take advantage of the newer SSD's top speeds.

I could care less about a few seconds of faster boot times, what I need is the ability to handle some Blender 3d workloads, and not be as susceptible to disk thrashing like on regular hard drives, load some monster 3d scenes in Blender 3d and max out the computers RAM memory and then see how the SSD drive handles the disk paging calls as the virtual memory begins to generate the page faults. Those high polygon mesh count scenes can eat up 8 gigs of RAM, and start taxing the page swap file pretty damn quick, when editing a large scene with millions of polygons. I can get my laptop thrashing in no time with the hard drive struggling to handle the swap file/page faults with a regular hard drive. This is a real world benchmark, if the system struggles handling virtual page swaps/page faults even with an SSD then that product may not be the one to choose. I'm sure there are synthetic benchmarks for page swaps/page fault handling, but still some real world testing with actual applications is always good to go along with the synthetic benchmarks.

April 15, 2015 | 04:12 PM - Posted by Anonymous (not verified)

Thrashing is caused by main memory being too small for the application working set, an SSD improves the situation, but is still much slower than RAM. Evicting small dirty 4KiB pages is also liable to lead to write amplfication due to the mismatch of memory and SSD page sizes.

A misconfigured system is not really a great real world benchmark at all. You'ld be optomising for something sane users would not hit.

Finally a 16GiB, laptop with an SSD and a HDD is not exactly a huge amount more expensive, any improvements over SATA with PCIe are marginal, compared to switching over to any kind of SSD for the main OS & Applications drive. You're waiting for no sensible reason!!

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Lines and paragraphs break automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockquote><p><br>
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.