Feedback

A Quick Look at AMD Ryzen X470 NVMe Storage Performance

Subject: Storage
Manufacturer: AMD
Tagged: x470, StoreMI, raid, NVMe, amd

NVMe RAID and StoreMI

With Ken testing all of the new AMD X470 goodness that we had floating around the office here at PCPer, I snuck in some quick storage testing to get a look at just how the new platform handled a typical power user NVMe RAID configuration. We will be testing a few different platform configurations:

  • ASUS Z270 w/ 7700K
    • 1x SSD behind chipset (PCH)
    • 2x SSD (RAID-0) behind chipset (PCH)
    • 1x SSD directly connected to CPU
  • AMD X470 w/ 2600X
    • 1x SSD via RAIDXpert bottom driver
    • 2x SSD (RAID-0) via RAIDXpert
    • 1x SSD via MS InBox NVMe driver

For the AMD system we tested, all M.2 ports were direct connected to the CPU. This should be the case for most systems since the AMD chipset has only a PCIe 2.0 x4 link which would cut most NVMe SSD bandwidth in half if passed through it. The difference on AMD is that installing the RAIDXpert software also installs a 'bottom driver' which replaces the Windows NVMe driver, while Intel's RST platform handles this process more in the chipset hardware (but is limited to PCIe 3.0 x4 DMI bandwidth). Now onto the results:

Random Read IOPS

View Full Size

For random IO, we see expected scaling from AMD, but do note that IOPS comes in ~40% lower than the same configuration on Intel's platform. This is critical as much of the IO seen in general use is random reads at lower queue depths. We'd like to see AMD doing better here, especially in the case where a single SSD was operating without the interference of the RAIDXpert driver, which was better, but still not able to match Intel.

Random Read Latency

View Full Size

This latency chart should better explain the IOPS performance seen above. Note that the across the board latency increases by ~10us on the X470 platform, followed by another ~20us when switching to the RAIDXpert driver. That combined ~30us is 50% of the 60us QD1 latency seen the Z270 platform (regardless of configuration).

Sequential Read

View Full Size

Ok, now we see the AMD platform stretch its legs a bit. Since Intel NVMe RAID is bottlenecked by its DMI link while AMD has all NVMe SSDs directly connected to the CPU, AMD is able to trounce Intel on sequentials, but there is a catch. Note the solid red line, which means no RAIDXpert software. That line tracks as it should, leveling off horizontally at a maximum for that SSD. Now look at the two dashed red lines and note how they fall off at ~QD8/16. It appears the RAIDXpert driver is interfering and limiting the ultimate throughput possible. This was even the case for a single SSD passing through the RAIDXpert bottom driver (configured as a JBOD volume).

StoreMI

View Full Size

AMD has also launched their answer to Intel RST caching. StoreMI is actually a more flexible solution that offers some unique advantages over Intel. Instead of copying a section of HDD data to the SSD cache, StoreMI combines the total available storage space of both the HDD and SSD, and is able to seamlessly shuffle the more active data blocks to the SSD. StoreMI also offers more cache capacity than Intel - up to 512GB SSD caches are possible (60GB limit on Intel). Lastly, the user can opt to donate 2GB of RAM as an additional caching layer.

View Full Size

AMD claims the typical speedups that one would expect with an SSD caching a much slower HDD. We have done some testing with StoreMI and can confirm the above slide's claims. Actively used applications and games end up running at close to SSD speeds (after the first execution, which comes from the HDD). StoreMI is not yet in a final state, but that is expected within the next week or two. We will revisit that topic with hard data once we have the final shipping product on-hand.


April 19, 2018 | 02:52 PM - Posted by Dark_wizzie

How do I run a nvme ssd on CPU lanes on z270? I thought they were all PCH, or did you go with one of those cards where you slot the ssd in and the card into the PCIE slot?

Also, isn't the cash value of a SSD the performance at a given scenario, with SSD latency being a major but not the sole contributor to it? I'm trying to understand why I should care about latencies when I can just look at say, QD 1 random reads.

I guess I'll chock up AMD's poor performance here to immature drivers/etc?

Thanks

April 20, 2018 | 01:44 PM - Posted by Allyn Malventano

Used an adapter to confirm CPU-connected speeds. Agreed QD1 random read performance is the major contributor to client workloads. Latency is included because it is painting the rest of the performance picture, especially at QD>1. AMD performance seems to come down to the platform adding some latency and then the RAIDXpert driver stack adding some more.

April 19, 2018 | 06:51 PM - Posted by Jann5s

Did x470 improve wrt x370?

April 20, 2018 | 01:45 PM - Posted by Allyn Malventano

I'd call them similar. (meaning Z270 and Z370 are similar).

April 20, 2018 | 06:42 AM - Posted by PersistentCache (not verified)

Does the cache persist/stay hot after restart or shutdown? On ZFS not a problem, but on Windows With all the updates, if the cache resets on boot, what's the point? :)

April 20, 2018 | 01:40 PM - Posted by Allyn Malventano

NAND caches generally persist, but StoreMI definitely will since it acts more like hybrid / combined storage, with the only copy of active data resting on the SSD.

April 20, 2018 | 01:02 PM - Posted by msroadkill612

Confusing, but 7GB/s~ sequential read from the native amd raid is not bad at all.

April 20, 2018 | 08:21 PM - Posted by msroadkill612

"StoreMI combines the total available storage space of both the HDD and SSD, and is able to seamlessly shuffle the more active data blocks to the SSD. StoreMI also offers more cache capacity than Intel - up to 512GB SSD caches are possible (60GB limit on Intel). Lastly, the user can opt to donate 2GB of RAM as an additional caching layer."

It does sound rather fun.

A variant would be a classy small nvme ssd caching a ho hum larger sata ssd.

Intuitively, I would be leery of using it as a system disk, unless restore is straight forward in a a failure, which is far more likely in a pair AND includes a HDD.

Even nattier would be if the "~hdd" could be a raid array, cached by ssd. I have seen such utiliies included in marvel chip cheap sata controllers (~$50)

https://www.amazon.com/Vantec-4-Channel-HyperDuo-Technology-UGT-ST644R/p...

April 23, 2018 | 08:54 PM - Posted by Allyn Malventano

Restoring would definitely not be straightforward - it's essentially a RAID-0 (if either drive fails, recovery will be extremely difficult). The risk level assumed should consider that half of the array is a spinning disk.

April 20, 2018 | 11:45 PM - Posted by Havor (not verified)

I really don't understand, why are there no trace benchmarks, synthetic benchmarking is really not that useful to see how a storage solution will behave in the real world on a desktop.

And i don't see much adaptation of AM4 in the server market. ;-)

And whats even the point of QD8 and higher, even with boot-up you seldom see QD4, ware traces can give you real world senario's that are useful for home/game users to buy there storage solution.

April 21, 2018 | 10:28 AM - Posted by Neieus (not verified)

Um.. I can provide more links but a simple Google search will take care of that for you.

https://azure.microsoft.com/en-us/blog/announcing-the-lv2-series-vms-pow...

April 22, 2018 | 09:57 AM - Posted by Havor (not verified)

That's about Epyc socket SP3 motherboards sure those be used in the server market.

But this article and i was talking about socket AM4, that are two totally different markets.

And unlike Intel that has it's certified whit C2xx server chipset togeter whit S115x Xeon CPU's, AMD dose not have a server grade AM4 line.

April 23, 2018 | 09:02 PM - Posted by Allyn Malventano

Device latency impacts workloads in very different ways. The testing here was at the fundamental level and was done to evaluate the fundamentals of the compared technologies. Comparing application traces across two different platforms might give a benefit to one platform over another regardless of the storage performance specifically.

You're right that QD's typically remain very low, especially for random performance. Some sequential loads do stretch up to QD32 (if the application performing the transfer is capable), and the AMD system showed a very odd performance falloff that is atypical, preventing the tested SSDs from reaching their rated performance when coupled with the RAIDXpert NVMe driver. Either way, I typically show sequential out to QD32 and limit random to a lower value so that the scale is not thrown off too badly by the higher IOPS seen at those unrealistically high QD's.

April 23, 2018 | 11:17 PM - Posted by Havor (not verified)

Comparing application traces across two different platforms might give a benefit to one platform over another regardless of the storage performance specifically.

But in this case you are testing the impact of generation differences, that should imho also include the rest of platform.

I would have liked to also see the standard PCmark Storage test, to see what those latancy's result in in the real world.

April 24, 2018 | 12:01 AM - Posted by Allyn Malventano

The rest of the platform is covered in Ken's review. This portion was storage-specific.

The very reason that I have not relied on PCMark's storage tests for years is that it historically did a poor job of isolating the storage-specific performance of a platform. Results varied wildly when testing the same exact SSD across multiple platforms, and it did an extremely poor job of indicating stuttery performance (which is part of the reason I developed Latency Percentile testing). They tried to rectify that with a storage-specific 'extended test', but that test takes things too far, performing full-span random access and other enterprise-grade tests that are unrealistic and unfair to modern SSDs employing caching techniques. This is likely the reason that PCMark discontinued the extended storage test in version 10 of their software. PCMark 10's simpler tests are decent, but they do not adequately bring a given SSD to a steady state condition, introducing increased run-to-run variance as the state of the SSD is manipulated by the test itself. In this case, it's 'too quick' of a test. For the 'quick tests' run for this article in the interest of time, the SSDs being tested were brought to a steady state condition which was maintained across all test platforms. That's also the reason for the lack of write results here - writing alters the conditioned state of the SSD (and therefore the results).

If trace tests were considered standard, then SSDs would be rated by that metric. Instead, they are rated by 4KB random IOPS (sometimes with latencies) and 128KB sequential transfers, which is what I demonstrated in this article. The same SSDs were tested on both platforms. AMD was slower in random while faster in sequential with the caveat of the odd falloff reported. The lower random performance will present as lower results in traces focusing on random IO, while traces focusing more on lower QD sequential transfers will favor the AMD RAID setup.

April 21, 2018 | 01:35 AM - Posted by James

The 2 GB of RAM doesn’t seem like a good idea. The OS should be using any free RAM you have as disk cache anyway.

April 21, 2018 | 08:55 AM - Posted by msroadkill612

If the OS is memory starved. Otherwise, it would be nice to have the override option to prioritise storage latency and speed.

April 21, 2018 | 08:55 PM - Posted by James

If you completely push blocks out of disk cache, then that means that the resident set size has exceeded your available system memory. This means that you will be swapping. In that case, it would be much better to use that 2 GB for the active application rather than as disk cache. Re-loading from SSD is quite fast. I use mostly Linux, so I don’t know how good windows is at disk caches in comparison. Linux generally follows the idea that unused RAM is wasted RAM, so it is quite aggressive at caching disk accesses. Also, if the implementation doesn’t disable caching of blocks in that 2 GB, the OS could be keeping a separate copy in memory in the OS disk cache. There is already a good RAM caching system in place, so just use that.

April 22, 2018 | 04:49 PM - Posted by Paul A. Mitchell (not verified)

Thanks, Allyn!

BTW, the ASUS Hyper M.2 x16 AIC
should work in one of the X470 chipset's
x16 PCIe slots, yes? no? maybe?

https://www.newegg.com/Product/Product.aspx?Item=9SIA4UG6B68401

April 22, 2018 | 04:56 PM - Posted by Paul A. Mitchell (not verified)

UPDATE:
https://www.asus.com/us/Motherboard-Accessory/HYPER-M-2-X16-CARD/specifi...

Compatible Models:

ROG RAMPAGE VI EXTREME
ROG RAMPAGE VI APEX
ROG STRIX X299-XE GAMING
ROG STRIX X299-E GAMING
PRIME X299-DELUXE
PRIME X299-A
TUF X299 MARK 1
TUF X299 MARK 2

... no AMD chipsets listed above.

April 23, 2018 | 08:56 PM - Posted by Allyn Malventano

This might change, but there were no options for PCIe bifurcation on the systems we tested. This means you would only see the first SSD installed in the x16 card.

April 24, 2018 | 12:52 AM - Posted by Paul A. Mitchell (not verified)

Thanks, Allyn!

May 12, 2018 | 09:39 AM - Posted by msroadkill612

That is a great pity. I was hoping that 4 port nvme card would run as 4x4x4x4 on an 8 lane slot running x16 pcie gen 2 at 4x4x4x4.

"there were no options for PCIe bifurcation on the systems we tested. This means you would only see the first SSD installed in the x16 card."

I dont understand why this important bifurcation matter is rarely remarked on in reviews and so obscurely documented?

Please answer me this Allyn, Is there a ~"bios emulator" app, that lets one play with bios settings of a proposed mobo purchase e.g.?

manuals are usually useless/outdated.

May 12, 2018 | 10:55 AM - Posted by msroadkill612

So are you effectively saying one could put an nvme pcie gen 3 x4 m.2 adapter in an 8 lane am4 slot (which includes APUs) and have a second true nvme port on an am4?

IE, the AM4 slots you have seen have no bifurcation, but are compatible with a single 4 lane device and wasting the remaining lanes.

How could one raid these two nvme ports on am4?

April 22, 2018 | 05:23 PM - Posted by Paul in Queens (not verified)

Allyn,

I asked on one of the mailbag Youtube videos to compare AMD straight CPU m.2 vs. Intel latency, so thank you for this. It looks like without the AMD RAID drivers installed AMD's m.2 is pretty quick, but still lags the Intel solution somewhat.

Does using StoreMI/Fuzedrive require one to install the slower RAID drivers? I am asking because I currently have a SATA TLC Sandisk Ultra II 480 GB as my system drive and I don't know if I should pay the $20 to Fuzedrive (on x370) it with a 32 GB Optane which I also have already installed and am currently using as a dedicated swapfile drive.

What kind of speedup would one get when putting an Optane in front of a slower TLC SSD? I see this listed as one of the three scenarios which is used when discussing StoreMI technology, but the unhelpful marketing always cite the speedup numbers achieved when tiering just about any SSD with a spinning platter.

April 23, 2018 | 08:59 PM - Posted by Allyn Malventano

I haven't done super detailed testing on StoreMI just yet, but remember that it is a software driver/layer added on top of the existing OS install. Expect it to be similar to the bottom driver thing as the best case. In reality, it is probably slightly worse since there is more overhead involved with block layer splitting of data between fast and slow storage. It does still do its job though - performance is more like the SSD than it is like the HDD for data that has been shifted over to the faster storage tier.

April 23, 2018 | 09:01 PM - Posted by Paul A. Mitchell (not verified)

I'll look forward to Allyn's expert answer to your excellent question.

From my own experience, I can offer some theoretical predictions, which may partially answer your question.

For one, most high-grade SATA SSDs have already reach the limits imposed by the 6G clock and 8b/10b legacy frame.

To illustrate, you will be hard-pressed to find any SATA SSD that READs or WRITEs faster than 560 MB/second. 560/600 is only 6.67% aggregate controller overhead, which is superb.

All NVMe SSDs have the advantage of an 8G clock and the 128b/130b jumbo frame. Thus, 8G / 8.125 = 984.6 MB/second for a single x1 PCIe 3.0 lane.

Your Optane uses x2 PCIe 3.0 lanes: 984.6 x 2 = 1,969.2 MB/second max bandwidth.

What Allyn's expert analysis shows is that Optane SSDs have another advantage in their superbly low latency.

So, using those numbers as a baseline, for purposes of theoretical predictions, the Optane "cache" should make a big difference, particularly as that cache fills up with frequently accessed files.

And, along the same lines, we could use the same numbers to predict an even better result from a U.2 Optane that uses x4 PCIe 3.0 lanes; plus, an Intel model 900P U.2 Optane has a much larger capacity that M.2 Optanes:

https://www.newegg.com/Product/Product.aspx?Item=N82E16820167438&Tpk=N82...

April 23, 2018 | 09:40 PM - Posted by Paul A. Mitchell (not verified)

p.s. A really fun thought experiment is to compare the state-of-the-art with 12G SAS SSDs, with a focus on rated performance (and overlooking the much higher retail prices).

When doing so, remember that the SAS protocol still uses the 8b/10b legacy frame (10 bits per byte), and many SAS devices are dual-ported, so they are also capable of parallel operations which SATA devices can NOT perform.

Thus, if such a SAS device were single-ported, we would predict a theoretical max bandwidth of 12G / 10 = 1.2 MB/second (exactly TWICE the current 6G SATA-III ceiling).

This Toshiba is rated at 1,900 MiB/sec doing READs:

http://www.tomsitpro.com/articles/toshiba-px04s-sas-ssd,2-905.html

There is also a good review of the Toshiba PX04S here:

http://www.storagereview.com/toshiba_px04s_enterprise_ssd_review

I would certainly like to see the next "SATA-IV" standard support the 128b/130b jumbo frame, and variable transmission speeds -- with an 8G default, and perhaps options for 12G and also 16G that has been announced for the PCIe 4.0 specification.

NVMe was a big leap forward for many reasons, one of which is that chipsets now "sync" with storage channels.

At PCIe 4.0, we can predict a theoretical max of 16G / 8.125 = 1,939.4 MB/second per x1 PCIe 4.0 lane (exactly TWICE the max bandwidth of a single x1 PCIe 3.0 lane).

Thus, x4 PCIe 4.0 lanes @ 1,939.4 MB/sec = 7,876.8 MB/sec.

And, even if aggregate controller overhead were HIGH at, say, 25%, measured READ speed = 7,876.8 x .75 ~= 5,900 MB/second. That's VERY FAST!! I wouldn't be at all surprised if Samsung's R&D labs are approaching those speeds right now, as we speak, in anticipation of PCIe 4.0 chipsets and add-on controllers.

April 23, 2018 | 09:51 PM - Posted by Paul A. Mitchell (not verified)

clerical correction:

16G / 8.125 = 1,969.2 MB/second

Sorry for the typos above.

April 24, 2018 | 02:30 PM - Posted by Allyn Malventano

If you create a site login you can directly edit your own posts :)

May 12, 2018 | 10:29 AM - Posted by msroadkill612

I agree their documentation via ~variants on a single example are inane.

if its ok to pair your ssdS as discussed, i suspect is the optane will do a lot more good in storemi than as a little used? swap drive.

It intelligently divides your files over the two devices based on usage and space considerations, so maybe your swap files reside on the optane anyway.

$20 seems reasonable to make a ~uselessly small drive useful, and the sata seem like an nvme for most jobs.

I also like pairing ssdS, as it diminishes the extra risks of failure in an array, by pairing far more reliable solid state devices.

Since a key benefit is tiering, this makes the contrasting speeds and price of nvme and sata ssdS a natural fit as a pair - a 250GB nvme w/ a 500GB sata ssd e.g.

Install windows to the 500GB sata, add the 250GB nvme as an accelerator under Storemi.

If not happy, its simple to revert to the single 500GB install if less than 500GB of data on the array.

April 23, 2018 | 05:26 PM - Posted by Johan Steyn (not verified)

PC Per, I have a problem. I like you guys, but these days I do look at you with some paranoia. I will try to get around it, but since you do have vested interest in Intel, I keep on judging you. I keep on asking why do you test this, why do you choose this benchmark etc.

I think I might be a bit unfair towards you (not sarcastically meant) and judging you too hard. Maybe you can help me and tell me if you are still doing business with Intel on the side as before. It really bothers me and taints my view of your objectivity.

This article seems fair enough though.

April 23, 2018 | 09:34 PM - Posted by Allyn Malventano

That's a fair issue that applies to more review sites than you might think. We disclose if we are working with companies as part of our reviews - a higher level of disclosure than others in the industry (some of which do similar contract work for companies). We have worked with both Intel *and* AMD on projects. The way my storage testing works, the numbers are what they are, and can be obtained by anyone else who runs similar tests. All I can do is test the things and report on the findings. It's up to you to trust that we can do so without allowing ourselves to be biased. Further, if anyone we work with attempted to influence or otherwise interfere with our review results, we would not only ignore them, but we would likely no longer work with them. I have turned down reviews of products whose companies wanted editorial control over my review. On multiple occasions, I have given poor ratings to the products of companies who we had worked with, even some who were advertising with us at that time. That doesn't mean we were biased against them, it just means the product performed poorly.

That said, as an enthusiast, I want all PC hardware (especially SSDs) to be awesome, and I will tend to get overly excited and geek out over the fastest things. Don't mistake favoring the faster / better products as a bias, not in the context you are going for at least. If AMD were to release an amazing SSD that wiped the floor with everyone else, you can bet that I would not hesitate to tell everyone about it (and probably buy one for my own system). Now would I plug that super-fast AMD SSD into an X470 system? Judging from the above results, probably not, as I would be giving up a fair chunk of responsiveness by doing so.

April 28, 2018 | 02:18 AM - Posted by johnbot (not verified)

StoreMI sounds like a perfect solution. Tired of having to move stuff between my ssd ad hdd all the time.

April 30, 2018 | 06:26 AM - Posted by Erik Emanuelsson (not verified)

For StoreMI, is the capacity included in that license for the fast tier storage 512GB like the article here states or is it 256GB like the AMD FAQ states? (https://www.amd.com/system/files/2018-04/AMD-StoreMI-FAQ.pdf)

It may very well have changed over time and I usually trust the numbers I see here over at PCPer, especially when they come from Allyn. I am asking as I have ordered components for a X470 based system and the SSD in that is a M.2 960 EVO 500GB that I would like to use as fast tier in StoreMI for the PC.
Will the default StoreMI license let me use that fully or will I have to pay Enmotus for a FuzeDrive Plus license to do that?

May 12, 2018 | 10:36 AM - Posted by msroadkill612

No size limit for large slow drive.

$20 storemi for x370 = 120GB limit for fast drive, 256GB limit for included w/ x470 version.

excess capacity is not wasted. it shows as a separate drive.

May 3, 2018 | 09:22 AM - Posted by luckz (not verified)

Does the Intel test system have all of the meltdown and spectre updates that strongly cut down on NVMe performance?

May 24, 2018 | 03:35 PM - Posted by AMDAnonym (not verified)

If you have 2T SSD as a main drive, is there any reason to buy a smaller and faster M.2 Optane drive with the SSD and StoreMI.
Or is StoreMI only to get more speed from HDD?
There seems very little information on this combo.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Lines and paragraphs break automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockquote><p><br>
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.