NVMe RAID and StoreMI
With Ken testing all of the new AMD X470 goodness that we had floating around the office here at PCPer, I snuck in some quick storage testing to get a look at just how the new platform handled a typical power user NVMe RAID configuration. We will be testing a few different platform configurations:
- ASUS Z270 w/ 7700K
- 1x SSD behind chipset (PCH)
- 2x SSD (RAID-0) behind chipset (PCH)
- 1x SSD directly connected to CPU
- AMD X470 w/ 2600X
- 1x SSD via RAIDXpert bottom driver
- 2x SSD (RAID-0) via RAIDXpert
- 1x SSD via MS InBox NVMe driver
For the AMD system we tested, all M.2 ports were direct connected to the CPU. This should be the case for most systems since the AMD chipset has only a PCIe 2.0 x4 link which would cut most NVMe SSD bandwidth in half if passed through it. The difference on AMD is that installing the RAIDXpert software also installs a 'bottom driver' which replaces the Windows NVMe driver, while Intel's RST platform handles this process more in the chipset hardware (but is limited to PCIe 3.0 x4 DMI bandwidth). Now onto the results:
Random Read IOPS
For random IO, we see expected scaling from AMD, but do note that IOPS comes in ~40% lower than the same configuration on Intel's platform. This is critical as much of the IO seen in general use is random reads at lower queue depths. We'd like to see AMD doing better here, especially in the case where a single SSD was operating without the interference of the RAIDXpert driver, which was better, but still not able to match Intel.
Random Read Latency
This latency chart should better explain the IOPS performance seen above. Note that the across the board latency increases by ~10us on the X470 platform, followed by another ~20us when switching to the RAIDXpert driver. That combined ~30us is 50% of the 60us QD1 latency seen the Z270 platform (regardless of configuration).
Ok, now we see the AMD platform stretch its legs a bit. Since Intel NVMe RAID is bottlenecked by its DMI link while AMD has all NVMe SSDs directly connected to the CPU, AMD is able to trounce Intel on sequentials, but there is a catch. Note the solid red line, which means no RAIDXpert software. That line tracks as it should, leveling off horizontally at a maximum for that SSD. Now look at the two dashed red lines and note how they fall off at ~QD8/16. It appears the RAIDXpert driver is interfering and limiting the ultimate throughput possible. This was even the case for a single SSD passing through the RAIDXpert bottom driver (configured as a JBOD volume).
AMD has also launched their answer to Intel RST caching. StoreMI is actually a more flexible solution that offers some unique advantages over Intel. Instead of copying a section of HDD data to the SSD cache, StoreMI combines the total available storage space of both the HDD and SSD, and is able to seamlessly shuffle the more active data blocks to the SSD. StoreMI also offers more cache capacity than Intel - up to 512GB SSD caches are possible (60GB limit on Intel). Lastly, the user can opt to donate 2GB of RAM as an additional caching layer.
AMD claims the typical speedups that one would expect with an SSD caching a much slower HDD. We have done some testing with StoreMI and can confirm the above slide's claims. Actively used applications and games end up running at close to SSD speeds (after the first execution, which comes from the HDD). StoreMI is not yet in a final state, but that is expected within the next week or two. We will revisit that topic with hard data once we have the final shipping product on-hand.
We've been hearing about Intel's VROC (NVMe RAID) technology for a few months now. ASUS started slipping clues in with their X299 motherboard releases starting back in May. The idea was very exciting, as prior NVMe RAID implementations on Z170 and Z270 platforms were bottlenecked by the chipset's PCIe 3.0 x4 DMI link to the CPU, and they also had to trade away SATA ports for M.2 PCIe lanes in order to accomplish the feat. X99 motherboards supported SATA RAID and even sported four additional ports, but they were left out of NVMe bootable RAID altogether. It would be foolish of Intel to launch a successor to their higher end workstation-class platform without a feature available in two (soon to be three) generations of their consumer platform.
To get a grip on what VROC is all about, lets set up some context with a few slides:
First, we have a slide laying out what the acronyms mean:
- VROC = Virtual RAID on CPU
- VMD = Volume Management Device
What's a VMD you say?
...so the VMD is extra logic present on Intel Skylake-SP CPUs, which enables the processor to group up to 16 lanes of storage (4x4) into a single PCIe storage domain. There are three VMD controllers per CPU.
VROC is the next logical step, and takes things a bit further. While boot support is restricted to within a single VMD, PCIe switches can be added downstream to create a bootable RAID possibly exceeding 4 SSDs. So long as the array need not be bootable, VROC enables spanning across multiple VMDs and even across CPUs!
Assembling the Missing Pieces
Unlike prior Intel storage technology launches, the VROC launch has been piecemeal at best and contradictory at worst. We initially heard that VROC would only support Intel SSDs, but Intel later published a FAQ that stated 'selected third-party SSDs' would also be supported. One thing they have remained steadfast on is the requirement for a hardware key to unlock RAID-1 and RAID-5 modes - a seemingly silly requirement given their consumer chipset supports bootable RAID-0,1,5 without any key requirement (and VROC only supports one additional SSD over Z170/Z270/Z370, which can boot from 3-drive arrays).
On the 'piecemeal' topic, we need three things for VROC to work:
- BIOS support for enabling VMD Domains for select groups of PCIe lanes.
- Hardware for connecting a group of NVMe SSDs to that group of PCIe lanes.
- A driver for OS mounting and managing of the array.
Let's run down this list and see what is currently available:
Check. Hardware for connecting multiple drives to the configured set of lanes?
Check (960 PRO pic here). Note that the ASUS Hyper M.2 X16 Card will only work on motherboards supporting PCIe bifurcation, which allows the CPU to split PCIe lanes into subgroups without the need of a PLX chip. You can see two bifurcated modes in the above screenshot - one intended for VMD/VROC, while the other (data) selection enables bifurcation without enabling the VMD controller. This option presents the four SSDs to the OS without the need of any special driver.
With the above installed, and the slot configured for VROC in the BIOS, we are greeted by the expected disappointing result:
Now for that pesky driver. After a bit of digging around the dark corners of the internet:
Check! (well, that's what it looked like after I rapidly clicked my way through the array creation)
Don't even pretend like you won't read the rest of this review! (click here now!)
Subject: Storage | May 31, 2017 - 08:58 PM | Allyn Malventano
Tagged: x299, VROC, Virtual RAID on CPU, raid, Intel, asus
Ken and I have been refreshing our Google search results ever since seeing the term 'VROC' slipped into the ASUS press releases. Virtual RAID on CPU (VROC) is a Skylake-X specific optional feature that is a carryover from Intel's XEON parts employing RSTe to create a RAID without the need for the chipset to tie it all together.
Well, we finally saw an article pop up over at PCWorld, complete with a photo of the elusive Hyper M.2 X16 card:
The theory is that you will be able to use the 1, 2, or 3 M.2 slots of an ASUS X299 motherboard, presumably passing through the chipset (and bottlenecked by DMI), or you can shift the SSDs over to a Hyper M.2 X16 card and have four piped directly to the Skylake-X CPU. If you don't have your lanes all occupied by GPUs, you can even add additional cards to scale up to a max theoretical 20-way RAID-0 supporting a *very* theoretical 128GBps.
A couple of gotchas here:
- Only works with Skylake-X (not Kaby Lake-X)
- RAID-1 and RAID-5 are only possible with a dongle (seriously?)
- VROC is supposedly only bootable when using Intel SSDs (what?)
Ok, so the first one is understandable given Kaby Lake-X will only have 16 PCIe lanes direclty off of the CPU.
The second is, well, annoying, but understandable once you consider that some server builders may want to capitalize on the RSTe-type technology without having to purchase server hardware. It's still a significant annoyance, because how long has it been since anyone has had to deal with a freaking hardware dongle to unlock a feature on a consumer part. That said, most enthusiasts are probably fine with RAID-0 for their SSD volume, given they would be going purely for increased performance.
The third essentially makes this awesome tech dead on arrival. Requiring only Intel branded M.2 SSDs for VROC bootability is a nail in the coffin. Enthusiasts are not going to want to buy 4 or 8 (or more) middle of the road Intel SSDs (the only M.2 NAND SSD available from Intel is the 600p) for their crazy RAID - they are going to go with something faster, and if that can't boot, that's a major issue.
More to follow as we learn more. We'll keep a lookout and keep you posted as we get official word from Intel on VROC!
Subject: Motherboards | May 30, 2017 - 08:00 AM | Ken Addison
Tagged: x299, VROC, Strix X299-E, ROG, Rampage VI Extreme, Rampage VI Apex, raid, NVMe, LiveDash, Intel, computex 2017, asus, 802.11ad, 10G
Hot on the heels of Intel's Core i9 Skylake-X and Kaby Lake-X announcements today, ASUS has released details on its X299 offerings. While you can find details on the more Prime and TUF ASUS Motherboards here, we're taking a look at the flagship Republic of Gamers products in this post.
Today ASUS is taking the wraps off of 3 X299 ROG Motherboards, the Rampage VI Extreme, Rampage VI Apex, and Strix X299-E.
One of the interesting features ASUS is talking about with these X299 boards is Intel VROC technology. While we'll have a post with some more details about VROC soon, essentially it allows for a bootable M.2 NVMe RAID to exist from the CPU PCIe lanes.
While NVMe RAID is supported on the Z170 and Z270 platforms, it depends on all data going through the chipset to function which creates a bottleneck. Using an add-in card in the PCIe slot of your motherboard, VROC claims to allow NVME SSDs to operate in a RAID away from the chipset, while still being bootable.
Rampage VI Extreme
As we've historically seen with ASUS ROG Motherboards, the "Extreme" model tends to be where we see innovative new features that will later find their way into the rest of ASUS's motherboard lineup, and the Rampage VI Extreme seems to be no different.
Networking seems to be a big focus for the Rampage VI Extreme, with the addition of 802.11ad wireless networking. While 802.11ad devices are just starting to come out, it's use of 60GHz wireless in order to hit speeds of up to 4.6GBps is very compelling. However, by using such high-frequency wireless technology, 802.11ad signals will not penetrate surfaces like walls and mostly depend on line of sight. This is more for fast file transfers within one room, with the rest of your house still utilizing 802.11ac.
Wired networking sees an upgrade too on the Rampage VI Extreme, with the addition of a 10 Gigabit NIC. We saw ASUS integrate 10G networking on the X99-E WS 10G late last year, and it's great to see continued commitment to bringing 10G to consumers.
In addition to the VROC add-in card for NVMe SSDs that we mentioned previously, the Rampage VI Extreme features 3 onboard M.2 slots (2 slots come from the use of the DIMM.2 module in one of the memory slots). ASUS says this will help clean up your chassis while still giving you maximum storage options. It's unclear if these SSDs are being routed through the chipset, or are going directly to the CPU using Intel's VROC technology which would provide more throughput.
For users looking for a bit of flair on their motherboard, in addition to built-in RGB lighting, the Rampage VI Extreme features a new LiveDash OLED display for displaying real-time system information on your motherboard. You can also customize this display to offer custom messages and graphics to complement your case mod.
Rampage VI Apex
A newer addition to the ROG family, the Apex motherboards are meant for maximum overclocking and performance. This means that you'll actually sacrifice some features from other X299 boards in order to get a lean product you can push to the edge.
For instance, the Rampage VI Apex only provide 1 DIMM slot per memory channel, which ASUS claims allows the optimal trace routing to improve performance and stability with the fastest memory kits. In addition, you'll also find 2 DIMM.2 slots to add a total of 4 PCIe M.2 SSDs to your system.
While it may not be meant for gamers, the Rampage VI Apex is sure to set some records in the high-end overclocking realm.
Unlike the uncompromising motherboards we've talked about so far, the Strix X299-E aims to bring a more entry-level motherboard to the ROG line.
Keeping the same design cues as the other ROG motherboards, the Strix X299-E also keeps a lot of the same features. Users can expect the same PCIe slots and headers as other ROG boards. Additionally, features like onboard RGB lighting with Aura Sync Software, USB 3.1 Gen2, SupremeFX audio, and Intel Gigabit networking make the Strix X299-E a compelling product that should suit the needs of most users.
ROG Rampage VI Extreme, Rampage VI Apex and ROG Strix X299-E motherboards will be available at leading resellers in North America starting in late June with the STRIX series arriving first.
Subject: Motherboards | May 30, 2017 - 08:00 AM | Ken Addison
Tagged: x299, VROC, skylake-x, raid, NVMe, LiveDash, kaby lake-x, Intel, HEDT, computex 2017, asus, 802.11ad
Alongside the announcement of Intel's Core i9 Skylake-X and Kaby Lake-X CPUs, ASUS has unveiled details of their X299 motherboards. While we've already taken a look updates to the Republic of Gamers line of products, ASUS also released details about their "Prime" and "TUF" X299 products.
The Prime line of motherboards from ASUS are their entry-level options for any given platform. However just because they are the lowest cost boards doesn't mean that they are lacking in features.
All X299 Prime motherboards feature 3-way SLI thanks to the additional PCIe lanes available in the X299 platform (supported 44 lane CPUs are required). These available x16 slots are all reinforced with ASUS SafeSlot technology to help prevent heavy GPUs from damaging your motherboard.
The new Prime motherboards also feature the Realtek S1220A which we first saw on ASUS's Z270 products. ASUS claims that the S1220A paired with high-quality audio components on board help produce an onboard sound that rivals some dedicated sound cards.
Additionally, all X299 boards feature RGB LED headers with Aura Sync compatibility for maximum customizability.
For the storage-focused, Intel VROC technology found on ASUS's X299 boards will allow for high-speed M.2 NVMe SSD RAIDs without being bottlenecked by chipset bandwidth, unlike on the Z270 platform.
Though the use of an add-in card users will be able to RAID multiple M.2 SSDs into a bootable array, utilizing full bandwidth from the available CPU PCIe lanes. ASUS even says that you can link multiple of these VROC cards together!
In addition to these features, the Prime X299-Deluxe adds some very exciting features
Wireless networking has seen an enhancement with the adoption of integrated 802.11ad Wi-Fi. This brand new wireless technology capable of 4.6Gbps transfer speeds uses spectrum in the 60Ghz range. While these high-speed radio waves cannot penetrate surfaces like walls, for applications like super fast file transfer between PCs in one room, or high-quality video streaming to wireless displays this should be ideal.
Along with Intel VROC support, the Prime X299-Deluxe has dual onboard M.2 Slots and a single U.2 Slot for high-speed storage options.
The new LiveDash display is a small OLED panel on the motherboard that allows you to display system statistics as well as custom animations and text for additional customization.
In addition to the features on the motherboard, ASUS is including their ThunderboltEX 3 expansion card with the Prime X299-Deluxe so that owners can utilize Thunderbolt 3 technology with up to 40Gbps of bandwidth.
ASUS TUF X299
The TUF line has always been focused on ultimate reliability and durability. This is accomplished with high-quality components and more stringent testing standards than other products.
The ASUS TUF X299 Mark 1 is the all-new flagship motherboard for the TUF line. Redesigned Thermal Armor design helps to streamline airflow across the motherboard while also providing cooling to the onboard M.2 slot.
The Fortifier backplate uses a carefully shaped metal plate to stiffen the board to prevent warping. A removable GPU holder is also supplied to help support the weight of heavy graphics cards.
The new version of ASUS TUF Detective software allows users to perform diagnostics wirelessly over an included USB Bluetooth adapter rather than depending on a wired connection like the previous implementation.
The TUF X299 Mark 2 removes the Thermal Armor and Fortifier, but retains all of the great reliability and durability aspects of the TUF Mark 2,
ASUS Prime X299-Deluxe, Prime X299-A and TUF X299 Mark 1 motherboards will be available at leading resellers in North America starting in late June
Introduction and Packaging
Data Robotics shipped their first product 10 years ago. Dubbed the Drobo (short for Data Robot), it was a 4-bay hot-swappable USB 2.0 connected external storage device. At a time where RAID was still a term mostly unknown to typical PC users, the Drobo was already pushing the concept of data redundancy past what those familiar with RAID were used to. BeyondRAID offered a form of redundant data storage that decoupled rigid RAID structures from fixed capacity disk packs. While most RAID volumes were 'dumb', BeyondRAID was aware of what was stored within its partitions, distributing that data in block format across the available disks. This not only significantly speed up rebuilding (only used portions of the disks need be recopied), it allowed for other cool tricks like the ability to mix drive capacities within the same array. Switching between parity levels could also be done on-the-fly and with significantly less effort than traditional RAID migrations.
While all of the above was great, the original Drobo saw performance hits from its block level management, which was limited by the processing overhead combined with the available processing power for such a device at the time. The first Drobo model was lucky to break 15 MB/s, which could not even fully saturate a USB 2.0 link. After the launch, requests for network attached capability led to the launch of the DroboShare, which could act as a USB to ethernet bridge. It worked but was still limited by the link speed of the connected Drobo. A Drobo FS launched a few years later, but it was not much quicker. Three years after that we got the 5N, which was finally a worthy contender in the space.
10 years and nearly a dozen models later, we now have the Drobo 5N2, which will replace the aging 5N. The newer model retains the same 5-bay form factor and mSATA bay for optional SSD cache but adds a second bondable Gigabit Ethernet port and upgrades most of the internals. Faster hardware specs and newer more capable firmware enables increased throughput and volume sizes up to 64TB. Since BeyondRAID is thin provisioned, you always make the volume as large as it can be and simply add disk capacity as the amount of stored content grows over time.
Subject: Storage | March 8, 2017 - 09:58 PM | Allyn Malventano
Tagged: xeon, raid, NAS, iosafe, fireproof
The Server 5 is a completely different twist for an ioSafe NAS. While previous units have essentially been a fireproof drive cage surrounding Synology NAS hardware, the Server 5 is a full blown Xeon D-1520 or D-1521 quad core HT, 16GB of DDR4, an Areca ARC-1225-8i hardware RAID controller (though only 5 ports are connected to the fireproof drive cage). ioSafe supports the Server 5 with Windows Server 2012 R2 or you can throw your preferred flavor of Linux on there. The 8-thread CPU and 16GB of RAM mean that you can have plenty of other services running straight off of this unit. It's not a particularly speedy CPU, but keep in mind that the Areca RAID card offloads all parity calculations from the host.
Overall the Server 5 looks nearly identical to the ioSafe 1515+, but with an extra inch or two of height added to the bottom to accommodate the upgraded hardware. The Server 5 should prove to be a good way to keep local enterprise / business data protected and available immediately after a disaster. While only the hard drives will be protected in a fire, they can be popped out of the charred housing and shifted to a backup Server 5 or just migrated to another Areca-driven NAS system. For those wondering what a typical post-fire ioSafe looks like, here ya go:
Note how clean the cage and drives are (and yes, they all still work)!
Press blast appears after the break.
Subject: Storage | June 21, 2016 - 02:43 PM | Allyn Malventano
Tagged: usb 3.0, Thunderbolt 2, raid, hdd, drobo, DAS, BeyondRAID, 5Dt, 5D
Today Drobo updated their 5D, shifting to Thunderbolt 2, an included mSATA caching SSD, and faster internals:
The new 5Dt (t for Turbo Edition) builds on the strengths of the 5D, which launched three years ago. The distinguishing features remain the same, as this is still a 5-bay model with USB 3.0, but the processor has been upgraded, as well as the USB 3.0 chipset, which was a bit finicky with some earlier implementations of the technology.
The changes present themselves at the rear, as we now have a pair of Thunderbolt 2 (20 Gb/s) ports which support display pass-through (up to 4k). Rates speeds climb to 540 MB/s read and 250 MB/s write when using HDDs. SSDs bump those figures up to 545 / 285 MB/s, respectively.
Another feature that has remained was their Hot Data Cache technology, but while the mSATA part was optional on the 5D, a 128GB unit comes standard and pre-installed on the 5Dt.
The Drobo 5Dt is available today starting at $899. That price is a premium over the 5D, but the increased performance specs, included SSD, and Thunderbolt connectivity come at a price.
The current (updated) Drobo product lineup.
Full press blast after the break.
NVMe was a great thing to happen to SSDs. The per-IO reduction in latency and CPU overhead was more than welcome, as PCIe SSDs were previously using the antiquated AHCI protocol, which was a carryover from the SATA HDD days. With NVMe came additional required support in Operating Systems and UEFI BIOS implementations. We did some crazy experiments with arrays of these new devices, but we were initially limited by the lack of native hardware-level RAID support to tie multiple PCIe devices together. The launch of the Z170 chipset saw a remedy to this, by including the ability to tie as many as three PCIe SSDs behind a chipset-configured array. The recent C600 server chipset also saw the addition of RSTe capability, expanding this functionality to enterprise devices like the Intel SSD P3608, which was actually a pair of SSDs on a single PCB.
Most Z170 motherboards have come with one or two M.2 slots, meaning that enthusiasts wanting to employ the 3x PCIe RAID made possible by this new chipset would have to get creative with the use of interposer / adapter boards (or use a combination of PCI and U.2 connected Intel SSD 750s). With the Samsung 950 Pro available, as well as the slew of other M.2 SSDs we saw at CES 2016, it’s safe to say that U.2 is going to push back into the enterprise sector, leaving M.2 as the choice for consumer motherboards moving forward. It was therefore only a matter of time before a triple-M.2 motherboard was launched, and that just recently happened - Behold the Gigabyte Z170X-SOC Force!
This new motherboard sits at the high end of Gigabyte’s lineup, with a water-capable VRM cooler and other premium features. We will be passing this board onto Morry for a full review, but this piece will be focusing on one section in particular:
I have to hand it to Gigabyte for this functional and elegant design choice. The space between the required four full length PCIe slots makes it look like it was chosen to fit M.2 SSDs in-between them. I should also note that it would be possible to use three U.2 adapters linked to three U.2 Intel SSD 750s, but native M.2 devices makes for a significantly more compact and consumer friendly package.
With the test system set up, let’s get right into it, shall we?
A quick look at storage
** This piece has been updated to reflect changes since first posting. See page two for PCIe RAID results! **
Our Intel Skylake launch coverage is intense! Make sure you hit up all the stories and videos that are interesting for you!
- The Intel Core i7-6700K Review - Skylake First for Enthusiasts (Video)
- Skylake vs. Sandy Bridge: Discrete GPU Showdown (Video)
- ASUS Z170-A Motherboard Preview
- Intel Skylake / Z170 Rapid Storage Technology Tested - PCIe and SATA RAID
When I saw the small amount of press information provided with the launch of Intel Skylake, I was both surprised and impressed. The new Z170 chipset was going to have an upgraded DMI link, nearly doubling throughput. DMI has, for a long time, been suspected as the reason Intel SATA controllers have pegged at ~1.8 GB/sec, which limits the effectiveness of a RAID with more than 3 SSDs. Improved DMI throughput could enable the possibility of a 6-SSD RAID-0 that exceeds 3GB/sec, which would compete with PCIe SSDs.
Speaking of PCIe SSDs, that’s the other big addition to Z170. Intel’s Rapid Storage Technology was going to be expanded to include PCIe (even NVMe) SSDs, with the caveat that they must be physically connected to PCIe lanes falling under the DMI-connected chipset. This is not as big of as issue as you might think, as Skylake does not have 28 or 40 PCIe lanes as seen with X99 solutions. Z170 motherboards only have to route 16 PCIe lanes from the CPU to either two (8x8) or three (8x4x4) PCIe slots, and the remaining slots must all hang off of the chipset. This includes the PCIe portion of M.2 and SATA Express devices.