Well here we are again with this Samsung 840 EVO slow down issue cropping up here, there, and everywhere. The story for this one is so long and convoluted that I’m just going to kick this piece off with a walk through of what was happening with this particular SSD, and what was attempted so far to fix it:
The Samsung 840 EVO is a consumer-focused TLC SSD. Normally TLC SSDs suffer from reduced write speeds when compared to their MLC counterparts, as writing operations take longer for TLC than for MLC (SLC is even faster). Samsung introduced a novel way of speeding things up with their TurboWrite caching method, which adds a fast SLC buffer alongside the slower flash. This buffer is several GB in size, and helps the 840 EVO maintain fast write speeds in most typical usage scenarios, but the issue with the 840 EVO is not its write speed – the problem is read speed. Initial reviews did not catch this issue as it only impacted data that had been stagnant for a period of roughly 6-8 weeks. As files aged their read speeds were reduced, starting from the speedy (and expected) 500 MB/sec and ultimately reaching a worst case speed of 50-100 MB/sec:
There were other variables that impacted the end result, which further complicated the flurry of reports coming in from seemingly everywhere. The slow speeds turned out to be the result of the SSD controller working extra hard to apply error correction to the data coming in from flash that was (reportedly) miscalibrated at the factory. This miscalibration caused the EVO to incorrectly adapt to cell voltage drifts over time (an effect that occurs in all flash-based storage – TLC being the most sensitive). Ambient temperature could even impact the slower read speeds as the controller was working outside of its expected load envelope and thermally throttled itself when faced with bulk amounts of error correction.
An example of file read speed slowing relative to age, thanks to a tool developed by Techie007.
Once the community reached sufficient critical mass to get Samsung’s attention, they issued a few statements and ultimately pushed out a combination firmware and tool to fix EVO’s that were seeing this issue. The 840 EVO Performance Restoration Tool was released just under two months after the original thread on the Overclock.net forums was started. Despite a quick update a few weeks later, that was not a bad turnaround considering Intel took three months to correct a firmware issue of one of their own early SSDs. While the Intel patch restored full performance to their X25-M, the Samsung update does not appear to be faring so well now that users have logged a few additional months after applying their fix.
Subject: Storage | January 26, 2015 - 10:27 PM | Jeremy Hellstrom
Tagged: Samsung, firmware, EVO, 840 evo
In the fall it was confirmed by Samsung that stale data on some 840 EVO drives would suffer performance degradation and released a tool to mitigate the issue which Al reviewed here. The Tech Report recently heard of some cases of drives slowing even with the new EXT0CB6Q firmware installed and decided to investigate. They took a 840 EVO 250GB SSD which had been filled with files to test the patch and was then left forgotten on a shelf for several months and tested the speeds. The benchmarks showed an average speed between 35-54MB/s far below what you would expect to see from an SSD but in line with what users have been reported. On the other hand another 840 EVO which has been in constant use since the firmware update shows no signs whatsoever of slowing down, though NTFS compression was recently used on the drive which could have refreshed the flash. Obviously more testing needs to be done, keep your eyes out for updates on this new development.
"In October, Samsung patched its 840 EVO SSD to address a problem that caused slow read speeds with old data. Recent reports suggest the issue isn't completely fixed, and the results of our own testing agree."
Here are some more Storage reviews from around the web:
- Plextor M6e Black Edition SSD @ The SSD Review
- OCZ Challenge update: All 5 ARC 100 SSD’s hit 200TB mark @ Kitgru
- Samsung's Portable SSD T1 @ The Tech Report
- Samsung Portable SSD T1 500GB @ Custom PC Review
- Samsung T1 SSD @ HardwareHeaven
- Samsung Portable SSD T1 @ The SSD Review
- Kingston SDCA3 microSDHC/SDXC UHS-I U3 Card @ The SSD Review
- Synology DiskStation DS215j 2-Bay NAS @ eTeknix
** Edit **
The tool is now available for download from Samsung here. Another note is that they intend to release an ISO / DOS version of the tool at the end of the month (for Lunix and Mac users). We assume this would be a file system agnostic version of the tool, which would either update all flash or wipe the drive. We suspect it would be the former.
** End edit **
As some of you may have been tracking, there was an issue with Samsung 840 EVO SSDs where ‘stale’ data (data which had not been touched for some period of time after writing it) saw slower read speeds as time since written extended beyond a period of weeks or months. The rough effect was that the read speed of old data would begin to slow roughly one month after written, and after a few more months would eventually reach a speed of ~50-100 MB/sec, varying slightly with room temperature. Speeds would plateau at this low figure, and more importantly, even at this slow speed, no users reported lost data while this effect was taking place.
An example of file read speeds slowing relative to file age.
Since we first published on this, we have been coordinating with Samsung to learn the root causes of this issue, how they will be fixed, and we have most recently been testing a pre-release version of the fix for this issue. First let's look at the newest statement from Samsung:
Because of an error in the flash management software algorithm in the 840 EVO, a drop in performance occurs on data stored for a long period of time AND has been written only once. SSDs usually calibrate changes in the statuses of cells over time via the flash management software algorithm. Due to the error in the software algorithm, the 840 EVO performed read-retry processes aggressively, resulting in a drop in overall read performance. This only occurs if the data was kept in its initial cell without changing, and there are no symptoms of reduced read performance if the data was subsequently migrated from those cells or overwritten. In other words, as the SSD is used more and more over time, the performance decrease disappears naturally. For those who want to solve the issue quickly, this software restores the read performance by rewriting the old data. The time taken to complete the procedure depends on the amount of data stored.
This partially confirms my initial theory in that the slow down was related to cell voltage drift over time. Here's what that looks like:
As you can see above, cell voltages will shift to the left over time. The above example is for MLC. TLC in the EVO will have not 4 but 8 divisions, meaning even smaller voltage shifts might cause the apparent flipping of bits when a read is attempted. An important point here is that all flash does this - the key is to correct for it, and that correction is what was not happening with the EVO. The correction is quite simple really. If the controller sees errors during reading, it follows a procedure that in part adapts to and adjusts for cell drift by adjusting the voltage thresholds for how the bits are interpreted. With the thresholds adapted properly, the SSD can then read at full speed and without the need for error correction. This process was broken in the EVO, and that adaptation was not taking place, forcing the controller to perform error correction on *all* data once those voltages had drifted near their default thresholds. This slowed the read speed tremendously. Below is a worst case example:
We are happy to say that there is a fix, and while it won't be public until
some time tomorrow now, we have been green lighted by Samsung to publish our findings.