Home,
Cost-Based Optimizer,
Benchmarks,
Server Systems,
Systems Architecture,
Processors,
Storage,
Storage Overview,
System View of Storage,
SQL Server View of Storage,
File Layout,
PCI-E, SAS, FC,
HDD,
SSD Technology,
RAID Controllers, Direct-Attach,
SAN, Dell MD3200,
EMC AX4, CX4, VNX,
V-Max,
HP P2000, EVA, P9000/VSP,
Hitachi AMS
SSD products: SATA/SAS SSDs, PCI-E SSDs,
Fusion iO, other SSD
PCI-E SSD Products Today (Fusion iO covered separately)
Below is a quick survey of SSD either currently available or expected in the near-term.
Intel DC P3(5,6,7)00 P320h PCI-E gen 3 SSDsUpdate 2014-06
Tom's itpro 2014 Jun 02 has this article Intel Intros New Data Center PCIe SSD Family and AnandTech 2014 Jun 03 Intel SSD DC P3700 Review: The PCIe SSD Transition Begins with NVMe. The DC P3500, 3600 and 3700 PCI-E SSDs are gen 3 with x4 width, and are based on the new NVMe standard.
| Model | DC P3500 | DC P3600 | DC P3700 |
|---|---|---|---|
| Interface | PCI-E g3 x4 | PCI-E g3 x4 | PCI-E g3 x4 |
| Form Factors | HHHL PCI-E & 2.5in SFF-8639 |
HHHL PCI-E & 2.5in SFF-8639 |
HHHL PCI-E & 2.5in SFF-8639 |
| Capacities | 400GB, 1.2TB, & 2.0TB |
400GB, 800GB, & 1.2TB |
400GB, 800GB, 1.6TB & 2.0TB |
| Raw Capacity | ? | ? | ? |
| Write Endurance | up to 1 PBW 0.3 DWPD |
up to 10.95 PBW 3 DWPD | up to 36.5 PBW 10 DWPD |
| 64K Read/Write (MB/s) | up to 2500/1700 | up to 2600/1700 | up to 2800/1900 |
| 4K Read/Write | up to 450K/35K | up to 450K/70K | up to 460K/160K |
| 4K 70/30 Mixed R/W | up to 85K | up to 170K | up to 250K |
| Pricing | $1.5/GB $599+ | $2/GB $783+ | $3/GB $1207+ |
| Intel DC P3700 | 400GB | 800GB | 1.6TB | 2.0TB |
|---|---|---|---|---|
| Sequential Read | 2,700 | 2,800 | 2,800 | 2,800 |
| Sequential Write | 1,200 | 1,900 | 1,900 | 1,900 |
| Random Read 4KB | 450,000 | 450,000 | 450,000 | 450,000 |
| Random Write 4KB | 75,000 | 90,000 | 150,000 | 175,000 |
| 4KB Read/Write 70/30 | 150,000 | 200,000 | 240,000 | 250,000 |
| Random Read 8KB | 260,000 | 280,000 | 280,000 | 280,000 |
| Random Write 8KB | 32,000 | 45,000 | 75,000 | 90,000 |
All performance specifications are "up to"
| Intel DC P3600 | 400GB | 800GB | 1.2TB | 1.6TB | 2.0TB |
|---|---|---|---|---|---|
| Sequential Read | 2,100 | 2,600 | 2,600 | 2,600 | 2,600 |
| Sequential Write | 550 | 1,000 | 1,250 | 1,600 | 1,600 |
| Random Read 4KB | 320,000 | 430,000 | 450,000 | 450,000 | 450,000 |
| Random Write 4KB | 30,000 | 50,000 | 50,000 | 56,000 | 56,000 |
| 4KB Read/Write 70/30 | 80,000 | 110,000 | 130,000 | 160,000 | 160,000 |
| Random Read 8KB | 180,000 | 240,000 | 260,000 | 260,000 | 260,000 |
| Random Write 8KB | 19,000 | 26,000 | 27,000 | 33,000 | 35,000 |
All performance specifications are "up to"
The AnandTech article cites 20µs average read/write latency. The P3700 uses HET MLC NAND. Presumably the P3600 and P3500 use ordinary MLC, with the difference in write endurance due to the degree of over-provisioning? The P3500, P3600 and P3700 over-provisioning is cited as 25%, 12 % and 7% respectively. AnandTech cites pricing as $3/GB, $1.95/GB and 1.495/GB respectively. Since we are not used car salesmen, I will cite this a $3, 2 and 1.5/GB. DRAM capacities range from 512M to 2.5GB.
As bandwidth tops out at 2.8GB/s, I am thinking that there are 8 or 9 NAND channels at 333MB/s on the back-end?
NVM Express (NVMe)
"NVM Express (NVMe) is register level interface that allows host software to communicate with a non-volatile memory subsystem. This Interface ... typically attached to the PCI Express interface."
Some objectives of NVMe are 1) allow OS boot from a PCI-E device (as opposed to SATA) and 2) completely re-architect the driver for operation at extremely high IOPS, far beyond anything considered for HDD storage arrays with emphasis on code-path efficiency and parallelism. Other capabilities are commands to match the nature of NAND, as previous interfaces were designed around hard disk storage.
NVMe and the other new storage initiatives do not take a hard position on the NAND controller interface, with plans to accommodate co-existence of both SATA/SAS and direct PCIE.
Micron P320h Update 2012-10
See the AnandTech Micron P320h PCIe SSD (700GB) Review review for details. Also see the ssd review micron-p320h-hhhl-700gb-pcie-enterprise-ssd-review. The Micron P320h is SLC. The pricing from CDW is about $15K per TB. This is definitely an interesting product.
| P320h | P320h | |
|---|---|---|
| Capacities (GB) | 350 | 700 |
| Raw Capacity (GB) | 512 | 1024 |
| Write Endurance (PB) | 25 | 50 |
| Max Read | 3,200MB/s | 3,200MB/s |
| Max Write | 1,900MB/s | 1,900MB/s |
| Random Read 4K | 785K IOPS | 785K IOPS |
| Random Write 4K | 205K IOPS | 205K IOPS |
| Pricing | $5,578 | $11,155 |
It is too bad Micron does not have an MLC product. For a transaction processing database, we might isolate a hot table to fit well with SLC pricing. However with MLC prices approaching $5K per TB for enterprise (or just better than consumer) grade products, it is feasible to put the entire database (except for archival tables) on SSD. In this case, the SLC write endurance is not necessary.
Below is the Micron controller, interfacing PCI-E directly to the NAND interface. There are 32 channels on the NAND side. Of course, a SATA SSD can have 8 NAND channels. So 4 SATA SSDs would have 32 NAND channels. The difference is that ONFI 2.0 allows up to 133MB/s and 2.1 allows 166 or 200MB/s per NAND channel, so 8 channels is more than the bandwidth of a single 6Gbps SAS lane (600MB/s net). The real benefit of interfacing directly from PCI-E to NAND is that this avoids the extra bandwidth matching step.
Using the figure below and the 350GB net capacity of the smaller model, the 0.875 factor for RAIN (RAI on NAND instead of disks) implies 400GB capacity prior to RAIN. The 0.78 factor for over-provisioning implies a raw capacity of 512GB. This would imply that the 350GB net capacity is binary, not decimal?
Micron also has a PCI-E SDD in 2.5in HDD form factor at 175 and 350GB.
| Capacities (GB) | 175 | 350 |
| Write Endurance (PB) | 25 | 50 |
| Max Read | 1,750MB/s | 1,750MB/s |
| Max Write | 1,100MB/s | 1,100MB/s |
| Random Read 4K | 415K IOPS | 415K IOPS |
| Random Write 4K | 145K IOPS | 145K IOPS |
| Pricing | ? | ? |
Intel 910 Series
The Intel 910 Series finally came out earlier this year. The 910 uses an LSI PCI-E to SAS controller followed by SAS to NAND controllers.
| 910 | 910 | |
|---|---|---|
| Capacities (GB) | 400 | 800 |
| Raw Capacity (GB) | 768? | 1536? |
| Write Endurance (PB) | 5-7 | 10-14 |
| Controllers | ? | ? |
| Max Read | 1,000MB/s | 2,000MB/s |
| Max Write | 750MB/s | 1,000MB/s |
| Random Read 4K | 90K IOPS | 180K IOPS |
| Random Write 4K | 38K IOPS | 75K IOPS |
| Pricing | $2,000 | $4,000 |
Some technology enthusiasts like to talk about the theoretical "advantages" of not being encumbered by the extra interface transition through SAS. I would point out that the LSI SAS controller is a very mature server product. Too many product vendors cite impressive performance numbers for their PCI-E SSD on the basis of a single adapter in the system.
In the server environment, performance from a single card is only one of several metrics. It is also critically important that the device have very low overhead. The objective is for the application (the SQL Server database engine in our case) to run with as little disruption as possible. The complete server system will have several PCI-E SSD devices. If the IO performance does not scale over multiple devices, then it is of limited value in servers. LSI has deep experience on server side as LSI SAS controllers are used in almost server systems. They are using the controller of choice for TPC benchmarks with proven ability to scale.
The NAND is 25nm with High Endurance Technology (HET). Write endurance is 2.5PB per 200GB module at 4KB and 3.5PB at 8KB write IO. This is 30X over the consumer SSDs?
Read and write latency is quoted as < 65us. The 800GB model can support 1500MB/s large block write if higher power is available. The architecture of the 800GB is shown below (from Intel Solid-State Drive 910 Series Product Specification).
The PCI-E bridge chip has an SAS interface on the backend. There are 4 SAS channels to a SAS/NAND ASIC. Each NAND module is 200GB. So the 400GB module has 2 SAS channels? It would seem that the Intel 910 uses half of the x4 SAS at 400GB and a full x4 SAS at 800GB?
Tom's hardware Intel SSD 910 Review: PCI Express-Based Enterprise Storage and Intel SSD 710 Tested: MLC NAND Flash Hits The Enterprise
LSI Nytro WarpDrive
There are several sub-brands of the LSI Nytro WarpDrive.
The MegaRAID Application Acceleration Cards has NAND flash plus software to cache hard drives
on the x4 SAS port.
The Application Acceleration Card is the straight PCI-E SSD.
There are 200 and 400GB SLC and 400, 800 and 1600GB MLC versions.
LSI also has Nytro XD Application Acceleration Storage Solution, a software product
for caching SAN or direct-attach storage on SSD.
| SLC | MLC | |
|---|---|---|
| Capacities (GB) | 200/400 | 400/800/1.6TB |
| Raw Capacity (GB) | ? | ? |
| Write Endurance (PB) | ? | ? |
| Controllers | ? | ? |
| Max Read | 2,000MB/s | 2,000MB/s |
| Max Write | 1,700MB/s | 1,000MB/s |
| Random Read 4K | 238K IOPS | 218K IOPS |
| Random Write 4K | 133K IOPS | 75K IOPS |
| Random Read 8K | 189K IOPS | 183K IOPS |
| Random Write 8K | 137K IOPS | 118K IOPS |
| Pricing | $6K/12K? | $6K/10K/$20K? |
I think the LSI came out in 2011, but I was not paying attention. The SLP-300 is rated at 1,400MB/s read, 1,200MB/s write for 64K sequential. 4K random is 150K read and 190K write. Interface is x8 PCI-E gen 2. Latency is < 50usec.
OCZ Z-Drive R4 R and C
OCZ PCI-E SSDs moved OCZ PCI-E SSDs
Micron Updated 2011-05
Anandtech details the Micron P320h PCIe SLC SSD with 350 & 700GB capacities. Specifications are 3GB/s read, 2GB/s write, 750K read & 298K write IOPS (140K for 350GB and 200K for 700GB models?). The IOPS numbers vary by source, so it is probably a complicated matter that is not being documented fully. Endurance is 25 & 50 petabytes respectively.
A feature of the P320h is Redundant Array of Independent NAND (RAIN). I believe this to be the correct evolutionary shift from HDD RAID, as the SSD is not fundamentally a single device, but rather multiple devices. So the right place for redundancy is inside the SSD. Computerworld reports RAIN is implemented as 7 channels of data plus 1 channel for parity. Read and Write latency is 50 μsec.
The table below is from the Micron p320 datasheet. I am not sure how the numbers work out. I am thinking there are currently SLC NAND 8-Gbit chips in production. It is possible to put 4 or 8 chips in one package. Then 8 chips of 8Gbit each is 64Gb or 8GB (so presumably NAND density is at the package level?). It would seem reasonable that an enterprise class SSD be over-provisioned, plus ECC and RAIN, so 64 x 8GB (= 512GB) packages in a 320GB product. Then the 700GB part would have to be 128 x 8GB or 64 x 16GB?
| Capacity | NAND Process | NAND Density | Package Count | Die per Package |
|---|---|---|---|---|
| 350GB | 34nm | 64Gb | 64 | 4 |
| 700GB | 34nm | 64Gb | 64 | 8 |
The Micron website says the P320h is sampling as of June 2011 with production in Q3. Computerworld reports that Micron intends to price around $16 per GB or $16K per TB. There is no mention of an MLC product.
Intel SSD
Various websites, AnandTech etc, discuss specifications for the upcoming Intel 710 and 720 series SSD, but the Intel website does not mention these on their SSD products page yet. The 720 is PCIe 2.0 SLC product with 2.2GB/s read, 1.8GB/s write, 180K IOPS read, 56K IOPS write.
Original 2010-10
HP
The HP specification sheet for their PCIe IO Accelerators also provides information on the Fusion-IO driver memory usage.
The amount of free RAM required by the driver depends on the size of the blocks used when writing to the drive. The smaller the blocks, the more RAM required. Here are the guidelines for each 80GB of storage:
| Average Block Size (bytes) | RAM usage (Megabytes) |
|---|---|
| 8,192 | 400 |
| 4,096 | 800 |
| 2,048 | 1,500 |
| 1,024 | 2,900 |
| 512 | 5,600 |
Give credit to the HP technical people really know what key information that should be documented is. This is too important to leave to the simpletons in !@#$%^&*. (IBM also puts out great Redbooks, give credit to IBM for spending money on important material, not just marketing rubbish).