Home, Optimizer, Benchmarks, Server Systems, Systems Architecture, Processors, Storage,
PCI-E, SAS, FC,
HDD,
SSD Technology,
RAID Controllers, Direct-Attach,
SAN, Dell MD3200,
CLARiiON AX4, CX4, VNX, V-Max,
HP P2000, EVA, P9000/VSP,
Hitachi AMS
SSD products: SATA/SAS SSDs, PCI-E SSDs,
Fusion iO, other SSD
PCI-E Nominal and Realizable Bandwidth (bi-directional)
PCI-E (or PCIe) bandwidth can be tricky to understand. It is perfectly legitamate to cite PCIe 1.0 as 2.5GT/s (giga-transfers/sec), 2.0 as 5GT/s and 3.0 as 8GT/s. PCIe 2.0 and 3.0 can also be referred to as gen 2 and gen 3. Notice the term GT/s as opposed to GHz. Engineers like to associated the term Hertz in GHz with the clock frequency. And it is popular to transfer data at a higher multiple of the clock. PCIe in fact has a reference clock at 100MHz. So we use the term giga-transfers per sec instead of GHz.
In high-speed data transmission, it is necessary to avoid having the transmitter and receiver drift out of synch due to runs of 0's or 1's. To do this without a separate clock signal, PCIe 1.0 and 2.0 encodes every 8 bits to 10b. The PCIe 3.0 encoding scheme is 128b/130b.
Next, we cannot simply send raw data. There must be information on the send, the recipient, and additional information. The PCIe protocol has 3 layers: physical, data link and transaction. The physical layer has a Frame, the data link layer has a Packet Sequence Number at the front and a CRC at the end, and the transaction layer has a header.
PCIe Protocol layers
PCIe Packet
Packet Header
PCIe Packet
By the time all is said and done, the table below shows the approximate net bandwidth for data in x4 and x8 slots.
| PCIe Gen | Raw bit rate | Encoding | Interconnect bandwidth | BW per lane per direction | BW per direction x8 | Net Data BW x4 | Net Data BW x8 |
|---|---|---|---|---|---|---|---|
| 1.0 | 2.5GT/s | 8b/10b | 2Gbps | 250MB/s | 2GB/s | 0.8GB/s | 1.6GB/s |
| 2.0 | 5.0GT/s | 8b/10b | 4Gbps | 500MB/s | 4GB/s | 1.6GB/s | 3.2GB/s |
| 3.0 | 8.0GT/s | 128b/130b | 8Gbps | 1GB/s | 8GB/s | 3.2GB/s? | 6.4GB/s? |
PCI-E 2.0 signals at 5.0Gbit/s, also with 8b/10b encoding for a bandwidth of 400MB/s per lane per direction. PCI-E 3.0 signals at 8.0Gbit/s, but with 128b/130b encoding for essentially double the 2.0 bandwidth.
Finally, note that there is difference between the maximum GT/s for each PCIe version and the PCIe version. A product can implement 2.5 or 5GT/s, but still be 3.0 compliant.
Systems of the Intel Core 2 processor architecture generation (Xeon 5100-5400 and Xeon 7300-7400 series) are almost exclusively PCI-E gen 1, as are the accompanying chipsets: the 5000P and 7300. The Intel 5400 MCH did support PCI-E gen 2, but no tier-1 system vendor produced a server with this chipset. (Supermicro, popular with white-box builders, did have 5400-based motherboards.) Systems of the Intel Nehalem generation and later have PCI-E gen 2. If someone could advise on when AMD Opteron transitioned from PCI-E gen 1 to gen 2, I would appreciate it.
The diagram below from the Intel IOH specification shows Intel 5520 IOH with 36 PCI-E gen 2 lanes, which are organized in 2 sets of 4 x4 and one set of 2 x2. Several combinations of x16, x8, x4 and even x2 slots can be configured. A server system should employ a proper combination of x8 and x4 slots. The x16 configuration is mostly for workstation graphics or special coprocessor, as there are no server disk or network adapters that can make effective use of x16 slots. Of course, there is the possible exception of Fusion-IO octal cards.
The diagram below shows more detail for the Intel IOH.
