Ever wondered what is the maximum bandwidth of a Dual 10Gbit LAN card, which can be reached using a dual 10Gbit ports card in a PCI Express 2.0 (Speed 5GT/s) and Width x4?
Here is the graph:
h3>SCREENSHOT 1) The bandwidth never exceeds 13.90Gbps (performed with only synthetic tests and mixed synthetic plus real http traffic).
As you can see the total of the two network ports is a little bit under 14Gbps. We are using intel dual-port controller:
Intel Corporation Ethernet Server Adapter X520-2
Even the dmesg reports the card is not in the right place:
[ 2.541813] ixgbe 0000:82:00.0: (Speed:5.0GT/s, Width: x4, Encoding Loss:20%) [ 2.541832] ixgbe 0000:82:00.0: This is not sufficient for optimal performance of this card. [ 2.541854] ixgbe 0000:82:00.0: For optimal performance, at least 20GT/s of bandwidth is required. [ 2.541876] ixgbe 0000:82:00.0: A slot with more lanes and/or higher speed is suggested. [ 2.541978] ixgbe 0000:82:00.0: MAC: 2, PHY: 19, SFP+: 5, PBA No: FFFFFF-0FF [ 2.541996] ixgbe 0000:82:00.0: 00:16:31:fd:03:b8 [ 2.543027] ixgbe 0000:82:00.0: Intel(R) 10 Gigabit Network Connection [ 2.694839] ixgbe 0000:82:00.1: Multiqueue Enabled: Rx Queue count = 48, Tx Queue count = 48 XDP Queue count = 0 [ 2.695531] ixgbe 0000:82:00.1: PCI Express bandwidth of 16GT/s available [ 2.696087] ixgbe 0000:82:00.1: (Speed:5.0GT/s, Width: x4, Encoding Loss:20%) [ 2.696631] ixgbe 0000:82:00.1: This is not sufficient for optimal performance of this card. [ 2.697181] ixgbe 0000:82:00.1: For optimal performance, at least 20GT/s of bandwidth is required. [ 2.697723] ixgbe 0000:82:00.1: A slot with more lanes and/or higher speed is suggested. [ 2.698352] ixgbe 0000:82:00.1: MAC: 2, PHY: 19, SFP+: 6, PBA No: FFFFFF-0FF [ 2.698890] ixgbe 0000:82:00.1: 00:16:31:fd:03:b9 [ 2.700436] ixgbe 0000:82:00.1: Intel(R) 10 Gigabit Network Connection
The controller is in the PCI Express slot – PCI 2.0 (Speed 5.0GT/s) Width x4, but the capability of the card is Speed 5GT/s, Width x8. This can be seen with “lspci -vvv” and the meanings with simple words:
- LnkCap – it is the device capability. In fact, this is the hightest possible speed of the device put in the slot.
- LnkSta – the actual speed of the PCI Express link.
If the device capacity (LnkCap) is higher than the actual speed (LnkSta) you could put the device in another slot with a higher capacity to take full advantage of the device.
In our case, the maximum bandwidth of the two ports of the Dual 10G port Intel card was just below 14Gbps (13.85Gbps ~ 13.95Gbps). After we move the very same card in another slot with the capability of Speed 5GT/s Width x8, the card’s maximum bandwidth increased to 19.20Gbps ~ 19.40Gbps.
SCREENSHOT 2) After changing the slot of the network card, which supports PCI 2.0 (5GT/s) Width x8, the bandwidth tops arround 19.40Gbps in synthetic tests (performed with iperf3).
