network | Any IT here? Help Me!

Dual 10Gbit network using PCI 2.0 (5GT/s) x4 – what is the maximum bandwidth?

Ever wondered what is the maximum bandwidth of a Dual 10Gbit LAN card, which can be reached using a dual 10Gbit ports card in a PCI Express 2.0 (Speed 5GT/s) and Width x4?
Here is the graph:
h3>SCREENSHOT 1) The bandwidth never exceeds 13.90Gbps (performed with only synthetic tests and mixed synthetic plus real http traffic).

main menu — Max graph bandwidth – below 14Gbps

As you can see the total of the two network ports is a little bit under 14Gbps. We are using intel dual-port controller:

Intel Corporation Ethernet Server Adapter X520-2

Even the dmesg reports the card is not in the right place:

[ 2.541813] ixgbe 0000:82:00.0: (Speed:5.0GT/s, Width: x4, Encoding Loss:20%)
[ 2.541832] ixgbe 0000:82:00.0: This is not sufficient for optimal performance of this card.
[ 2.541854] ixgbe 0000:82:00.0: For optimal performance, at least 20GT/s of bandwidth is required.
[ 2.541876] ixgbe 0000:82:00.0: A slot with more lanes and/or higher speed is suggested.
[ 2.541978] ixgbe 0000:82:00.0: MAC: 2, PHY: 19, SFP+: 5, PBA No: FFFFFF-0FF
[ 2.541996] ixgbe 0000:82:00.0: 00:16:31:fd:03:b8
[ 2.543027] ixgbe 0000:82:00.0: Intel(R) 10 Gigabit Network Connection
[ 2.694839] ixgbe 0000:82:00.1: Multiqueue Enabled: Rx Queue count = 48, Tx Queue count = 48 XDP Queue count = 0
[ 2.695531] ixgbe 0000:82:00.1: PCI Express bandwidth of 16GT/s available
[ 2.696087] ixgbe 0000:82:00.1: (Speed:5.0GT/s, Width: x4, Encoding Loss:20%)
[ 2.696631] ixgbe 0000:82:00.1: This is not sufficient for optimal performance of this card.
[ 2.697181] ixgbe 0000:82:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
[ 2.697723] ixgbe 0000:82:00.1: A slot with more lanes and/or higher speed is suggested.
[ 2.698352] ixgbe 0000:82:00.1: MAC: 2, PHY: 19, SFP+: 6, PBA No: FFFFFF-0FF
[ 2.698890] ixgbe 0000:82:00.1: 00:16:31:fd:03:b9
[ 2.700436] ixgbe 0000:82:00.1: Intel(R) 10 Gigabit Network Connection

The controller is in the PCI Express slot – PCI 2.0 (Speed 5.0GT/s) Width x4, but the capability of the card is Speed 5GT/s, Width x8. This can be seen with “lspci -vvv” and the meanings with simple words:

LnkCap – it is the device capability. In fact, this is the hightest possible speed of the device put in the slot.
LnkSta – the actual speed of the PCI Express link.

If the device capacity (LnkCap) is higher than the actual speed (LnkSta) you could put the device in another slot with a higher capacity to take full advantage of the device.

In our case, the maximum bandwidth of the two ports of the Dual 10G port Intel card was just below 14Gbps (13.85Gbps ~ 13.95Gbps). After we move the very same card in another slot with the capability of Speed 5GT/s Width x8, the card’s maximum bandwidth increased to 19.20Gbps ~ 19.40Gbps.

SCREENSHOT 2) After changing the slot of the network card, which supports PCI 2.0 (5GT/s) Width x8, the bandwidth tops arround 19.40Gbps in synthetic tests (performed with iperf3).

Keep on reading!

Gentoo Minimal Installation CD (amd64 aka x86_64) – booting (in UEFI mode)

Here is the process of booting from a Gentoo Minimal Installation CD amd64 (x86_64) with UEFI mode enabled. This is not an installation guide!
You can download the CD from here: https://www.gentoo.org/downloads/ Here is the ISO file: http://distfiles.gentoo.org/releases/amd64/autobuilds/20180911T214502Z/install-amd64-minimal-20180911T214502Z.iso or you can check it in some mirror like leaseweb – http://mirror.leaseweb.com/gentoo/releases/amd64/autobuilds/current-install-amd64-minimal/install-amd64-minimal-20180911T214502Z.iso.

Our motherboard is Asus ROG Zenith Extreme motherboard using X399 chipset https://www.asus.com/th/Motherboards/ROG-ZENITH-EXTREME/ and you’ll see the BIOS options for it, but they do not differ much with the other motherboard boot options. Here also setup the network and start up an openssh server to manage our Gentoo Linux installation – (coming soon).

SCREENSHOT 1) Starting the machine

Keep on reading!

Pages: 123

Ryzen Threadripper kernel problems with PCIe – wifi, network, video problems

If you have Ryzen Threadripper (in my case 1950X) and Ubuntu 17.10 (because the others versions do not boot at all!) with the latest kernel 4.15.x and the default vmlinuz-4.13.0-21-generic you might experience frequently wifi, network cards and video cards problems like wifi disconnects and network cards lost connections and some kind of freezes for a second or two of the video! If you have Ryzen Threadripper 1950X and or ASUS ROG ZENITH EXTREME (or any other motherboard with the chipset AMD X399) and ubuntu 17.10 or probably any linux distro with any kernel you might experience really unstable system. Look at your dmesg and if you see something like

[   18.669950] dpc 0000:00:01.1:pcie010: DPC containment event, status:0x1f00 source:0x0000
[   18.669966] pcieport 0000:00:01.1: AER: Corrected error received: id=0000
[   18.669970] pcieport 0000:00:01.1: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0009(Receiver ID)
[   18.669973] pcieport 0000:00:01.1:   device [1022:1453] error status/mask=00000040/00006000
[   18.669975] pcieport 0000:00:01.1:    [ 6] Bad TLP               
[   23.805640] dpc 0000:00:01.1:pcie010: DPC containment event, status:0x1f00 source:0x0000
[   23.805664] pcieport 0000:00:01.1: AER: Corrected error received: id=0000
[   23.805669] pcieport 0000:00:01.1: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0009(Receiver ID)
[   23.805674] pcieport 0000:00:01.1:   device [1022:1453] error status/mask=00000040/00006000
[   23.805678] pcieport 0000:00:01.1:    [ 6] Bad TLP               
[   23.838698] dpc 0000:00:01.1:pcie010: DPC containment event, status:0x1f00 source:0x0000
[   23.838716] pcieport 0000:00:01.1: AER: Corrected error received: id=0000
[   23.838720] pcieport 0000:00:01.1: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0009(Receiver ID)
[   23.838725] pcieport 0000:00:01.1:   device [1022:1453] error status/mask=00000040/00006000
[   23.838728] pcieport 0000:00:01.1:    [ 6] Bad TLP               
[   36.124101] atlantic: link change old 0 new 100
[   36.124222] IPv6: ADDRCONF(NETDEV_CHANGE): enp65s0: link becomes ready
[   52.294413] dpc 0000:00:01.1:pcie010: DPC containment event, status:0x1f00 source:0x0000
[   52.294436] pcieport 0000:00:01.1: AER: Corrected error received: id=0000
[   52.294441] pcieport 0000:00:01.1: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0009(Receiver ID)
[   52.294446] pcieport 0000:00:01.1:   device [1022:1453] error status/mask=00000040/00006000
[   52.294449] pcieport 0000:00:01.1:    [ 6] Bad TLP               
[   53.275256] dpc 0000:00:01.1:pcie010: DPC containment event, status:0x1f00 source:0x0000
[   53.275289] pcieport 0000:00:01.1: AER: Corrected error received: id=0000
[   53.275293] pcieport 0000:00:01.1: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0009(Receiver ID)
[   53.275298] pcieport 0000:00:01.1:   device [1022:1453] error status/mask=00000040/00006000
[   53.275301] pcieport 0000:00:01.1:    [ 6] Bad TLP               
[   53.418493] dpc 0000:00:01.1:pcie010: DPC containment event, status:0x1f00 source:0x0000
[   53.418517] pcieport 0000:00:01.1: AER: Corrected error received: id=0000
[   53.418522] pcieport 0000:00:01.1: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0009(Receiver ID)
[   53.418527] pcieport 0000:00:01.1:   device [1022:1453] error status/mask=00000040/00006000
[   53.418530] pcieport 0000:00:01.1:    [ 6] Bad TLP               
[   53.649938] dpc 0000:00:01.1:pcie010: DPC containment event, status:0x1f00 source:0x0000
[   53.649962] pcieport 0000:00:01.1: AER: Corrected error received: id=0000
[   53.649967] pcieport 0000:00:01.1: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0009(Receiver ID)
[   53.649972] pcieport 0000:00:01.1:   device [1022:1453] error status/mask=00000040/00006000
[   53.649975] pcieport 0000:00:01.1:    [ 6] Bad TLP               
[  113.294411] dpc 0000:00:01.1:pcie010: DPC containment event, status:0x1f00 source:0x0000
[  113.294434] pcieport 0000:00:01.1: AER: Corrected error received: id=0000
[  113.294439] pcieport 0000:00:01.1: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0009(Receiver ID)
[  113.294444] pcieport 0000:00:01.1:   device [1022:1453] error status/mask=00000040/00006000
[  113.294450] pcieport 0000:00:01.1:    [ 6] Bad TLP               
[  265.469397] dpc 0000:00:01.1:pcie010: DPC containment event, status:0x1f00 source:0x0000
[  265.469419] pcieport 0000:00:01.1: AER: Corrected error received: id=0000
[  265.469424] pcieport 0000:00:01.1: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0009(Transmitter ID)
[  265.469429] pcieport 0000:00:01.1:   device [1022:1453] error status/mask=00001000/00006000
[  265.469438] pcieport 0000:00:01.1:    [12] Replay Timer Timeout  
[  533.031982] dpc 0000:00:01.1:pcie010: DPC containment event, status:0x1f00 source:0x0000
[  533.032004] pcieport 0000:00:01.1: AER: Corrected error received: id=0000
[  533.032009] pcieport 0000:00:01.1: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0009(Receiver ID)
[  533.032014] pcieport 0000:00:01.1:   device [1022:1453] error status/mask=00000080/00006000
[  533.032022] pcieport 0000:00:01.1:    [ 7] Bad DLLP

Apparently everything on the pci express could be affected! So till the kernel team fixes the issues, you can use the following workaround: add to the kernel boot parameters

pcie_aspm=off

Under Ubuntu you can add in the file

/etc/default/grub

the following line

GRUB_CMDLINE_LINUX="pcie_aspm=off"

And then execute

update-grub

the changes to take effect in the grub configuration. Then reboot the machine!

Under CentOS 7 the configuration file and the line to put are the same just the update of the grub configuration is different:

For UEFI-based systems execute:

grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

For legacy-BIOS systems execute:
```
grub2-mkconfig -o /boot/grub2/grub.cfg
```

*You should know what is your setup, probably UEFI-based, you could check in your boot directory, if there is /boot/efi/EFI/centos/grub.cfg you probably should use the option for UEFI-based, if you do not have any “efi” directory under “/boot” use the second option for legacy-BIOS.
Then reboot the machine!