It’s been a while since a new tcp congestion has been developed (by Google), which has the potential to replace the default cubic one! The TCP BBR tries to keep being fast and improves the network connectivity when there is packet loss. Enabling it on the server does not require it to enable it on the client to function BBR algorithm properly. So probably we’ll see the switch to this new tcp congestion control in the next major linux distros releases.
Here we present how to enable TCP BBR on your CentOS 7 box. As you know the CentOS 7 uses kernel 3.10.X and the TCP BBR was introduced in version 4.9+ kernels and in version 4.13 and on it has a tcp-level pacing so you do not need the qdisc “fq” to be enabled, when using TCP BBR. So we need a new kernel version 4.9 and on and the easiest and safest way to do it to use the mainline kernel from https://www.elrepo.org/. We have a detailed howto for this if you need more information on using mainline kernel with your CentOS 7.
Here are the steps, you need to be root to execute the following commands:
STEP 1) Update your system and reboot
yum clean all yum -y update reboot
STEP 2) Install the mainline kernel from https://www.elrepo.org/
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
wget http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum -y install ./elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum list available --disablerepo='*' --enablerepo=elrepo-kernel
yum install -y --enablerepo=elrepo-kernel kernel-ml
awk -F\' /^menuentry/{print\$2} /boot/grub2/grub.cfg
grub2-set-default 0
grub2-mkconfig -o /boot/grub2/grub.cfg
grub2-editenv list
reboot
Here we loaded the new (mainline) kernel:
[root@srv ~]# uname -a Linux srv.local 4.18.5-1.el7.elrepo.x86_64 #1 SMP Fri Aug 24 11:35:05 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
If you use this method with grub2-set-default you should set in “/etc/default/grub” (before executing the “grub2-mkconfig”):
GRUB_DEFAULT=saved
Of course you can just list the kernels with:
awk -F\' /^menuentry/{print\$2} /boot/grub2/grub.cfg
and set the ID of the new kernel with “grub2-set-default ID” as above. And then generate the configuration with “grub2-mkconfig”.
STEP 3) Enable the TCP BBR and save the options for the next reboot
[root@srv ~]# sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_congestion_control = bbr #if you kernel is under version 4.9, uncomment the following line #sysctl -w net.core.default_qdisc=fq
Here you can save the options for the next reboots:
#uncomment the line below if you kernel under version 4.9 #echo 'net.core.default_qdisc=fq' >> /etc/sysctl.d/99-sysctl.conf echo 'net.ipv4.tcp_congestion_control=bbr' >> /etc/sysctl.d/99-sysctl.conf
Here is the whole output of the above steps
[root@srv ~]# yum clean all
Loaded plugins: fastestmirror, langpacks
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
Cleaning repos: base epel extras nginx updates
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Cleaning up list of fastest mirrors
[root@srv ~]# yum -y update
Loaded plugins: fastestmirror, langpacks
base | 3.6 kB 00:00:00
epel/x86_64/metalink | 8.3 kB 00:00:00
epel | 3.2 kB 00:00:00
extras | 3.4 kB 00:00:00
nginx | 2.9 kB 00:00:00
updates | 3.4 kB 00:00:00
(1/8): base/7/x86_64/group_gz | 166 kB 00:00:00
(2/8): epel/x86_64/primary | 3.6 MB 00:00:00
(3/8): updates/7/x86_64/primary_db | 5.2 MB 00:00:00
(4/8): extras/7/x86_64/primary_db | 186 kB 00:00:00
(5/8): epel/x86_64/group_gz | 88 kB 00:00:00
(6/8): epel/x86_64/updateinfo | 937 kB 00:00:00
(7/8): nginx/7/x86_64/primary_db | 116 kB 00:00:00
(8/8): base/7/x86_64/primary_db | 5.9 MB 00:00:03
Determining fastest mirrors
* base: repos-va.psychz.net
* epel: mirror.cogentco.com
* extras: mirror.cogentco.com
* updates: ewr.edge.kernel.org
Resolving Dependencies
--> Running transaction check
---> Package OpenIPMI-modalias.x86_64 0:2.0.19-15.el7 will be updated
---> Package OpenIPMI-modalias.x86_64 0:2.0.23-2.el7 will be an update
---> Package abrt.x86_64 0:2.1.11-48.el7.centos will be updated
---> Package abrt.x86_64 0:2.1.11-50.el7.centos will be an update
....
....
Transaction Summary
============================================================================================================================================================================
Install 2 Packages (+1 Dependent package)
Upgrade 283 Packages
Total download size: 311 M
Downloading packages:
No Presto metadata available for nginx
No Presto metadata available for base
(1/2): updates/7/x86_64/prestodelta | 538 kB 00:00:00
(2/2): epel/x86_64/prestodelta | 611 B 00:00:00
Delta RPMs reduced 77 M of updates to 31 M (60% saved)
(1/286): audit-libs-python-2.7.6-3.el7_2.8.1-3.el7_5.1.x86_64.drpm | 34 kB 00:00:00
(2/286): audit-libs-2.7.6-3.el7_2.8.1-3.el7_5.1.x86_64.drpm | 48 kB 00:00:00
....
....
Complete!
[root@srv ~]# reboot
[root@srv ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
[root@srv ~]# wget http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
--2018-08-29 08:45:41-- http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
Resolving www.elrepo.org (www.elrepo.org)... 69.195.83.87
Connecting to www.elrepo.org (www.elrepo.org)|69.195.83.87|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8656 (8.5K) [application/x-rpm]
Saving to: ‘elrepo-release-7.0-3.el7.elrepo.noarch.rpm’
100%[==================================================================================================================================>] 8,656 --.-K/s in 0s
2018-08-29 08:45:41 (82.5 MB/s) - ‘elrepo-release-7.0-3.el7.elrepo.noarch.rpm’ saved [8656/8656]
[root@srv ~]# yum -y install ./elrepo-release-7.0-3.el7.elrepo.noarch.rpm
Loaded plugins: fastestmirror, langpacks
Examining ./elrepo-release-7.0-3.el7.elrepo.noarch.rpm: elrepo-release-7.0-3.el7.elrepo.noarch
Marking ./elrepo-release-7.0-3.el7.elrepo.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package elrepo-release.noarch 0:7.0-3.el7.elrepo will be installed
--> Finished Dependency Resolution
Dependencies Resolved
============================================================================================================================================================================
Package Arch Version Repository Size
============================================================================================================================================================================
Installing:
elrepo-release noarch 7.0-3.el7.elrepo /elrepo-release-7.0-3.el7.elrepo.noarch 5.2 k
Transaction Summary
============================================================================================================================================================================
Install 1 Package
Total size: 5.2 k
Installed size: 5.2 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : elrepo-release-7.0-3.el7.elrepo.noarch 1/1
Verifying : elrepo-release-7.0-3.el7.elrepo.noarch 1/1
Installed:
elrepo-release.noarch 0:7.0-3.el7.elrepo
Complete!
[root@srv ~]#
[root@srv ~]# yum list available --disablerepo='*' --enablerepo=elrepo-kernel
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* elrepo-kernel: reflector.westga.edu
elrepo-kernel | 2.9 kB 00:00:00
elrepo-kernel/primary_db | 1.8 MB 00:00:00
Available Packages
kernel-lt.x86_64 4.4.153-1.el7.elrepo elrepo-kernel
kernel-lt-devel.x86_64 4.4.153-1.el7.elrepo elrepo-kernel
kernel-lt-doc.noarch 4.4.153-1.el7.elrepo elrepo-kernel
kernel-lt-headers.x86_64 4.4.153-1.el7.elrepo elrepo-kernel
kernel-lt-tools.x86_64 4.4.153-1.el7.elrepo elrepo-kernel
kernel-lt-tools-libs.x86_64 4.4.153-1.el7.elrepo elrepo-kernel
kernel-lt-tools-libs-devel.x86_64 4.4.153-1.el7.elrepo elrepo-kernel
kernel-ml.x86_64 4.18.5-1.el7.elrepo elrepo-kernel
kernel-ml-devel.x86_64 4.18.5-1.el7.elrepo elrepo-kernel
kernel-ml-doc.noarch 4.18.5-1.el7.elrepo elrepo-kernel
kernel-ml-headers.x86_64 4.18.5-1.el7.elrepo elrepo-kernel
kernel-ml-tools.x86_64 4.18.5-1.el7.elrepo elrepo-kernel
kernel-ml-tools-libs.x86_64 4.18.5-1.el7.elrepo elrepo-kernel
kernel-ml-tools-libs-devel.x86_64 4.18.5-1.el7.elrepo elrepo-kernel
perf.x86_64 4.18.5-1.el7.elrepo elrepo-kernel
python-perf.x86_64
[root@srv ~]# yum install -y --enablerepo=elrepo-kernel kernel-ml
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: repos-va.psychz.net
* elrepo: iad.mirror.rackspace.com
* elrepo-kernel: iad.mirror.rackspace.com
* epel: mirror.cogentco.com
* extras: mirror.cogentco.com
* updates: ewr.edge.kernel.org
elrepo | 2.9 kB 00:00:00
elrepo/primary_db | 494 kB 00:00:00
Resolving Dependencies
--> Running transaction check
---> Package kernel-ml.x86_64 0:4.18.5-1.el7.elrepo will be installed
--> Finished Dependency Resolution
Dependencies Resolved
============================================================================================================================================================================
Package Arch Version Repository Size
============================================================================================================================================================================
Installing:
kernel-ml x86_64 4.18.5-1.el7.elrepo elrepo-kernel 45 M
Transaction Summary
============================================================================================================================================================================
Install 1 Package
Total download size: 45 M
Installed size: 203 M
Downloading packages:
kernel-ml-4.18.5-1.el7.elrepo.x86_64.rpm | 45 MB 00:00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : kernel-ml-4.18.5-1.el7.elrepo.x86_64 1/1
Verifying : kernel-ml-4.18.5-1.el7.elrepo.x86_64 1/1
Installed:
kernel-ml.x86_64 0:4.18.5-1.el7.elrepo
Complete!
[root@srv ~]# awk -F\' /^menuentry/{print\$2} /boot/grub2/grub.cfg
CentOS Linux (4.18.5-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-862.11.6.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-693.21.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-f6659a14d3de4b758a66a90e9d0bf948) 7 (Core)
[root@srv ~]# grub2-set-default 0
[root@srv ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.18.5-1.el7.elrepo.x86_64
Found initrd image: /boot/initramfs-4.18.5-1.el7.elrepo.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-862.11.6.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-862.11.6.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-693.21.1.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-693.21.1.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-693.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-693.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-f6659a14d3de4b758a66a90e9d0bf948
Found initrd image: /boot/initramfs-0-rescue-f6659a14d3de4b758a66a90e9d0bf948.img
done
[root@srv ~]# grub2-editenv list
saved_entry=0
[root@srv ~]# reboot
[root@srv ~]# uname -a
Linux srv.local 4.18.5-1.el7.elrepo.x86_64 #1 SMP Fri Aug 24 11:35:05 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@srv ~]# sysctl -w net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_congestion_control = bbr
[root@srv ~]# sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = bbr
[root@srv ~]# #echo 'net.core.default_qdisc=fq' >> /etc/sysctl.d/99-sysctl.conf
[root@srv ~]# echo 'net.ipv4.tcp_congestion_control=bbr' >> /etc/sysctl.d/99-sysctl.conf