Enable BBR tcp congestion control under CentOS 7

It’s been a while since a new tcp congestion has been developed (by Google), which has the potential to replace the default cubic one! The TCP BBR tries to keep being fast and improves the network connectivity when there is packet loss. Enabling it on the server does not require it to enable it on the client to function BBR algorithm properly. So probably we’ll see the switch to this new tcp congestion control in the next major linux distros releases.

Here we present how to enable TCP BBR on your CentOS 7 box. As you know the CentOS 7 uses kernel 3.10.X and the TCP BBR was introduced in version 4.9+ kernels and in version 4.13 and on it has a tcp-level pacing so you do not need the qdisc “fq” to be enabled, when using TCP BBR. So we need a new kernel version 4.9 and on and the easiest and safest way to do it to use the mainline kernel from https://www.elrepo.org/. We have a detailed howto for this if you need more information on using mainline kernel with your CentOS 7.

Here are the steps, you need to be root to execute the following commands:

STEP 1) Update your system and reboot

yum clean all
yum -y update
reboot

STEP 2) Install the mainline kernel from https://www.elrepo.org/

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
wget http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum -y install ./elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum list available --disablerepo='*' --enablerepo=elrepo-kernel
yum install -y --enablerepo=elrepo-kernel kernel-ml
awk -F\' /^menuentry/{print\$2} /boot/grub2/grub.cfg
grub2-set-default 0
grub2-mkconfig -o /boot/grub2/grub.cfg
grub2-editenv list
reboot

Here we loaded the new (mainline) kernel:

[root@srv ~]# uname -a
Linux srv.local 4.18.5-1.el7.elrepo.x86_64 #1 SMP Fri Aug 24 11:35:05 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

If you use this method with grub2-set-default you should set in “/etc/default/grub” (before executing the “grub2-mkconfig”):

GRUB_DEFAULT=saved

Of course you can just list the kernels with:

awk -F\' /^menuentry/{print\$2} /boot/grub2/grub.cfg

and set the ID of the new kernel with “grub2-set-default ID” as above. And then generate the configuration with “grub2-mkconfig”.

STEP 3) Enable the TCP BBR and save the options for the next reboot

[root@srv ~]# sysctl -w net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_congestion_control = bbr
#if you kernel is under version 4.9, uncomment the following line
#sysctl -w net.core.default_qdisc=fq

Here you can save the options for the next reboots:

#uncomment the line below if you kernel under version 4.9
#echo 'net.core.default_qdisc=fq' >> /etc/sysctl.d/99-sysctl.conf
echo 'net.ipv4.tcp_congestion_control=bbr' >> /etc/sysctl.d/99-sysctl.conf

Here is the whole output of the above steps

[root@srv ~]# yum clean all
Loaded plugins: fastestmirror, langpacks
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
Cleaning repos: base epel extras nginx updates
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Cleaning up list of fastest mirrors
[root@srv ~]# yum -y update
Loaded plugins: fastestmirror, langpacks
base                                                                                                                                                 | 3.6 kB  00:00:00     
epel/x86_64/metalink                                                                                                                                 | 8.3 kB  00:00:00     
epel                                                                                                                                                 | 3.2 kB  00:00:00     
extras                                                                                                                                               | 3.4 kB  00:00:00     
nginx                                                                                                                                                | 2.9 kB  00:00:00     
updates                                                                                                                                              | 3.4 kB  00:00:00     
(1/8): base/7/x86_64/group_gz                                                                                                                        | 166 kB  00:00:00     
(2/8): epel/x86_64/primary                                                                                                                           | 3.6 MB  00:00:00     
(3/8): updates/7/x86_64/primary_db                                                                                                                   | 5.2 MB  00:00:00     
(4/8): extras/7/x86_64/primary_db                                                                                                                    | 186 kB  00:00:00     
(5/8): epel/x86_64/group_gz                                                                                                                          |  88 kB  00:00:00     
(6/8): epel/x86_64/updateinfo                                                                                                                        | 937 kB  00:00:00     
(7/8): nginx/7/x86_64/primary_db                                                                                                                     | 116 kB  00:00:00     
(8/8): base/7/x86_64/primary_db                                                                                                                      | 5.9 MB  00:00:03     
Determining fastest mirrors
 * base: repos-va.psychz.net
 * epel: mirror.cogentco.com
 * extras: mirror.cogentco.com
 * updates: ewr.edge.kernel.org
Resolving Dependencies
--> Running transaction check
---> Package OpenIPMI-modalias.x86_64 0:2.0.19-15.el7 will be updated
---> Package OpenIPMI-modalias.x86_64 0:2.0.23-2.el7 will be an update
---> Package abrt.x86_64 0:2.1.11-48.el7.centos will be updated
---> Package abrt.x86_64 0:2.1.11-50.el7.centos will be an update
....
....
Transaction Summary
============================================================================================================================================================================
Install    2 Packages (+1 Dependent package)
Upgrade  283 Packages

Total download size: 311 M
Downloading packages:
No Presto metadata available for nginx
No Presto metadata available for base
(1/2): updates/7/x86_64/prestodelta                                                                                                                  | 538 kB  00:00:00     
(2/2): epel/x86_64/prestodelta                                                                                                                       |  611 B  00:00:00     
Delta RPMs reduced 77 M of updates to 31 M (60% saved)
(1/286): audit-libs-python-2.7.6-3.el7_2.8.1-3.el7_5.1.x86_64.drpm                                                                                   |  34 kB  00:00:00     
(2/286): audit-libs-2.7.6-3.el7_2.8.1-3.el7_5.1.x86_64.drpm                                                                                          |  48 kB  00:00:00     
....
....

Complete!
[root@srv ~]# reboot
[root@srv ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
[root@srv ~]# wget http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
--2018-08-29 08:45:41--  http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
Resolving www.elrepo.org (www.elrepo.org)... 69.195.83.87
Connecting to www.elrepo.org (www.elrepo.org)|69.195.83.87|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8656 (8.5K) [application/x-rpm]
Saving to: ‘elrepo-release-7.0-3.el7.elrepo.noarch.rpm’

100%[==================================================================================================================================>] 8,656       --.-K/s   in 0s      

2018-08-29 08:45:41 (82.5 MB/s) - ‘elrepo-release-7.0-3.el7.elrepo.noarch.rpm’ saved [8656/8656]

[root@srv ~]# yum -y install ./elrepo-release-7.0-3.el7.elrepo.noarch.rpm
Loaded plugins: fastestmirror, langpacks
Examining ./elrepo-release-7.0-3.el7.elrepo.noarch.rpm: elrepo-release-7.0-3.el7.elrepo.noarch
Marking ./elrepo-release-7.0-3.el7.elrepo.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package elrepo-release.noarch 0:7.0-3.el7.elrepo will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================================================
 Package                             Arch                        Version                                 Repository                                                    Size
============================================================================================================================================================================
Installing:
 elrepo-release                      noarch                      7.0-3.el7.elrepo                        /elrepo-release-7.0-3.el7.elrepo.noarch                      5.2 k

Transaction Summary
============================================================================================================================================================================
Install  1 Package

Total size: 5.2 k
Installed size: 5.2 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : elrepo-release-7.0-3.el7.elrepo.noarch                                                                                                                   1/1 
  Verifying  : elrepo-release-7.0-3.el7.elrepo.noarch                                                                                                                   1/1 

Installed:
  elrepo-release.noarch 0:7.0-3.el7.elrepo                                                                                                                                  

Complete!
[root@srv ~]#
[root@srv ~]# yum list available --disablerepo='*' --enablerepo=elrepo-kernel
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * elrepo-kernel: reflector.westga.edu
elrepo-kernel                                                                                                                                        | 2.9 kB  00:00:00     
elrepo-kernel/primary_db                                                                                                                             | 1.8 MB  00:00:00     
Available Packages
kernel-lt.x86_64                                                                      4.4.153-1.el7.elrepo                                                     elrepo-kernel
kernel-lt-devel.x86_64                                                                4.4.153-1.el7.elrepo                                                     elrepo-kernel
kernel-lt-doc.noarch                                                                  4.4.153-1.el7.elrepo                                                     elrepo-kernel
kernel-lt-headers.x86_64                                                              4.4.153-1.el7.elrepo                                                     elrepo-kernel
kernel-lt-tools.x86_64                                                                4.4.153-1.el7.elrepo                                                     elrepo-kernel
kernel-lt-tools-libs.x86_64                                                           4.4.153-1.el7.elrepo                                                     elrepo-kernel
kernel-lt-tools-libs-devel.x86_64                                                     4.4.153-1.el7.elrepo                                                     elrepo-kernel
kernel-ml.x86_64                                                                      4.18.5-1.el7.elrepo                                                      elrepo-kernel
kernel-ml-devel.x86_64                                                                4.18.5-1.el7.elrepo                                                      elrepo-kernel
kernel-ml-doc.noarch                                                                  4.18.5-1.el7.elrepo                                                      elrepo-kernel
kernel-ml-headers.x86_64                                                              4.18.5-1.el7.elrepo                                                      elrepo-kernel
kernel-ml-tools.x86_64                                                                4.18.5-1.el7.elrepo                                                      elrepo-kernel
kernel-ml-tools-libs.x86_64                                                           4.18.5-1.el7.elrepo                                                      elrepo-kernel
kernel-ml-tools-libs-devel.x86_64                                                     4.18.5-1.el7.elrepo                                                      elrepo-kernel
perf.x86_64                                                                           4.18.5-1.el7.elrepo                                                      elrepo-kernel
python-perf.x86_64
[root@srv ~]# yum install -y --enablerepo=elrepo-kernel kernel-ml
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: repos-va.psychz.net
 * elrepo: iad.mirror.rackspace.com
 * elrepo-kernel: iad.mirror.rackspace.com
 * epel: mirror.cogentco.com
 * extras: mirror.cogentco.com
 * updates: ewr.edge.kernel.org
elrepo                                                                                                                                               | 2.9 kB  00:00:00     
elrepo/primary_db                                                                                                                                    | 494 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package kernel-ml.x86_64 0:4.18.5-1.el7.elrepo will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================================================
 Package                               Arch                               Version                                           Repository                                 Size
============================================================================================================================================================================
Installing:
 kernel-ml                             x86_64                             4.18.5-1.el7.elrepo                               elrepo-kernel                              45 M

Transaction Summary
============================================================================================================================================================================
Install  1 Package

Total download size: 45 M
Installed size: 203 M
Downloading packages:
kernel-ml-4.18.5-1.el7.elrepo.x86_64.rpm                                                                                                             |  45 MB  00:00:01     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : kernel-ml-4.18.5-1.el7.elrepo.x86_64                                                                                                                     1/1 
  Verifying  : kernel-ml-4.18.5-1.el7.elrepo.x86_64                                                                                                                     1/1 

Installed:
  kernel-ml.x86_64 0:4.18.5-1.el7.elrepo                                                                                                                                    

Complete!
[root@srv ~]# awk -F\' /^menuentry/{print\$2} /boot/grub2/grub.cfg
CentOS Linux (4.18.5-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-862.11.6.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-693.21.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-f6659a14d3de4b758a66a90e9d0bf948) 7 (Core)
[root@srv ~]# grub2-set-default 0
[root@srv ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.18.5-1.el7.elrepo.x86_64
Found initrd image: /boot/initramfs-4.18.5-1.el7.elrepo.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-862.11.6.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-862.11.6.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-693.21.1.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-693.21.1.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-693.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-693.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-f6659a14d3de4b758a66a90e9d0bf948
Found initrd image: /boot/initramfs-0-rescue-f6659a14d3de4b758a66a90e9d0bf948.img
done
[root@srv ~]# grub2-editenv list
saved_entry=0
[root@srv ~]# reboot
[root@srv ~]# uname -a
Linux srv.local 4.18.5-1.el7.elrepo.x86_64 #1 SMP Fri Aug 24 11:35:05 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@srv ~]# sysctl -w net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_congestion_control = bbr
[root@srv ~]# sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = bbr
[root@srv ~]# #echo 'net.core.default_qdisc=fq' >> /etc/sysctl.d/99-sysctl.conf
[root@srv ~]# echo 'net.ipv4.tcp_congestion_control=bbr' >> /etc/sysctl.d/99-sysctl.conf

Leave a Reply

Your email address will not be published. Required fields are marked *