Enable BBR tcp congestion control under CentOS 7

Author:

It’s been a while since a new tcp congestion has been developed (by Google), which has the potential to replace the default cubic one! The TCP BBR tries to keep being fast and improves the network connectivity when there is packet loss. Enabling it on the server does not require it to enable it on the client to function BBR algorithm properly. So probably we’ll see the switch to this new tcp congestion control in the next major linux distros releases.

Here we present how to enable TCP BBR on your CentOS 7 box. As you know the CentOS 7 uses kernel 3.10.X and the TCP BBR was introduced in version 4.9+ kernels and in version 4.13 and on it has a tcp-level pacing so you do not need the qdisc “fq” to be enabled, when using TCP BBR. So we need a new kernel version 4.9 and on and the easiest and safest way to do it to use the mainline kernel from https://www.elrepo.org/. We have a detailed howto for this if you need more information on using mainline kernel with your CentOS 7.

Here are the steps, you need to be root to execute the following commands:

STEP 1) Update your system and reboot

1
2
3
yum clean all
yum -y update
reboot

STEP 2) Install the mainline kernel from https://www.elrepo.org/

1
2
3
4
5
6
7
8
9
10
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
wget http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum -y install ./elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum list available --disablerepo='*' --enablerepo=elrepo-kernel
yum install -y --enablerepo=elrepo-kernel kernel-ml
awk -F\' /^menuentry/{print\$2} /boot/grub2/grub.cfg
grub2-set-default 0
grub2-mkconfig -o /boot/grub2/grub.cfg
grub2-editenv list
reboot

Here we loaded the new (mainline) kernel:

1
2
[root@srv ~]# uname -a
Linux srv.local 4.18.5-1.el7.elrepo.x86_64 #1 SMP Fri Aug 24 11:35:05 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

If you use this method with grub2-set-default you should set in “/etc/default/grub” (before executing the “grub2-mkconfig”):

1
GRUB_DEFAULT=saved

Of course you can just list the kernels with:

1
awk -F\' /^menuentry/{print\$2} /boot/grub2/grub.cfg

and set the ID of the new kernel with “grub2-set-default ID” as above. And then generate the configuration with “grub2-mkconfig”.

STEP 3) Enable the TCP BBR and save the options for the next reboot

1
2
3
4
[root@srv ~]# sysctl -w net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_congestion_control = bbr
#if you kernel is under version 4.9, uncomment the following line
#sysctl -w net.core.default_qdisc=fq

Here you can save the options for the next reboots:

1
2
3
#uncomment the line below if you kernel under version 4.9
#echo 'net.core.default_qdisc=fq' >> /etc/sysctl.d/99-sysctl.conf
echo 'net.ipv4.tcp_congestion_control=bbr' >> /etc/sysctl.d/99-sysctl.conf

Here is the whole output of the above steps

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
[root@srv ~]# yum clean all
Loaded plugins: fastestmirror, langpacks
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
Cleaning repos: base epel extras nginx updates
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Cleaning up list of fastest mirrors
[root@srv ~]# yum -y update
Loaded plugins: fastestmirror, langpacks
base                                                                                                                                                 | 3.6 kB  00:00:00    
epel/x86_64/metalink                                                                                                                                 | 8.3 kB  00:00:00    
epel                                                                                                                                                 | 3.2 kB  00:00:00    
extras                                                                                                                                               | 3.4 kB  00:00:00    
nginx                                                                                                                                                | 2.9 kB  00:00:00    
updates                                                                                                                                              | 3.4 kB  00:00:00    
(1/8): base/7/x86_64/group_gz                                                                                                                        | 166 kB  00:00:00    
(2/8): epel/x86_64/primary                                                                                                                           | 3.6 MB  00:00:00    
(3/8): updates/7/x86_64/primary_db                                                                                                                   | 5.2 MB  00:00:00    
(4/8): extras/7/x86_64/primary_db                                                                                                                    | 186 kB  00:00:00    
(5/8): epel/x86_64/group_gz                                                                                                                          |  88 kB  00:00:00    
(6/8): epel/x86_64/updateinfo                                                                                                                        | 937 kB  00:00:00    
(7/8): nginx/7/x86_64/primary_db                                                                                                                     | 116 kB  00:00:00    
(8/8): base/7/x86_64/primary_db                                                                                                                      | 5.9 MB  00:00:03    
Determining fastest mirrors
 * base: repos-va.psychz.net
 * epel: mirror.cogentco.com
 * extras: mirror.cogentco.com
 * updates: ewr.edge.kernel.org
Resolving Dependencies
--> Running transaction check
---> Package OpenIPMI-modalias.x86_64 0:2.0.19-15.el7 will be updated
---> Package OpenIPMI-modalias.x86_64 0:2.0.23-2.el7 will be an update
---> Package abrt.x86_64 0:2.1.11-48.el7.centos will be updated
---> Package abrt.x86_64 0:2.1.11-50.el7.centos will be an update
....
....
Transaction Summary
============================================================================================================================================================================
Install    2 Packages (+1 Dependent package)
Upgrade  283 Packages
 
Total download size: 311 M
Downloading packages:
No Presto metadata available for nginx
No Presto metadata available for base
(1/2): updates/7/x86_64/prestodelta                                                                                                                  | 538 kB  00:00:00    
(2/2): epel/x86_64/prestodelta                                                                                                                       |  611 B  00:00:00    
Delta RPMs reduced 77 M of updates to 31 M (60% saved)
(1/286): audit-libs-python-2.7.6-3.el7_2.8.1-3.el7_5.1.x86_64.drpm                                                                                   |  34 kB  00:00:00    
(2/286): audit-libs-2.7.6-3.el7_2.8.1-3.el7_5.1.x86_64.drpm                                                                                          |  48 kB  00:00:00    
....
....
 
Complete!
[root@srv ~]# reboot
[root@srv ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
[root@srv ~]# wget http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
--2018-08-29 08:45:41--  http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
Resolving www.elrepo.org (www.elrepo.org)... 69.195.83.87
Connecting to www.elrepo.org (www.elrepo.org)|69.195.83.87|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8656 (8.5K) [application/x-rpm]
Saving to: ‘elrepo-release-7.0-3.el7.elrepo.noarch.rpm’
 
100%[==================================================================================================================================>] 8,656       --.-K/s   in 0s     
 
2018-08-29 08:45:41 (82.5 MB/s) - ‘elrepo-release-7.0-3.el7.elrepo.noarch.rpm’ saved [8656/8656]
 
[root@srv ~]# yum -y install ./elrepo-release-7.0-3.el7.elrepo.noarch.rpm
Loaded plugins: fastestmirror, langpacks
Examining ./elrepo-release-7.0-3.el7.elrepo.noarch.rpm: elrepo-release-7.0-3.el7.elrepo.noarch
Marking ./elrepo-release-7.0-3.el7.elrepo.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package elrepo-release.noarch 0:7.0-3.el7.elrepo will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
============================================================================================================================================================================
 Package                             Arch                        Version                                 Repository                                                    Size
============================================================================================================================================================================
Installing:
 elrepo-release                      noarch                      7.0-3.el7.elrepo                        /elrepo-release-7.0-3.el7.elrepo.noarch                      5.2 k
 
Transaction Summary
============================================================================================================================================================================
Install  1 Package
 
Total size: 5.2 k
Installed size: 5.2 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : elrepo-release-7.0-3.el7.elrepo.noarch                                                                                                                   1/1
  Verifying  : elrepo-release-7.0-3.el7.elrepo.noarch                                                                                                                   1/1
 
Installed:
  elrepo-release.noarch 0:7.0-3.el7.elrepo                                                                                                                                 
 
Complete!
[root@srv ~]#
[root@srv ~]# yum list available --disablerepo='*' --enablerepo=elrepo-kernel
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * elrepo-kernel: reflector.westga.edu
elrepo-kernel                                                                                                                                        | 2.9 kB  00:00:00    
elrepo-kernel/primary_db                                                                                                                             | 1.8 MB  00:00:00    
Available Packages
kernel-lt.x86_64                                                                      4.4.153-1.el7.elrepo                                                     elrepo-kernel
kernel-lt-devel.x86_64                                                                4.4.153-1.el7.elrepo                                                     elrepo-kernel
kernel-lt-doc.noarch                                                                  4.4.153-1.el7.elrepo                                                     elrepo-kernel
kernel-lt-headers.x86_64                                                              4.4.153-1.el7.elrepo                                                     elrepo-kernel
kernel-lt-tools.x86_64                                                                4.4.153-1.el7.elrepo                                                     elrepo-kernel
kernel-lt-tools-libs.x86_64                                                           4.4.153-1.el7.elrepo                                                     elrepo-kernel
kernel-lt-tools-libs-devel.x86_64                                                     4.4.153-1.el7.elrepo                                                     elrepo-kernel
kernel-ml.x86_64                                                                      4.18.5-1.el7.elrepo                                                      elrepo-kernel
kernel-ml-devel.x86_64                                                                4.18.5-1.el7.elrepo                                                      elrepo-kernel
kernel-ml-doc.noarch                                                                  4.18.5-1.el7.elrepo                                                      elrepo-kernel
kernel-ml-headers.x86_64                                                              4.18.5-1.el7.elrepo                                                      elrepo-kernel
kernel-ml-tools.x86_64                                                                4.18.5-1.el7.elrepo                                                      elrepo-kernel
kernel-ml-tools-libs.x86_64                                                           4.18.5-1.el7.elrepo                                                      elrepo-kernel
kernel-ml-tools-libs-devel.x86_64                                                     4.18.5-1.el7.elrepo                                                      elrepo-kernel
perf.x86_64                                                                           4.18.5-1.el7.elrepo                                                      elrepo-kernel
python-perf.x86_64
[root@srv ~]# yum install -y --enablerepo=elrepo-kernel kernel-ml
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: repos-va.psychz.net
 * elrepo: iad.mirror.rackspace.com
 * elrepo-kernel: iad.mirror.rackspace.com
 * epel: mirror.cogentco.com
 * extras: mirror.cogentco.com
 * updates: ewr.edge.kernel.org
elrepo                                                                                                                                               | 2.9 kB  00:00:00    
elrepo/primary_db                                                                                                                                    | 494 kB  00:00:00    
Resolving Dependencies
--> Running transaction check
---> Package kernel-ml.x86_64 0:4.18.5-1.el7.elrepo will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
============================================================================================================================================================================
 Package                               Arch                               Version                                           Repository                                 Size
============================================================================================================================================================================
Installing:
 kernel-ml                             x86_64                             4.18.5-1.el7.elrepo                               elrepo-kernel                              45 M
 
Transaction Summary
============================================================================================================================================================================
Install  1 Package
 
Total download size: 45 M
Installed size: 203 M
Downloading packages:
kernel-ml-4.18.5-1.el7.elrepo.x86_64.rpm                                                                                                             |  45 MB  00:00:01    
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : kernel-ml-4.18.5-1.el7.elrepo.x86_64                                                                                                                     1/1
  Verifying  : kernel-ml-4.18.5-1.el7.elrepo.x86_64                                                                                                                     1/1
 
Installed:
  kernel-ml.x86_64 0:4.18.5-1.el7.elrepo                                                                                                                                   
 
Complete!
[root@srv ~]# awk -F\' /^menuentry/{print\$2} /boot/grub2/grub.cfg
CentOS Linux (4.18.5-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-862.11.6.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-693.21.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-f6659a14d3de4b758a66a90e9d0bf948) 7 (Core)
[root@srv ~]# grub2-set-default 0
[root@srv ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.18.5-1.el7.elrepo.x86_64
Found initrd image: /boot/initramfs-4.18.5-1.el7.elrepo.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-862.11.6.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-862.11.6.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-693.21.1.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-693.21.1.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-693.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-693.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-f6659a14d3de4b758a66a90e9d0bf948
Found initrd image: /boot/initramfs-0-rescue-f6659a14d3de4b758a66a90e9d0bf948.img
done
[root@srv ~]# grub2-editenv list
saved_entry=0
[root@srv ~]# reboot
[root@srv ~]# uname -a
Linux srv.local 4.18.5-1.el7.elrepo.x86_64 #1 SMP Fri Aug 24 11:35:05 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@srv ~]# sysctl -w net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_congestion_control = bbr
[root@srv ~]# sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = bbr
[root@srv ~]# #echo 'net.core.default_qdisc=fq' >> /etc/sysctl.d/99-sysctl.conf
[root@srv ~]# echo 'net.ipv4.tcp_congestion_control=bbr' >> /etc/sysctl.d/99-sysctl.conf
Categories: CentOS 7
Tags: , , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *