Tracking the network latency of the servers’ network is not an easy job. Most monitoring software is capable to monitor the state of the server, but how to monitor the state of the connectivity and the network latency and even the Internet connectivity with some respectful addresses like 1.1.1.1 or 8.8.8.8? It should be easy to do it with ICMP and ping command but using the collectd daemon and one of its plugins offers collectd-ping from https://collectd.org/wiki/index.php/Plugin:Ping to save all the history in a time series back-end and using grafana – https://grafana.com/ (or other graphs/histograms and etc software) to make graphs.
Using the collectd-ping plugin in conjunction with grafana may reach the similar effect as using the old and gold smokeping.
CentOS 7 included the collectd-ping plugin in its official repository, but in CentOS 8 the plugin is missing! Under Cent OS 8 the CentOS SIG OpsTools https://wiki.centos.org/SpecialInterestGroup/OpsTools includes the collectd-ping plugin in their repository. More on SIG and OpsTools may be obtained in the later page. In general, it is safe to use this repository it would not break user’s system.
Here is how to install and configure it. Real grafana examples are also included at the end.
The example here assumes there is a grafana server installed with influxdb backend.
STEP 1) Add OpsTools repository and install the collectd and collectd-ping.
The OpsTools repository is installed with centos-release-opstools package.
Here is what is going to install:
dnf install -y centos-release-opstools dnf install -y collectd collectd-ping
The output of the installation process.
[root@srv ~]# dnf install -y centos-release-opstools Last metadata expiration check: 0:04:50 ago on Tue May 17 10:43:35 2022. Dependencies resolved. ============================================================================================================== Package Architecture Version Repository Size ============================================================================================================== Installing: centos-release-opstools noarch 1-12.el8 extras 10 k Transaction Summary ============================================================================================================== Install 1 Package Total download size: 10 k Installed size: 1.7 k Downloading Packages: centos-release-opstools-1-12.el8.noarch.rpm 46 kB/s | 10 kB 00:00 -------------------------------------------------------------------------------------------------------------- Total 10 kB/s | 10 kB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : centos-release-opstools-1-12.el8.noarch 1/1 Verifying : centos-release-opstools-1-12.el8.noarch 1/1 Installed: centos-release-opstools-1-12.el8.noarch Complete!
Then the installation of collectd-ping is possible:
[root@srv ~]# dnf install -y collectd collectd-ping Last metadata expiration check: 0:00:18 ago on Tue May 17 10:50:19 2022. Dependencies resolved. ============================================================================================================== Package Arch Version Repository Size ============================================================================================================== Installing: collectd x86_64 5.12.0-7.el8s centos-opstools 707 k collectd-ping x86_64 5.12.0-7.el8s centos-opstools 33 k Installing dependencies: liboping x86_64 1.10.0-15.el8s centos-opstools 78 k perl-Carp noarch 1.42-396.el8 baseos 30 k perl-Data-Dumper x86_64 2.167-399.el8 baseos 58 k perl-Digest noarch 1.17-395.el8 appstream 27 k perl-Digest-MD5 x86_64 2.55-396.el8 appstream 37 k perl-Encode x86_64 4:2.97-3.el8 baseos 1.5 M perl-Errno x86_64 1.28-421.el8 baseos 76 k perl-Exporter noarch 5.72-396.el8 baseos 34 k perl-File-Path noarch 2.15-2.el8 baseos 38 k perl-File-Temp noarch 0.230.600-1.el8 baseos 63 k perl-Getopt-Long noarch 1:2.50-4.el8 baseos 63 k perl-HTTP-Tiny noarch 0.074-1.el8 baseos 58 k perl-IO x86_64 1.38-421.el8 baseos 142 k perl-MIME-Base64 x86_64 3.15-396.el8 baseos 31 k perl-Net-SSLeay x86_64 1.88-1.module_el8.4.0+517+be1595ff appstream 379 k perl-PathTools x86_64 3.74-1.el8 baseos 90 k perl-Pod-Escapes noarch 1:1.07-395.el8 baseos 20 k perl-Pod-Perldoc noarch 3.28-396.el8 baseos 86 k perl-Pod-Simple noarch 1:3.35-395.el8 baseos 213 k perl-Pod-Usage noarch 4:1.69-395.el8 baseos 34 k perl-Scalar-List-Utils x86_64 3:1.49-2.el8 baseos 68 k perl-Socket x86_64 4:2.027-3.el8 baseos 59 k perl-Storable x86_64 1:3.11-3.el8 baseos 98 k perl-Term-ANSIColor noarch 4.06-396.el8 baseos 46 k perl-Term-Cap noarch 1.17-395.el8 baseos 23 k perl-Text-ParseWords noarch 3.30-395.el8 baseos 18 k perl-Text-Tabs+Wrap noarch 2013.0523-395.el8 baseos 24 k perl-Time-Local noarch 1:1.280-1.el8 baseos 34 k perl-URI noarch 1.73-3.el8 appstream 116 k perl-Unicode-Normalize x86_64 1.25-396.el8 baseos 82 k perl-constant noarch 1.33-396.el8 baseos 25 k perl-interpreter x86_64 4:5.26.3-421.el8 baseos 6.3 M perl-libnet noarch 3.11-3.el8 appstream 121 k perl-libs x86_64 4:5.26.3-421.el8 baseos 1.6 M perl-macros x86_64 4:5.26.3-421.el8 baseos 72 k perl-parent noarch 1:0.237-1.el8 baseos 20 k perl-podlators noarch 4.11-1.el8 baseos 118 k perl-threads x86_64 1:2.21-2.el8 baseos 61 k perl-threads-shared x86_64 1.58-2.el8 baseos 48 k yajl x86_64 2.1.0-11.el8 appstream 41 k Installing weak dependencies: perl-IO-Socket-IP noarch 0.39-5.el8 appstream 47 k perl-IO-Socket-SSL noarch 2.066-4.module_el8.4.0+517+be1595ff appstream 298 k perl-Mozilla-CA noarch 20160104-7.module_el8.3.0+416+dee7bcef appstream 15 k Enabling module streams: perl 5.26 perl-IO-Socket-SSL 2.066 perl-libwww-perl 6.34 Transaction Summary ============================================================================================================== Install 45 Packages Total download size: 13 M Installed size: 38 M Downloading Packages: (1/45): collectd-ping-5.12.0-7.el8s.x86_64.rpm 72 kB/s | 33 kB 00:00 (2/45): liboping-1.10.0-15.el8s.x86_64.rpm 156 kB/s | 78 kB 00:00 (3/45): perl-Digest-1.17-395.el8.noarch.rpm 212 kB/s | 27 kB 00:00 (4/45): perl-Digest-MD5-2.55-396.el8.x86_64.rpm 407 kB/s | 37 kB 00:00 (5/45): perl-IO-Socket-IP-0.39-5.el8.noarch.rpm 484 kB/s | 47 kB 00:00 (6/45): collectd-5.12.0-7.el8s.x86_64.rpm 950 kB/s | 707 kB 00:00 (7/45): perl-Mozilla-CA-20160104-7.module_el8.3.0+416+dee7bcef.noarch.rpm 193 kB/s | 15 kB 00:00 (8/45): perl-IO-Socket-SSL-2.066-4.module_el8.4.0+517+be1595ff.noarch.rpm 1.5 MB/s | 298 kB 00:00 (9/45): perl-URI-1.73-3.el8.noarch.rpm 785 kB/s | 116 kB 00:00 (10/45): perl-libnet-3.11-3.el8.noarch.rpm 827 kB/s | 121 kB 00:00 (11/45): yajl-2.1.0-11.el8.x86_64.rpm 796 kB/s | 41 kB 00:00 (12/45): perl-Net-SSLeay-1.88-1.module_el8.4.0+517+be1595ff.x86_64.rpm 1.4 MB/s | 379 kB 00:00 (13/45): perl-Carp-1.42-396.el8.noarch.rpm 244 kB/s | 30 kB 00:00 (14/45): perl-Data-Dumper-2.167-399.el8.x86_64.rpm 367 kB/s | 58 kB 00:00 (15/45): perl-Errno-1.28-421.el8.x86_64.rpm 1.1 MB/s | 76 kB 00:00 (16/45): perl-Exporter-5.72-396.el8.noarch.rpm 614 kB/s | 34 kB 00:00 (17/45): perl-File-Path-2.15-2.el8.noarch.rpm 640 kB/s | 38 kB 00:00 (18/45): perl-File-Temp-0.230.600-1.el8.noarch.rpm 746 kB/s | 63 kB 00:00 (19/45): perl-Getopt-Long-2.50-4.el8.noarch.rpm 792 kB/s | 63 kB 00:00 (20/45): perl-HTTP-Tiny-0.074-1.el8.noarch.rpm 342 kB/s | 58 kB 00:00 (21/45): perl-MIME-Base64-3.15-396.el8.x86_64.rpm 249 kB/s | 31 kB 00:00 (22/45): perl-PathTools-3.74-1.el8.x86_64.rpm 861 kB/s | 90 kB 00:00 (23/45): perl-IO-1.38-421.el8.x86_64.rpm 355 kB/s | 142 kB 00:00 (24/45): perl-Pod-Escapes-1.07-395.el8.noarch.rpm 730 kB/s | 20 kB 00:00 (25/45): perl-Pod-Perldoc-3.28-396.el8.noarch.rpm 1.9 MB/s | 86 kB 00:00 (26/45): perl-Encode-2.97-3.el8.x86_64.rpm 2.0 MB/s | 1.5 MB 00:00 (27/45): perl-Pod-Usage-1.69-395.el8.noarch.rpm 62 kB/s | 34 kB 00:00 (28/45): perl-Scalar-List-Utils-1.49-2.el8.x86_64.rpm 129 kB/s | 68 kB 00:00 (29/45): perl-Socket-2.027-3.el8.x86_64.rpm 1.8 MB/s | 59 kB 00:00 (30/45): perl-Pod-Simple-3.35-395.el8.noarch.rpm 349 kB/s | 213 kB 00:00 (31/45): perl-Storable-3.11-3.el8.x86_64.rpm 2.2 MB/s | 98 kB 00:00 (32/45): perl-Term-ANSIColor-4.06-396.el8.noarch.rpm 290 kB/s | 46 kB 00:00 (33/45): perl-Text-ParseWords-3.30-395.el8.noarch.rpm 140 kB/s | 18 kB 00:00 (34/45): perl-Term-Cap-1.17-395.el8.noarch.rpm 137 kB/s | 23 kB 00:00 (35/45): perl-Text-Tabs+Wrap-2013.0523-395.el8.noarch.rpm 1.2 MB/s | 24 kB 00:00 (36/45): perl-Time-Local-1.280-1.el8.noarch.rpm 1.2 MB/s | 34 kB 00:00 (37/45): perl-Unicode-Normalize-1.25-396.el8.x86_64.rpm 2.8 MB/s | 82 kB 00:00 (38/45): perl-constant-1.33-396.el8.noarch.rpm 464 kB/s | 25 kB 00:00 (39/45): perl-macros-5.26.3-421.el8.x86_64.rpm 260 kB/s | 72 kB 00:00 (40/45): perl-parent-0.237-1.el8.noarch.rpm 368 kB/s | 20 kB 00:00 (41/45): perl-podlators-4.11-1.el8.noarch.rpm 815 kB/s | 118 kB 00:00 (42/45): perl-threads-2.21-2.el8.x86_64.rpm 1.1 MB/s | 61 kB 00:00 (43/45): perl-threads-shared-1.58-2.el8.x86_64.rpm 1.2 MB/s | 48 kB 00:00 (44/45): perl-libs-5.26.3-421.el8.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 (45/45): perl-interpreter-5.26.3-421.el8.x86_64.rpm 2.4 MB/s | 6.3 MB 00:02 -------------------------------------------------------------------------------------------------------------- Total 2.0 MB/s | 13 MB 00:06 CentOS-OpsTools - collectd 1.0 MB/s | 1.0 kB 00:00 Importing GPG key 0x51BC2A13: Userid : "CentOS OpsTools SIG (https://wiki.centos.org/SpecialInterestGroup/OpsTools) <security@centos.org>" Fingerprint: 7872 8176 9AD7 3878 85EE A649 4FD9 5327 51BC 2A13 From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-OpsTools Key imported successfully Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : perl-Digest-1.17-395.el8.noarch 1/45 Installing : perl-Digest-MD5-2.55-396.el8.x86_64 2/45 Installing : perl-Data-Dumper-2.167-399.el8.x86_64 3/45 Installing : perl-libnet-3.11-3.el8.noarch 4/45 Installing : perl-Net-SSLeay-1.88-1.module_el8.4.0+517+be1595ff.x86_64 5/45 Installing : perl-URI-1.73-3.el8.noarch 6/45 Installing : perl-Pod-Escapes-1:1.07-395.el8.noarch 7/45 Installing : perl-Mozilla-CA-20160104-7.module_el8.3.0+416+dee7bcef.noarch 8/45 Installing : perl-IO-Socket-IP-0.39-5.el8.noarch 9/45 Installing : perl-Time-Local-1:1.280-1.el8.noarch 10/45 Installing : perl-IO-Socket-SSL-2.066-4.module_el8.4.0+517+be1595ff.noarch 11/45 Installing : perl-Term-ANSIColor-4.06-396.el8.noarch 12/45 Installing : perl-Term-Cap-1.17-395.el8.noarch 13/45 Installing : perl-File-Temp-0.230.600-1.el8.noarch 14/45 Installing : perl-Pod-Simple-1:3.35-395.el8.noarch 15/45 Installing : perl-HTTP-Tiny-0.074-1.el8.noarch 16/45 Installing : perl-podlators-4.11-1.el8.noarch 17/45 Installing : perl-Pod-Perldoc-3.28-396.el8.noarch 18/45 Installing : perl-Text-ParseWords-3.30-395.el8.noarch 19/45 Installing : perl-Pod-Usage-4:1.69-395.el8.noarch 20/45 Installing : perl-MIME-Base64-3.15-396.el8.x86_64 21/45 Installing : perl-Storable-1:3.11-3.el8.x86_64 22/45 Installing : perl-Getopt-Long-1:2.50-4.el8.noarch 23/45 Installing : perl-Errno-1.28-421.el8.x86_64 24/45 Installing : perl-Socket-4:2.027-3.el8.x86_64 25/45 Installing : perl-Encode-4:2.97-3.el8.x86_64 26/45 Installing : perl-Carp-1.42-396.el8.noarch 27/45 Installing : perl-Exporter-5.72-396.el8.noarch 28/45 Installing : perl-libs-4:5.26.3-421.el8.x86_64 29/45 Installing : perl-Scalar-List-Utils-3:1.49-2.el8.x86_64 30/45 Installing : perl-parent-1:0.237-1.el8.noarch 31/45 Installing : perl-macros-4:5.26.3-421.el8.x86_64 32/45 Installing : perl-Text-Tabs+Wrap-2013.0523-395.el8.noarch 33/45 Installing : perl-Unicode-Normalize-1.25-396.el8.x86_64 34/45 Installing : perl-File-Path-2.15-2.el8.noarch 35/45 Installing : perl-IO-1.38-421.el8.x86_64 36/45 Installing : perl-PathTools-3.74-1.el8.x86_64 37/45 Installing : perl-constant-1.33-396.el8.noarch 38/45 Installing : perl-threads-1:2.21-2.el8.x86_64 39/45 Installing : perl-threads-shared-1.58-2.el8.x86_64 40/45 Installing : perl-interpreter-4:5.26.3-421.el8.x86_64 41/45 Installing : liboping-1.10.0-15.el8s.x86_64 42/45 Installing : yajl-2.1.0-11.el8.x86_64 43/45 Installing : collectd-5.12.0-7.el8s.x86_64 44/45 Running scriptlet: collectd-5.12.0-7.el8s.x86_64 44/45 Installing : collectd-ping-5.12.0-7.el8s.x86_64 45/45 Running scriptlet: collectd-ping-5.12.0-7.el8s.x86_64 45/45 Verifying : collectd-5.12.0-7.el8s.x86_64 1/45 Verifying : collectd-ping-5.12.0-7.el8s.x86_64 2/45 Verifying : liboping-1.10.0-15.el8s.x86_64 3/45 Verifying : perl-Digest-1.17-395.el8.noarch 4/45 Verifying : perl-Digest-MD5-2.55-396.el8.x86_64 5/45 Verifying : perl-IO-Socket-IP-0.39-5.el8.noarch 6/45 Verifying : perl-IO-Socket-SSL-2.066-4.module_el8.4.0+517+be1595ff.noarch 7/45 Verifying : perl-Mozilla-CA-20160104-7.module_el8.3.0+416+dee7bcef.noarch 8/45 Verifying : perl-Net-SSLeay-1.88-1.module_el8.4.0+517+be1595ff.x86_64 9/45 Verifying : perl-URI-1.73-3.el8.noarch 10/45 Verifying : perl-libnet-3.11-3.el8.noarch 11/45 Verifying : yajl-2.1.0-11.el8.x86_64 12/45 Verifying : perl-Carp-1.42-396.el8.noarch 13/45 Verifying : perl-Data-Dumper-2.167-399.el8.x86_64 14/45 Verifying : perl-Encode-4:2.97-3.el8.x86_64 15/45 Verifying : perl-Errno-1.28-421.el8.x86_64 16/45 Verifying : perl-Exporter-5.72-396.el8.noarch 17/45 Verifying : perl-File-Path-2.15-2.el8.noarch 18/45 Verifying : perl-File-Temp-0.230.600-1.el8.noarch 19/45 Verifying : perl-Getopt-Long-1:2.50-4.el8.noarch 20/45 Verifying : perl-HTTP-Tiny-0.074-1.el8.noarch 21/45 Verifying : perl-IO-1.38-421.el8.x86_64 22/45 Verifying : perl-MIME-Base64-3.15-396.el8.x86_64 23/45 Verifying : perl-PathTools-3.74-1.el8.x86_64 24/45 Verifying : perl-Pod-Escapes-1:1.07-395.el8.noarch 25/45 Verifying : perl-Pod-Perldoc-3.28-396.el8.noarch 26/45 Verifying : perl-Pod-Simple-1:3.35-395.el8.noarch 27/45 Verifying : perl-Pod-Usage-4:1.69-395.el8.noarch 28/45 Verifying : perl-Scalar-List-Utils-3:1.49-2.el8.x86_64 29/45 Verifying : perl-Socket-4:2.027-3.el8.x86_64 30/45 Verifying : perl-Storable-1:3.11-3.el8.x86_64 31/45 Verifying : perl-Term-ANSIColor-4.06-396.el8.noarch 32/45 Verifying : perl-Term-Cap-1.17-395.el8.noarch 33/45 Verifying : perl-Text-ParseWords-3.30-395.el8.noarch 34/45 Verifying : perl-Text-Tabs+Wrap-2013.0523-395.el8.noarch 35/45 Verifying : perl-Time-Local-1:1.280-1.el8.noarch 36/45 Verifying : perl-Unicode-Normalize-1.25-396.el8.x86_64 37/45 Verifying : perl-constant-1.33-396.el8.noarch 38/45 Verifying : perl-interpreter-4:5.26.3-421.el8.x86_64 39/45 Verifying : perl-libs-4:5.26.3-421.el8.x86_64 40/45 Verifying : perl-macros-4:5.26.3-421.el8.x86_64 41/45 Verifying : perl-parent-1:0.237-1.el8.noarch 42/45 Verifying : perl-podlators-4.11-1.el8.noarch 43/45 Verifying : perl-threads-1:2.21-2.el8.x86_64 44/45 Verifying : perl-threads-shared-1.58-2.el8.x86_64 45/45 Installed: collectd-5.12.0-7.el8s.x86_64 collectd-ping-5.12.0-7.el8s.x86_64 liboping-1.10.0-15.el8s.x86_64 perl-Carp-1.42-396.el8.noarch perl-Data-Dumper-2.167-399.el8.x86_64 perl-Digest-1.17-395.el8.noarch perl-Digest-MD5-2.55-396.el8.x86_64 perl-Encode-4:2.97-3.el8.x86_64 perl-Errno-1.28-421.el8.x86_64 perl-Exporter-5.72-396.el8.noarch perl-File-Path-2.15-2.el8.noarch perl-File-Temp-0.230.600-1.el8.noarch perl-Getopt-Long-1:2.50-4.el8.noarch perl-HTTP-Tiny-0.074-1.el8.noarch perl-IO-1.38-421.el8.x86_64 perl-IO-Socket-IP-0.39-5.el8.noarch perl-IO-Socket-SSL-2.066-4.module_el8.4.0+517+be1595ff.noarch perl-MIME-Base64-3.15-396.el8.x86_64 perl-Mozilla-CA-20160104-7.module_el8.3.0+416+dee7bcef.noarch perl-Net-SSLeay-1.88-1.module_el8.4.0+517+be1595ff.x86_64 perl-PathTools-3.74-1.el8.x86_64 perl-Pod-Escapes-1:1.07-395.el8.noarch perl-Pod-Perldoc-3.28-396.el8.noarch perl-Pod-Simple-1:3.35-395.el8.noarch perl-Pod-Usage-4:1.69-395.el8.noarch perl-Scalar-List-Utils-3:1.49-2.el8.x86_64 perl-Socket-4:2.027-3.el8.x86_64 perl-Storable-1:3.11-3.el8.x86_64 perl-Term-ANSIColor-4.06-396.el8.noarch perl-Term-Cap-1.17-395.el8.noarch perl-Text-ParseWords-3.30-395.el8.noarch perl-Text-Tabs+Wrap-2013.0523-395.el8.noarch perl-Time-Local-1:1.280-1.el8.noarch perl-URI-1.73-3.el8.noarch perl-Unicode-Normalize-1.25-396.el8.x86_64 perl-constant-1.33-396.el8.noarch perl-interpreter-4:5.26.3-421.el8.x86_64 perl-libnet-3.11-3.el8.noarch perl-libs-4:5.26.3-421.el8.x86_64 perl-macros-4:5.26.3-421.el8.x86_64 perl-parent-1:0.237-1.el8.noarch perl-podlators-4.11-1.el8.noarch perl-threads-1:2.21-2.el8.x86_64 perl-threads-shared-1.58-2.el8.x86_64 yajl-2.1.0-11.el8.x86_64 Complete!
STEP 2) Configure and use collectd and colletcd-ping
. The example here uses the network plugin to send the information collectd collects to influxdb on IP: 192.168.0.100, but in fact, collectd may be used with many other back-end servers like Prometheus, Graphite, Mysql and etc, and even the very collectd daemon may store the data locally in rrd format!
Here are the all collectd-ping options explained – https://collectd.org/documentation/manpages/collectd.conf.5.shtml#plugin_ping.
In the main /etc/collectd.conf:
- Uncomment the with loading the collectd-ping plugin.
- Configure the collectd-ping plugin in a section starting with <Plugin ping> and ending with </Plugin ping>
Here is just this part of the configuration (/etc/collectd.conf):
.....
LoadPlugin ping
.....
.....
<Plugin ping>
Host "srv1.example.com"
Host "srv3.example.com"
Host "srv4.example.com"
Host "srv6.example.com"
Host "srv8.example.com"
Host "srv9.example.com"
Host "srv10.example.com"
Interval 1.0
Timeout 0.9
TTL 255
# SourceAddress "1.2.3.4"
# AddressFamily "any"
# Device "eth0"
# MaxMissed -1
</Plugin ping>
Then start the collectd daemon.
[root@srv ~]# systemctl start collectd
[root@srv ~]# systemctl status collectd
● collectd.service - Collectd statistics daemon
Loaded: loaded (/usr/lib/systemd/system/collectd.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2022-05-17 11:10:35 UTC; 2s ago
Docs: man:collectd(1)
man:collectd.conf(5)
Main PID: 3697 (collectd)
Tasks: 11 (limit: 11381)
Memory: 1.1M
CGroup: /system.slice/collectd.service
└─3697 /usr/sbin/collectd
STEP 3) Grafana example graphs.
SCREENSHOT 1) There are two example graphs – with the pings time to live and how many drops are counted per 10 seconds.
The legends includes the Mean, Last, Max and Min values. Grafana and influxdb may show more sophisticated processing like standard deviation and etc.
SCREENSHOT 2) A selected server to show only one server ping graph.
The srv19 graph is selected and shown. The last 6 hours the latency is quite unstable and varies a lot. The bottom graph show the ping drops per 10 seconds with ping interval of 1 second, so 0.100 means 1 ping drop and 0.200 means 2 ping drops per 10 seconds.
