systemd | Any IT here? Help Me!

Automatically start zram swap device on boot with systemd zram-generator under CentOS Stream 9

zram-generator package will install necessary tools to automate the creation on boot the compressed RAM devices. This article focuses on compressed swap devices.

As of writing this article, the latest version in the package system under CentOS Stream 9 is 0.32. The the latest version on the original page of the software is much higher number 1.1.2 and many of the following configuration options are marked as OBSOLETE, but they work in the 0.32 version included in CentOS Stream 9 (and the new configuration options does not!). That’s why it is important to check the included sample configuration file.
The package installs no configuration file, just a sample configuration file – /usr/share/doc/zram-generator/zram-generator.conf.example.
zram in the kernel space – https://docs.kernel.org/admin-guide/blockdev/zram.html

STEP 1) Install the zram-generator

It is easy and straightforward, just a single package:

[root@srv) ~]# dnf install -y zram-generator
Last metadata expiration check: 3:42:20 ago on Fri 20 Oct 2023 05:18:32 AM UTC.
Dependencies resolved.
==============================================================================================================
 Package                      Architecture         Version                      Repository               Size
==============================================================================================================
Installing:
 zram-generator               x86_64               0.3.2-7.el9                  appstream               409 k

Transaction Summary
==============================================================================================================
Install  1 Package

Total download size: 409 k
Installed size: 983 k
Downloading Packages:
zram-generator-0.3.2-7.el9.x86_64.rpm                                         2.1 MB/s | 409 kB     00:00    
--------------------------------------------------------------------------------------------------------------
Total                                                                         1.3 MB/s | 409 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                      1/1 
  Installing       : zram-generator-0.3.2-7.el9.x86_64                                                    1/1 
  Running scriptlet: zram-generator-0.3.2-7.el9.x86_64                                                    1/1 
  Verifying        : zram-generator-0.3.2-7.el9.x86_64                                                    1/1 

Installed:
  zram-generator-0.3.2-7.el9.x86_64                                                                           

Complete!

STEP 2) Create a configuration and start the service

By default, there is no configuration file. The best place for it is /etc/systemd/zram-generator.conf.
Here is an example, which will fulfill the following:

Set the maximum RAM of the system, because it is a hard-coded 9G, by default. And if not redefined the device will be with 9G max memory no matter how much is tuned by the other options.
Set the RAM of the compressed device.
Set the fraction of the ram, which may be used by the device. Again as with the above configuration option, it is important, because it limits the maximum available memory to allocate for the compressed device.
Set the type of the device – filesystem or swap device.

There is the real world example configuration – /etc/systemd/zram-generator.conf

[root@srv ~]# cat /etc/systemd/zram-generator.conf
[zram0]
host-memory-limit = none
max-zram-size = 32768
zram-fraction = 1.0
compression-algorithm = zstd
swap-priority = 100
fs-type = swap

Keep on reading!

Edit with systemctl edit to add restart on fail to a service – nfs-ganesha

A quick tip how to edit a service unit file under a c system like CentOS Stream 9 or Ubuntu. The best way is to edit it with the the tool “systemctl edit [service_name]”, which will trigger the default editor to open a temporary copy of the systemd unit file with the service name used with it. The default editor in the console is controlled by “EDITOR” variable and may be changed prior using the systemctl edit. After a successful manipulation of the system unit file the new one will be installed and a reload of the systemd unit files will be triggered with “systemctl daemon-reload” automatically. Indeed, it is just a text edit of a text file, which will do several actions when using “systemctl edit” command.

systemd options ro restart a service on fail are:

[Service]
Restart=on-failure
RestartSec=5s

Here, the example is to add a restart-on-fail functionality to the nfs-ganesha service (NFS service). The systemctl edit may be used for many other changes to the systemd unit file under the console and it is the easiest and proper way.

SCREENSHOT 1) Use “systemctl edit” to edit a copy of the systemd override unit file.

do not insert anything at the end of the comments or below the second red line comments – “### Lines below this comment will be discarded”. This temporary override file includes a systemd unit file of the service, which is opened for editing. The result override.conf file will only include the added lines, no other comments shown below the second red line.

Keep on reading!

Run podman/docker InfluxDB 1.8 container to collect statistics from collectd

Yet another article on the topic of the InfluxDB 1.8 and collectd gathering statistics, in continuation of the articles Monitor and analyze with Grafana, influxdb 1.8 and collectd under Ubuntu 22.04 LTS and Monitor and analyze with Grafana, influxdb 1.8 and collectd under CentOS Stream 9. This time, the InfluxDB runs in a container created with podman or docker software.

Here are the important points to mind when running InfluxDB 1.8 in a docker/podman container:
Keep on reading!

Save iptables rules over reboots on Ubuntu 16 and Ubuntu 18 – persistent iptables rules

Moving towards the firewalld software and especially the systemd some good old init scripts got missing! For example, one of those good scripts is the init script for iptables firewall, which allows saving iptables rules and during boot, it loads them again. With the init iptables script we have persistence of the iptables rules. Meanwhile, we can always call the init script with “save” argument to update the currently saved rules. Many different Linux distributions have this init script – “/etc/init.d/iptables”, but in systemd world, it has been removed and replaced with nothing (probably, because you are encouraged to use firewalld, which is not a bad thing!).

There are two packages “iptables-persistent” and “netfilter-persistent”, which work together to have iptables persistence over reboots. The rules are saved and restored automatically during system startup.

First, install “iptables-persistent” and “netfilter-persistent” with

sudo apt install netfilter-persistent iptables-persistent

During the iptables–persistent installation the setup asks the user to save the current iptables rules. Hit “Yes” if you want to save the current iptables rules, which will be automatically loaded the next time the system starts up.

So it is safe to install it on a live system – the current iptables rules won’t be deleted.
Second, ensure the boot script to restore the iptables rules is enabled

sudo systemctl enable netfilter-persistent

Additional information

Saving the current state of the iptables rules:

myuser@myubuntupc:~$ sudo /usr/sbin/netfilter-persistent save
run-parts: executing /usr/share/netfilter-persistent/plugins.d/15-ip4tables save
run-parts: executing /usr/share/netfilter-persistent/plugins.d/25-ip6tables save

Restore the original state of the iptables rules:

sudo systemctl restart netfilter-persistent

And all commands you can do – start, stop, restart, reload, flush, save. You can use the script directly (it is not mandatory to use systemctl to restart, i.e. restore rules and etc.)

myuser@myubuntupc:~$ sudo /usr/sbin/netfilter-persistent
Usage: /usr/sbin/netfilter-persistent (start|stop|restart|reload|flush|save)

The script netfilter-persistent executes 2 other scripts as plugins:

/usr/share/netfilter-persistent/plugins.d/15-ip4tables
/usr/share/netfilter-persistent/plugins.d/25-ip6tables

The iptables rules are saved respectively in files

/etc/iptables/rules.v4
/etc/iptables/rules.v6

And you can always edit them manually or save/restore with iptables-save and iptables-restore redirecting the output to the above files.

It’s normal the state of the “active (exited)”. The service is “enabled” as you can see (by default the setup automatically enables the service on Ubuntu, but always check it to be sure, it’s the firewall!).

myuser@myubuntupc:~$ sudo systemctl status netfilter-persistent
● netfilter-persistent.service - netfilter persistent configuration
   Loaded: loaded (/lib/systemd/system/netfilter-persistent.service; enabled; vendor preset: enabled)
   Active: active (exited) since Thu 2019-01-17 20:44:08 EST; 14min ago
 Main PID: 666 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/netfilter-persistent.service

Jan 17 20:44:08 myubuntupc systemd[1]: Starting netfilter persistent configuration...
Jan 17 20:44:08 myubuntupc netfilter-persistent[666]: run-parts: executing /usr/share/netfilter-persistent/plugins.d/15-ip4tables start
Jan 17 20:44:08 myubuntupc netfilter-persistent[666]: run-parts: executing /usr/share/netfilter-persistent/plugins.d/25-ip6tables start
Jan 17 20:44:08 myubuntupc systemd[1]: Started netfilter persistent configuration.

Make systemd to save logs on the disk

On some Linux distributions, systemd log files are not saved on your disk, but only temporary in the memory and when you reboot all logs are discarded. So the systemd logs are not persistent, which could lead to missing important information if you want to check them when you are booted in a rescue disk or even if you just reboot your server. for exmaple,

if some important service failed to boot and your server is unreachable and you boot in rescue CD you do not have logs to check why the service failed and the (error) output of the process of starting the services!

Here is how you can enable the systemd logs to be persistent i.e. save them on the disk. This is tested on CentOS 7, which by default saves the systemd logs on memory!

STEP 1) Prepare the systemd log directory

mkdir -p /var/log/journal/
systemd-tmpfiles --create --prefix /var/log/journal/

STEP 2) Edit systemd configuration and reload the daemon

And ensure your configuration uses “Storage=persistent” in /etc/systemd/journald.conf

grep Storage /etc/systemd/journald.conf
Storage=persistent
systemctl restart systemd-journald

The last line with systemctl restart could be replace with

killall -USR1 systemd-journald

if you do not want to lose all your current logs in memory!

Bonus – systemd logs from multiple reboots

Here we have logs from 5 reboots. Here you can also see what are the right owner (systemd-journal) and Selinux labels of the “/var/log/journal/”

[root@srv ~]# ls -altrZ /var/log/journal/
drwxr-sr-x+ root systemd-journal system_u:object_r:var_log_t:s0   dbd91181db6b4c9f900d9b3a1651a8d5
drwxr-sr-x+ root systemd-journal system_u:object_r:var_log_t:s0   .
drwxr-xr-x. root root            system_u:object_r:var_log_t:s0   ..
[root@srv ~]# journalctl --disk-usage
Archived and active journals take up 112.0M on disk.
[root@srv ~]# journalctl --list-boots
-4 ec4146b78ac944b8a8d4116f259e09ee Thu 2019-06-06 23:39:14 UTC—Thu 2019-06-06 23:39:37 UTC
-3 ae3d39db626c4592aa84cc68072fbb32 Thu 2019-06-06 23:41:03 UTC—Thu 2019-06-06 23:42:13 UTC
-2 68c1ca07c05b4d59adcc9888c50f4065 Thu 2019-06-06 23:42:57 UTC—Fri 2019-06-07 00:13:27 UTC
-1 f7e8da6aaa8740faa05c4985c92023fd Fri 2019-06-07 00:14:08 UTC—Fri 2019-06-07 00:16:33 UTC
 0 45c00dc29e1a48298d9f87f5421468b4 Fri 2019-06-07 00:17:13 UTC—Mon 2019-06-10 01:39:17 UTC
[root@srv ~]# journalctl --boot=-2
-- Logs begin at Thu 2019-06-06 23:39:14 UTC, end at Mon 2019-06-10 01:39:17 UTC. --
Jun 06 23:42:57 srv systemd-journal[133]: Runtime journal is using 8.0M (max allowed 1.5G, trying to leave 2.3G free of 15.6G available → current limit 1.5G).
Jun 06 23:42:57 srv kernel: microcode: microcode updated early to revision 0x710, date = 2013-06-17
Jun 06 23:42:57 srv kernel: Initializing cgroup subsys cpuset
Jun 06 23:42:57 srv kernel: Initializing cgroup subsys cpu
Jun 06 23:42:57 srv kernel: Initializing cgroup subsys cpuacct
Jun 06 23:42:57 srv kernel: Linux version 3.10.0-514.10.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 S
Jun 06 23:42:57 srv kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7.x86_64 root=UUID=c9bec791-c77d-4189-b18a-9ddc728ee782 ro crashkernel=auto r
Jun 06 23:42:57 srv kernel: e820: BIOS-provided physical RAM map:
....
....
[root@srv ~]# journalctl --boot=-2 -u auditd
-- Logs begin at Thu 2019-06-06 23:39:14 UTC, end at Mon 2019-06-10 01:50:18 UTC. --
Jun 06 23:43:05 srv systemd[1]: Starting Security Auditing Service...
Jun 06 23:43:05 srv auditd[694]: Started dispatcher: /sbin/audispd pid: 698
Jun 06 23:43:05 srv audispd[698]: priority_boost_parser called with: 4
Jun 06 23:43:05 srv audispd[698]: max_restarts_parser called with: 10
Jun 06 23:43:05 srv audispd[698]: audispd initialized with q_depth=150 and 1 active plugins
Jun 06 23:43:05 srv augenrules[695]: /sbin/augenrules: No change
Jun 06 23:43:05 srv auditd[694]: Init complete, auditd 2.6.5 listening for events (startup state enable)
Jun 06 23:43:05 srv augenrules[695]: No rules
Jun 06 23:43:05 srv augenrules[695]: enabled 1
Jun 06 23:43:05 srv augenrules[695]: failure 1
Jun 06 23:43:05 srv augenrules[695]: pid 694
Jun 06 23:43:05 srv augenrules[695]: rate_limit 0
Jun 06 23:43:05 srv augenrules[695]: backlog_limit 320
Jun 06 23:43:05 srv augenrules[695]: lost 0
Jun 06 23:43:05 srv augenrules[695]: backlog 1
Jun 06 23:43:05 srv systemd[1]: Started Security Auditing Service.
Jun 06 23:56:48 srv auditd[694]: The audit daemon is exiting.
Jun 06 23:56:49 srv systemd[1]: Starting Security Auditing Service...
Jun 06 23:56:49 srv auditd[24744]: Started dispatcher: /sbin/audispd pid: 24746
Jun 06 23:56:49 srv audispd[24746]: audispd initialized with q_depth=250 and 1 active plugins
Jun 06 23:56:49 srv auditd[24744]: Init complete, auditd 2.8.4 listening for events (startup state enable)
Jun 06 23:56:49 srv augenrules[24750]: /sbin/augenrules: No change
Jun 06 23:56:49 srv augenrules[24750]: No rules
Jun 06 23:56:49 srv augenrules[24750]: enabled 1
Jun 06 23:56:49 srv augenrules[24750]: failure 1
Jun 06 23:56:49 srv augenrules[24750]: pid 24744
Jun 06 23:56:49 srv augenrules[24750]: rate_limit 0
Jun 06 23:56:49 srv augenrules[24750]: backlog_limit 320
Jun 06 23:56:49 srv augenrules[24750]: lost 0
Jun 06 23:56:49 srv augenrules[24750]: backlog 1
Jun 06 23:56:49 srv systemd[1]: Started Security Auditing Service.
Jun 07 00:13:26 srv systemd[1]: Stopping Security Auditing Service...
Jun 07 00:13:26 srv systemd[1]: Stopped Security Auditing Service.

Now you have logs of your booting process!

The systemd log files are accessible even if you’ve booted from a rescue CD and you chroot in your system!

Be careful with the disk free space when using disk storage for your systemd logs – Clear or delete systemd logs.

simple time synchronization of a server (laptop, desktop) using built-in systemd-timesyncd service

Here we offer you a relatively new way of keeping your server’s time (or your computer and laptop) synchronized with a reliable time service on the Internet.

systemd has a built-in feature – a small daemon (systemd-timesyncd) to periodically to contact NTP servers and keep the server’s clock synchronized with them!

Of course, you must use systemd in your Linux distribution. This article is for those Linux systems using systemd, not for upstart (sysvinit, openrc, upstart, runit and so on). Most of the modern Linux distributions use the systemd like Fedora, Ubuntu, CentOS, RedHat, Gentoo, SuSe and many more.

Once there were not many options to keep your server’s clock synced with NTP servers. Now we have simpler programs (some of which by the way could act as clients only!!!) – chrony, openntpd, systemd-timesyncd and more.
This time synchronization service is not going to open server port 123, it does not have the server capabilities of an NTP server. So you won’t need any firewall rules (like for ntpd). It is a simple client service to sync your time and keep it synchronized all the time with accuracy not more than 100ms.

Do not expect complex clock discipline like training or compensating. It just sets the time according to a selected time server from the configuration file in “/etc/systemd/timesyncd.conf”. The polling interval is automatically adjusted in minimal and maximal values from the configuration file and the daemon decides which is the actual interval based on the near-term drift it thinks. Possible back running clock if it needs to set in the past. The quality of the clock source could not be checked, so

in any case, you may not expect more than 100ms accuracy.

Of course, this service is actively developed and it has already many changes from the base client once it was!

Here is how you can enable it. Here are the steps:
Keep on reading!

systemd service freezes in activating (start-post) status – mysqld or other services

We’ve experienced this with the MySQL server under CentOS 7, but you can have this state with other services!
After updating our MySQL we tried to start it up, but the service got this strange state after “systemctl start” returned:

[root@mysql2 ~]# systemctl start mysqld
Job for mysqld.service failed because a timeout was exceeded. See "systemctl status mysqld.service" and "journalctl -xe" for details.

The timeout is big it’s something like 5 to 10 minutes and so it is typical (do not do it!) to type “ctrl+c” and you end up without this message and a strange state of the mysql:

[root@mysql2 ~]# systemctl status mysqld
● mysqld.service - MySQL Community Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: activating (start-post) since Fri 2018-11-09 09:00:55 UTC; 6min ago
  Process: 8333 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
  Process: 8321 ExecStartPre=/usr/bin/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
 Main PID: 8333 (code=exited, status=0/SUCCESS);         : 8334 (mysql-systemd-s)
   CGroup: /user.slice/user-0.slice/session-2395.scope/system.slice/mysqld.service
           └─control
             ├─ 8334 /bin/bash /usr/bin/mysql-systemd-start post
             └─10152 sleep 1

Nov 09 09:00:55 mysql2.mytv.bg systemd[1]: Starting MySQL Community Server...
Nov 09 09:00:56 mysql2.mytv.bg mysqld_safe[8333]: 181109 09:00:56 mysqld_safe Logging to '/var/log/mysqld.log'.
Nov 09 09:00:56 mysql2.mytv.bg mysqld_safe[8333]: 181109 09:00:56 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql

Meanwhile with “pstree”:

[root@mysql2 ~]# pstree
systemd─┬─agetty
        ├─crond
        ├─dbus-daemon
        ├─mysql-systemd-s───sleep
        ├─rsyslogd───2*[{rsyslogd}]
        ├─sshd─┬─sshd───bash───systemctl─┬─systemctl
        │      │                         └─systemd-tty-ask
        │      └─sshd───bash───pstree
        ├─systemd-journal
        └─systemd-logind

So as you can see no mysqld process! Apparently systemctl had tried to start MySQL server process and it failed.
So the first thing to do was to check the MySQL logs. In our case it was a obsolete option in my.cnf:

2018-11-09 09:10:57 11384 [ERROR] /usr/sbin/mysqld: unknown variable 'default-character-set=utf8'
2018-11-09 09:10:57 11384 [ERROR] Aborting

The interesting part is that

the service got “Active: activating (start-post)” and when you fix the problem you cannot “systemctl start mysqld” it just start to wait for the current timeout.

In fact this state means “I’m trying to start the service…” and it is in an endless loop to start the service and if you the service has a big start timeout like 5-10 minutes you must wait for the next iteration of the loop to start the service successfully (if you fixed the problem!). And if you want not to wait you must execute first stop to the service and then start – you’ll not wait for any timeout and you can check immediately if the service was started successfully:

[root@mysql2 ~]# systemctl status mysqld
● mysqld.service - MySQL Community Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: activating (start-post) since Fri 2018-11-09 09:20:56 UTC; 2min 50s ago
  Process: 13208 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
  Process: 13196 ExecStartPre=/usr/bin/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
 Main PID: 13208 (code=exited, status=0/SUCCESS);         : 13209 (mysql-systemd-s)
   CGroup: /user.slice/user-0.slice/session-2395.scope/system.slice/mysqld.service
           └─control
             ├─13209 /bin/bash /usr/bin/mysql-systemd-start post
             └─14357 sleep 1

Nov 09 09:20:56 mysql2.mytv.bg systemd[1]: Starting MySQL Community Server...
Nov 09 09:20:56 mysql2.mytv.bg mysqld_safe[13208]: 181109 09:20:56 mysqld_safe Logging to '/var/log/mysqld.log'.
Nov 09 09:20:56 mysql2.mytv.bg mysqld_safe[13208]: 181109 09:20:56 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
[root@mysql2 ~]# systemctl start mysqld
Job for mysqld.service failed because a timeout was exceeded. See "systemctl status mysqld.service" and "journalctl -xe" for details.
[root@mysql2 ~]# systemctl status mysqld
● mysqld.service - MySQL Community Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-11-09 09:30:59 UTC; 2s ago
  Process: 15656 ExecStartPost=/usr/bin/mysql-systemd-start post (code=exited, status=0/SUCCESS)
  Process: 15643 ExecStartPre=/usr/bin/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
 Main PID: 15655 (mysqld_safe)
   CGroup: /user.slice/user-0.slice/session-2395.scope/system.slice/mysqld.service
           ├─15655 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
           └─16243 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mysqld.log --open-files-limit=10000...

Nov 09 09:30:56 mysql2.mytv.bg systemd[1]: Starting MySQL Community Server...
Nov 09 09:30:57 mysql2.mytv.bg mysqld_safe[15655]: 181109 09:30:57 mysqld_safe Logging to '/var/log/mysqld.log'.
Nov 09 09:30:57 mysql2.mytv.bg mysqld_safe[15655]: 181109 09:30:57 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Nov 09 09:30:59 mysql2.mytv.bg systemd[1]: Started MySQL Community Server.

As you can see we even received error again that the service cannot be started and immediately after that the service status is in normal “active (running)” state! And we waited for around 10 minutes! You can see the times in the logs above.
So to summarize it up:

If you have a service in “activating (start-post)” the service cannot be started because of an error, check and fix the problem and then issue “stop and start”:

[root@mysql2 ~]# systemctl start mysqld
Job for mysqld.service failed because a timeout was exceeded. See "systemctl status mysqld.service" and "journalctl -xe" for details.
[root@mysql2 ~]# systemctl status mysqld
● mysqld.service - MySQL Community Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: activating (start-post) since Fri 2018-11-09 10:05:20 UTC; 2min 17s ago
  Process: 23601 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
  Process: 23589 ExecStartPre=/usr/bin/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
 Main PID: 23601 (code=exited, status=0/SUCCESS);         : 23602 (mysql-systemd-s)
   CGroup: /user.slice/user-0.slice/session-2395.scope/system.slice/mysqld.service
           └─control
             ├─23602 /bin/bash /usr/bin/mysql-systemd-start post
             └─24646 sleep 1

Nov 09 10:05:20 mysql2.mytv.bg systemd[1]: Starting MySQL Community Server...
Nov 09 10:05:21 mysql2.mytv.bg mysqld_safe[23601]: 181109 10:05:21 mysqld_safe Logging to '/var/log/mysqld.log'.
Nov 09 10:05:21 mysql2.mytv.bg mysqld_safe[23601]: 181109 10:05:21 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
[root@mysql2 ~]# systemctl stop mysqld
[root@mysql2 ~]# systemctl status mysqld
● mysqld.service - MySQL Community Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Fri 2018-11-09 10:07:52 UTC; 4s ago
  Process: 23602 ExecStartPost=/usr/bin/mysql-systemd-start post (code=killed, signal=TERM)
  Process: 23601 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
  Process: 23589 ExecStartPre=/usr/bin/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
 Main PID: 23601 (code=exited, status=0/SUCCESS)

Nov 09 10:05:20 mysql2.mytv.bg systemd[1]: Starting MySQL Community Server...
Nov 09 10:05:21 mysql2.mytv.bg mysqld_safe[23601]: 181109 10:05:21 mysqld_safe Logging to '/var/log/mysqld.log'.
Nov 09 10:05:21 mysql2.mytv.bg mysqld_safe[23601]: 181109 10:05:21 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Nov 09 10:07:52 mysql2.mytv.bg systemd[1]: Stopped MySQL Community Server.
[root@mysql2 ~]# systemctl start mysqld
[root@mysql2 ~]# systemctl status mysqld
● mysqld.service - MySQL Community Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-11-09 10:08:06 UTC; 3s ago
  Process: 24711 ExecStartPost=/usr/bin/mysql-systemd-start post (code=exited, status=0/SUCCESS)
  Process: 24698 ExecStartPre=/usr/bin/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
 Main PID: 24710 (mysqld_safe)
   CGroup: /user.slice/user-0.slice/session-2395.scope/system.slice/mysqld.service
           ├─24710 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
           └─25298 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mysqld.log --open-files-limit=10000...

Nov 09 10:08:04 mysql2.mytv.bg systemd[1]: Starting MySQL Community Server...
Nov 09 10:08:04 mysql2.mytv.bg mysqld_safe[24710]: 181109 10:08:04 mysqld_safe Logging to '/var/log/mysqld.log'.
Nov 09 10:08:04 mysql2.mytv.bg mysqld_safe[24710]: 181109 10:08:04 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Nov 09 10:08:06 mysql2.mytv.bg systemd[1]: Started MySQL Community Server.

Clear or delete systemd logs

Systemd linux distros use Journald service to collect and store logs in the system. Here are a couple of tips if you have problems with the space they occupy. It is good that all systemd linux distros support it – CentOS 7, Ubuntu 16+, Fedora, OpenSuse and so on.

TIP 1) Remove the old archive logs older than 10 days with

Time based removal of old logs. It will remove the old files. This command won’t change the configuration, so it just has a temporary effect.

journalctl --vacuum-time=10d

TIP 2) Remove the old archive logs greater than 1G

Size based removal of old logs. It will reduce the size of the logs to this specified size. This command won’t change the configuration, so it just has a temporary effect.

journalctl --vacuum-size=1024M

TIP 3) Show disk usage

[root@srv ~]# journalctl --disk-usage
Archived and active journals take up 785.5M on disk.

TIP 4) Show all logs and information for them

Where are the log files, the size they occupy, the time period of the entries in them and more:

[root@srv0 ~]# journalctl --header
File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system.journal
File ID: 7deaff4610a94c82aab85386597e825b
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ONLINE
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 1239405
Tail Sequential Number: 1341511
Head Realtime Timestamp: mon 2018-06-25 10:09:09 UTC
Tail Realtime Timestamp: tue 2018-06-28 00:07:10 UTC
Tail Monotonic Timestamp: 1month 2d 21h 6min 47.287s
Objects: 258136
Entry Objects: 102107
Data Objects: 129585
Data Hash Table Fill: 72.5%
Field Objects: 36
Field Hash Table Fill: 10.8%
Tag Objects: 0
Entry Array Objects: 26406
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-000000000011512d-00056f3f3ff56172.journal
File ID: a8e1f042bc144df78f37005b0b555a82
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 1134893
Tail Sequential Number: 1239404
Head Realtime Timestamp: fri 2018-06-22 18:32:10 UTC
Tail Realtime Timestamp: mon 2018-06-25 10:09:09 UTC
Tail Monotonic Timestamp: 1month 7h 8min 45.991s
Objects: 263445
Entry Objects: 104512
Data Objects: 131953
Data Hash Table Fill: 73.8%
Field Objects: 34
Field Hash Table Fill: 10.2%
Tag Objects: 0
Entry Array Objects: 26944
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-00000000000fc083-00056f0d6ef70320.journal
File ID: 007272b1f181487d83116bee96c40c30
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 1032323
Tail Sequential Number: 1134892
Head Realtime Timestamp: wed 2018-06-20 07:06:10 UTC
Tail Realtime Timestamp: fri 2018-06-22 18:32:10 UTC
Tail Monotonic Timestamp: 4w 2h 1min 46.980s
Objects: 263998
Entry Objects: 102570
Data Objects: 133456
Data Hash Table Fill: 74.7%
Field Objects: 37
Field Hash Table Fill: 11.1%
Tag Objects: 0
Entry Array Objects: 27933
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-00000000000e3f67-00056ee09376d287.journal
File ID: 4c268268a6e34d6297c7ae9ca01fc31b
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: yes
Head Sequential Number: 933735
Tail Sequential Number: 1032322
Head Realtime Timestamp: mon 2018-06-18 01:35:09 UTC
Tail Realtime Timestamp: wed 2018-06-20 07:06:10 UTC
Tail Monotonic Timestamp: 3w 4d 14h 35min 47.248s
Objects: 261498
Entry Objects: 98588
Data Objects: 134059
Data Hash Table Fill: 75.0%
Field Objects: 40
Field Hash Table Fill: 12.0%
Tag Objects: 0
Entry Array Objects: 28809
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-00000000000cbf3b-00056eb3ea255b2a.journal
File ID: 7019a721bebe4e00b76410a6e7052c6d
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: yes
Head Sequential Number: 835387
Tail Sequential Number: 933734
Head Realtime Timestamp: fri 2018-06-15 20:18:10 UTC
Tail Realtime Timestamp: mon 2018-06-18 01:35:09 UTC
Tail Monotonic Timestamp: 3w 2d 9h 4min 45.972s
Objects: 260794
Entry Objects: 98348
Data Objects: 134060
Data Hash Table Fill: 75.0%
Field Objects: 33
Field Hash Table Fill: 9.9%
Tag Objects: 0
Entry Array Objects: 28351
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-00000000000b2fac-00056e827d3375dd.journal
File ID: 61c16e8acc364d97b8c0c7cc2afcb6ec
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 733100
Tail Sequential Number: 835386
Head Realtime Timestamp: wed 2018-06-13 09:20:08 UTC
Tail Realtime Timestamp: fri 2018-06-15 20:18:10 UTC
Tail Monotonic Timestamp: 3w 3h 47min 46.827s
Objects: 264298
Entry Objects: 102287
Data Objects: 134043
Data Hash Table Fill: 75.0%
Field Objects: 33
Field Hash Table Fill: 9.9%
Tag Objects: 0
Entry Array Objects: 27933
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-000000000009966a-00056e4cbbb6ebfc.journal
File ID: c90df7bdc4654ab7a4ac955c74a779a3
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 628330
Tail Sequential Number: 733099
Head Realtime Timestamp: sun 2018-06-10 17:12:09 UTC
Tail Realtime Timestamp: wed 2018-06-13 09:19:10 UTC
Tail Monotonic Timestamp: 2w 4d 16h 48min 46.993s
Objects: 263295
Entry Objects: 104770
Data Objects: 131816
Data Hash Table Fill: 73.7%
Field Objects: 37
Field Hash Table Fill: 11.1%
Tag Objects: 0
Entry Array Objects: 26670
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-000000000007fd20-00056e16f3135f30.journal
File ID: 32915d651dd84728b9e4a33a97b480cf
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 523552
Tail Sequential Number: 628329
Head Realtime Timestamp: fri 2018-06-08 01:02:10 UTC
Tail Realtime Timestamp: sun 2018-06-10 17:12:09 UTC
Tail Monotonic Timestamp: 2w 2d 41min 46.200s
Objects: 263287
Entry Objects: 104778
Data Objects: 131696
Data Hash Table Fill: 73.7%
Field Objects: 36
Field Hash Table Fill: 10.8%
Tag Objects: 0
Entry Array Objects: 26775
Disk usage: 98.1M
[root@srv0 ~]#

* Deleting logs

[root@srv0 ~]# journalctl --vacuum-size=128M
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a33241-000570199db5be3c.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a34463-0005701a0b24d0fc.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3568c-0005701a79e388c7.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a36899-0005701ae529f439.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a37ac8-0005701b50781f62.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a38cde-0005701bbd3b60c3.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a39ef6-0005701c28dcf40e.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3b0f1-0005701c9416ee94.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3c309-0005701cffe19015.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3d521-0005701d6c79b9cb.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3e748-0005701dd7c60e79.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3f92d-0005701e48413eb8.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a40b0a-0005701ec83a0c74.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a41ce2-0005701f3fdcadc6.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a42f04-0005701fabfefbca.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a4410e-000570201757a256.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a60a8f-0005702ac025bb0c.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a7d254-000570359f2bd7eb.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a99a34-000570405b5e812e.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000ab639b-0005704b05553c7d.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000ad2c3e-00057055cb403fae.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000aef5c8-000570607686fd78.journal (120.0M).
Vacuuming done, freed 960.0M of archived journals from /var/log/journal/9be717e698354ec481abb641cf4085c3.

systemd can be used to run your application as a daemon

A nice feature of systemd is the scripts could be daemonized without any modifications of the scripts like standard stdin, stdout, stderr to be redirected to null (and even in some languages is not that simple!).
So if you want to run your ordinary script as daemon under a systemd linux distro, create a systemd service file in

/etc/systemd/system/myscript.service

Replace “myscript” with the nme of your script.

[Install]
WantedBy=multi-user.target

[Unit]
Description=<my_script_description>
After=syslog.target
After=network.target

[Service]
Type=simple
User=<run_as_user>
Group=<run_as_group_group>
ExecStart=<your_path_and_script_name_here>

# Give the script some time to startup
TimeoutSec=300
#EnvironmentFile=/etc/<filename>

[Install]
WantedBy=multi-user.target

So change the following according your needs:

<my_script_description> – description of your script/program with couple of words
<run_as_user> – the user under which will be executed
<run_as_group_group> – the group under which will be executed
<your_path_and_script_name_here> – the absolute path and filename to the script or program to be executed
Uncomment “EnvironmentFile” line if you would like to have different environment file for your script

And then you can run it with

systemctl start myscript

And you can enable it to start at boot

systemctl enable myscript

And also you can see the output of the script on syslog or with

systemctl status myscript

Here is an example of running a python script as a daemon with systemd.
The python script is

/usr/local/bin/mypythonscript.py

and the service name is

/etc/systemd/system/mypythonscript.service

Description=My python script daemon
After=syslog.target
After=network.target

[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/bin/python /usr/local/bin/mypythonscript.py

# Give the script some time to startup
TimeoutSec=300

[Install]
WantedBy=multi-user.target

And you can see handful of information for the state of your script with

systemctl status mypythonscript

The uptime of your script and output if any

* mypythonscript.service
   Loaded: loaded (/etc/systemd/system/mypythonscript.service; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-01-04 13:58:34 UTC; 1 months 12 days ago
 Main PID: 29654 (python)
   CGroup: /system.slice/mypythonscript.service
           `-37421 /usr/bin/python /usr/local/bin/mypythonscript.py

Jan 12 10:05:21 srv.local systemd[1]: [/etc/systemd/system/mypythonscript.service:3] Assignment outside of section. Ignoring.
Jan 12 10:05:21 srv.local systemd[1]: [/etc/systemd/system/mypythonscript.service:1] Assignment outside of section. Ignoring.