On some Linux distributions, systemd log files are not saved on your disk, but only temporary in the memory and when you reboot all logs are discarded. So the systemd logs are not persistent, which could lead to missing important information if you want to check them when you are booted in a rescue disk or even if you just reboot your server. for exmaple,
if some important service failed to boot and your server is unreachable and you boot in rescue CD you do not have logs to check why the service failed and the (error) output of the process of starting the services!
Here is how you can enable the systemd logs to be persistent i.e. save them on the disk. This is tested on CentOS 7, which by default saves the systemd logs on memory!
STEP 1) Prepare the systemd log directory
mkdir -p /var/log/journal/ systemd-tmpfiles --create --prefix /var/log/journal/
STEP 2) Edit systemd configuration and reload the daemon
And ensure your configuration uses “Storage=persistent” in /etc/systemd/journald.conf
grep Storage /etc/systemd/journald.conf Storage=persistent systemctl restart systemd-journald
The last line with systemctl restart could be replace with
killall -USR1 systemd-journald
if you do not want to lose all your current logs in memory!
Bonus – systemd logs from multiple reboots
Here we have logs from 5 reboots. Here you can also see what are the right owner (systemd-journal) and Selinux labels of the “/var/log/journal/”
[root@srv ~]# ls -altrZ /var/log/journal/ drwxr-sr-x+ root systemd-journal system_u:object_r:var_log_t:s0 dbd91181db6b4c9f900d9b3a1651a8d5 drwxr-sr-x+ root systemd-journal system_u:object_r:var_log_t:s0 . drwxr-xr-x. root root system_u:object_r:var_log_t:s0 .. [root@srv ~]# journalctl --disk-usage Archived and active journals take up 112.0M on disk. [root@srv ~]# journalctl --list-boots -4 ec4146b78ac944b8a8d4116f259e09ee Thu 2019-06-06 23:39:14 UTC—Thu 2019-06-06 23:39:37 UTC -3 ae3d39db626c4592aa84cc68072fbb32 Thu 2019-06-06 23:41:03 UTC—Thu 2019-06-06 23:42:13 UTC -2 68c1ca07c05b4d59adcc9888c50f4065 Thu 2019-06-06 23:42:57 UTC—Fri 2019-06-07 00:13:27 UTC -1 f7e8da6aaa8740faa05c4985c92023fd Fri 2019-06-07 00:14:08 UTC—Fri 2019-06-07 00:16:33 UTC 0 45c00dc29e1a48298d9f87f5421468b4 Fri 2019-06-07 00:17:13 UTC—Mon 2019-06-10 01:39:17 UTC [root@srv ~]# journalctl --boot=-2 -- Logs begin at Thu 2019-06-06 23:39:14 UTC, end at Mon 2019-06-10 01:39:17 UTC. -- Jun 06 23:42:57 srv systemd-journal[133]: Runtime journal is using 8.0M (max allowed 1.5G, trying to leave 2.3G free of 15.6G available → current limit 1.5G). Jun 06 23:42:57 srv kernel: microcode: microcode updated early to revision 0x710, date = 2013-06-17 Jun 06 23:42:57 srv kernel: Initializing cgroup subsys cpuset Jun 06 23:42:57 srv kernel: Initializing cgroup subsys cpu Jun 06 23:42:57 srv kernel: Initializing cgroup subsys cpuacct Jun 06 23:42:57 srv kernel: Linux version 3.10.0-514.10.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 S Jun 06 23:42:57 srv kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7.x86_64 root=UUID=c9bec791-c77d-4189-b18a-9ddc728ee782 ro crashkernel=auto r Jun 06 23:42:57 srv kernel: e820: BIOS-provided physical RAM map: .... .... [root@srv ~]# journalctl --boot=-2 -u auditd -- Logs begin at Thu 2019-06-06 23:39:14 UTC, end at Mon 2019-06-10 01:50:18 UTC. -- Jun 06 23:43:05 srv systemd[1]: Starting Security Auditing Service... Jun 06 23:43:05 srv auditd[694]: Started dispatcher: /sbin/audispd pid: 698 Jun 06 23:43:05 srv audispd[698]: priority_boost_parser called with: 4 Jun 06 23:43:05 srv audispd[698]: max_restarts_parser called with: 10 Jun 06 23:43:05 srv audispd[698]: audispd initialized with q_depth=150 and 1 active plugins Jun 06 23:43:05 srv augenrules[695]: /sbin/augenrules: No change Jun 06 23:43:05 srv auditd[694]: Init complete, auditd 2.6.5 listening for events (startup state enable) Jun 06 23:43:05 srv augenrules[695]: No rules Jun 06 23:43:05 srv augenrules[695]: enabled 1 Jun 06 23:43:05 srv augenrules[695]: failure 1 Jun 06 23:43:05 srv augenrules[695]: pid 694 Jun 06 23:43:05 srv augenrules[695]: rate_limit 0 Jun 06 23:43:05 srv augenrules[695]: backlog_limit 320 Jun 06 23:43:05 srv augenrules[695]: lost 0 Jun 06 23:43:05 srv augenrules[695]: backlog 1 Jun 06 23:43:05 srv systemd[1]: Started Security Auditing Service. Jun 06 23:56:48 srv auditd[694]: The audit daemon is exiting. Jun 06 23:56:49 srv systemd[1]: Starting Security Auditing Service... Jun 06 23:56:49 srv auditd[24744]: Started dispatcher: /sbin/audispd pid: 24746 Jun 06 23:56:49 srv audispd[24746]: audispd initialized with q_depth=250 and 1 active plugins Jun 06 23:56:49 srv auditd[24744]: Init complete, auditd 2.8.4 listening for events (startup state enable) Jun 06 23:56:49 srv augenrules[24750]: /sbin/augenrules: No change Jun 06 23:56:49 srv augenrules[24750]: No rules Jun 06 23:56:49 srv augenrules[24750]: enabled 1 Jun 06 23:56:49 srv augenrules[24750]: failure 1 Jun 06 23:56:49 srv augenrules[24750]: pid 24744 Jun 06 23:56:49 srv augenrules[24750]: rate_limit 0 Jun 06 23:56:49 srv augenrules[24750]: backlog_limit 320 Jun 06 23:56:49 srv augenrules[24750]: lost 0 Jun 06 23:56:49 srv augenrules[24750]: backlog 1 Jun 06 23:56:49 srv systemd[1]: Started Security Auditing Service. Jun 07 00:13:26 srv systemd[1]: Stopping Security Auditing Service... Jun 07 00:13:26 srv systemd[1]: Stopped Security Auditing Service.
Now you have logs of your booting process!
The systemd log files are accessible even if you’ve booted from a rescue CD and you chroot in your system!
Be careful with the disk free space when using disk storage for your systemd logs – Clear or delete systemd logs.