Upgrading to systemd-utils and resolving systemd-tmpfiles soft blocking systemd-utils

main menu
News migrating to systemd-utils

Recently a new Gentoo package was added – sys-apps/systemd-utils, which should replace three separate packages sys-apps/systemd-tmpfiles, sys-boot/systemd-boot and sys-fs/udev. As the sys-fs/udev is one of the core packages in the OpenRC world and it can trigger multiple blocking errors and dependency problems.
The sys-apps/systemd-utils has three USE variables for the three replacing packages – udev, boot, and tmpfiles, so the user might add them in make.conf file.
It’s worth noting when upgrading to the sys-apps/systemd-utils, the sys-apps/systemd-tmpfiles and sys-boot/systemd-boot will be removed, but sys-fs/udev is staying in the system with version sys-fs/udev-250 and above. The sys-fs/udev-250 would always pull in sys-apps/systemd-utils[udev] as a dependency. Despite the names including systemd, these packages do not depend on systemd and are meant to be used in OpenRC system.
There is Gentoo news about this upgrade and it can be read with eselect. Check out the article ending for more information.
Here is what a blocking could look like:

root@srv ~ # emerge -vau world
......
......
[ebuild     U  ] dev-lang/php-8.0.27:8.0::gentoo [8.0.25:8.0::gentoo] USE="acl bcmath berkdb bzip2 calendar cli ctype curl enchant exif fileinfo filter flatfile fpm ftp gd gdbm gmp iconv imap intl ipv6 jit mhash mysql mysqli nls opcache pcntl pdo phar posix readline session session-mm sharedmem simplexml soap sockets sqlite ssl tidy tokenizer truetype unicode xml xmlreader xmlwriter xslt zip zlib -apache2 -apparmor -argon2 -cdb -cgi -cjk -coverage -debug -embed -ffi -firebird -inifile -iodbc -kerberos -ldap -ldap-sasl -libedit -lmdb -mssql -oci8-instant-client -odbc -phpdbg -postgres -qdbm (-selinux) -snmp -sodium -spell -systemd -sysvipc -test -threads -tokyocabinet -webp -xpm" 10549 KiB
[ebuild     UD ] sys-apps/man-pages-posix-2013a::gentoo [2017a::gentoo] 909 KiB
[ebuild     U  ] sys-apps/openrc-0.46::gentoo [0.45.2-r1::gentoo] USE="ncurses netifrc pam unicode -audit -bash -debug -newnet (-selinux) -sysv-utils" 242 KiB
[blocks B      ] sys-fs/eudev ("sys-fs/eudev" is soft blocking sys-apps/systemd-utils-251.10-r1)
[blocks B      ] sys-apps/systemd-utils[udev] ("sys-apps/systemd-utils[udev]" is soft blocking sys-fs/eudev-3.2.11-r3)

Total: 113 packages (97 upgrades, 1 downgrade, 15 new), Size of downloads: 554077 KiB
Conflict: 3 blocks (2 unsatisfied)

 * Error: The above package list contains packages which cannot be
 * installed at the same time on the same system.

  (sys-fs/eudev-3.2.11-r3:0/0::gentoo, installed) pulled in by
    sys-fs/eudev required by @selected 

  (sys-apps/systemd-utils-251.10-r1:0/0::gentoo, ebuild scheduled for merge) pulled in by
    sys-apps/systemd-utils[tmpfiles] required by (sys-apps/systemd-tmpfiles-250:0/0::gentoo, ebuild scheduled for merge) USE="" ABI_X86="(64)"
    sys-apps/systemd-utils[tmpfiles] required by (virtual/tmpfiles-0-r3:0/0::gentoo, installed) USE="" ABI_X86="(64)"
    sys-apps/systemd-utils[udev] required by (virtual/udev-217-r5:0/0::gentoo, installed) USE="" ABI_X86="(64)"


For more information about Blocked Packages, please refer to the following
section of the Gentoo Linux x86 Handbook (architecture is irrelevant):

https://wiki.gentoo.org/wiki/Handbook:X86/Working/Portage#Blocked_packages
root@srv ~ # emerge -va sys-apps/systemd-utils


These are the packages that would be merged, in order:

Calculating dependencies... done!
Dependency resolution took 0.84 s.

[ebuild  N     ] acct-group/audio-0-r1::gentoo  0 KiB
[ebuild  N     ] acct-group/cdrom-0-r1::gentoo  0 KiB
[ebuild  N     ] acct-group/dialout-0-r1::gentoo  0 KiB
[ebuild  N     ] acct-group/disk-0-r1::gentoo  0 KiB
[ebuild  N     ] acct-group/kmem-0-r1::gentoo  0 KiB
[ebuild  N     ] acct-group/lp-0-r1::gentoo  0 KiB
[ebuild  N     ] acct-group/sgx-0::gentoo  0 KiB
[ebuild  N     ] acct-group/tape-0-r1::gentoo  0 KiB
[ebuild  N     ] acct-group/tty-0-r1::gentoo  0 KiB
[ebuild  N     ] acct-group/video-0-r1::gentoo  0 KiB
[ebuild  N     ] sys-apps/systemd-utils-251.10-r1::gentoo  USE="acl kmod (split-usr) tmpfiles udev -boot (-selinux) -sysusers -test" ABI_X86="(64) -32 (-x32)" 11194 KiB
[blocks B      ] <sys-apps/systemd-tmpfiles-250 ("<sys-apps/systemd-tmpfiles-250" is soft blocking sys-apps/systemd-utils-251.10-r1)
[blocks B      ] sys-fs/eudev ("sys-fs/eudev" is soft blocking sys-apps/systemd-utils-251.10-r1)
[blocks B      ] sys-apps/systemd-utils[udev] ("sys-apps/systemd-utils[udev]" is soft blocking sys-fs/eudev-3.2.11-r3)

Total: 11 packages (11 new), Size of downloads: 11194 KiB
Conflict: 3 blocks (3 unsatisfied)

 * Error: The above package list contains packages which cannot be
 * installed at the same time on the same system.

  (sys-apps/systemd-tmpfiles-249.9:0/0::gentoo, installed) pulled in by
    sys-apps/systemd-tmpfiles required by @selected 

  (sys-apps/systemd-utils-251.10-r1:0/0::gentoo, ebuild scheduled for merge) pulled in by
    sys-apps/systemd-utils

  (sys-fs/eudev-3.2.11-r3:0/0::gentoo, installed) pulled in by
    sys-fs/eudev required by @selected 


For more information about Blocked Packages, please refer to the following
section of the Gentoo Linux x86 Handbook (architecture is irrelevant):

https://wiki.gentoo.org/wiki/Handbook:X86/Working/Portage#Blocked_packages

A screenshot from the command line, which is colored properly.

main menu
Unresolved Soft blocking tmpfiles and eudev

To solve the soft blocking, first, deselect the old packages, which are soft blocking the sys-apps/systemd-utils. In the above example, the problematic packages are sys-fs/eudev and sys-apps/systemd-tmpfiles. Deselect the packages and try to build only sys-apps/systemd-utils:
Keep on reading!

Make systemd to save logs on the disk

On some Linux distributions, systemd log files are not saved on your disk, but only temporary in the memory and when you reboot all logs are discarded. So the systemd logs are not persistent, which could lead to missing important information if you want to check them when you are booted in a rescue disk or even if you just reboot your server. for exmaple,

if some important service failed to boot and your server is unreachable and you boot in rescue CD you do not have logs to check why the service failed and the (error) output of the process of starting the services!

Here is how you can enable the systemd logs to be persistent i.e. save them on the disk. This is tested on CentOS 7, which by default saves the systemd logs on memory!

STEP 1) Prepare the systemd log directory

mkdir -p /var/log/journal/
systemd-tmpfiles --create --prefix /var/log/journal/

STEP 2) Edit systemd configuration and reload the daemon

And ensure your configuration uses “Storage=persistent” in /etc/systemd/journald.conf

grep Storage /etc/systemd/journald.conf
Storage=persistent
systemctl restart systemd-journald

The last line with systemctl restart could be replace with

killall -USR1 systemd-journald

if you do not want to lose all your current logs in memory!

Bonus – systemd logs from multiple reboots

Here we have logs from 5 reboots. Here you can also see what are the right owner (systemd-journal) and Selinux labels of the “/var/log/journal/”

[root@srv ~]# ls -altrZ /var/log/journal/
drwxr-sr-x+ root systemd-journal system_u:object_r:var_log_t:s0   dbd91181db6b4c9f900d9b3a1651a8d5
drwxr-sr-x+ root systemd-journal system_u:object_r:var_log_t:s0   .
drwxr-xr-x. root root            system_u:object_r:var_log_t:s0   ..
[root@srv ~]# journalctl --disk-usage
Archived and active journals take up 112.0M on disk.
[root@srv ~]# journalctl --list-boots
-4 ec4146b78ac944b8a8d4116f259e09ee Thu 2019-06-06 23:39:14 UTC—Thu 2019-06-06 23:39:37 UTC
-3 ae3d39db626c4592aa84cc68072fbb32 Thu 2019-06-06 23:41:03 UTC—Thu 2019-06-06 23:42:13 UTC
-2 68c1ca07c05b4d59adcc9888c50f4065 Thu 2019-06-06 23:42:57 UTC—Fri 2019-06-07 00:13:27 UTC
-1 f7e8da6aaa8740faa05c4985c92023fd Fri 2019-06-07 00:14:08 UTC—Fri 2019-06-07 00:16:33 UTC
 0 45c00dc29e1a48298d9f87f5421468b4 Fri 2019-06-07 00:17:13 UTC—Mon 2019-06-10 01:39:17 UTC
[root@srv ~]# journalctl --boot=-2
-- Logs begin at Thu 2019-06-06 23:39:14 UTC, end at Mon 2019-06-10 01:39:17 UTC. --
Jun 06 23:42:57 srv systemd-journal[133]: Runtime journal is using 8.0M (max allowed 1.5G, trying to leave 2.3G free of 15.6G available → current limit 1.5G).
Jun 06 23:42:57 srv kernel: microcode: microcode updated early to revision 0x710, date = 2013-06-17
Jun 06 23:42:57 srv kernel: Initializing cgroup subsys cpuset
Jun 06 23:42:57 srv kernel: Initializing cgroup subsys cpu
Jun 06 23:42:57 srv kernel: Initializing cgroup subsys cpuacct
Jun 06 23:42:57 srv kernel: Linux version 3.10.0-514.10.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 S
Jun 06 23:42:57 srv kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7.x86_64 root=UUID=c9bec791-c77d-4189-b18a-9ddc728ee782 ro crashkernel=auto r
Jun 06 23:42:57 srv kernel: e820: BIOS-provided physical RAM map:
....
....
[root@srv ~]# journalctl --boot=-2 -u auditd
-- Logs begin at Thu 2019-06-06 23:39:14 UTC, end at Mon 2019-06-10 01:50:18 UTC. --
Jun 06 23:43:05 srv systemd[1]: Starting Security Auditing Service...
Jun 06 23:43:05 srv auditd[694]: Started dispatcher: /sbin/audispd pid: 698
Jun 06 23:43:05 srv audispd[698]: priority_boost_parser called with: 4
Jun 06 23:43:05 srv audispd[698]: max_restarts_parser called with: 10
Jun 06 23:43:05 srv audispd[698]: audispd initialized with q_depth=150 and 1 active plugins
Jun 06 23:43:05 srv augenrules[695]: /sbin/augenrules: No change
Jun 06 23:43:05 srv auditd[694]: Init complete, auditd 2.6.5 listening for events (startup state enable)
Jun 06 23:43:05 srv augenrules[695]: No rules
Jun 06 23:43:05 srv augenrules[695]: enabled 1
Jun 06 23:43:05 srv augenrules[695]: failure 1
Jun 06 23:43:05 srv augenrules[695]: pid 694
Jun 06 23:43:05 srv augenrules[695]: rate_limit 0
Jun 06 23:43:05 srv augenrules[695]: backlog_limit 320
Jun 06 23:43:05 srv augenrules[695]: lost 0
Jun 06 23:43:05 srv augenrules[695]: backlog 1
Jun 06 23:43:05 srv systemd[1]: Started Security Auditing Service.
Jun 06 23:56:48 srv auditd[694]: The audit daemon is exiting.
Jun 06 23:56:49 srv systemd[1]: Starting Security Auditing Service...
Jun 06 23:56:49 srv auditd[24744]: Started dispatcher: /sbin/audispd pid: 24746
Jun 06 23:56:49 srv audispd[24746]: audispd initialized with q_depth=250 and 1 active plugins
Jun 06 23:56:49 srv auditd[24744]: Init complete, auditd 2.8.4 listening for events (startup state enable)
Jun 06 23:56:49 srv augenrules[24750]: /sbin/augenrules: No change
Jun 06 23:56:49 srv augenrules[24750]: No rules
Jun 06 23:56:49 srv augenrules[24750]: enabled 1
Jun 06 23:56:49 srv augenrules[24750]: failure 1
Jun 06 23:56:49 srv augenrules[24750]: pid 24744
Jun 06 23:56:49 srv augenrules[24750]: rate_limit 0
Jun 06 23:56:49 srv augenrules[24750]: backlog_limit 320
Jun 06 23:56:49 srv augenrules[24750]: lost 0
Jun 06 23:56:49 srv augenrules[24750]: backlog 1
Jun 06 23:56:49 srv systemd[1]: Started Security Auditing Service.
Jun 07 00:13:26 srv systemd[1]: Stopping Security Auditing Service...
Jun 07 00:13:26 srv systemd[1]: Stopped Security Auditing Service.

Now you have logs of your booting process!

The systemd log files are accessible even if you’ve booted from a rescue CD and you chroot in your system!

Be careful with the disk free space when using disk storage for your systemd logs – Clear or delete systemd logs.