Failed to start Security Audit Service, Authorization Manager and Login Service

A power outrage caused one of our servers to shut down unexpectedly and after it had been powered up the server did not show up. The server was unreachable and apparently, the network did not bring up the interfaces.
Loading the IPMI KVM Console and rebooting the server there were three errors on the screen during the boot up of the CentOS 7:

[FAILED] Failed to start Security Audit Service.
See 'systemctl status auditd.service' for details.
....
....
[FAILED] Failed to start Authorization Manager.
See 'systemctl status polkit.service' for details.
....
....
[FAILED] Failed to start Login Service.
See 'systemctl status systemd-logind.service' for details.

And after the above last line, the system stopped loading.
The disks are clean, but there was no login service, so you cannot log in to the server through the keyboard and the monitor! There was no network as mentioned above, which meant no logging at all in the server. You might not know, but if auditd service is enabled you probably use Selinux!

STEP 1) Failed to start the three important services – Security Audit Service, Authorization Manager and Login Service.

So we ended up with unability to log in our server.

main menu

Not sure what exactly caused this problem (seems strange a perfectly working Selinux enabled CentOS 7 server to have miss-labeled files in the root only because of an unexpected shutdown), but to be able to fix the issue and bring back your server to life

you need a rescue CD/USB/DVD/PXE Server to boot from and mount the disks and relabel your root file system.

STEP 1) Boot from a rescue CD/USB/DVD/PXE Server.

In our case, we used the IPMI KVM Console and mounted a Gentoo ISO disk and then booted from it to have a bash shell in our system. Our root resides on software RAID 1, so cat the /proc/mdstat and mount your root file system somewhere (/mnt/gentoo is there by default…)

STEP 2) Booted in our rescue Gentoo CD and mount your root file system.

main menu
Rescue Gentoo CD

STEP 2) create a file “.autorelabel” in the mounting point of your root file system.

So in our case, we mounted our CentOS 7 root file system in /mnt/gentoo and you must create a file with patch “/mnt/gentoo/.autorelabel”. umount and reboot. And a few minutes later your server will be back from the dead. A quick and handful advice – edit your /etc/fstab to mount only the root file system by commenting out all other big storage mounts – of course, if it is possible. We have big storage with millions of files in /mnt/storage-01 and we put the “#” to comment out the line with it – we do not want to wait for relabeling this file system, because the problem apparently is in our root file system! If it is possible (it is highly recommended) to relabel only the root file system in such situations to be able to regain shell control over your server fast.

Bonus – booted in rescue but no logs

OK, we booted to the rescue and tried to see what was the error (with journalctl in chrooted /mnt/gentoo), which did not allow auditd, polkit and systemd-logind to fail to start, but it appeared by default the systemd logs are not persistent on the disk in CentOS 7, so when you reboot in rescue you do not have systemd logs from the last boot! As a piece of additional advice here you may consider enabling persistent systemd logs!

rsync and selinux – opendir failed: Permission denied

Selinux could sometime mess up with your setup. Let’s say you configured your rsync daemon but still, you get the error related to permissions when executing the rsync to copy files!

rsync: opendir "/." (in backup2) failed: Permission denied (13)

Apparently, the rsync client connects to the server and it finds there is a section name “backup2”, but still no permission despite you explicitly set in the section uid and ig to be root (uid=0 and gid=0 in the section)!

The most common reason is

selinux denies rsync process to open the directory exported by the path in your rsync configuration file.

By default, Selinux will deny access to any of the files and directories in your system! In most cases here what can you help:

setsebool -P rsync_export_all_ro=1

rsync_export_all_ro will export any files and directories read-only and requests like above will not be denied.
The capital letter “-P” is to set it permanently for the system over reboots.
Keep on reading!

Selinux permission denied and no log in audit.log

So you execute a script and get a “Permission denied” and you know you have enabled SELinux. OK to disable the selinux is not an option (and never will be), so the first thing to check is the audit log to see what is the error and what the selinux tools will offer to solve it.

But there are no entries in the audit log when you execute your script!

So you decide to temporarily disable the selinux to check if this permission denied issues is still caused by it with:

setenforce 0

And the script just executes fine no error! Then again you put back the Enforcing with:

setenforce 1
./myscript
Permission Denied

And NO added lines in audit.log (/var/log/audit/audit.log in our system!). Apparently the logging is just fine, because it got sometime entries, but when executing our script, which is just a simple:

 
find /mnt/storA/servers/webroots/

After some research it appeared that

not all AVC denials may be logged when SELinux denies access.

Too many applications and system libraries check for permissions, which might not use or even need after that and the logging could grow exponentially or be less informative for the real cause of a problem!
Keep on reading!

Technical details of a default CentOS 6.10 minimal installation

This article is for those of you who do not want to install a how new operating system only to discover some technical details about the default installation like, disk layout, packages included, software versions and so on. Here we are going to review in several sections what is like to have a default installation of CentOS 6.10.
Despite the old kernel, as you can see branch 2.6.x (2.6.32) it detects successfully the new RYZEN AMD and the system is stable (though we could not boot in UEFI mode!).

Software

The CentOS 6.10 you can have

  • Xorg X server – 1.17.4
  • GNOME (the GUI) – 2.28.2
  • K Desktop Environment – Workspace 4.3.4
  • linux kernel – 2.6.32 (2.6.32-754.el6)
  • linux-firmware – no version, multiple packages for the drivers
  • QT – 4.6.2
  • libc – 2.12
  • gnu gcc – 4.4.7
  • coreutils – 8.4
  • yum – 3.2.29
  • cups – 1.4.2
  • MySQL server – 5.1.73
  • PHP – 5.3.3
  • python – 2.6.6
  • perl – 5.10.1
  • ruby – 1.8.7.374
  • OpenJDK – 1.8.0.191.b12 and also includes 1.6.0.41 and 1.7.0.191

Note: Not all of the above software comes installed by default.
Installed packages are 268 occupying 717M space:

[root@srv ~]# yum list installed|wc -l
268
[root@srv ~]# df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_srv-lv_root
                       50G  717M   46G   2% /

Keep on reading!

Technical details of a default Fedora 29 Workstation installation

This article is for those of you who do not want to install a how new operating system only to discover some technical details about the default installation like, disk layout, packages included, software versions and so on. Here we are going to review in several sections what is like to have a default installation of Fedora Workstation 29.

Software

The Fedora 29 Workstation you can have

  • Xorg X server – 1.20.1
  • GNOME (the GUI) – 3.30.1
  • linux kernel – 4.18.16
  • linux-firmware – 20180913 (87.git44d4fca9.fc29)
  • QT – 5.11.1
  • libc – 2.28
  • gnu gcc – 8.2.1 / installed by default
  • coreutils – 8.30
  • dnf – 4.0.4
  • cups – 2.2.8
  • MySQL server – 8.0.12
  • PHP – 7.2.11
  • Go Lang – 1.11
  • python2.7 – 2.7.15
  • python3 (default) – 3.7.0 / installed by default
  • perl – 5.28.0 / installed by default
  • ruby – 2.5.1
  • OpenJDK – “11-ea” 2018-09-25 / installed by default

Note: Not all of the above software comes installed by default.
Installed packages are 1613 occupying 5.9G space:

[root@localhost ~]# dnf list installed|wc -l
1613
[root@localhost ~]# df -h /
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/fedora-root   49G  5.9G   41G  13% /

Keep on reading!

Gentoo update tips when updating packages with blocks and masked files

It’s not so rear to have

blocks or masked files

when using Gentoo emerge system, but it is not complex and in most cases it is easy to resolve

To summarize it up at the beginning and then we are going to show you other articles using these advises here where you’ll see what are the steps we took to resolve the conflicts and masked packages:

  • Use verbose,verbose-conflicts and backtrack with emerge
  • Remove only big GUI packages, which have really big dependency graph like office suites or development IDEs
  • Remove obsolete packages – you do not need them, they can just make problems when updating, because emerge will take into consideration their requirements and dependencies and your update could be impossible!
  • Do not update everything with one line, you could update only the base libraries like QT, which are very important for the Linux GUI in general
  • Include explicitly packages, which block our updates in the emerge line! You could specify packages with the versions.
  • use tools like “equery” (part of app-portage/gentoolkit) for checking dependencies and/or which packages depend on the queried package. You can use it with specific version for the package. “qlist” (part of app-portage/portage-utils) also is a handful tool.
  • Sometimes when updating a group or a package with big dependency graph it is much easier to drop the -“u” update argument and to rebuild some packages with the updates.
  • In rear cases you can use “–nodeps” when updating or installing a new package (we do not need and show this one here!)
  • Do NOT rebuild the entire system with “emerge -v world” every time when you rebuild glibc, gcc, it is not mandatory to do it to have a healthy system.
  • Add or remove USE flags if needed – emerge will show you information about it. Use package.use, package.mask, package.unmask and so on.
  • use qlist to update/re(build) to pull currently installed packages with some name or category (categories)
    emerge -v $(qlist -IC|grep <NAME>)
    

    and for update just add “u” to “-v”:

    emerge -vu $(qlist -IC|grep <NAME>)
    

Keep on reading!

How to proper enable the selinux in a CentOS7 installed server

These days many dedicated servers are offered with automation installation of operating systems and it have never been so easy and fast to pay a server and to get it up in minutes! Yes, we are talking for dedicated machines not virtual servers or cloud ones, but many cloud ones are in the same situation, when the host uses full virtualization.
It is fast, you can choose from many different linux distros and the installation is unattended and happens immediately, but in most cases the selinux is disabled, because is more easy for the support, for the user, for the admin, and for the offered preinstalled software…If you do not have some strange software in most cases it is advisable to enable the selinux, because it is of great security enhancement for your server and for the software in general. It is very simple to enable selinux, but there is an additional step, which if you omit, the server will probably get unusable (probably you won’t be able even to ssh it or login). These steps are tested under CentOS7, but probably works in all other distros, which support selinux!
So here are the steps:

STEP 1) Enable selinux in configuration

Edit the configuration file in

/etc/selinux/config

SELINUX=enforcing

STEP 2) relabel the file system

When using the selinux, there are labels (extended attributes of the file system), which are additional layer of security. Every system comes with prebuild rules instructing what label is set in which file or directory, so when you enable the selinux you must relabel the entire file system (or at least the root partition, to be able to boot normally). This is done with just a line of code below:

touch /.autorelabel

STEP 3) reboot

To take effect the changes made above the server must be rebooted.

reboot

The init process will find the file from step 2 “/.autorelabel” and will initiate a proper relabel according the current selinux rules file, then the server will be rebooted automatically again, the relabel could take time and it depends on the number of files you have in your server, just keep patient.
After the second reboot (which is automatically after the relabeling)

4) Recommendations

USE SELINUX, do not disable it! In most cases it is really simple to configure it in minutes for the need of your special software and for the generic one bet it there are rules offered in the distro’s packet system.

4) Post install check

You can check if the selinux is enabled with

[root@srv ~]# getenforce 
Enforcing
[root@srv ~]#