Installing gitlab-ee (and gitlab-ce) under CentOS 7 with enabled SELinux (i.e. enforcing mode) looped endlessly the container in restarting the installation process! There were multiple errors for missing sockets in the podman logs of the gitlab container. Here are some of the errors:
Missing postgresql unix socket in “/var/opt/gitlab/postgresql”:
Recipe: gitlab::database_migrations
* bash[migrate gitlab-rails database] action run
[execute] rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:53:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
.....
.....
Running handlers:
There was an error running gitlab-ctl reconfigure:
bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20200915-35-lemic5" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:53:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" "/tmp/chef-script20200915-35-lemic5" ----
Ran "bash" "/tmp/chef-script20200915-35-lemic5" returned 1
Missing redis socket in
Running handlers: There was an error running gitlab-ctl reconfigure: redis_service[redis] (redis::enable line 19) had an error: RuntimeError: ruby_block[warn pending redis restart] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/resources/service.rb line 65) had an error: RuntimeError: Execution of the command `/opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket INFO` failed with a non-zero exit code (1) stdout: stderr: Could not connect to Redis at /var/opt/gitlab/redis/redis.socket: No such file or directory
It should be noted that the /var/opt/gitlab directory has been mapped in /mnt/storage/podman/gitlab/data. GlusterFS is used for /mnt/storage, so the gitlab files resides on a GlusterFS volume.
ERROR 1) Cannot create unix socket.
Checking the /var/log/audit/audit.log reveiled the problem immediately:
type=SYSCALL msg=audit(1600207399.900:13213426): arch=c000003e syscall=133 success=no exit=-13 a0=7f8c38ff7760 a1=c1c0 a2=0 a3=0 items=0 ppid=1 pid=11789 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="glfs_iotwr001" exe="/usr/sbin/glusterfsd" subj=system_u:system_r:glusterd_t:s0 key=(null)
type=PROCTITLE msg=audit(1600207399.900:13213426): proctitle=2F7573722F7362696E2F676C7573746572667364002D73006C737276312E73746F65762E6575002D2D766F6C66696C652D696400564F4C312E6C737276312E73746F65762E65752E6D6E742D73746F72616765312D676C757374657266732D6172626974657231002D70002F7661722F72756E2F676C75737465722F766F6C73
type=AVC msg=audit(1600207400.935:13213427): avc: denied { create } for pid=11782 comm="glfs_iotwr003" name="redis.socket" scontext=system_u:system_r:glusterd_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=sock_file permissive=0
type=SYSCALL msg=audit(1600207400.935:13213427): arch=c000003e syscall=133 success=no exit=-13 a0=7fef580607e0 a1=c1c0 a2=0 a3=0 items=0 ppid=1 pid=11782 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="glfs_iotwr003" exe="/usr/sbin/glusterfsd" subj=system_u:system_r:glusterd_t:s0 key=(null)
type=PROCTITLE msg=audit(1600207400.935:13213427): proctitle=2F7573722F7362696E2F676C7573746572667364002D73006C737276312E73746F65762E6575002D2D766F6C66696C652D696400564F4C312E6C737276312E73746F65762E65752E6D6E742D73746F72616765312D676C757374657266732D627269636B31002D70002F7661722F72756E2F676C75737465722F766F6C732F56
type=AVC msg=audit(1600207400.935:13213428): avc: denied { create } for pid=11789 comm="glfs_iotwr001" name="redis.socket" scontext=system_u:system_r:glusterd_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=sock_file permissive=0
Which problem may be solved with:
#============= glusterd_t ============== #!!!! WARNING: 'unlabeled_t' is a base type. allow glusterd_t unlabeled_t:sock_file create;
To solve the issue of the sample lines above from the /var/log/audit/audit.log included the above rule in a file with name audit-log-socket-create-denied-glusterfs.log and then transffered in a SELinux module with audit2allow. Then imported the SELinux module with semodule:
[root@srv ~]# cat audit-log-socket-create-denied-glusterfs.log|audit2allow -M audit-log-socket-create-denied-glusterfs ******************** IMPORTANT *********************** To make this policy package active, execute: semodule -i audit-log-socket-create-denied-glusterfs.pp [root@lsrv1 ~]# semodule -i audit-log-socket-create-denied-glusterfs.pp
ERROR 2) Cannot use setattr, unlink and link on the unix socket.
But then the gitlab’ redis and postgresql still do not create the unix socket and the services are not started! Again, there is a deny record in the /var/log/audit/audit.log for setattr, unlink and link. So we searched with ausearch:
[root@srv ~]# ausearch -c 'glfs_iotwr001' --raw | audit2allow
#============= glusterd_t ==============
#!!!! This avc is allowed in the current policy
allow glusterd_t unlabeled_t:sock_file { create link setattr unlink };
[root@srv ~]# ausearch -c 'glfs_iotwr001' --raw | audit2allow -M unix-socket-unlabeled-all
******************** IMPORTANT ***********************
To make this policy package active, execute:
semodule -i unix-socket-unlabeled-all.pp
[root@srv ~]# semodule -i unix-socket-unlabeled-all.pp
The command with ausearch searches the whole default audit file in /var/log/audit/audit.log and it finds all denied requests including those, which have already been fixed like “create“. But it is good to have all SELinux rules in one file – unix-socket-unlabeled-all.
Make the SELinux module with audit2allow and then import the module. And now the gitlab podman container could create the unix sockets for the redis server, gitaly, postgresql and so on.
Here is what would look like the denied reuqest before importing the last SELinux module:
type=AVC msg=audit(1600209302.567:13222910): avc: denied { setattr } for pid=11789 comm="glfs_iotwr001" name="test-13bf4376.sock" dev="md3" ino=135268758 scontext=system_u:system_r:glusterd_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1600209517.408:13224124): avc: denied { link } for pid=11789 comm="glfs_iotwr001" name="test-14b0dc17.sock" dev="md3" ino=135268898 scontext=system_u:system_r:glusterd_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=sock_file permissive=0
type=SYSCALL msg=audit(1600209517.408:13224124): arch=c000003e syscall=265 success=no exit=-13 a0=ffffffffffffff9c a1=7f8c38ff7750 a2=ffffffffffffff9c a3=7f8c38ff7410 items=0 ppid=1 pid=11789 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="glfs_iotwr001" exe="/usr/sbin/glusterfsd" subj=system_u:system_r:glusterd_t:s0 key=(null)
type=AVC msg=audit(1600209517.757:13224131): avc: denied { unlink } for pid=11789 comm="glfs_iotwr001" name="redis.socket" dev="md3" ino=135268225 scontext=system_u:system_r:glusterd_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=sock_file permissive=0
type=SYSCALL msg=audit(1600209517.757:13224131): arch=c000003e syscall=87 success=no exit=-13 a0=7f8c38ff7670 a1=7f8c44007bd0 a2=7f8c38ff7670 a3=0 items=0 ppid=1 pid=11789 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="glfs_iotwr001" exe="/usr/sbin/glusterfsd" subj=system_u:system_r:glusterd_t:s0 key=(null)
ERROR 3) Cannot use create, setattr, write, unlink and link on the unix socket for container_t in fusefs_t.
Still couple of security errors for redis, postgresql and gitaly:
Couple of more denied requests for the container request because the use of fuse filesystem GlusterFS (mounted by fusefs):
type=AVC msg=audit(1600245312.188:13266182): avc: denied { create } for pid=30546 comm="redis-server" name="redis.socket" scontext=system_u:system_r:container_t:s0:c430,c692 tcontext=system_u:object_r:fusefs_t:s0 tclass=sock_file permissive=0
type=SYSCALL msg=audit(1600245312.188:13266182): arch=c000003e syscall=49 success=no exit=-13 a0=6 a1=7ffd2aa63d80 a2=6e a3=373 items=0 ppid=30344 pid=30546 auid=0 uid=997 gid=997 euid=997 suid=997 fsuid=997 egid=997 sgid=997 fsgid=997 tty=(none) ses=3900 comm="redis-server" exe="/opt/gitlab/embedded/bin/redis-server" subj=system_u:system_r:container_t:s0:c430,c692 key=(null)
type=PROCTITLE msg=audit(1600245312.188:13266182): proctitle=2F6F70742F6769746C61622F656D6265646465642F62696E2F72656469732D736572766572002F7661722F6F70742F6769746C61622F72656469732F72656469732E636F6E66
type=AVC msg=audit(1600245502.494:13266482): avc: denied { write } for pid=31994 comm="gitaly" name="ruby.1" dev="fuse" ino=13517449456874575942 scontext=system_u:system_r:container_t:s0:c430,c692 tcontext=system_u:object_r:fusefs_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1600245685.485:13266624): avc: denied { setattr } for pid=386 comm="postgres" name=".s.PGSQL.5432" dev="fuse" ino=9397654564698525528 scontext=system_u:system_r:container_t:s0:c430,c692 tcontext=system_u:object_r:fusefs_t:s0 tclass=sock_file permissive=0
type=SYSCALL msg=audit(1600245685.485:13266624): arch=c000003e syscall=90 success=no exit=-13 a0=7ffffdc8f130 a1=1ff a2=6e a3=253 items=0 ppid=32616 pid=386 auid=0 uid=996 gid=996 euid=996 suid=996 fsuid=996 egid=996 sgid=996 fsgid=996 tty=(none) ses=3900 comm="postgres" exe="/opt/gitlab/embedded/postgresql/11/bin/postgres" subj=system_u:system_r:container_t:s0:c430,c692 key=(null)
type=AVC msg=audit(1600244868.338:13265739): avc: denied { unlink } for pid=28492 comm="redis-server" name="redis.socket" dev="fuse" ino=13033874487204667813 scontext=system_u:system_r:container_t:s0:c310,c387 tcontext=system_u:object_r:fusefs_t:s0 tclass=sock_file permissive=0
type=SYSCALL msg=audit(1600244868.338:13265739): arch=c000003e syscall=87 success=no exit=-13 a0=7fe4bbe2b640 a1=7fe4bd142778 a2=17 a3=1d4 items=0 ppid=25991 pid=28492 auid=0 uid=997 gid=997 euid=997 suid=997 fsuid=997 egid=997 sgid=997 fsgid=997 tty=(none) ses=3900 comm="redis-server" exe="/opt/gitlab/embedded/bin/redis-server" subj=system_u:system_r:container_t:s0:c310,c387 key=(null)
And analyzing the audit.log with ausearch gives what should be done:
[root@lsrv1 ~]# ausearch -c 'gitaly' --raw|audit2allow
#============= container_t ==============
allow container_t fusefs_t:sock_file { create write };
[root@lsrv1 ~]# ausearch -c 'redis-server' --raw|audit2allow
#============= container_t ==============
allow container_t fusefs_t:sock_file { create setattr unlink };
[root@lsrv1 ~]# ausearch -c 'postgres' --raw|audit2allow
#============= container_t ==============
allow container_t fusefs_t:sock_file { setattr unlink };
Parse the whole file to generate a rule for the all denied records (or just separate the errors in another file):
[root@srv ~]# cat /var/log/audit/audit.log|audit2allow
#============= container_t ==============
allow container_t fusefs_t:sock_file { create setattr unlink write };
[root@srv ~]# cat /var/log/audit/audit.log|audit2allow -M unix-socket-container-fusefs
******************** IMPORTANT ***********************
To make this policy package active, execute:
semodule -i unix-socket-container-fusefs.pp
[root@srv ~]# semodule -i unix-socket-container-fusefs.pp
After importing the SELinux unix-socket-unlabeled-all.pp and unix-socket-container-fusefs.pp rules’ modules gitlab proceeds with the installation normally!