gitlab in podman cannot create unix sockets in glusterfs because of SELinux

Installing gitlab-ee (and gitlab-ce) under CentOS 7 with enabled SELinux (i.e. enforcing mode) looped endlessly the container in restarting the installation process! There were multiple errors for missing sockets in the podman logs of the gitlab container. Here are some of the errors:
Missing postgresql unix socket in “/var/opt/gitlab/postgresql”:

Recipe: gitlab::database_migrations
  * bash[migrate gitlab-rails database] action run
    [execute] rake aborted!
              PG::ConnectionBad: could not connect to server: No such file or directory
                Is the server running locally and accepting
                connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:53:in `block (3 levels) in <top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              Tasks: TOP => gitlab:db:configure
              (See full trace by running task with --trace)
    
    
    Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
.....
.....
Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200915-35-lemic5" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:53:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR: 
---- End output of "bash"  "/tmp/chef-script20200915-35-lemic5" ----
Ran "bash"  "/tmp/chef-script20200915-35-lemic5" returned 1

Missing redis socket in

Running handlers:
There was an error running gitlab-ctl reconfigure:

redis_service[redis] (redis::enable line 19) had an error: RuntimeError: ruby_block[warn pending redis restart] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/resources/service.rb line 65) had an error: RuntimeError: Execution of the command `/opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket INFO` failed with a non-zero exit code (1)
stdout: 
stderr: Could not connect to Redis at /var/opt/gitlab/redis/redis.socket: No such file or directory

It should be noted that the /var/opt/gitlab directory has been mapped in /mnt/storage/podman/gitlab/data. GlusterFS is used for /mnt/storage, so the gitlab files resides on a GlusterFS volume.

ERROR 1) Cannot create unix socket.

Checking the /var/log/audit/audit.log reveiled the problem immediately:
Keep on reading!

Docker change the port mapping of an existing container

Unfortunately, it is not possible to change the port mapping (forwarded ports from the hosts to the container) of an existing RUNNING container!

Not only that, but you cannot change the mapped ports (forwarded ports) even when the container is stopped, so think twice when you run or start a container from the image you’ve chosen. Of course, you can always use docker’s commit command, which just creates a new image from you (running, in a sense of changes fro the original image) container and then you can run the new image with new mapped ports!

Still, there is a solution not involving the creation of new docker images and containers, but just to edit manually a configuration file while the Docker service is stopped.

So if you have several docker containers running you should stop all of them! When the Docker service stops, edit the “hostconfig.json” file! Here is the whole procedure:

  1. Stop the container.
  2. Stop the Docker container service.
  3. Edit the container’s file – hostconfig.json (usually in /var/lib/docker/containers/[ID]/hostconfig.json) and add or replace ports.
  4. Start the Docker container service.
  5. Start the docker container.

Real World Example

myuser@srv:~# sudo docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                  PORTS                                                                                    NAMES
a9e21e92e2dd        gitlab/gitlab-runner:latest   "/usr/bin/dumb-init …"   2 days ago          Up 33 hours                                                                                                      gitlab-runner
5d025e7f93a4        gitlab/gitlab-ce:latest       "/assets/wrapper"        3 days ago          Up 34 hours (healthy)   0.0.0.0:80->80/tcp, 0.0.0.0:4567->4567/tcp, 0.0.0.0:1022->22/tcp   gitlab
myuser@srv:~# sudo docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                  PORTS                                                                                    NAMES
a9e21e92e2dd        gitlab/gitlab-runner:latest   "/usr/bin/dumb-init …"   2 days ago          Up 33 hours                                                                                                      gitlab-runner
5d025e7f93a4        gitlab/gitlab-ce:latest       "/assets/wrapper"        3 days ago          Up 34 hours (healthy)   0.0.0.0:80->80/tcp, 0.0.0.0:4567->4567/tcp, 0.0.0.0:1022->22/tcp   gitlab
myuser@srv:~# sudo docker stop gitlab-runner
gitlab-runner
myuser@srv:~# sudo docker stop gitlab
gitlab
myuser@srv:~# sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
myuser@srv:~# systemctl stop docker
myuser@srv:~# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Thu 2019-11-14 21:54:57 UTC; 5s ago
     Docs: https://docs.docker.com
  Process: 2340 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=0/SUCCESS)
 Main PID: 2340 (code=exited, status=0/SUCCESS)

Nov 14 21:54:33 srv dockerd[2340]: time="2019-11-14T21:54:33.308531424Z" level=warning msg="a9e21e92e2dd297a68f68441353fc3bda39d0bb5564b60d402ae651fa80f5c72 cleanu
Nov 14 21:54:46 srv dockerd[2340]: time="2019-11-14T21:54:46.394643530Z" level=info msg="Container 5d025e7f93a45a50dbbaa87c55d7cdbbf6515bbe1d45ff599074f1cdcf320a0c
Nov 14 21:54:46 srv dockerd[2340]: time="2019-11-14T21:54:46.757171067Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete ty
Nov 14 21:54:47 srv dockerd[2340]: time="2019-11-14T21:54:47.031709355Z" level=warning msg="5d025e7f93a45a50dbbaa87c55d7cdbbf6515bbe1d45ff599074f1cdcf320a0c cleanu
Nov 14 21:54:57 srv systemd[1]: Stopping Docker Application Container Engine...
Nov 14 21:54:57 srv dockerd[2340]: time="2019-11-14T21:54:57.439296168Z" level=info msg="Processing signal 'terminated'"
Nov 14 21:54:57 srv dockerd[2340]: time="2019-11-14T21:54:57.447803201Z" level=info msg="Daemon shutdown complete"
Nov 14 21:54:57 srv dockerd[2340]: time="2019-11-14T21:54:57.449422219Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled
Nov 14 21:54:57 srv dockerd[2340]: time="2019-11-14T21:54:57.449576789Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled
Nov 14 21:54:57 srv systemd[1]: Stopped Docker Application Container Engine.
myuser@srv:~# cat /var/lib/docker/containers/5d025e7f93a45a50dbbaa87c55d7cdbbf6515bbe1d45ff599074f1cdcf320a0c/hostconfig.json 
{"Binds":["/srv/gitlab/config:/etc/gitlab","/srv/gitlab/logs:/var/log/gitlab","/srv/gitlab/data:/var/opt/gitlab"],"ContainerIDFile":"","LogConfig":{"Type":"json-file","Config":{}},"NetworkMode":"default","PortBindings":{"22/tcp":[{"HostIp":"","HostPort":"1022"}],"4567/tcp":[{"HostIp":"","HostPort":"4567"}],"80/tcp":[{"HostIp":"","HostPort":"80"}]},"RestartPolicy":{"Name":"always","MaximumRetryCount":0},"AutoRemove":false,"VolumeDriver":"","VolumesFrom":null,"CapAdd":null,"CapDrop":null,"Capabilities":null,"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IpcMode":"private","Cgroup":"","Links":null,"OomScoreAdj":0,"PidMode":"","Privileged":false,"PublishAllPorts":false,"ReadonlyRootfs":false,"SecurityOpt":null,"UTSMode":"","UsernsMode":"","ShmSize":67108864,"Runtime":"runc","ConsoleSize":[0,0],"Isolation":"","CpuShares":0,"Memory":0,"NanoCpus":0,"CgroupParent":"","BlkioWeight":0,"BlkioWeightDevice":[],"BlkioDeviceReadBps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteIOps":null,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpusetCpus":"","CpusetMems":"","Devices":[],"DeviceCgroupRules":null,"DeviceRequests":null,"KernelMemory":0,"KernelMemoryTCP":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":null,"OomKillDisable":false,"PidsLimit":null,"Ulimits":null,"CpuCount":0,"CpuPercent":0,"IOMaximumIOps":0,"IOMaximumBandwidth":0,"MaskedPaths":["/proc/asound","/proc/acpi","/proc/kcore","/proc/keys","/proc/latency_stats","/proc/timer_list","/proc/timer_stats","/proc/sched_debug","/proc/scsi","/sys/firmware"],"ReadonlyPaths":["/proc/bus","/proc/fs","/proc/irq","/proc/sys","/proc/sysrq-trigger"]
myuser@srv:~# nano /var/lib/docker/containers/5d025e7f93a45a50dbbaa87c55d7cdbbf6515bbe1d45ff599074f1cdcf320a0c/hostconfig.json 
myuser@srv:~# systemctl start docker
myuser@srv:~# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-11-14 22:12:06 UTC; 2s ago
     Docs: https://docs.docker.com
 Main PID: 4693 (dockerd)
    Tasks: 54
   CGroup: /system.slice/docker.service
           ├─4693 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
           ├─4867 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 4567 -container-ip 172.17.0.3 -container-port 4567
           ├─4881 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip 172.17.0.3 -container-port 443
           ├─4895 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 1022 -container-ip 172.17.0.3 -container-port 22
           └─4907 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.17.0.3 -container-port 80

Nov 14 22:12:04 srv dockerd[4693]: time="2019-11-14T22:12:04.034007956Z" level=warning msg="Your kernel does not support swap memory limit"
Nov 14 22:12:04 srv dockerd[4693]: time="2019-11-14T22:12:04.034062799Z" level=warning msg="Your kernel does not support cgroup rt period"
Nov 14 22:12:04 srv dockerd[4693]: time="2019-11-14T22:12:04.034074070Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Nov 14 22:12:04 srv dockerd[4693]: time="2019-11-14T22:12:04.034361581Z" level=info msg="Loading containers: start."
Nov 14 22:12:04 srv dockerd[4693]: time="2019-11-14T22:12:04.344354207Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Dae
Nov 14 22:12:05 srv dockerd[4693]: time="2019-11-14T22:12:05.916782317Z" level=info msg="Loading containers: done."
Nov 14 22:12:05 srv dockerd[4693]: time="2019-11-14T22:12:05.988204406Z" level=info msg="Docker daemon" commit=9013bf583a graphdriver(s)=overlay2 version=19.03.4
Nov 14 22:12:05 srv dockerd[4693]: time="2019-11-14T22:12:05.988317448Z" level=info msg="Daemon has completed initialization"
Nov 14 22:12:06 srv dockerd[4693]: time="2019-11-14T22:12:06.010801856Z" level=info msg="API listen on /var/run/docker.sock"
Nov 14 22:12:06 srv systemd[1]: Started Docker Application Container Engine.
myuser@srv:~# sudo docker start gitlab-runner
gitlab-runner
myuser@srv:~# sudo docker start gitlab
gitlab
myuser@srv:~# sudo docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                             PORTS                                                                                    NAMES
a9e21e92e2dd        gitlab/gitlab-runner:latest   "/usr/bin/dumb-init …"   2 days ago          Up 19 seconds                                                                                                               gitlab-runner
5d025e7f93a4        gitlab/gitlab-ce:latest       "/assets/wrapper"        3 days ago          Up 19 seconds (health: starting)   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:4567->4567/tcp, 0.0.0.0:1022->22/tcp   gitlab
myuser@srv:~# wget --no-check-certificate https://192.168.0.238/
--2019-11-14 22:13:30--  https://192.168.0.238/
Connecting to 192.168.0.238:443... connected.
    WARNING: certificate common name ‘gitlab.ahelpme.com’ doesn't match requested host name ‘192.168.0.238’.
HTTP request sent, awaiting response... 302 Found
Location: https://192.168.0.238/users/sign_in [following]
--2019-11-14 22:13:30--  https://192.168.0.238/users/sign_in
Reusing existing connection to 192.168.0.238:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html                                     [ <=>                                                                                     ]  12.41K  --.-KB/s    in 0s      

2019-11-14 22:13:31 (134 MB/s) - ‘index.html’ saved [12708]


Change the ports or add more ports in “PortBindings”. The syntax is pretty straightforward just mind the comas, [] and {}.

"PortBindings":{"22/tcp":[{"HostIp":"","HostPort":"1022"}],"4567/tcp":[{"HostIp":"","HostPort":"4567"}],"80/tcp":[{"HostIp":"","HostPort":"80"}]}

Here we change the mapping from “host port 1022 to 22” to “host port 2222 to 22” just replacing the “1022” to “2222”:

"PortBindings":{"22/tcp":[{"HostIp":"","HostPort":"2222"}],"4567/tcp":[{"HostIp":"","HostPort":"4567"}],"80/tcp":[{"HostIp":"","HostPort":"80"}]}

And the second example is in addition to the 2222 change we want to add another mapping “host from 443 to 443” (open the HTTPS), just add new group with the above syntax:

"PortBindings":{"22/tcp":[{"HostIp":"","HostPort":"2222"}],"4567/tcp":[{"HostIp":"","HostPort":"4567"}],"80/tcp":[{"HostIp":"","HostPort":"80"}],"443/tcp":[{"HostIp":"","HostPort":"443"}]}

A note!

Probably there may be an idea not to be easy to add mapped ports when you think one of the main Docker goals is to isolate services per a Docker instance. It sounds strange to have a docker container for one service exporting a number of ports (or a single port) and later why you would need to expose another port? For another service in the same container, but you should use a separate container, not the same one!
But more and more Docker containers are used also to deliver a fine-tuned environment of a whole platform, which provides multiple services in a single docker container. Let’s take an example – GitLab, which offers installation in a Docker container hosting more than 10 services in a single container!

docker and dind service (.gitlab-ci.yml) with self-signed certificate and x509: certificate signed by unknown authority

When using GitLab and the CI/CD for building docker images you may stumble on such error using the “docker:dind” (dind stands for docker in docker) image:

$ docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $REGISTRY_URL
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://gitlab.ahelpme.com:4567/v2/: x509: certificate signed by unknown authority
ERROR: Job failed: exit code 1

In our case, because “docker build” command needs a docker service to be running and the GitLab runner needs to provide this docker service so docker:dind is our best option! A self-signed certificate could be really difficult to use in such a big platform as GitLab, but no matter whatever might be the reasons to use docker service in a docker container you may need to use a custom registry with a self-signed certificate!

There are two options to use self-signed certificates with docker:

  1. Add the self-signed certificate in “/etc/docker/certs.d/[custom_registry]/ca.crt”. custom_registry must include the port, for example: “/etc/docker/certs.d/gitlab.example.com\:4567/ca.crt” and restart the docker service! This could be difficult when you use GitLab CI/CD and .gitlab-ci.yml
  2. Add “–insecure-registry” in docker configuration and restart. Apperantly it is easier than the first option when using GitLab CI/CD .gitlab-ci.yml.

The solution

In the GitLab CI/CD file .gitlab-ci.yml add two options (entrypoint, command) to the services, which provides the “dind” (docker in docker). The start of your should start with something like:

image: docker:18.09.7
services:
  - name: docker:18.09.7-dind
    entrypoint: ["dockerd-entrypoint.sh"]
    command: ["--insecure-registry", "gitlab.ahelpme.com:4567"]

Of course, replace the “gitlab.ahelpme.com:4567” with your custom docker registry domain.

Real world example – failed job in gitlab-runner

Keep on reading!

Install gitlab-ce (community edition) in docker container with HTTPS and docker registry

This article is a howto install of the official docker gitlab-ce (GitLab Community Edition). GitLab maintains a docker image in the Docker registry and this is the best way to install GitLab.
In this article you are going to learn how:

  • to install the GitLab CE in docker
  • to enable HTTPS (SSL) web support to your GitLab
  • to enable the docker registry functionality of GitLab

To install GitLab docker image in your Linux distribution all you need is a working docker environment and started docker daemon. As you know, installing software with docker will allow you to keep your main system clean and let you use a fined tuned installation from the official developer (creator). As mentioned already, the GitLab maintains an official GitLab image in the Docker Registry so you may expect everything to work smoothly and better than if you make an installation in a clean Linux distribution like CentOS, Ubuntu and so on. In this article, we will include the most important docker commands to control and configure the GitLab docker container and even if you are not familiar with the Docker software they are simple enough to use them and prefer this method over GitLab normal installation.

GitLab has integrated the Docker Container Registry in GitLab Container Registry and now with GitLab you can have a local Docker registry containing all project’s docker images!

Just to note, the Docker Registry is the place for the Docker (aka Linux) images.
Using GitLab Container Registry with CI/CD (continuous integration and continuous delivery) you can create automatically test, staging, development and production docker images.
Keep on reading!