gitlab in podman cannot create unix sockets in glusterfs because of SELinux

Installing gitlab-ee (and gitlab-ce) under CentOS 7 with enabled SELinux (i.e. enforcing mode) looped endlessly the container in restarting the installation process! There were multiple errors for missing sockets in the podman logs of the gitlab container. Here are some of the errors:
Missing postgresql unix socket in “/var/opt/gitlab/postgresql”:

Recipe: gitlab::database_migrations
  * bash[migrate gitlab-rails database] action run
    [execute] rake aborted!
              PG::ConnectionBad: could not connect to server: No such file or directory
                Is the server running locally and accepting
                connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:53:in `block (3 levels) in <top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              Tasks: TOP => gitlab:db:configure
              (See full trace by running task with --trace)
    
    
    Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
.....
.....
Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200915-35-lemic5" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:53:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR: 
---- End output of "bash"  "/tmp/chef-script20200915-35-lemic5" ----
Ran "bash"  "/tmp/chef-script20200915-35-lemic5" returned 1

Missing redis socket in

Running handlers:
There was an error running gitlab-ctl reconfigure:

redis_service[redis] (redis::enable line 19) had an error: RuntimeError: ruby_block[warn pending redis restart] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/resources/service.rb line 65) had an error: RuntimeError: Execution of the command `/opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket INFO` failed with a non-zero exit code (1)
stdout: 
stderr: Could not connect to Redis at /var/opt/gitlab/redis/redis.socket: No such file or directory

It should be noted that the /var/opt/gitlab directory has been mapped in /mnt/storage/podman/gitlab/data. GlusterFS is used for /mnt/storage, so the gitlab files resides on a GlusterFS volume.

ERROR 1) Cannot create unix socket.

Checking the /var/log/audit/audit.log reveiled the problem immediately:
Keep on reading!

docker and dind service (.gitlab-ci.yml) with self-signed certificate and x509: certificate signed by unknown authority

When using GitLab and the CI/CD for building docker images you may stumble on such error using the “docker:dind” (dind stands for docker in docker) image:

$ docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $REGISTRY_URL
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://gitlab.ahelpme.com:4567/v2/: x509: certificate signed by unknown authority
ERROR: Job failed: exit code 1

In our case, because “docker build” command needs a docker service to be running and the GitLab runner needs to provide this docker service so docker:dind is our best option! A self-signed certificate could be really difficult to use in such a big platform as GitLab, but no matter whatever might be the reasons to use docker service in a docker container you may need to use a custom registry with a self-signed certificate!

There are two options to use self-signed certificates with docker:

  1. Add the self-signed certificate in “/etc/docker/certs.d/[custom_registry]/ca.crt”. custom_registry must include the port, for example: “/etc/docker/certs.d/gitlab.example.com\:4567/ca.crt” and restart the docker service! This could be difficult when you use GitLab CI/CD and .gitlab-ci.yml
  2. Add “–insecure-registry” in docker configuration and restart. Apperantly it is easier than the first option when using GitLab CI/CD .gitlab-ci.yml.

The solution

In the GitLab CI/CD file .gitlab-ci.yml add two options (entrypoint, command) to the services, which provides the “dind” (docker in docker). The start of your should start with something like:

image: docker:18.09.7
services:
  - name: docker:18.09.7-dind
    entrypoint: ["dockerd-entrypoint.sh"]
    command: ["--insecure-registry", "gitlab.ahelpme.com:4567"]

Of course, replace the “gitlab.ahelpme.com:4567” with your custom docker registry domain.

Real world example – failed job in gitlab-runner

Keep on reading!

Install gitlab-ce (community edition) in docker container with HTTPS and docker registry

This article is a howto install of the official docker gitlab-ce (GitLab Community Edition). GitLab maintains a docker image in the Docker registry and this is the best way to install GitLab.
In this article you are going to learn how:

  • to install the GitLab CE in docker
  • to enable HTTPS (SSL) web support to your GitLab
  • to enable the docker registry functionality of GitLab

To install GitLab docker image in your Linux distribution all you need is a working docker environment and started docker daemon. As you know, installing software with docker will allow you to keep your main system clean and let you use a fined tuned installation from the official developer (creator). As mentioned already, the GitLab maintains an official GitLab image in the Docker Registry so you may expect everything to work smoothly and better than if you make an installation in a clean Linux distribution like CentOS, Ubuntu and so on. In this article, we will include the most important docker commands to control and configure the GitLab docker container and even if you are not familiar with the Docker software they are simple enough to use them and prefer this method over GitLab normal installation.

GitLab has integrated the Docker Container Registry in GitLab Container Registry and now with GitLab you can have a local Docker registry containing all project’s docker images!

Just to note, the Docker Registry is the place for the Docker (aka Linux) images.
Using GitLab Container Registry with CI/CD (continuous integration and continuous delivery) you can create automatically test, staging, development and production docker images.
Keep on reading!