Category: software

syslog – UDP local to rsyslog and send remote with TCP and compression

This article is to show how to log Nginx’s access logs locally using UDP to the local rsyslog daemon, which will send the logs to a remote rsyslog server using TCP and compression. In general, logs could generate a lot of traffic and using UDP over distant locations could result in packet loss respectively logs’ lines loss. The idea here is to log messages locally using UDP (non-blocking way) to a local Syslog server, which will send the stream to a remote central Syslog server using TCP connections to be sure no packets are lost. In addition, we are going to enable local caching (if the remote server is temporary unreachable) and compression between the two Syslog servers.
Our goal is to use

  • UDP for our client program (Nginx in the case) for non-blocking log writes.
  • TCP between our local machine and the remote syslog server – to be sure not to lose messages on bad connectivity.
  • local caching for our client machine – not to lose messages if the remote syslog is temporary unreachable.
  • compression between the local machine and the remote syslog server.

The configuration and the commands are tested on CentOS 7, CentOS 8 and Ubuntu 18 LTS. Check out UDP remote logging here – nginx remote logging to UDP rsyslog server (CentOS 7).

STEP 1) Configure client’s local rsyslog to accept UDP log messages only if the messages’ tags are “nginx”

Couple of things should be enabled in the local client-size rsyslog daemon:

  • rsyslog to accept UDP messages. Uncomment or add the following under section “Modules” (probably the first section in the file?) in /etc/rsyslog.conf 
    $ModLoad imudp
$UDPServerRun 514

    or

    module(load="imudp")
input(type="imudp" port="514")

    The first is the old syntax, which is still supported and the second is the new syntax. For simplicity, all of the following configuration will be using the new syntax, because the old one is depricated.

  • Add a rule to catch the tag containing “nginx” and execute action to forward the messages to the remote server 
    if ($syslogtag == 'nginx:') then {
action(type="omfwd" target="10.10.10.10" port="10514" protocol="tcp" compression.Mode="single" ZipLevel="9"
queue.filename="forwarding" queue.spoolDirectory="/var/log" queue.size="1000000" queue.type="LinkedList" queue.maxFileSize="1g" queue.SaveOnShutdown="on"
action.resumeRetryCount="-1")
& stop
}

    • The options are almost self-explanatory, the important ones are there is no retry limit count of reconnecting to the server, there is in-disk cache of maximum 1G if the remote server is unavailable and the compression per message is turned on. More on actions – https://www.rsyslog.com/doc/v8-stable/configuration/actions.html, the forward module – https://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html and the queue – https://www.rsyslog.com/doc/v8-stable/rainerscript/queue_parameters.html

And restart the rsyslog:

systemctl restart rsyslog

Keep on reading!

Mirror the official Ubuntu repositories using aptly

This article is to show mainly how to work with aptly by mirroring an official Ubuntu mirror. If you want to know how to install and a brief description of what is aptly you may want to read our previous article – Install aptly under Ubuntu 18 LTS with Nginx serving the packages and the first steps

What we are going to do – this is what you need to have a mirror of an external application repository:

  1. Install aptly in Ubuntu 18 LTS
  2. Create a mirror in aptly
  3. Create a snapshot of the mirror created before
  4. Publish the snapshot to be used in other servers.

and at the last step there is an example how to use the mirror in your local machines.

STEP 1) Install aptly in Ubuntu 18.04 LTS.

As mentioned already you may follow our article on the subject – Install aptly under Ubuntu 18 LTS with Nginx serving the packages and the first steps. The following steps are based on this installation!
The aptly home directory is in “/srv/aptly”. We use the “aptly” user and change to it to manipulate the aptly installation.

STEP 2) Create a mirror in aptly.

Prepare the keys (aptly needs to have the Ubuntu keys in its trustedkeys keyring):

aptly@srv:~$ gpg --no-default-keyring --keyring /usr/share/keyrings/ubuntu-archive-keyring.gpg --export | gpg --no-default-keyring --keyring trustedkeys.gpg --import
gpg: key 3B4FE6ACC0B21F32: 3 signatures not checked due to missing keys
gpg: key 3B4FE6ACC0B21F32: public key "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" imported
gpg: key D94AA3F0EFE21092: 3 signatures not checked due to missing keys
gpg: key D94AA3F0EFE21092: public key "Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>" imported
gpg: key 871920D1991BC93C: 1 signature not checked due to a missing key
gpg: key 871920D1991BC93C: public key "Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>" imported
gpg: Total number processed: 3
gpg:               imported: 3
gpg: public key of ultimately trusted key 212A3D20E4D3351D not found
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u

Probably you would like to have “main” and “universe” for the three bionic, bionic updates and bionic security.
First, main and universe for bionic repository. main is ~16G and universe is ~136, these numbers will vary in future when more packages are added.
Two commands are need for the aptly mirror:

  1. create – create the mirror.
  2. update – download the repository contents locally.

Keep on reading!

Pages: Page 1, Page 2

nginx and proxy_cache not growing cache despite max_size is bigger – shared memory zone to blame

One of our big Nginx cache servers has recently been upgraded to have 70T storage, which is pretty good storage for a proxy. And in a hurry to configure the big storage we changed only the “max_size” option of proxy_cache_path directive! After a week in production, the proxy reached 23T and it just stopped growing with no apparent reason! Space and Nginx max_size were OK 75T total space and 70T for the proxy cache, but no Nginx had not added more space after reaching 23T occupied space for two days, which was impossible because all files were kept for 5 years and 200G per day were generated. No errors in the logs and we even use “virtual host traffic status module” – Live status information like used space and more for nginx proxy cache, but still no clue why it did not grow above this threshold of 23T! And it began to remove cached objects!

proxy_cache_path /mnt/cache levels=1:2 keys_zone=STATIC:900m inactive=42600h max_size=70000g;

It appeared we exhausted the shared memory zone limit for the zone! And Nginx cache just stopped growing.

According to the Nginx manual “One megabyte (of shared memory zone), a zone can store about 8 thousand keys“. Apparently, after 23T of files, we have passed 7 200 000 keys and exhausted the limit we configured in the proxy_cache_path line!

The solution is really simple just increase the limit for the shared memory zone for the zone.

proxy_cache_path /mnt/cache levels=1:2 keys_zone=STATIC:4000m inactive=42600h max_size=70000g;

In the past, with small cache (15T) it was enough to have 900Mbytes for the cache’s shared memory. Now we set it to 4000 Mbytes to be able to store approximately 32 000 000 keys. We have 23T and 900M of shared memory for keys (for our setup, your setup may differ a lot!!!) and setting it to 4000M, which is more than 4 times bigger than before it will probably be enough for the rest free storage to be used at full extent.
Be careful this operation will trigger the “Nginx cache loader” to load the cache index and may produce IO during this operation!

Nginx shared memory zone size

Nginx workers use shared mappings – mmap, which is different from the SYSV and POSIX shared memory (so you cannot use ipcs tool to check for shared memory). You should check how many memory currently the process is using. Here is how you can get the size of the shared memory zone occupied by the Nginx processes and as you can see each Nginx worker is around 900M of column “RSS” (Resident Set Size):

[root@srv ~]# ps -o rss,pid,comm,user,cmd -C nginx
  RSS   PID COMMAND         USER     CMD
904888 3979 nginx           nginx    nginx: worker process
905116 3980 nginx           nginx    nginx: worker process
904828 3981 nginx           nginx    nginx: worker process
905176 3982 nginx           nginx    nginx: worker process
905196 3983 nginx           nginx    nginx: worker process
905008 3984 nginx           nginx    nginx: worker process
904908 3985 nginx           nginx    nginx: worker process
905372 3986 nginx           nginx    nginx: worker process
905088 3987 nginx           nginx    nginx: worker process
902688 3988 nginx           nginx    nginx: worker process
904932 3989 nginx           nginx    nginx: worker process
905032 3990 nginx           nginx    nginx: worker process
26452  3991 nginx           nginx    nginx: cache manager process
33928  8148 nginx           root     nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf

For a single Nginx process:

[root@srv ~]# cat /proc/3981/status |grep RssShmem
RssShmem:         894240 kB

You can check the occupied inodes of your file system with df to get approximately how many files you have:

[root@srv ~]# df -i
Filesystem         Inodes   IUsed      IFree IUse% Mounted on
devtmpfs         16452656     715   16451941   1% /dev
tmpfs            16455999       1   16455998   1% /dev/shm
tmpfs            16455999    1153   16454846   1% /run
tmpfs            16455999      17   16455982   1% /sys/fs/cgroup
/dev/md1          2076704   39687    2037017   2% /
/dev/md3       1214685184 6897020 1207788164   1% /mnt/cache
tmpfs            16455999       5   16455994   1% /run/user/0

inodes around 6 897 020 and not growing for days. This number is very close to the maximum keys, which 900M key shared memory zone may store!

Two days after changing the key shared memory zone limit to 4000Mbytes:

proxy_cache_path /mnt/cache levels=1:2 keys_zone=STATIC:4000m inactive=42600h max_size=70000g;

The Nginx workers passed 900Mbytes RSS (Resident Set Size) and it reached 1Gbyte. The occupied cached sized grew with 1T and continued to grow.

[root@srv ~]# ps -o rss,pid,comm,user,cmd -C nginx
  RSS   PID COMMAND         USER     CMD
52256  8148 nginx           root     nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
1005624 16899 nginx         nginx    nginx: worker process
1005948 16900 nginx         nginx    nginx: worker process
1005936 16901 nginx         nginx    nginx: worker process
1005912 16902 nginx         nginx    nginx: worker process
1005832 16903 nginx         nginx    nginx: worker process
1005836 16904 nginx         nginx    nginx: worker process
1005868 16905 nginx         nginx    nginx: worker process
1005932 16906 nginx         nginx    nginx: worker process
1005796 16907 nginx         nginx    nginx: worker process
1005980 16908 nginx         nginx    nginx: worker process
1005848 16909 nginx         nginx    nginx: worker process
1005888 16910 nginx         nginx    nginx: worker process
26328 16911 nginx           nginx    nginx: cache manager process

The occupied inodes also increased to 7 484 291, which means the cache added around 700 000 new files.

[root@srv ~]# df -i
Filesystem         Inodes   IUsed      IFree IUse% Mounted on
devtmpfs         16452656     715   16451941    1% /dev
tmpfs            16455999       1   16455998    1% /dev/shm
tmpfs            16455999    1153   16454846    1% /run
tmpfs            16455999      17   16455982    1% /sys/fs/cgroup
/dev/md1          2076704   39690    2037014    2% /
/dev/md3       1214685184 7484582 1207200602    1% /mnt/cache
tmpfs            16455999       5   16455994    1% /run/user/0

KDE Plasma windows force resize – iKVM virtual keyboard

If you happen to use KDE Plasma these days and you encounter view problems like you cannot see the whole viewpoint of a window (especially JAVA/GTK based programs?).

KDE Plasma Desktop offers the ability to force a window to expand to new dimensions.

STEP 1) The Java-based iKVM program window has a handful virtual keyboard.

It could be used to “click on” specific key combinations, which otherwise could be caught by your system. But in sometimes the virtual keyboad window is trimmed and you lose some important keys like Ctrl, Alt, Space, arrow keys and more (the last line of buttons).

main menu
iKVM virtual keyboard trimmed keys

Keep on reading!

aplty – unable to find control.tar.gz part in package – change deb package compression from xz to gzip

We upgraded to a new version of Ubuntu and our CI (continuous integration) scripts began to throw errors when uploading packages to out aptly repository:

"Report":{"Warnings":["Unable to read file /srv/aptly/.aptly/upload/mysoft/mysoft-6.15-pk19.deb: unable to find control.tar.gz part in package /srv/aptly/.aptly/upload/mysoft/mysoft-6.15-pk19.deb"],"Added":[],"Removed":[]}}

But if the same command:

dpkg-deb --build $PKGNAME

is executed on our older Ubuntu 16 everyhting is perfect and no error when uploading the package in the repository.

It turns out the new version of dpkg 1.19.0 the dpkg-deb will compress the deb file with XZ by default. Before version 1.19 the default compression is gzip.

You may upgrade your aptly installation to 1.20 and above or just fix your script to use “-Zgzip” with dpkg-deb

dpkg-deb -Zgzip --build $PKGNAME

This command will force the dpkg-deb to use gzip to compress the debian package.

Change the compression of existing deb package

Thanks to the aptly bug report – https://github.com/aptly-dev/aptly/issues/655 (hopes this link stays forever) you may have a workarround to decompress and comrpess an existing package with antoher algorythm

dpkg-deb -R package.deb tmp
rm package.deb
fakeroot dpkg-deb -Zgzip -b tmp package.deb
rm -rf tmp

docker mysql – Fatal error: Please read “Security” section of the manual to find out how to run mysqld as root!

Pulling the official MySQL image from the docker registry https://hub.docker.com/r/mysql/mysql-server to start a MySQL instance with your configuration file (and MySQL binary files). Adding the “–volume” option for the configuration directory (or file) and MySQL binary files and you stumble on the error:

2019-12-03 01:13:38 0 [Note] mysqld (mysqld 5.6.46-log) starting as process 67 ...
2019-12-03 01:13:38 67 [ERROR] Fatal error: Please read "Security" section of the manual to find out how to run mysqld as root!

2019-12-03 01:13:38 67 [ERROR] Aborting

2019-12-03 01:13:38 67 [Note] Binlog end
2019-12-03 01:13:38 67 [Note] mysqld: Shutdown complete

Apparently, the server option is not configured to run properly as a root user and you do not want to run it, but why it keeps insisting to run it as root?

Because of the entry point script will execute only “mysqld” as a command, which expects to have a “user” option in the “[mysqld]” section of your my.cnf configuration file!

Do not miss the user option in my.cnf! This is how the MySQL server will be using the “mysql” username not the root!

user=mysql

Typical error, because it is not so common to include the username in my.cnf configuration file of the mysqld process to run as. If you use the official docker MySQL image to create your configuration file you would not encounter the above error, but if you use an existing (probably old and from non virtualized environment) my.cnf make sure to include the username, which should be used to run the mysqld process as.

Here is our command to execute the container:

docker run --privileged -d -v /mnt/storage/docker/mysql-slave/files:/var/lib/mysql -v /mnt/storage/docker/mysql-slave/etc/my.cnf:/etc/my.cnf mysql/mysql-server:5.6

Build docker image with custom Dockerfile name – docker build requires exactly 1 argument

Docker uses the Dockerfile to build docker images, but what if you want to change the name and (or) the path of this file?
By default “docker build” command uses a file named Dockerfile on the same directory you execute the “docker build“. There is an option to change the path and name of this special file:

  -f, --file string             Name of the Dockerfile (Default is 'PATH/Dockerfile')

And the “-f” may include path and file name but it is mandatory to specify the path at the end “docker build” usually the current directory (context by the docker terminology) by adding “.” (the dot at the end of the command)

So if you want to build with a docker file mydockerfile in the current directory you must execute:

docker build -f mydockerfile .

If your file is in a sub-directory execute:

docker build -f subdirectory/mydockerfile .

The command will create a docker image in your local repository. Here is the output of the first command:

root@srv:~/docker# docker build -f mydockerfile .
Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM ubuntu:bionic-20191029
bionic-20191029: Pulling from library/ubuntu
7ddbc47eeb70: Pull complete 
c1bbdc448b72: Pull complete 
8c3b70e39044: Pull complete 
45d437916d57: Pull complete 
Digest: sha256:6e9f67fa63b0323e9a1e587fd71c561ba48a034504fb804fd26fd8800039835d
Status: Downloaded newer image for ubuntu:bionic-20191029
 ---> 775349758637
Step 2/3 : MAINTAINER test@example.com
 ---> Running in 5fa42bca749c
Removing intermediate container 5fa42bca749c
 ---> 0a1ffa1728f4
Step 3/3 : RUN apt-get update && apt-get upgrade -y && apt-get install -y git wget
 ---> Running in 2e35040f247c
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
.....
.....
Processing triggers for ca-certificates (20180409) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Removing intermediate container 2e35040f247c
 ---> 2382809739a4
Successfully built 2382809739a4

Here is the image:

REPOSITORY                            TAG                 IMAGE ID            CREATED              SIZE
root@srv:~# docker images
<none>                                <none>              2382809739a4        About a minute ago   186MB

Build command with custom name and registry URL and TAG

root@srv:~# docker build -t gitlab.ahelpme.com:4567/root/ubuntu-project/ubuntu18-manual-base:v0.1 -f mydockerfile .
Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM ubuntu:bionic-20191029
 ---> 775349758637
Step 2/3 : MAINTAINER test@example.com
 ---> Using cache
 ---> 0a1ffa1728f4
Step 3/3 : RUN apt-get update && apt-get upgrade -y && apt-get install -y git wget
 ---> Using cache
 ---> 2382809739a4
Successfully built 2382809739a4
Successfully tagged gitlab.ahelpme.com:4567/root/ubuntu-project/ubuntu18-manual-base:v0.1
root@srv:~# docker push gitlab.ahelpme.com:4567/root/ubuntu-project/ubuntu18-manual-base:v0.1
The push refers to repository [gitlab.ahelpme.com:4567/root/ubuntu-project/ubuntu18-manual-base]
7cebba4bf6c3: Pushed 
e0b3afb09dc3: Pushed 
6c01b5a53aac: Pushed 
2c6ac8e5063e: Pushed 
cc967c529ced: Pushed 
v0.1: digest: sha256:acf42078bf46e320c402f09c6417a3dae8992ab4f4f685265486063daf30cb13 size: 1364

the registry URL is “gitlab.ahelpme.com:4567” and the project path is “/root/ubuntu-project/” and the name of the image is “ubuntu18-manual-base” with tag “v0.1“. The build command uses the cache from our first build example here (because the docker file is the same).

Typical errors with “-f”

Two errors you may encounter when trying the “-f” to change the name of the default Dockerfile name:

$ docker build -t gitlab.ahelpme.com:4567/root/ubuntu-project/ubuntu18-manual-base:v0.1 -f mydockerfile subdirectory/
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /builds/dev/docker-containers/mydockerfile: no such file or directory

$ docker build -t gitlab.ahelpme.com:4567/root/ubuntu-project/ubuntu18-manual-base:v0.1 -f subdirectory/mydockerfile
"docker build" requires exactly 1 argument.
See 'docker build --help'.

Usage:  docker build [OPTIONS] PATH | URL | -

First, you might think the -f would take the path and file name and this should be enough, but the errors above appears!

Our example Dockerfile

This is our simple example docker file:

FROM ubuntu:bionic-20191029
MAINTAINER test@example.com

RUN apt-get update && apt-get upgrade -y && apt-get install -y git wget

We are using the official docker image from Ubuntu. Always use official docker images!

Docker change the port mapping of an existing container

Unfortunately, it is not possible to change the port mapping (forwarded ports from the hosts to the container) of an existing RUNNING container!

Not only that, but you cannot change the mapped ports (forwarded ports) even when the container is stopped, so think twice when you run or start a container from the image you’ve chosen. Of course, you can always use docker’s commit command, which just creates a new image from you (running, in a sense of changes fro the original image) container and then you can run the new image with new mapped ports!

Still, there is a solution not involving the creation of new docker images and containers, but just to edit manually a configuration file while the Docker service is stopped.

So if you have several docker containers running you should stop all of them! When the Docker service stops, edit the “hostconfig.json” file! Here is the whole procedure:

  1. Stop the container.
  2. Stop the Docker container service.
  3. Edit the container’s file – hostconfig.json (usually in /var/lib/docker/containers/[ID]/hostconfig.json) and add or replace ports.
  4. Start the Docker container service.
  5. Start the docker container.

Real World Example

myuser@srv:~# sudo docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                  PORTS                                                                                    NAMES
a9e21e92e2dd        gitlab/gitlab-runner:latest   "/usr/bin/dumb-init …"   2 days ago          Up 33 hours                                                                                                      gitlab-runner
5d025e7f93a4        gitlab/gitlab-ce:latest       "/assets/wrapper"        3 days ago          Up 34 hours (healthy)   0.0.0.0:80->80/tcp, 0.0.0.0:4567->4567/tcp, 0.0.0.0:1022->22/tcp   gitlab
myuser@srv:~# sudo docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                  PORTS                                                                                    NAMES
a9e21e92e2dd        gitlab/gitlab-runner:latest   "/usr/bin/dumb-init …"   2 days ago          Up 33 hours                                                                                                      gitlab-runner
5d025e7f93a4        gitlab/gitlab-ce:latest       "/assets/wrapper"        3 days ago          Up 34 hours (healthy)   0.0.0.0:80->80/tcp, 0.0.0.0:4567->4567/tcp, 0.0.0.0:1022->22/tcp   gitlab
myuser@srv:~# sudo docker stop gitlab-runner
gitlab-runner
myuser@srv:~# sudo docker stop gitlab
gitlab
myuser@srv:~# sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
myuser@srv:~# systemctl stop docker
myuser@srv:~# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Thu 2019-11-14 21:54:57 UTC; 5s ago
     Docs: https://docs.docker.com
  Process: 2340 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=0/SUCCESS)
 Main PID: 2340 (code=exited, status=0/SUCCESS)

Nov 14 21:54:33 srv dockerd[2340]: time="2019-11-14T21:54:33.308531424Z" level=warning msg="a9e21e92e2dd297a68f68441353fc3bda39d0bb5564b60d402ae651fa80f5c72 cleanu
Nov 14 21:54:46 srv dockerd[2340]: time="2019-11-14T21:54:46.394643530Z" level=info msg="Container 5d025e7f93a45a50dbbaa87c55d7cdbbf6515bbe1d45ff599074f1cdcf320a0c
Nov 14 21:54:46 srv dockerd[2340]: time="2019-11-14T21:54:46.757171067Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete ty
Nov 14 21:54:47 srv dockerd[2340]: time="2019-11-14T21:54:47.031709355Z" level=warning msg="5d025e7f93a45a50dbbaa87c55d7cdbbf6515bbe1d45ff599074f1cdcf320a0c cleanu
Nov 14 21:54:57 srv systemd[1]: Stopping Docker Application Container Engine...
Nov 14 21:54:57 srv dockerd[2340]: time="2019-11-14T21:54:57.439296168Z" level=info msg="Processing signal 'terminated'"
Nov 14 21:54:57 srv dockerd[2340]: time="2019-11-14T21:54:57.447803201Z" level=info msg="Daemon shutdown complete"
Nov 14 21:54:57 srv dockerd[2340]: time="2019-11-14T21:54:57.449422219Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled
Nov 14 21:54:57 srv dockerd[2340]: time="2019-11-14T21:54:57.449576789Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled
Nov 14 21:54:57 srv systemd[1]: Stopped Docker Application Container Engine.
myuser@srv:~# cat /var/lib/docker/containers/5d025e7f93a45a50dbbaa87c55d7cdbbf6515bbe1d45ff599074f1cdcf320a0c/hostconfig.json 
{"Binds":["/srv/gitlab/config:/etc/gitlab","/srv/gitlab/logs:/var/log/gitlab","/srv/gitlab/data:/var/opt/gitlab"],"ContainerIDFile":"","LogConfig":{"Type":"json-file","Config":{}},"NetworkMode":"default","PortBindings":{"22/tcp":[{"HostIp":"","HostPort":"1022"}],"4567/tcp":[{"HostIp":"","HostPort":"4567"}],"80/tcp":[{"HostIp":"","HostPort":"80"}]},"RestartPolicy":{"Name":"always","MaximumRetryCount":0},"AutoRemove":false,"VolumeDriver":"","VolumesFrom":null,"CapAdd":null,"CapDrop":null,"Capabilities":null,"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IpcMode":"private","Cgroup":"","Links":null,"OomScoreAdj":0,"PidMode":"","Privileged":false,"PublishAllPorts":false,"ReadonlyRootfs":false,"SecurityOpt":null,"UTSMode":"","UsernsMode":"","ShmSize":67108864,"Runtime":"runc","ConsoleSize":[0,0],"Isolation":"","CpuShares":0,"Memory":0,"NanoCpus":0,"CgroupParent":"","BlkioWeight":0,"BlkioWeightDevice":[],"BlkioDeviceReadBps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteIOps":null,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpusetCpus":"","CpusetMems":"","Devices":[],"DeviceCgroupRules":null,"DeviceRequests":null,"KernelMemory":0,"KernelMemoryTCP":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":null,"OomKillDisable":false,"PidsLimit":null,"Ulimits":null,"CpuCount":0,"CpuPercent":0,"IOMaximumIOps":0,"IOMaximumBandwidth":0,"MaskedPaths":["/proc/asound","/proc/acpi","/proc/kcore","/proc/keys","/proc/latency_stats","/proc/timer_list","/proc/timer_stats","/proc/sched_debug","/proc/scsi","/sys/firmware"],"ReadonlyPaths":["/proc/bus","/proc/fs","/proc/irq","/proc/sys","/proc/sysrq-trigger"]
myuser@srv:~# nano /var/lib/docker/containers/5d025e7f93a45a50dbbaa87c55d7cdbbf6515bbe1d45ff599074f1cdcf320a0c/hostconfig.json 
myuser@srv:~# systemctl start docker
myuser@srv:~# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-11-14 22:12:06 UTC; 2s ago
     Docs: https://docs.docker.com
 Main PID: 4693 (dockerd)
    Tasks: 54
   CGroup: /system.slice/docker.service
           ├─4693 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
           ├─4867 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 4567 -container-ip 172.17.0.3 -container-port 4567
           ├─4881 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip 172.17.0.3 -container-port 443
           ├─4895 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 1022 -container-ip 172.17.0.3 -container-port 22
           └─4907 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.17.0.3 -container-port 80

Nov 14 22:12:04 srv dockerd[4693]: time="2019-11-14T22:12:04.034007956Z" level=warning msg="Your kernel does not support swap memory limit"
Nov 14 22:12:04 srv dockerd[4693]: time="2019-11-14T22:12:04.034062799Z" level=warning msg="Your kernel does not support cgroup rt period"
Nov 14 22:12:04 srv dockerd[4693]: time="2019-11-14T22:12:04.034074070Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Nov 14 22:12:04 srv dockerd[4693]: time="2019-11-14T22:12:04.034361581Z" level=info msg="Loading containers: start."
Nov 14 22:12:04 srv dockerd[4693]: time="2019-11-14T22:12:04.344354207Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Dae
Nov 14 22:12:05 srv dockerd[4693]: time="2019-11-14T22:12:05.916782317Z" level=info msg="Loading containers: done."
Nov 14 22:12:05 srv dockerd[4693]: time="2019-11-14T22:12:05.988204406Z" level=info msg="Docker daemon" commit=9013bf583a graphdriver(s)=overlay2 version=19.03.4
Nov 14 22:12:05 srv dockerd[4693]: time="2019-11-14T22:12:05.988317448Z" level=info msg="Daemon has completed initialization"
Nov 14 22:12:06 srv dockerd[4693]: time="2019-11-14T22:12:06.010801856Z" level=info msg="API listen on /var/run/docker.sock"
Nov 14 22:12:06 srv systemd[1]: Started Docker Application Container Engine.
myuser@srv:~# sudo docker start gitlab-runner
gitlab-runner
myuser@srv:~# sudo docker start gitlab
gitlab
myuser@srv:~# sudo docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                             PORTS                                                                                    NAMES
a9e21e92e2dd        gitlab/gitlab-runner:latest   "/usr/bin/dumb-init …"   2 days ago          Up 19 seconds                                                                                                               gitlab-runner
5d025e7f93a4        gitlab/gitlab-ce:latest       "/assets/wrapper"        3 days ago          Up 19 seconds (health: starting)   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:4567->4567/tcp, 0.0.0.0:1022->22/tcp   gitlab
myuser@srv:~# wget --no-check-certificate https://192.168.0.238/
--2019-11-14 22:13:30--  https://192.168.0.238/
Connecting to 192.168.0.238:443... connected.
    WARNING: certificate common name ‘gitlab.ahelpme.com’ doesn't match requested host name ‘192.168.0.238’.
HTTP request sent, awaiting response... 302 Found
Location: https://192.168.0.238/users/sign_in [following]
--2019-11-14 22:13:30--  https://192.168.0.238/users/sign_in
Reusing existing connection to 192.168.0.238:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html                                     [ <=>                                                                                     ]  12.41K  --.-KB/s    in 0s      

2019-11-14 22:13:31 (134 MB/s) - ‘index.html’ saved [12708]

Change the ports or add more ports in “PortBindings”. The syntax is pretty straightforward just mind the comas, [] and {}.

"PortBindings":{"22/tcp":[{"HostIp":"","HostPort":"1022"}],"4567/tcp":[{"HostIp":"","HostPort":"4567"}],"80/tcp":[{"HostIp":"","HostPort":"80"}]}

Here we change the mapping from “host port 1022 to 22” to “host port 2222 to 22” just replacing the “1022” to “2222”:

"PortBindings":{"22/tcp":[{"HostIp":"","HostPort":"2222"}],"4567/tcp":[{"HostIp":"","HostPort":"4567"}],"80/tcp":[{"HostIp":"","HostPort":"80"}]}

And the second example is in addition to the 2222 change we want to add another mapping “host from 443 to 443” (open the HTTPS), just add new group with the above syntax:

"PortBindings":{"22/tcp":[{"HostIp":"","HostPort":"2222"}],"4567/tcp":[{"HostIp":"","HostPort":"4567"}],"80/tcp":[{"HostIp":"","HostPort":"80"}],"443/tcp":[{"HostIp":"","HostPort":"443"}]}

A note!

Probably there may be an idea not to be easy to add mapped ports when you think one of the main Docker goals is to isolate services per a Docker instance. It sounds strange to have a docker container for one service exporting a number of ports (or a single port) and later why you would need to expose another port? For another service in the same container, but you should use a separate container, not the same one!
But more and more Docker containers are used also to deliver a fine-tuned environment of a whole platform, which provides multiple services in a single docker container. Let’s take an example – GitLab, which offers installation in a Docker container hosting more than 10 services in a single container!

docker and dind service (.gitlab-ci.yml) with self-signed certificate and x509: certificate signed by unknown authority

When using GitLab and the CI/CD for building docker images you may stumble on such error using the “docker:dind” (dind stands for docker in docker) image:

$ docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $REGISTRY_URL
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://gitlab.ahelpme.com:4567/v2/: x509: certificate signed by unknown authority
ERROR: Job failed: exit code 1

In our case, because “docker build” command needs a docker service to be running and the GitLab runner needs to provide this docker service so docker:dind is our best option! A self-signed certificate could be really difficult to use in such a big platform as GitLab, but no matter whatever might be the reasons to use docker service in a docker container you may need to use a custom registry with a self-signed certificate!

There are two options to use self-signed certificates with docker:

  1. Add the self-signed certificate in “/etc/docker/certs.d/[custom_registry]/ca.crt”. custom_registry must include the port, for example: “/etc/docker/certs.d/gitlab.example.com\:4567/ca.crt” and restart the docker service! This could be difficult when you use GitLab CI/CD and .gitlab-ci.yml
  2. Add “–insecure-registry” in docker configuration and restart. Apperantly it is easier than the first option when using GitLab CI/CD .gitlab-ci.yml.

The solution

In the GitLab CI/CD file .gitlab-ci.yml add two options (entrypoint, command) to the services, which provides the “dind” (docker in docker). The start of your should start with something like:

image: docker:18.09.7
services:
  - name: docker:18.09.7-dind
    entrypoint: ["dockerd-entrypoint.sh"]
    command: ["--insecure-registry", "gitlab.ahelpme.com:4567"]

Of course, replace the “gitlab.ahelpme.com:4567” with your custom docker registry domain.

Real world example – failed job in gitlab-runner

Keep on reading!

Install gitlab-ce (community edition) in docker container with HTTPS and docker registry

This article is a howto install of the official docker gitlab-ce (GitLab Community Edition). GitLab maintains a docker image in the Docker registry and this is the best way to install GitLab.
In this article you are going to learn how:

  • to install the GitLab CE in docker
  • to enable HTTPS (SSL) web support to your GitLab
  • to enable the docker registry functionality of GitLab

To install GitLab docker image in your Linux distribution all you need is a working docker environment and started docker daemon. As you know, installing software with docker will allow you to keep your main system clean and let you use a fined tuned installation from the official developer (creator). As mentioned already, the GitLab maintains an official GitLab image in the Docker Registry so you may expect everything to work smoothly and better than if you make an installation in a clean Linux distribution like CentOS, Ubuntu and so on. In this article, we will include the most important docker commands to control and configure the GitLab docker container and even if you are not familiar with the Docker software they are simple enough to use them and prefer this method over GitLab normal installation.

GitLab has integrated the Docker Container Registry in GitLab Container Registry and now with GitLab you can have a local Docker registry containing all project’s docker images!

Just to note, the Docker Registry is the place for the Docker (aka Linux) images.
Using GitLab Container Registry with CI/CD (continuous integration and continuous delivery) you can create automatically test, staging, development and production docker images.
Keep on reading!