Author: neoX

Upload files and directories with swift in OpenStack

First, you need to install

swift command line utility

and here is how to do it: Install OpenStack swift client only
In general, you will need:

  1. username (–os-username) – Username
  2. password (–os-password) – Password
  3. authentication url (–os-auth-url) – The URL address, which authorize your requests, it generates a security token for your operations. Always use https!
  4. tenant name (–os-tenant-name) – Tenant is like a project.

All of the above information should be available from your OpenStack administrator.
For the examples we assume there is a container “mytest” (it’s like a main directory from the root). You cannot upload files in the root, because this is the place for containers only i.e. directories. You must always upload files under container (i.e. directory aka folder).

To upload a single file with swift cli execute:

myuser@myserver:~$ swift --os-username myuser --os-tenant-name mytenant --os-password mypass --os-auth-url https://auth-url.example.com/v2.0/ upload mytest ./file1.log 
file1.log

Keep on reading!

aptly – ERROR: unable to remove: published repo with storage:prefix/distribution ./mytest-stable not found

Sometimes the user manual may be unclear and you came here searching for a solution of dropping a published repository.
We have aptly version: 1.3.0 and here is the right syntax to remove a published repository.

First list the published repositories and reverse the “/” replacing it with space

The commands will be:

aptly publish list
Published repositories:
  * <name-distribution>/<release> [amd64] publishes {main: [xenial-<name>]: Some description}
aptly publish drop -force-drop <release> <name-distribution>

“name-distribution” is the “http://aptly.example.com/[name-distribution]” in the URL. For example, the repository URL of myrepo is “http://aptly.example.com/myrepo” and the name-distribution is “myrepo”.

A real world example

root@srv-aptly:~# aptly --config="/etc/aptly/.aptly.conf" publish list
Published repositories:
  * myrepo/stable [amd64] publishes {main: [xenial-myrepo]: Stable myrepo packages}
  * test/test [amd64] publishes {test: [test]: Test repo}
root@srv-aptly:~# aptly --config="/etc/aptly/.aptly.conf" publish list --raw
myrepo stable
test test

We want to remove “myrepo/stable”:

root@srv-aptly:~# aptly --config="/etc/aptly/.aptly.conf" publish drop -force-drop stable myrepo
Removing /etc/aptly/.aptly/public/etc/dists...
Removing /etc/aptly/.aptly/public/etc/pool...

The published repository has been removed successfully.
root@srv-aptly:~#

The wrong syntax

You might have tried it that’s why you came here:

root@srv-aptly:~# aptly --config="/etc/aptly/.aptly.conf" publish list           
Published repositories:
  * myrepo/stable [amd64] publishes {main: [xenial-myrepo]: Stable myrepo packages}
  * test/test [amd64] publishes {test: [test]: Test repo}
root@srv-aptly:~# aptly --config="/etc/aptly/.aptly.conf" publish list --raw
myrepo stable
test test
root@srv-aptly:~# aptly --config="/etc/aptly/.aptly.conf" publish drop myrepo
ERROR: unable to remove: published repo with storage:prefix/distribution ./myrepo not found
root@srv-aptly:~# aptly --config="/etc/aptly/.aptly.conf" publish drop myrepo stable
ERROR: unable to remove: published repo with storage:prefix/distribution stable/myrepo not found
root@srv-aptly:~# aptly --config="/etc/aptly/.aptly.conf" publish drop myrepo-stable
ERROR: unable to remove: published repo with storage:prefix/distribution ./myrepo-stable not found
root@srv-aptly:~# aptly --config="/etc/aptly/.aptly.conf" publish drop -force-drop myrepo-stable
ERROR: unable to remove: published repo with storage:prefix/distribution ./myrepo-stable not found
root@srv-aptly:~# aptly --config="/etc/aptly/.aptly.conf" publish drop -force-drop myrepo stable
ERROR: unable to remove: published repo with storage:prefix/distribution stable/myrepo not found
root@srv-aptly:~#

aptly mirror – gpgv: Can’t check signature: public key not found

If you want to mirror repositories from your current aptly server to a new server you must import the GPG key from your old server because you are going to encounter the following error:

gpgv: Signature made Fri 22 Apr 2019 17:35:04 AM UTC using DSA key ID FDC7A25E
gpgv: Can't check signature: public key not found

Looks like some keys are missing in your trusted keyring, you may consider importing them from keyserver:

gpg --no-default-keyring --keyring trustedkeys.gpg --keyserver pool.sks-keyservers.net --recv-keys 181482CCFDC7A25E

Sometimes keys are stored in repository root in file named Release.key, to import such key:

wget -O - https://some.repo/repository/Release.key | gpg --no-default-keyring --keyring trustedkeys.gpg --import

ERROR: unable to fetch mirror: verification of detached signature failed: exit status 2

And the mirror command fails. The problem is

you must import the GPG key from your old server in trustedkeys.gpg (even if you have already imported it in the new server with apt-key!!!)

Here is how to list, export and import it (we are going to import it in default and trustedkeys.gpg, because it is more convenient, but it is not mandatory to be in the default).
Keep on reading!

Install OpenStack swift client only

To manage files in the OpenStack cloud you need the swift client. This is not so tiny python tool (a lot of dependencies), which offers by means of files (as files are objects for the OpenStack):

  • delete – Delete files, directories and sub-directories. Be careful with a simple command you can delete all your files at once. “Delete a container or objects within a container.”
  • download – Download files to your local storage. “Download objects from containers.”
  • list – List all files in the main directory (i.e. the container), cannot be listed files in sub-directories. The output will be a file with path per line or you can enable extended output to show file size, time modified, the type of the file and the full file path. “Lists the containers for the account or the objects for a container.”
  • post – Creates containers and updates metadata, could create seurity keys for temporary urls – “Updates meta information for the account, container, or object; creates containers if not present.”
  • copy – Copy files to a new place within a container or to a different container. “Copies object, optionally adds meta”
  • stat – Display information for files, container and your account. No information is available per directory. Most typically you will read the file length, the type and modifcation date. “Displays information for the account, container, or object”
  • upload – uploads files and whole directories in a container – “Uploads files or directories to the given container.” You can read our article here – Upload files and directories with swift in OpenStack
  • capabilities – List the configuration options of your account like file size limits, file amount limits , limit requests per seconds and so on and which additional middleware your instance supports like bulk_delete, bulk_upload (the name are self-explainatory) – “List cluster capabilities.”
  • tempurl – Generate a temporary url for a file protected with time validity and a key – “Create a temporary URL.”
  • auth – Show your authentication token and storage url – “Display auth related environment variables.”

The text in the hyphens is from the swift help command. If you do not know what is a container with simple words these are the main sub-directories in your account if you use the list command without any argument (not adding a name after “list”):

myuser@myserver:~$ swift --os-username myusr --os-tenant-name myusr --os-password mypass --os-auth-url https://auth01.example.com:5000/v2.0 list
mycontainer1
anyname

The best way is to install the swift client from the package system of your Linux Distribution. The package system ensures the programs you install and upgrade are consistent within dependencies upgrades.

When you install this package “python-swiftclient” it depends on multiple python packages, which might be upgraded later, too, but the package system of a Linux distribution ensure the programs of python-swiftclient will work with them

. On the contrary, if you install it manually even with “pip” as it is offered in OpenStack site it is unsure what will happen and even your client program from “python-swiftclient” could stop working because of an upgraded dependancy library. Of course, the packages from the official package system could be a little bit older (but probably more stable) than the manual installs from the OpenStack official site. Still if you would like to install it with “pip” here you can check how you can do it: https://www.swiftstack.com/docs/integration/python-swiftclient.html
Keep on reading!

Expand disk and the root partition of the QEMU virtual server

This article is to show how easy is to grow the size of QEMU virtual disk and its partitions (along with ext4 file system). Of course, you can use this article as an example of expanding the partitions of a physical disk.

Our setup is a QEMU virtual server using a raw image of 20G and the steps are as follow:

  1. Stop the virtual server
  2. Resize with qemu-img the raw image of the virtual server
  3. Start the virtual server
  4. Get a root ssh shell (probably by using openssh)
  5. Use parted to resize the partition (and fix the GPT of the disk – not the disk is larger, so the GPT table need fixing).
  6. Use resize2fs to resize the

STEP 1) Power off your virtual server.

The best way is to power it off within the server with the “poweroff” command. Be careful to check whether the host server killed the QEMU process. It is almost certain if the VNC port is released, the QEMU process has been exited.
If you use virsh (i.e. libvirt), you may execute:

virsh shutdown my-private-vm-01
virsh destroy my-private-vm-01

The destroy command ensures there is no QEMU process, which still operates over the image disk file. But it is dangerous for your data if you issue it on a running virtual server, because it may lose the unsaved data.
If you use QEMU manually wait for the process to exit or if you have enabled the management console connect to it using telnet and just quit – this will destory the QEMU virtual server process – again be careful with unsaved data.

[root@lsrv1 ~]# ps axuf|grep qemu
root     15575  2.3 50.1 13061032 8112212 ?    Sl   May08 1522:27 qemu-system-x86_64 -enable-kvm -smp 4,maxcpus=8 -daemonize -vnc :30 -cdrom /mnt/vm/isos/CentOS-7-x86_64-Minimal-1810.iso -drive file=/mnt/vm/images/templatesrv-wordpress.bin,cache=none,aio=threads,if=virtio -boot c -net nic,model=virtio,macaddr=00:00:00:00:00:30 -net tap,ifname=tap30,script=no,downscript=no -balloon virtio -m 8144 -monitor telnet:127.0.0.1:5830,server,nowait
[root@srv-host ~]# telnet 127.0.0.1 5830
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
QEMU 2.0.0 monitor - type 'help' for more information
(qemu) q
Connection closed by foreign host.

If you use a web interface (for example WebVirtMgr) check whether the virtual server is in power-off state.

STEP 2) Resize the image file of the virtual server.

Find where are located the virtual servers’ image files in your installation and use qemu-img. We want to increse the size with 174GB to 200GB.

qemu-img resize my-private-vm-01.img +174GB

STEP 3) Start your server.

Start your server by issuing a command with virsh or QEMU (qemu-system-x86_64) or from a web interface if use one (like WebVirtMgr).
*virsh and libvirt:

virsh start my-private-vm-01

*Manual start of QEMU emulator – qemu-system-x86_64:

qemu-system-x86_64 -enable-kvm -smp 4,maxcpus=8 -daemonize -vnc :30 -cdrom /mnt/vm/isos/CentOS-7-x86_64-Minimal-1810.iso -drive file=/mnt/vm/images/templatesrv-wordpress.bin,cache=none,aio=threads,if=virtio -boot c -net nic,model=virtio,macaddr=00:00:00:00:00:30 -net tap,ifname=tap30,script=no,downscript=no -balloon virtio -m 8144 -monitor telnet:127.0.0.1:5830,server,nowait

Or just use the web browser and start the virtual server from WebVirtMgr if it is what you use.

STEP 4) Open a shell to your server.

We use openssh client to connect to our server.

STEP 5) Use parted to resize the partition.

The program “parted” will report that the partition table does not use the whole available disk, which is perfectly normal because we’ve just increased the disk size. Just confirm to fix the GPT partition table:

parted /dev/vda
GNU Parted 3.2
Using /dev/vda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Warning: Not all of the space available to /dev/vda appears to be used, you can fix the GPT to use all of the space (an extra 367001600 blocks) or continue with the current
setting? 
Fix/Ignore? Fix                                                           
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 215GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name  Flags
 1      1049kB  2097kB  1049kB                        bios_grub
 2      2097kB  4096MB  4094MB  linux-swap(v1)
 3      4096MB  24.0GB  19.9GB  ext4

(parted) resizepart 3 -1                                                  
Warning: Partition /dev/vda3 is being used. Are you sure you want to continue?
parted: invalid token: -1
Yes/No? Yes
End?  [24.0GB]? -1
(parted) p
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 215GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system     Name  Flags
 1      1049kB  2097kB  1049kB                        bios_grub
 2      2097kB  4096MB  4094MB  linux-swap(v1)
 3      4096MB  215GB   211GB   ext4

(parted) q                                                                
Information: You may need to update /etc/fstab.

There is a warning the partition is in use but it is perfectly OK to continue.
*parted: reports “invalid token: -1”, but it is accepted for the “End” parameter.

STEP 6) Resize the ext4 file system online

Use the tool resize2fs to resize EXT4.

resize2fs /dev/vda3
resize2fs 1.42.13 (17-May-2015)
Filesystem at /dev/vda3 is mounted on /; on-line resizing required
old_desc_blocks = 2, new_desc_blocks = 13
The filesystem on /dev/vda3 is now 51428620 (4k) blocks long.

To check the resize operation:

srv1-vm ~ # dmesg|grep EXT4
[  449.330140] EXT4-fs (vda3): resizing filesystem from 4859392 to 51428620 blocks
[  449.936044] EXT4-fs (vda3): resized filesystem to 51428620
srv1-vm ~ # df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           798M  3.6M  795M   1% /run
/dev/vda3       193G  5.7G  180G   4% /
tmpfs           3.9G  196K  3.9G   1% /dev/shm

Output log of the whole resize operation

srv-vm1 ~ # parted /dev/vda
GNU Parted 3.2
Using /dev/vda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 26.8GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name  Flags
 1      1049kB  2097kB  1049kB                        bios_grub
 2      2097kB  4096MB  4094MB  linux-swap(v1)
 3      4096MB  24.0GB  19.9GB  ext4

(parted) q                                                                
srv-vm1 ~ # poweroff
Connection to srv-vm1 closed by remote host.
Connection to srv-vm1 closed.
myuser@gw1:~$ sshh srv1-host
srv1-host ~ # cd /mnt/vm/images
srv1-host images # qemu-img resize srv-vm1.img +174GB
Image resized.
srv1-host images # logout
Connection to srv1-host closed.
myuser@gw1:~$ sshh srv-vm1
srv-vm1 ~ # df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           798M  3.5M  795M   1% /run
/dev/vda3        19G  5.7G   12G  33% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           798M     0  798M   0% /run/user/0
srv-vm1 ~ # parted /dev/vda
GNU Parted 3.2
Using /dev/vda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Warning: Not all of the space available to /dev/vda appears to be used, you can fix the GPT to use all of the space (an extra 367001600 blocks) or continue with the current
setting? 
Fix/Ignore? Fix                                                           
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 215GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name  Flags
 1      1049kB  2097kB  1049kB                        bios_grub
 2      2097kB  4096MB  4094MB  linux-swap(v1)
 3      4096MB  24.0GB  19.9GB  ext4

(parted) q
Information: You may need to update /etc/fstab.

srv-vm1 ~ # parted /dev/vda                                           
GNU Parted 3.2
Using /dev/vda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) resizepart 3 -1                                                  
Warning: Partition /dev/vda3 is being used. Are you sure you want to continue?
parted: invalid token: -1                                                 
Yes/No? Yes                                                               
End?  [24.0GB]? -1                                                        
(parted) p                                                                
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 215GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name  Flags
 1      1049kB  2097kB  1049kB                        bios_grub
 2      2097kB  4096MB  4094MB  linux-swap(v1)
 3      4096MB  215GB   211GB   ext4

(parted) q                                                                
Information: You may need to update /etc/fstab.

srv-vm1 ~ # resize2fs /dev/vda3
resize2fs 1.42.13 (17-May-2015)
Filesystem at /dev/vda3 is mounted on /; on-line resizing required
old_desc_blocks = 2, new_desc_blocks = 13
The filesystem on /dev/vda3 is now 51428620 (4k) blocks long.

srv-vm1 ~ # dmesg|grep EXT4
[  449.330140] EXT4-fs (vda3): resizing filesystem from 4859392 to 51428620 blocks
[  449.936044] EXT4-fs (vda3): resized filesystem to 51428620

srv-vm1 ~ # touch /forcefsck
srv-vm1 ~ # reboot

We rebooted the virtual machine with force check for precaution, but it is not reqiured.

Bonus – physical disks setup

Probably you would need an additional first step of copying your old disk to the new disk – basically, there are two ways to do it:

  • blind copy everything with a hardware or the Linux “dd” command.
  • use gparted to copy the GPT table and the partitions to the new disk

ansible making a link: error – refusing to convert from file to symlink

A quick notice for your ansible scripts and as a reminder the right syntax for making a link with ansible is:

- name: change version
  file: src="path-to-existing-file-or-directory" dest="path-to-the-name-of-the-symlink" state=link
  • src must be existing file on the file system with the full path. The link will point to this file!
  • dest must be the name of the link with the full path. The setup will create or change the where this link points to.

Common error is to swap the src and dst

Here is an example of this error:

TASK [PHP-prepare : change version] ****************************************
fatal: [localhost]: FAILED! => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "msg": "refusing to convert from file to symlink for /usr/bin/php7.2", "owner": "root", "path": "/usr/bin/php7.2", "size": 4488224, "state": "file", "uid": 0}

The bad ansible code:

- name: change version
  file: src="/etc/alternatives/php" dest="/usr/bin/php7.2" state=link

The right ansible code:

- name: change version
  file: src="/usr/bin/php7.2" dest="/etc/alternatives/php" state=link

Online resize of a root ext4 file system – increase the space

Here you can see how to online resize your root ext4 file system. The free space of your partition will be increased after the operation. The size of the root file system will grow not shrink. Of course, this could have been any other partition, not exactly the root one, but in most cases, such operations on the root are the more complex and dangerous – SO ALWAYS do backups before such operations!

All services work properly and no shut down of services, no reboot, or umount is required during the resize operation.

Still, we rebooted the server once to force check the file system as a precaution, because it was possible and this server was not in production. The reboot of the server after this kind of resizing is not mandatory.
The following method is tested on CentOS 7, Ubuntu 16 LTS, and Gentoo with kernel 4.15 kernel. So we can assume you may have no problems if your system is newer than ours.

Summary

  1. Partition resize – Use resizepart in parted command. All Linux distributions have this package with the same name as the needed command “parted”
  2. File system resize – Use resize2fs from the E2fsprogs package. All Linux distributions include this package mostly with the same name of the package.

STEP 1) Expand the partition, which holds the root partition.

Let’s assume you have changed your disk and now there is more unallocated space to be used or somehow the space of the disk is increased. Look below for a real-world example with one of our virtual servers.

root@srv1 ~ # parted /dev/sda
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Model: Model: ATA Samsung SSD 850 (scsi)
Disk /dev/sda: 215GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name  Flags
 1      1049kB  2097kB  1049kB                        bios_grub
 2      2097kB  4096MB  4094MB  linux-swap(v1)
 3      4096MB  24.0GB  19.9GB  ext4
(parted) resizepart 3 -1                                                  
Warning: Partition /dev/sda3 is being used. Are you sure you want to continue?
parted: invalid token: -1                                                 
Yes/No? Yes                                                               
End?  [24.0GB]? -1                                                        
(parted) p                                                                
Model: Model: ATA Samsung SSD 850 (scsi)
Disk /dev/sda: 215GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name  Flags
 1      1049kB  2097kB  1049kB                        bios_grub
 2      2097kB  4096MB  4094MB  linux-swap(v1)
 3      4096MB  215GB   211GB   ext4

(parted) q                                                                
Information: You may need to update /etc/fstab.

As you can see from the first print command the partition number 3 is 19.9GB and after the resize command with “-1” is 211GB. There is a warning about the partition is used, but it is normal and not critical.

STEP 2) Resize the file system, on which we expanded the partition.

You need to install E2fsprogs. All Linux distributions have this package, here are some of them:

  • CentOS 7 – e2fsprogs
  • Ubuntu – e2fsprogs
  • Gentoo – sys-fs/e2fsprogs

After installing the e2fsprogs package you will have the online ext4 resizing tool – resize2fs.

root@srv ~ # resize2fs /dev/sda3
resize2fs 1.42.13 (17-May-2015)
Filesystem at /dev/sda3 is mounted on /; on-line resizing required
old_desc_blocks = 2, new_desc_blocks = 13
The filesystem on /dev/sda3 is now 51428620 (4k) blocks long.

Check if everything is OK with

root@srv ~ # dmesg|grep EXT4
[  449.330140] EXT4-fs (vda3): resizing filesystem from 4859392 to 51428620 blocks
[  449.936044] EXT4-fs (vda3): resized filesystem to 51428620
root@srv ~ # df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           798M  3.5M  795M   1% /run
/dev/sda3       193G  3.4G  182G   2% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           798M     0  798M   0% /run/user/0

Bonus – you can force check the file system on the next reboot

Probably it is a good idea to force check the file system integrity on the next boot. This step is not mandatory and you may skip it.
For Ubuntu you can do:

root@srv ~ # touch /forcefsck
root@srv ~ # reboot

Bonus 2

Fixing the GPT. Newer versions may display warning the GPT table is not using the whole disk space and to fix it. Just type fix to add the new unallocated disk space:

root@srv ~ # parted /dev/sda
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Warning: Not all of the space available to /dev/sda appears to be used, you can fix the GPT to use all of the space (an extra 188743680 blocks) or continue with the current setting? 
Fix/Ignore? Fix                                                           
Model: Virtio Block Device (virtblk)
Disk /dev/sda: 118GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name  Flags
 1      1049kB  2097kB  1049kB                        bios_grub
 2      2097kB  17.2GB  17.2GB  ext4
 3      17.2GB  21.5GB  4293MB  linux-swap(v1)

(parted)

zswap – enable – the write back compressed RAM of your swap

Zswap is an interesting way to extend your swap space with a memory write-back cache. Here is our simple explanation:
When this feature is enabled an amount of machine’s RAM is put aside and when the system needs to use swap space it will first write to this area and when it is full then it will use the disk. The data is also compressed on-the-fly by the kernel when saved in the memory allocated for the zswap device. The data is not compressed when saved on the disk. So it may happen your disk won’t be touch at all if the data could fit in the compressed memory pool. In addition, if the memory pool is full or it is at the maximum allowed space and no further extension is possible it occurs data evictions to the disk swap space using least recently used (LRU) algorithm.
Of course, it is a little bit more complex like it compresses only pages and there two handlers, which stores up to 2 compress pages in 1 and another one stores up to 3 pages in 1 (as to understand it even if you sometimes could store more compressed pages in let’s say 5 in 1 page it would not happen, the current memory allocator will compress pages up to what it is configured).
The most important piece of information is:

zswap uses RAM to make a compressed pool, which is first used when a swap out request is made. No writing to the disk is made.

You can effectively increase the amount of RAM using this feature because it’s like you have the ability to compress part of your RAM and the current algorithms show 2x to 3x times compression ratio. So separating 20% of 2G RAM of your virtual server for the zswap device you end up with 1.6G RAM + 400M zswap with the average compression ratio of 2x you may have 2.4G before the swap process touches your disks.

There are multiple cases where this feature is very handful such as:

  • virtualization – virtual servers – increase your RAM
  • reduce IO to the slow disks such as hard drives
  • reduce IO to the flash-based storage, which may increase their life
  • database or DNS servers could have great benefits because the compression ratio could be around 3x (i.e. 3 compressed pages stored in 1 real page)

If you do not know what is 1 page in computer terminology – it is the smallest unit of data for memory management and in most cases it is 4K, of course there are additional sizes 8K, 16K and more. You can see more here – https://en.wikipedia.org/wiki/Page_(computer_memory)

Enable zswap

To enable zswap device you must do the following:
Boot your kernel with the kernel parameter (reset is required, on some old kernels of 3.x this is the only option):

zswap.enabled=1

Or just enable it from /proc filesystem (runtime enable, not possible in old kernels):

echo 1 > /sys/module/zswap/parameters/enabled

When you disable it by setting to 0 it will not immediately decompress all pages and remove the pool. The pages in the pool must be invalidated or fault back to the memory. You may force the removal of all the compressed pages and the pool by deactivating the swap device by

swapoff -a

To turn off all swap devices and it will return all swap out pages into memory including the ones in the zswap compressed memory pool. The pool will be removed.
Keep on reading!

Ubuntu with PHP 7.2 and mcrypt module

As mentioned in our previous article PHP 7.2 or PHP 7.3 with mcrypt – manual build the PHP versions 7.2 and 7.3 do not include PHP mcrypt module. The mcrypt module was part of PHP 5 till 7.1, in which it was deprecated and removed in 7.2.
In this article we show how to build mcrypt module for Ubuntu based on our previous article showed above. Because of the great popularity of Ubuntu and it has no PHP mcrypt package in Ubuntu package system unlike other Linux distributions (like Gentoo, which created a package) we decided to make this article.

For our purpose we use Ubuntu 18.04.2 LTS and here is what the steps to have the mcrypt PHP module:

STEP 1) Update and install mcrypt library and header development packet

sudo apt update -y
sudo apt install -y libmcrypt-dev

STEP 2) Install the GNU GCC build utility and the PHP dev packet

This is the compiler to build the module.

sudo apt install -y build-essential
sudo apt install -y php7.2-dev

STEP 3) Download the PHP mcrypt module and build it.

cd
mkdir mcrypt-php-module-manual
cd mcrypt-php-module-manual
wget https://pecl.php.net/get/mcrypt-1.0.2.tgz
tar xzf mcrypt-1.0.2.tgz
cd mcrypt-1.0.2
phpize
aclocal
libtoolize --force
autoheader
autoconf
./configure
make
sudo make install

STEP 4) Load the module in the PHP configuration (we use PHP-FPM and PHP-CLI) and block future PHP versions to be installed when apt update is used.

Because we compile the PHP mcrypt module for the specific currently installed PHP we do not want to upgrade our PHP when there is an update and the mcrypt module to fail to load. Each change of the PHP version (upgrade) would require a recompile against the current PHP version. To see more for holding and unholding Ubuntu packages – apt-mark – upgrade with the exception of certain packages Of course, if there is an update for PHP you must install it just recompile the mcrypt package, too!

echo "extension=mcrypt.so" > 20-mcrypt.ini
sudo cp 20-mcrypt.ini /etc/php/7.2/cli/conf.d/20-mcrypt.ini
sudo cp 20-mcrypt.ini /etc/php/7.2/fpm/conf.d/20-mcrypt.ini
sudo apt-mark hold php-cli php7.2-cli php-fpm php7.2-fpm

Keep on reading!

PHP 7.2 or PHP 7.3 with mcrypt – manual build

Newer PHP versions do not include PHP mcrypt library. The mcrypt module was part of PHP 5 till 7.1, in which it was deprecated and removed in 7.2. If you open the php.net documentation for mcrypt PHP functions you will see:

This function has been DEPRECATED as of PHP 7.1.0 and REMOVED as of PHP 7.2.0. Relying on this function is highly discouraged.

The mcrypt module is now in PHP PECL (repository for PHP Extensions) in https://pecl.php.net/package/mcrypt. As you can see in the description, this is a legacy module, which

Provides bindings for the unmaintained libmcrypt.

, so it is strongly recommended to replace it with OpenSSL (for example).
Still, if you need this legacy module – mcrypt and :

You may want to manually build the mcrypt module for your current installed PHP. Of course, the generic dependencies are:

  1. libmcrypt and its headers (if the Linux distribution) splits the binary and the headers
  2. GNU GCC
  3. PHP 7.2+
  4. download the latest mcrypt module source from https://pecl.php.net/package/mcrypt. For example, now it is https://pecl.php.net/get/mcrypt-1.0.2.tgz
mkdir /root/mcrypt-php-module-manual
cd /root/mcrypt-php-module-manual
wget https://pecl.php.net/get/mcrypt-1.0.2.tgz
tar xzf mcrypt-1.0.2.tgz
cd mcrypt-1.0.2
phpize
aclocal
libtoolize --force
autoheader
autoconf
./configure
make
make install

Do not use “make -j N” (“make -j 8”, for example), because it may fail to compile.
Keep on reading!