There two main options to change the location of all the containers’ storages:
- “mount bind” the new location to the default storage directory (look Note 1)
- Change the path of the location in the configuration file /etc/containers/storage.conf
You should stop all your containers though it is not mandatory.
You should stop the containers (if any) and copy the directory, because when reconfigured the storage path podman won’t access the ones in the old path – containers and images!
STEP 1) Change the storage path in the podman configuration file.
If the SELinux has been disabled, which should not be done, it is just a matter of changing a path option in the configuration file /etc/containers/storage.conf
# Primary Read/Write location of container storage graphroot = "/var/lib/containers/storage"
Change it to whatever path you like. Mostly, it should point to the big storage device. In our case, the big storage is mounted under “/mnt/mystorage/virtual/storage”. Change the options to:
# Primary Read/Write location of container storage graphroot = "/mnt/mystorage/virtual/storage"
Check the running configuration with:
[root@lsrv1 mystorage]# podman info host: BuildahVersion: 1.12.0-dev CgroupVersion: v1 Conmon: package: conmon-2.0.8-1.el7.x86_64 path: /usr/bin/conmon version: 'conmon version 2.0.8, commit: f85c8b1ce77b73bcd48b2d802396321217008762' Distribution: distribution: '"centos"' version: "7" MemFree: 191963136 MemTotal: 16563531776 OCIRuntime: name: runc package: runc-1.0.0-67.rc10.el7_8.x86_64 path: /usr/bin/runc version: 'runc version spec: 1.0.1-dev' SwapFree: 7857680384 SwapTotal: 8581541888 arch: amd64 cpus: 8 eventlogger: journald hostname: lsrv1 kernel: 3.10.0-1062.9.1.el7.x86_64 os: linux rootless: false uptime: 607h 10m 53.36s (Approximately 25.29 days) registries: blocked: null insecure: null search: - registry.access.redhat.com - registry.fedoraproject.org - registry.centos.org - docker.io store: ConfigFile: /etc/containers/storage.conf ContainerStore: number: 0 GraphDriverName: overlay GraphOptions: {} GraphRoot: /mnt/mystorage/virtual/storage GraphStatus: Backing Filesystem: extfs Native Overlay Diff: "true" Supports d_type: "true" Using metacopy: "false" ImageStore: number: 0 RunRoot: /var/run/containers/storage VolumePath: /mnt/mystorage/virtual/storage/volumes
STEP 2*) Add SELinux rules if it is in enforcing mode.
*And only if you are using SELinux in enforcing mode you must transfer the rules for the new location. Just list the SELinux rules, pick only those with the old path, and add the same rules with the new path.
First, check the status of the SELinux with:
[root@srv ~]# getenforce Enforcing
Add the rules for the new path:
[root@lsrv1 ~]# semanage fcontext -l|grep "/var/lib/containers" /var/lib/containers(/.*)? all files system_u:object_r:container_var_lib_t:s0 /var/lib/containers/home(/.*)? all files system_u:object_r:openshift_var_lib_t:s0 /var/lib/containers/atomic(/.*)? all files <<None>> /var/lib/containers/overlay(/.*)? all files system_u:object_r:container_share_t:s0 /var/lib/containers/overlay2(/.*)? all files system_u:object_r:container_share_t:s0 /var/lib/containers/overlay-layers(/.*)? all files system_u:object_r:container_share_t:s0 /var/lib/containers/overlay-images(/.*)? all files system_u:object_r:container_share_t:s0 /var/lib/containers/overlay2-layers(/.*)? all files system_u:object_r:container_share_t:s0 /var/lib/containers/overlay2-images(/.*)? all files system_u:object_r:container_share_t:s0 /var/lib/containers/storage/overlay(/.*)? all files system_u:object_r:container_share_t:s0 /var/lib/containers/storage/overlay2(/.*)? all files system_u:object_r:container_share_t:s0 /var/lib/containers/storage/volumes/[^/]*/.* all files system_u:object_r:container_file_t:s0 /var/lib/containers/storage/overlay-layers(/.*)? all files system_u:object_r:container_share_t:s0 /var/lib/containers/storage/overlay-images(/.*)? all files system_u:object_r:container_share_t:s0 /var/lib/containers/storage/overlay2-layers(/.*)? all files system_u:object_r:container_share_t:s0 /var/lib/containers/storage/overlay2-images(/.*)? all files system_u:object_r:container_share_t:s0
It’s simple engough – add an equivalent directory path with semanage and then relabel the path.
[root@lsrv1 mystorage]# semanage fcontext -a -e /var/lib/containers /mnt/mystorage/virtual/storage [root@lsrv1 mystorage]# restorecon -Rv /mnt/mystorage/virtual/storage restorecon reset /mnt/mystorage/virtual/storage context unconfined_u:object_r:mnt_t:s0->unconfined_u:object_r:container_var_lib_t:s0
Note, the target is not allowed to end with ‘/’.
Note 1 – mount bind
Use mount bind to change the directory to new location.
mount -o bind /mnt/mystorage/virtual/storage /var/lib/containers
And add it to the /etc/fstab to mount it always on boot:
/mnt/mystorage/virtual/storage /var/lib/containers bind bind 0 0
First, rename (or copy/rsync the content of it to a temporary location) /var/lib/containers and then “mount bind” it and then copy back the original content from temporary location to preseve all the podman data before the change.
Just to note when using SELinux you should check STEP 2) above and add equivalent direcotry, becuase if relabeling occurs on the storage device could overwrite the SELinux context labels.
*Note 2 – podman temporary data
There is one more storage options in the file, which is used for temporary data and it is located in /var/run (links to /run – a tmpfs directory).
# Temporary storage location runroot = "/var/run/containers/storage"
Apparently, it stores temporary files like container’s specific confguration file and data for running containers. The data does not occupy much space so the temporary storage directory may not be moved to the big storage device. Still, if the temporary storage directory should be moved to another location and SELinux is enabled the STEP 2) must be taken into consideration.
Here is what kind of data to expect:
[root@lsrv1 storage]# ls -altr /var/run/containers/storage/overlay-containers/65deb46b790795505e94f618f2f9b4a381d56a241a5d976dda9ccb1ba5e6021f/userdata/ total 20 drwx------. 3 root root 60 28 May 9,42 .. -rw-r--r--. 1 root root 179 28 May 9,42 resolv.conf -rw-r--r--. 1 root root 361 28 May 9,42 hosts -rw-r--r--. 1 root root 12 28 May 9,42 hostname -rw-r--r--. 1 root root 0 28 May 9,42 .containerenv drwxr-xr-x. 3 root root 60 28 May 9,42 run -rw-r--r--. 1 root root 5 28 May 9,42 conmon.pid -rw-r--r--. 1 root root 0 28 May 9,42 oci-log -rw-r--r--. 1 root root 5 28 May 9,42 pidfile
Hey,
I think in “Note 1 – mount bind” you should have:
mount -o bind /mnt/mystorage/virtual/storage /var/lib/containers
and
/mnt/mystorage/virtual/storage /var/lib/containers bind bind 0 0
Take a look on my test on RHEL8:
[root@vmhive-docker containers]# df -lh /var/lib/containers/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel_vmhive--docker-root 50G 19G 32G 37% /
[root@vmhive-docker containers]# df -lh /home/containers/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel_vmhive--docker-home 142G 1.1G 141G 1% /home
[root@vmhive-docker containers]# mount -o bind /home/containers /var/lib/containers
[root@vmhive-docker containers]# df -lh /var/lib/containers/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel_vmhive--docker-home 142G 1.1G 141G 1% /var/lib/containers
[root@vmhive-docker containers]# df -lh /home/containers/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel_vmhive--docker-home 142G 1.1G 141G 1% /home
Regards,
Łukasz
Yes, you are absolutely right and I updated it above! I am not sure why I swap the two directories (first source and second is the [target] directory). I even use the bind option in one of my servers…the right way, but here it got wrong…Thank you.
Thank you for this blog post. It’s really helpful for someone who starts his journey with podman!
Hi,
I am not sure about if Note1 and Note2 are really needed… If I understand it correctly, they are mentionated only as an alternative to do Step1, since your are doing a link of the original directory…
Is that correct ?
Thks
Hi.
Tried this method (2024) on Fedora Server 40 and found the following. Changing graphroot is the way to go. Creating a bind mount caused havoc on the system (SELinux likely cause). Maybe there is another way of doing this but this is the best tutorial I’ve found.