Transfer only a list of files with rsync

Transferring a list of files from one server to another maybe not so easy as it looks like, if the list consists of files with symlinks in the paths. Consider the following example:

/mnt/storage1/dir1/subdir1/file1
/mnt/storage1/dir1/subdir2/file2
/mnt/storage1/dir1/subdir3/file3

But what if the subdir3 is a symlink to a sub-directory:

/mnt/storage1/dir1/subdir3 -> /mnt/storage2/dir1/subdir3

STEP 1) Generate a list of only files following the symlinks if they exist.

The best way is to use Linux command find:

find -L /mnt/storage1/ -type f &> find.files.log

The option “-L” instructs the find to follow symbolic links and the test for the file (-type f) will always match against the type of the file that the symbolic link points to and not the symlink itself.

STEP 2) rsync the list of the files

There are several options (–copy-links –copy-dirlinks), which must be used with rsync command to be able to transfer the list of files, which may include symlink directories in the files’ paths:

rsync --copy-links --copy-dirlinks --partial --files-from=/root/find.files.log --verbose --progress --stats --times --perms --owner --group 10.10.10.10::storage /

The command above uses the rsync daemon started on the source server with shared root under the name “root” in the rsync configuration. Here is a sample rsync daemon configuration (/etc/rsyncd.conf):

pid file = /run/rsyncd.pid
use chroot = yes
read only = yes
hosts allow = 10.10.10.10/24
hosts deny = *

[root]
        uid=0
        gid=0
        path = /
        comment = root partition
        exclude = /proc /sys

Of course, a relative path could be used in the rsync daemon configuration if the file generated is also a relative path. For simplicity, here the whole real path is used.

In addition, the rsync could be used without a rsync daemon, but in conjunction with ssh daemon:

rsync --rsh='ssh -p 22 -l root' --copy-links --copy-dirlinks --partial --files-from=/root/find.files.log --verbose --progress --stats --times --perms --owner --group 10.10.10.10:/ /

Again, because the files in the find.files.log are with absolute paths the root / must be used in the rsync command. Relative paths may be used, too.

ansible making a link: error – refusing to convert from file to symlink

A quick notice for your ansible scripts and as a reminder the right syntax for making a link with ansible is:

- name: change version
  file: src="path-to-existing-file-or-directory" dest="path-to-the-name-of-the-symlink" state=link
  • src must be existing file on the file system with the full path. The link will point to this file!
  • dest must be the name of the link with the full path. The setup will create or change the where this link points to.

Common error is to swap the src and dst

Here is an example of this error:

TASK [PHP-prepare : change version] ****************************************
fatal: [localhost]: FAILED! => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "msg": "refusing to convert from file to symlink for /usr/bin/php7.2", "owner": "root", "path": "/usr/bin/php7.2", "size": 4488224, "state": "file", "uid": 0}

The bad ansible code:

- name: change version
  file: src="/etc/alternatives/php" dest="/usr/bin/php7.2" state=link

The right ansible code:

- name: change version
  file: src="/usr/bin/php7.2" dest="/etc/alternatives/php" state=link