Transferring a list of files from one server to another maybe not so easy as it looks like, if the list consists of files with symlinks in the paths. Consider the following example:
/mnt/storage1/dir1/subdir1/file1 /mnt/storage1/dir1/subdir2/file2 /mnt/storage1/dir1/subdir3/file3
But what if the subdir3 is a symlink to a sub-directory:
/mnt/storage1/dir1/subdir3 -> /mnt/storage2/dir1/subdir3
STEP 1) Generate a list of only files following the symlinks if they exist.
The best way is to use Linux command find:
find -L /mnt/storage1/ -type f &> find.files.log
The option “-L” instructs the find to follow symbolic links and the test for the file (-type f) will always match against the type of the file that the symbolic link points to and not the symlink itself.
STEP 2) rsync the list of the files
There are several options (–copy-links –copy-dirlinks), which must be used with rsync command to be able to transfer the list of files, which may include symlink directories in the files’ paths:
rsync --copy-links --copy-dirlinks --partial --files-from=/root/find.files.log --verbose --progress --stats --times --perms --owner --group 10.10.10.10::storage /
The command above uses the rsync daemon started on the source server with shared root under the name “root” in the rsync configuration. Here is a sample rsync daemon configuration (/etc/rsyncd.conf):
pid file = /run/rsyncd.pid
use chroot = yes
read only = yes
hosts allow = 10.10.10.10/24
hosts deny = *
[root]
uid=0
gid=0
path = /
comment = root partition
exclude = /proc /sys
Of course, a relative path could be used in the rsync daemon configuration if the file generated is also a relative path. For simplicity, here the whole real path is used.
In addition, the rsync could be used without a rsync daemon, but in conjunction with ssh daemon:
rsync --rsh='ssh -p 22 -l root' --copy-links --copy-dirlinks --partial --files-from=/root/find.files.log --verbose --progress --stats --times --perms --owner --group 10.10.10.10:/ /
Again, because the files in the find.files.log are with absolute paths the root / must be used in the rsync command. Relative paths may be used, too.
