Transfer only a list of files with rsync

Transferring a list of files from one server to another maybe not so easy as it looks like, if the list consists of files with symlinks in the paths. Consider the following example:

/mnt/storage1/dir1/subdir1/file1
/mnt/storage1/dir1/subdir2/file2
/mnt/storage1/dir1/subdir3/file3

But what if the subdir3 is a symlink to a sub-directory:

/mnt/storage1/dir1/subdir3 -> /mnt/storage2/dir1/subdir3

STEP 1) Generate a list of only files following the symlinks if they exist.

The best way is to use Linux command find:

find -L /mnt/storage1/ -type f &> find.files.log

The option “-L” instructs the find to follow symbolic links and the test for the file (-type f) will always match against the type of the file that the symbolic link points to and not the symlink itself.

STEP 2) rsync the list of the files

There are several options (–copy-links –copy-dirlinks), which must be used with rsync command to be able to transfer the list of files, which may include symlink directories in the files’ paths:

rsync --copy-links --copy-dirlinks --partial --files-from=/root/find.files.log --verbose --progress --stats --times --perms --owner --group 10.10.10.10::storage /

The command above uses the rsync daemon started on the source server with shared root under the name “root” in the rsync configuration. Here is a sample rsync daemon configuration (/etc/rsyncd.conf):

pid file = /run/rsyncd.pid
use chroot = yes
read only = yes
hosts allow = 10.10.10.10/24
hosts deny = *

[root]
        uid=0
        gid=0
        path = /
        comment = root partition
        exclude = /proc /sys

Of course, a relative path could be used in the rsync daemon configuration if the file generated is also a relative path. For simplicity, here the whole real path is used.

In addition, the rsync could be used without a rsync daemon, but in conjunction with ssh daemon:

rsync --rsh='ssh -p 22 -l root' --copy-links --copy-dirlinks --partial --files-from=/root/find.files.log --verbose --progress --stats --times --perms --owner --group 10.10.10.10:/ /

Again, because the files in the find.files.log are with absolute paths the root / must be used in the rsync command. Relative paths may be used, too.

rsync daemon in CentOS 8

Apparently, the team behind the CentOS 8 decided to split the rsync functionality to two packages – one for the client-side and for the server-side, despite the binary rsync is only one and offers the client-size and server-side.
So there two packages in CentOS 8:

  1. rsync – provides the client-side and server-side as usual
  2. rsync-daemon – provides configuration example file and the systemd to start it as a service.

So if you wonder where is your rsync service after installing the rsync package under CentOS 8 you must install additional package “rsync-daemon”.

Of course, you may just create anywhere “rsyncd.conf” (the best place for the configuration file is in /etc, but could be placed anywhere with “–config=PATH/FILE” option) file and start the daemon as usual with “–daemon” option included to have the rsync server-side up and running.

rsync --daemon --config=/etc/rsyncd.conf

Just create yourself the configuration “/etc/rsyncd.conf” file.

Install the rsync program – the client and the daemon

dnf install rsync

Install the configuration and systemd files

dnf install rsync-daemon

rsync and rsync-daemon files

The files included in the two packages:

[root@srv ~]# dnf repoquery -l rsync-daemon
Last metadata expiration check: 0:33:02 ago on Wed 22 Jan 2020 02:57:06 PM UTC.
/etc/rsyncd.conf
/etc/sysconfig/rsyncd
/usr/lib/systemd/system/rsyncd.service
/usr/lib/systemd/system/rsyncd.socket
/usr/lib/systemd/system/rsyncd@.service
/usr/share/man/man5/rsyncd.conf.5.gz
[root@srv ~]# dnf repoquery -l rsync
Last metadata expiration check: 0:33:06 ago on Wed 22 Jan 2020 02:57:06 PM UTC.
/usr/bin/rsync
/usr/lib/.build-id
/usr/lib/.build-id/6e
/usr/lib/.build-id/6e/bdbf5032a966a6293cea003621ba766ab36ff1
/usr/share/doc/rsync
/usr/share/doc/rsync/NEWS
/usr/share/doc/rsync/OLDNEWS
/usr/share/doc/rsync/README
/usr/share/doc/rsync/support
/usr/share/doc/rsync/support/Makefile
/usr/share/doc/rsync/support/atomic-rsync
/usr/share/doc/rsync/support/cvs2includes
/usr/share/doc/rsync/support/deny-rsync
/usr/share/doc/rsync/support/file-attr-restore
/usr/share/doc/rsync/support/files-to-excludes
/usr/share/doc/rsync/support/git-set-file-times
/usr/share/doc/rsync/support/instant-rsyncd
/usr/share/doc/rsync/support/logfilter
/usr/share/doc/rsync/support/lsh
/usr/share/doc/rsync/support/lsh.sh
/usr/share/doc/rsync/support/mapfrom
/usr/share/doc/rsync/support/mapto
/usr/share/doc/rsync/support/mnt-excl
/usr/share/doc/rsync/support/munge-symlinks
/usr/share/doc/rsync/support/rrsync
/usr/share/doc/rsync/support/rsync-no-vanished
/usr/share/doc/rsync/support/rsync-slash-strip
/usr/share/doc/rsync/support/rsyncstats
/usr/share/doc/rsync/support/savetransfer.c
/usr/share/doc/rsync/tech_report.tex
/usr/share/licenses/rsync
/usr/share/licenses/rsync/COPYING
/usr/share/man/man1/rsync.1.gz