A great piece of software is
lftp – sophisticated file transfer program
This little console tool could ease your life significantly with many enhancements to the simple FTP protocol. This tip is for those how what to list all their files in a directory or the entire FTP account, but do not have ls command with recursive abilities. So the only option is to manually go through all the directories to fetch the listing information of the directories, but this could be automatically done by
lftp using the custom command “find” and if you add “-l” argument the output is like “ls -al” – file or directory, file permissions, user and group, file size, date and file name are shown on single line for each file.
Just execute the command with proper credentials and the starting directory of your choice. The command output could even be piped to another command.
Using lftp with interactive shell:
myuser@local ~ # lftp -u "myuser,myuserpassword" 10.10.10.10 lftp myuser.myuser@10.10.10.10:/> ls drwxr-xr-x 1 myuser myuser 0 Apr 13 14:10 web drwxr-xr-x 1 myuser myuser 0 Apr 13 14:10 backups lftp myuser.myuser@10.10.10.10:/web> cd web lftp myuser.myuser@10.10.10.10:/web> ls drwxr-xr-x 1 myuser myuser 0 Apr 13 14:10 public lftp myuser.myuser@10.10.10.10:/web> cd public lftp myuser.myuser@10.10.10.10:/web/public> ls drwxr-xr-x 1 myuser myuser 0 Apr 13 14:11 0 drwxr-xr-x 1 myuser myuser 0 Apr 13 14:11 1000 drwxr-xr-x 1 myuser myuser 0 Apr 13 14:11 10000 drwxr-xr-x 1 myuser myuser 0 Apr 13 14:11 11000 .... .... drwxr-xr-x 1 myuser myuser 0 Apr 13 14:17 8000 drwxr-xr-x 1 myuser myuser 0 Apr 13 14:17 9000 lftp myuser.myuser@10.10.10.10:/web/public> find ./ ./0/ ./0/1/ ./0/1/1_small.txt ./0/1/1_medium.txt ./0/1/1_big.txt ./0/11/ ./0/11/11_small.txt ./0/11/11_medium.txt ./0/11/11_big.txt ./0/118/ ./0/118/118_small.txt ./0/118/118_medium.txt ./0/118/118_big.txt ./0/120/ ./0/120/120_small.txt ./0/120/120_medium.txt ./0/120/120_big.txt ^CInterrupt lftp myuser.myuser@10.10.10.10:/web/public> find -l drwxr-xr-x - - ./ drwxr-xr-x myuser/myuser 0 2019-04-13 14:11:30 ./0/ drwxr-xr-x myuser/myuser 0 2019-04-13 14:18:30 ./0/1/ -rw-r--r-- myuser/myuser 24116232 2018-10-22 15:08:30 ./0/1/1_small.txt -rw-r--r-- myuser/myuser 42922383 2018-10-22 15:13:30 ./0/1/1_medium.txt -rw-r--r-- myuser/myuser 103514401 2018-10-22 15:13:30 ./0/1/1_big.txt drwxr-xr-x myuser/myuser 0 2019-04-13 14:18:30 ./0/11/ -rw-r--r-- myuser/myuser 28493047 2018-10-22 16:08:30 ./0/11/11_small.txt -rw-r--r-- myuser/myuser 50273126 2018-10-22 16:08:30 ./0/11/11_medium.txt -rw-r--r-- myuser/myuser 106139598 2018-10-22 16:09:30 ./0/11/11_big.txt drwxr-xr-x myuser/myuser 0 2019-04-13 14:18:30 ./0/118/ -rw-r--r-- myuser/myuser 28422439 2018-10-22 18:04:30 ./0/118/118_small.txt -rw-r--r-- myuser/myuser 50954506 2018-10-22 18:09:30 ./0/118/118_medium.txt -rw-r--r-- myuser/myuser 105554784 2018-10-22 18:11:30 ./0/118/118_big.txt drwxr-xr-x myuser/myuser 0 2019-04-13 14:18:30 ./0/120/ -rw-r--r-- myuser/myuser 26327340 2018-10-23 01:18:30 ./0/120/120_small.txt -rw-r--r-- myuser/myuser 46865921 2018-10-23 01:20:30 ./0/120/120_medium.txt -rw-r--r-- myuser/myuser 102516584 2018-10-23 01:20:30 ./0/120/120_big.txt ^CInterrupt lftp myuser.myuser@10.10.10.10:/web/public> exit
As you can see “find -l” has the same output as “ls -al” but the command output recursively all sub-directories (in fact, invisible for the user the command just makes the ftp client to change the directory and list it for every sub-directory recursively!).
The output could be interrupted with Ctrl+C, you’ll be still in the lftp shell and connected to the FTP server if you were before.
Scripts
You can execute this command as a script and redirect the output to file or another program like awk, grep an so on. Just pass the commands in a string to “-e” lftp command argument like:
lftp -u "myuser.myuser,myuserpass" -e "find -l /;exit" 10.10.10.10 &> all_files.log lftp -u "myuser.myuser,myuserpass" -e "find -l web/public/;exit" 10.10.10.10 &> all_files_web.log
or redirect it to multiple pipes like awk (to have only the data you need – file size), egrep (to remove the directories) and a cycle to sum the total file size of selected files like. Note the while is in a group, because each pipe starts a shell, so won’t have $total after the while (and it will output only 0).
myuser@srv ~ # total=0;lftp -u "myuser.myuser,myuserpass" -e "find -l web/public/;exit" 10.10.10.10|egrep -v "^d" |grep "_big"|awk '{print $3}'|(while read one; do total=$((total+one));done;echo $total)
6452345191
myuser@srv ~ #
Missing “ls -R”
So many FTP implementations lack the list directories recursively:
“-R” is interpreted as a file name, not an argument.
lftp -u "myuser.myuser,myuserpass" 10.10.10.10 lftp myuser.myuser@10.10.10.10:~> ls -R ls: ls -R: Access failed: 550 No such file or directory. (-R) lftp myuser.myuser@10.10.10.10:/>
lftp with find command completely eliminates the lack of “ls -R”!