List all your files (and directories) with file size over FTP without ls -R (recursive)

A great piece of software is

lftp – sophisticated file transfer program

This little console tool could ease your life significantly with many enhancements to the simple FTP protocol. This tip is for those how what to list all their files in a directory or the entire FTP account, but do not have ls command with recursive abilities. So the only option is to manually go through all the directories to fetch the listing information of the directories, but this could be automatically done by

lftp using the custom command “find” and if you add “-l” argument the output is like “ls -al” – file or directory, file permissions, user and group, file size, date and file name are shown on single line for each file.

Just execute the command with proper credentials and the starting directory of your choice. The command output could even be piped to another command.

Using lftp with interactive shell:

myuser@local ~ # lftp -u "myuser,myuserpassword" 10.10.10.10
lftp myuser.myuser@10.10.10.10:/> ls
drwxr-xr-x   1 myuser  myuser         0 Apr 13 14:10 web
drwxr-xr-x   1 myuser  myuser         0 Apr 13 14:10 backups
lftp myuser.myuser@10.10.10.10:/web> cd web
lftp myuser.myuser@10.10.10.10:/web> ls
drwxr-xr-x   1 myuser  myuser         0 Apr 13 14:10 public
lftp myuser.myuser@10.10.10.10:/web> cd public
lftp myuser.myuser@10.10.10.10:/web/public> ls
drwxr-xr-x   1 myuser  myuser         0 Apr 13 14:11 0
drwxr-xr-x   1 myuser  myuser         0 Apr 13 14:11 1000
drwxr-xr-x   1 myuser  myuser         0 Apr 13 14:11 10000
drwxr-xr-x   1 myuser  myuser         0 Apr 13 14:11 11000
....
....
drwxr-xr-x   1 myuser  myuser         0 Apr 13 14:17 8000
drwxr-xr-x   1 myuser  myuser         0 Apr 13 14:17 9000
lftp myuser.myuser@10.10.10.10:/web/public> find
./
./0/
./0/1/                                                              
./0/1/1_small.txt                                                 
./0/1/1_medium.txt
./0/1/1_big.txt
./0/11/
./0/11/11_small.txt                                                
./0/11/11_medium.txt
./0/11/11_big.txt
./0/118/
./0/118/118_small.txt                                                 
./0/118/118_medium.txt
./0/118/118_big.txt
./0/120/
./0/120/120_small.txt                                               
./0/120/120_medium.txt
./0/120/120_big.txt
^CInterrupt
lftp myuser.myuser@10.10.10.10:/web/public> find -l
drwxr-xr-x                      - - ./
drwxr-xr-x  myuser/myuser     0 2019-04-13 14:11:30 ./0/        
drwxr-xr-x  myuser/myuser     0 2019-04-13 14:18:30 ./0/1/        
-rw-r--r--  myuser/myuser 24116232 2018-10-22 15:08:30 ./0/1/1_small.txt
-rw-r--r--  myuser/myuser 42922383 2018-10-22 15:13:30 ./0/1/1_medium.txt
-rw-r--r--  myuser/myuser 103514401 2018-10-22 15:13:30 ./0/1/1_big.txt
drwxr-xr-x  myuser/myuser     0 2019-04-13 14:18:30 ./0/11/
-rw-r--r--  myuser/myuser 28493047 2018-10-22 16:08:30 ./0/11/11_small.txt
-rw-r--r--  myuser/myuser 50273126 2018-10-22 16:08:30 ./0/11/11_medium.txt
-rw-r--r--  myuser/myuser 106139598 2018-10-22 16:09:30 ./0/11/11_big.txt
drwxr-xr-x  myuser/myuser     0 2019-04-13 14:18:30 ./0/118/
-rw-r--r--  myuser/myuser 28422439 2018-10-22 18:04:30 ./0/118/118_small.txt
-rw-r--r--  myuser/myuser 50954506 2018-10-22 18:09:30 ./0/118/118_medium.txt
-rw-r--r--  myuser/myuser 105554784 2018-10-22 18:11:30 ./0/118/118_big.txt
drwxr-xr-x  myuser/myuser     0 2019-04-13 14:18:30 ./0/120/
-rw-r--r--  myuser/myuser 26327340 2018-10-23 01:18:30 ./0/120/120_small.txt
-rw-r--r--  myuser/myuser 46865921 2018-10-23 01:20:30 ./0/120/120_medium.txt
-rw-r--r--  myuser/myuser 102516584 2018-10-23 01:20:30 ./0/120/120_big.txt
^CInterrupt
lftp myuser.myuser@10.10.10.10:/web/public> exit

As you can see “find -l” has the same output as “ls -al” but the command output recursively all sub-directories (in fact, invisible for the user the command just makes the ftp client to change the directory and list it for every sub-directory recursively!).
The output could be interrupted with Ctrl+C, you’ll be still in the lftp shell and connected to the FTP server if you were before.

Scripts

You can execute this command as a script and redirect the output to file or another program like awk, grep an so on. Just pass the commands in a string to “-e” lftp command argument like:

lftp -u "myuser.myuser,myuserpass" -e "find -l /;exit" 10.10.10.10 &> all_files.log
lftp -u "myuser.myuser,myuserpass" -e "find -l web/public/;exit" 10.10.10.10 &> all_files_web.log

or redirect it to multiple pipes like awk (to have only the data you need – file size), egrep (to remove the directories) and a cycle to sum the total file size of selected files like. Note the while is in a group, because each pipe starts a shell, so won’t have $total after the while (and it will output only 0).

 
myuser@srv ~ # total=0;lftp -u "myuser.myuser,myuserpass" -e "find -l web/public/;exit" 10.10.10.10|egrep -v "^d" |grep "_big"|awk '{print $3}'|(while read one; do total=$((total+one));done;echo $total)
6452345191
myuser@srv ~ # 

Missing “ls -R”

So many FTP implementations lack the list directories recursively:
“-R” is interpreted as a file name, not an argument.

lftp -u "myuser.myuser,myuserpass" 10.10.10.10
lftp myuser.myuser@10.10.10.10:~> ls -R
ls: ls -R: Access failed: 550 No such file or directory. (-R)
lftp myuser.myuser@10.10.10.10:/>

lftp with find command completely eliminates the lack of “ls -R”!

Leave a Reply

Your email address will not be published.