Delete the zookeeper logs and snapshot with FileTxnSnapLog from the command-line

The new version of zookeeper has the ability to auto-purge the logs and snapshots (if autopurge.snapRetainCount and autopurge.purgeInterval are enabled in the configuration), but if an older version is used or the administrator would like to force freeing space, there is a way using the command-line to remove the zookeepers logs and snapshots.
The manual shows how to do it (https://archive.cloudera.com/cdh4/cdh/4/zookeeper/zookeeperAdmin.html#sc_advancedConfiguration):

 java -cp zookeeper.jar:lib/slf4j-api-1.6.1.jar:lib/slf4j-log4j12-1.6.1.jar:lib/log4j-1.2.15.jar:conf \
     org.apache.zookeeper.server.PurgeTxnLog <dataDir> <snapDir> -n <count>
  • Load all needed jars with the current installed versions in zookeeper install directory appended with /lib
  • use the function org.apache.zookeeper.server.PurgeTxnLog
  • Append three parameters “[dataDir] [snapDir] -n [count]”. is in fact /datalog directory, is the directory where the snapshots are kept.

A more clear and detailed syntax:

java -cp [zookeeper & slf4j-api & slf4j-log4j12 & log4j & ... ].jar:conf org.apache.zookeeper.server.PurgeTxnLog \
     <base_datalog_dir> <base_snapshot_dir> <count>

Here is an example with zookeeper 3.7.0:

root@zoo1:~# cd /apache-zookeeper-3.7.0-bin/lib
root@zoo1:/apache-zookeeper-3.7.0-bin/lib# java -cp zookeeper-3.7.0.jar:slf4j-api-1.7.30.jar:slf4j-log4j12-1.7.30.jar:log4j-1.2.17.jar:zookeeper-jute-3.7.0.jar:snappy-java-1.1.7.7.jar:conf org.apache.zookeeper.server.PurgeTxnLog /datalog/ /data/ -n 3
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.server.persistence.FileTxnSnapLog).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Removing file: Sep 9, 2021, 6:07:13 AM  /datalog/version-2/log.4d6800000001
Removing file: Sep 9, 2021, 5:06:57 AM  /data/version-2/snapshot.4d6700c1986c
Removing file: Sep 9, 2021, 6:07:13 AM  /data/version-2/snapshot.4d6800010b26
Removing file: Sep 9, 2021, 4:45:10 AM  /data/version-2/snapshot.4d6700bef73a
Removing file: Sep 9, 2021, 5:44:23 AM  /data/version-2/snapshot.4d6700c62032
Removing file: Sep 9, 2021, 5:31:04 AM  /data/version-2/snapshot.4d6700c48e0f
Removing file: Sep 9, 2021, 5:37:37 AM  /data/version-2/snapshot.4d6700c55610
Removing file: Sep 9, 2021, 5:14:36 AM  /data/version-2/snapshot.4d6700c29039
Removing file: Sep 9, 2021, 5:22:33 AM  /data/version-2/snapshot.4d6700c387b3
Removing file: Sep 9, 2021, 4:57:02 AM  /data/version-2/snapshot.4d6700c0571e
Removing file: Sep 9, 2021, 5:56:59 AM  /data/version-2/snapshot.4d6700c63856

A message for removing files should be printed, if not probably the directories’ paths are wrong!

This is how the zookeeper is organized in the system:

  1. Zookeeper installation directory is /apache-zookeeper-3.7.0-bin meaning the lib is under: /apache-zookeeper-3.7.0-bin/lib, where the jars are placed:
    root@zoo1:/apache-zookeeper-3.7.0-bin/lib# ls /apache-zookeeper-3.7.0-bin/lib/
    audience-annotations-0.12.0.jar               jline-2.14.6.LICENSE.txt                               netty-transport-native-unix-common-4.1.59.Final.LICENSE.txt
    commons-cli-1.4.jar                           jline-2.14.6.jar                                       netty-transport-native-unix-common-4.1.59.Final.jar
    jackson-annotations-2.10.5.jar                log4j-1.2.17.LICENSE.txt                               simpleclient-0.9.0.LICENSE.txt
    jackson-core-2.10.5.jar                       log4j-1.2.17.jar                                       simpleclient-0.9.0.jar
    jackson-databind-2.10.5.1.jar                 metrics-core-4.1.12.1.jar                              simpleclient_common-0.9.0.jar
    javax.servlet-api-3.1.0.jar                   metrics-core-4.1.12.1.jar_LICENSE.txt                  simpleclient_common-0.9.0_LICENSE.txt
    jetty-http-9.4.38.v20210224.LICENSE.txt       netty-buffer-4.1.59.Final.LICENSE.txt                  simpleclient_hotspot-0.9.0.jar
    jetty-http-9.4.38.v20210224.jar               netty-buffer-4.1.59.Final.jar                          simpleclient_hotspot-0.9.0_LICENSE.txt
    jetty-io-9.4.38.v20210224.LICENSE.txt         netty-codec-4.1.59.Final.LICENSE.txt                   simpleclient_servlet-0.9.0.jar
    jetty-io-9.4.38.v20210224.jar                 netty-codec-4.1.59.Final.jar                           simpleclient_servlet-0.9.0_LICENSE.txt
    jetty-security-9.4.38.v20210224.LICENSE.txt   netty-common-4.1.59.Final.LICENSE.txt                  slf4j-1.7.30.LICENSE.txt
    jetty-security-9.4.38.v20210224.jar           netty-common-4.1.59.Final.jar                          slf4j-api-1.7.30.jar
    jetty-server-9.4.38.v20210224.LICENSE.txt     netty-handler-4.1.59.Final.LICENSE.txt                 slf4j-log4j12-1.7.30.jar
    jetty-server-9.4.38.v20210224.jar             netty-handler-4.1.59.Final.jar                         snappy-java-1.1.7.7.jar
    jetty-servlet-9.4.38.v20210224.LICENSE.txt    netty-resolver-4.1.59.Final.LICENSE.txt                snappy-java-1.1.7.7.jar_LICENSE.txt
    jetty-servlet-9.4.38.v20210224.jar            netty-resolver-4.1.59.Final.jar                        zookeeper-3.7.0.jar
    jetty-util-9.4.38.v20210224.LICENSE.txt       netty-transport-4.1.59.Final.LICENSE.txt               zookeeper-jute-3.7.0.jar
    jetty-util-9.4.38.v20210224.jar               netty-transport-4.1.59.Final.jar                       zookeeper-prometheus-metrics-3.7.0.jar
    jetty-util-ajax-9.4.38.v20210224.LICENSE.txt  netty-transport-native-epoll-4.1.59.Final.LICENSE.txt
    jetty-util-ajax-9.4.38.v20210224.jar          netty-transport-native-epoll-4.1.59.Final.jar
    
  2. The datalog directory is under /datalog:
    root@zoo1:/apache-zookeeper-3.7.0-bin/lib# ls -altr /datalog/
    total 12
    drwxr-xr-x 1 root      root      4096 Sep  9 05:56 ..
    drwxr-xr-x 3 zookeeper root      4096 Sep  9 05:56 .
    drwxr-xr-x 2 zookeeper zookeeper 4096 Sep  9 07:04 version-2
    root@zoo1:/apache-zookeeper-3.7.0-bin/lib# ls -altr /datalog/version-2/
    total 131852
    drwxr-xr-x 3 zookeeper root          4096 Sep  9 05:56 ..
    -rw-r--r-- 1 zookeeper zookeeper 67108880 Sep  9 06:18 log.4d6800010b28
    -rw-r--r-- 1 zookeeper zookeeper 67108880 Sep  9 06:29 log.4d6800025751
    -rw-r--r-- 1 zookeeper zookeeper 67108880 Sep  9 06:39 log.4d680003b40f
    -rw-r--r-- 1 zookeeper zookeeper 67108880 Sep  9 06:52 log.4d680004d2d8
    drwxr-xr-x 3 zookeeper zookeeper     4096 Sep  9 06:52 .
    -rw-r--r-- 1 zookeeper zookeeper 67108880 Sep  9 07:03 log.4d6800064eda
    
  3. The snapshot directory is under /data
    root@zoo1:/apache-zookeeper-3.7.0-bin/lib# ls -altr /data/
    total 968
    -rw-r--r-- 1 zookeeper root           2 Aug 31 07:47 myid
    drwxr-xr-x 3 zookeeper root        4096 Aug 31 07:55 .
    drwxr-xr-x 1 root      root        4096 Sep  9 05:56 ..
    drwxr-xr-x 3 zookeeper zookeeper 974848 Sep  9 07:04 version-2
    root@zoo1:/apache-zookeeper-3.7.0-bin# ls -altr /data/version-2/
    total 79004
    drwxr-xr-x 3 zookeeper root         4096 Aug 31 07:55 ..
    -rw-r--r-- 1 zookeeper zookeeper 6402486 Sep  9 04:45 snapshot.4d6700bef73a
    -rw-r--r-- 1 zookeeper zookeeper 6068093 Sep  9 04:57 snapshot.4d6700c0571e
    -rw-r--r-- 1 zookeeper zookeeper 6905356 Sep  9 05:06 snapshot.4d6700c1986c
    -rw-r--r-- 1 zookeeper zookeeper 6484358 Sep  9 05:14 snapshot.4d6700c29039
    -rw-r--r-- 1 zookeeper zookeeper 6319179 Sep  9 05:22 snapshot.4d6700c387b3
    -rw-r--r-- 1 zookeeper zookeeper 6362239 Sep  9 05:31 snapshot.4d6700c48e0f
    -rw-r--r-- 1 zookeeper zookeeper 6280967 Sep  9 05:37 snapshot.4d6700c55610
    -rw-r--r-- 1 zookeeper zookeeper 6251946 Sep  9 05:44 snapshot.4d6700c62032
    -rw-r--r-- 1 zookeeper zookeeper       5 Sep  9 05:56 acceptedEpoch
    -rw-r--r-- 1 zookeeper zookeeper 6208681 Sep  9 05:56 snapshot.4d6700c63856
    -rw-r--r-- 1 zookeeper zookeeper       5 Sep  9 05:56 currentEpoch
    -rw-r--r-- 1 zookeeper zookeeper 7442360 Sep  9 06:07 snapshot.4d6800010b26
    -rw-r--r-- 1 zookeeper zookeeper 7666290 Sep  9 06:18 snapshot.4d680002574f
    -rw-r--r-- 1 zookeeper zookeeper 7467034 Sep  9 06:29 snapshot.4d680003b40d
    drwxr-xr-x 2 root      root         4096 Sep  9 06:32 version-2
    drwxr-xr-x 3 zookeeper zookeeper  974848 Sep  9 06:32 .
    

Do not use for directories /datalog/version-2 or /data/version-2 it is wrong and no files will be removed!

Troubleshooting

If executing the above line outputs a missing java class like below, the easiest way is just to search for the name in the [zookeeper-install-directory]/lib/ with tools like grep or any other text file search tool.

root@zoo1:/apache-zookeeper-3.7.0-bin/lib# java -cp zookeeper-3.7.0.jar:slf4j-api-1.7.30.jar:slf4j-log4j12-1.7.30.jar:log4j-1.2.17.jar:zookeeper-jute-3.7.0.jar:conf org.apache.zookeeper.server.PurgeTxnLog /datalog/ /data/ -n 3
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.server.persistence.FileTxnSnapLog).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.NoClassDefFoundError: org/xerial/snappy/SnappyInputStream
        at org.apache.zookeeper.server.persistence.FileSnap.findNValidSnapshots(FileSnap.java:171)
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.findNValidSnapshots(FileTxnSnapLog.java:568)
        at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:82)
        at org.apache.zookeeper.server.PurgeTxnLog.main(PurgeTxnLog.java:192)
Caused by: java.lang.ClassNotFoundException: org.xerial.snappy.SnappyInputStream
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
        at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
        ... 4 more
root@zoo1:/apache-zookeeper-3.7.0-bin/lib# grep -ir SnappyInputStream
grep: snappy-java-1.1.7.7.jar: binary file matches

So there is a match with the name SnappyInputStream in jar file snappy-java-1.1.7.7.jar, so just include it in the java -cp command.

Delete millions of files slowly without loading the server

There a situations when we need to delete a great deal of files from our filesystem and if we just execute

rm -Rf

the server will surely get loaded and the service it provides will degrade! What if you cannot reformat the filesystem, because the server use it extensively, but you need to delete let’s say a couple of millions file from it? We can use find and usleep (in most linux distro this program is installed by an additional package). The idea is to delete files one by one tuning the pause between every delete. Here you can execute this command in the background or a screen:

find /mnt/storage/old/ -type f -exec echo {} \; -exec rm {} \; -exec usleep 200000 \;

usleep accepts microseconds, so 200000 microseconds are 0.2 seconds. You can tune it precisely with a step of just a microsecond. In the real world under the bash console we probably will use values of max 1/10 of a second around above 100000 microseconds. Execute the command and then watch your server load and tune.

  • usleep in CentOS 7 is installed with package “initscripts”, which is installed by default
  • usleep in Ubuntu is missing and probably won’t find any safe place to download a package to install, but it can be sort of replace with “sleep <floating_point_number>s”, GNU sleep could accept floating point number for the delay and when added “s” at the end it could sleep for a fractions of a seconds. So the command for the Ubuntu is slightly changed:
    find /mnt/storage/old/ -type f -exec echo {} \; -exec rm {} \; -exec sleep 0.2s \;
    
  • not GNU version of sleep require NUMBER, so the smallest sleep is only 1 second, which is too big for the purpose. Check your man manual to see if your system has GNU sleep command.