Delete the zookeeper logs and snapshot with FileTxnSnapLog from the command-line

Author:

The new version of zookeeper has the ability to auto-purge the logs and snapshots (if autopurge.snapRetainCount and autopurge.purgeInterval are enabled in the configuration), but if an older version is used or the administrator would like to force freeing space, there is a way using the command-line to remove the zookeepers logs and snapshots.
The manual shows how to do it (https://archive.cloudera.com/cdh4/cdh/4/zookeeper/zookeeperAdmin.html#sc_advancedConfiguration):

 java -cp zookeeper.jar:lib/slf4j-api-1.6.1.jar:lib/slf4j-log4j12-1.6.1.jar:lib/log4j-1.2.15.jar:conf \
     org.apache.zookeeper.server.PurgeTxnLog <dataDir> <snapDir> -n <count>
  • Load all needed jars with the current installed versions in zookeeper install directory appended with /lib
  • use the function org.apache.zookeeper.server.PurgeTxnLog
  • Append three parameters “[dataDir] [snapDir] -n [count]”. is in fact /datalog directory, is the directory where the snapshots are kept.

A more clear and detailed syntax:

java -cp [zookeeper & slf4j-api & slf4j-log4j12 & log4j & ... ].jar:conf org.apache.zookeeper.server.PurgeTxnLog \
     <base_datalog_dir> <base_snapshot_dir> <count>

Here is an example with zookeeper 3.7.0:

root@zoo1:~# cd /apache-zookeeper-3.7.0-bin/lib
root@zoo1:/apache-zookeeper-3.7.0-bin/lib# java -cp zookeeper-3.7.0.jar:slf4j-api-1.7.30.jar:slf4j-log4j12-1.7.30.jar:log4j-1.2.17.jar:zookeeper-jute-3.7.0.jar:snappy-java-1.1.7.7.jar:conf org.apache.zookeeper.server.PurgeTxnLog /datalog/ /data/ -n 3
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.server.persistence.FileTxnSnapLog).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Removing file: Sep 9, 2021, 6:07:13 AM  /datalog/version-2/log.4d6800000001
Removing file: Sep 9, 2021, 5:06:57 AM  /data/version-2/snapshot.4d6700c1986c
Removing file: Sep 9, 2021, 6:07:13 AM  /data/version-2/snapshot.4d6800010b26
Removing file: Sep 9, 2021, 4:45:10 AM  /data/version-2/snapshot.4d6700bef73a
Removing file: Sep 9, 2021, 5:44:23 AM  /data/version-2/snapshot.4d6700c62032
Removing file: Sep 9, 2021, 5:31:04 AM  /data/version-2/snapshot.4d6700c48e0f
Removing file: Sep 9, 2021, 5:37:37 AM  /data/version-2/snapshot.4d6700c55610
Removing file: Sep 9, 2021, 5:14:36 AM  /data/version-2/snapshot.4d6700c29039
Removing file: Sep 9, 2021, 5:22:33 AM  /data/version-2/snapshot.4d6700c387b3
Removing file: Sep 9, 2021, 4:57:02 AM  /data/version-2/snapshot.4d6700c0571e
Removing file: Sep 9, 2021, 5:56:59 AM  /data/version-2/snapshot.4d6700c63856

A message for removing files should be printed, if not probably the directories’ paths are wrong!

This is how the zookeeper is organized in the system:

  1. Zookeeper installation directory is /apache-zookeeper-3.7.0-bin meaning the lib is under: /apache-zookeeper-3.7.0-bin/lib, where the jars are placed: 
    root@zoo1:/apache-zookeeper-3.7.0-bin/lib# ls /apache-zookeeper-3.7.0-bin/lib/
audience-annotations-0.12.0.jar               jline-2.14.6.LICENSE.txt                               netty-transport-native-unix-common-4.1.59.Final.LICENSE.txt
commons-cli-1.4.jar                           jline-2.14.6.jar                                       netty-transport-native-unix-common-4.1.59.Final.jar
jackson-annotations-2.10.5.jar                log4j-1.2.17.LICENSE.txt                               simpleclient-0.9.0.LICENSE.txt
jackson-core-2.10.5.jar                       log4j-1.2.17.jar                                       simpleclient-0.9.0.jar
jackson-databind-2.10.5.1.jar                 metrics-core-4.1.12.1.jar                              simpleclient_common-0.9.0.jar
javax.servlet-api-3.1.0.jar                   metrics-core-4.1.12.1.jar_LICENSE.txt                  simpleclient_common-0.9.0_LICENSE.txt
jetty-http-9.4.38.v20210224.LICENSE.txt       netty-buffer-4.1.59.Final.LICENSE.txt                  simpleclient_hotspot-0.9.0.jar
jetty-http-9.4.38.v20210224.jar               netty-buffer-4.1.59.Final.jar                          simpleclient_hotspot-0.9.0_LICENSE.txt
jetty-io-9.4.38.v20210224.LICENSE.txt         netty-codec-4.1.59.Final.LICENSE.txt                   simpleclient_servlet-0.9.0.jar
jetty-io-9.4.38.v20210224.jar                 netty-codec-4.1.59.Final.jar                           simpleclient_servlet-0.9.0_LICENSE.txt
jetty-security-9.4.38.v20210224.LICENSE.txt   netty-common-4.1.59.Final.LICENSE.txt                  slf4j-1.7.30.LICENSE.txt
jetty-security-9.4.38.v20210224.jar           netty-common-4.1.59.Final.jar                          slf4j-api-1.7.30.jar
jetty-server-9.4.38.v20210224.LICENSE.txt     netty-handler-4.1.59.Final.LICENSE.txt                 slf4j-log4j12-1.7.30.jar
jetty-server-9.4.38.v20210224.jar             netty-handler-4.1.59.Final.jar                         snappy-java-1.1.7.7.jar
jetty-servlet-9.4.38.v20210224.LICENSE.txt    netty-resolver-4.1.59.Final.LICENSE.txt                snappy-java-1.1.7.7.jar_LICENSE.txt
jetty-servlet-9.4.38.v20210224.jar            netty-resolver-4.1.59.Final.jar                        zookeeper-3.7.0.jar
jetty-util-9.4.38.v20210224.LICENSE.txt       netty-transport-4.1.59.Final.LICENSE.txt               zookeeper-jute-3.7.0.jar
jetty-util-9.4.38.v20210224.jar               netty-transport-4.1.59.Final.jar                       zookeeper-prometheus-metrics-3.7.0.jar
jetty-util-ajax-9.4.38.v20210224.LICENSE.txt  netty-transport-native-epoll-4.1.59.Final.LICENSE.txt
jetty-util-ajax-9.4.38.v20210224.jar          netty-transport-native-epoll-4.1.59.Final.jar
  2. The datalog directory is under /datalog: 
    root@zoo1:/apache-zookeeper-3.7.0-bin/lib# ls -altr /datalog/
total 12
drwxr-xr-x 1 root      root      4096 Sep  9 05:56 ..
drwxr-xr-x 3 zookeeper root      4096 Sep  9 05:56 .
drwxr-xr-x 2 zookeeper zookeeper 4096 Sep  9 07:04 version-2
root@zoo1:/apache-zookeeper-3.7.0-bin/lib# ls -altr /datalog/version-2/
total 131852
drwxr-xr-x 3 zookeeper root          4096 Sep  9 05:56 ..
-rw-r--r-- 1 zookeeper zookeeper 67108880 Sep  9 06:18 log.4d6800010b28
-rw-r--r-- 1 zookeeper zookeeper 67108880 Sep  9 06:29 log.4d6800025751
-rw-r--r-- 1 zookeeper zookeeper 67108880 Sep  9 06:39 log.4d680003b40f
-rw-r--r-- 1 zookeeper zookeeper 67108880 Sep  9 06:52 log.4d680004d2d8
drwxr-xr-x 3 zookeeper zookeeper     4096 Sep  9 06:52 .
-rw-r--r-- 1 zookeeper zookeeper 67108880 Sep  9 07:03 log.4d6800064eda
  3. The snapshot directory is under /data 
    root@zoo1:/apache-zookeeper-3.7.0-bin/lib# ls -altr /data/
total 968
-rw-r--r-- 1 zookeeper root           2 Aug 31 07:47 myid
drwxr-xr-x 3 zookeeper root        4096 Aug 31 07:55 .
drwxr-xr-x 1 root      root        4096 Sep  9 05:56 ..
drwxr-xr-x 3 zookeeper zookeeper 974848 Sep  9 07:04 version-2
root@zoo1:/apache-zookeeper-3.7.0-bin# ls -altr /data/version-2/
total 79004
drwxr-xr-x 3 zookeeper root         4096 Aug 31 07:55 ..
-rw-r--r-- 1 zookeeper zookeeper 6402486 Sep  9 04:45 snapshot.4d6700bef73a
-rw-r--r-- 1 zookeeper zookeeper 6068093 Sep  9 04:57 snapshot.4d6700c0571e
-rw-r--r-- 1 zookeeper zookeeper 6905356 Sep  9 05:06 snapshot.4d6700c1986c
-rw-r--r-- 1 zookeeper zookeeper 6484358 Sep  9 05:14 snapshot.4d6700c29039
-rw-r--r-- 1 zookeeper zookeeper 6319179 Sep  9 05:22 snapshot.4d6700c387b3
-rw-r--r-- 1 zookeeper zookeeper 6362239 Sep  9 05:31 snapshot.4d6700c48e0f
-rw-r--r-- 1 zookeeper zookeeper 6280967 Sep  9 05:37 snapshot.4d6700c55610
-rw-r--r-- 1 zookeeper zookeeper 6251946 Sep  9 05:44 snapshot.4d6700c62032
-rw-r--r-- 1 zookeeper zookeeper       5 Sep  9 05:56 acceptedEpoch
-rw-r--r-- 1 zookeeper zookeeper 6208681 Sep  9 05:56 snapshot.4d6700c63856
-rw-r--r-- 1 zookeeper zookeeper       5 Sep  9 05:56 currentEpoch
-rw-r--r-- 1 zookeeper zookeeper 7442360 Sep  9 06:07 snapshot.4d6800010b26
-rw-r--r-- 1 zookeeper zookeeper 7666290 Sep  9 06:18 snapshot.4d680002574f
-rw-r--r-- 1 zookeeper zookeeper 7467034 Sep  9 06:29 snapshot.4d680003b40d
drwxr-xr-x 2 root      root         4096 Sep  9 06:32 version-2
drwxr-xr-x 3 zookeeper zookeeper  974848 Sep  9 06:32 .

Do not use for directories /datalog/version-2 or /data/version-2 it is wrong and no files will be removed!

Troubleshooting

If executing the above line outputs a missing java class like below, the easiest way is just to search for the name in the [zookeeper-install-directory]/lib/ with tools like grep or any other text file search tool. 

root@zoo1:/apache-zookeeper-3.7.0-bin/lib# java -cp zookeeper-3.7.0.jar:slf4j-api-1.7.30.jar:slf4j-log4j12-1.7.30.jar:log4j-1.2.17.jar:zookeeper-jute-3.7.0.jar:conf org.apache.zookeeper.server.PurgeTxnLog /datalog/ /data/ -n 3
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.server.persistence.FileTxnSnapLog).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.NoClassDefFoundError: org/xerial/snappy/SnappyInputStream
        at org.apache.zookeeper.server.persistence.FileSnap.findNValidSnapshots(FileSnap.java:171)
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.findNValidSnapshots(FileTxnSnapLog.java:568)
        at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:82)
        at org.apache.zookeeper.server.PurgeTxnLog.main(PurgeTxnLog.java:192)
Caused by: java.lang.ClassNotFoundException: org.xerial.snappy.SnappyInputStream
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
        at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
        ... 4 more
root@zoo1:/apache-zookeeper-3.7.0-bin/lib# grep -ir SnappyInputStream
grep: snappy-java-1.1.7.7.jar: binary file matches

So there is a match with the name SnappyInputStream in jar file snappy-java-1.1.7.7.jar, so just include it in the java -cp command.

Categories: zookeeper
Tags: , , , , , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *