Deleting a node with a subtree in a Zookeeper is pretty easy and it could be done even with the Zookeeper cli from the command-line.
Deleting a node, which has a subtree could be done with the command “deleteall” but the command should be executing on the leader in the Zookeeper cluster!
If the user receives the following error:
root@zk01:/apache-zookeeper-3.7.0-bin# ./bin/zkCli.sh [zk: localhost:2181(CONNECTED) 25] deleteall /clickhouse/statdata/collectd_metrics/replicas/replica_2 Failed to delete some node(s) in the subtree!
First, check if the Zookeeper node is the leader, and second, execute the deleteall command on the path, which should be deleted. Using the zkServer.sh in the Zookeeper bin directory to check whether the Zookeeper node is the leader one:
root@zk03:/apache-zookeeper-3.7.0-bin# ./bin/zkServer.sh status ZooKeeper JMX enabled by default Using config: /conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: leader root@zk03:/apache-zookeeper-3.7.0-bin# ./bin/zkCli.sh [zk: localhost:2181(CONNECTED) 0] deleteall /clickhouse/statdata/collectd_metrics/replicas/replica_2 [zk: localhost:2181(CONNECTED) 1]
No error, when the command deleteall is executed in the leader. No additional output either.
Deleting with the “delete” command will result in an error, too:
[zk: localhost:2181(CONNECTED) 0] delete /clickhouse/statdata/collectd_metrics/replicas/replica_2 Node not empty: /clickhouse/statdata/collectd_metrics/replicas/replica_2
Zookeeper cli output
The follower’s output with delete, deleteall and list command ls:
root@zk01:/apache-zookeeper-3.7.0-bin# ./bin/zkServer.sh status ZooKeeper JMX enabled by default Using config: /conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: follower root@zk01:/apache-zookeeper-3.7.0-bin# ./bin/zkCli.sh Connecting to localhost:2181 2021-09-21 04:24:34,369 [myid:] - INFO [main:Environment@98] - Client environment:zookeeper.version=3.7.0-e3704b390a6697bfdf4b0bef79e3da7a4f6bac4b, built on 2021-03-17 09:46 UTC 2021-09-21 04:24:34,373 [myid:] - INFO [main:Environment@98] - Client environment:host.name=zk01 2021-09-21 04:24:34,373 [myid:] - INFO [main:Environment@98] - Client environment:java.version=11.0.12 2021-09-21 04:24:34,375 [myid:] - INFO [main:Environment@98] - Client environment:java.vendor=Oracle Corporation 2021-09-21 04:24:34,376 [myid:] - INFO [main:Environment@98] - Client environment:java.home=/usr/local/openjdk-11 2021-09-21 04:24:34,376 [myid:] - INFO [main:Environment@98] - Client environment:java.class.path=/apache-zookeeper-3.7.0-bin/bin/../zookeeper-server/target/classes:/apache-zookeeper-3.7.0-bin/bin/../build/classes:/apache-zookeeper-3.7.0-bin/bin/../zookeeper-server/target/lib/*.jar:/apache-zookeeper-3.7.0-bin/bin/../build/lib/*.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/zookeeper-prometheus-metrics-3.7.0.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/zookeeper-jute-3.7.0.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/zookeeper-3.7.0.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/snappy-java-1.1.7.7.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/slf4j-log4j12-1.7.30.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/slf4j-api-1.7.30.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/simpleclient_servlet-0.9.0.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/simpleclient_hotspot-0.9.0.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/simpleclient_common-0.9.0.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/simpleclient-0.9.0.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/netty-transport-native-unix-common-4.1.59.Final.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/netty-transport-native-epoll-4.1.59.Final.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/netty-transport-4.1.59.Final.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/netty-resolver-4.1.59.Final.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/netty-handler-4.1.59.Final.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/netty-common-4.1.59.Final.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/netty-codec-4.1.59.Final.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/netty-buffer-4.1.59.Final.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/metrics-core-4.1.12.1.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/log4j-1.2.17.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jline-2.14.6.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jetty-util-ajax-9.4.38.v20210224.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jetty-util-9.4.38.v20210224.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jetty-servlet-9.4.38.v20210224.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jetty-server-9.4.38.v20210224.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jetty-security-9.4.38.v20210224.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jetty-io-9.4.38.v20210224.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jetty-http-9.4.38.v20210224.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jackson-databind-2.10.5.1.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jackson-core-2.10.5.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jackson-annotations-2.10.5.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/commons-cli-1.4.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/audience-annotations-0.12.0.jar:/apache-zookeeper-3.7.0-bin/bin/../zookeeper-*.jar:/apache-zookeeper-3.7.0-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/conf: 2021-09-21 04:24:34,376 [myid:] - INFO [main:Environment@98] - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib 2021-09-21 04:24:34,376 [myid:] - INFO [main:Environment@98] - Client environment:java.io.tmpdir=/tmp 2021-09-21 04:24:34,376 [myid:] - INFO [main:Environment@98] - Client environment:java.compiler=<NA> 2021-09-21 04:24:34,377 [myid:] - INFO [main:Environment@98] - Client environment:os.name=Linux 2021-09-21 04:24:34,377 [myid:] - INFO [main:Environment@98] - Client environment:os.arch=amd64 2021-09-21 04:24:34,377 [myid:] - INFO [main:Environment@98] - Client environment:os.version=5.4.0-42-generic 2021-09-21 04:24:34,377 [myid:] - INFO [main:Environment@98] - Client environment:user.name=root 2021-09-21 04:24:34,377 [myid:] - INFO [main:Environment@98] - Client environment:user.home=/root 2021-09-21 04:24:34,377 [myid:] - INFO [main:Environment@98] - Client environment:user.dir=/apache-zookeeper-3.7.0-bin 2021-09-21 04:24:34,378 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.free=55MB 2021-09-21 04:24:34,380 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.max=256MB 2021-09-21 04:24:34,380 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.total=64MB 2021-09-21 04:24:34,385 [myid:] - INFO [main:ZooKeeper@637] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@7946e1f4 2021-09-21 04:24:34,390 [myid:] - INFO [main:X509Util@77] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2021-09-21 04:24:34,397 [myid:] - INFO [main:ClientCnxnSocket@239] - jute.maxbuffer value is 1048575 Bytes 2021-09-21 04:24:34,409 [myid:] - INFO [main:ClientCnxn@1726] - zookeeper.request.timeout value is 0. feature enabled=false Welcome to ZooKeeper! 2021-09-21 04:24:34,439 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1171] - Opening socket connection to server localhost/127.0.0.1:2181. 2021-09-21 04:24:34,441 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1173] - SASL config status: Will not attempt to authenticate using SASL (unknown error) JLine support is enabled 2021-09-21 04:24:34,454 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1005] - Socket connection established, initiating session, client: /127.0.0.1:40846, server: localhost/127.0.0.1:2181 2021-09-21 04:24:34,467 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1438] - Session establishment complete on server localhost/127.0.0.1:2181, session id = 0x1006ad505c30007, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: localhost:2181(CONNECTED) 0] delete /clickhouse/statdata/collectd_metrics/replicas/replica_2 Node not empty: /clickhouse/statdata/collectd_metrics/replicas/replica_2 [zk: localhost:2181(CONNECTED) 1] deleteall /clickhouse/statdata/collectd_metrics/replicas/replica_2 Failed to delete some node(s) in the subtree! [zk: localhost:2181(CONNECTED) 2] ls /clickhouse/statdata/collectd_metrics/replicas/replica_2/parts [2021_5155978_5164111_2367, 2021_5155978_5164112_2368, 2021_5155978_5164113_2369, 2021_5155978_5164114_2370, 2021_5155978_5164115_2371, 2021_5155978_5164116_2372, 2021_5155978_5164117_2373]
Deleting the node subtree in the Zookeeper leader:
root@zk03:/apache-zookeeper-3.7.0-bin# ./bin/zkServer.sh status ZooKeeper JMX enabled by default Using config: /conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: leader root@zk03:/apache-zookeeper-3.7.0-bin# ./bin/zkCli.sh Connecting to localhost:2181 2021-09-21 04:41:55,397 [myid:] - INFO [main:Environment@98] - Client environment:zookeeper.version=3.7.0-e3704b390a6697bfdf4b0bef79e3da7a4f6bac4b, built on 2021-03-17 09:46 UTC 2021-09-21 04:41:55,401 [myid:] - INFO [main:Environment@98] - Client environment:host.name=zk03 2021-09-21 04:41:55,401 [myid:] - INFO [main:Environment@98] - Client environment:java.version=11.0.12 2021-09-21 04:41:55,404 [myid:] - INFO [main:Environment@98] - Client environment:java.vendor=Oracle Corporation 2021-09-21 04:41:55,404 [myid:] - INFO [main:Environment@98] - Client environment:java.home=/usr/local/openjdk-11 2021-09-21 04:41:55,404 [myid:] - INFO [main:Environment@98] - Client environment:java.class.path=/apache-zookeeper-3.7.0-bin/bin/../zookeeper-server/target/classes:/apache-zookeeper-3.7.0-bin/bin/../build/classes:/apache-zookeeper-3.7.0-bin/bin/../zookeeper-server/target/lib/*.jar:/apache-zookeeper-3.7.0-bin/bin/../build/lib/*.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/zookeeper-prometheus-metrics-3.7.0.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/zookeeper-jute-3.7.0.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/zookeeper-3.7.0.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/snappy-java-1.1.7.7.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/slf4j-log4j12-1.7.30.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/slf4j-api-1.7.30.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/simpleclient_servlet-0.9.0.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/simpleclient_hotspot-0.9.0.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/simpleclient_common-0.9.0.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/simpleclient-0.9.0.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/netty-transport-native-unix-common-4.1.59.Final.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/netty-transport-native-epoll-4.1.59.Final.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/netty-transport-4.1.59.Final.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/netty-resolver-4.1.59.Final.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/netty-handler-4.1.59.Final.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/netty-common-4.1.59.Final.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/netty-codec-4.1.59.Final.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/netty-buffer-4.1.59.Final.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/metrics-core-4.1.12.1.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/log4j-1.2.17.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jline-2.14.6.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jetty-util-ajax-9.4.38.v20210224.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jetty-util-9.4.38.v20210224.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jetty-servlet-9.4.38.v20210224.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jetty-server-9.4.38.v20210224.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jetty-security-9.4.38.v20210224.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jetty-io-9.4.38.v20210224.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jetty-http-9.4.38.v20210224.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jackson-databind-2.10.5.1.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jackson-core-2.10.5.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/jackson-annotations-2.10.5.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/commons-cli-1.4.jar:/apache-zookeeper-3.7.0-bin/bin/../lib/audience-annotations-0.12.0.jar:/apache-zookeeper-3.7.0-bin/bin/../zookeeper-*.jar:/apache-zookeeper-3.7.0-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/conf: 2021-09-21 04:41:55,405 [myid:] - INFO [main:Environment@98] - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib 2021-09-21 04:41:55,405 [myid:] - INFO [main:Environment@98] - Client environment:java.io.tmpdir=/tmp 2021-09-21 04:41:55,405 [myid:] - INFO [main:Environment@98] - Client environment:java.compiler=<NA> 2021-09-21 04:41:55,405 [myid:] - INFO [main:Environment@98] - Client environment:os.name=Linux 2021-09-21 04:41:55,405 [myid:] - INFO [main:Environment@98] - Client environment:os.arch=amd64 2021-09-21 04:41:55,405 [myid:] - INFO [main:Environment@98] - Client environment:os.version=5.4.0-42-generic 2021-09-21 04:41:55,406 [myid:] - INFO [main:Environment@98] - Client environment:user.name=root 2021-09-21 04:41:55,406 [myid:] - INFO [main:Environment@98] - Client environment:user.home=/root 2021-09-21 04:41:55,406 [myid:] - INFO [main:Environment@98] - Client environment:user.dir=/apache-zookeeper-3.7.0-bin 2021-09-21 04:41:55,406 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.free=56MB 2021-09-21 04:41:55,408 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.max=256MB 2021-09-21 04:41:55,408 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.total=64MB 2021-09-21 04:41:55,414 [myid:] - INFO [main:ZooKeeper@637] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@7946e1f4 2021-09-21 04:41:55,419 [myid:] - INFO [main:X509Util@77] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2021-09-21 04:41:55,427 [myid:] - INFO [main:ClientCnxnSocket@239] - jute.maxbuffer value is 1048575 Bytes 2021-09-21 04:41:55,439 [myid:] - INFO [main:ClientCnxn@1726] - zookeeper.request.timeout value is 0. feature enabled=false Welcome to ZooKeeper! 2021-09-21 04:41:55,473 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1171] - Opening socket connection to server localhost/127.0.0.1:2181. 2021-09-21 04:41:55,477 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1173] - SASL config status: Will not attempt to authenticate using SASL (unknown error) JLine support is enabled 2021-09-21 04:41:55,496 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1005] - Socket connection established, initiating session, client: /127.0.0.1:41234, server: localhost/127.0.0.1:2181 2021-09-21 04:41:55,513 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1438] - Session establishment complete on server localhost/127.0.0.1:2181, session id = 0x3006ad4fdbc0002, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: localhost:2181(CONNECTED) 0] deleteall /clickhouse/statdata/collectd_metrics/replicas/replica_2 [zk: localhost:2181(CONNECTED) 1] 2021-09-21 04:42:53,061 [myid:] - ERROR [main:ServiceUtils@42] - Exiting JVM with code 0 root@zk03:/apache-zookeeper-3.7.0-bin#