How to backup Zookeeper?

Question

I read that copying the data directory will work. But this is a combination of magazines and snapshots. How do people create zookeeper backup infrastructure? Export? Dump? Custom script? What are the best practices?

Answer 1

We modify the zkConfig.py script, which is the provided project when installing zookeeper. It allows you to interact with zookeeper through a python script.

We modify scripts to dump and save all contents every day, and then back up files. Although I would be curious to hear other solutions.

Answer 2

Zookeeper records a snapshot when it determines that it has enough transactions and each new snapshot completely replaces the old ones. Thus, the last snapshot + transaction log since the snapshot is enough to restore the current state. To simplify the calculations, you can simply back up the last 3 snapshots (in case the last binding is corrupted) and transaction logs from the timestamp corresponding to the earliest snapshot. The links below contain more detailed information.

• http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_dataFileManagement
• http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance

Answer 3

I had the same requirement, and it turned out that most of the available parameters either did not work, or require a lot of configuration.

The best option I found was Guano, which is a small Java application that visits every node in a tree recursively, starting with the declared node and uploads it to the appropriate directory structure, so you get a directory structure of simple files that are structured as actual wood.

You can also restore these backups by asking them to restore recursively from anywhere in this tree. I think this is not bad for both backup and reconnaissance. For example, I immediately used ack from the root to find all the files with the record that I cared about.

This is a simple addition to the correct backup by simply setting it as a cron job and adding a zip step to compress the entire backup to the archive, as well as handle any necessary rotation.

The tool has several drawbacks:

• Since it stands on Github, the original is not compiled due to the lack of several imported goods. Several people have made PR or forks that fix this problem, for example https://github.com/feldoh/guano , which is my fork in which I also improved the documents. I also now precompiled the jar and will call the binaries at https://bintray.com/feldoh/Guano/guano .
• It only deletes data, which is good for research, but loses metadata such as mTime or version of data. Admittedly, the recovery should probably be counted as an update, so I cannot say that it is really bad, but it is not a real recovery at a particular point in time.

NB: I created my own Zookeeper editor, as I had problems finding one of those that worked and met my needs. Depending on when you read this, https://github.com/feldoh/JZookeeperEdit may also have an export function. Questions 13/14 cover this planned function.

Answer 4

Netflix provided a solution for this exhibitor . This is a ZooKeeper collaborative process, such as monitoring, backup / restore, cleanup, and visualization.