Cassandra incremental backup cleaning guidelines

Question

We have an incremental backup on our Cassandra cluster. The “backup” folders under the data folders now contain a lot of data, and some of them have millions of files.

According to the documentation : "DataStax recommends setting up a process to clear incremental backup hard links every time you create a new snapshot."

I don’t understand how best to clean these files. Can they just be deleted when taking a snapshot or delete files that are older than a certain period?

My thought was to be safe, to run a regular script to delete files older than 30 days:

find [Cassandra data root]/*/*/backups -type f -mtime +30 -delete 

Am I too careful? We do not care about the long history of backups.

Thanks.

Answer 1

You are probably too careful, although this is not always bad, but there are a number of considerations. A good template is to have several snapshots (for example, weekly snapshots returning to a certain period) and all backups during this period of time so that you can restore known states. For example, if for some reason your last snapshot is not working for some reason, if you still have the previous snapshot + all sstables since then, you can use it.

You can delete all backups created after snapshot as an act of taking snapshots of snapshots and hard links to all sstables in the snapshot directory. Just make sure your snapshots actually happen and end (this is a pretty solid process with hard links) before getting rid of old snapshots and deleting backups.

You should also check your recovery process, as this will give you a good idea of ​​what you need. You should be able to recover from the last snapshot + sstables saved since that time. It would be nice to start a new cluster and try to recover data from your snapshots + backups, or perhaps try this process in a test environment.

I would like to point out this article: 'Cassandra and Backups' as a good cassandra backup and restore run.