Cassandra Tombstoning Warnings and Failures Failed

Question

We run the DB Titan Graph server, supported by Cassandra, as a permanent repository and encounter a problem with reaching the limit on the thresholds of Tombstones of Cassandra, which periodically causes errors / timeout when data is accumulated. The seal seems to be unable to keep up with the amount of tombstones added.

Our use case supports:

• High read / write bandwidth.
• High sensitivity to reading.
• Frequent updates to node values ​​in Titan. whereby the rows are updated in Kassandra.

Given the above use cases, we are already optimizing Cassandra to aggressively do the following:

• Aggressive sealing using seal alignment strategies
• Using tombstone_compaction_interval after 60 seconds.
• Using tombstone_threshold should be 0.01
• Setting gc_grace_seconds to 1800

Despite the following optimizations, we still see warnings in Cassandra logs similar to: [WARN] (ReadStage: 7510) org.apache.cassandra.db.filter.SliceQueryFilter: read 0 live and 10,350 tombstones in .graphindex (see tombstone_warn_threshold ) 8001 columns requested, slices = [00-ff], delInfo = {deletedAt = -9223372036854775808, localDeletion = 2147483647}

Sometimes, as time progresses, we also see that the failure threshold is violated and causes errors.

In our cassandra.yaml file, the tombstone_warn_threshold value will be 10000, and the tombstone_failure_threshold value will be much higher than the 250,000 recommended, without any real benefits.

Any help that might point to the correct configurations would be greatly appreciated if there was room for further optimizations. Thank you in advance for your time and help.

Answer 1

Sounds like the root of your problem - your data model. You have done everything you can to mitigate the receipt of a TombstoneOverwhelmingException. Since your data model requires frequent updates that lead to the creation of tombstones, a possible sequential store such as Cassandra may not be entirely suitable for your use case. When we encounter these problems, we had to modify our data model to better fit Cassandra's strengths.

About the removal of http://www.slideshare.net/planetcassandra/8-axel-liljencrantz-23204252 (slides 34-39)

Answer 2

Tombstones are not compressed until gc_grace_seconds is configured in the table for that tombstone. Thus, even increasing the compaction interval, your tombstones will not be removed until gc_grace_seconds expires, with a default value of 10 days. You can try to set gc_grace_seconds to a lower value and do repairs more often (usually you plan to make repairs every gc_grace_seconds_in_days - 1 day).

Answer 3

The variables that you configured help you to expire the tombstones, but it is worth noting that although the tombstones cannot be cleaned before gc_grace_seconds, Cassandra does not guarantee that the tombstones will be cleaned in gc_grace_seconds. In fact, the tombstones do not compact until a clot containing the tombstone is detached, and even then it will not be removed if there is another sstable containing the shaded cell.

This leads to the fact that the tombstones potentially persist for a very long time, especially if you use sstables, which are rarely compressed (say, very large STCS-sstables). To solve this problem, there are tools such as the JMX endpoint for forced UserDefinedCompaction - if you do not know how to use the JMX endpoints, tools for this automatically exist, for example http://www.encql.com/purge-cassandra-tombstones/

Answer 4

So everything is right here. If you frequently repair and compactly reduce your gc_grace_seconds number.

However, it is also worth considering that Inserting Nulls is equivalent to deletion. This will increase the number of tombstones. Instead, you will want to insert UNSET_VALUE if you use prepared statements. It is probably too late for you, but if someone else comes here.