Error reading saved cache and system table when starting Cassandra

Question

When starting Cassandra Daemon, I encountered the following exception. I run from 1.2 trunk.

WARN 14:47:51,038 error reading saved cache /home/manuzhang/cassandra/saved_caches/system-local-KeyCache-b.db java.lang.NullPointerException at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:141) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:237) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:340) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:312) at org.apache.cassandra.db.Table.initCf(Table.java:332) at org.apache.cassandra.db.Table.<init>(Table.java:265) at org.apache.cassandra.db.Table.open(Table.java:110) at org.apache.cassandra.db.Table.open(Table.java:88) at org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:284) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:168) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:318) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:361) 

here where the caches are stored:

 manuzhang@manuzhang-U24E :~/cassandra/saved_caches$ ls -l total 12 -rw-rw-r-- 1 manuzhang manuzhang 156 Aug 7 13:09 system-local-KeyCache-b.db -rw-rw-r-- 1 manuzhang manuzhang 60 Aug 7 13:09 system-schema_columnfamilies-KeyCache-b.db -rw-rw-r-- 1 manuzhang manuzhang 60 Aug 7 13:09 system-schema_columns-KeyCache-b.db 

In addition, system table files cannot be loaded.

 ERROR 17:03:16,637 Fatal exception during initialization org.apache.cassandra.config.ConfigurationException: Found system table files, but they couldn't be loaded! at org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:303) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:201) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:349) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:392) 

Now I can reproduce the failure of the boot systems table for every three Cassandra launches (after that I clear all the files). An exception is thrown here:

 /** * One of three things will happen if you try to read the system table: * 1. files are present and you can read them: great * 2. no files are there: great (new node is assumed) * 3. files are present but you can't read them: bad * @throws ConfigurationException */ public static void checkHealth() throws ConfigurationException { Table table; try { table = Table.open(Table.SYSTEM_TABLE); } catch (AssertionError err) { // this happens when a user switches from OPP to RP. ConfigurationException ex = new ConfigurationException("Could not read system table!"); ex.initCause(err); throw ex; } ColumnFamilyStore cfs = table.getColumnFamilyStore(LOCAL_CF); String req = "SELECT cluster_name FROM system.%s WHERE key='%s'"; UntypedResultSet result = processInternal(String.format(req, LOCAL_CF, LOCAL_KEY)); if (result.isEmpty() || !result.one().has("cluster_name")) { // this is a brand new node if (!cfs.getSSTables().isEmpty()) throw new ConfigurationException("Found system table files, but they couldn't be loaded!"); // no system files. this is a new node. req = "INSERT INTO system.%s (key, cluster_name) VALUES ('%s', '%s')"; processInternal(String.format(req, LOCAL_CF, LOCAL_KEY, DatabaseDescriptor.getClusterName())); return; } String savedClusterName = result.one().getString("cluster_name"); if (!DatabaseDescriptor.getClusterName().equals(savedClusterName)) throw new ConfigurationException("Saved cluster name " + savedClusterName + " != configured name " + DatabaseDescriptor.getClusterName()); } 

Three runs exactly match the three conditions of the comment.

“No files” on first run, as this is a new node.

In the second run, "there are files and you can read them."

In the third run, "there are files, but you cannot read them," and I checked that both result.isEmpty() and result.one.has("cluster_name") return false .

In fact, I am confused by the exception that "it is impossible to load." What does it mean? I do not think that this is a problem with the file system, since access rights to r / w are granted to the current user.

The above problems disappear after deleting all related files, but I do not want to do this every time Cassandra starts.

This upset me for quite some time.

An unrelated issue is that I don't think Cassandra @stackoverflow has received enough attention from the community. Do you agree?

Any ideas or suggestions would be appreciated.

Thanks.

Answer 1

I had this problem in 2 scenarios.

• I tried changing the separator without deleting the cluster data (I can not do this). Also explain the mailing list .
• I started the cassandra process as superuser the first time sudo ./cassandra , which created the necessary data / log / cache directories with permissions only for the superuser, and then restarted cassandra and started the process as a normal user (and thus, there was no permission to use files in directories created by the superuser process).

I know that you solved the problem, but it may be useful for other developers.