I have a Cassandra 1.2 cluster on my local machine with changed hosts:
#cassandra nodes myip myip (Node 1)<---- seed 1 myip 127.0.0.3 (Node 2) myip 127.0.0.4 (Node 3)<---- seed 2 myip 127.0.0.5 (Node 4) myip 127.0.0.6 (Node 5)
... therefore the cluster has a total of 5 nodes. I use Virtual Nodes, so I am not havo to set initial_token anywhere.
The actions that I performed:
- Starting seed nodes (1 and 3)
- Start nodes 2 and 4.
- Disable Node 3.
- Try joining Node 5.
The problem is the 4th step, when I try to join Node 5 Cassandra throws this exception:
java.lang.IllegalStateException: unable to find sufficient sources for streaming range (-7204018021044318047,-6965052533844316617] at org.apache.cassandra.dht.RangeStreamer.getRangeFetchMap(RangeStreamer.java:205) at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:129) at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81) at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:890) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:659) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:514) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:411) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:278) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:366) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:409) java.lang.IllegalStateException: unable to find sufficient sources for streaming range (-7204018021044318047,-6965052533844316617] at org.apache.cassandra.dht.RangeStreamer.getRangeFetchMap(RangeStreamer.java:205) at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:129) at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81) at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:890) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:659) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:514) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:411) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:278) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:366) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:409) Exception encountered during startup: unable to find sufficient sources for streaming range (-7204018021044318047,-6965052533844316617] ERROR 16:55:47,634 Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:321) at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:479) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.lang.Thread.run(Thread.java:722)
The range (-7204018021044318047, -6965052533844316617] just belongs to the deceased Node (Node 3). Due to the fact that I use virtual nodes, I believe that the ring should be rebalanced:
Cluster rebalancing is no longer required when adding or removing nodes. When a Node joins a cluster, it takes responsibility for an even portion of the data from other nodes in the cluster. If a node fails, the load is distributed evenly among other nodes in the cluster.
But I'm wrong, obviously :).
What can I do to join a new Node in an existing cluster when the previous Node is down? Please note that I have 256 tokens per node, moving nodetool is not an option (I think).
Thanks in advance, any information would be welcome!