Cassandra resets the connection with a large load on the insert

Question

I have a cassandra cluster of 8 nodes, cassandra 1.0.8.

I am trying to make many small inserts in a loop with batch_mutate (). After a while (~ 200K inserts), the server drops the connection with the following exception:

org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147) at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:157) at org.apache.cassandra.thrift.Cassandra$Client.send_batch_mutate(Cassandra.java:998) at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:986) at org.scale7.cassandra.pelops.Mutator$1.execute(Mutator.java:46) at org.scale7.cassandra.pelops.Mutator$1.execute(Mutator.java:42) at org.scale7.cassandra.pelops.Operand.tryOperation(Operand.java:56) at org.scale7.cassandra.pelops.Mutator.execute(Mutator.java:51) ... Caused by: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145) ... 25 more 

Otherwise, the cluster is functioning normally. Server logs are clean.

What can cause this problem?

Thanks!

Answer 1

I found a reason: the size of the packet mutation exceeded the size of the TFramedTransport frame. There are two possible solutions: either increase the configuration property "thrift_framed_transport_size_in_mb" in the cassandra.yaml file, or use a smaller batch size.