Elasticearch not working after full disk

Question

I made a space, restarted the service (not good), and then rebooted, but I still get this in elasticsearch.log:

[2015-02-16 13:35:19,625][WARN ][cluster.action.shard ] [Server] [logstash-2015.02.16][1] sending failed shard for [logstash-2015.02.16][1], node[PFamB-ZJS7CwSdyyAcP_8A], [P], s[INITIALIZING], indexUUID [tZ3I9HZ6TDaZSicIuGWRWQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[logstash-2015.02.16][1] failed to recover shard]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: ElasticsearchIllegalArgumentException[No version type match [83]]; ]] [2015-02-16 13:35:19,625][WARN ][cluster.action.shard ] [Server] [logstash-2015.02.16][1] received shard failed for [logstash-2015.02.16][1], node[PFamB-ZJS7CwSdyyAcP_8A], [P], s[INITIALIZING], indexUUID [tZ3I9HZ6TDaZSicIuGWRWQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[logstash-2015.02.16][1] failed to recover shard]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: ElasticsearchIllegalArgumentException[No version type match [83]]; ]] [2015-02-16 13:35:43,570][DEBUG][action.index ] [Server] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-16 13:36:10,757][DEBUG][action.index ] [Server] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] 

What should I do?

Answer 1

Bigfoot's solution is the only thing that works.

While the observed stack trace seems similar to: https://github.com/elastic/elasticsearch/issues/12055

This extract request should fix the problem: https://github.com/elastic/elasticsearch/pull/9797

But upgrading to version 1.5.0 also does not do the trick.

So the only thing that works is:

 find /var/lib/elasticsearch/elasticsearch/nodes/ -name "*.recovering" 

And delete all recoverable files. Of course, this has side effects.

Answer 2

I also have a problem with the error "There is no version type matching" in the logs.

(Also after filling the disc)

Tried a big boot solution for manually deleting recovery files and it worked.

Perhaps this can have some negative side effects.