When will LogStash exceed queue bandwidth and discard messages?

Question

I use LogStash to collect logs from my service. The data volume is so large (20 GB / day) that I am afraid that some data will be reset at peak times.

So I asked a question here and decided to add Redis as a buffer between ELB and LogStash to prevent data loss.

However, I am interested to know about when LogStash will exceed the queue bandwidth and cancel messages?

Since I conducted several experiments, and the result shows that LogStash can fully process all the data without any loss, for example a local file → LogStash → local file, netcat → LogStash → local file.

Can someone give me a convincing example when LogStash will eventually delete messages? So I can better understand why we need a buffer in front of it.

Answer 1

As far as I know, the Logstash queue is very small. See here .

Logstash sets each queue size to 20. This means only 20 events can be pending into the next phase.
This helps reduce any data loss and in general avoids logstash trying to act as a data storage   
system. These internal queues are not for storing messages long-term.

As you say, the size of your daily logs is 20 GB. This is quite a lot. Therefore, it is recommended that you install redis before logstash. Another advantage for installing redis is that there is an error and shutdown while working with the journal rack, redis can buffer the logs for you, otherwise all your logs will be deleted.