I currently have an architecture with a file bit as the sender of a log that sends logs to an instance of the log indexer, and then to managed elastic search in AWS. Due to persistent TCP connections, I cannot load balance using instances of the AWS ELB index with multiple registrars, since file elements always select instances and send them there. So I decided to use redis. Now, seeing how difficult it is to scale redis and make it a highly accessible component on the ELK stack, I want to ask what is even a redis point. I read it a million times, it acts like a buffer, but if filebeats stops sending logs to logstash, if logstash cannot handle the load, why do we need a buffer. Filebeat is smart enough to know to stop sending logs. Logstash is smart enough to stop sending logs to elastic search if elastic search goes down. Thus, the pipeline stops. I really don't understand that redis acts as a buffer in every standard ELK architecture.
source share