Docker Fluentd Multi-Line Registration Driver

Question

I am trying to create a centralized logging system using fluentd for a docker environment. Currently, I can send docker logs to fluentd using the fluentd docker logging driver, which is a much cleaner solution compared to reading the docker log file using the in_tail method. However, I am currently facing a multi-line log issue.

As can be seen from the figure above, the multiline log is faulty, which is very confusing for the user. Can this be solved?

Thanks.

Cw

Answer 1

Take a look at the multiline parsing in your documentation: http://docs.fluentd.org/articles/parser-plugin-overview#

Basically you should specify a regular expression that will match the start of a new log message, and this will allow flentd to combine multi-line log events into a single message.

Example for regular java stacktrace from their docs:

format multiline format_firstline /\d{4}-\d{1,2}-\d{1,2}/ format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/

Answer 2

I know that this is not so and will "answer" to the current one. But this guide solves the problem with logstash: http://www.labouisse.com/how-to/2015/09/14/elk-and-docker-1-8

JSON support by adding

  json { source => "log_message" target => "json" } 

to your filter after parsing the log line

I did not find a solution for fluentd, so went with this solution instead

Updated link

Answer 3

Using a smooth plugin helped me fix the problem.

Adding these lines to flu-conf

  <filter **> @type concat key log stream_identity_key container_id multiline_start_regexp /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3} multiline_end_regexp /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3} </filter> 

Where my regular expression checks DateTimeStamp in the logs where each line starts, as well as the date and timestamp (note the "log":"2017-09-21 15:03:27.289 ) below

 2017-09-21T15:03:27Z tag {"container_id":"11b0d89723b9c812be65233adbc51a71507bee04e494134258b7af13f089087f","container_name":"/bel_osc.1.bc1k2z6lke1d7djeq5s28xjyl","source":"stdout","log":"2017-09-21 15:03:27.289 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet Engine: Apache Tomcat/8.5.6"} 2017-09-21T15:03:28Z tag {"container_id":"11b0d89723b9c812be65233adbc51a71507bee04e494134258b7af13f089087f","container_name":"/bel_osc.1.bc1k2z6lke1d7djeq5s28xjyl","source":"stdout","log":"2017-09-21 15:03:28.191 INFO 1 --- [ost-startStop-1] oaccC[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext"} 

In addition, I had to add below lines to the Dockerfile to install the plugin

 RUN ["gem", "install", "fluent-plugin-concat", "--version", "2.1.0"] #Works with Fluentd v0.14-debian 

Although this regular expression does not work well when an exception occurs, it is still much better than before. Fluentd Link, for reference .