Logstash grok filter for logs with arbitrary value attribute pairs

Question

(This is related to my other question logstash grok filter for custom logs )

I have a log file whose lines look something like this:

14:46:16.603 [http-nio-8080-exec-4] INFO METERING - msg=93e6dd5e-c009-46b3-b9eb-f753ee3b889a CREATE_JOB job=a820018e-7ad7-481a-97b0-bd705c3280ad data=71b1652e-16c8-4b33-9a57-f5fcb3d5de92 14:46:17.378 [http-nio-8080-exec-3] INFO METERING - msg=c1ddb068-e6a2-450a-9f8b-7cbc1dbc222a SET_STATUS job=a820018e-7ad7-481a-97b0-bd705c3280ad status=ACTIVE final=false 

I built a template that matched the first line:

 %{TIME:timestamp} %{NOTSPACE:http} %{WORD:loglevel}%{SPACE}%{WORD:logtype} - msg=%{NOTSPACE:msg}%{SPACE}%{WORD:action}%{SPACE}job=%{NOTSPACE:job}%{SPACE}data=%{NOTSPACE:data} 

but is it obvious that it only works for rows with data= at the end, compared to status= and final= at the end of the second row, or other attribute pairs on other rows? How do I set up a template that says that after a certain point there will be an arbitrary pair foo=bar that I want to recognize and output as attribute / value pairs in the output file?

Answer 1

You can change your grok template to have all pairs of key values ​​in one field ( kvpairs ):

 %{TIME:timestamp} %{NOTSPACE:http} %{WORD:loglevel}%{SPACE}%{WORD:logtype} - %{GREEDYDATA:kvpairs} 

You can then use the kv filter to parse key value pairs.

 kv { source => "kvpairs" remove_field => [ "kvpairs" ] # Delete the field afterwards } 

Unfortunately, you have some simple values ​​inside your kv pairs (e.g. CREATE_JOB ). You can analyze them with grok and use one kv filter for values ​​before and another kv filter for values ​​after these simple values.