Logstash: Preserving Value in Events

Question

I have a date that is present only once in each log file, and I am trying to add this date to all the following events after it has been matched once, which to some extent acts as a global variable. (The date is at the top of the document, and I cannot use multiline or make changes to the name or contents of the file)

For this, my approach is to use a grep filter with drop => false .

 grok { patterns_dir => "[...]" match => [ "message", "%{DATELINE}" ] tag_on_failure => [ ] } grep { add_field => { "grepdate" => "%{mydate}" } drop => false } date { locale => "en" timezone => "Europe/Paris" match => [ "grepdate", "yyyyMMdd" ] target => "grepdate" } 

Regular expression:

 DATELINE (= Date: (?<mydate>[0-9]{8})) 

I noticed that the grepdate field grepdate correctly added to all events - this is what I want - but the value of this field is not the date itself (value %{mydate} ), but the actual string is "%{mydate}" , unless the actual match is executed for the first time (when analyzing the actual date in my log file, the grepdate field contains the correct value)

What can I do to fix this?

Any help is greatly appreciated.

Edit:

Now I'm trying to find a solution involving using the memorize plugin. However, I get the following error:

You cannot use more than one working filter, because the following plugins do not work with several workers: remember

Is there a way to make this filter thread safe?

Answer 1

Perhaps you should use the official aggregate filter , since memorize not official and will not work with Logstash> 2.0 .

It will look like this:

 # same as what you have now grok { patterns_dir => "[...]" match => [ "message", "%{DATELINE}" ] tag_on_failure => [ "not_date_line" ] } # add a fictional taskId field to correlate all lines mutate { add_field => { "taskId" => "all" } } # if we're processing the first line, remember the date if "not_date_line" not in [tags] { aggregate { task_id => "%{taskId}" code => "map['mydate'] = event['mydate']" } } # if we're processing the next lines, add the date else { aggregate { task_id => "%{taskId}" code => "event['mydate'] = map['mydate']" map_action => "update" timeout => 0 } } 

All your events will have a mydate field with the date that was in the first line of the log.