How to debug date match?

Question

I match events like

[Sun Jan 11 10:43:35 2015][3205.51466981] user idp : testing 10.234.22.220 (10.234.22.220) [61673782] 

from

 %{SYSLOG5424SD:timestamp}%{GREEDYDATA}user %{WORD:user} : testing %{HOST:ip} 

This works, I see various fields in elasticsearch / kibana. In particular, the timestamp in the above example maps to [Sun Jan 11 10:43:35 2015]

Now I would like to use this match with date to have @timestamp right.

I tried using in filter

  date { match => [ "timestamp", "SYSLOG5424SD" ] } 

but this causes logstash to crash with the exit offering an error report file - I opened the ticket .

In the meantime, I tried explicitly matching the pattern with

  date { match => [ "timestamp", "\[EEE MMM dd HH:mm:ss y\]" ] } 

As you suspect - it never matches, @timestamp set at the time the event is logstash.

Can you identify the problem or is there a smart way to debug such cases?

Answer 1

The time stamp matching specified by the date filter is not based on regular expressions or grok expressions. This is why installing SYSLOG5424SD does not work there. In addition to the few special cases listed in the documentation, you can only use tokens recognized by the Joda-Time library. See the joda.time.format.DateTimeFormat class documentation .

You were very close to figure this out - just don't avoid the square brackets:

 date { match => ["timestamp", "[EEE MMM dd HH:mm:ss y]"] } 

Again, Joda-Time patterns are not regular expressions, so you don't have to do anything special to match literals with a square bracket. Quoting the Joda-Time documentation:

Any characters in the pattern that are not within the ranges ['a' .. 'z'] and ['A' .. 'Z'] will be treated as quoted text. For example, characters such as ':', '.', '', '#' AND '?' appear in the resulting text of the time, even if they are not enclosed in single quotes.

Answer 2

Regarding your second question: yes, there is a smart way to debug such cases, there is the grok online debugger ( http://grokdebug.herokuapp.com/ ) and the joda time debugger I created, inspired by the first: https: // java-time- parse-debugger.herokuapp.com/

Refresh . Since users are encouraged to upgrade to the Java 8 DateTime API, I ported this debugging web application to use the Java 8 DateTime API without JodaTime.