Elasticsearch: field extract number

Question

I use elasticsearch and kibana to store my logs. Now I want to extract the number from the field and save it in a new field.

So, for example, having this:

accountExist lead time: 1046 ms

I would like to extract the number (1046) and see it in a new field in the kiban.

Is it possible? as? thanks for the help

Answer 1

You will need to do this before / during indexing.

Inside Elasticsearch, you can get what you need during indexing:

• Define a new analyzer using Pattern Analyzer to wrap a regular expression (for your purposes, writing down consecutive numbers in a string is a good answer on this ).
• Create a new numeric field in the mapping to store the retrieved times.
• Use copy_to to copy the log message from the input field to the new numerical field from (2), where the new analyzer will analyze it.

API analysis can be useful for testing purposes.

Answer 2

As long as you do not perform actions, if you should avoid reindexing, you can use scripts in kiban.

Introduction here: https://www.elastic.co/blog/using-painless-kibana-scripted-fields

• enable painless regex support by putting the following in your elasticsearch.yaml:

  script.painless.regex.enabled: true

• elasticsearch reboot
• create a new script field in Kibana via Management → Index Patterns → Scripted Fields
• choose a painless language and number as type
• create the actual script, for example:

  def logMsg = params ['_ source'] ['log_message'];
 if (logMsg == null) {
  return -10000;
 }
 def m = /.*accountExist execution time: ([0-9] +) ms. * $ /. matcher (params ['_ source'] ['log_message']);
 if (m.matches ()) {
    return Integer.parseInt (m.group (1))
 } else {
    return -10000
 }

• you must completely reload the site in order for the new fields to be completed, just re-performing the search on the open discovery site will not receive the new fields. (It almost made me give up trying to make it work.)
• use script in search or visualization

Although I understand that it does not work for script fields for millions of log entries, my usecase is a very specific log entry that is logged 10 times a day and I use only the received fields to create a visualization or analysis when I shorten it in advance candidates for regular requests.

It would be interesting if it is possible that these fields will be calculated only in those situations where you need them (or they make sense and can be calculated for a start, that is, to make the "-1000 return" unnecessary). Currently they will be applied and displayed for each journal entry.
You can create the fields written by the script inside queries, such as: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-script-fields.html , but this seems too much too big than under the hood to easily maintain: /