Elasticsearch - what if the fields have the same name but multiple display

Question

I use Elasticsearch to store data sent from several sources outside my system, i.e. I do not control the incoming data - I just get the json document and save it. I do not have logstash with its filters in the middle, only ES and Kibana. Each data source sent its own data type, and they are all stored in the same index (for each tenant), but in different types. However, since I cannot manage the data transferred to me, it is possible to receive documents of different types with a field with the same name and different structure.
For example , suppose I have type1 and type2 with an FLD field, which is an object in both cases, but the structure of this object is not the same. In particular, FLD.name is a string field in type1, but an object in type2. And in this case, when data of type 1 arrives, they are successfully saved, but when data of type 2 arrives, they are rejected:

Failed to put index mappings [[myindex]], enter [type2]
java.lang.IllegalArgumentException: Mapper for [FLD] conflicts with existing mapping of other types. [Cannot combine object mapping (FLD.name) with object mapping [FLD.name]]

ES documentation clearly states that fields with the same name in the same index in different types of mappings displayed inside the same field inside must have the same display ( see here ).

My question is , what can I do in this case? I would prefer to keep all types in the same index. Is it possible to add a unique suffix for each type in the field names or something like this? Any other solution? I am new to Elasticsearch, so maybe I am missing something simple ... Thanks in advance.

Answer 1

There is no way to make an arbitrary JSON index without preprocessing before indexing it - even Dynamic Templates are quite flexible.

You can smooth nested objects into key-value pairs and use the Nested data type , Multi-fields and ignore_malformed to index arbitrary JSON (even with type conflicts), as described here . Unfortunately, Elasticsearch can still throw an exception during a request if you try, for example, to match a string with kv_pairs.value.long , so you must select the appropriate fields depending on the format of the value.