I am trying to use Logstash to convert XML to JSON for ElasticSearch. I can get values read and sent to ElasticSearch. The problem is that all values come out as arrays. I would like them to come out like strings. I know that I can do replace for each field separately, but then I ran into a problem when nested fields have 3 depth levels.
XML
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <acs2:SubmitTestResult xmlns:acs2="http://tempuri.org/" xmlns:acs="http://schemas.sompleace.org" xmlns:acs1="http://schemas.someplace.org"> <acs2:locationId>Location Id</acs2:locationId> <acs2:userId>User Id</acs2:userId> <acs2:TestResult> <acs1:CreatedBy>My Name</acs1:CreatedBy> <acs1:CreatedDate>2015-08-07</acs1:CreatedDate> <acs1:Output>10.5</acs1:Output> </acs2:TestResult> </acs2:SubmitTestResult>
Logstash Configuration
input { file { path => "/var/log/logstash/test.xml" } } filter { multiline { pattern => "^\s\s(\s\s|\<\/acs2:SubmitTestResult\>)" what => "previous" } if "multiline" in [tags] { mutate { replace => ["message", '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>%{message}'] } xml { target => "SubmitTestResult" source => "message" } mutate { remove_field => ["message", "@version", "host", "@timestamp", "path", "tags", "type"] remove_field => ["entry", "[SubmitTestResult][xmlns:acs2]", "[SubmitTestResult][xmlns:acs]", "[SubmitTestResult][xmlns:acs1]"] # This works replace => [ "[SubmitTestResult][locationId]", "%{[SubmitTestResult][locationId]}" ] # This does NOT work replace => [ "[SubmitTestResult][TestResult][CreatedBy]", "%{[SubmitTestResult][TestResult][CreatedBy]}" ] } } } output { stdout { codec => "rubydebug" } elasticsearch { index => "xmltest" cluster => "logstash" } }
Result
{ "_index": "xmltest", "_type": "logs", "_id": "AU8IZBURkkRvuur_3YDA", "_version": 1, "found": true, "_source": { "SubmitTestResult": { "locationId": "Location Id", "userId": [ "User Id" ], "TestResult": [ { "CreatedBy": [ "My Name" ], "CreatedDate": [ "2015-08-07" ], "Output": [ "10.5" ] } ] } } }
As you can see, the output is an array for each element (except for the locationId, which I replaced). I try not to do a replacement for each item. Is there any way to configure the configuration to ensure that the output works correctly? If not, how can I get 3 levels in depth replace ?
- UPDATE -
I figured out how to get to level 3 in the test results. Replace:
replace => [ "[SubmitTestResult][TestResult][0][CreatedBy]", "%{[SubmitTestResult][TestResult][0][CreatedBy]}" ]