How to implement module or integration tests for logstash configuration?

Question

With logstash 1.2.1, you can now conditionally do different things. Even an earlier conf file can be more complicated if you manage many log files and implement metric extraction.

After watching this comprehensive example, I really wondered how I can detect any breakdowns in this configuration?

Any ideas.

Answer 1

To check the syntax, there is --configtest :

 java -jar logstash.jar agent --configtest --config <yourconfigfile> 

To check the configuration logic, you can write rspec tests. This is an example rspec file for checking the haproxy log filter:

 require "test_utils" describe "haproxy logs" do extend LogStash::RSpec config <<-CONFIG filter { grok { type => "haproxy" add_tag => [ "HTTP_REQUEST" ] pattern => "%{HAPROXYHTTP}" } date { type => 'haproxy' match => [ 'accept_date', 'dd/MMM/yyyy:HH:mm:ss.SSS' ] } } CONFIG sample({'@message' => '<150>Oct 8 08:46:47 syslog.host.net haproxy[13262]: 10.0.1.2:44799 [08/Oct/2013:08:46:44.256] frontend-name backend-name/server.host.net 0/0/0/1/2 404 1147 - - ---- 0/0/0/0/0 0/0 {client.host.net||||Apache-HttpClient/4.1.2 (java 1. 5)} {text/html;charset=utf-8|||} "GET /app/status HTTP/1.1"', '@source_host' => '127.0.0.1', '@type' => 'haproxy', '@source' => 'tcp://127.0.0.1:60207/', }) do insist { subject["@fields"]["backend_name"] } == [ "backend-name" ] insist { subject["@fields"]["http_request"] } == [ "/app/status" ] insist { subject["tags"].include?("HTTP_REQUEST") } insist { subject["@timestamp"] } == "2013-10-08T06:46:44.256Z" reject { subject["@timestamp"] } == "2013-10-08T06:46:47Z" end end 

This will be based on the given filter configuration, run the input samples and check if the expected result is obtained.

To run the test, save the test as haproxy_spec.rb and run `logstash rspec:

 java -jar logstash.jar rspec haproxy_spec.rb 

There are many sample examples in the original Logstash repository.

Answer 2

since logstash has been updated and now the command will be something like (specify the folder)

 /opt/logstash/bin/logstash agent --configtest -f /etc/logstash/logstash-indexer/conf.d 

If you see some kind of warning, but the error message is mixed together and you do not know which one has the problem. You must check the file one by one.