Configure logstash to read logs from Amazon S3 Recycle Bin

I am trying to configure logstash to read the logs that are generated in my Amazon S3 but were not successful. The following are the details:

  • I installed logstash for ec2 instance
  • My logs - all gz files in s3 bucket
  • The conf file looks like this:
  input {
     s3 {
       access_key_id => "MY_ACCESS_KEY_ID"
       bucket => "MY_BUCKET"
       region => "MY_REGION"
       secret_access_key => "MY_SECRET_ACESS_KEY"
       prefix => "/"
       type => "s3"
       add_field => {source => gzfiles}
     }
   }

   filter {
     if [type] == "s3" {
       csv {
         columns => ["date", "time", "x-edge-location", "sc-bytes", "c-ip", "cs-method", "Host", "cs-uri-stem", "sc-status", "Referer", "User-Agent", "cs-uri-query", "Cookie", "x-edge-result-type", "x-edge-request-id"]  
       }
     }

     if ([message] = ~ / ^ # /) {
       drop {}
     } 
   }

   output {
     elasticsearch {
       host => "ELASTICSEARCH_URL" protocol => "http"
     } 
   }
+7
amazon-s3 amazon-web-services logstash logstash-configuration logstash-file
source share

No one has answered this question yet.

See related questions:

208
Create a bucket in Amazon S3
135
How are you looking for amazon s3 bucket?
127
How to configure SSL for Amazon S3 Bucket
one
logstash does not retrieve data from a log file
one
How to get Logstash to analyze newly added / registered content on file input in real time
0
Logstash locks the log file
0
Reading PDF content from LOGSTASH
0
Reading a log file from a given path using logstash
0
logstash 6.2.4 convert IIS logs to unknown elasticsearch settings
-2
Logstash {: exception => # <URI :: InvalidComponentError: invalid component (expected port component): 443>

All Articles