Elastic Beanstalk CloudWatch Log streaming stops working - How to debug

Question

My resilient Beanstalk environment stops node.js streaming events for CloudWatch logs. Streaming is great for watching minutes in a new instance. After viewing minutes, no more logs appear on CloudWatch.

I configured AWS Elastic Beanstalk for the log stream in CloudWatch under "Configuration"> "Software Configuration"> "CloudWatch Logs"> "Log Stream" (true). I turned off the log stream and reactivated it as a test. A look at cloudwatch

• The latest eb activity log is about 10 minutes.
• Error log is not available (in no instance)
• nginx / access.log is the view in seconds
• nodejs.log is about an hour (short after restarting the instance)

Each health check writes a log entry of every kind of seconds in nodejs.log.

I did not find any logs in the ec2 instance regarding log streaming.

• Are there any similar problems?
• How can I create node.ls node logs with elastic beanstalk flow in CloudWatch logs.

--- EDIT

[ ec2-user@ip- ###-##-##-## log]$ cat /var/log/awslogs.log 2017-03-07 11:01:05,928 - cwlogs.push.stream - INFO - 31861 - Thread-1 - Detected file rotation, notifying reader 2017-03-07 11:01:05,928 - cwlogs.push.stream - INFO - 31861 - Thread-1 - Reader is still alive. 2017-03-07 11:01:05,928 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/error.log*'. 2017-03-07 11:01:05,928 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/access.log*'. 2017-03-07 11:01:06,052 - cwlogs.push.reader - INFO - 31861 - Thread-8 - No data is left. Reader is leaving. 2017-03-07 11:01:10,929 - cwlogs.push.stream - INFO - 31861 - Thread-1 - Removing dead reader [2177a5cce5ed29525de329bfdc292ff1, /var/log/nginx/access.log] 2017-03-07 11:01:10,929 - cwlogs.push.stream - INFO - 31861 - Thread-1 - Starting reader for [92257964a10edeb586f084f4f2ba35de, /var/log/nginx/access.log] 2017-03-07 11:01:10,930 - cwlogs.push.reader - INFO - 31861 - Thread-11 - Start reading file from 0. 2017-03-07 11:01:10,930 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/error.log*'. 2017-03-07 11:01:10,930 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/access.log*'. 2017-03-07 11:01:15,931 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/error.log*'. 2017-03-07 11:01:15,931 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/access.log*'. 2017-03-07 11:01:16,788 - cwlogs.push.publisher - INFO - 31861 - Thread-7 - Log group: /aws/elasticbeanstalk/production/var/log/nginx/access.log, log stream: i-0bd24767864801e2c, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1488884470930, 'start_position': 0L, 'end_position': 114L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1488884472931, 'start_position': 341L, 'end_position': 454L}, 'source_id': '92257964a10edeb586f084f4f2ba35de', 'num_of_events': 4, 'batch_size_in_bytes': 554} 2017-03-07 11:01:20,932 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/error.log*'. 2017-03-07 11:01:20,932 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/access.log*'. 2017-03-07 11:01:25,933 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/error.log*'. 2017-03-07 11:01:25,933 - cwlogs.push.stream - WARNING - 31861 - Thread-1 - No file is found with given path '/var/log/httpd/access.log*'. 2017-03-07 11:01:27,881 - cwlogs.push.publisher - INFO - 31861 - Thread-7 - Log group: /aws/elasticbeanstalk/production/var/log/nginx/access.log, log stream: i-0bd24767864801e2c, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1488884481933, 'start_position': 454L, 'end_position': 568L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1488884482934, 'start_position': 568L, 'end_position': 681L}, 'source_id': '92257964a10edeb586f084f4f2ba35de', 'num_of_events': 2, 'batch_size_in_bytes': 277} 

Answer 1

When Andrew (@ andrew-ferk) and I activated log streaming, he created all the log groups and streams in CloudWatch with the current log. After we deployed again, we noticed that the magazines had stopped. This is because aws hashes the first line of the log. If he saw this hash before processing this file, as if it had already been processed

If you use npm start , the name of your versioned application will be shown on the first lines.

You can add CMD date && npm start to your dockerfile to run a new first line each time, or run npm in silent mode (if your first output is unique).

In addition, according to their documents, you must add some rules to your elastic beanstalk before you enable the AWS-Docs feature .

 { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:GetLogEvents", "logs:PutLogEvents", "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:PutRetentionPolicy" ], "Resource": [ "*" ] } ] } 

Answer 2

The following frequently asked questions may be helpful:

• CloudWatch Logs Agent Frequently Asked Questions
• Why can't I send log data to CloudWatch Logs using the awslogs agent?

Some things to check if you are streaming custom log files:

1. eb ssh to an eb ssh instance and look at /var/log/awslogs.log . If it doesn’t even mention “Downloading additional configurations from (your awslogs configuration file)”, make sure that you install your configuration file correctly and also restart the awslogs service after installing it (presumably using .ebextensions . See “Custom file magazine "). Streaming ”in Using Elastic Beanstalk with Amazon CloudWatch Logs . For information on restarting the awslogs service , see the commands section in the logs-streamtocloudwatch-linux.config file .
2. CloudWatch Log Agent is in state. If the first few lines of your log file are empty or never change, you may need to set file_fingerprint_lines . See CloudWatch Logs Agent Reference .