The number of processes on remote machines on AWS

Question

I am developing an auto-scale system for my application that runs on Amazon EC2 instances. The application reads messages from SQS and processes them.

The Auto Scaling system will track two things:

• The number of messages in SQS,
• The total number of processes running on all EC2 machines.

For example, if the number of messages in SQS exceeds 3000, I want the system to automatically scale, create a new instance of EC2, put the code on it, and whenever the number of messages is below 2000, I want the system to complete the instance of EC2.

I do this with Ruby and capistrano. My question is:

I cannot find a method for determining the number of processes running on all EC2 machines and store the number inside a variable. could you help me?

Answer 1

You might want to use cron and the CloudWatch API to manually enter numbers in CloudWatch as part of an autoscaling policy. By numbers, I mean the number of processes from each instance of ps aux | grep your_process | wc -l ps aux | grep your_process | wc -l

CloudWatch allows you to set an alarm for this manual metric, aggregated by SUM from nr processes through all running instances or using auto-scaling group.

Something to get you started:

Manually pushing RAM memory metrics: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/mon-scripts-perl.html

Another one: http://aws.typepad.com/aws/2011/05/amazon-cloudwatch-user-defined-metrics.html

For memory, this looks simple, as amazon already provides scripts for this. For processes, you may need to dig into these scripts or read official API docs

EDIT:

If you are now worried about a single point of failure in the surveillance system, and you have a list of servers, it might be preferable to examine them in parallel from a remote server:

 rm ~/count.log # SSH in parallel for ROW in `cat ~/ListofIP.txt` do IP=`echo ${ROW} | sed 's/\./ /g' | awk '{print $1}'` ssh -i /path/to/keyfile root@ ${IP} "ps -ef | grep process_name.rb | grep -v grep | wc -l" >> ~/count.log & done # Wait for totals while [ ! `wc -l ~/ListofIP.txt` -eq `wc -l ~/count.log` ] do wait 1 done # Sum up numbers from ~/count.log # Push TO CloudWatch