Shutting down an EC2 instance if it is idle right in front of another billable hour

Question

At unpredictable times (user request), I need to start working with intensive memory. To do this, I get an instance or instance on request and tag it as non_idle . When the task is completed (which may take several hours), I give it the idle tag. Due to the AWS billing model in hours, I want to keep this instance alive until another billable hour arrives if another job happens. If the job arrives, the instance must be reused and marked as non_idle . If the task is not completed during this time, the instance must complete.

Does AWS offer a turnkey solution for this? As far as I know, CloudWatch cannot set alarms that should start at a specific time, not to mention the use of CPUUtilization or instance tags. Otherwise, perhaps I could just set up a java timer or scala actor for each instance created, which starts every hour after the instance is created and checks the idle tag.

Answer 1

There is no readily available AWS solution for this fine-grained optimization, but you can use the existing building blocks for your own creation based on the launch time of the current instance (see Dmitry Samsky’s solution for finding out How long did this EC2 instance run? ).

Game "Chicken"

Shlomo Svidler studied this optimization in his article Play Chicken with Spot Instances , albeit a slightly different motivation in the context of Amazon EC2 Spot Instances :

AWS Spot instances have an interesting economic feature that makes the game a little bit on the system. Like all EC2 instances, when you initiate the termination of Spot Instance, then you pay for the entire hour, even if you use less than an hour. But when AWS completes the instance because the spot price exceeds the purchase price, you do not pay for the current hour.

Of course, the mechanics are the same, so you can just reuse the script that he built, i.e. execute this script instead of or in addition to marking the instance as idle :

 #! /bin/bash t=/tmp/ec2.running.seconds.$$ if wget -q -O $t http://169.254.169.254/latest/meta-data/local-ipv4 ; then # add 60 seconds artificially as a safety margin let runningSecs=$(( `date +%s` - `date -r $t +%s` ))+60 rm -f $t let runningSecsThisHour=$runningSecs%3600 let runningMinsThisHour=$runningSecsThisHour/60 let leftMins=60-$runningMinsThisHour # start shutdown one minute earlier than actually required let shutdownDelayMins=$leftMins-1 if [[ $shutdownDelayMins > 1 && $shutdownDelayMins < 60 ]]; then echo "Shutting down in $shutdownDelayMins mins." # TODO: Notify off-instance listener that the game of chicken has begun sudo shutdown -h +$shutdownDelayMins else echo "Shutting down now." sudo shutdown -h now fi exit 0 fi echo "Failed to determine remaining minutes in this billable hour. Terminating now." sudo shutdown -h now exit 1 

After completing the task, you can cancel the scheduled completion instead of or in addition to marking the instance with non_idle as follows:

 sudo shutdown -c 

This is also a red button emergency command during testing / operation, see, for example, Shlomo Warning:

Make sure you really understand what this script does before using this. If you mistakenly plan to close the instance, you can cancel it with this command, run on the instance: sudo shutdown -c

Adding CloudWatch to the game

You could take another Shlomo approach by integrating with Amazon CloudWatch , which recently added the Use Amazon CloudWatch option to detect and turn off unused Amazon EC2 instances , see the Amazon CloudWatch intro blog post - Signal actions for details:

Today we give you the opportunity to stop or end your EC2 when a CloudWatch alarm is triggered . You can use this as (detect an abnormal state and then act) or as part of your application processing logic (expect the expected condition and then act). [emphasis mine]

Your use case is listed in the Application Integration section:

You can also create CloudWatch alarms based on Custom Metrics , which you monitor an instance individually. You could, for example, measure calls to your own web services APIs, page requests or posting messages per minute and respond as desired.

Thus, you can use this new feature to publish custom metrics to CloudWatch to indicate whether the instance should end (there is an idle ) based on and Dmitriy detects the start time and reset the metric again when the task arrives, and the instance should continue to work (this is non_idle ) - for example, EC2 will take care of completion, 2 of 3 automation steps would be moved from the instance to the operating environment, and accordingly the management and visibility of the automation process would be improved.