Geek Answers Handbook

AWS auto-scaling groups and non-ELB health checks

We have auto-scaling groups for one of our cloud-forming stacks that have a processor-based signal that determines when to scale instances.

This is great, but we recently expanded it from one node to three, and one of these nodes could not boot via cfn-init. After reducing the workload and downsizing the group to one node, he killed two good instances and left the partially loaded node with the only remaining instance. This meant that we stopped processing the work until someone logged in and restarted the boot process.

Obviously, this is not perfect. What is the best way to notify the auto-scaling group that the node is not healthy when it is not behind the ELB?

Since this is only a bootstrap, I would really like it to return to the auto-scaling group that failed in this node and completed it, and the new node is deployed in its place.

+7
amazon-web-services autoscaling
source share
2 answers

A colleague just showed me http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/as-configure-healthcheck.html , which looks convenient.

If you have your own health check system, you can use the information from your health check system to set the health state of instances in the automatic scaling group.

UPDATE I managed to start this work during startup.

Here is what my UserData section for ASG looks like:

#!/bin/bash -v set -x export AWS_DEFAULT_REGION=us-west-1 cfn-init --region us-west-1 --stack bapi-prod --resource LaunchConfiguration -v if [[ $? -ne 0 ]]; then export INSTANCE=`curl http://169.254.169.254/latest/meta-data/instance-id` aws autoscaling set-instance-health \ --instance-id $INSTANCE \ --health-status Unhealthy fi
+7
source share
  cfn-init --region us-west-1 --stack bapi-prod --resource LaunchConfiguration -v if [[ $? -ne 0 ]]; then export INSTANCE=`curl http://169.254.169.254/latest/meta-data/instance-id` aws autoscaling set-instance-health \ --instance-id $INSTANCE \ --health-status Unhealthy fi

Can also be done as single line. For example, I use the following in Terraform:

 runcmd: - /tmp/runcmd-puppet.sh || { export INSTANCE=`curl http://169.254.169.254/latest/meta-data/instance-id`; aws autoscaling --region eu-west-1 set-instance-health --instance-id $INSTANCE --health-status Unhealthy; }
0
source share

More articles:


All Articles