How do people handle connection loss through scaling in amazon auto-scaling group?

Question

As I understand it, when Amazon automatically scales groups down, any connections open to the completed instance are simply lost - there is no graceful termination.

I wonder how others handle this.

My thinking is that the initiator of the connection should handle the failure, since it must deal with the situation where the instance is not working and not intentionally terminating.

Any thoughts?

Thanks,

Pete

Answer 1

The way I did this is the hook of the life cycle. Which can interrupt the completion process for a set time (default is 1 hour).

It is intended to resume after your work is completed, but the timeout worked to drain the hacked connection.

You have the option to add a hook to your Auto Scaling group of instances in this state in the "Termination: Pending" state. This state allows you to access these instances until they are complete.

source: http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AutoScalingGroupLifecycle.html

con: tuning through the CLI, but not so bad.

How to do it: http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/adding-lifecycle-hooks.html

When creating IAM, you will need this policy:

{ "Version": "2012-10-17", "Statement": [ { "Action": [ "autoscaling:PutLifecycleHook", "autoscaling:DeleteLifecycleHook", "autoscaling:RecordLifecycleActionHeartbeat", "autoscaling:CompleteLifecycleAction", "autoscaling:DescribeAutoscalingGroups", "autoscaling:DescribeLifecycleHooks", "autoscaling:PutInstanceInStandby", "autoscaling:PutInstanceInService", "iam:AddRoleToInstanceProfile", "iam:CreateInstanceProfile", "iam:CreateRole", "iam:PassRole", "iam:ListInstanceProfiles", "ec2:Describe*" ], "Effect": "Allow", "Resource": "*" } ] } 

Good luck

Answer 2

If you use a load balancer, you can enable the connection drain option on the Instances tab. It allows you to set the time to wait for connections to close before the instance is completed. Max. 3600 seconds. See https://aws.amazon.com/blogs/aws/elb-connection-draining-remove-instances-from-service-with-care/

Answer 3

You typically use an elastic load balancer (ELB) in front of your autoscaled instances. The load balancer will stop sending requests to the instance to be completed. If you use, for example, the following format:

 as-put-scaling-policy MyScaleDownPolicy --auto-scaling-group MyAutoScalingGroup --adjustment=-1 --type ChangeInCapacity --cooldown 300 

You will have enough time for your instance to complete processing the requests it had before it is complete.

For more details see: http://docs.amazonwebservices.com/AutoScaling/latest/DeveloperGuide/US_SetUpASLBApp.html

Note that you must have an ELB group for these instances. From AWS Auto-scaling docs :

After automatic scaling determines which particular instance will be completed, it checks whether the instance is part of the elastic load balancing group. If so, Auto Scaling instructs load balancing to remove the instance from the load balancing group and wait for the deletion to complete. If Auto Scaling determines that the instance is not part of the elastic load balancing group, attempt to automatically scale the instance by completing the system shutdown scripts.

Answer 4

There is no solution in the EC2 / AutoscalingGroups search space. I did not expect that pending removal from the ELB to complete means that existing connections will be closed. ELBs use their own non-configurable timeouts, but even assuming the group will wait for this when the ELB reports that any open connections are optimistic.

A potential answer is to fork out other AWS offerings such as SQS and SNS. These two can support most of the termination policies, as they can also be triggered by CloudWatch signals. The missing parts are employee selection, recovery time, and alarm reset.