EC2 tools / automation strategies?

Question

What tools or strategies do you use to automate the actions of EC2?

I need to be able to run several instances of EC2, provide it with various programs (primarily Python packages), interact with S3 (primarily load data), and run various tasks. I will do this both on request and on schedule.

I am trying to decide if I should:

• Create AMI with all the software I downloaded or
• Run a simple AMI instance for vanilla linux and scp my software for it

To ensure and automate Boto looks good. Or I could write something with Paramiko . Recommend one or the other that I should look for?

I'm mainly looking for tips / success stories, let me know what works with you.

Answer 1

To answer your bullets about choosing AMI, I would say that it depends on how much software you install.

I was successful with a hybrid approach where I create AMI and download my heavier and more stable software. This is what it takes to start the installer, or takes a long time to install (remember that if you reinstall the package each time as part of the startup process, you pay for the installation each time). Then I download small and unstable software during setup / startup. This bucket contains most of the application code, data, etc. This way I can change my application and not touch the AMI.

The advantages of this approach:

• No need to pay to run the same software thousands of times.
• AMI can remain fairly stable over time.
• Software may be used that requires intervention or interaction with the graphical interface for installation.

The main disadvantages:

• Your version of the AMI operating system will become obsolete over time.
• Your AMI may not be flexible regarding the type of instance / architecture on which it will run. For example, you can create it on a 32-bit OS and thereby prevent it from starting on types of High CPU instances or vice versa. Thus, you can block yourself in the pricing scheme.

I do not use Python, so I can not comment on any of the APIs that you referenced.

Answer 2

AWS has just released the System Manager package, which includes an Automation Service that will (among other things) handle your AMI use cases .

Answer 3

This question was asked some time ago, but I believe that my answer may be useful to other users. I believe the best automation tools available on the market are provided by Cloud Management platforms. For example, they offer automatic scaling, integration of configuration software (Chef / Puppet), database replication, dns management ...

The most popular cloud-based software products are Scalr (disclaimer: I work there), RightScale and enStratus. Scalr is open source and released under the Apache 2 license.

As for your specific question about AMI, cloud-based management platforms usually provide preconfigured AMIs (in Scalr we call them roles). If you want to create your own AMI built on an existing instance, you can create snpashots and use them as a template for future instances.