What to bake in AMI AWS and what to provide with cloud-init?

Question

I use AWS Cloudformation to configure many network infrastructure elements (VPC, SecurityGroups, Subnets, Autoscaling groups, etc.) for my web application. I want the whole process to be automated. I want to press a button and be able to run it all.

I have successfully created a Cloudformation template that configures this entire network infrastructure. However, EC2 instances are currently running without any necessary software. Now I'm trying to figure out how best to get this software.

To do this, I create AMI using Packer.io . But some people instead encouraged me to use the Cloud-Init. What heuristics should I use to decide what to bake in the AMI and / or what to configure using the Cloud-Init?

For example, I want to preconfigure an EC2 instance to allow me ( saqib ) to log in without a password from my own laptop. Therefore, EC2 must have a user. This user must have a home directory. And in this home directory there should be a .ssh/known_hosts file containing encrypted codes. Should I bake these directories in OMI? Or should I use cloud-init to configure them? And how should I make a decision in this and other similar cases?

Answer 1

I like to separate computer software from the environment.

In general, I use the following as a guide:

Build phase

• Create a base machine image with something like Packer, including all the software you need to run your application. Create an AMI from this.
• Install the application (s) on the base machine image by creating the application image. Add a tag and version of this artifact. Do not embed environment-specific environments such as database connections, etc., as this does not allow you to reuse this AMI in different environments.
• Make sure all services are stopped.

Release phase

• Hide the medium consisting of images and the required infrared rays using something like CFN.
• Use Cloud-Init user-data to configure the application environment (database connections, log senders, etc.), and then launch applications / services

This approach provides maximum flexibility and clearly separates the various problems of continuous piping.

Answer 2

One of the important factors determining how you should build servers, AMI and infrastructure planning is the answer to the question: at what stage will I need a new instance?

The answer to this question will determine how much you bake in the AMI, and how much you build after loading.

NOTE. . My experience is with Chef Server, so I will use Chef terminology, but the concepts are the same for any other configuration management stack.

A general rule is to consider your "infrastructure as code." This means that we think about launching instances, creating users on this computer, and managing the known_hosts and SSH files as well as your application code. The ability to track changes in the infrastructure in the source code simplifies management, redistribution, and even CI.

A true chef Introduction covers the terminology of a chef cookbooks, recipes, resources, etc. It shows you how to create a simple LAMP stack, and how you can easily restart it with a single command.

So, given the example in your question, at a high level, I would do the following:

• Run the base UI-Ubuntu Linux AMI (currently 14.04) with a cloud-based information script.
• In the UserData section of the Instance configuration, download the Chef client installation process.
• Run the recipe to create a user.
• Run the recipe to create a known_hosts file for the user.

Tools like Chef are used because you can break the infrastructure down into small blocks of code that perform specific functions. There are numerous Cookbooks already built-in and affordable that perform the basic building blocks of creating services, installing software packages, etc.

All that is said, there are several times when you need to deviate from best practices in the interests of your specific domain and requirements. There may be situations in which you still have to bake all the benefits of infrastructure management in AMI.

Assume that your application performs image processing and requires the use of ImageMagick. Suppose you need to create an ImageMagick from source code. If you did this with chef recipes, it could add another 7 minutes by simply compiling ImageMagick at the normal instance load time. If the expectation of 10-12 minutes is too high for a new instance to appear on the network, you may want to bake your own AMI, which ImageMagick has already compiled and installed.

This is an acceptable solution, but you should keep in mind that managing your own fleet of pre-processed AMIs adds extra infrastructure overhead. You will need to update your custom AMIs as new AMIs are released, you expand to different types of instances and different areas of AWS.