Introduction to AWS CloudFormation with Example Project Walk-Through

In this tutorial, I am going to show you how to write an AWS CloudFormation Template that deploys a server on EC2.

In 2020, I wrote an AWS CloudFormation Template to deploy an IPsec VPN server on EC2, using Lin Song's setup-ipsec-vpn deployment script. And I am also going to use that example to walk you through the steps to create your own custom CloudFormation template. The source code for the template can be found here.

Note: This article is written in the spirit that the readers may get a better idea of how the template is designed and written.

What is AWS CloudFormation?

I believe whoever reading this must be familiar with Amazon Web Services (AWS), the global cloud platform giant providing a large collection of services.

CloudFormation being one of the services provided by AWS, can deploy a set of cloud resources with custom parameters that you can specify in a template file. For example, you can write a CloudFormation template that launches simultaneously three EC2 instances shipped with the software that you like. It basically takes in a blueprint of your cloud architecture and turns that into a reality.

If you frequently need to deploy a given service on the cloud, you may write a CloudFormation template for that particular service so that you don't have to do the manual labour of setting up EC2 instances by hand. In that scenario, CloudFormation saves a lot of time.

Microsoft's Azure also has Automation, a similar functionality (as opposed to a service) that does the same thing for Azure. Since the setup-ipsec-vpn project does also have an Azure automation quick-deploy template, a personal remark has to be made here that the Azure's Automation has a much more robust and clean standard which saves a lot of work. For example, Azure's Automation can directly grab a piece of code off a remote Git repository on the public Internet, whereas AWS CloudFormation needs the user to either upload the template as a file or specify an S3 bucket where the template is stored. Lately AWS added a "Git repository" option as the source of the template, but not every repository on the public Internet is eligible to be used. It would be great if one day CloudFormation can ease up the process of template importation so that a remote Git repo can be used as an upstream template source.

How does CloudFormation work?

In order to create the resources that you need, CloudFormation needs a template file. And the template file is where you specify the resources you want and the required parameters.

The template file can be either written in JSON or yaml, and must follow a set of predefined syntax to order to be properly accepted by CloudFormation.

After creating the template file, you upload it to CloudFormation and a stack is created which contains all the resources defined in the template.

Every stack may take a while to be fully deployed, and sometimes it may fail to deploy due to errors in the execution of the template, in which case CloudFormation will attempt to "roll-back" the deployment by deleting the resources already created.

After the stack is fully deployed, you can then use the services as you desired.

If you want to delete the stack some time after deployment, you can do so while deleting all the resources explicitly defined in the template. In certain cases resources may fail to be deleted.

Use the Graphical Template Editor

If you are not familiar with typing up the code yourself, you can choose to use the CloudFormation's graphical template editor to quickly create and validate a template without writing a line of code. The graphical template provides a GUI to create and manipulate cloud resources, and then converts the logic into a template file. However, that appraoch has certain limitations.

The graphical template editor does indeed provide a handy function of converting between the templates written in JSON and in yaml. However, per personal experiences, that functionality tends to be faulty once complex logic is involved in the template. The editor also can help you verify the format of the template.

In this tutorial, we are going to show you how to write a template without using the graphical template editor.

The Anatomy of a Template

The entire CloudFormation template is either in JSON (JavaScript Object Notation) or yaml. And at the root level of the object, there are the following six entries:

• Metadata: includes certain data that is required by the CloudFormation graphical template editor and possibly other cloud-side services.
• AWSTemplateFormatVersion: defines the version of the format this file is written in.
• Mappings: inludes user-defined mappings that can be referenced in the template as variables.
• Resources: includes the cloud resources that the user wants to deploy.
• Parameters: includes the parameters for the deployment like the size of the VM, which can be referenced like variables in the template.
• Outputs: defines the information that is to be displayed under the "Ouputs" tab when the stack is successfully deployed.

Normally, if you are writing the template without using the graphical editor, only the latter four items are to be populated.

The Metadata section, however, can also be used nonconventionally as a comment section which may include copyright information or the purpose of the template.

The Mappings section can include a set of user-defined variables, which can be easily referenced by using intrinsic functions within the template. For example you can define respective installation command of software for different operating systems.

The Resources section can include the declaration of the cloud resources to be deployed. They are not limited to just EC2 instances, and almost every single AWS service is supported by AWS CloudFormation. For more information, please refer to AWS CloudFormation Documentation.

Example Project: EC2 Server Deployment

Suppose now that you want to deploy an EC2 instance carrying a specific software (e.g. VPN server, Email server or Web server). How can you do that with CloudFormation template?

The conventional approach or the "manual solution" is to use the AWS CLI or web-portal and select the kind of instance you want to create.(i.e. instance size, OS, location, etc...) After that, you would also have to log into the instance with SSH and configure the programs that you need in a terminal.

The manual solution is highly inefficient and time consuming, which is not ideal if you frequently need the same setup of software on an EC2 to go. CloudFormation comes in handy for exactly that use case.

In this tutorial, I am going to use the CloudFormation template that I wrote in 2020 for the setup-ipsec-vpn project, as an example. The objective of the example template is to install and set up the libreswan IPsec VPN server using Lin Song's installation script.

I will in the following chapter sequentially go through the various parts of the template file in detail.

Principle Logic

The principle logic of this template is to first create an EC2 instance and then run the setup script (user-data) on it. CloudFormation waits for the script's completion and if successful presents the user's generated credentials in the "Outputs" tab on the web portal.

Miscellaneous actions are executed along the way to make the whole thing work. For example, an S3 bucket has to be created to store the credentials for user to download, and the user's private key for accessing the instance over SSH has to be retrieved in plain-text and displayed in the "Outputs" tab upon successful stack creation.

Note: that the example template linked here is not the latest version becaseu I have to make sure the line numbers don't change when the file on Github is being constantly updated.

Metadata Section (Line 1-23)

The metadata section contains a comment paragraph that declares the purpose of this template and some information about the project it belongs to and the author's contact information.

Although the original purpose of the metadata section is to pass through some parameters to the CloudFormation, it can also be used to display a README as in the example template.

If you have used the graphical template editor to create the template code, you might see certain attributes in the metadata section that are automatically generated by the editor. In using the template as is, you may remove such attributes from the metadata section to save some space, as they are not used by the CloudFormation stack creation wizard.

Template Format Version (Line 24)

This single line specifies the version number for the format that's used by the template file. The value 2010-09-09 is usually the same in all templates. Just make sure that this property is there in your template and everything will be fine.

Mappings Section (Line 25-52)

The mappings section includes the mappings between the instance OS distribution version and the corresponding one-liner command to install the certain packages required by aws-cfn-bootstrap, the various "helper scripts" to interface with the CloudFormation service during deployment.

For Debian and Ubuntu options, the one-liner is more or less the same. And since Amazon Linux 2 has all the helper scripts built in, I only need to make sure the correct directories are included in the $PATH. The template used to support versions of CentOS, but these were phased out due to their respective EOL(s) (End of Life) were reached. At the moment, four distributions are supported in total.

Resources Section (Line 53-619)

The Resources section includes all the cloud resources to deploy in the stack. I will go through each and every one of them below.

IAMInstanceProfile (Line 53-71)

This object is an IAM Instance Profile, which is another way of saying an IAM (Identity Access Management) account that is created by the root account and has certain assigned privileges. In this case, it's being assigned the Role with the name S3ExecutionRole, which is another resource in the template.

An IAM Instance Profile is required by the EC2 Instance resource that is going to be declared later in the document.

Ikev2S3Bucket (Line 72-102)

This object is an S3 Bucket, and is used to store the credentials that the user needs in order to set up his client to connect to the VPN server once the deployment is finished.

OpenBucketPolicy (Line 103-132)

This object is a Bucket Policy for the S3 bucket that is going to store the user's connection credentials for download. It contains an ACL (Access Control List) statement that makes the bucket in question permissive to public download action.

VpnVpc (Line 133-139)

This object is a VPC that contains the subnet where the EC2 instance dwells, with the IP block of 10.0.0.0/24.

VpnSubnet (Line 140-162)

This object is the subnet that lives inside the VPC resource defined above, and it also contains the EC2 instance that runs the VPN server program. It has the same CIDR of 10.0.0.0/24.

VpnRouteTable and PublicInternetRoute (Line 163-192)

VpnRouteTable is a route table that is assigned to the VPC resource defined above, and it contains the necessary routing rules that direct the traffic correctly in and out of the VPC.

PublicInternetRoute is a route table rule that allows access to the public Internet for the resources inside the VPN (i.e. EC2 instance).

VpnInstance (Line 193-276)

This is the EC2 instance that runs the VPN server software. As you can see , starting at line 204, a user-data script is attached directly inline in the template for the VM to run at launch. According to AWS standards, the script, if attached in-line, will have to be combined line by line with the Fn::Join intrinsic function, and then encoded by Fn::Base64. Inside the script, Fn::Sub is frequently used to substitute variable names with the template values at runtime. AWS has specified a format in which variables inside the template can be referenced by attached scripts using Fn::Sub.

Another design in the user-data script is that if the script fails or stalls beyong 15 minutes after launch, CloudFormation automatically rolls back the resources already created, deleting the failed stack in the process. That is achieved at line 195-198, with a CreationPolicy dubbed with a ResourceSignal which has a Timeout of 15 minutes (i.e. PT15M). Within the attached script, at line 210 and line 230, is the use of one of the "Helper Scripts", cfn-signal which tells CloudFormation to start the roll-back in case of stall or failure. If everything goes well, the script reaches its end (line 230), which acknowledges CloudFormation of the successful execution of the script. In case something goes wrong, an error will be caught by a trap clause defined at line 210, indicating explicitly an error has occurred.

KeyPair (Line 277-292)

A key pair that the user uses to authenticate himself when accessing the server over SSH.

VpnSecurityGroup (Line 293-329)

A network security group that allows certain traffic to pass through between the Internet and the EC2 instance. Port 22 is opened for TCP traffic in order to use SSH. Port 500 and 4500 are opened are UDP traffic in order to make the IPsec VPN work.

VpcInternetGateway (Line 330-337)

Internet Gateway resource for the VPC, granting Internet access to the VPC in use.

SubnetRouteTableAssociation (Line 338-349)

Subnet Route Table Association, which associates the subnet resource with the route table resource created earlier.

KeyPairDisplayFunction (Line 350-418) and KeyPairDisplayFunctionInfo (Line 419-437)

This is the first of a pair of Python emdedded inline functions that are executed when the stack is created. CloudFormation template allows such Lambda scripts to be embedded in the template as singular cloud resources. That provides immense extensibility to the template, where you can execute commands that are not supported natively by the template's syntax.

First of all, the purpose of this KeyPairDisplayFunction is to retrieve the created key pair's private key material in plain-text, and to be referenced by an Fn::GetAtt for display in the Outputs tab. This script also does the job of generating a unique name for the S3 bucket that stores the generated user credentials.

Two resources are needed, one for the storage of the inline function, and the other for the storage of the returned values which invokes the prior function resource.

AMIInfo (Line 438-458) and AMIInfoFunction (Line 459-517)

This is the second of a pair of Python embedded inline functions that are executed when the stack is created Its job is to obtain the AMI ID of the latest version of the selected GNU/Linux distribution to be used on the server. The template by default only takes in fixed values but the distro images are being updated periodically so a dynamic sorting function like this is needed.

Similar to how the prior function works, AMIInfoFunction carries the inline Python function and IAMInfo stores the return values of the function and invokes the function when called.

LambdaExecutionRole (Line 518-553)

This is the IAM role that is created for the execution of Lambda function, which has the same privilege as the root user basically. Its privileges could have been narrowed down to just what it needs but for simplicity I just granted it maximum privilege.

S3ExecutionRole (Line 554-606)

This is the IAM role that is created for the execution of the user-date script that uploads the generated user credential files to the designated S3 bucket. It has the privilege of doing s3:PutObject upon the Ikev2S3Bucket resource.

InternetGatewayAttachment (Line 607-618)

Attachment resource that associates the Internet Gateway resource the VPC.

Parameters Section (Line 620-667)

The Parameters section includes the parameters that the template takes in during creation to customize the cloud resources. The parameter values are specified by the user either using AWS CLI or the AWS Management Console, whichever being used.

In our case of deploying a VPN server, the following customization parameters are used:

1. VPN Username: the login username of the VPN server.
2. VPN PSK (Pre-Shared Key): a set of characters as an additional authentication secret.
3. VPN Password: the login password of the VPN server.
4. Server OS: the GNU/Linux distribution to be used on the VPN server. The available options are a few of the distributions supported by Lin Song's script. Every new distro added requires testing so I have only made available some of the most commonly used ones.
5. EC2 Instance Type: the size of the EC2 instance to use for the VPN server. The options available are some of the most commonly used and most regionally available instance types that I found. Not every instance type is supported in each AWS region. If you are creating an instance of a type unsupported by the region, the stack creation will fail. It is recommended to use regions that were not recently established (e.g. domestic US regions).

For the first three parameter items, each is defined with two properties (i.e. Type and Description), and for the latter two, two more properties are defined (i.e. AllowedValues and Default).

• Type specifies the value type for this parameter, which can be a String or other available value types.
• Description is what is shown to user as a description of the parameter during stack creation (by the stack creation wizard).
• AllowedValues is a list of strings that the user has to choose from, implying that this parameter takes in an option.
• Default is the default value for this parameter, pre-filled when the webpage is loaded.

The first two properties are REQUIRED for each parameter while the latter two properties are OPTIONAL.

Note that there is a way to validate the input of a parameter by using regular expressions (e.g. length). For more information, please consult CloudFormation documentation.

One vitally important parameter especially for a VPN server, the location of the server (AWS Region) is missing in the parameters section. That is because you don't choose the region as a parameter but by visiting the correct regional AWS Management Console Web pages. Or just pay attention to the top-right corner of the browser viewport, which tells you the AWS Region (e.g. Tokyo) you are currently at.

Outputs Section (Line 668-741)

The Outputs section defines the certain values generated by a successful deployment that are to be displayed under the "Outputs" tab once the deployment is successful.

In our case, the Outputs section displays the several following attributes of a created stack:

• The Public IP Address of the VPN Server: the IPv4 address for the user to connect to the server.
• VPN Username: VPN login username specified by user at stack launch.
• VPN Password: VPN login password specified by user at stack launch.
• VPN PSK (Pre-Shared Key): a set of characters that the user specfied at launch to be used as an additional authentication secret.
• EC2 Private Key ID: AWS EC2 doesn't allow plain-password login to the instances, and you have to use a generated private key for SSH login. This is the ID of the private key that can be retrieved through the use of the AWS CLI.
• EC2 Private Key Material: alternatively the private key material is also exported in plain-text format. In order to use it you have to properly format the data in a text editor.
• Next-Step Guide: provides a link to Github pages for further instructions on setting up the VPN client software.
• Warning for Debian Users: Debian GNU/Linux distribution, since version 10, has removed certain components required by IPsec/L2TP mode of VPN in its cloud-oriented system images, which is by default used by AWS EC2. So if you have chose Debian as the system at stack launch, you will not be able to use the VPN in IPsec/L2TP mode. To use the IPsec/L2TP mode, you have to choose a distro image of Debian that is the normal "desktop version" as opposed to the "cloud version".
• IKEv2 Credential Download Link: The bucket download link to the IKEv2 credential profiles. The various credentials are compressed into a .zip package that is encrypted with the password of the VPN server.

As you can see in the code, there are two available properties that can be defined under each output item:

1. Description: gives the user an idea of what the attribute is about.
2. Value: a set of string or a numerical value.

However, it's important to note that the attribute name of each item as defined in the template (e.g. 1VPNAddress) is also shown in the final "Outputs" tab.

You can easily use the reference intrinsic function to display the attributes from within the template, or you can set your own set of characters.

In the case of the last ouput item in the example template, an "Fn::Join" intrinsic function along with "Fn::GetAtt" produces the URL for the credential download link.

That wraps up the template file which I just elaborated on in detail. In order to use the template, you can follow the instructions on this page.

Install "Helper Scripts" on Your Computer

I mentioned before that in order for a template to be properly accepted by CloudFormation, the template has to be validated to have no formatting issues. That can be done when you use the stack creation wizard (i.e. bad templates get rejected) or when you use the graphical template editor.

However, AWS now provides a utility called cfn-lint to do the validation locally on your Dev computer. That program is among the many "Helper Scripts" that were designed to be used in companion with the CloudFormation service. For more information on how to install that on your computer, refer to this documentation.

Quick-Deploy Link and "Launch Stack" Icon

Once you've written a template yourself, you can let the stack creation wizard create a "Quick-deploy" link for you, after filling out the required parameters. However, to make that work, you have to use an already-uploaded template file in an S3 bucket. With the generated link, you can launch a stack of a pre-defined set of parameters with a few clicks. Such links can be shared amongst friends or displayed on Web pages.

As one of the "Best Practices", AWS has franchised a CloudFormation "Launch Stack" icon for you to embed in your HTML(s). They used to have a more 3D design but now it's more flat. Point the embedded icon to your "Quick-deploy" link or just the stack creation wizard.

Conclusion

AWS CloudFormation provides an All-in-One solution to create complex logics of cloud resources at a click. I personally used it only briefly since AWS is still relatively more costy than the indie VPS providers out there. However for experimental purposes, I do still use EC2 and other AWS services, sometimes through Cloudformation.

Readers should also remember that CloudFormation does not only support EC2, but almost all AWS services (e.g. RDS instance, VPC, Elastic IP and more). So there are inifinite possibilities at how you use it to create your own cloud architecture.

It's important to note that the various CloudFormation standards are constantly being revised. New services are being introduced and adopted to CloudFormation on a daily basis. At the inception of the example project, Key Pairs could not be created inline as a cloud resource because of security reasons, so back then the Key Pair had to be exclusively created sort of in a "hack" within an embedded Python script snippet. AWS changed that a few years after. So please let me know if you spotted some new changes that this article does not yet cover.

Further attempts were made by me to write up a similar deployment template for the well-known Seafile project, an open-source CMS storage solution. Despite being rejected by the Seafile team for increased maintenance burden , a few improvements were made in this template that does a similar job. Specifically a LogGroup and a LogStream of the service Cloudwatch were added through cloud-init such that the logs of the user-data script can be easily stored and analyzed on the web portal, a relief to the debugging process. Also since Seahub, Seafile's frontend, is a website, I added a Cloudfront CDN for that in the template. I gave up testing those two TO-DOs because I realized this would break the divine law of "Keep it simple, Keep it stupid", since nevertheless the Cloudfront feature has a certificate importation problem that could not be resolved. Despite none of those two were fully tested to work, they can serve as potential future improvements to this kind of deployment template.

References

AWS CloudFormation provides a collection of very useful documentation and tutorials on how to write your own template. Every single type of cloud resources is documented on the AWS's website. I have tried my best to link the various keywords in this article to the appropriate sources. For a complete reference of how CloudFormation works in detail, please consult the following location:

• AWS CloudFormation Documentation