In this tutorial, we'll look at how to deploy a Django app to AWS ECS with Terraform.
Dependencies:
- Django v3.2.9
- Docker v20.10.10
- Python v3.9.0
- Terraform v1.0.11
Contents
Objectives
By the end of this tutorial, you will be able to:
- Explain what Terraform is and how you can use it to write infrastructure as code
- Utilize the ECR Docker image registry to store images
- Create the required Terraform configuration for spinning up an ECS cluster
- Spin up AWS infrastructure via Terraform
- Deploy a Django app to a cluster of EC2 instances manged by an ECS Cluster
- Use Boto3 to update an ECS Service
- Configure AWS RDS for data persistence
- Create an HTTPS listener for an AWS load balancer
Terraform
Terraform is an infrastructure as code (IaC) tool used for building, changing, and versioning infrastructure through code. It uses a high-level declarative configuration language that lets you describe the desired state of your cloud or on-prem infrastructure for running an application. Think of it as the single source of truth for your infrastructure, which makes it easy to create, update, and delete resources safely and efficiently. After describing the end state of your infrastructure, Terraform generates a plan and then executes it -- e.g., provision and spin up the necessary infrastructure.
If you're new to Terraform, review the Introduction to Terraform article and go through the Getting Started guide.
Make sure you have Terraform installed before moving on:
$ terraform -v Terraform v1.0.11 on darwin_amd64
In this tutorial, using Terraform, we'll develop the high-level configuration files required to deploy a Django application to ECS. Once configured, we'll run a single command to set up the following AWS infrastructure:
- Networking:
- VPC
- Public and private subnets
- Routing tables
- Internet Gateway
- Key Pairs
- Security Groups
- Load Balancers, Listeners, and Target Groups
- IAM Roles and Policies
- ECS:
- Task Definition (with multiple containers)
- Cluster
- Service
- Launch Config and Auto Scaling Group
- RDS
- Health Checks and Logs
Amazon's Elastic Container Service (ECS) is a fully managed container orchestration platform that's used to manage and run containerized applications on clusters of EC2 instances.
If you're new to ECS, it's recommended to experiment with it in the web console first. Rather than configuring all the underlying network resources, IAM roles and policies, and logs manually, let ECS create these for you. You'll just need to set up ECS, a Load Balancer, Listener, Target Group, and RDS. Once you feel comfortable then move on to an infrastructure as code tool like Terraform.
Architecture diagram:
Project Setup
Let's start by setting up a quick Django project.
Create a new project directory along with a new Django project:
$ mkdir django-ecs-terraform && cd django-ecs-terraform
$ mkdir app && cd app
$ python3.10 -m venv env
$ source env/bin/activate
(env)$ pip install django==3.2.9
(env)$ django-admin startproject hello_django .
(env)$ python manage.py migrate
(env)$ python manage.py runserver
Navigate to http://localhost:8000/ to view the Django welcome screen. Kill the server once done, and then exit from the virtual environment. Go ahead and remove it as well. We now have a simple Django project to work with.
Add a requirements.txt file:
Django==3.2.9
gunicorn==20.1.0
Add a Dockerfile as well:
# pull official base image
FROM python:3.9.0-slim-buster
# set work directory
WORKDIR /usr/src/app
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install dependencies
RUN pip install --upgrade pip
COPY ./requirements.txt .
RUN pip install -r requirements.txt
# copy project
COPY . .
For testing purposes, set DEBUG
to True
and allow all hosts in the settings.py file:
DEBUG = True
ALLOWED_HOSTS = ['*']
Next, build and tag the image and spin up a new container:
$ docker build -t django-ecs .
$ docker run \
-p 8007:8000 \
--name django-test \
django-ecs \
gunicorn hello_django.wsgi:application --bind 0.0.0.0:8000
Ensure you can view the welcome screen again at http://localhost:8007/.
Stop and remove the container once done:
$ docker stop django-test
$ docker rm django-test
Add a .gitignore file to the project root:
__pycache__
.DS_Store
*.sqlite3
Your project structure should now look like this:
├── .gitignore
└── app
├── Dockerfile
├── hello_django
│ ├── __init__.py
│ ├── asgi.py
│ ├── settings.py
│ ├── urls.py
│ └── wsgi.py
├── manage.py
└── requirements.txt
For a more detailed look at how to containerize a Django app, review the Dockerizing Django with Postgres, Gunicorn, and Nginx blog post.
ECR
Before jumping into Terraform, let's push the Docker image to Elastic Container Registry (ECR), a private Docker image registry.
Navigate to the ECR console, and add a new repository called "django-app". Keep the tags mutable. For more on this, review the Image Tag Mutability guide.
Back in your terminal, build and tag the image again:
$ docker build -t <AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com/django-app:latest .
Be sure to replace
<AWS_ACCOUNT_ID>
with your AWS account ID.We'll be using the
us-west-1
region throughout this course. Feel free to change this if you'd like.
Authenticate the Docker CLI to use the ECR registry:
$ aws ecr get-login --region us-west-1 --no-include-email
This command will provide an auth token. Copy and paste the entire docker login
command to authenticate.
Push the image:
$ docker push <AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com/django-app:latest
Terraform Setup
Add a "terraform" folder to your project's root. We'll add each of our Terraform configuration files to this folder.
Next, add a new file to "terraform" called 01_provider.tf:
provider "aws" {
region = var.region
}
Here, we defined the AWS provider. You'll need to provide your AWS credentials in order to authenticate. Define them as environment variables:
$ export AWS_ACCESS_KEY_ID="YOUR_AWS_ACCESS_KEY_ID"
$ export AWS_SECRET_ACCESS_KEY="YOUR_AWS_SECRET_ACCESS_KEY"
We used a string interpolated value for region
, which will be read in from a variables.tf file. Go ahead and add this file to the "terraform" folder and add the following variable to it:
# core
variable "region" {
description = "The AWS region to create resources in."
default = "us-west-1"
}
Feel free to update the variables as you go through this tutorial based on your specific requirements.
Run terraform init
to create a new Terraform working directory and download the AWS provider.
With that we can start defining each piece of the AWS infrastructure.
AWS Resources
Next, let's configure the following AWS resources:
- Networking:
- VPC
- Public and private subnets
- Routing tables
- Internet Gateway
- Key Pairs
- Security Groups
- Load Balancers, Listeners, and Target Groups
- IAM Roles and Policies
- ECS:
- Task Definition (with multiple containers)
- Cluster
- Service
- Launch Config and Auto Scaling Group
- Health Checks and Logs
You can find each of the Terraform configuration files in the django-ecs-terraform repo on GitHub.
Network Resources
Let's define our network resources in a new file called 02_network.tf:
# Production VPC
resource "aws_vpc" "production-vpc" {
cidr_block = "10.0.0.0/16"
enable_dns_support = true
enable_dns_hostnames = true
}
# Public subnets
resource "aws_subnet" "public-subnet-1" {
cidr_block = var.public_subnet_1_cidr
vpc_id = aws_vpc.production-vpc.id
availability_zone = var.availability_zones[0]
}
resource "aws_subnet" "public-subnet-2" {
cidr_block = var.public_subnet_2_cidr
vpc_id = aws_vpc.production-vpc.id
availability_zone = var.availability_zones[1]
}
# Private subnets
resource "aws_subnet" "private-subnet-1" {
cidr_block = var.private_subnet_1_cidr
vpc_id = aws_vpc.production-vpc.id
availability_zone = var.availability_zones[0]
}
resource "aws_subnet" "private-subnet-2" {
cidr_block = var.private_subnet_2_cidr
vpc_id = aws_vpc.production-vpc.id
availability_zone = var.availability_zones[1]
}
# Route tables for the subnets
resource "aws_route_table" "public-route-table" {
vpc_id = aws_vpc.production-vpc.id
}
resource "aws_route_table" "private-route-table" {
vpc_id = aws_vpc.production-vpc.id
}
# Associate the newly created route tables to the subnets
resource "aws_route_table_association" "public-route-1-association" {
route_table_id = aws_route_table.public-route-table.id
subnet_id = aws_subnet.public-subnet-1.id
}
resource "aws_route_table_association" "public-route-2-association" {
route_table_id = aws_route_table.public-route-table.id
subnet_id = aws_subnet.public-subnet-2.id
}
resource "aws_route_table_association" "private-route-1-association" {
route_table_id = aws_route_table.private-route-table.id
subnet_id = aws_subnet.private-subnet-1.id
}
resource "aws_route_table_association" "private-route-2-association" {
route_table_id = aws_route_table.private-route-table.id
subnet_id = aws_subnet.private-subnet-2.id
}
# Elastic IP
resource "aws_eip" "elastic-ip-for-nat-gw" {
vpc = true
associate_with_private_ip = "10.0.0.5"
depends_on = [aws_internet_gateway.production-igw]
}
# NAT gateway
resource "aws_nat_gateway" "nat-gw" {
allocation_id = aws_eip.elastic-ip-for-nat-gw.id
subnet_id = aws_subnet.public-subnet-1.id
depends_on = [aws_eip.elastic-ip-for-nat-gw]
}
resource "aws_route" "nat-gw-route" {
route_table_id = aws_route_table.private-route-table.id
nat_gateway_id = aws_nat_gateway.nat-gw.id
destination_cidr_block = "0.0.0.0/0"
}
# Internet Gateway for the public subnet
resource "aws_internet_gateway" "production-igw" {
vpc_id = aws_vpc.production-vpc.id
}
# Route the public subnet traffic through the Internet Gateway
resource "aws_route" "public-internet-igw-route" {
route_table_id = aws_route_table.public-route-table.id
gateway_id = aws_internet_gateway.production-igw.id
destination_cidr_block = "0.0.0.0/0"
}
Here, we defined the following resources:
Add the following variables as well:
# networking
variable "public_subnet_1_cidr" {
description = "CIDR Block for Public Subnet 1"
default = "10.0.1.0/24"
}
variable "public_subnet_2_cidr" {
description = "CIDR Block for Public Subnet 2"
default = "10.0.2.0/24"
}
variable "private_subnet_1_cidr" {
description = "CIDR Block for Private Subnet 1"
default = "10.0.3.0/24"
}
variable "private_subnet_2_cidr" {
description = "CIDR Block for Private Subnet 2"
default = "10.0.4.0/24"
}
variable "availability_zones" {
description = "Availability zones"
type = list(string)
default = ["us-west-1b", "us-west-1c"]
}
Run terraform plan
to generate and show the execution plan based on the defined configuration.
Security Groups
Moving on, to protect the Django app and ECS cluster, let's configure Security Groups in a new file called 03_securitygroups.tf:
# ALB Security Group (Traffic Internet -> ALB)
resource "aws_security_group" "load-balancer" {
name = "load_balancer_security_group"
description = "Controls access to the ALB"
vpc_id = aws_vpc.production-vpc.id
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
# ECS Security group (traffic ALB -> ECS, ssh -> ECS)
resource "aws_security_group" "ecs" {
name = "ecs_security_group"
description = "Allows inbound access from the ALB only"
vpc_id = aws_vpc.production-vpc.id
ingress {
from_port = 0
to_port = 0
protocol = "-1"
security_groups = [aws_security_group.load-balancer.id]
}
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
Take note of the inbound rule on the Security Group associated with the ECS cluster for port 22. This is so we can SSH into an EC2 instance to run the initial DB migrations and add a super user.
Load Balancer
Next, let's configure an Application Load Balancer (ALB) along with the appropriate Target Group and Listener.
04_loadbalancer.tf:
# Production Load Balancer
resource "aws_lb" "production" {
name = "${var.ecs_cluster_name}-alb"
load_balancer_type = "application"
internal = false
security_groups = [aws_security_group.load-balancer.id]
subnets = [aws_subnet.public-subnet-1.id, aws_subnet.public-subnet-2.id]
}
# Target group
resource "aws_alb_target_group" "default-target-group" {
name = "${var.ecs_cluster_name}-tg"
port = 80
protocol = "HTTP"
vpc_id = aws_vpc.production-vpc.id
health_check {
path = var.health_check_path
port = "traffic-port"
healthy_threshold = 5
unhealthy_threshold = 2
timeout = 2
interval = 5
matcher = "200"
}
}
# Listener (redirects traffic from the load balancer to the target group)
resource "aws_alb_listener" "ecs-alb-http-listener" {
load_balancer_arn = aws_lb.production.id
port = "80"
protocol = "HTTP"
depends_on = [aws_alb_target_group.default-target-group]
default_action {
type = "forward"
target_group_arn = aws_alb_target_group.default-target-group.arn
}
}
Add the required variables:
# load balancer
variable "health_check_path" {
description = "Health check path for the default target group"
default = "/ping/"
}
# ecs
variable "ecs_cluster_name" {
description = "Name of the ECS cluster"
default = "production"
}
So, we configured our load balancer and listener to listen for HTTP requests on port 80. This is temporary. After we verify that our infrastructure and application are set up correctly, we'll update the load balancer to listen for HTTPS requests on port 443.
Take note of the path URL for the health check: /ping/
.
IAM Roles
05_iam.tf:
resource "aws_iam_role" "ecs-host-role" {
name = "ecs_host_role_prod"
assume_role_policy = file("policies/ecs-role.json")
}
resource "aws_iam_role_policy" "ecs-instance-role-policy" {
name = "ecs_instance_role_policy"
policy = file("policies/ecs-instance-role-policy.json")
role = aws_iam_role.ecs-host-role.id
}
resource "aws_iam_role" "ecs-service-role" {
name = "ecs_service_role_prod"
assume_role_policy = file("policies/ecs-role.json")
}
resource "aws_iam_role_policy" "ecs-service-role-policy" {
name = "ecs_service_role_policy"
policy = file("policies/ecs-service-role-policy.json")
role = aws_iam_role.ecs-service-role.id
}
resource "aws_iam_instance_profile" "ecs" {
name = "ecs_instance_profile_prod"
path = "/"
role = aws_iam_role.ecs-host-role.name
}
Add a new folder in "terraform" called "policies". Then, add the following role and policy definitions:
ecs-role.json:
{
"Version": "2008-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": [
"ecs.amazonaws.com",
"ec2.amazonaws.com"
]
},
"Effect": "Allow"
}
]
}
ecs-instance-role-policy.json:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecs:*",
"ec2:*",
"elasticloadbalancing:*",
"ecr:*",
"cloudwatch:*",
"s3:*",
"rds:*",
"logs:*"
],
"Resource": "*"
}
]
}
ecs-service-role-policy.json:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"elasticloadbalancing:Describe*",
"elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
"ec2:Describe*",
"ec2:AuthorizeSecurityGroupIngress",
"elasticloadbalancing:RegisterTargets",
"elasticloadbalancing:DeregisterTargets"
],
"Resource": [
"*"
]
}
]
}
Logs
06_logs.tf:
resource "aws_cloudwatch_log_group" "django-log-group" {
name = "/ecs/django-app"
retention_in_days = var.log_retention_in_days
}
resource "aws_cloudwatch_log_stream" "django-log-stream" {
name = "django-app-log-stream"
log_group_name = aws_cloudwatch_log_group.django-log-group.name
}
Add the variable:
# logs
variable "log_retention_in_days" {
default = 30
}
Key Pair
07_keypair.tf:
resource "aws_key_pair" "production" {
key_name = "${var.ecs_cluster_name}_key_pair"
public_key = file(var.ssh_pubkey_file)
}
Variable:
# key pair
variable "ssh_pubkey_file" {
description = "Path to an SSH public key"
default = "~/.ssh/id_rsa.pub"
}
ECS
Now, we can configure our ECS cluster.
08_ecs.tf:
resource "aws_ecs_cluster" "production" {
name = "${var.ecs_cluster_name}-cluster"
}
resource "aws_launch_configuration" "ecs" {
name = "${var.ecs_cluster_name}-cluster"
image_id = lookup(var.amis, var.region)
instance_type = var.instance_type
security_groups = [aws_security_group.ecs.id]
iam_instance_profile = aws_iam_instance_profile.ecs.name
key_name = aws_key_pair.production.key_name
associate_public_ip_address = true
user_data = "#!/bin/bash\necho ECS_CLUSTER='${var.ecs_cluster_name}-cluster' > /etc/ecs/ecs.config"
}
data "template_file" "app" {
template = file("templates/django_app.json.tpl")
vars = {
docker_image_url_django = var.docker_image_url_django
region = var.region
}
}
resource "aws_ecs_task_definition" "app" {
family = "django-app"
container_definitions = data.template_file.app.rendered
}
resource "aws_ecs_service" "production" {
name = "${var.ecs_cluster_name}-service"
cluster = aws_ecs_cluster.production.id
task_definition = aws_ecs_task_definition.app.arn
iam_role = aws_iam_role.ecs-service-role.arn
desired_count = var.app_count
depends_on = [aws_alb_listener.ecs-alb-http-listener, aws_iam_role_policy.ecs-service-role-policy]
load_balancer {
target_group_arn = aws_alb_target_group.default-target-group.arn
container_name = "django-app"
container_port = 8000
}
}
Take a look at the user_data
field in the aws_launch_configuration
. Put simply, user_data is a script that is run when a new EC2 instance is launched. In order for the ECS cluster to discover new EC2 instances, the cluster name needs to be added to the ECS_CLUSTER
environment variable within the /etc/ecs/ecs.config config file within the instance. In other words, the following script will run when a new instance is bootstrapped allowing it to be discovered by the cluster:
#!/bin/bash
echo ECS_CLUSTER='production-cluster' > /etc/ecs/ecs.config
For more on this discovery process, check out Amazon ECS Container Agent Configuration guide.
Add a "templates" folder to the "terraform" folder, and then add a new template file called django_app.json.tpl:
[
{
"name": "django-app",
"image": "${docker_image_url_django}",
"essential": true,
"cpu": 10,
"memory": 512,
"links": [],
"portMappings": [
{
"containerPort": 8000,
"hostPort": 0,
"protocol": "tcp"
}
],
"command": ["gunicorn", "-w", "3", "-b", ":8000", "hello_django.wsgi:application"],
"environment": [],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/django-app",
"awslogs-region": "${region}",
"awslogs-stream-prefix": "django-app-log-stream"
}
}
}
]
Here, we defined our container definition associated with the Django app.
Add the following variables as well:
# ecs
variable "ecs_cluster_name" {
description = "Name of the ECS cluster"
default = "production"
}
variable "amis" {
description = "Which AMI to spawn."
default = {
us-west-1 = "ami-0bd3976c0dbacc605"
}
}
variable "instance_type" {
default = "t2.micro"
}
variable "docker_image_url_django" {
description = "Docker image to run in the ECS cluster"
default = "<AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com/django-app:latest"
}
variable "app_count" {
description = "Number of Docker containers to run"
default = 2
}
Again, be sure to replace
<AWS_ACCOUNT_ID>
with your AWS account ID.Refer to the Linux Amazon ECS-optimized AMIs guide to find a list of AMIs with Docker pre-installed.
Since we added the Template provider, run terraform init
again to download the new provider.
Auto Scaling
09_auto_scaling.tf:
resource "aws_autoscaling_group" "ecs-cluster" {
name = "${var.ecs_cluster_name}_auto_scaling_group"
min_size = var.autoscale_min
max_size = var.autoscale_max
desired_capacity = var.autoscale_desired
health_check_type = "EC2"
launch_configuration = aws_launch_configuration.ecs.name
vpc_zone_identifier = [aws_subnet.private-subnet-1.id, aws_subnet.private-subnet-2.id]
}
New variables:
# auto scaling
variable "autoscale_min" {
description = "Minimum autoscale (number of EC2)"
default = "1"
}
variable "autoscale_max" {
description = "Maximum autoscale (number of EC2)"
default = "10"
}
variable "autoscale_desired" {
description = "Desired autoscale (number of EC2)"
default = "4"
}
Test
outputs.tf:
output "alb_hostname" {
value = aws_lb.production.dns_name
}
Here, we configured an outputs.tf file along with an output value called alb_hostname
. After we execute the Terraform plan, to spin up the AWS infrastructure, the load balancer's DNS name will be outputted to the terminal.
Ready?!? View then execute the plan:
$ terraform plan
$ terraform apply
You should see the health check failing with a 404:
service production-service (instance i-013f1192da079b0bf) (port 49153)
is unhealthy in target-group production-tg due to
(reason Health checks failed with these codes: [404])
This is expected since we haven't set up a /ping/ handler in the app yet.
Django Health Check
Add the following middleware to app/hello_django/middleware.py:
from django.http import HttpResponse
from django.utils.deprecation import MiddlewareMixin
class HealthCheckMiddleware(MiddlewareMixin):
def process_request(self, request):
if request.META['PATH_INFO'] == '/ping/':
return HttpResponse('pong!')
Add the class to the middleware config in settings.py:
MIDDLEWARE = [
'hello_django.middleware.HealthCheckMiddleware', # new
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
]
This middleware is used to handle requests to the /ping/ URL before ALLOWED_HOSTS
is checked. Why is this necessary?
The health check request comes from the EC2 instance. Since we don't know the private IP beforehand, this will ensure that the /ping/ route always returns a successful response even after we restrict ALLOWED_HOSTS
.
It's worth noting that you could toss Nginx in front of Gunicorn and handle the health check in the Nginx config like so:
location /ping/ { access_log off; return 200; }
To test locally, build the new image and then spin up the container:
$ docker build -t django-ecs .
$ docker run \
-p 8007:8000 \
--name django-test \
django-ecs \
gunicorn hello_django.wsgi:application --bind 0.0.0.0:8000
Make sure http://localhost:8007/ping/ works as expected:
pong!
Stop and remove the container once done:
$ docker stop django-test
$ docker rm django-test
Next, update ECR:
$ docker build -t <AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com/django-app:latest .
$ docker push <AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com/django-app:latest
Let's add a quick script to update the Task Definition and Service so that the new Tasks use the new image that we just pushed.
Create a "deploy" folder in the project root. Then, add an update-ecs.py file to that newly created folder:
import boto3
import click
def get_current_task_definition(client, cluster, service):
response = client.describe_services(cluster=cluster, services=[service])
current_task_arn = response["services"][0]["taskDefinition"]
response = client.describe_task_definition(taskDefinition=current_task_arn)
return response
@click.command()
@click.option("--cluster", help="Name of the ECS cluster", required=True)
@click.option("--service", help="Name of the ECS service", required=True)
def deploy(cluster, service):
client = boto3.client("ecs")
response = get_current_task_definition(client, cluster, service)
container_definition = response["taskDefinition"]["containerDefinitions"][0].copy()
response = client.register_task_definition(
family=response["taskDefinition"]["family"],
volumes=response["taskDefinition"]["volumes"],
containerDefinitions=[container_definition],
)
new_task_arn = response["taskDefinition"]["taskDefinitionArn"]
response = client.update_service(
cluster=cluster, service=service, taskDefinition=new_task_arn,
)
if __name__ == "__main__":
deploy()
So, this script will create a new revision of the Task Definition and then update the Service so it uses the revised Task Definition.
Create and activate a new virtual environment. Then, install Boto3 and Click:
$ pip install boto3 click
Add your AWS credentials along with the default region:
$ export AWS_ACCESS_KEY_ID="YOUR_AWS_ACCESS_KEY_ID"
$ export AWS_SECRET_ACCESS_KEY="YOUR_AWS_SECRET_ACCESS_KEY"
$ export AWS_DEFAULT_REGION="us-west-1"
Run the script like so:
$ python update-ecs.py --cluster=production-cluster --service=production-service
The Service should start two new Tasks based on the revised Task Definition and register them with the associated Target Group. This time the health checks should pass. You should now be able to view your application using the DNS hostname that was outputted to your terminal:
Outputs:
alb_hostname = production-alb-1008464563.us-west-1.elb.amazonaws.com
RDS
Next, let's configure RDS so we can use Postgres for our production database.
Add a new Security Group to 03_securitygroups.tf to ensure that only traffic from your ECS instance can talk to the database:
# RDS Security Group (traffic ECS -> RDS)
resource "aws_security_group" "rds" {
name = "rds-security-group"
description = "Allows inbound access from ECS only"
vpc_id = aws_vpc.production-vpc.id
ingress {
protocol = "tcp"
from_port = "5432"
to_port = "5432"
security_groups = [aws_security_group.ecs.id]
}
egress {
protocol = "-1"
from_port = 0
to_port = 0
cidr_blocks = ["0.0.0.0/0"]
}
}
Next, add a new file called 10_rds.tf for setting up the database itself:
resource "aws_db_subnet_group" "production" {
name = "main"
subnet_ids = [aws_subnet.private-subnet-1.id, aws_subnet.private-subnet-2.id]
}
resource "aws_db_instance" "production" {
identifier = "production"
name = var.rds_db_name
username = var.rds_username
password = var.rds_password
port = "5432"
engine = "postgres"
engine_version = "12.3"
instance_class = var.rds_instance_class
allocated_storage = "20"
storage_encrypted = false
vpc_security_group_ids = [aws_security_group.rds.id]
db_subnet_group_name = aws_db_subnet_group.production.name
multi_az = false
storage_type = "gp2"
publicly_accessible = false
backup_retention_period = 7
skip_final_snapshot = true
}
Variables:
# rds
variable "rds_db_name" {
description = "RDS database name"
default = "mydb"
}
variable "rds_username" {
description = "RDS database username"
default = "foo"
}
variable "rds_password" {
description = "RDS database password"
}
variable "rds_instance_class" {
description = "RDS instance type"
default = "db.t2.micro"
}
Note that we left off the default value for the password. More on this in a bit.
Since we'll need to know the address for the instance in our Django app, add a depends_on
argument to the aws_ecs_task_definition
in 08_ecs.tf:
resource "aws_ecs_task_definition" "app" {
family = "django-app"
container_definitions = data.template_file.app.rendered
depends_on = [aws_db_instance.production]
}
Next, we need to update the DATABASES
config in settings.py:
if 'RDS_DB_NAME' in os.environ:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': os.environ['RDS_DB_NAME'],
'USER': os.environ['RDS_USERNAME'],
'PASSWORD': os.environ['RDS_PASSWORD'],
'HOST': os.environ['RDS_HOSTNAME'],
'PORT': os.environ['RDS_PORT'],
}
}
else:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}
Add the import:
import os
Update the environment
section in the django_app.json.tpl template:
"environment": [
{
"name": "RDS_DB_NAME",
"value": "${rds_db_name}"
},
{
"name": "RDS_USERNAME",
"value": "${rds_username}"
},
{
"name": "RDS_PASSWORD",
"value": "${rds_password}"
},
{
"name": "RDS_HOSTNAME",
"value": "${rds_hostname}"
},
{
"name": "RDS_PORT",
"value": "5432"
}
],
Update the vars passed to the template in 08_ecs.tf:
data "template_file" "app" {
template = file("templates/django_app.json.tpl")
vars = {
docker_image_url_django = var.docker_image_url_django
region = var.region
rds_db_name = var.rds_db_name
rds_username = var.rds_username
rds_password = var.rds_password
rds_hostname = aws_db_instance.production.address
}
}
Add Psycopg2 to the requirements file:
Django==3.2.9
gunicorn==20.1.0
psycopg2-binary==2.9.2
Update the Dockerfile to install the appropriate packages required for Psycopg2:
# pull official base image
FROM python:3.9.0-slim-buster
# set work directory
WORKDIR /usr/src/app
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install psycopg2 dependencies
RUN apt-get update \
&& apt-get -y install gcc postgresql \
&& apt-get clean
# install dependencies
RUN pip install --upgrade pip
COPY ./requirements.txt .
RUN pip install -r requirements.txt
# copy project
COPY . .
Alright. Build the Docker image and push it up to ECR. Then, to update the ECS Task Definition, create the RDS resources, and update the Service, run:
$ terraform apply
Since we didn't set a default for the password, you'll be prompted to enter one:
var.rds_password
RDS database password
Enter a value:
Rather than having to pass a value in each time, you could set an environment variable like so:
$ export TF_VAR_rds_password=foobarbaz
$ terraform apply
Keep in mind that this approach, of using environment variables, keeps sensitive variables out of the .tf files, but they are still stored in the terraform.tfstate file in plain text. So, be sure to keep this file out of version control. Since keeping it out of version control doesn't work if other people on your team need access to it, look to either encrypting the secrets or using a secret store like Vault or AWS Secrets Manager.
After the new Tasks are registered with the Target Group, SSH into an EC2 instance where one of the Tasks is running:
$ ssh ec2-user@<instance-ip>
# ssh [email protected]
Grab the container ID via docker ps
, and use it to apply the migrations:
$ docker exec -it <container-id> python manage.py migrate
# docker exec -it 73284cda8a87 python manage.py migrate
You may want to create a super user as well. Once done, exit from the SSH session. You'll probably want to remove the following inbound rule from the ECS Security group if you don't need SSH access any longer:
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
Domain and SSL Certificate
Assuming you've generated and validated a new SSL certificate from AWS Certificate Manager, add the certificate's ARN to your variables:
# domain
variable "certificate_arn" {
description = "AWS Certificate Manager ARN for validated domain"
default = "ADD YOUR ARN HERE"
}
Update the default listener associated with the load balancer in 04_loadbalancer.tf so that it listens for HTTPS requests on port 443 (as opposed to HTTP on port 80):
# Listener (redirects traffic from the load balancer to the target group)
resource "aws_alb_listener" "ecs-alb-http-listener" {
load_balancer_arn = aws_lb.production.id
port = "443"
protocol = "HTTPS"
ssl_policy = "ELBSecurityPolicy-2016-08"
certificate_arn = var.certificate_arn
depends_on = [aws_alb_target_group.default-target-group]
default_action {
type = "forward"
target_group_arn = aws_alb_target_group.default-target-group.arn
}
}
Apply the changes:
$ terraform apply
Make sure to point your domain at the load balancer using a CNAME record. Make sure you can view your application.
Nginx
Next, let's add Nginx into the mix to handle requests for static files appropriately.
In the project root, create the following files and folders:
└── nginx
├── Dockerfile
└── nginx.conf
Dockerfile:
FROM nginx:1.19.0-alpine
RUN rm /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/conf.d
EXPOSE 80
nginx.conf:
upstream hello_django {
server django-app:8000;
}
server {
listen 80;
location / {
proxy_pass http://hello_django;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
}
Here, we set up a single location block, routing all traffic to the Django app. We'll set up a new location block for static files in the next section.
Create a new repo in ECR called "nginx", and then build and push the new image:
$ docker build -t <AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com/nginx:latest .
$ docker push <AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com/nginx:latest
Add the following variable to the ECS section of the variables file:
variable "docker_image_url_nginx" {
description = "Docker image to run in the ECS cluster"
default = "<AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com/nginx:latest"
}
Add the new container definition to the django_app.json.tpl template:
[
{
"name": "django-app",
"image": "${docker_image_url_django}",
"essential": true,
"cpu": 10,
"memory": 512,
"links": [],
"portMappings": [
{
"containerPort": 8000,
"hostPort": 0,
"protocol": "tcp"
}
],
"command": ["gunicorn", "-w", "3", "-b", ":8000", "hello_django.wsgi:application"],
"environment": [
{
"name": "RDS_DB_NAME",
"value": "${rds_db_name}"
},
{
"name": "RDS_USERNAME",
"value": "${rds_username}"
},
{
"name": "RDS_PASSWORD",
"value": "${rds_password}"
},
{
"name": "RDS_HOSTNAME",
"value": "${rds_hostname}"
},
{
"name": "RDS_PORT",
"value": "5432"
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/django-app",
"awslogs-region": "${region}",
"awslogs-stream-prefix": "django-app-log-stream"
}
}
},
{
"name": "nginx",
"image": "${docker_image_url_nginx}",
"essential": true,
"cpu": 10,
"memory": 128,
"links": ["django-app"],
"portMappings": [
{
"containerPort": 80,
"hostPort": 0,
"protocol": "tcp"
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/nginx",
"awslogs-region": "${region}",
"awslogs-stream-prefix": "nginx-log-stream"
}
}
}
]
Pass the variable to the template in 08_ecs.tf:
data "template_file" "app" {
template = file("templates/django_app.json.tpl")
vars = {
docker_image_url_django = var.docker_image_url_django
docker_image_url_nginx = var.docker_image_url_nginx
region = var.region
rds_db_name = var.rds_db_name
rds_username = var.rds_username
rds_password = var.rds_password
rds_hostname = aws_db_instance.production.address
}
}
Add the new logs to 06_logs.tf:
resource "aws_cloudwatch_log_group" "nginx-log-group" {
name = "/ecs/nginx"
retention_in_days = var.log_retention_in_days
}
resource "aws_cloudwatch_log_stream" "nginx-log-stream" {
name = "nginx-log-stream"
log_group_name = aws_cloudwatch_log_group.nginx-log-group.name
}
Update the Service so it points to the nginx
container instead of django-app
:
resource "aws_ecs_service" "production" {
name = "${var.ecs_cluster_name}-service"
cluster = aws_ecs_cluster.production.id
task_definition = aws_ecs_task_definition.app.arn
iam_role = aws_iam_role.ecs-service-role.arn
desired_count = var.app_count
depends_on = [aws_alb_listener.ecs-alb-http-listener, aws_iam_role_policy.ecs-service-role-policy]
load_balancer {
target_group_arn = aws_alb_target_group.default-target-group.arn
container_name = "nginx"
container_port = 80
}
}
Apply the changes:
$ terraform apply
Make sure the app can still be accessed from the browser.
Now that we're dealing with two containers, let's update the deploy function to handle multiple container definitions in update-ecs.py:
@click.command()
@click.option("--cluster", help="Name of the ECS cluster", required=True)
@click.option("--service", help="Name of the ECS service", required=True)
def deploy(cluster, service):
client = boto3.client("ecs")
container_definitions = []
response = get_current_task_definition(client, cluster, service)
for container_definition in response["taskDefinition"]["containerDefinitions"]:
new_def = container_definition.copy()
container_definitions.append(new_def)
response = client.register_task_definition(
family=response["taskDefinition"]["family"],
volumes=response["taskDefinition"]["volumes"],
containerDefinitions=container_definitions,
)
new_task_arn = response["taskDefinition"]["taskDefinitionArn"]
response = client.update_service(
cluster=cluster, service=service, taskDefinition=new_task_arn,
)
Static Files
Set the STATIC_ROOT
in your settings.py file:
STATIC_URL = '/staticfiles/'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
Also, turn off debug mode:
DEBUG = False
Update the Dockerfile so that it runs the collectstatic
command at the end:
# pull official base image
FROM python:3.9.0-slim-buster
# set work directory
WORKDIR /usr/src/app
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install psycopg2 dependencies
RUN apt-get update \
&& apt-get -y install gcc postgresql \
&& apt-get clean
# install dependencies
RUN pip install --upgrade pip
COPY ./requirements.txt .
RUN pip install -r requirements.txt
# copy project
COPY . .
# collect static files
RUN python manage.py collectstatic --no-input
Next, let's add a shared volume to the Task Definition and update the Nginx conf file.
Add the new location block to nginx.conf:
upstream hello_django {
server django-app:8000;
}
server {
listen 80;
location /staticfiles/ {
alias /usr/src/app/staticfiles/;
}
location / {
proxy_pass http://hello_django;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
}
Add the volume to aws_ecs_task_definition
in 08_ecs.tf:
resource "aws_ecs_task_definition" "app" {
family = "django-app"
container_definitions = data.template_file.app.rendered
depends_on = [aws_db_instance.production]
volume {
name = "static_volume"
host_path = "/usr/src/app/staticfiles/"
}
}
Add the volume to the container definitions in the django_app.json.tpl template:
[
{
"name": "django-app",
"image": "${docker_image_url_django}",
"essential": true,
"cpu": 10,
"memory": 512,
"links": [],
"portMappings": [
{
"containerPort": 8000,
"hostPort": 0,
"protocol": "tcp"
}
],
"command": ["gunicorn", "-w", "3", "-b", ":8000", "hello_django.wsgi:application"],
"environment": [
{
"name": "RDS_DB_NAME",
"value": "${rds_db_name}"
},
{
"name": "RDS_USERNAME",
"value": "${rds_username}"
},
{
"name": "RDS_PASSWORD",
"value": "${rds_password}"
},
{
"name": "RDS_HOSTNAME",
"value": "${rds_hostname}"
},
{
"name": "RDS_PORT",
"value": "5432"
}
],
"mountPoints": [
{
"containerPath": "/usr/src/app/staticfiles",
"sourceVolume": "static_volume"
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/django-app",
"awslogs-region": "${region}",
"awslogs-stream-prefix": "django-app-log-stream"
}
}
},
{
"name": "nginx",
"image": "${docker_image_url_nginx}",
"essential": true,
"cpu": 10,
"memory": 128,
"links": ["django-app"],
"portMappings": [
{
"containerPort": 80,
"hostPort": 0,
"protocol": "tcp"
}
],
"mountPoints": [
{
"containerPath": "/usr/src/app/staticfiles",
"sourceVolume": "static_volume"
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/nginx",
"awslogs-region": "${region}",
"awslogs-stream-prefix": "nginx-log-stream"
}
}
}
]
Now, each container will share a directory named "staticfiles".
Build the new images and push them up to ECR:
$ docker build -t <AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com/django-app:latest .
$ docker push <AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com/django-app:latest
$ docker build -t <AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com/nginx:latest .
$ docker push <AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com/nginx:latest
Apply the changes:
$ terraform apply
Static files should now load correctly.
Allowed Hosts
Finally, let's lock down our application for production:
ALLOWED_HOSTS = os.getenv('ALLOWED_HOSTS', '').split()
Add the ALLOWED_HOSTS
environment variable to the container definition:
"environment": [
{
"name": "RDS_DB_NAME",
"value": "${rds_db_name}"
},
{
"name": "RDS_USERNAME",
"value": "${rds_username}"
},
{
"name": "RDS_PASSWORD",
"value": "${rds_password}"
},
{
"name": "RDS_HOSTNAME",
"value": "${rds_hostname}"
},
{
"name": "RDS_PORT",
"value": "5432"
},
{
"name": "ALLOWED_HOSTS",
"value": "${allowed_hosts}"
}
],
Pass the variable to the template in 08_ecs.tf:
data "template_file" "app" {
template = file("templates/django_app.json.tpl")
vars = {
docker_image_url_django = var.docker_image_url_django
docker_image_url_nginx = var.docker_image_url_nginx
region = var.region
rds_db_name = var.rds_db_name
rds_username = var.rds_username
rds_password = var.rds_password
rds_hostname = aws_db_instance.production.address
allowed_hosts = var.allowed_hosts
}
}
Add the variable to the ECS section of the variables file, making sure to add your domain name:
variable "allowed_hosts" {
description = "Domain name for allowed hosts"
default = "YOUR DOMAIN NAME"
}
Build the new image and push it up to ECR:
$ docker build -t <AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com/django-app:latest .
$ docker push <AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com/django-app:latest
Apply:
$ terraform apply
Test it out one last time.
Bring the infrastructure down once done:
$ terraform destroy
Conclusion
This tutorial looked at how to use Terraform to spin up the required AWS infrastructure for running a Django app on ECS.
While the initial configuration is complex, large teams with complicated infrastructure requirements, will benefit from Terraform. It provides a readable, central source of truth for your infrastructure, which should result in quicker feedback cycles.
Next steps:
- Configure CloudWatch alarms for scaling containers out and in.
- Store user-uploaded files on Amazon S3
- Set up multi-stage Docker builds and use a non-root user in the Docker container
- Rather than routing traffic on port 80 to Nginx, add a listener for port 443.
- Run through the entire Django deployment checklist.
- If you're planing to host multiple applications, you may want to move any "common" resources shared across the applications to a separate Terraform stack so that if you're regularly making modifications, your core AWS services will not be not affected.
- Take a look at ECS Fargate. This can help simply your infrastructure since you don't have to manage the actual cluster.
You can find the final code in the django-ecs-terraform repo.