Awesome
terraform-ecs-fargate-airflow
A Terraform template for provisioning Apache Airflow workflows on AWS ECS Fargate.
This template assumes you already have an airflow cluster up and running in AWS with the FargateEcsOperator operator installed (with fargate support). The template will output a deploy-airflow.sh script that does the following:
- builds your code into a container image (assumes you have a Dockerfile in the root of your project)
- pushes your container image to ECR
- copies your DAG (referencing your container) to an S3 location to be deployed (assumes you have a process for copying the DAGs from your S3 location into your airflow directory)
The templates are designed to be customized. The optional components can be removed by simply deleting the .tf file.
Components
base
These components are shared by all environments.
| Name | Description | Optional |
|---|---|---|
| main.tf | AWS provider, output | |
| state.tf | S3 bucket backend for storing Terraform remote state | |
| ecr.tf | ECR repository for application (all environments share) |
env/dev
These components are for a specific environment. There should be a corresponding directory for each environment that is needed.
| Name | Description | Optional |
|---|---|---|
| main.tf | Terrform remote state, AWS provider, output | |
| ecs.tf | ECS Cluster, Service, Task Definition, ecsTaskExecutionRole, CloudWatch Log Group | |
| nsg.tf | NSG for Task | |
| role-airflow.tf | Attaches policies to the airflow role that allow it to integrate with fargate | |
| airflow.tf | Generates the deploy-airflow.sh deployment script | |
| dag.tf | Generates a default DAG py file | |
| dashboard.tf | CloudWatch dashboard: CPU, memory, and HTTP-related metrics | |
| role.tf | Application Role for container | |
| cicd.tf | IAM user that can be used by CI/CD systems | Yes |
| secretsmanager.tf | Add a base secret to Secretsmanager | Yes |
| ecs-event-stream.tf | Add an ECS event log dashboard | Yes |
Usage
Typically, the base Terraform will only need to be run once, and then should only need changes very infrequently. After the base is built, each environment can be built.
# Move into the base directory
$ cd base
# Sets up Terraform to run
$ terraform init
# Executes the Terraform run
$ terraform apply
# Now, move into the dev environment
$ cd ../env/dev
# Sets up Terraform to run
$ terraform init
# Executes the Terraform run
$ terraform apply
Important (after initial terraform apply)
The generated base .tfstate is not stored in the remote state S3 bucket. Ensure the base .tfstate is checked into your infrastructure repo. The default Terraform .gitignore generated by GitHub will ignore all .tfstate files; you'll need to modify this!
fargate-create
Alternatively you can use the fargate-create CLI to scaffold new projects based on this template.
install
curl -s get-fargate-create.turnerlabs.io | sh
create an input vars file (terraform.tfvars)
# app/env to scaffold
app = "my-app"
environment = "dev"
schedule_expression = "@hourly"
airflow_dag_s3_bucket = "s3://my-dags/"
airflow_role = "airflow_role"
region = "us-east-1"
aws_profile = "default"
saml_role = "admin"
vpc = "vpc-123"
private_subnets = "subnet-123,subnet-456"
public_subnets = "subnet-789,subnet-012"
tags = {
application = "my-app"
environment = "dev"
team = "my-team"
customer = "my-customer"
contact-email = "me@example.com"
}
$ fargate-create -f terraform.tfvars -t git@github.com:turnerlabs/terraform-ecs-fargate-airflow
Deploy DAG to airflow
cd iac/env/dev
./deploy-airflow.sh