Awesome

This Terraform deploys a stateless containerised sshd bastion service on AWS with IAM based authentication:

This module requires Terraform >/=1.3.0 Older versions were previously supported going back to Terraform 0.11.x with module version to ~> v4.0

N.B. If you are using a newer version of this module when you have an older version deployed, please review the changelog!

Overview

This plan provides socket-activated sshd-containers with one container instantiated per connection and destroyed on connection termination or else after 12 hours- to deter things like reverse tunnels etc. The host assumes an IAM role, inherited by the containers, allowing it to query IAM users and request their ssh public keys lodged with AWS.

It is essential to limit incoming service traffic to whitelisted ports. If you do not then internet background noise will exhaust the host resources and/ or lead to rate limiting from amazon on the IAM identity calls- resulting in denial of service.

It is possible to replace the components in userdata and the base AMI with components of your own choosing. The following describes deployment with all sections as provided by module defaults.

The actual call for public keys is made with a GO binary, which is built during host instance initial launch and made available via shared volume in the docker image. In use the Docker container queries AWS for users with ssh keys at runtime, creates local linux user accounts for them and handles their login. The users who may access the bastion service may be restricted to membership of a defined AWS IAM group which is not set up or managed by this plan. When the connection is closed the container exits. This means that users log in as themselves and manage their own ssh keys using the AWS web console or CLI. For any given session they will arrive in a vanilla Ubuntu container with passwordless sudo and can install whatever applications and frameworks might be required for that session. Because the IAM identity checking and user account population is done at container run time and the containers are called on demand, there is no delay between creating an account with a public ssh key on AWS and being able to access the bastion. If users have more than one ssh public key then their account will be set up so that any of them may be used- AWS allows up to 5 keys per user. Aside from the resources provided by AWS and remote public repositories this plan is entirely self contained. There is no reliance on registries, build chains etc. Although it is possible to deploy this module as is without doing so, the standard/default configuration requires either a nat gateway on the VPC or a public IP address, in order for the instance to retrieve packages and dependencies to build the binary and containerised service

This plan is also published on the Terraform Community Module Registry

You may find it more convenient to call it in your plan directly from the Terraform Community Module Registry

With thanks and acknowledgments to all contributors!

Quick start

Ivan Mesic has kindly contributed an example use of this module creating a VPC and a bastion instance within it - see /examples/full-with-public-ip

examples/custom-outbound-security-group is for more specialist use cases demonstrating how to run the service on a different port with an external supplied security group for external ingress and egress. This would not be necessary for most users.

Custom sections:

You can specify a custom base AMI to use for the service host if you wish with var.custom_ami_id.

Userdata has been divided into sections which are individually applicable. Each is a HEREDOC and may be excluded by assigning any non-empty value to the relevant section variable. The value given is used simply for a logic test and not passed into userdata. If you ignore all of these variables then historic/ default behaviour continues and everything is built on the host instance on first boot (allow 3 minutes on t2.medium).

The variables for these sections are:

• custom_ssh_populate - any value excludes default ssh_populate script used on container launch from userdata

• custom_authorized_keys_command - any value excludes default Go binary iam-authorized-keys built from source from userdata

• custom_docker_setup - any value excludes default docker installation and container build from userdata

• custom_systemd - any value excludes default systemd and hostname change from userdata

• extra_user_data* - (optional, several variables) you may supply your own user data here - appended following above sections

If you exclude any section then you must replace it with equivalent functionality, either in your base AMI or extra_user_data* for a working service. Especially if you are not replacing all sections then be mindful that the systemd service expects docker to be installed and to be able to call the docker container as sshd_worker. The service container in turn references the ssh_populate script which calls iam-authorized-keys from a specific location.

You can supply a list of one or more security groups to attach to the host instance launch configuration within the module if you wish. This can be supplied together with or instead of a whitelisted range of CIDR blocks. Starting with release 8.1 it is possible to use this to also passlist egress ports exclusively if var.custom_outbound_security_group = true (default false). It may be useful in an enterprise setting to have security groups with rules managed separately from the bastion plan but of course if you do not assign either a suitable security group or whitelist then you may not be able to reach the service!

Starting with release 8.1 it is possible to assign a custom port for the containerised ssh bastion service, e.g. port 443. This may be useful for advanced users and must match security group ingress and egress- see examples/custom-outbound-security-group

Load Balancer health check port may be optionally set to either the containerised service port (by default port 22) or port 2222 (EC2 host sshd). Port 2222 is the default. If you are deploying a large number of bastion instances, all of them checking into the same parent account for IAM queries in response to load balancer health checks on port 22 causes IAM rate limiting from AWS. Using the modified EC2 host sshd of port 2222 avoids this issue, is recommended for larger deployments and is now default. The host sshd is set to port 2222 as part of the service setup so this healthcheck is not entirely invalid. Security group rules, target groups and load balancer listeners are conditionally created to support any combination of access/healthcheck on port 2222 or not.

Ability to assume a role in another account

The ability to assume a role to source IAM users from another account has been integrated with conditional logic. If you supply the ARN for a role for the bastion service to assume (typically in another account) ${var.assume_role_arn} then this plan will create an instance profile, role and policy along with each bastion to make use of it. A matching sample policy and trust relationship is given as an output from the plan to assist with application in the other account. If you do not supply this arn then this plan presumes IAM lookups in the same account and creates an appropriate instance profile, role and policies for each bastion in the same AWS account. 'Each bastion' here refers to a combination of environment, AWS account, AWS region and VPCID determined by deployment. This is a high availability service, but if you are making more than one independent deployment using this same module within such a combination then you can specify "service_name" to avoid resource collision.

If you are seeking a solution for ECS hosts then you are recommended to the Widdix project. This offers IAM authentication for local users with a range of features suitable for a long-lived stateful host built as an AMI or with configuration management tools.

Service deployed by this plan (presuming default userdata)

This plan creates a network load balancer and autoscaling group with an optional DNS entry and an optional public IP for the service.

Default, partial and complete customisation of hostname

You can overwrite the suggested hostname entirely with

var.bastion_host_name

You can instead customise just the last part of the hostname if you like with

bastion_vpc_name

By default this is the vpc ID via the magic default value of 'vpc_id' with the format

name = "${var.environment_name}-${data.aws_region.current.name}-${var.vpc}-bastion-service.${var.dns_domain}"

e.g.

module default:

dev-ap-northeast-1-vpc-1a23b456d7890-bastion-service.yourdomain.com

but you can pass a custom string, or an empty value to omit this. e.g.

bastion_vpc_name = "compute"

gives

dev-ap-northeast-1-compute-bastion-service.yourdomain.com

and

bastion_vpc_name = ""

gives

dev-ap-northeast-1-bastion-service.yourdomain.com

In any event this ensures a consistent and obvious naming format for each combination of AWS account and region that does not collide if multiple vpcs are deployed per region.

The container shell prompt is set similarly but with a systemd incremented counter, e.g. for

aws_user

you might see

aws_user@demo-eu-west-1-vpc-06ef3237ac609eb4b-0:~$ 

and a subsequent container might have

aws_user@demo-eu-west-1-vpc-06ef3237ac609eb4b-1:~$ 

In the case that

bastion_vpc_name = ""

the service container shell prompt is set similar to

you@dev-ap-northeast-1_3

In use

It is considered normal to see very highly incremented counters if the load balancer health checks are conducted on the service port. It is essential to limit incoming service traffic to whitelisted ports. If you do not then internet background noise will exhaust the host resources and/ or lead to rate limiting from amazon on the IAM identity calls- resulting in denial of service.

The host is set to run the latest patch release at deployment of Debian Bullseye - unless you specify a custom AMI. Debian was chosen originally because the socket activation requires systemd but Ubuntu 16.04 did not automatically set up DHCP for additional elastic network interfaces (see version 1 series).

The host sshd is available on port 2222 and uses standard ec2 ssh keying. The default login username for Debian AMI's is 'admin'. If you do not whitelist any access to this port directly from the outside world (plan default) then it may be convenient to access from a container during development, e.g. with

sudo apt install -y curl; ssh -p2222 admin@`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`

Make sure that your agent forwarding is active before attempting this!

It is advised to deploy to production without ec2 keys to increase security.

If you are interested in specifying your own AMI then be aware that there are many subtle differences in systemd implementations between different versions, e.g. it is not possible to use Amazon Linux 2 because we need (from Systemd):

• RunTimeMaxSec to limit the service container lifetime. This was introduced with Systemd version 229 (feb 2016) whereas Amazon Linux 2 uses version 219 (Feb 2015) This is a critical requirement.

IAM user names and Linux user names

with thanks to michaelwittig and the Widdix project

IAM user names may be up to 64 characters long.

Linux user names may only be up to 32 characters long.

Allowed characters for IAM user names are:

alphanumeric, including the following common characters: plus (+), equal (=), comma (,), period (.), at (@), underscore (_), and hyphen (-).

Allowed characters for Linux user names are (POSIX ("Portable Operating System Interface for Unix") standard (IEEE Standard 1003.1 2008)):

alphanumeric, including the following common characters: period (.), underscore (_), and hyphen (-).

Therefore, characters that are allowed in IAM user names but not in Linux user names:

plus (+), equal (=), comma (,), at (@).

This solution will use the following mapping for those special characters in iam usernames when creating linux user accounts on the sshd_worker container:

• + => plus
• = => equal
• , => comma
• @ => at

So for example if we have an iam user called test@+=,test (which uses all of the disputed characters)

this username would translate to testatplusequalcommatest and they would need to shell in, e.g. with

ssh testatplusequalcommatest@dev-eu-west-1-bastion-service.yourdomain.com

Users should be aware that:

• They are logging on as themselves using an identity based on their AWS IAM identity
• They must manage their own ssh keys using the AWS interface(s), e.g. in the web console under IAM/Users/Security credentials and 'Upload SSH public key'.
• The ssh server key is set at container build time. This means that it will change whenever the bastion host is respawned

The following is referenced in "message of the day" on the container:

• They have an Ubuntu userland with passwordless sudo within the container, so they can install whatever they find useful for that session
• Every connection is given a newly instanced container, nothing persists to subsequent connections. Even if they make a second connection to the service from the same machine at the same time it will be a separate container.
• When they close their connection that container terminates and is removed
• If they leave their connection open then the host will kill the container after 12 hours

Logging

The sshd-worker container is launched with -v /dev/log:/dev/log This causes logging information to be recorded in the host systemd journal which is not directly accessible from the container. It is thus simple to see who logged in and when by interrogating the host (if you have access!), e.g.

journalctl | grep 'Accepted publickey'

gives information such as

April 27 14:05:02 dev-eu-west-1-bastion-host sshd[7294]: Accepted publickey for aws_user from 192.168.168.0 port 65535 ssh2: RSA SHA256:*****************************

N.B. It appears that calling client IP addresses are no longer visible - see issue 45. The reason for this is unclear.

Starting with release 3.8 it is possible to use the output giving the name of the role created for the service and to append additional user data. This means that you can call this module from a plan specifying your preferred logging solution, e.g. AWS cloudwatch.

Note that:

• ssh keys are called only at login- if an account or ssh public key is deleted from AWS whilst a user is logged in then that session will continue until otherwise terminated.

Components (using default userdata)

EC2 Host OS (debian) with:

• Systemd docker unit
• Systemd service template unit
• IAM Profile connected to EC2 host
• golang
• go binary compiled from code included in plan and supplied as user data - sourced from Fullscreen project

IAM Role

This and all of the following are prefixed with ${var.service_name} to ensure uniqueness. An appropriate set is created depending on whether or not an external role to assume is referenced for IAM identity checks.

• IAM role
• IAM policies
• IAM instance profile

Docker container 'sshd_worker' - built at host launch time using generic ubuntu image, we add awscli; sshd and sudo.

Go binary and forked to a companion repo.

The files in question on the host deploy thus:

/opt
├── golang
│   ├── bin
│   ├── pkg
│   └── src
├── iam_helper
│   ├── iam-authorized-keys-command
│   └── ssh_populate.sh
└── sshd_worker
    └── Dockerfile

• golang is the source and build directory for the go binary
• iam-helper is made available as a read-only volume to the docker container as /opt.
• iam-authorized-keys-command is the Go binary that gets the users and ssh public keys from aws - it is built during bastion deployment. Built directly on the host - we may look at a dockerised build in future
• ssh_populate.sh is the container entry point and populates the local user accounts using the go binary
• sshd_worker/Dockerfile is obviously the docker build configuration. It uses Ubuntu 22.04 from the public Docker registry and installs additional public packages.

Sample policy for other accounts

If you supply the ARN for an external role for the bastion service to assume ${var.assume_role_arn} then a matching sample policy and trust relationship is given as an output from the plan to assist with application in that other account for typical operation.

The DNS entry (if created) for the service is also displayed as an output of the format

name = "${var.environment_name}-${data.aws_region.current.name}-${var.vpc}-bastion-service.${var.dns_domain}"

Inputs and Outputs

These have been generated with terraform-docs

Requirements

Providers

Modules

No modules.

Resources

Inputs

Outputs