Awesome

Requirements

• Bash (v3+)
• Terraform
• AWS CLI (only for aws_bootstrap command)

About

Terraformsh is a Bash script that makes it easier to run Terraform by performing common steps for you. It also makes it easy to keep your configuration DRY and deploy infrastructure based on a directory hierarchy of environments. See DRY_CODE.md for more details.

Unlike Terragrunt, this script includes no DSL or code generation. All it does is make it easier to call Terraform. See PHILOSOPHY.md for more details.

Terraformsh will detect and use Terraform -var-files and -backend-config configuration files across a directory hierarchy. It also has its own configuration file so you don't have to remember any command other than terraformsh itself (removing the need for a Makefile, though you can still use one if you want).

You can override any options with environment variables, command-line options and config files. Good conventions like using .plan files for changes are the default.

How it works

Basic operation

Change to the directory of a Terraform module and run terraformsh with any Terraform commands and arguments you'd normally use.

   $ cd root-modules/aws/common/
   $ terraformsh plan
   $ terraformsh apply

Terraformsh will run dependent Terraform commands when necessary. If you run terraformsh plan, Terraformsh will first run terraform validate, but before that terraform get, but before that terraform init. Terraformsh passes relevant options to each command as necessary, and you can also override those options.

Automatic plan files

When certain commands are run (plan, apply, plan_destroy, destroy) Terraformsh will use the appropriate options to create a plan file. This way you can be sure that an apply or destroy operation will only happen on a plan that has been saved to a file and reviewed. (You can disable this automatic behavior by setting USE_PLANFILE=0 as an environment or configuration variable)

The plan files are, by default, written to the directory where you ran Terraformsh, with a naming convention like tfsh.92h39d9hd9.plan. You can override this by setting environent or configuration variable TF_PLANFILE and TF_DESTROY_PLANFILE.

Multiple commands as arguments

You can pass multiple Terraform commands as options and it'll run them in the order you specify.

Not sure what that looks like? Use the dry-run mode:

    $ ./terraformsh -N plan apply
    ./terraformsh: WARNING: No -b option passed! Potentially using only local state.

    + terraform init -input=false -reconfigure -force-copy
    + terraform get -update=true
    + terraform validate
    + terraform plan -input=false -out=/home/vagrant/git/PUBLIC/terraformsh/tf.104900abc1.plan
    + terraform init -input=false -reconfigure -force-copy
    + terraform apply -input=false /home/vagrant/git/PUBLIC/terraformsh/tf.104900abc1.plan

Change directory at runtime

You can tell Terraformsh to change to a module's directory before running commands so you don't have to do it yourself (later versions of Terraform have an option for this, but earlier ones don't):

    $ ./terraformsh -C ../../../root-modules/aws/common/ plan

Passing Terraform tfvars files

You can pass Terraform configuration files using the -f or -b options.

    $ terraformsh -C ../../../root-modules/aws/common/ \
        -f terraform.tfvars.json \
        -f override.auto.tfvars.json \
        -b backend.tfvars \
        -b backend-key.tfvars \
        plan approve apply

To make this even simpler, if you pass any argument to Terraformsh after the initial OPTIONS, and they match TFVARS ('*.backend.tfvars', '*.backend.sh.tfvars', '*.tfvars.json', '*.tfvars', '*.sh.tfvars.json', '*.sh.tfvars'), they will be automatically loaded with the -f and -b options.

    # Assuming you already have 'something.tfvars' and 'something.backend.tfvars'
    # in your current working directory, run the following:
    $ terraformsh -C ../../../root-modules/aws/common/ \
        *.tfvars \
        plan approve apply

Finally, if in any parent directory of where you ran Terraformsh, there are files named backend.sh.tfvars, terraform.sh.tfvars.json, or terraform.sh.tfvars, those will also be loaded automatically (you can disable this with the -I option).

    $ mkdir -p some/configs/here
    $ cd some
    $ touch terraform.sh.tfvars
    $ cd configs
    $ touch backend.sh.tfvars
    $ cd here
    $ touch terraform.sh.tfvars
    $ terraformsh -N plan apply
    + terraform init -input=false -reconfigure -force-copy -backend-config /home/vagrant/git/PUBLIC/terraformsh/some/configs/backend.sh.tfvars
    + terraform get -update=true
    + terraform validate -var-file /home/vagrant/git/PUBLIC/terraformsh/some/terraform.sh.tfvars -var-file /home/vagrant/git/PUBLIC/terraformsh/some/configs/here/terraform.sh.tfvars
    + terraform plan -var-file /home/vagrant/git/PUBLIC/terraformsh/some/terraform.sh.tfvars -var-file /home/vagrant/git/PUBLIC/terraformsh/some/configs/here/terraform.sh.tfvars -input=false -out=/home/vagrant/git/PUBLIC/terraformsh/some/configs/here/tf.019c25e289.plan
    + terraform init -input=false -reconfigure -force-copy -backend-config /home/vagrant/git/PUBLIC/terraformsh/some/configs/backend.sh.tfvars
    + terraform apply -input=false /home/vagrant/git/PUBLIC/terraformsh/some/configs/here/tf.019c25e289.plan

Environment Variables / Configuration

Don't want to remember what options to pass to terraformsh? You don't have to! You can capture anything you want Terraformsh to do in a config file that is automatically loaded.

The config file format is just a bash script. Therefore you can do things like 'export' arbitrary environment variables for Terraform to load, or even run custom code.

It's highly recommended that you do not set environment variables like Terraform's TF_VAR_*, otherwise you will have a mix of variables set in both config files and environment variables, and it will make it difficult to track down where/how a variable is being set. Stick to static variables in *.tfvars or *.tfvars.json files, and load dynamic variables from Terraform with a data source.

You can set the following variables in a config file (any of: /etc/terraformsh, ~/.terraformshrc, .terraformshrc, terraformsh.conf), or set them as environment variables before you call Terraformsh:

DEBUG=1                     # Enable bash tracing
TERRAFORM=terraform         # The name of the terraform executable
TF_PLANFILE=                # Automatically populated by terraformsh
TF_DESTROY_PLANFILE=        # Automatically populated by terraformsh
TF_BOOTSTAP_PLANFILE=       # Automatically populated by terraformsh
PUSH_ERRORED_TFSTATE=0      # Don't push errored.tfstate on failed apply
USE_PLANFILE=0              # Don't use a plan file for each apply/destroy
INHERIT_TFFILES=0           # Don't inherit tfvars files in parent directories
NO_DEP_CMDS=1               # Don't run dependent commands automatically
NO_CLEANUP_TMP=1            # Don't clean up temporary TF_DATA_DIR
DRYRUN=1                    # Enable dry-run mode
CD_DIR=             # The directory to change to before running terraform commands

The environment variable TF_DATA_DIR is automatically overridden by Terraformsh. A new temporary directory is created for the data dir, based on both the name of the directory you ran Terraformsh from, and the Terraform module directory you run terraform against (the -C option). If you pass your own TF_DATA_DIR environment variable, Terraformsh will use that instead.

The following can be set in the Terraformsh config file as Bash arrays, or you can set them by passing them to -E, such as -E "PLAN_ARGS=(-no-color -input=false)".

VARFILES=()                     # files to pass to -var-file
BACKENDVARFILES=()              # files to pass to -backend-config
CMDS=()                         # the commands for terraformsh to run
PLAN_ARGS=(-input=false)        # the arguments for 'terraform plan'
APPLY_ARGS=(-input=false)       # the arguments for 'terraform apply'
PLANDESTROY_ARGS=(-input=false) # arguments for 'plan -destroy'
DESTROY_ARGS=(-input=false)     # arguments for 'terraform destroy'
REFRESH_ARGS=(-input=false)     # arguments for 'terraform refresh'
INIT_ARGS=(-input=false -reconfigure -force-copy)  # arguments for 'terraform init'
OH12UPGRADE_ARGS=(-yes)         # arguments for '0.12upgrade'
OH13UPGRADE_ARGS=(-yes)         # arguments for '0.13upgrade'
IMPORT_ARGS=(-input=false)      # arguments for 'terraform import'
GET_ARGS=(-update=true)         # arguments for 'terraform get'
STATE_ARGS=()                   # arguments for 'terraform state'
VALIDATE_ARGS=()                # arguments for 'terraform validate'
WORKSPACE_ARGS=()               # arguments for 'terraform workspace'
CONSOLE_ARGS=()                 # arguments for 'terraform console
OUTPUT_ARGS=()                  # arguments for 'terraform output'
TAINT_ARGS=()                   # arguments for 'terraform taint'
UNTAINT_ARGS=()                 # arguments for 'terraform untaint'
FORCEUNLOCK_ARGS=(-force)       # arguments for 'terraform forceunlock'
SHOW_ARGS=()                    # arguments for 'terraform show'

To use the 'aws_bootstrap' command, pass the '-b FILE' option and make sure the file(s) have the following variables:

bucket          - The S3 bucket your Terraform state will live in
dynamodb_table  - The DynamoDB table your Terraform state will be managed in

An example file: .terraformshrc-example

Interactive troubleshooting

Need to troubleshoot some problem by just running 'terraform' yourself? No problem, use the shell command. It will drop you into a Bash shell after first changing to the correct directory and running terraform init and terraform get with all the environment variables set up for you (including the automatic TF_DATA_DIR).

    $ ./terraformsh -N -C ../../../root-modules/aws/common/ shell
    + cd "../../../root-modules/aws/common/"
    ./terraformsh: WARNING: No -b option passed! Potentially using only local state.

    + terraform init -input=false -reconfigure -force-copy
    + terraform get -update=true
    + bash -i -l

You can even get Terraformsh to explicitly ask you for confirmation before moving to the next command with the approve command (since the default is to pass -input=false to each command for easier use in automation).

Are you working in a hierarchy of config files, and want to grep all the parent directories? Use the built-in revgrep command:

    $ terraformsh revgrep -H -e "gcp_project_id"
    terraformsh: Info: Found terraform command 'revgrep'
    terraformsh: Warning: '-H' is not a valid command; passing as an option instead
    terraformsh: Warning: 'project_id' is not a valid command; passing as an option instead
    + cd "/home/vagrant/my-repo/env/product/dev/nonprod/us-west1/tf-state/bootstrap"
    /home/vagrant/git/my-repo/env/product/dev/nonprod/terraform.sh.tfvars:gcloud_project_id = "123456789"

Want to output one of Terraformsh's plan files as JSON?

    $ terraformsh show -json "$(pwd)/tf.b063520160.plan"

More Examples

There are many ways to use Terraformsh, whether you pass all the options via environment variables/command-line options, or keep all the commands in a configuration file and load everything automatically.

• Run 'plan' using a .terraformshrc file that has all the above options, but override terraformsh's internal arguments to 'terraform plan':

   $ terraformsh -E 'PLAN_ARGS=("-compact-warnings" "-no-color" "-input=false")' \
     plan
  

• Run 'plan' on a module and pass any configs found in these directories:

   $ terraformsh -C root-modules/my-database/ \
      *.tfvars \
      env/my-database/*.tfvars \
      plan
  

• Run 'plan' on a module, implicitly loading configuration files from parent directories:

   $ pwd
   /home/vagrant/git/some-repo/env/non-prod/us-east-2/my-database
   $ echo 'CD_DIR=../../../../modules/my-database/' > terraformsh.conf
   $ echo 'aws_account_id = "0123456789"' > ../../terraform.sh.tfvars
   $ echo 'region = "us-east-2"' > ../terraform.sh.tfvars
   $ echo 'database_name = "some database"' > terraform.sh.tfvars
   $ terraformsh plan
  

• You've applied some Terraform using local state, and now you want to migrate it to a remote backend. After you add your new backend tf code, you'd run:

   $ terraformsh -E "INIT_ARGS=()" init -force-copy -migrate-state plan apply
  

Having trouble?

• Problem: I'm using Terraformsh from two different shell sessions, in the same directory, running the same commands, but one of them is working and the other isn't. What's going on?

  Solution: Something's wrong with your environment variables in one of the sessions. If both TF_DATA_DIR and TF_TMPDIR are set to something starting with /tmp/tfsh., then you probably used terraformsh shell and forgot to exit.

terraformsh v0.14
Usage: ./terraformsh [OPTIONS] [TFVARS] COMMAND [..]

Options

Pass these OPTIONS before any others (see examples); do not pass them after TFVARS or COMMANDs.

-f FILE         A file passed to Terraform's -var-file option.
                  ( config: VARFILES= )
-b FILE         A file passed to Terraform's -backend-config option.
                  ( config: BACKENDVARFILES= )
-C DIR          Change to directory DIR.
                  ( config: CD_DIR= )
-c file         Specify a '.terraformshrc' configuration file to load.
-E EXPR         Evaluate an expression in bash ('eval EXPR').
-I              Disables automatically loading any 'terraform.sh.tfvars',
                'terraform.sh.tfvars.json', or 'backend.sh.tfvars' files
                found while recursively searching parent directories.
                  ( config: INHERIT_TFFILES=0 )
-P              Do not use '.plan' files for plan/apply/destroy commands.
                  ( config: USE_PLANFILE=0 )
-D              Don't run 'dependency' commands (e.g. don't run "terraform
                init" before "terraform apply").
                  ( config: NO_DEP_CMDS=1 )
-N              Dry-run mode (don't execute anything).
                  ( config: DRYRUN=1 )
-n              Don't remove the temporary TF_DATA_DIR.
                  ( config: NO_CLEANUP_TMP=1 )
-v              Verbose mode.
                  ( config: DEBUG=1 )
-h              This help screen.

Commands

The following are Terraform commands that terraformsh provides wrappers for (there's some Terraformsh-specific logic behind the scenes). Other Terraform commands not listed here are passed through to Terraform verbatim.

plan              Run init, get, validate, `terraform plan @VARFILE_ARG -out $TF_PLANFILE`
apply             Run init, get, validate, `terraform apply $TF_PLANFILE`
plan_destroy      Run init, get, validate, `terraform plan -destroy -out=$TF_DESTROY_PLANFILE`
destroy           Run init, get, validate, `terraform apply $TF_DESTROY_PLANFILE`
refresh           Run init, `terraform refresh`
validate          Run init, get, `terraform validate`
init              Run clean_modules, `terraform init @BACKENDVARFILE_ARG`
get               Run init, `terraform get [..]`
show              Run init, `terraform show [..]`
import            Run init, `terraform import [..]`
state             Run init, `terraform state [..]`
taint             Run init, `terraform taint [..]`
untaint           Run init, `terraform untaint [..]`
output            Run init, refresh, `terraform output [..]`
console           Run init, `terraform console [..]`
workspace         Run init, `terraform workspace [..]`
force-unlock      Run init, `terraform force-unlock [..]`
0.12upgrade       Run init, `terraform 0.12upgrade [..]`
0.13upgrade       Run init, `terraform 0.13upgrade [..]`

The following commands are specific to terraformsh:

shell             Run init, get, and `bash -i -l`
clean             Remove '.terraform/modules/*', terraform.tfstate files, and .plan files
clean_modules     Run `rm -v -rf .terraform/modules/*`
approve           Prompts the user to approve the next step, or the program will exit with an error.
aws_bootstrap     Looks for 'bucket' and 'dynamodb_table' in your '-b' file options.
                  If found, creates the bucket and table and initializes your Terraform state with them.
revgrep           Run 'grep' on files in all parent directories
env               Run 'env' command with optional arguments

All arguments after a COMMAND are evaluated for whether they match a Terraform or Terraformsh command; if they don't, they are assumed to be options and are passed to the first recognized command that precedes them.