Awesome
[EXPERIMENTAL] terraform-controller
Use K8s to Run Terraform
NOTE: We are actively experimenting with this in the open. Consider this ALPHA software and subject to change.
Terraform-controller - This is a low level tool to run Git controlled Terraform modules in Kubernetes. The controller manages the TF state file using Kubernetes as a remote statefile backend (requires Terraform 0.13.4)! You can have changes auto-applied or wait for an explicit "OK" before running.
There are two parts to the stack, the controller and the executor.
The controller creates three CRDs and runs controllers for modules and executions. A module is the building block and is the same as a terraform module. This is referenced from an execution which is used to combine all information needed to run Terraform. The execution combines Terraform variables and environment variables from secrets and/or config maps to provide to the executor.
The executor is a job that runs Terraform. Taking input from the execution run CRD the executor runs terraform init, terraform plan and terraform create/destroy depending on the context.
Executions have a 1-to-many relationship with execution runs, as updates or changes are made in the module or execution additional runs are created to update the terraform resources.
Deploying
Use provided manifests kubectl create -f ./manifests to deploy to an existing k8s cluster. Manifests will create all CRDs necessary and a Deployment with the rancher/terraform-controller image.
Verify
~ kubectl get all -n terraform-controller
NAME READY STATUS RESTARTS AGE
pod/terraform-controller-8494cf85c5-x97sn 1/1 Running 0 17s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/terraform-controller 1/1 1 1 18s
NAME DESIRED CURRENT READY AGE
replicaset.apps/terraform-controller-8494cf85c5 1 1 1 18s
Namespace
Everything is put in the terraform-controller namespace with these provided manifests. Edit metadata.namespace in files to change name space or remove to run in default. You will need to update the args for the command in the deployment to update or remove --namespace argument for the executable. Passing in the flag limits the controller to only watching CRD objects in it's namespace, remove this param to let the terraform-controller see all CRD objects in any namespace.
Quickstart Appliance + k3s
Use (k3d)[https://github.com/rancher/k3d/releases] to spin up small (k3s)[https://github.com/rancher/k3s] clusters for a quick start for using the Terraform Controller. The appliance image comes pre-built with the deployment manifests and will auto-create verything the Terraform Controller needs when they boot.
~ k3d create --name terraform-controller --image rancher/terraform-controller-appliance
~ export KUBECONFIG="$(k3d get-kubeconfig --name='terraform-controller')"
~ kubectl get all -n terraform-controller
NAME READY STATUS RESTARTS AGE
pod/terraform-controller-d774bbd44-w4mzk 0/1 Pending 0 1s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/terraform-controller 0/1 1 0 14s
NAME DESIRED CURRENT READY AGE
replicaset.apps/terraform-controller-d774bbd44 1 1 0 1s
~ kubectl get crd | grep terraformcontroller
executionruns.terraformcontroller.cattle.io 2019-05-22T17:19:01Z
executions.terraformcontroller.cattle.io 2019-05-22T17:19:01Z
modules.terraformcontroller.cattle.io 2019-05-22T17:19:01Z
~ k3d delete --name terraform
Example
With the controller running you can run the provided example in the ./example directory which shows you how to run a basic Digital Ocean Terraform module which takes a do_token and do_name and creates a droplet.
Modify ./example/00-secret.yaml with your Digital Ocean API token and desired droplet name.
Run kubectl create -f ./example -n terraform-controller to create all envvars/secrets/module and the execution which will automatically run. The controller creates Jobs for the Terraform runs so to access logs check the pod logs for the executor create and destroy jobs. This example is setup to auto-confirm and auto-delete when the CRD object is destroyed.
Delete the droplet by deleting the CRD kubectl delete -f ./example/20-deployment.yaml -n terraform-controller.
Approving a Plan
In ./example/20-state.yaml its pre-configured to auto-approve and auto-delete when you make the execution CRD. You can turn off spec.destroyOnDelete and spec.autoConfirm and do these by hand doing the following.
To get the plan check logs of the pods used to run the job.
kubectl logs [executer-pod-name] -n terraform-controller
Assuming the action Terraform is going to perform is correct annotate the Execution Run to approve the changes:
kubectl annotate executionruns.terraform-controller.cattle.io [execution-run-name] -n terraform-controller approved="yes" --overwrite
Once the job completes, you can see the outputs from Terraform by checking the Execution Run:
kubectl get executionruns.terraform-controller.cattle.io [execution-run-name] -n terraform-controller -o yaml
With destroyOnDelete turned off you will have to delete the Droplet by hand as a destroy job will not kick off.
Building Custom Execution Environment
Create a Dockerfile
FROM rancher/terraform-controller-executor:v0.0.3 #Or whatever the release is
RUN curl https://myurl.com/get-some-binary
Build that image and push to a registry.
When creating the execution define the image:
apiVersion: terraformcontroller.cattle.io/v1
kind: Execution
metadata:
name: cluster-create
spec:
moduleName: cluster-modules
destroyOnDelete: true
autoConfirm: false
image: cloudnautique/tf-executor-rancher2-provider:v0.0.3 # Custom IMAGE
variables:
SecretNames:
- my-secret
envConfigNames:
- env-config
If you already have an execution, edit the CR via kubectl and add the image field.
Building
make
Local Execution
Use ./bin/terraform-controller
Running the Executor in Docker - Useful for testing the Executor
docker run -d -v "/Path/To/Kubeconfig:/root/.kube/config" -e "KUBECONFIG=/root/.kube/config" -e "EXECUTOR_RUN_NAME=RUN_NAME" -e "EXECUTOR_ACTION=create" rancher/terraform-controller-executor:dev
License
Copyright (c) 2019 Rancher Labs, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.