Awesome
Terraform Budget GCP
These modules create infrastructure on the Google Cloud Platform (GCP) that can be run for less than 10$ a month.( YMMV ) This repo contains instructions on how to actually create these resources. Read the companion article to understand what choices were made to cut costs.
It guides on how to create all the resources plus pre-requisites from scratch.
Resources created include;
- A Domain Name using Freenom
- A GCP Organization
- A GCP Seed Project from which other projects will be created.
- A GCP Project for our resources.
- A GKE Private Cluster.
- A Google Container Instance running Traefik.
- A Google CloudRun Service running Vault.
Compatibility
This module is meant for use with Terraform 0.12.
Requirements
- A Domain Name.
- A Google Cloud Platform Account.
- A Valid Debit or Credit Card to enable billing.
Setup
Pre-Requisites
You need the following before you can proceed. Skip if you already have these.
1. Get a Domain Name
Step 1
- Visit https://www.freenom.com.
- Search for a domain you like and click **Check Availability.
- If the domain name is available click Get it now! and then click **Checkout.
- Set the period to 12 months. Then click Continue.
- Check I have read and agree to the Terms & Conditions. Then click Complete Order.
- Provide your email and set up a password.
- You will receive an email notification to complete your order.
- Click on the confirmation link in the email and and provide your information to complete registration.
- You should be able to view your registered domain
You cannot use cloudflare API if your domain is a
.cf, .ga, .gq, .ml, or .tk
Skip Steps 2 & 3 if you have any of these.
Step 2
- Visit https://dash.cloudflare.com/sign-up.
- Click Add a site in Account Dashboard.
- Type in the domain you created above and click Add site.
- Select the Free Plan and click Confirm plan.
- Cloudflare will scan for existing DNS records. Wait until it finishes, and click Continue.
- Cloudflare will give you two nameservers to set up in Freenom.
Step 3
- Go back to Freenom, Click Services > My Domains. Click Manage Domain on the domain that you’re configuring.
- Click Management Tools > Nameservers > Use custom nameservers (enter below). Now enter the nameservers provided by Cloudflare, and click Change Nameservers.
- Go back to Cloudflare, click Done, check nameservers. It may take a while, you will receive an email once your domain has been added.
- Click Profile > API Tokens > Create Token
- On Edit zone DNS, Click Use Template > Under Zone Resources > Select your domain
- Click Continue To Summary > Create Token then copy and save the created token.
Reference
</details>2. Sign up for a GCP Account
<details> <summary>Click to expand</summary>- Visit Google Cloud to create your account.
Provide the required information and you should be greeted with this page.
<a href="https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https://github.com/nufailtd/terraform-budget-gcp&open_in_editor=README.md&cloudshell_tutorial=tutorial.md" target="_blank"><img alt="Open this project in Cloud Shell" src ="https://gstatic.com/cloudssh/images/open-btn.svg"></a>
You only need to click on this button once.
If you already did this, click here.
Then run below to resume the tutorial
cloudshell launch-tutorial -d ~/cloudshell_open/terraform-budget-gcp/tutorial.md
Create the GCP Seed Project
Please skip this for now and return when/if you successfully create an organization.
We are able to deploy our resources without it.
Switch to the seed_project
folder;
cd ~/cloudshell_open/terraform-budget-gcp/seed_project
Create a file terraform.tfvars
with the following information from the organization you just created;
org_id = "<ORGANIZATION_ID>"
billing_account = "<BILLING_ACCOUNT_ID>"
group_org_admins = "<admin@domain.com>"
group_billing_admins = "<billing@domain.com>"
default_region = "us-central1"
sa_enable_impersonation = true
Then perform the following commands:
terraform init
- to get the plugins
you@cloudshell:~/cloudshell_open/terraform-budget-gcp/seed_project$ terraform init
terraform plan
- to see the infrastructure plan
you@cloudshell:~/cloudshell_open/terraform-budget-gcp/seed_project$ terraform plan
terraform apply --auto-approve
- to apply the infrastructure build
you@cloudshell:~/cloudshell_open/terraform-budget-gcp/seed_project$ terraform apply --auto-approve
</details>
Create the GCP Project
Switch to the budget_gcp_project
folder;
cd ~/cloudshell_open/terraform-budget-gcp/budget_gcp_project
Create a terraform.tfvars
file in the budget_gcp_project
directory by running
./create-tfvars.sh myproject
Example;
user@cloudshell:~/cloudshell_open/terraform-budget-gcp/budget_gcp_project$ ./create-tfvars.sh myproject
## This section has been auto-generated by ./create-tfvars.sh ##
email = "user@gmail.com"
billing_account = "02E280-9E2C47-1DF365"
name = "myproject"
## This section has been auto-generated by ./create-tfvars.sh ##
Then perform the following commands:
terraform init
- to get the plugins
you@cloudshell:~/cloudshell_open/terraform-budget-gcp/budget_gcp_project$ terraform init
terraform plan
- to see the infrastructure plan
you@cloudshell:~/cloudshell_open/terraform-budget-gcp/budget_gcp_project$ terraform plan
terraform apply --auto-approve
- to apply the infrastructure build
you@cloudshell:~/cloudshell_open/terraform-budget-gcp/budget_gcp_project$ terraform apply --auto-approve
This operation will output the project_id
to be used in the next steps.
Before creating our resources it will be convenient to complete the following manual operations.
This will help us automate the final step.
<a name="create-oauth2-client"></a>
Create an Oauth2 Client in GCP
<details> <summary>Click to expand</summary>-
Log in to your Google Cloud account and go to the APIs & services.
Select the project you created above.
Navigate to OAuth consent screen using the left-hand menu and select External -
Configure your Oauth Consent Screen
Under App information Fill in App Name
Select your email in User support email
Enter your email in Developer contact information
Click Save & Continue
Proceed to next page and under scopes make sure you select the following scopes only
openid, profile, email
Use the Filter table on the right to search for the scopes and check each of the scopes then click UpdateClick Save & Continue
Under Test Users Add your email as a test user and complete.Click Save & Continue
A successful configuration should like below -
Create New Credentials.
On the Credentials page, click Create credentials and select OAuth client ID. -
Configure Client ID
On the Create OAuth client ID page, select Web application.
Under Authorized redirect URIs set the following parameters substitutingdomain.com
for domain you created above
https://authenticate.domain.com/oauth2/callback
https://vault.domain.com/ui/vault/auth/oidc/oidc/callback
-
Click Create to proceed. The [Client ID] and [Client Secret] settings will be displayed.
Download the client_secret file to be used in a later configuration.
<a name="create-zone"></a>
Create a zone in CloudDNS
<details> <summary>Click to expand</summary>-
In your Google Cloud account go to Net services.
Select the project created above then navigate to Cloud DNS using the left-hand menu and click Create zone
-
Create New Zone.
On the Create a DNS Zone page, enter a Zone Name (a nice approach is to replace the periods in your domain with a dash) then enter the domain you created above in DNS name. -
Get Nameservers.
Nameservers in the format ns-cloud-xx.googledomains.com will be presented to you.
Take note of them as you will use them in the next step.
<a name="update-nameservers"></a>
Update Nameservers in Freenom (Or your Domain Name Registrar)
<details> <summary>Click to expand</summary>-
Login to your Freenom account, Click Services > My Domains. Click Manage Domain on the domain that you’re configuring.
-
Click Management Tools > Nameservers > Use custom nameservers (enter below). Now enter the nameservers from CloudDNS, and click Change Nameservers.
-
Confirm the nameservers have been set successfully.
Create resources in the GCP Project
The variables for this step are derived from the previous steps, namely:
oauth2 ClientID
oauth2 ClientSecret
cloudflare Token
(optional)domain
project_id
Make sure you are in the root folder
cd ~/cloudshell_open/terraform-budget-gcp
Create a file terraform.tfvars
by running the following command;
cp tfvars.example terraform.tfvars
In your cloud shell editor, open the file terraform.tfvars
and change the values in "<>"
to match your values from the previous steps.
project_id = "<project_name>"
region = "us-central1"
zones = ["us-central1-a"]
cluster_name = "kluster"
domain = "<your domain>"
domain_filter = "<your domain>"
run_post_install = false
email = "<your email>"
dns_auth = [
{
name = "provider"
value = "google"
}
]
# OIDC Configuration
oidc_config = [
{
name = "authenticate.idp.provider"
value = "google"
},
{
name = "authenticate.idp.clientID"
value = "<[project_id]-[hash].apps.googleusercontent.com>"
},
{
name = "authenticate.idp.clientSecret"
value = "<mysecret>"
},
{
name = "authenticate.idp.url"
value = "https://accounts.google.com"
}
]
Then perform the following commands:
It is recommended to run this before performing the subsequent commands.
This ensures that you use the project's service account to create resources.
The token expires every hour so you'll have to re-issue these command if you get a not allowed error.
Simply run;
. ./auth.sh [your_project_id]
;
you@cloudshell:~/cloudshell_open/terraform-budget-gcp$ . ./auth.sh [your_project_id]
terraform init
- to get the plugins
you@cloudshell:~/cloudshell_open/terraform-budget-gcp$ terraform init
terraform plan
- to see the infrastructure plan
you@cloudshell:~/cloudshell_open/terraform-budget-gcp$ terraform plan
terraform apply --auto-approve
- to apply the infrastructure build
you@cloudshell:~/cloudshell_open/terraform-budget-gcp$ terraform apply --auto-approve
Important!!
Once the above step completes successfully, perform the following command:
./post-install.sh
you@cloudshell:~/cloudshell_open/terraform-budget-gcp$ ./post-install.sh
- This will, in order, do the following;
- Set
run_post_install = true
interraform.tfvars
- Enable
vault-config
module in file vault-config.tf - Run
terraform init
to get plugins - Run
terraform plan
to display proposed changes
- Set
Note: It is possible to run these commands manually and skip the ./post-install.sh
script.
This script just offers convenience.
Then run:
terraform apply --auto-approve
- to apply the infrastructure build
you@cloudshell:~/cloudshell_open/terraform-budget-gcp$ terraform apply --auto-approve
We do this in 2 steps because of some limitations in terraform that will cause an error if certain resources do not exist.
If you encounter any errors, try re-running the command
terraform apply --auto-approve
.
It is safe to do so.
Sucesss
Congratulations!! You did it.
Some of the services automatically deployed can be found in the following urls (substituting your.domain of course);
- Vault https://vault.your.domain
- Pomerium https://authenticate.your.domain/.pomerium/
- Traefik https://dash.your.domain
- Ghost https://blog.your.domain
Check them out below
File structure
The project has the following folders and files:
- /: root folder
- /modules/cert/: - creates certificated for our domain.
- /modules/cloud-run/: creates a ghost blog cloudrun service
- /modules/custom-nat/: creates an internet gateway for our private cluster
- /modules/dns/: creates domain records for our domain name
- /modules/kubeip/: assigns a static IP to instances in our cluster *not used*
- /modules/pomerium-app/: secures our applications to provide access to from allowed users only
- /modules/test-workload-identity/: tests the google workload identity feature
- /modules/traefik-sa/: creates a kubernetes service account used by traefik-vm
- /modules/traefik-vm/: creates an instance with a public ip to route traffic to our cluster
- /modules/vault-cloud-run/: creates a vault application running in cloudrun
- /modules/vault-sa/: creates a kubernetes service account to be used by the vault application
- /modules/workload-identity/: allows us to access google apis without requiring us to save key files
- /tfvars.example: an example of a file to generate terraform.tfvars
- /variables.tf: variables used by our main.tf file
- /main.tf/: creates gke cluster and the rest of the modules
- /outputs.tf: displays created resources
- /README.md: this file
- /auth.sh: authenticates your servicce account
- /post-install.sh: creates resources that could not be created in the first run
- /destroy.sh: destroys resources
Acknowledgements
This project builds upon some excellent work from other projects. Here are some of them;
- Sandeep Dinesh - Get 3X Capacity for your Kubernetes Cluster
- Niko Kosonen - How to: Kubernetes for Cheap on Google Cloud
- Henrik Hoegh - Proxying Kubernetes services with Traefik
- Ahmet B - Cloud Run Static IP
- Seth Vargo - Vault Init
- Mike Brancato - Vault Google Cloud Run Module
Clean up
You now have GCP resources to play around with on the cheap.
However, if you want to delete these resources and stop charges accruing to your account, run
./destroy.sh
you@cloudshell:~/cloudshell_open/terraform-budget-gcp$ ./destroy.sh
to destroy the built infrastructure.