Home

Awesome

T.A.D.S. boilerplate <!-- omit in toc -->

Build Status License: MIT Tweet

<div align="center"><strong>The power of Ansible and Terraform + the simplicity of Swarm = DevOps on :fire::fire::fire:</strong></div> <br />

T.A.D.S. logo

:tada: What is it?

A boilerplate to create a full Infrastructure as Code (IaC) repository, from provisioning to deployment with:

It handles different environments:

On top of that, it features:

With T.A.D.S., you will be able to onboard a new developer on your project in less than 3 minutes, with just 3 commands! Even if you have a complex microservices architecture. Forget about your outdated wikis or installation procedures, they are no longer needed! See the example user README to get a preview of what your new procedures could look like.

Example of a fresh development environment setup with T.A.D.S. in 02:30!

<div align="center"><sub>Example of a fresh development environment setup with T.A.D.S. in 02:30!</sub></div>

:dart: Who is it for?

If you recognize yourself into some of these statements, this project is definitely for you:

On the contrary, this project might not be for you if:

... but don't be sad, I am thinking of creating a similar project for K8s ;) Tell me if you want to help!

:muscle: Philosophy

:lock: Knowledge prerequisites

Before going further, I assume that you already have the knowledge and practice with Docker Swarm mode, Ansible, Terraform, and Infrastructure as Code in general. If it is not the case, I urge you to study and practice before. You can use this material as a starter:

:lock: Technical prerequisites

Have a look at Install the required dependencies for installation procedures.

OS X: It should not be that hard to make the project run on OS X. PRs are welcome! I am also thinking of creating a dockerized version of the project to improve compatibility.

:rocket: Quick start

1. Make this repository yours

Clone this repo, create your own and push the code to it.

git clone --single-branch https://github.com/Thomvaill/tads-boilerplate.git <YOUR_PROJECT_NAME>
cd <YOUR_PROJECT_NAME>
git remote set-url origin <YOUR_REPO_URL>
git push

2. Install the required dependencies

This will install Ansible, Vagrant, Virtualbox and Terraform on your local machine:

./tads install-dependencies

You can also manually install the dependencies if your preferer.

3. Provision your local machine and deploy the example stack

  1. Copy ansible/group_vars/localhost_overrides.sample.yml to ansible/group_vars/localhost_overrides.yml
  2. Add ansible_user variable with your user name
ansible_user: <yourUserName>
  1. Provision and Deploy
./tads ansible-playbook localhost provision
./tads ansible-playbook localhost deploy

The first ./tads command will:

And the second one will deploy traefik and example_app stacks. If everything went well, you are now able to access it at this URL: https://yourcompany.localhost/

4. Write your own Docker Swarm Compose files

Now that the example stack is running on your machine, you can deploy your own services.

First, you probably need to change the domains dict in ansible/group_vars/all.yml. This file contains all Ansible variables default values. These values can be overridden later in other group_vars files. You are free to add your variables in it.

Then, you can write your own Docker Swarm Compose files, following this naming convention: ansible/stacks/<STACK_NAME>/<STACK_NAME>.yml.j2 These files are Jinja2 templates. You are highly encouraged to use Ansible variables in them, so your template file can be used across all your environments. Have a look at ansible/stacks/example_app/example_app.yml.j2 to see a good example.

Finally, do not forget to add your new stacks to ansible/deploy.yml.

To help you with the ansible/group_vars directory, here is a representation of Ansible groups:

all
├── dev
|   ├── localhost
|   └── vagrant
├── production
├── staging
└── any_other_remote_environment...

Each group has its _overrides counterpart, which enables you to override some variables locally in a xxx_overrides.yml file, which is not versionned. Have a look at .sample.yml files to see some examples.

While developing, perform some ./tads ansible-playbook localhost deploy to test your deployment. Do not forget to run ./tads ansible-playbook localhost provision again if you have changed domain names.

Tips:

5. Test on a Vagrant cluster

Now that you are happy with your localhost environment, you should test the provisioning and the deployment on an environment which looks more like a production environment. For instance, on localhost, you can have just one node! And maybe you forgot some dependencies that are already installed on your computer. With Vagrant, you will be able to test your stacks on a fresh 3 nodes Swarm cluster.

  1. Copy vagrant/vagrant.sample.yml to vagrant/vagrant.yml and adjust its settings
  2. Run ./tads vagrant up
  3. Run ./tads ansible-playbook vagrant all

Now, you will be able to test your stacks deployed on Vagrant. If you have kept the example app, you can test it on https://yourcompany.test/

Tips:

6. Edit and encrypt your production environment variables

Before going further, you should edit your production group_vars files:

When you are done, do not commit production_encrypted.yml! You have to encrypt it first:

The first command has generated a random key in ansible/vault_keys/production. You must not commit this file. You should keep it in a safe place, and share it with your authorized team members securely. If you lose it, you won't be able to decrypt your files anymore. The second one has encrypted your file with AES-256. You can now commit it.

You can still edit this file by running ./tads ansible-vault production edit ansible/group_vars/production_encrypted.yml. Always check that you do not commit an unencrypted version of this file by mistake.

7.a. Create, provision and deploy your production environment with Terraform

Now that everything is fine locally, it is time to create and deploy your production environment!

The terraform/environments/production is an AWS example. PRs are welcome for other providers! To make it work, you should:

Terraform will use this default profile credentials.

Then, you can run ./tads terraform production init and ./tads terraform production apply. This example will create:

The CLI will also create the corresponding Ansible inventory for you in ansible/inventories/production from Terraform outputs. You should commit it. You should also commit the Terraform state file, or better: use a remote state.

Then, you have to create an alias in Route53 to the ELB.

Finally, you can run ./tads ansible-playbook production all and your website will be available!

Disclaimer:

7.b. Provision and deploy your production environment to an existing infrastructure

If you don't want to use a cloud provider, you can use classic Virtual Machines. For a production environment, you should have at least 3 manager nodes, so 3 VMs. They should be fresh installs. Ubuntu server 18.04 or Debian 9 is fine.

  1. Make sure you can SSH into the VMs with your key pair
  2. Copy ansible/inventories/production.sample-baremetal to ansible/inventories/production
  3. Edit it
  4. Run ./tads ansible-playbook production all and your website will be available!

8. Add other remote environments

You can add other remote environments, like production.

For Terraform, you just have to duplicate terraform/environments/production to the directory of your choice, eg staging. After editing it, you can run ./tads terraform staging apply, it will create the ansible/inventories/staging inventory file.

For an existing bare metal infrastructure, you just have to create the ansible/inventories/staging inventory file.

Then, in Ansible, you have to create these files:

Then, create the ansible-vault key and encrypt the file:

Finally, provision and deploy! ./tads ansible-playbook staging all

9. Make your team members autonomous

It is one of this project's goals: DevOps is not a job, it is a mindset! Now that you have a beautiful IaC, it is time to onboard your team members.

  1. Replace this README.md by README.example.md and customize it, so your team can use the project easily
  2. Make your developers use this project to configure their machine and develop
  3. Let your developers update/create stacks on their own, show them how to test their changes locally
  4. Enjoy :-)

:question: FAQ

Where is the companion CLI documentation? <!-- omit in toc -->

There is no documentation of the CLI since you will probably modify it, or add new commands! To get some help, just run ./tads. Do not hesitate also to have a look at the source into the scripts directory. This CLI is just a wrapper of Terraform, Ansible and Vagrant commands.

What if I don't want to deploy all the stacks locally? <!-- omit in toc -->

Use Ansible tags! Example if you just want to deploy the traefik and myapp stack: ./tads ansible-playbook localhost deploy --tags stack-traefik,stack-myapp.

How to do Continuous Delivery / Deployment? <!-- omit in toc -->

I have not taken the time to develop this feature properly yet. But basically what you can do is:

How to manage persistant storage? <!-- omit in toc -->

This problematic is beyond the scope of this project and depends a lot on your infrastructure / cloud provider. I advise you to have a look at REX-Ray.

This might be a future feature to implement this plugin in the boilerplate.

Contributing

Pull Requests are more than welcome! Please read CONTRIBUTING.md for more details.

Development:

./tads install-dependencies --dev
make lint
make test

Acknowledgments

License

This project is licensed under the MIT license, Copyright (c) 2019 Thomas Vaillant. For more information see LICENSE file.