Home

Awesome

git2params

A simple tool to publish and update your application configuration from a Git repo to the SSM Parameter Store.

Why?

SSM Parameter Store has several nice properties for configuration storage:

Features

Requirements

Before you can use git2params, you need the following installed:

How does it work?

git2params

  1. Developer pushes changes to a git repo

  2. The repo triggers a webhook to an API Gateway endpoint

  3. API Gateway triggers a Lambda function, which:

    1. Reads the SSH key from SSM Parameter Store with path $SYSTEM_PARAM_PREFIX/ssh-key

    2. Clones the git repository locally

    3. Reads latest revision ID from SSM Parameter Store with path $SYSTEM_PARAM_PREFIX/revision:

    4. If revision ID does not exist in the store:

      1. Uploads all files from git to the store with path $GIT_REPO_NAME/path/to/key

    5. If revision ID exists:

      1. Take git diff of commits: retreived revision ID versus latest revision ID

      2. Uploads all added and modified files to the store

      3. Deletes all removed files from the store

  4. Sends SNS message to the configured topic

All parametes are encryped with the default SSM key. For each parameter, the Description field contains the ID of the commit that last modified this file.

Usage

  1. Clone this repository

  2. Create your configurations file based on parameters_example.yml

  3. Define environments in the serverless.yml file by adding custom keys like

    prod: ${file(./prod-config.yml)
testing: ${file(./testing-config.yml)}

    specifing the path to config that you created in previous step

    If you don't know what you are doing, please do not modify any other value in serverless.yml as it can break your deployment, IAM permissions, etc.

  4. Generate an SSH key with ssh-keygen -t rsa -b 4096 -f ~/.ssh/my_git_repo

  5. Add the generated public key to your git repository as a read-only Deployment key. See how to add deployment key for Bitbucket/Github/Gitlab.

  6. Upload the generated private key to the EC2 Parameters:

    aws ssm put-parameter \
    --name /git2params/repo_name/ssh-key \
    --value "$(cat ~/.ssh/my_git_repo)" \
    --overwrite \
    --type SecureString

    The SSH key must be uploaded via CLI or API because the web UI breaks new lines, invaliditing the key.

  7. Deploy by running make deploy ENVIRONMENT=prod, where ENVIRONMENT should match the keys defined in Step 3

  8. If the deployment is successful, you'll see an API Gateway endpoint in the response:

    endpoints:
    POST - https://xxxxxxxx.execute-api.<region>.amazonaws.com/<environment>/params/update

  9. Copy the endpoint and configure the webhook on your git repository (Webhook configurations for GitHub, BitBucket and GitLab).

WARNING: During first run if your git repository contains a lot of files (or you added many files in one push), function execution can fail with a timeout, without processing all the files. Therefore, it's recommended to execute first run locally with make local ENVIRONMENT=prod (IAM user with all the needed permissions required).

SNS Notifications

{
    "added": {
        "errors": [
            {
                "Commit": "9561d3a405c372eec509d15ad52d9cd77b9c3119",
                "CommitURL": "https://bitbucket.org/user/my-configs/commits/9561d3a405c372eec509d15ad52d9cd77b9c3119",
                "Error": "YAML format problem: mapping values are not allowed here\n in \"<string>\", line 1, column 5:\n c: x: f:\n ^",
                "Time": "Fri Aug 25 14:04:06 2017",
                "Key": "/my-configs/test.yml",
                "KeyURL": "https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Parameters:Name=[Equals]/my-configs/test.yml",
                "Author": "user <user@example.com>",
                "Message": "commit message"
            }
        ],
        "success": [
            {
                "Commit": "e9cd55dad2c5c0d77233b9c58e61fb6126949bcb",
                "CommitURL": "https://bitbucket.org/user/my-configs/commits/e9cd55dad2c5c0d77233b9c58e61fb6126949bcb",
                "Time": "Fri Aug 11 13:21:10 2017",
                "Key": "/my-configs/test",
                "KeyURL": "https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Parameters:Name=[Equals]/my-configs/test",
                "Author": "user <user@example.com>",
                "Message": "commit message"
            }
        ]
    },
    "modified": {
        "errors": [
            {
                "Commit": "ccf5d88ef6eb5940cd1d62f2e96b7b69e82354de",
                "CommitURL": "https://bitbucket.org/user/my-configs/commits/ccf5d88ef6eb5940cd1d62f2e96b7b69e82354de",
                "Error": "JSON format problem: Expecting , delimiter: line 3 column 2 (char 16)",
                "Time": "Fri Aug 25 14:00:32 2017",
                "Key": "/my-configs/test.json",
                "KeyURL": "https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Parameters:Name=[Equals]/my-configs/test.json",
                "Author": "user <user@example.com>",
                "Message": "commit message"
            }
        ],
        "success": [
            {
                "Commit": "e9cd55dad2c5c0d77233b9c58e61fb6126949bcb",
                "CommitURL": "https://bitbucket.org/user/my-configs/commits/e9cd55dad2c5c0d77233b9c58e61fb6126949bcb",
                "Time": "Fri Aug 11 13:21:10 2017",
                "Key": "/my-configs/test_new",
                "KeyURL": "https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Parameters:Name=[Equals]/my-configs/test_new",
                "Author": "user <user@example.com>",
                "Message": "commit message"
            }
        ]
    },
    "removed": {
        "errors": [
            "/my-configs/key1",
            "/my-configs/key2"
        ],
        "success": [
            "/my-configs/key3",
            "/my-configs/key4"
        ]
    },
    "type": "git2params"
}

Known Limitations

TODO