Home

Awesome

DEPRECATED

This repository is no longer actively maintained.

Secretary

Travis CI Coverage Status Go Report Card

Secretary helps solve the problem of secrets distribution and authorization in highly dynamic container and VM environments. NaCL and AWS Key Management Service (KMS) are supported crypto backends and can be mixed freely.

System Components

Design

In a standalone setup the secretary client performs decryption using either local NaCL keys or by calling the AWS Key Management Service.

In Mesos clusters it may not be desirable to have all slave nodes hold master keys or access KMS directly. A container would instead call secretary daemon which authenticates its signature and performs the decryption in a central place. The secretary daemon queries Marathon to retrieve a containers public keys and determine what secrets it may access.

Encryption is done at configuration time through public keys or by calling KMS. This enables delegation of secrets management to non-admin users and help keep configuration, secrets and software versions together throughout the delivery pipeline.

NaCL Crypto

Secretary uses NaCL boxes through the golang crypto/nacl package. Boxes are encrypted and signed using modern and strong public key cryptography.

Secretary uses 4 distinct key pairs for encrypting secrets and authenticating service instances.

Amazon AWS KMS

Secretary can encrypt and decrypt secrets using AWS Key Management Service which provides hardware security modules (HSMs) for key storage and access control, as well as audit logs of key usage.

KMS coupled with IAM roles and CloudTrail provides access control and audit trails at the instance level. AWS EC2 instances could then use secretary to decrypt secrets embedded into user-data or VM images.

Compared to Centralized Systems?

Benefits of using public key cryptography compared to centrally managed token-based systems like Vault or KeyWhiz

Initial Secret Problem?

In token-based systems a problem occurs where the token that gives access to secrets needs to be securely managed. Any holder of a token can use it to request the plaintext secrets. A token should typically not be checked into source control or it will be available to anyone with access to the config repo.

Secretary mitigates this problem by storing encrypted secrets in the config repo and keeping them encrypted all the way into the runtime environment. Secrets are only ever decrypted inside the container at startup and stored in environment variables visible only to the service.

Secrets can only be accessed by a process that holds both the deploy and service private keys. The deploy key is generated for each single deployment and is available only to specific containers. While the service key is available on slave nodes or embedded into a single application image.

What is needed to get the secrets?

In the runtime env:

Or with access to the config repo:

Getting Started

The master and config key pairs are created once and for each environment using secretary genkeys, which defaults to put keys into the ./keys/ directory. Provision all the keys to each master nodes, including the highly sensitive master-private-key.

Store master-public-key and config private/public key in the config repo together with other environment config and encrypted secrets. This enables users with access to the config repo to encrypt secrets and store them in the config.

Generate a new deploy key for each deployment and insert it into the env element of the Marathon app config. Lighter will perform this step automatically given this config example

someenv/globals.yml - stored in the Lighter config repo

secretary:
  url: 'https://secretary-daemon-loadbalancer:5070'
  master:
    publickey: 'someenv/keys/master-public-key.pem'

someenv/myservice.yml - stored in the Lighter config repo

maven:
  groupid: "com.example"
  artifactid: "myservice"
  classifier: "marathon"
  resolve: [1.0.0,2.0.0)
override:
  env:
    DATABASE_USERNAME: "myservice"
    DATABASE_PASSWORD: "ENC[NACL,NVnSkhxA010D2yOWKRFog0jpUvHQzmkmKKHmqAbHAnz8oGbPEFkDfyKHQHGO7w==]"
    DATABASE_URL: "jdbc:mysql://hostname:3306/schema"

Service Key In Docker Image

The service key is optional but adds extra security. It is required by secretary daemon to authenticate a service if its Marathon app env defines the $SERVICE_PUBLIC_KEY variable.

At build time generate a new service key using e.g. secretary genkeys service and embed the service-private-key into the Docker image. Ensure it's chmod 0600 root-only readable and that a new key is created for each build/release.

The service-public-key needs to be available in the Marathon app env as $SERVICE_PUBLIC_KEY so that secretary daemon can find it when querying Marathon. A solution could be deploying a template JSON app config to a Maven repository and use Lighter to pull it down at deployment time. For example

myservice-1.0.0-marathon.json - deployed to Maven

{
  "id": "/myproduct/mysubsystem/myservice"
  "env" {
    "SERVICE_PUBLIC_KEY": "rEmz7Rt6tUnlC4TKYeNzePYg+p1ePAw4BAtfJAY4zzs="
  }
}

Service Key In VM Image

Generate a service key using e.g. secretary genkeys service and embed the service-private-key into the VM image. Ensure that the service-public-key is available in the Marathon app env as $SERVICE_PUBLIC_KEY. A Lighter config could look like like

someenv/globals.yml - stored in the Lighter config repo

variables:
  secretary.service.publickey: "WvDT+V2fB5ZKkbAmHaFh2XqDXC/veVsl1FKSE/HzxC0="

someenv/myservice.yml

override:
  env:
    SERVICE_PUBLIC_KEY: "%{secretary.service.publickey}"
    SERVICE_PRIVATE_KEY: /service/keys/service-private-key.pem
  container:
    volumes:
     - containerPath: "/service/keys"
       hostPath: "/etc/secretary/service-keys"
       mode: "RO"

Runtime Config

An runtime config automatically expanded by Lighter might look like

{
    "id": "/myproduct/mysubsystem/myservice"
    ...
    "env" {
        "SECRETARY_URL": "https://secretary-daemon-loadbalancer:5070",
        "MASTER_PUBLIC_KEY": "MX+S1xWkxfKlZUvzaEhBLkIVWEkwIrEaD9uKXVC5IGE=",
        "DEPLOY_PUBLIC_KEY": "0k+v11LV3SOr+XiFJ/ug0KcPPhwkXnVirmO65nAd1LI=",
        "DEPLOY_PRIVATE_KEY": "rEmz7Rt6tUnlC4TKYeNzePYg+p1ePAw4BAtfJAY4zzs=",
        "SERVICE_PUBLIC_KEY": "/1fbWGMTaR+lLQJnEsmxdfwWybKOpPQpyWB3FpNmOF4=",
        "SERVICE_PRIVATE_KEY": "/service/keys/service-private-key.pem"
        "DATABASE_USERNAME": "myservice",
        "DATABASE_PASSWORD": "ENC[NACL,SLXf+O9iG48uyojT0Zg30Q8/uRV8DizuDWMWtgL5PmTU54jxp5cTGrYeLpd86rA=]",
        "DATABASE_URL": "jdbc:mysql://hostname:3306/schema"
        "BUCKET_TOKEN": "ENC[KMS,RP+BAwEBCmttc1BheWxvYWQB/4IAAQMBEEVuY3J5cHRlZERhdGFLZXkBCgABBU5vbmNlA==]"
    }
    ...
}

Container Startup Sequence

Docker images should embed the secretary executable. Use it as an entrypoint with `secretary exec --". It will then decrypt all command line arguments and environment variables.

Dockerfile

# Install secretary
ENV SECRETARY_VERSION x.y.z
RUN curl -fsSLo /usr/bin/secretary "https://github.com/meltwater/secretary/releases/download/${SECRETARY_VERSION}/secretary-`uname -s`-`uname -m`" && \
    chmod +x /usr/bin/secretary

ENTRYPOINT ["/usr/bin/secretary", "exec", "--"]
CMD ["/bin/echo", "ENC[NACL,XXXXXX==]"] # This can be replaced in Dockerfiles that inherit from this one

The complete decryption sequence could be described as

  1. client asks the secretary daemon for the DATABASE_PASSWORD secret to be decrypted. This exchange is encrypted/authenticated using master-public-key, deploy-private-key and service-private-key.
  2. daemon retrieves SERVICE_PUBLIC_KEY and DEPLOY_PUBLIC_KEY from Marathon and uses it to authenticate the service.
  3. daemon validates that the service has access to the given secret by checking the env segment of its Marathon app config.
  4. daemon decrypts the secret using master-private-key and authenticates with config-public-key.
  5. daemon re-encrypts the plaintext secret with service-public-key and deploy-public-key, signed with master-private-key before sending it back to the client.
  6. client decrypts the secret using deploy-private-key and service-private-key, authenticating with master-public-key.
  7. client outputs a sh script export DATABASE_PASSWORD='secret' fragment that is sourced into the service environment.

Installation

Place a secretary script in the root of your configuration repo. Replace the SECRETARY_VERSION with a version from the releases page.

#!/bin/bash
set -e

SECRETARY_VERSION="x.y.z"

BASEDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
SECRETARY="$BASEDIR/target/secretary-`uname -s`-`uname -m`-${SECRETARY_VERSION}"

if [ ! -x "$SECRETARY" ]; then
    mkdir -p $(dirname "$SECRETARY")
    curl -sSfLo "$SECRETARY" https://github.com/meltwater/secretary/releases/download/${SECRETARY_VERSION}/secretary-`uname -s`-`uname -m`
    chmod +x "$SECRETARY"
fi

exec "$SECRETARY" $@

Command Line Usage

# Avoid secrets ending up in bash history
set +o history

# Generate master and config key pairs
./secretary genkeys

# Generate example deploy and service key pairs
./secretary genkeys mydeploy myservice

# Generate an example service key
./secretary genkeys myservice

# Encrypt for writing into deployment config files
echo -n secret | ./secretary encrypt

# Encrypt using Amazon AWS KMS
# Note: You need AWS credentials setup in ~/.aws/credentials or envvars $AWS_ACCESS_KEY, $AWS_SECRET_ACCESS_KEY, $AWS_REGION
echo -n secret | ./secretary encrypt --kms-key-id=12345678-1234-1234-1234-123456789012

# Decrypt (requires access to master-private-key)
echo -n <encrypted> | ./secretary decrypt

# Decrypt and substitute encrypted environment variables
eval $(./secretary decrypt -e)

# Decrypt all encrypted substrings in file
cat /path/to/secrets | ./secretary decrypt

Secretary Daemon

Deploy several instances of the secretary daemon to trusted master nodes and create a load balancer in front of them to ensure high availability. The daemon defaults to bind to 5070/tcp. The secretary daemon is stateless and can be load balanced freely.

The daemon has an HTTP health check endpoint at /v1/status that will respond with HTTP 200 OK if all is well. This could be used to point a load balancers health check mechanism at.

TLS Support

In order to enable end to end encryption, you can supply the ssl certificate through environment variable: TLS_KEY_FILE and TLS_CERT_FILE or with the following options:

secretary daemon --tls-key-file <path to key file> --tls-cert-file <path to cert file>

Systemd

Create a Systemd unit file in /etc/systemd/system/secretary.service with contents like below.

[Unit]
Description=Secretary secrets distribution
After=docker.service
Requires=docker.service

[Install]
WantedBy=multi-user.target

[Service]
Environment=IMAGE=meltwater/secretary:latest NAME=secretary

# Allow docker pull to take some time
TimeoutStartSec=600

# Restart on failures
KillMode=none
Restart=always
RestartSec=15

ExecStartPre=-/usr/bin/docker kill ${NAME}
ExecStartPre=-/usr/bin/docker rm ${NAME}
ExecStartPre=-/usr/bin/docker pull ${IMAGE}
ExecStart=/usr/bin/docker run --name=${NAME} \
    -p 5070:5070 \
    -v /etc/secretary/master-keys:/keys \
    -e MARATHON_URL=http://marathon-host:8080 \
    $IMAGE

ExecStop=/usr/bin/docker stop $NAME

Puppet Hiera

Using the garethr-docker module

classes:
  - docker::run_instance

docker::run_instance:
  'secretary':
    image: 'meltwater/secretary:latest'
    ports:
      - '5070:5070'
    volumes:
      - '/etc/secretary/master-keys:/keys'
    env:
      - 'MARATHON_URL=http://marathon-host:8080'

Amazon AWS KMS

When interacting with KMS to encrypt or decrypt secrets you or the instance needs access to the AWS API and the specific KMS key. Key access is managed via the AWS IAM console and can be both on the KMS API level as well as fine grained permissions for each key.

For workstation access to encrypt secrets you typically need AWS credentials setup in ~/.aws/credentials or environment variables $AWS_ACCESS_KEY, $AWS_SECRET_ACCESS_KEY and $AWS_REGION so that secretary can interact with the KMS API.

AWS EC2 instances should use IAM roles rather than access keys, to grant them access to the KMS API and the specific KMS keys.

Secrets in user-data

When using CoreOS cloud-config and passing secrets in the user-data section.

In the examples replace the SECRETARY_VERSION with a version from the releases page. You also need to replace the e59c5534e4e6fb3c2ad0d3c075d9e2fa664889b9 sha1sum with one that is calculated from the exact version you intend to use. This can be done like

curl -sSL https://github.com/meltwater/secretary/releases/download/${SECRETARY_VERSION}/secretary-Linux-x86_64 | sha1sum -

Embedded Secretary binary

This CoreOS user-data example writes out /etc/environment.encrypted with encrypted secrets and forwards them into a Docker container as encrypted environment variables. The Docker image embeds the secretary binary and its startup script decrypts the environment using eval $(secretary decrypt -e)

#cloud-config
---
coreos:
  units:
  - name: myservice.service
    command: start
    content: |
      [Unit]
      After=docker.service decrypt.service
      Requires=docker.service decrypt.service

      [Install]
      WantedBy=multi-user.target

      [Service]
      EnvironmentFile=/etc/environment.encrypted
      Environment=IMAGE=myservice:latest NAME=myservice

      # Allow docker pull to take some time
      TimeoutStartSec=600

      # Restart on failures
      KillMode=none
      Restart=always
      RestartSec=15

      # Start Docker container
      ExecStartPre=-/usr/bin/docker kill ${NAME}
      ExecStartPre=-/usr/bin/docker rm ${NAME}
      ExecStartPre=-/bin/sh -c 'if ! docker images | tr -s " " : | grep "^${IMAGE}:"; then docker pull "${IMAGE}"; fi'      
      ExecStart=/usr/bin/docker run --name ${NAME} \
        -e "DATABASE_PASSWORD=${DATABASE_PASSWORD}" \
        -e "API_KEY=${API_KEY}" \
        ${IMAGE}

write_files:
  - path: "/etc/environment.encrypted"
    permissions: "0600"
    owner: "root"
    content: |
      DATABASE_PASSWORD=ENC[KMS,RP+BAwEBCmttc1BheWxvYWQB/4IAAQMBEEVuY3J5cHRlZERhdGFLZXkBCgABBU5vbmNlA==]
      API_KEY=ENC[KMS,SLXf+O9iG48uyojT0Zg30Q8/uRV8DizuDWMWtgL5PmTU54jxp5cTGrYeLpd86rA==]

External Secretary binary

This CoreOS user-data example writes out /etc/environment.encrypted with encrypted secrets. Then uses secretary and KMS to decrypt them and forwards the secrets into a Docker container as unencrypted environment variables.

#cloud-config
---
coreos:
  units:
  - name: myservice.service
    command: start
    content: |
      [Unit]
      After=docker.service decrypt.service
      Requires=docker.service decrypt.service

      [Install]
      WantedBy=multi-user.target

      [Service]
      EnvironmentFile=/etc/environment.encrypted
      Environment=IMAGE=myservice:latest NAME=myservice SECRETARY_VERSION=x.y.z

      # Allow docker pull to take some time
      TimeoutStartSec=600

      # Restart on failures
      KillMode=none
      Restart=always
      RestartSec=15

      # Download and verify signature of secretary binary
      ExecStartPre=/bin/sh -c '\
        if [ ! -f /tmp/secretary ]; then \
          curl -sSLo /tmp/secretary https://github.com/meltwater/secretary/releases/download/${SECRETARY_VERSION}/secretary-Linux-x86_64 && \
          chmod +x /tmp/secretary; \
        fi'
      ExecStartPre=/bin/sh -c 'echo e59c5534e4e6fb3c2ad0d3c075d9e2fa664889b9 /tmp/secretary | sha1sum -c -'

      # Start Docker container
      ExecStartPre=-/usr/bin/docker kill ${NAME}
      ExecStartPre=-/usr/bin/docker rm ${NAME}
      ExecStartPre=-/bin/sh -c 'if ! docker images | tr -s " " : | grep "^${IMAGE}:"; then docker pull "${IMAGE}"; fi'      
      ExecStart=/bin/sh -c 'eval $(/tmp/secretary decrypt -e) && docker run --name myservice \
        -e "DATABASE_PASSWORD=${DATABASE_PASSWORD}" \
        -e "API_KEY=${API_KEY}" \
        myservice:latest'

write_files:
  - path: "/etc/environment.encrypted"
    permissions: "0600"
    owner: "root"
    content: |
      DATABASE_PASSWORD=ENC[KMS,RP+BAwEBCmttc1BheWxvYWQB/4IAAQMBEEVuY3J5cHRlZERhdGFLZXkBCgABBU5vbmNlA==]
      API_KEY=ENC[KMS,SLXf+O9iG48uyojT0Zg30Q8/uRV8DizuDWMWtgL5PmTU54jxp5cTGrYeLpd86rA==]