Home

Awesome

DEPRECATED

This project has been deprecated.

Storing and managing secrets in source code is subject to a number of risks and drawbacks compared to proper solutions for managing secrets. While this and similar related projects utilizing the operator pattern allow to leverage built-in functionality of Kubernetes Secrets and might look attractive and even convenient at first glance, the following reasons should encourage people using more robust solutions (in no particular order):

Secreter

Cryptography API

Core cryptography API is freezed and is unlikely to be changed. It is open for extension and adding more providers. Thorough independent security audit is badly needed and welcome!

Kubernetes API and CLI

The basic features have been completed, and while no breaking API changes are currently planned, the API can change in a backwards incompatible way before the project is declared stable.

Overview

Secreter consists of two components:

Secrets encrypted with the Secreter CLI can only be decrypted by the Encrypted Secrets Controller. Controller watches for new and changed EncryptedSecret objects and decrypts them on the fly creating Secret objects accordingly.

When used in conjunction with encrypting Secret Data at Rest it will create a perfect solution where the actual secrets are known in their raw format only to the workloads they are explicitly bound to inside the cluster. Outside of it, kube-apiserver would store the secrets encrypted in etcd, secreter allows storing encrypted secrets in VCS, CI/CD, et al.

Features

Getting Started

Deploying the Secreter operator

  1. Create all the necessary resources and deploy the operator:

    kubectl apply -Rf deploy

  2. Verify that the operator is running:

    $ kubectl -n secreter get deployment
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
secreter   1/1     1            1           5m

  3. Apply an example Secret Encryption Config:

    kubectl apply -f example/k8s_v1alpha1_secretencryptionconfig_cr.yaml

Getting CLI

Official Secreter CLI binaries can be found on the Releases page.

Configuring Secreter

The SecretEncryptionConfig CRD is used for configuring Secreter.

Important considerations

Configuration overview

Let's go through the sample SecretEncryptionConfig:

apiVersion: k8s.amaiz.com/v1alpha1
kind: SecretEncryptionConfig
metadata:
  name: example
  namespace: secreter
providers:
  - name: primary
    curve25519:
      keyStore:
        name: my-curve25519-keystore
  - name: secondary
    gcpkms:
      projectID: my-kms-project
      locationID: global
      keyRingID: my-keyring
      cryptoKeyID: my-key
      credentials:
        - secretKeyRef:
            name: my-kms-project-creds
            key: creds.json
status:
  publicKey: 2fe57da347cd62431528daac5fbb290730fff684afc4cfc2ed90995f58cb3b74

namespace: secreter matches the namespace where the secreter operator has been deployed.

providers is an ordered list. Only one provider may be specified per entry.

status.publicKey contains public key (or any other public data required for encryption).

In the above config there are two providers configured , named primary and secondary for simplicity. The first element is a primary provider that is supposed to be used for encryption. primary provider's section configures the built-in curve25519 provider. Name of the keyStore is the name of the Kubernetes secret containing public and primary keys. Operator will create the key store if it does not find one. The secondary provider contains the gcpkms configuration with the reference to the secret with the the service account credentials. credentials is an ordered list allowing you to seamlessly rotate the credentials when needed.

Configuring providers

Curve25519
<...>
providers:
  - name: primary
    curve25519:
      keyStore:
        name: my-curve25519-keystore

Operator will create a Kubernetes Secret my-curve25519-keystore with the keystore if it does not exist. No configuration other than that is needed.

GCP KMS

To add a GCP KMS provider you will need to pass projectID, locationID, keyRingID and cryptoKeyID. Those are required for symmetric encryption. In order to use asymmetric encryption be sure to pass cryptoKeyVersion as well. Please refer to the GCP KMS Object hierarchy for details.

credentials hold the GCP service account credentials. Once you've created and obtained the credentials json create a Kubernetes Secret in the operator namespace:

$ kubectl -n secreter create secret generic my-kms-project-creds --from-file creds.json
secret/my-kms-project-creds created

Once created you can refer to this secret within the secret encryption config. credentials is a list so that you can refer to multiple service account credentials.

<...>
providers:
  - name: my-gcp
    gcpkms:
      projectID: my-kms-project
      locationID: global
      keyRingID: my-keyring
      cryptoKeyID: my-key
      cryptoKeyVersion: 1 # optional field, use with asymmetric encryption only.
      credentials:
        - secretKeyRef:
            name: my-kms-project-creds
            key: creds.json
  <...>

To be able to encrypt secrets using GCP KMS you will need to:

  1. install the Google Cloud SDK.

  2. acquire new Application Default Credentials:

    gcloud auth application-default login
AWS KMS

In order to add AWS KMS provider you will need to pass keyID. You can use either ARN or the AWS KMS key alias.

credentials hold the list of AWS security credentials. Once you've created and obtained the access key ID (something like AKIAIOSFODNN7EXAMPLE) and a secret access key (something like wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY) create a Kubernetes Secret in the operator namespace:

$ kubectl -n secreter create secret generic my-aws-creds --from-literal aws_access_key_id=AKIAIOSFODNN7EXAMPLE --from-literal aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
secret/my-kms-project-creds created

Once created you can refer to this secret within the secret encryption config. credentials is a list so that you can refer to multiple service account credentials.

<...>
providers:
  - name: example-awskms
    awskms:
      keyID: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab
      credentials:
        - accessKeyID:
            secretKeyRef:
              key: aws_access_key_id
              name: my-aws-creds
          secretAccessKey:
            secretKeyRef:
              key: aws_secret_access_key
              name: my-aws-creds
  <...>

To be able to encrypt secrets using AWS KMS you will need to obtain AWS credentials for your account and put them into ~/.aws/credentials, e.g.:

[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Key rotation

Regularly rotating keys is a security best practice and there is a number of ways keys can be rotated for the selected Secret encryption config.

Important note: rotating keys will not re-encrypt existing encrypted secrets. Please refer to the "Encrypting Secrets" section for details about how to re-encrypt a secret.

Provider rotation

The easiest way to rotate keys is to add a new primary provider:

apiVersion: k8s.amaiz.com/v1alpha1
kind: SecretEncryptionConfig
metadata:
  name: example
  namespace: secreter
providers:
  - name: new-primary # <- add as the first element to make primary
    curve25519:
      keyStore:
        name: my-new-curve25519-keystore
  - name: my-previous-primary-provider
    curve25519:
      keyStore:
        name: my-curve25519-keystore
  <...>

This will create a new curve25519 provider named new-primary. It will be used for encrypting new data and updating existing encrypted secrets. The second provider, named my-previous-primary-provider, will still be used for decryption. This is the recommended way to manually rotate keys for the curve25519 provider.

Curve25519 automatic key rotation

Not implemented. To be announced soon.

KMS providers

Some KMS providers (like Google Cloud KMS) support automatic key rotation.

Regularly rotating credentials for KMS providers is important as well.

Below is an example flow for rotating GCP KMS credentials.

  1. Create a secret in the secreter namespace with the newly created credentials file:

    $ kubectl -n secreter create secret generic my-new-creds --from-file creds.json
secret/my-new-creds created

  2. Add a reference to the newly created secret in your kms provider configuration section and comment out the previous key:

    apiVersion: k8s.amaiz.com/v1alpha1
kind: SecretEncryptionConfig
metadata:
  name: example
  namespace: secreter
providers:
- name: primary
  curve25519:
    keyStore:
      name: my-curve25519-keystore
- name: secondary
  gcpkms:
    projectID: my-kms-project
    locationID: global
    keyRingID: my-keyring
    cryptoKeyID: my-key
    credentials:
    - secretKeyRef:
        name: my-new-creds
        key: creds.json
    # - secretKeyRef:
    #     name: my-kms-project-creds
    #     key: creds.json

  3. Apply the new config and check everything is working as expected.

  4. Once you are happy with the new credentials feel free to delete the commented section and the old my-kms-project-creds secret:

    $ kubectl delete secret my-kms-project-creds
secret "my-kms-project-creds" deleted

Encrypting Secrets

Secrets can be encrypted in a number of ways. Below you will find a couple of examples.

The same techniques can be applied to re-encrypt secrets in case the primary provider has changed or the primary key has been rotated.

Please run secreter help encrypt for detailed description and examples.

Encrypted secrets decryption

Encrypted secrets have a one-to-one mapping with the corresponding secrets. Treat an EncryptedSecret resource as the encrypted form of the Secret with the same name, namespace, .metadata.labels, data map and type.

EncryptedSecret does not have a stringData map though. If the Secret to be encrypted contains this stringData map the data it consists of will be encrypted and put into the data map.

The data map of an EncryptedSecret will become the data map of the corresponding Secret. Encrypted Secret resource will be set as the owner for the decrypted Secret. Values of this map are decrypted on the best effort basis. In case any value fails to get decrypted it will appear in the encrypted secret's status.failedToDecrypt map with the reason for failure. Once all the values of the encrypted secret are decrypted its status will reflect this state:

$ kubectl get encryptedsecrets.k8s.amaiz.com
NAME   DECRYPTED
test   true

Updating Encrypted Secrets

Updating encrypted secrets is similar to the encryption operation but the config passed should match the config used during initial encryption.

Please run secreter help update for detailed description and examples.

If it is needed to use different secret encryption config then the recommended way is to encrypt the secret using the desired config and to replace the encrypted secret.

Uninstalling Secreter operator

The most important thing to bear in mind before deletion is that all of the decrypted secrets are owned by their corresponding EncryptedSecret objects. It is recommended to make a backup of the essentially important secrets before proceeding.

  1. Scale down the operator:

    kubectl -n secreter scale deployment secreter --replicas 0

  2. Remove the owner references for each secret, e.g.:

    $ kubectl patch secret test --type json -p '[{"op": "remove", "path": "/metadata/ownerReferences"}]'
secret/test patched

  3. Delete the namespace with the operator:

    $ kubectl delete namespace secreter
namespace "secreter" deleted

  4. Delete CRDs. Kubernetes will garbage collect all operator-managed resources:

    $ kubectl delete crd secretencryptionconfigs.k8s.amaiz.com encryptedsecrets.k8s.amaiz.com
customresourcedefinition.apiextensions.k8s.io "secretencryptionconfigs.k8s.amaiz.com" deleted
customresourcedefinition.apiextensions.k8s.io "encryptedsecrets.k8s.amaiz.com" deleted

Design and goals

The main rationale behind starting this project was to create a Kubernetes-native set of tools for managing secrets in a secure and protected fashion. Kubernetes secrets contain base64-encoded byte arrays which makes it impossible to store them along with other resources and manage them in a declarative way. Some investigation showed that there are various approaches to managing secrets in Kubernetes world but all of them are complex multi-step systems implying a lot of manual preparatory work hence prone to introducing fragility and human errors. Many approaches suggest encrypting whole files whereas all is needed is encrypting the data map values.

After inspecting the already implemented feature of encrypting data at rest it became clear that encrypting secrets on client side together with proper RBAC and encrypting data at rest would create a complete straightforward solution for securing and managing Kubernetes secrets where secrets in their decrypted way are known only to certain workloads (via explicit referencing) or to certain people/service accounts via RBAC.

TODO: add a diagram showing relations and workflow for better understanding of the concept.

Cryptography overview

One of the major goals was to avoid legacy crypto algorithms as well as new shiny but lacking good track record and assessment approaches and to keep the number of options to a minimum to make things simple and clear.

All secret data is encrypted using hybrid (or envelope) encryption.

AEAD is used for encrypting the data itself. Currently only XChaCha20-Poly1305 is used for AEAD. It is designed to avoid key and nonce reuse. Key and nonce are derived for input key material using HKDF. Please refer to the package documentation for detailed description.

All providers are supposed to use XChaCha20-Poly1305 for encrypting raw secret data.

Providers: