Home

Awesome

vals

vals is a tool for managing configuration values and secrets.

It supports various backends including:

ToC:

Usage

CLI

vals is a Helm-like configuration "Values" loader with support for various sources and merge strategies

Usage:
  vals [command]

Available Commands:
  eval          Evaluate a JSON/YAML document and replace any template expressions in it and prints the result
  exec          Populates the environment variables and executes the command
  env           Renders environment variables to be consumed by eval or a tool like direnv
  get           Evaluate a string value passed as the first argument and replace any expressiosn in it and prints the result
  ksdecode      Decode YAML document(s) by converting Secret resources' "data" to "stringData" for use with "vals eval"
  version       Print vals version

Use "vals [command] --help" for more information about a comman

vals has a collection of providers that each an be referred with a URI scheme looks ref+<TYPE>.

For this example, use the Vault provider.

Let's start by writing some secret value to Vault:

$ vault kv put secret/foo mykey=myvalue

Now input the template of your YAML and refer to vals' Vault provider by using ref+vault in the URI scheme:

$ VAULT_TOKEN=yourtoken VAULT_ADDR=http://127.0.0.1:8200/ \
  echo "foo: ref+vault://secret/data/foo?proto=http#/mykey" | vals eval -f -

Voila! vals, replacing every reference to your secret value in Vault, produces the output looks like:

foo: myvalue

Which is equivalent to that of the following shell script:

VAULT_TOKEN=yourtoken  VAULT_ADDR=http://127.0.0.1:8200/ cat <<EOF
foo: $(vault kv get -format json secret/foo | jq -r .data.data.mykey)
EOF

Save the YAML content to x.vals.yaml and running vals eval -f x.vals.yaml does produce output equivalent to the previous one:

foo: myvalue

Helm

Use value references as Helm Chart values, so that you can feed the helm template output to vals -f - for transforming the refs to secrets.

$ helm template mysql-1.3.2.tgz --set mysqlPassword='ref+vault://secret/data/foo#/mykey' | vals ksdecode -o yaml -f - | tee manifests.yaml
apiVersion: v1
kind: Secret
metadata:
  labels:
    app: release-name-mysql
    chart: mysql-1.3.2
    heritage: Tiller
    release: release-name
  name: release-name-mysql
  namespace: default
stringData:
  mysql-password: ref+vault://secret/data/foo#/mykey
  mysql-root-password: vZQmqdGw3z
type: Opaque

This manifest is safe to be committed into your version-control system(GitOps!) as it doesn't contain actual secrets.

When you finally deploy the manifests, run vals eval to replace all the refs to actual secrets:

$ cat manifests.yaml | ~/p/values/bin/vals eval -f - | tee all.yaml
apiVersion: v1
kind: Secret
metadata:
    labels:
        app: release-name-mysql
        chart: mysql-1.3.2
        heritage: Tiller
        release: release-name
    name: release-name-mysql
    namespace: default
stringData:
    mysql-password: myvalue
    mysql-root-password: 0A8V1SER9t
type: Opaque

Finally run kubectl apply to apply manifests:

$ kubectl apply -f all.yaml

This gives you a solid foundation for building a secure CD system as you need to allow access to a secrets store like Vault only from servers or containers that pulls safe manifests and runs deployments.

In other words, you can safely omit access from the CI to the secrets store.

Go

import "github.com/helmfile/vals"

secretsToCache := 256 // how many secrets to keep in LRU cache
runtime, err := vals.New(secretsToCache)
if err != nil {
  return nil, err
}

valsRendered, err := runtime.Eval(map[string]interface{}{
    "inline": map[string]interface{}{
        "foo": "ref+vault://127.0.0.1:8200/mykv/foo?proto=http#/mykey",
        "bar": map[string]interface{}{
            "baz": "ref+vault://127.0.0.1:8200/mykv/foo?proto=http#/mykey",
        },
    },
})

Now, vals contains a map[string]interface{} representation of the below:

cat <<EOF
foo: $(vault read mykv/foo -o json | jq -r .mykey)
  bar:
    baz: $(vault read mykv/foo -o json | jq -r .mykey)
EOF

Expression Syntax

vals finds and replaces every occurrence of ref+BACKEND://PATH[?PARAMS][#FRAGMENT][+] URI-like expression within the string at the value position with the retrieved secret value.

BACKEND is the identifier of one of the supported backends.

PATH is the backend-specific path for the secret to be retried.

PARAMS is key-value pairs where the key and the value are combined using the intermediate "=" character while key-value pairs are combined using "&" characters. It's supposed to be the "query" component of the URI as defined in RFC3986.

FRAGMENT is a path-like expression that is used to extract a single value within the secret. When a fragment is specified, vals parse the secret value denoted by the PATH into a YAML or JSON object, and traverses the object following the fragment, and uses the value at the path as the final secret value. It's supposed to be the "fragment" componet of the URI as defined in RFC3986.

Finally, the optional trailing + is the explicit "end" of the expression. You usually don't need it, as if omitted, it treats anything after ref+ and before the new-line or the end-of-line as an expression to be evaluated. An explicit + is handy when you want to do a simple string interpolation. That is, foo ref+SECRET1+ ref+SECRET2+ bar evaluates to foo SECRET1_VALUE SECRET2_VALUE bar.

Although we mention the RFC for the sake of explanation, PARAMS and FRAGMENT might not be fully RFC-compliant as, under the hood, we use a simple regexp that seemed to work for most of use-cases.

The regexp is defined as DefaultRefRegexp in our code base.

Please see the relevant unit test cases for exactly which patterns are supposed to work with vals.

Supported Backends

Please see pkg/providers for the implementations of all the providers. The package names corresponds to the URI schemes.

Vault

Authentication

The auth_method or VAULT_AUTH_METHOD envar configures how vals authenticates to HashiCorp Vault. Currently only these options are supported:

Examples:

AWS

There are four providers for AWS:

Both provider have support for specifying AWS region and profile via envvars or options:

AWS SSM Parameter Store

The first form result in a GetParameter call and result in the reference to be replaced with the value of the parameter.

The second form is handy but fairly complex.

For the second form, you can optionally specify recursive=true to enable the recursive option of the GetParametersByPath API.

Let's say you had a number of parameters like:

NAME        VALUE
/foo/bar    {"BAR":"VALUE"}
/foo/bar/a  A
/foo/bar/b  B

On the other hand,

AWS Secrets Manager

The third form allows you to reference a secret in another AWS account (if your cross-account secret permissions are configured).

Examples:

AWS S3

Examples:

AWS KMS

Decrypts the URL-safe base64-encoded ciphertext using AWS KMS. Note that URL-safe base64 encoding is the same as "traditional" base64 encoding, except it uses _ and - in place of / and +, respectively. For example, to get a URL-safe base64-encoded ciphertext using the AWS CLI, you might run

aws kms encrypt \
  --key-id alias/example \
  --plaintext $(echo -n "hello, world" | base64 -w0) \
  --query CiphertextBlob \
  --output text |
  tr '/+' '_-'

Valid values for alg include:

Valid value formats for key include:

For ciphertext encrypted with a symmetric key, the key field may be omitted. For ciphertext encrypted with a key in your own account, a plain key id or alias can be used. If the encryption key is from another AWS account, you must use the key or alias ARN.

Use the context parameter to optionally specify the encryption context used when encrypting the ciphertext. Format it by URL-encoding the JSON representation of the encryption context. For example, if the encryption context is {"foo":"bar","hello":"world"}, then you would represent that as context=%7B%22foo%22%3A%22bar%22%2C%22hello%22%2C%22world%22%7D.

Examples:

Google GCS

Examples:

GCP Secrets Manager

Examples:

NOTE: Got an error like expand gcpsecrets://project/secret-name?version=1: failed to get secret: rpc error: code = PermissionDenied desc = Request had insufficient authentication scopes.?

In some cases like you need to use an alternative credentials or project, you'll likely need to set GOOGLE_APPLICATION_CREDENTIALS and/or GCP_PROJECT envvars.

GCP KMS

Decrypts the URL-safe base64-encoded ciphertext using GCP KMS. Note that URL-safe base64 encoding is the same as "traditional" base64 encoding, except it uses _ and - in place of / and +, respectively. For example, to get a URL-safe base64-encoded ciphertext using the GCP CLI, you might run

echo test | gcloud kms encrypt \
  --project myproject \
  --location global \
  --keyring mykeyring \
  --key mykey \
  --plaintext-file - \
  --ciphertext-file - \
  | base64 -w0 \
  | tr '/+' '_-'

Google Sheets

Examples:

Terraform (tfstate)

Options:

aws_profile: If non-empty, vals tries to let tfstate-lookup to use the specified AWS profile defined in the well-known ~/.credentials file. az_subscription_id: If non-empty, vals tries to let tfstate-lookup to use the specified Azure Subscription ID.

Examples:

When you're using terraform-aws-vpc to define a module "vpc" resource and you wanted to grab the first vpc ARN created by the module:

$ tfstate-lookup -s ./terraform.tfstate module.vpc.aws_vpc.this[0].arn
arn:aws:ec2:us-east-2:ACCOUNT_ID:vpc/vpc-0cb48a12e4df7ad4c

$ echo 'foo: ref+tfstate://terraform.tfstate/module.vpc.aws_vpc.this[0].arn' | vals eval -f -
foo: arn:aws:ec2:us-east-2:ACCOUNT_ID:vpc/vpc-0cb48a12e4df7ad4c

You can also grab a Terraform output by using output.OUTPUT_NAME like:

$ tfstate-lookup -s ./terraform.tfstate output.mystack_apply

which is equivalent to the following input for vals:

$ echo 'foo: ref+tfstate://terraform.tfstate/output.mystack_apply' | vals eval -f -

Remote backends like S3, GCS and AzureRM blob store are also supported. When a remote backend is used in your terraform workspace, there should be a local file at ./terraform/terraform.tfstate that contains the reference to the backend:

{
    "version": 3,
    "serial": 1,
    "lineage": "f1ad69de-68b8-9fe5-7e87-0cb70d8572c8",
    "backend": {
        "type": "s3",
        "config": {
            "access_key": null,
            "acl": null,
            "assume_role_policy": null,
            "bucket": "yourbucketnname",

Just specify the path to that file, so that vals is able to transparently make the remote state contents available for you.

Terraform in GCS bucket (tfstategs)

Examples:

It allows to use Terraform state stored in GCS bucket with the direct URL to it. You can try to read the state with command:

$ tfstate-lookup -s gs://bucket-with-terraform-state/terraform.tfstate google_compute_disk.instance.source_image_id
5449927740744213880

which is equivalent to the following input for vals:

$ echo 'foo: ref+tfstategs://bucket-with-terraform-state/terraform.tfstate/google_compute_disk.instance.source_image_id' | vals eval -f -

Terraform in S3 bucket (tfstates3)

Examples:

It allows to use Terraform state stored in AWS S3 bucket with the direct URL to it. You can try to read the state with command:

$ tfstate-lookup -s s3://bucket-with-terraform-state/terraform.tfstate module.vpc.aws_vpc.this[0].arn
arn:aws:ec2:us-east-2:ACCOUNT_ID:vpc/vpc-0cb48a12e4df7ad4c

which is equivalent to the following input for vals:

$ echo 'foo: ref+tfstates3://bucket-with-terraform-state/terraform.tfstate/module.vpc.aws_vpc.this[0].arn' | vals eval -f -

Terraform in AzureRM Blob storage (tfstateazurerm)

Examples:

It allows to use Terraform state stored in Azure Blob storage given the resource group, storage account, container name and blob name. You can try to read the state with command:

$ tfstate-lookup -s azurerm://my_rg/my_storage_account/terraform-backend/unique.terraform.tfstate output.virtual_network.name

which is equivalent to the following input for vals:

$ echo 'foo: ref+tfstateazurerm://my_rg/my_storage_account/terraform-backend/unique.terraform.tfstate/output.virtual_network.name' | vals eval -f -

Terraform in Terraform Cloud / Terraform Enterprise (tfstateremote)

Examples:

It allows to use Terraform state stored in Terraform Cloud / Terraform Enterprise given the resource group, the organization and the workspace. You can try to read the state with command (with exported variable TFE_TOKEN):

$ tfstate-lookup -s remote://app.terraform.io/myorg/myworkspace output.virtual_network.name

which is equivalent to the following input for vals:

$ echo 'foo: ref+tfstateremote://app.terraform.io/myorg/myworkspace/output.virtual_network.name' | vals eval -f -

SOPS

Note: When using an inline base64-encoded sops "file", be sure to use URL-safe Base64 encoding. URL-safe base64 encoding is the same as "traditional" base64 encoding, except it uses _ and - in place of / and +, respectively. For example, you might use the following command: sops -e <(echo "foo") | base64 -w0 | tr '/+' '_-'

Examples:

Echo

Echo provider echoes the string for testing purpose. Please read the original proposal to get why we might need this.

Examples:

File

File provider reads a local text file, or the value for the specific path in a YAML/JSON file.

Examples:

Azure Key Vault

Retrieve secrets from Azure Key Vault. Path is used to specify the vault and secret name. Optionally a specific secret version can be retrieved.

VAULT-NAME is either a simple name if operating in AzureCloud (vault.azure.net) or the full endpoint dns name when operating against non-default azure clouds (US Gov Cloud, China Cloud, German Cloud). Examples:

Authentication

Vals aquires Azure credentials though Azure CLI or from environment variables. The easiest way is to run az login. Vals can then aquire the current credentials from az without further set up.

Other authentication methods require information to be passed in environment variables. See Azure SDK docs and auth.go for the full list of supported environment variables.

For example, if using client credentials the required env vars are AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID and possibly AZURE_ENVIRONMENT in case of accessing an Azure GovCloud.

The order in which authentication methods are checked is:

  1. Client credentials
  2. Client certificate
  3. Username/Password
  4. Azure CLI or Managed identity (set environment AZURE_USE_MSI=true to enabled MSI)

EnvSubst

Environment variables substitution.

Examples:

GitLab Secrets

For this provider to work you require an access token exported as the environment variable GITLAB_TOKEN.

Examples:

1Password

For this provider to work a working service account token is required. The following env var has to be configured:

1Password is organized in vaults and items. An item can have multiple fields with or without a section. Labels can be set on fields and sections. Vaults, items, sections and labels can be accessed by ID or by label/name (and IDs and labels can be mixed and matched in one URL).

If a section does not have a label the field is only accessible via the section ID. This does not hold true for some default fields which may have no section at all (e.g.username and password for a Login item).

See Secret reference syntax for more information.

Caution: vals-expressions are parsed as URIs. For the 1Password provider the host component of the URI identifies the vault. Therefore vaults containing certain characters not allowed in the host component (e.g. whitespaces, see RFC-3986 for details) can only be accessed by ID.

Examples:

1Password Connect

For this provider to work you require a working and accessible 1Password connect server. The following env vars have to be configured:

1Password is organized in vaults and items. An item can have multiple fields with or without a section. Labels can be set on fields and sections. Vaults, items, sections and labels can be accessed by ID or by label/name (and IDs and labels can be mixed and matched in one URL).

If a section does not have a label the field is only accessible via the section ID. This does not hold true for some default fields which may have no section at all (e.g.username and password for a Login item).

Caution: vals-expressions are parsed as URIs. For the 1Password connect provider the host component of the URI identifies the vault (by ID or name). Therefore vaults containing certain characters not allowed in the host component (e.g. whitespaces, see RFC-3986 for details) can only be accessed by ID.

Examples:

Doppler

Examples:

(DOPPLER_TOKEN set as environment variable)

Pulumi State

Obtain value in state pulled from Pulumi Cloud REST API:

Environment variables:

Examples:

Kubernetes

Fetch value from Kubernetes:

Authentication to the Kubernetes cluster is done by referencing the local kubeconfig file. The path to the kubeconfig can be specified as a URI parameter, read from the KUBECONFIG environment variable or the provider will attempt to read $HOME/.kube/config. The Kubernetes context can be specified as a URI parameteter.

Environment variables:

Examples:

NOTE: This provider only supports kind "Secret" or "ConfigMap" in apiVersion "v1" at this time.

Conjur

This provider retrieves the value of secrets stored in Conjur. It's based on the https://github.com/cyberark/conjur-api-go lib.

The following env vars have to be configured:

Example:

HCP Vault Secrets

This provider retrieves the value of secrets stored in HCP Vault Secrets.

It is based on the HashiCorp Cloud Platform Go SDK lib.

Environment variables:

Parameters:

Parameters are optional and can be passed as query parameters in the URI, taking precedence over environment variables.

Example:

ref+hcpvaultsecrets://APPLICATION_NAME/SECRET_NAME[?client_id=HCP_CLIENT_ID&client_secret=HCP_CLIENT_SECRET&organization_id=HCP_ORGANIZATION_ID&organization_name=HCP_ORGANIZATION_NAME&project_id=HCP_PROJECT_ID&project_name=HCP_PROJECT_NAME&version=2]

Bitwarden

This provider retrieves the secrets stored in Bitwarden. It uses the Bitwarden Vault-Management API that is included in the Bitwarden CLI by executing bw serve.

Environment variables:

Parameters:

Parameters are optional and can be passed as query parameters in the URI, taking precedence over environment variables.

Examples:

HTTP JSON

This provider retrieves values stored in JSON hosted by a HTTP frontend.

This provider is built on top of jsonquery and xpath packages.

Given the diverse array of JSON structures that can be encountered, utilizing jsonquery with XPath presents a more effective approach for handling this variability in data structures.

This provider requires an xpath to be provided.

Do not include the protocol scheme i.e. http/https. Provider defaults to scheme https (http is available, see below)

Examples:

Fetch string value

ref+httpjson://<domain>/<path>?[insecure=false&floatAsInt=false]#/<xpath>

Let's say you want to fetch the below JSON object from https://api.github.com/users/helmfile/repos:

[
    {
        "name": "chartify"
    },
    {
        "name": "go-yaml"
    }
]

# To get name="chartify" using https protocol you would use:
ref+httpjson://api.github.com/users/helmfile/repos#///*[1]/name

# To get name="go-yaml" using https protocol you would use:
ref+httpjson://api.github.com/users/helmfile/repos#///*[2]/name

# To get name="go-yaml" using http protocol you would use:
ref+httpjson://api.github.com/users/helmfile/repos?insecure=true#///*[2]/

Fetch integer value

ref+httpjson://<domain>/<path>?[insecure=false&floatAsInt=false]#/<xpath>

Let's say you want to fetch the below JSON object from https://api.github.com/users/helmfile/repos:

[
    {
        "id": 251296379
    }
]

# Running the following will return: 2.51296379e+08
ref+httpjson://api.github.com/users/helmfile/repos#///*[1]/id

# Running the following will return: 251296379
ref+httpjson://api.github.com/users/helmfile/repos?floatAsInt=true#///*[1]/id

Advanced Usages

Discriminating config and secrets

vals has an advanced feature that helps you to do GitOps.

GitOps is a good practice that helps you to review how your change would affect the production environment.

To best leverage GitOps, it is important to remove dynamic aspects of your config before reviewing.

On the other hand, vals's primary purpose is to defer retrieval of values until the time of deployment, so that we won't accidentally git-commit secrets. The flip-side of this is, obviously, that you can't review the values themselves.

Using ref+<value uri> and secretref+<value uri> in combination with vals eval --exclude-secret helps it.

By using the secretref+<uri> notation, you tell vals that it is a secret and regular ref+<uri> instances are for config values.

myconfigvalue: ref+awsssm://myconfig/value
mysecretvalue: secretref+awssecrets://mysecret/value

To leverage GitOps most by allowing you to review the content of ref+awsssm://myconfig/value only, you run vals eval --exclude-secret to generate the following:

myconfigvalue: MYCONFIG_VALUE
mysecretvalue: secretref+awssecrets://mysecret/value

This is safe to be committed into git because, as you've told to vals, awsssm://myconfig/value is a config value that can be shared publicly.

Non-Goals

Complex String-Interpolation / Template Functions

In the early days of this project, the original author has investigated if it was a good idea to introduce string interpolation like feature to vals:

foo: xx${{ref "ref+vault://127.0.0.1:8200/mykv/foo?proto=http#/mykey" }}
bar:
  baz: yy${{ref "ref+vault://127.0.0.1:8200/mykv/foo?proto=http#/mykey" }}

But the idea had abandoned due to that it seemed to drive the momentum to vals being a full-fledged YAML templating engine. What if some users started wanting to use vals for transforming values with functions? That's not the business of vals.

Instead, use vals solely for composing sets of values that are then input to another templating engine or data manipulation language like Jsonnet and CUE.

Note though, vals does have support for simple string interpolation like usage. See Expression Syntax for more information.

Merge

Merging YAMLs is out of the scope of vals. There're better alternatives like Jsonnet, Sprig, and CUE for the job.