Awesome

NACP - Nomad Admission Control Proxy

A proxy infront of the Nomad API that allows to perform mutation and validation on the job data.

nacp

How

It intercepts the Nomad API calls that include job data (plan, register, validate) and performs mutation and validation on the job data. The job data is at that point is already transformed from HCL to JSON. If any errors occur the proxy will return the error to the Nomad API caller. Warnings are attached to the Nomad response when they come back from the actual Nomad API.

Currently validation comes into two flavors:

Embedded OPA rules
Webhooks

Mutation

During the mutation phase the job data is modified by the configured mutators.

OPA

The opa mutator uses the OPA policy engine to perform the mutation. The OPA rule is expects to return a JSONPatch object. The JSONPatch object is then applied to the job data. It can also return errors and warnings. An example rego could look like this:

package hello_world_meta
import future.keywords

patch contains ops if [

   input.Name == "greeting_job"
   ops:= {
        "op": "add",
        "path": "/Meta",
        "value": {
            "hello": "world"
        }
    }
]

errors contains msg if {

    input.Name == "silent_job"
    msg := "cannot greet"
}

warnings contains msg if {

  input.Name == "had_no_coffee_yet_job"
  msg := "you should have coffee first"
}

For the embedded you also have to define the query that is used to extract the patch from the OPA response:

mutator "opa_json_patch" "hello_world_opa_mutator" {

    opa_rule {
        query = <<EOH
        patch = data.hello_world_meta.patch
        errors = data.hello_world_meta.errors
        warnings = data.hello_world_meta.warnings
        EOH
        filename = "hello_world_meta.rego"
    }
}

Webhook

The webhook mutator sends the job data to a configured endpoint and expects a JSONPatch object in return. It can also return errors and warnings. The JSONPatch object is then applied to the job data. An example response could look like this:

{
  "patch": [
    {
      "op": "add",
      "path": "/Meta",
      "value": {
        "hello": "world"
      }
    }
  ],
  "errors": [
    "some error"
  ],
  "warnings": [
    "some warning"
  ]
}

The webhook mutator can be configured with the following options:

mutator "json_patch_webhook" "hello_world_webhook_mutator" {

  webhook {
    endpoint = "http://example.org/send/job/here"
    method = "POST"
  }

}

Hint: You can also setup the OPA server as a webhook mutator. You can use the system main package to run the OPA server as a webhook mutator.

Validation

During the validation phase the job data is validated by the configured validators. If any errors occur the proxy will return the error to the Nomad API caller. Warnings are attached to the Nomad response when they come back from the actual Nomad API.

OPA

The opa validator uses the OPA policy engine to perform the validation. The OPA rule is expects to return a list of errors and warnings. An example rego could look like this:

package costcenter_meta

import future.keywords.contains
import future.keywords.if

errors contains msg if {

	not input.Meta.costcenter
	msg := "Every job must have a costcenter metadata label"
}

errors contains msg if {
	value := input.Meta.costcenter

	not startswith(value, "cccode-")
	msg := sprintf("Costcenter code must start with `cccode-`; found `%v`", [value])
}

Then configure the validator in the config file:

validator "opa" "costcenter_opa_validator" {

    opa_rule {
        query = <<EOH
        errors = data.costcenter_meta.errors
        warnings = data.costcenter_meta.warnings
        EOH
        filename = "costcenter_meta.rego"
    }
}

Webhook

The webhook validator sends the job data to a configured endpoint and expects a list of errors and warnings in return.

The response should include potential errors and warnings:

{
  "errors": [
    "some error"
  ],
  "warnings": [
    "some warning"
  ]
}

The webhook validator can be configured with the following options:

validator "webhook" "some_webhook_validator" {

  webhook {
    endpoint = "http://example.org/send/job/here"
    method = "POST"
  }

}

More Examples

Checkout the examples folder for more examples.

Usage

Run Proxy

$ nacp -config config.hcl

It will launch per default on port 6464.

Send Job to Nomad via Proxy

NOMAD_ADDR=http://localhost:6464 nomad job run job.hcl

Other Configuration

NACP Server

The NACP server can be configured with the following options:

server {
  # The address the server will listen on
  bind = "0.0.0.0"
  port = 6464

  tls { # If this is present nomad will use TLS
    # The path to the certificate file
    cert_file = "cert.pem"
    # The path to the private key file
    key_file = "key.pem"

    # The path to the CA certificate file
    ca_file = "ca.pem"
  }
}

Nomad Upstream

The Nomad upstream can be configured with the following options:

nomad {
  # The address of the Nomad API
  address = "http://localhost:4646"

  tls { # If this is present nomad will use TLS
    # The path to the certificate file
    cert_file = "cert.pem"
    # The path to the private key file
    key_file = "key.pem"

    # The path to the CA certificate file
    ca_file = "ca.pem"
  }
}

Notation

Image signature validation can be done in two ways. Either by the notation validator or via the opa by using the notation_verify_image function which returns either true if the image is valid or false if the image is not valid. See example/notation for an example.

Both validators expect a notation block. E.g.:

...
validator "opa" "notation_opa_validator" {

  opa_rule {
      ...
  }
  notation {
    repo_plain_http   = false
    trust_store_dir   = "/some/path/to/truststore"
    trust_policy_file = "/some/path/to/trustpolicy.json"
    credential_store_file = "/some/path/to/credentialstore.json"
  }
}

The credential_store_file refers to the [oras' credential file] (https://docs.docker.com/engine/reference/commandline/cli/#docker-cli-configuration-file-configjson-properties)

e.g.:

{
  "auths": {
    "https://my-registy.example.org": {
      "auth": "<base64 encoded username:password>"
    }
  }
}

Note

This work was inspired by the internal Nomad Admission Controller