Awesome

Addon Manager

Addons are critical components within a Kubernetes cluster that provide critical services needed by applications like DNS, Ingress, Metrics, Logging, etc. Addon Manager provides a CRD for lifecycle management of such addons using Argo Workflows.

AddonManagerImg

Important!

This is Alpha state software, anything can change at any time. Our deprecation and removal policy during Alpha is very aggressive, see below:

Any changes to the api types will result in an iterative version bump e.g., v1alpha1 -> v1alpha2
API versions will NOT be backward compatible, all new api versions will immediately replace the older version
All api kinds within the group addonmgr.keikoproj.io will be versioned the same e.g., v1alphaX

Dependencies

kubernetes-1.12+
kubectl-1.14+

Installation

To use: kubectl kustomize github.com/keikoproj/addon-manager.git/config/default | kubectl apply -f -

Usage example

An Addon describes a kubernetes resource-based application that is deployed to a cluster. The Addon CRD defines a spec with some optional and required fields, and a lifecycle where most of the addon may be contained. Internally, Argo Workflows is used for each lifecycle step to specify and submit the resources. The design spec for the Addon CRD is as follows:

apiVersion: addonmgr.keikoproj.io/v1alpha1
kind: Addon
metadata:
  name: fluentd-addon
  namespace: addon-manager-system
spec:
  pkgName: core/fluentd
  pkgVersion: v0.0.1
  pkgType: composite
  pkgDescription: Company fluentd addon.
  pkgDeps: 
    argoproj/workflows: v2.2.1
  params:
    namespace: mynamespace
    clusterContext:
      clusterName: "my-test-cluster"
      clusterRegion: "us-west-2"
    data:
      hec_splunk_server: hec.splunk.example.com
  selector:
    matchLabels:
      app.kubernetes.io/name: fluentd
      app.kubernetes.io/version: "1.0.0"
  lifecycle:
    prereqs:
      template: |
        apiVersion: argoproj.io/v1alpha1
        kind: Workflow
        ...
    install:
      template: |
        apiVersion: argoproj.io/v1alpha1
        kind: Workflow
        ...
    delete:
      template: | 
        apiVersion: argoproj.io/v1alpha1
        kind: Workflow
        ...
    validate:
      template: |
        apiVersion: argoproj.io/v1alpha1
        kind: Workflow
        ...

To submit: kubectl apply -f addon.yaml

The 4 stages of the addon lifecycle (prereqs, install, validate, delete) have 3 fields, Name, Role, and Template. The template is specified as an inline <a href="https://github.com/argoproj/argo/" target="_blank">Argo Workflow alt [^] </a>. The workflow should specify the Kubernetes resources being submitted as part of each step, and include the appropriate kubernetes client commands that submit those resources to the server. Here's an example of such

...
    prereqs:
      template: |
        apiVersion: argoproj.io/v1alpha1
        kind: Workflow
        metadata:
          generateName: prereqs-minion-manager-
        spec:
          entrypoint: entry
          serviceAccountName: addon-manager-workflow-installer-sa
          templates:
          - name: entry
            steps:
            - - name: prereq-resources
                template: submit
                arguments:
                  artifacts:
                  - name: doc
                    path: /tmp/doc
                    raw:
                      data: |
                        apiVersion: v1
                        kind: Namespace
                        metadata:
                          name: "{{workflow.parameters.namespace}}"
                        ---
                        apiVersion: v1
                        kind: ServiceAccount
                        metadata:
                          name: example-sa
                          namespace: "{{workflow.parameters.namespace}}"
          - name: submit
            inputs:
              artifacts:
                - name: doc
                  path: /tmp/doc
            container:
              image: expert360/kubectl-awscli:v1.11.2
              command: [sh, -c]
              args: ["kubectl apply -f /tmp/doc"]

The Addon Manager provides parameter injection to all lifecycle workflows to get rid of the need to input the parameters into each workflow. Before the workflows are run, all key-value pairs in the addon spec.params are made into global workflow parameters. This means their values are accessible like so: "{{workflow.parameters.NAME}}". It's important to make sure when referencing parameters that the templated value is escaped, either with quotes or if it's included in a string literal; if this is not done, you may experience a failed workflow due to the worflow failing to be parsed correctly.

Generally, there are a set of best practices defined that make defining an Addon CR straightforward:

Each addon (with a few exceptions) should be deployed to its own namespace. This is done by specifying a namespace name in spec.params.namespace, and then templating that into each lifecycle workflow where there are namespaced resources, like in the example above.
Best-practice for the prereqs lifecycle step workflow: if the addon has any namespaced resources, the namespace in question should be specified first in the prereqs workflow. Following the namespace, all other non-deployable resources should be specified by priotity in the prereqs workflow as well.
Best-practice for the install lifecycle step workflow: all deployable resources (deployments, services, statefulsets, replicasets, daemonsets, etc.) should be supplied as part of this workflow.

Get Addons

kubectl get addons -n addon-manager-system

NAME                       PACKAGE                    VERSION       STATUS           AGE
addon-manager-argo-addon   addon-argo-workflow        v2.2.1        Succeeded        14m
cluster-autoscaler         cluster-autoscaler-addon   v0.1          Pending          1m
event-router               event-router               v0.2          Pending          1m
external-dns               external-dns               v0.2          Pending          1m
fluentd                    core/fluentd-addon         v0.0.1        Pending          1m
...

Delete Addon

To delete: kubectl delete -f addon.yaml

Addonctl

The Addon Manager is distributed with the addonctl binary which allows a default Addon CR generation given spec parameters yaml resource files, and python scripts. Pre-alpha currently, this tool can be more useful for initial addon generation, allowing the user to then manually modify it.

To use: addonctl --help

A control plane for managing addons

Usage:
  addonctl [command]

Available Commands:
  create      Create the addon resource with the supplied arguments
  help        Help about any command

Flags:
  -c, --channel string          Channel for the addon package
      --cluster-name string     Name of the cluster context being used
      --cluster-region string   Cluster region
      --deps string             Comma seperated dependencies list in the format 'pkgName:pkgVersion'
      --desc string             Description of the addon
      --dryrun                  Outputs the addon spec but doesn't submit
  -h, --help                    help for addonctl
      --install string          File or directory of resource yaml to submit as install step
  -n, --namespace string        Namespace where the addon will be deployed
  -p, --params string           Params to supply to the resource yaml
      --prereqs string          File or directory of resource yaml to submit as prereqs step
      --secrets string          Comma seperated list of secret names which are validated as part ofthe addon-manager-system namespace
      --selector string         Selector applied to all resources?
  -t, --type string             Addon package type
  -v, --version string          Addon package version

Use "addonctl [command] --help" for more information about a command.

Addonctl Example

addonctl create my-addon -n my-addon-ns \
  --type composite \
  --version v0.2 \
  --cluster-name my.cluster.k8s.local \
  --cluster-region us-west-2 \
  --selector app:myaddon \
  --prereqs ./prereq_resources.yaml \
  --install ./install_resources.yaml \
  --dryrun

❤ Contributing ❤

Please see CONTRIBUTING.md.

Developer Guide

Please see DEVELOPER.md.

Other KeikoProj Projects

Instance Manager - Kube Forensics - Active Monitor - Upgrade Manager - Minion Manager - Governor