Awesome

Workbench Helm chart

NDS Labs Workbench is an open-source platform for hosting and launching pre-packages apps for educational or workshop environments.

TL;DR

$ git clone https://github.com/nds-org/workbench-helm-chart && cd workbench-helm-chart/ 
$ helm dep up
$ helm upgrade --install workbench -n workbench --create-namespace .

Introduction

This chart bootstraps a Workbench webui and apiserver on a Kubernetes cluster using the Helm package manager.

Prerequisites

• Kubernetes 1.22+
• helm v3.7.0 or later (required)
• make + kubectl docker + git (optional, for Developing the Chart)
• yarn (optional, for local dev/compilation)

Installing the Chart

Download dependency subcharts by running the following:

$ helm dep up

Then install a new release with Helm:

$ helm upgrade --install workbench -n workbench --create-namespace .

Uninstalling the Chart

To shut down all Workbench dependencies, webui, and apiserver:

$ helm uninstall workbench -n workbench

NOTE: this does not shutdown or affect UserApps

Configuration

The following table lists the configurable parameters of the Workbench chart and their default values.

Controller

These options affect the Deployment resource created by this chart.

Ingress

These options affect the Ingress resources created by this chart.

Workbench Options

These options affect the internals of Workbench and the customization of the WebUI.

Frontend: Domain + Auth + UI Customizations

Backend: Domain + Keycloak + MongoDB + UserApp Kubernetes Config

Misc

Other miscellaneous top-level configuration options.

Currently Unused?

These options are currently present, but may not yet be used.

Subcharts

• MongoDB
• Keycloak
• OAuth2 Proxy
• NFS Subdir External Provisioner
• NFS Server Provisioner
• NGINX Ingress Controller

MongoDB

To run a local mongodb alongside Workbench, you can set mongodb.enabled to true in the values.yaml:

mongodb:
  enabled: true
  # ... include any other config values from the mongodb chart
  auth:
    rootUser: workbench
    rootPassword: workbench

See https://artifacthub.io/packages/helm/bitnami/mongodb for configuration options

For more info about MongoDB, see https://www.mongodb.com/docs/manual/tutorial/getting-started/

Keycloak + PostgreSQL

To run a local keycloak alongside Workbench, you can set keycloak.enabled to true in the values.yaml:

keycloak:
  enabled: true
  # ... include any other config values from the keycloak chart
  httpRelativePath: "/auth/"
  auth:
     adminUser: "admin"
     adminPassword: "workbench"
  proxyAddressForwarding: true

See https://artifacthub.io/packages/helm/bitnami/keycloak for configuration options

For more info about Keycloak, see https://www.keycloak.org/docs/11.0/getting_started/

OAuth2 Proxy [+ Redis]

To run a local OAuth2 Proxy alongside Workbench, you can set oauth2-proxy.enabled to true in the values.yaml:

oauth2-proxy:
  enabled: true
  # ... include any other config values from the oauth2-proxy chart
  extraArgs:
    - --provider=keycloak-oidc

See https://artifacthub.io/packages/helm/bitnami/oauth2-proxy for configuration options

For more info about OAuth2 Proxy, see https://oauth2-proxy.github.io/oauth2-proxy/docs/configuration/overview

For more info about configuring specific providers, see https://oauth2-proxy.github.io/oauth2-proxy/docs/configuration/oauth_provider/

ReadWriteMany Volumes (NFS)

You'll need a StorageClass on your cluster that supports ReadWriteMany.

If you already have a volume provisioner running that supports ReadWriteMany, you can skip this section.

NOTE: You should only need the subdir OR the server, but you do not need to run both.

(Experimental) NFS Subdir External Provisioner: use an existing NFS server to provision RWM volumes

The NFS Subdir External Provisioner is a small piece of code that will talk to an existing NFS server to provision RWM volumes.

To run a local NFS Subdir External Provisioner alongside Workbench, you can set nfs-subdir-external-provisioner.enabled to true in the values.yaml:

nfs-subdir-external-provisioner:
  enabled: true
  nfs:
    # Use in-cluster DNS to resolve
    # server: "workbench-nfs-server-provisioner.workbench.svc.cluster.local"
    
    # Use external hostname
    server: "taiga-nfs.ncsa.illinois.edu"
    path: "/taiga/ncsa/radiant/<PROJECT_ID>"
    mountOptions:
    - tcp
    - nfsvers=3

See https://artifacthub.io/packages/helm/nfs-subdir-external-provisioner/nfs-subdir-external-provisioner for configuration options

NFS Server Provisioner: run your own NFS server to provision RWM volumes

To run a local NFS Server Provisioner alongside Workbench, you can set nfs-client-provisioner.enabled to true in the values.yaml:

nfs-server-provisioner:
  enabled: true
  # ... include any other config values from the nfs-server-provisioner chart

See https://artifacthub.io/packages/helm/kvaps/nfs-server-provisioner for configuration options

Ingress Controller (NGINX)

To run a local NGINX Ingress Controller alongside Workbench, you can set ingress-nginx.enabled to true in the values.yaml:

ingress-nginx:
  enabled: true
  # ... include any other config values from the ingress-nginx chart
  controller:
    kind: Deployment

See https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx for configuration options

For more info about NGINX Ingress Controller, see https://kubernetes.github.io/ingress-nginx/deploy/

Advanced Configuration (Optional)

Enable TLS with Wildcard DNS certs

1. Install cert-manager Helm chart: jetstack/cert-manager
2. Add an Issuer that support DNS-01 (see below):

apiVersion: cert-manager.io/v1
kind: Issuer 
metadata:
  name: letsencrypt-staging
  namespace: workbench
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: email@example.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging
    # Enable the DNS-01 challenge provider
    solvers:
    - dns01:
        # ...

3. Include tls section in the top-level ingress section of values.yaml:

ingress:
  class: "nginx"
  tls:
    - hosts:
      - "local.ndslabs.org"
      - "*.local.ndslabs.org"

WARNING: Do not include the same issuer or cluster-issuer annotation on multiple ingress rules.

WARNING: LetsEncrypt will rate limit you if you request too many certs are requested for the same domain.

To avoid this, use the staging environment (as above) for testing.

When you are ready (after testing) to move from staging to real certs, you can use the production environment.

DNS-01 via ACMEDNS

If your provider does not support DNS-01 requests (e.g. Google Domains), you can use ACMEDNS:

1. Register with acmedns for a unique set of credentials: curl -XPOST https://auth.acme-dns.io/register
   • This will return a set of credentials as a JSON blob:

{"username":"_username_","password":"_password_","fulldomain":"_id_.auth.acme-dns.io","subdomain":"_id_","allowfrom":[]}

2. Build up an acmedns.json file using this JSON blob:
   • You'll need to copy and paste this JSON value multiple times to build it up:

{
  "local.ndslabs.org": {"username":"_username_","password":"_password_","fulldomain":"_id_.auth.acme-dns.io","subdomain":"_id_","allowfrom":[]},
  "*.local.ndslabs.org": {"username":"_username_","password":"_password_","fulldomain":"_id_.auth.acme-dns.io","subdomain":"_id_","allowfrom":[]}
}

3. Create a secret from the acmedns.json file:

$ kubectl create secret -n cert-manager acme-dns --from-file=acmedns.json`

4. Point an Issuer at the acme-dns secret:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-staging
  namespace: workbench
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: email@example.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - dns01:
        acmeDNS:
          host: https://auth.acme-dns.io
          accountSecretRef:
            name: acme-dns
            key: acmedns.json 

5. Add the issuer annotation to your ingress.api.annotations section:

ingress:
  class: "nginx"
  tls:
    # ....
  api:
    annotations:
      cert-manager.io/issuer: "acmedns-staging"
      ingress.kubernetes.io/ssl-redirect: "true"
      ingress.kubernetes.io/force-ssl-redirect: "true"

Keycloak Realm Import

Download and import realm.json for a preconfigured workbench-dev realm:

$ kubectl create configmap keycloak-realm --from-file=realm.json -n workbench

Then add the following to your values.yaml:

  extraEnvVars:
    - name: KEYCLOAK_EXTRA_ARGS
      value: "-Dkeycloak.import=/config/realm.json"
  extraVolumeMounts:
    - name: config
      mountPath: "/config"
      readOnly: true
  extraVolumes:
    - name: config
      configMap:
        name: keycloak-realm
        items:
        - key: "realm.json"
          path: "realm.json"

Developing the Chart (Optional)

Clone the Git repository locally:

$ git clone https://github.com/nds-org/workbench-helm-chart && cd workbench-helm-chart/

You can then use the Makefile helper script:

$ make help         # Print help message
$ make check_all    # verify that dependencies are installed

To install/uninstall the chart:

$ make dep          # Fetch Helm dependencies
$ make template     # Debug the Helm chart template
$ make install      # Install the Helm chart
$ make uninstall    # Uninstall the Helm chart

To debug Pods startup/runtime:

$ make describe
$ make target=api logs
$ make target=webui logs
$ make target=nginx logs
$ make target=proxy logs
$ make target=keycloak logs
$ make target=mongo logs

To build/push local Docker images:

$ make clone    # clone from git repos
$ make pull     # pull from git upstream
$ make build    # build docker images
$ make push     # push to docker hub - note: this also performs a "make build"
$ make restart  # delete the running Pod so it restart with new docker images

Optional: Makefile Configuration

Change the default parameters of the Makefile by editing the included .env file:

$ cat .env
# Success/failure symbols
SUCCESS=[✔]
FAILED=[x]

# Helm chart config
NAMESPACE=workbench
NAME=workbench
CHART_PATH=.

# Docker image config
APISERVER_IMAGE=ndslabs/apiserver:python
WEBUI_IMAGE=ndslabs/webui:react

# Upstream Git repo/branch config
APISERVER_REPO=https://github.com/nds-org/workbench-apiserver-python
WEBUI_REPO=https://github.com/nds-org/workbench-webui
APISERVER_UPSTREAM_BRANCH=main
WEBUI_UPSTREAM_BRANCH=main

# Set this to empty to disable creating keycloak-realm ConfigMap
REALM_IMPORT=realm_import

Optional: Map source into running containers (local dev / hostpath only)

Run make clone and make pull to grab the latest source code.

Use your favorite IDE(s) or local tools use them to import the src/webui source code and run make compile.

This will produce a new folder src/webui/build containing compiled artifacts that can be mounted directly into the running webui container.

Finally, add the following to values.yaml and run make again:

controller:
  extraEnv:
    webui: []
    
    # Enable auto-reload of Python when source changes
    apiserver:
    - name: DEBUG
      value: "true"

  # Mount source code into respective containers
  extraVolumeMounts:
    webui:
    - mountPath: /usr/share/nginx/html/
      name: webuisrc
    apiserver:
    - mountPath: /app/
      name: apisrc

  # Point the extraVolumes at your local machine (hostpath only)
  extraVolumes:
    - name: webuisrc
      hostPath:
        path: /full/path/to/your/workbench-helm-chart/src/webui/build
    - name: apisrc
      hostPath:
        path: /full/path/to/your/workbench-helm-chart/src/apiserver

Now you can modify the webui or apiserver in any way that you see fit, then navigate to https://kubernetes.docker.internal to immediately test your changes.

NOTE: You'll need to re-run make compile after any modifications to the webui.

This will trigger the build step (during which you will get a 500 error) that will refresh the files in src/webui/build.

Cleaning Up

To delete all of the associated cluster volumes:

$ make clean

NOTE: this will delete your Keycloak Realm and/or MongoDB database and all user data

(OPTIONAL) Last step is to delete the namespace that was created by the helm install step:

$ make clean_all

NOTE: if you're using cert-manager, backup any necessary secrets to avoid being ratelimited by the LetsEncrypt API

TODO

• wait-for startup ordering for keycloak <- oauth2-proxy + apiserver (OIDC discovery)
• source chart somewhere.. NCSA Harbor? github pages?
• Verbose configuration documentation
• Adjust webui to speak _oauth2_proxy instead of / addition in speaking keycloak for OIDC
• MongoDB replication
• Git release workflows? CI? Github Actions?