Home

Awesome

ceil: Auto-provisioned RPi cluster running K8S on bare-metal

Enter make help to see available commands.

Why the name? intval(ceil(M_PI)) === 4 which is the number of k8s nodes of the ceil cluster - flowers to mlande for gifting the name.

See branch max for the Mini PC (amd64) variant.

Author: Helmut Hoffer von Ankershoffen né Oertel

Goals

Tasks

Phase 0: Hardware

alt text

Phase 1: Foundation

Phase 2: Storage and Loadbalancing

Phase 3: Router

Phase 4: PiWatch

Phase 5: PiPHP

Phase 6: Auto-Scaling

Phase 7: Mesh-Networking (waiting for ARM images from CNCF et al)

Phase 8: GitOps and Progressive Delivery (waiting for ARM images from CNCF et al)

Phase 9: CI and emphemeral test environments (waiting for ARM images from CNCF et al)

Phase 10: A/B testing (waiting for ARM images from CNCF et al)

Phase 11: Sharing is caring

Layers and tools

Install this repository

  1. Fork this repository and clone to your workstation
  2. Walk all files with suffix .tpl, create a copy in the same directory without said suffix and enter specifics where invited by capital letters

Provision RPIs

  1. Prepare you workstation by installing Ansible, kubectl, helm etc. using homebrew: make prepare-mac
  2. Pull the hypriot image (which is not stored in GitHub): make pull-image
  3. Flash RPIs (insert SD cards in your workstation): make {router,one,two,three,four}-provision
  4. Insert SD cards into slots of respective RPIs
  5. Insert thumb drives into USB ports of RPIs
  6. Start RPIs by plugging in the USB charger

Setup router

  1. Make a DHCP reservation for ceil-router on your home or company WiFi router with IP address 192.168.0.100 - it will register as ceil-router at your WiFi router
  2. Set up a static route to the k8s subnet 11.0.0.0 with 192.168.0.100 as gateway in your company or home wifi router - if this is not achievable use make workstation-route-add to add a route on your workstation.
  3. For VPN setup port forwarding (sometimes called "virtual server") in your company or home wifi router for port 1194 (or whatever you configured in router/roles/vpn/defaults/main.yml) to 192.168.0.100
  4. Add 192.168.0.100 as the first nameserver for the (WiFi) connection of your workstation using system settings
  5. Reboot ceil-router to pickup its IP address via make router-reboot - it will register via ZeroConf/Avahi on your workstation as ceil-router.local
  6. Check via make router-check-ip if the IP address has been picked up
  7. Setup networking services on router using make router-setup
  8. Wait for 1 minute than check if the k8s nodes (ceil-{one,two,three,four}.dev) have picked up their designated IP addresses from the router in the range 11.0.0.101 to 11.0.0.104: make k8s-check-ip

Notes:

Setup K8S and execute all deployments

  1. Execute make setup to setup K8S inc. persistence and deploy everything at once - takes ca. 45 minutes.

Notes:

Alternatively you can execute the setup and deploy steps one-by-one as described below

Interact, open dashboards and UIs

  1. Establish proxy to cluster (leave open in separate terminal): make k8s-proxy
  2. List nodes: make nodes-show
  3. List pods: make pods-show
  4. Generate bearer token for accessing K8S dashboard: make k8s-dashboard-bearer-token-show
  5. Access K8S dashboard in your browser and enter token: make k8s-dashboard-open
  6. Open Traefik UI in your browser: make traefik-ui-open
  7. Show webpage in your browser: make httpd-open
  8. Open Prometheus UI in your browser: make prometheus-open
  9. Open Grafana dashboards in your browser: make grafana-open

Notes:

Setup K8S inc. persistence and helm/tiller

  1. Setup K8S cluster inc. persistence via GlusterFS+Heketi and helm/tiller for later deployments: make k8s-setup.

Notes:

Deploy

  1. Execute all deployments using make all-deploy or deploy step by step as documented below.
  2. Interact, open dashboards and UIs as documented above.

Delete deployments

  1. All deployments provide an individual make target for deleting the deployment, e.g. ngrok-delete. Execute make help to see all commands.
  2. Execute make all-delete to delete all deployments at once

Remove K8S inc. persistence and helm/tiller

  1. Execute make k8s-remove.

Teardown

  1. Execute make teardown to delete all deployments and remove K8S.

Obstacles

Additional references