Awesome
Kubernetes Controller Development
Since January 2023 one of my tasks is developing Kubernetes Controllers and CRDs with Go.
This is not an introduction to the topic.
Basics
First, you need to learnthe basics Go: go.dev/learn
You need to learn the basics of Kubernetes: kubernetes.io
You need to learn Kubernetes Client Go. Unfortunately only same examples exist.
Learn the API Conventions
Things I learned
Here are some things I learned.
status.conditions are great.
Cluster-API has a more precise way of handling conditions. The proposal explains it: Condition Proposal
cluster-api has some nice helpers: conditions
I wrote a small tool to check all conditions of all resources in a cluster: check-conditions
CRDs
Custom Resource Definitions are a way to extend the API schema of Kubernetes. It is like adding new tables to relational database.
You should know one big drawback of CRDs: CRDs are not namespaces. This means you can only install one version of your CRD in the cluster.
You can't install version v1 in namespace "foo" and version v2 in namespace "bar".
If you just need a way to configure your application, then a ConfigMap might be enough. This has the benefit that you can run two versions of your applications in parallel.
Related: Should I use a ConfigMap or a custom resource?
If you want to create a CRD, then read the newbie friendly docs of Kubebuilder
CRDs: make run
If you write a CRD and your own controller, then it is likely that you will use Tilt to run your controller in a development cluster (in most cases a kind cluster on your device).
But remember: a controller is just a Kubernetes API client. You can run it anywhere you like. If you created your project with Kubebuilder, you can use make run to
run the controller without any container. This makes it easy for debugging with delve.
Unfortunately, if you use web-hooks, then this will not work, except you set up a way so that the API server can reach your webhook.
CRDs: status will get lost upon backup+restore
Guideline: A controller does not read its own status.
After a backup+restore the status will be empty. The controller need to check the actual state and then create a matching status.
CRDs: no foreign keys
In a relational database you can create a foreign key, and you can be shure that the foreign key will never be broken.
I admit, that I miss this feature in the Kubernetes API.
CRDs: Webhooks: Order of restore
Imagine you have a CRD which represents a datacenter.
And you have a second CRD which represents a server.
We created a validating web hook which ensures that for a server spec.datacenter contains a string which has a corresponding datacenter.metadata.name.
The web hook works fine.
Except during a restore.
Disabling the web hook during restore is a manual task which I would like to avoid. This is not solved yet.
Client.Get(key, r.Client.Get(ctx, key, &myresource) finds nothing?
If r.Client.Get(key, r.Client.Get(ctx, key, &myresource) does not find a resource, but you can see it with kubectl, then two things could be wrong:
- RBAC
- Client Cache
To check RBAC, you can use that (if your Go code can't find a secret):
kubectl describe secret \
--as=system:serviceaccount:your-namespace-of-the-controller:your-serviceaccount \
-n namespace-of-secret \
your-secret
You can find the name of the service-account by looking at the pod.
Client Caching: check ctrl.NewManager() (usualy in main.go). Maybe there is some filter which makes the cache hide a lot of objects which would be visible otherwise.
Tools
Tools I use for developing with Golang and Kubernetes
- vscode
- kubespy Tools for observing Kubernetes resources in real time
- kubectl stern Multi pod and container log tailing for Kubernetes
- check-conditions my tool the check all conditions of all resources in a cluster
Related
Feedback
I love feedback and I love to hear from you. Just create an issue and tell me what's on your mind.