Home

Awesome

Usernetes: Kubernetes without the root privileges (Generation 2)

Usernetes (Gen2) deploys a Kubernetes cluster inside Rootless Docker, so as to mitigate potential container-breakout vulnerabilities.

Note

Usernetes (Gen2) has significantly diverged from the original Usernetes (Gen1), which did not require Rootless Docker to be installed on hosts.

See the gen1 branch for the original Usernetes (Gen1).

Usernetes (Gen2) is similar to Rootless kind and Rootless minikube, but Usernetes (Gen 2) supports creating a cluster with multiple hosts.

Components

Requirements

Host operating systemMinimum version
Ubuntu (recommended)22.04
Rocky Linux9
AlmaLinux9
Fedora(?)
Container EngineMinimum version
Rootless Docker (recommended)v20.10
Rootless Podmanv4.x
Rootless nerdctlv1.6
curl -o install.sh -fsSL https://get.docker.com
sudo sh install.sh
dockerd-rootless-setuptool.sh install
sudo loginctl enable-linger $(whoami)
sudo mkdir -p /etc/systemd/system/user@.service.d

sudo tee /etc/systemd/system/user@.service.d/delegate.conf <<EOF >/dev/null
[Service]
Delegate=cpu cpuset io memory pids
EOF

sudo systemctl daemon-reload
sudo tee /etc/modules-load.d/usernetes.conf <<EOF >/dev/null
br_netfilter
vxlan
EOF

sudo systemctl restart systemd-modules-load.service
sudo tee /etc/sysctl.d/99-usernetes.conf <<EOF >/dev/null
net.ipv4.conf.default.rp_filter = 2
EOF

sudo sysctl --system

Use scripts in ./init-host for automating these steps.

Usage

See make help.

# Bootstrap a cluster
make up
make kubeadm-init
make install-flannel

# Enable kubectl
make kubeconfig
export KUBECONFIG=$(pwd)/kubeconfig
kubectl get pods -A

# Multi-host
make join-command
scp join-command another-host:~/usernetes
ssh another-host make -C ~/usernetes up kubeadm-join
make sync-external-ip

# Debug
make logs
make shell
make kubeadm-reset
make down-v
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

The container engine defaults to Docker. To change the container engine, set export CONTAINER_ENGINE=podman or export CONTAINER_ENGINE=nerdctl.

Limitations

Advanced topics

Network

When CONTAINER_ENGINE is set to nerdctl, bypass4netns can be enabled for accelerating connect(2) syscalls. The acceleration currently does not apply to VXLAN packets.

containerd-rootless-setuptool.sh install-bypass4netnsd
export CONTAINER_ENGINE=nerdctl
make up

Misc