Home

Awesome

k3sup ๐Ÿš€ (said 'ketchup')

<img src="docs/assets/k3sup.png" width="20%" alt="k3sup logo">

k3sup is a light-weight utility to get from zero to KUBECONFIG with k3s on any local or remote VM. All you need is ssh access and the k3sup binary to get kubectl access immediately.

The tool is written in Go and is cross-compiled for Linux, Windows, MacOS and even on Raspberry Pi.

How do you say it? Ketchup, as in tomato.

Sponsor this License: MIT build Github All Releases

Contents:

What's this for? ๐Ÿ’ป

This tool uses ssh to install k3s to a remote Linux host. You can also use it to join existing Linux hosts into a k3s cluster as agents. First, k3s is installed using the utility script from Rancher, along with a flag for your host's public IP so that TLS works properly. The kubeconfig file on the server is then fetched and updated so that you can connect from your laptop using kubectl.

You may wonder why a tool like this needs to exist when you can do this sort of thing with bash.

k3sup was developed to automate what can be a very manual and confusing process for many developers, who are already short on time. Once you've provisioned a VM with your favourite tooling, k3sup means you are only 60 seconds away from running kubectl get pods on your own computer. If you are a local computer, you can bypass SSH with k3sup install --local

Do you use k3sup?

k3sup was created by Alex Ellis - the founder of OpenFaaS ยฎ & inlets.

<a href="https://github.com/sponsors/alexellis/"> <img alt="Sponsor this project" src="https://github.com/alexellis/alexellis/blob/master/sponsor-today.png" width="90%"> </a>

Want to see continued development? Sponsor alexellis on GitHub

Uses

Bootstrapping Kubernetes

Conceptual architecture Conceptual architecture, showing k3sup running locally against any VM such as AWS EC2 or a VPS such as DigitalOcean.

Download k3sup (tl;dr)

k3sup is distributed as a static Go binary. You can use the installer on MacOS and Linux, or visit the Releases page to download the executable for Windows.

curl -sLS https://get.k3sup.dev | sh
sudo install k3sup /usr/local/bin/

k3sup --help

A note for Windows users

Windows users can use k3sup install and k3sup join with a normal "Windows command prompt".

Demo ๐Ÿ“ผ

In the demo I install Kubernetes (k3s) onto two separate machines and get my kubeconfig downloaded to my laptop each time in around one minute.

  1. Ubuntu 18.04 VM created on DigitalOcean with ssh key copied automatically
  2. Raspberry Pi 4 with my ssh key copied over via ssh-copy-id

Watch the demo:

asciicast

Usage โœ…

The k3sup tool is a client application which you can run on your own computer. It uses SSH to connect to remote servers and creates a local KUBECONFIG file on your disk. Binaries are provided for MacOS, Windows, and Linux (including ARM).

Pre-requisites for k3sup servers and agents

Some Linux hosts are configured to allow sudo to run without having to repeat your password. For those which are not already configured that way, you'll need to make the following changes if you wish to use k3sup:

# sudo visudo

# Then add to the bottom of the file
# replace "alex" with your username i.e. "ubuntu"
alex ALL=(ALL) NOPASSWD: ALL

In most circumstances, cloud images for Ubuntu and other distributions will not require this step.

As an alternative, if you only need a single server you can log in interactively and run k3sup install --local instead of using SSH.

๐Ÿ‘‘ Setup a Kubernetes server with k3sup

You can setup a server and stop here, or go on to use the join command to add some "agents" aka nodes or workers into the cluster to expand its compute capacity.

Provision a new VM running a compatible operating system such as Ubuntu, Debian, Raspbian, or something else. Make sure that you opt-in to copy your registered SSH keys over to the new VM or host automatically.

Note: You can copy ssh keys to a remote VM with ssh-copy-id user@IP.

Imagine the IP was 192.168.0.1 and the username was ubuntu, then you would run this:

export IP=192.168.0.1
k3sup install --ip $IP --user ubuntu

# Or use a hostname and SSH key for EC2
export HOST="ec2-3-250-131-77.eu-west-1.compute.amazonaws.com"
k3sup install --host $HOST --user ubuntu \
  --ssh-key $HOME/ec2-key.pem

Other options for install:

See even more install options by running k3sup install --help.

export KUBECONFIG=`pwd`/kubeconfig
kubectl get node

Note that you should always use pwd/ so that a full path is set, and you can change directory if you wish.

Checking if a cluster is ready

There are various ways to confirm whether a cluster is ready to use.

K3sup runs the "kubectl get nodes" command using a KUBECONFIG file, and looks for the "Ready" status on each node, including agents/workers.

Install K3s directly on the node and then check if it's ready:

k3sup install \
  --local \
  --context localk3s

k3sup ready \
  --context localk3s \
  --kubeconfig ./kubeconfig

Check a remote server saved to a local file:

k3sup install \
  --ip 192.168.0.101 \
  --user pi

k3sup ready \
  --context default \
  --kubeconfig ./kubeconfig

Check a merged context in your default KUBECONFIG:

k3sup install \
  --ip 192.168.0.101 \
  --user pi \
  --context pik3s \
  --merge \
  --local-path $HOME/.kube/config

# $HOME/.kube/config is a default for kubeconfig
k3sup ready --context pik3s

Merging clusters into your KUBECONFIG

You can also merge the remote config into your main KUBECONFIG file $HOME/.kube/config, then use kubectl config get-contexts or kubectx to manage it.

The default "context" name for the remote k3s cluster is default, however you can override this as below.

For example:

k3sup install \
  --ip $IP \
  --user $USER \
  --merge \
  --local-path $HOME/.kube/config \
  --context my-k3s

Here we set a context of my-k3s and also merge into our main local KUBECONFIG file, so we could run kubectl config use-context my-k3s or kubectx my-k3s.

๐Ÿ˜ธ Join some agents to your Kubernetes server

Let's say that you have a server, and have already run the following:

export SERVER_IP=192.168.0.100
export USER=root

k3sup install --ip $SERVER_IP --user $USER

Next join one or more agents to the cluster:

export AGENT_IP=192.168.0.101

export SERVER_IP=192.168.0.100
export USER=root

k3sup join --ip $AGENT_IP --server-ip $SERVER_IP --user $USER

Please note that if you are using different usernames for SSH'ing to the agent and the server that you must provide the username for the server via the --server-user parameter.

That's all, so with the above command you can have a two-node cluster up and running, whether that's using VMs on-premises, using Raspberry Pis, 64-bit ARM or even cloud VMs on EC2.

Use your hardware authentication / 2FA or SSH Agent

You may wish to use the ssh-agent utility if:

Run the following to set SSH_AUTH_SOCK:

$ eval $(ssh-agent)
Agent pid 2641757

Optionally, if your key is encrypted, run: ssh-add ~/.ssh/id_rsa

Now run any k3sup command, and your SSH key will be requested from the ssh-agent instead of from the usual location.

You can also specify an SSH key with --ssh-key if you want to use a specific key-pair.

K3sup plan for automation

A new command was added to k3sup to help with automating large amounts of nodes.

k3sup plan reads a JSON input file containing hosts, and will generate an installation command for a number of servers and agents.

Example input file:

[
  {
    "hostname": "node-a-1",
    "ip": "192.168.129.138"
  },
  {
    "hostname": "node-a-2",
    "ip": "192.168.129.128"
  },
  {
    "hostname": "node-a-3",
    "ip": "192.168.129.131"
  },
  {
    "hostname": "node-a-4",
    "ip": "192.168.129.130"
  },
  {
    "hostname": "node-a-5",
    "ip": "192.168.129.127"
  }
]

The following will create 1x primary server, with 2x additional servers within a HA etcd cluster, the last two nodes will be added as agents:

k3sup plan \
  devices.json \
  --user ubuntu \
  --servers 3 \
  --server-k3s-extra-args "--disable traefik" \
  --background > bootstrap.sh

Then make the file executable and run it:

chmod +x bootstrap.sh
./bootstrap.sh

Watch a demo with dozens of Firecracker VMs: Testing Kubernetes at Scale with bare-metal

The initial version of k3sup plan has a reduced set of flags. Flags such as --k3s-version and --datastore are not available, but feel free to propose an issue with what you need.

Create a multi-master (HA) setup with external SQL

The easiest way to test out k3s' multi-master (HA) mode with external storage, is to set up a Mysql server using DigitalOcean's managed service.

Before:

mysql://doadmin:80624d3936dfc8d2e80593@db-mysql-lon1-90578-do-user-6456202-0.a.db.ondigitalocean.com:25060/defaultdb?ssl-mode=REQUIRED

After:

mysql://doadmin:80624d3936dfc8d2e80593@tcp(db-mysql-lon1-90578-do-user-6456202-0.a.db.ondigitalocean.com:25060)/defaultdb

Note that we've removed ?ssl-mode=REQUIRED and wrapped the host/port in tcp().

export DATASTORE="mysql://doadmin:80624d3936dfc8d2e80593@tcp(db-mysql-lon1-90578-do-user-6456202-0.a.db.ondigitalocean.com:25060)/defaultdb

You can prefix this command with two spaces, to prevent it being cached in your bash history.

Generate a token used to encrypt data (If you already have a running node this can be retrieved by logging into a running node and looking in /var/lib/rancher/k3s/server/token)

# Best option for a token:
export TOKEN=$(openssl rand -base64 64)

# Fallback for no openssl, on a Linux host:
export TOKEN=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 64)

# Failing that, then try:
export TOKEN=$(head -c 64 /dev/urandom|shasum| cut -d - -f 1)

Imagine we have the following three VMs, two will be servers, and one will be an agent.

export SERVER1=104.248.135.109
export SERVER2=104.248.25.221
export AGENT1=104.248.137.25

k3sup install --user root --ip $SERVER1 --datastore="${DATASTORE}" --token=${TOKEN}

k3sup install --user root --ip $SERVER2 --datastore="${DATASTORE}" --token=${TOKEN}

You can join the agent to either server, the datastore is not required for this step.

k3sup join --user root --server-ip $SERVER1 --ip $AGENT1

Please note that if you are using different usernames for SSH'ing to the agent and the server that you must provide the username for the server via the --server-user parameter.

If you run kubectl get node, you'll now see two masters/servers and one agent, however, we joined the agent to the first server. If the first server goes down, the agent will effectively also go offline.

kubectl get node

NAME              STATUS                        ROLES    AGE     VERSION
k3sup-1           Ready                         master   73s     v1.19.6+k3s1
k3sup-2           Ready                         master   2m31s   v1.19.6+k3s1
k3sup-3           Ready                         <none>   14s     v1.19.6+k3s1

There are two ways to prevent a dependency on the IP address of any one host. The first is to create a TCP load-balancer in the cloud of your choice, the second is for you to create a DNS round-robbin record, which contains all of the IPs of your servers.

In your DigitalOcean dashboard, go to the Networking menu and click "Load Balancer", create one in the same region as your Droplets and SQL server. Select your two Droplets, i.e. 104.248.34.61 and 142.93.175.203, and use TCP with port 6443.

If you want to run k3sup join against the IP of the LB, then you should also add TCP port 22

Make sure that the health-check setting is also set to TCP and port 6443. Wait to get your IP, mine was: 174.138.101.83

Save the LB into an environment variable:

export LB=174.138.101.83

Now use ssh to log into both of your servers, and edit their config files at /etc/systemd/system/k3s.service, update the lines --tls-san and the following address, to that of your LB:

ExecStart=/usr/local/bin/k3s \
    server \
        '--tls-san' \
        '104.248.135.109' \

Becomes:

ExecStart=/usr/local/bin/k3s \
    server \
        '--tls-san' \
        '174.138.101.83' \

Now run:

sudo systemctl daemon-reload && \
  sudo systemctl restart k3s-agent

And repeat these steps on the other server.

You can update the agent manually, via ssh and edit /etc/systemd/system/k3s-agent.service.env on the host, or use k3sup join again, but only if you added port 22 to your LB:

k3sup join --user root --server-ip $LB --ip $AGENT1

Finally, regenerate your KUBECONFIG file with the LB's IP, instead of one of the servers:

k3sup install --skip-install --ip $LB

Log into the first server, and stop k3s sudo systemctl stop k3s, then check that kubectl still functions as expected:

export KUBECONFIG=`pwd`/kubeconfig
kubectl get node -o wide

NAME              STATUS                        ROLES    AGE   VERSION
k3sup-1           NotReady                      master   23m   v1.19.6+k3s1
k3sup-2           Ready                         master   25m   v1.19.6+k3s1
k3sup-3           Ready                         <none>   22m   v1.19.6+k3s1

You've just simulated a failure of one of your masters/servers, and you can still access kubectl. Congratulations on building a resilient k3s cluster.

Create a multi-master (HA) setup with embedded etcd

In k3s v1.19.5+k3s1 a HA multi-master (multi-server in k3s terminology) configuration is available called "embedded etcd". A quorum of servers will be required, which means having an odd number of nodes and least three. See more

Note the --cluster flag

export SERVER_IP=192.168.0.100
export USER=root

k3sup install \
  --ip $SERVER_IP \
  --user $USER \
  --cluster \
  --k3s-version v1.19.1+k3s1

Note the new --server flag

export USER=root
export SERVER_IP=192.168.0.100
export NEXT_SERVER_IP=192.168.0.101

k3sup join \
  --ip $NEXT_SERVER_IP \
  --user $USER \
  --server-user $USER \
  --server-ip $SERVER_IP \
  --server \
  --k3s-version v1.19.1+k3s1

Now check kubectl get node:

kubectl get node
NAME              STATUS   ROLES    AGE     VERSION
paprika-gregory   Ready    master   8m27s   v1.19.2-k3s
cave-sensor       Ready    master   27m     v1.19.2-k3s

If you used --no-extras on the initial installation you will also need to provide it on each join:

export USER=root
export SERVER_IP=192.168.0.100
export NEXT_SERVER_IP=192.168.0.101

k3sup join \
  --ip $NEXT_SERVER_IP \
  --user $USER \
  --server-user $USER \
  --server-ip $SERVER_IP \
  --server \
  --no-extras \
  --k3s-version v1.19.1+k3s1

๐Ÿ‘จโ€๐Ÿ’ป Micro-tutorial for Raspberry Pi (2, 3, or 4) ๐Ÿฅง

In a few moments you will have Kubernetes up and running on your Raspberry Pi 2, 3 or 4. Stand by for the fastest possible install. At the end you will have a KUBECONFIG file on your local computer that you can use to access your cluster remotely.

Conceptual architecture, showing k3sup running locally against bare-metal ARM devices.

export KUBECONFIG=`pwd`/kubeconfig
kubectl get node -o wide

You now have kubectl access from your laptop to your Raspberry Pi running k3s.

If you want to join some nodes, run export IP="" for each additional RPi, followed by:

Remember all these commands are run from your computer, not the RPi.

Now where next? I would recommend my detailed tutorial where I spend time looking at how to flash the SD card, deploy k3s, deploy OpenFaaS (for some useful microservices), and then get incoming HTTP traffic.

Try it now: Will it cluster? K3s on Raspbian

Caveats on security

If you are using public cloud, then make sure you see the notes from the Rancher team on setting up a Firewall or Security Group.

k3s docs: k3s configuration / open ports

Contributing

Blog posts & tweets

Blogs posts, tutorials, and Tweets about k3sup (#k3sup) are appreciated. Please send a PR to the README.md file to add yours.

Contributing via GitHub

Before contributing code, please see the CONTRIBUTING guide. Note that k3sup uses the same guide arkade

Both Issues and PRs have their own templates. Please fill out the whole template.

All commits must be signed-off as part of the Developer Certificate of Origin (DCO)

License

MIT

๐Ÿ“ข What are people saying about k3sup?

Checkout the Announcement tweet

Similar tools & glossary

Glossary:

Related tools:

Troubleshooting and support

Maybe the problem is with K3s?

If you're having issues, it's likely that this is a problem with K3s, and not with k3sup. How do we know that? K3sup is a very mature project and has a few use-cases that it generally performs very well.

Rancher provides support for K3s on their Slack in the #k3s channel. This should be your first port of call. Your second port of call is to raise an issue with the K3s maintainers in the K3s repo

Do you want to install a specific version of K3s? See k3sup install --help and the --k3s-version and --k3s-channel flags.

Is your system ready to run Kubernetes? K3s requires certain Kernel modules to be available, run k3s check-config and check the output. Alex tests K3sup with Raspberry Pi OS and Ubuntu LTS on a regular basis.

Common issues

The most common problem is that you missed a step, fortunately it's relatively easy to get the logs from the K3s service and it should tell you what's wrong.

Note: Passing --no-deploy to --k3s-extra-args was deprecated by the K3s installer in K3s 1.17. Use --disable instead or --no-extras.

Getting access to your KUBECONFIG

You may have run into an issue where sudo access is required for kubectl access.

You should not run kubectl on your server or agent nodes. k3sup is designed to rewrite and/or merge your cluster's config to your local KUBECONFIG file. You should run kubectl on your laptop / client machine.

If you've lost your kubeconfig, you can use k3sup install --skip-install. See also the various flags for merging and setting a context name.

Smart cards and 2FA

Warning: issues requesting support for smart cards / 2FA will be closed immediately. The feature has been proven to work, and is provided as-is.

You can use a smart card or 2FA security key such as a Yubikey. You must have your ssh-agent configured correctly, at that point k3sup will defer to the agent to make connections on MacOS and Linux. Find out more

Misc note on iptables

Note added by Eduardo Minguez Perez

Currently there is an issue in k3s involving iptables >= 1.8 that can affect the network communication. See the k3s issue and the corresponding kubernetes one for more information and workarounds. The issue has been observed in Debian Buster but it can affect other distributions as well.