Home

Awesome

Automated build of HA k3s Cluster with kube-vip and MetalLB

Fully Automated K3S etcd High Availability Install

This playbook will build an HA Kubernetes cluster with k3s, kube-vip and MetalLB via ansible.

This is based on the work from this fork which is based on the work from k3s-io/k3s-ansible. It uses kube-vip to create a load balancer for control plane, and metal-lb for its service LoadBalancer.

If you want more context on how this works, see:

πŸ“„ Documentation (including example commands)

πŸ“Ί Watch the Video

πŸ“– k3s Ansible Playbook

Build a Kubernetes cluster using Ansible with k3s. The goal is easily install a HA Kubernetes cluster on machines running:

on processor architecture:

βœ… System requirements

πŸš€ Getting Started

🍴 Preparation

First create a new directory based on the sample directory within the inventory directory:

cp -R inventory/sample inventory/my-cluster

Second, edit inventory/my-cluster/hosts.ini to match the system information gathered above

For example:

[master]
192.168.30.38
192.168.30.39
192.168.30.40

[node]
192.168.30.41
192.168.30.42

[k3s_cluster:children]
master
node

If multiple hosts are in the master group, the playbook will automatically set up k3s in HA mode with etcd.

Finally, copy ansible.example.cfg to ansible.cfg and adapt the inventory path to match the files that you just created.

This requires at least k3s version 1.19.1 however the version is configurable by using the k3s_version variable.

If needed, you can also edit inventory/my-cluster/group_vars/all.yml to match your environment.

☸️ Create Cluster

Start provisioning of the cluster using the following command:

ansible-playbook site.yml -i inventory/my-cluster/hosts.ini

After deployment control plane will be accessible via virtual ip-address which is defined in inventory/group_vars/all.yml as apiserver_endpoint

πŸ”₯ Remove k3s cluster

ansible-playbook reset.yml -i inventory/my-cluster/hosts.ini

You should also reboot these nodes due to the VIP not being destroyed

βš™οΈ Kube Config

To copy your kube config locally so that you can access your Kubernetes cluster run:

scp debian@master_ip:/etc/rancher/k3s/k3s.yaml ~/.kube/config

If you get file Permission denied, go into the node and temporarly run:

sudo chmod 777 /etc/rancher/k3s/k3s.yaml

Then copy with the scp command and reset the permissions back to:

sudo chmod 600 /etc/rancher/k3s/k3s.yaml

You'll then want to modify the config to point to master IP by running:

sudo nano ~/.kube/config

Then change server: https://127.0.0.1:6443 to match your master IP: server: https://192.168.1.222:6443

πŸ”¨ Testing your cluster

See the commands here.

Variables

Role(s)VariableTypeDefaultRequiredDescription
downloadk3s_versionstring❌RequiredK3s binaries version
k3s_agent, k3s_server, k3s_server_postapiserver_endpointstring❌RequiredVirtual ip-address configured on each master
k3s_agentextra_agent_argsstringnullNot requiredExtra arguments for agents nodes
k3s_agent, k3s_servergroup_name_masterstringnullNot requiredName othe master group
k3s_agentk3s_tokenstringnullNot requiredToken used to communicate between masters
k3s_agent, k3s_serverproxy_envdictnullNot requiredInternet proxy configurations
k3s_agent, k3s_serverproxy_env.HTTP_PROXYstring❌RequiredHTTP internet proxy
k3s_agent, k3s_serverproxy_env.HTTPS_PROXYstring❌RequiredHTTP internet proxy
k3s_agent, k3s_serverproxy_env.NO_PROXYstring❌RequiredAddresses that will not use the proxies
k3s_agent, k3s_server, resetsystemd_dirstring/etc/systemd/systemNot requiredPath to systemd services
k3s_custom_registriescustom_registries_yamlstring❌RequiredYAML block defining custom registries. The following is an example that pulls all images used in this playbook through your private registries. It also allows you to pull your own images from your private registry, without having to use imagePullSecrets in your deployments. If all you need is your own images and you don't care about caching the docker/quay/ghcr.io images, you can just remove those from the mirrors: section.
k3s_server, k3s_server_postcilium_bgpbool~Not requiredEnable cilium BGP control plane for LB services and pod cidrs. Disables the use of MetalLB.
k3s_server, k3s_server_postcilium_ifacestring❌Not requiredThe network interface used for when Cilium is enabled
k3s_serverextra_server_argsstring""Not requiredExtra arguments for server nodes
k3s_serverk3s_create_kubectl_symlinkboolfalseNot requiredCreate the kubectl -> k3s symlink
k3s_serverk3s_create_crictl_symlinkbooltrueNot requiredCreate the crictl -> k3s symlink
k3s_serverkube_vip_arpbooltrueNot requiredEnables kube-vip ARP broadcasts
k3s_serverkube_vip_bgpboolfalseNot requiredEnables kube-vip BGP peering
k3s_serverkube_vip_bgp_routeridstring"127.0.0.1"Not requiredDefines the router ID for the kube-vip BGP server
k3s_serverkube_vip_bgp_asstring"64513"Not requiredDefines the AS for the kube-vip BGP server
k3s_serverkube_vip_bgp_peeraddressstring"192.168.30.1"Not requiredDefines the address for the kube-vip BGP peer
k3s_serverkube_vip_bgp_peerasstring"64512"Not requiredDefines the AS for the kube-vip BGP peer
k3s_serverkube_vip_bgp_peerslist[]Not requiredList of BGP peer ASN & address pairs
k3s_serverkube_vip_bgp_peers_groupslist['k3s_master']Not requiredInventory group in which to search for additional kube_vip_bgp_peers parameters to merge.
k3s_serverkube_vip_ifacestring~Not requiredExplicitly define an interface that ALL control nodes should use to propagate the VIP, define it here. Otherwise, kube-vip will determine the right interface automatically at runtime.
k3s_serverkube_vip_tag_versionstringv0.7.2Not requiredImage tag for kube-vip
k3s_serverkube_vip_cloud_provider_tag_versionstringmainNot requiredTag for kube-vip-cloud-provider manifest when enable
k3s_server, k3_server_postkube_vip_lb_ip_rangestring~Not requiredIP range for kube-vip load balancer
k3s_server, k3s_server_postmetal_lb_controller_tag_versionstringv0.14.3Not requiredImage tag for MetalLB
k3s_servermetal_lb_speaker_tag_versionstringv0.14.3Not requiredImage tag for MetalLB
k3s_servermetal_lb_typestringnativeNot requiredUse FRR mode or native. Valid values are frr and native
k3s_serverretry_countint20Not requiredAmount of retries when verifying that nodes joined
k3s_serverserver_init_argsstring❌Not requiredArguments for server nodes
k3s_server_postbpf_lb_algorithmstringmaglevNot requiredBPF lb algorithm
k3s_server_postbpf_lb_modestringhybridNot requiredBPF lb mode
k3s_server_postcalico_blocksizeint26Not requiredIP pool block size
k3s_server_postcalico_ebpfboolfalseNot requiredUse eBPF dataplane instead of iptables
k3s_server_postcalico_encapsulationstringVXLANCrossSubnetNot requiredIP pool encapsulation
k3s_server_postcalico_natOutgoingstringEnabledNot requiredIP pool NAT outgoing
k3s_server_postcalico_nodeSelectorstringall()Not requiredIP pool node selector
k3s_server_postcalico_ifacestring~Not requiredThe network interface used for when Calico is enabled
k3s_server_postcalico_tagstringv3.27.2Not requiredCalico version tag
k3s_server_postcilium_bgp_my_asnint64513Not requiredLocal ASN for BGP peer
k3s_server_postcilium_bgp_peer_asnint64512Not requiredBGP peer ASN
k3s_server_postcilium_bgp_peer_addressstring~Not requiredBGP peer address
k3s_server_postcilium_bgp_neighborslist[]Not requiredList of BGP peer ASN & address pairs
k3s_server_postcilium_bgp_neighbors_groupslist['k3s_all']Not requiredInventory group in which to search for additional cilium_bgp_neighbors parameters to merge.
k3s_server_postcilium_bgp_lb_cidrstring192.168.31.0/24Not requiredBGP load balancer IP range
k3s_server_postcilium_exportPodCIDRbooltrueNot requiredExport pod CIDR
k3s_server_postcilium_hubblebooltrueNot requiredEnable Cilium Hubble
k3s_server_postcilium_hubblebooltrueNot requiredEnable Cilium Hubble
k3s_server_postcilium_modestringnativeNot requiredInner-node communication mode (choices are native and routed)
k3s_server_postcluster_cidrstring10.52.0.0/16Not requiredInner-cluster IP range
k3s_server_postenable_bpf_masqueradebooltrueNot requiredUse IP masquerading
k3s_server_postkube_proxy_replacementbooltrueNot requiredReplace the native kube-proxy with Cilium
k3s_server_postmetal_lb_available_timeoutstring240sNot requiredWait for MetalLB resources
k3s_server_postmetal_lb_ip_rangestring192.168.30.80-192.168.30.90Not requiredMetalLB ip range for load balancer
k3s_server_postmetal_lb_controller_tag_versionstringv0.14.3Not requiredImage tag for MetalLB
k3s_server_postmetal_lb_modestringlayer2Not requiredMetallb mode (choices are bgp and layer2)
k3s_server_postmetal_lb_bgp_my_asnstring~Not requiredBGP ASN configurations
k3s_server_postmetal_lb_bgp_peer_asnstring~Not requiredBGP peer ASN configurations
k3s_server_postmetal_lb_bgp_peer_addressstring~Not requiredBGP peer address
lxccustom_reboot_commandstring~Not requiredCommand to run on reboot
prereqsystem_timezonestringnullNot requiredTimezone to be set on all nodes
proxmox_lxc, reset_proxmox_lxcproxmox_lxc_ct_idslist❌RequiredProxmox container ID list
raspberrypistatestringpresentNot requiredIndicates whether the k3s prerequisites for Raspberry Pi should be set up (possible values are present and absent)

Troubleshooting

Be sure to see this post on how to troubleshoot common problems

Testing the playbook using molecule

This playbook includes a molecule-based test setup. It is run automatically in CI, but you can also run the tests locally. This might be helpful for quick feedback in a few cases. You can find more information about it here.

Pre-commit Hooks

This repo uses pre-commit and pre-commit-hooks to lint and fix common style and syntax errors. Be sure to install python packages and then run pre-commit install. For more information, see pre-commit

🌌 Ansible Galaxy

This collection can now be used in larger ansible projects.

Instructions:

collections:
  - name: ansible.utils
  - name: community.general
  - name: ansible.posix
  - name: kubernetes.core
  - name: https://github.com/techno-tim/k3s-ansible.git
    type: git
    version: master

Thanks 🀝

This repo is really standing on the shoulders of giants. Thank you to all those who have contributed and thanks to these repos for code and ideas: