Awesome

What is kube-loxilb ?

kube-loxilb is loxilb's implementation of kubernetes service load-balancer spec which includes support for load-balancer class, advanced IPAM (shared or exclusive) etc. kube-loxilb runs as a deloyment set in kube-system namespace. This component runs inside k8s cluster to gather information about k8s nodes/reachability/LB services etc but in itself does not implement packet/session load-balancing. It is done by loxilb which usually runs outside the cluster as an external-LB.

Many users frequently ask us whether it is possible to run the actual packet/session load-balancing inside the cluster (in worker-nodes or master-nodes). The answer is "yes". loxilb can be run in-cluster or as an external entity. The preferred way is to run <b>kube-loxilb</b> component inside the cluster and provision <b>loxilb</b> docker in any external node/vm as mentioned in this guide. The rationale is to provide users a similar look and feel whether running loxilb in an on-prem or public cloud environment. Public-cloud environments usually run load-balancers/firewalls externally in order to provide a seamless and safe environment for the cloud-native workloads. But users are free to choose any mode (in-cluster mode or external mode) as per convenience and their system architecture. The following blogs give detailed steps for :

1. Running loxilb in external node with AWS EKS
2. Running in-cluster LB with K3s for on-prem use-cases

This usually leads to another query - Who will be responsible for managing the external node ? On public cloud(s), it is as simple as spawning a new instance in your VPC and launch loxilb docker in it. For on-prem cases, you need to run loxilb docker in a spare node/vm as applicable. loxilb docker is a self-contained entity and easily managed with well-known tools like docker, containerd, podman etc. It can be independently restarted/upgraded anytime and kube-loxilb will make sure all the k8s LB services are properly configured each time. When deploying in-cluster mode, everything is managed by Kubernetes itself with little to no manual intervention.

How is kube-loxilb different from loxi-ccm ?

Another loxilb component known as loxi-ccm also provides implementation of kubernetes load-balancer spec but it runs as a part of cloud-provider and provides load-balancer life-cycle management as part of it. If one needs to integrate loxilb with their existing cloud-provider implementation, they can use or include loxi-ccm as a part of it. Else, kube-loxilb is the right component to use for all scenarios. It also has the latest loxilb features integrated as development is currently focused on it.

kube-loxilb is a standalone implementation of kubernetes load-balancer spec which does not depend on cloud-provider. It runs as a kube-system deployment and provisions load-balancer rules in loxilb based on load-balancer class. It only acts on load-balancers services for the LB classes that is provided by itself. This along with loxilb's modularized architecture also allows us to have different load-balancers working together in the same K8s environment. In future, loxi-ccm and kube-loxilb will share the same code base but currently they are maintained separately.

Overall topology

• For external mode, the overall topology including all components should be similar to the following :

• For in-cluster mode, the overall topology including all components should be similar to the following :

How to deploy kube-loxilb ?

• If you have chosen external-mode, please make sure loxilb docker is downloaded and installed properly in a node external to your cluster. One can follow guides here or refer to various other documentation . It is important to have network connectivity from this node to the k8s cluster nodes (where kube-loxilb will eventually run) as seen in the above figure. (PS - This step can be skipped if running in-cluster mode)

• Download the kube-loxilb config yaml :

wget https://github.com/loxilb-io/kube-loxilb/raw/main/manifest/ext-cluster/kube-loxilb.yaml

• Modify arguments as per user's needs :

        args:
            - --loxiURL=http://12.12.12.1:11111
            - --cidrPools=defaultPool=123.123.123.1/24
            #- --cidrPools=defaultPool=123.123.123.1/24,pool2=124.124.124.1/24
            #- --cidr6Pools=defaultPool=3ffe::1/96
            #- --monitor
            #- --setBGP=65100
            #- --extBGPPeers=50.50.50.1:65101,51.51.51.1:65102
            #- --enableBGPCRDs
            #- --setRoles=0.0.0.0
            #- --setLBMode=1
            #- --setUniqueIP=false

The arguments have the following meaning :

Many of the above flags and arguments can be overriden on a per-service basis based on loxilb specific annotation as mentioned below.

• kube-loxilb supported annotations:

• Apply the yaml after making necessary changes :

kubectl apply -f kube-loxilb.yaml

• The above should make sure kube-loxilb is successfully running. Check kube-loxilb is running :

k8s@master:~$ sudo kubectl get pods -A
NAMESPACE         NAME                                        READY   STATUS    RESTARTS   AGE
kube-system       local-path-provisioner-84db5d44d9-pczhz     1/1     Running   0          16h
kube-system       coredns-6799fbcd5-44qpx                     1/1     Running   0          16h
kube-system       metrics-server-67c658944b-t4x5d             1/1     Running   0          16h
kube-system       kube-loxilb-5fb5566999-ll4gs                1/1     Running   0          14h

• Finally to create service LB for a workload, we can use and apply the following template yaml

(<b>Note</b> - Check <b>loadBalancerClass</b> and other <b>loxilb</b> specific annotation) :

        apiVersion: v1
        kind: Service
        metadata:
          name: iperf-service
          annotations:
           # If there is a need to do liveness check from loxilb
           loxilb.io/liveness: "yes"
           # Specify LB mode - one of default, onearm or fullnat 
           loxilb.io/lbmode: "default"
           # Specify loxilb IPAM mode - one of ipv4, ipv6 or ipv6to4 
           loxilb.io/ipam: "ipv4"
           # Specify cidr pool for allocating externalIP
           # If not specified defaults to "defaultPool"
           # loxilb.io/poolSelect: "pool1"
           # Specify cidr pool of secondary networks for multi-homing
           # Only valid for SCTP currently
           # loxilb.io/poolSelectSecondary: "pool2,pool3"
           # Specify a static externalIP for this service
           # loxilb.io/staticIP: "123.123.123.2"
        spec:
          loadBalancerClass: loxilb.io/loxilb
          selector:
            what: perf-test
          ports:
            - port: 55001
              targetPort: 5001
          type: LoadBalancer
        ---
        apiVersion: v1
        kind: Pod
        metadata:
          name: iperf1
          labels:
            what: perf-test
        spec:
          containers:
            - name: iperf
              image: eyes852/ubuntu-iperf-test:0.5
              command:
                - iperf
                - "-s"
              ports:
                - containerPort: 5001

Users can change the above as per their needs.

• Verify LB service is created

k8s@master:~$ sudo kubectl get svc
NAME                TYPE           CLUSTER-IP    EXTERNAL-IP         PORT(S)             AGE
kubernetes          ClusterIP      10.43.0.1     <none>              443/TCP             13h
iperf1              LoadBalancer   10.43.8.156   llb-192.168.80.20   55001:5001/TCP      8m20s

• For more example yaml templates, kindly refer to kube-loxilb's manifest directory

Additional steps to deploy loxilb (in-cluster) mode

To run loxilb in-cluster mode, the URL argument in kube-loxilb.yaml needs to be commented out:

        args:
            #- --loxiURL=http://12.12.12.1:11111
            - --cidrPools=defaultPool=123.123.123.1/24

This enables a self-discovery mode of kube-loxilb where it can find and reach loxilb pods running inside the cluster. Last but not the least we need to create the loxilb pods in cluster :

sudo kubectl apply -f https://github.com/loxilb-io/kube-loxilb/raw/main/manifest/in-cluster/loxilb.yaml

Once all the pods are created, the same can be verified as follows (you can see both kube-loxilb and loxilb components running:

k8s@master:~$ sudo kubectl get pods -A
NAMESPACE         NAME                                        READY   STATUS    RESTARTS   AGE
kube-system       local-path-provisioner-84db5d44d9-pczhz     1/1     Running   0          16h
kube-system       coredns-6799fbcd5-44qpx                     1/1     Running   0          16h
kube-system       metrics-server-67c658944b-t4x5d             1/1     Running   0          16h
kube-system       kube-loxilb-5fb5566999-ll4gs                1/1     Running   0          14h
kube-system       loxilb-lb-mklj2                             1/1     Running   0          13h
kube-system       loxilb-lb-stp5k                             1/1     Running   0          13h
kube-system       loxilb-lb-j8fc6                             1/1     Running   0          13h
kube-system       loxilb-lb-5m85p                             1/1     Running   0          13h

Thereafter, the process of service creation remains the same as explained in previous sections.

How to use kube-loxilb CRDs ?

Kube-loxilb provides Custom Resource Definition (CRD). Current the following operations are supported (which would be continually updated):

• Add a BGP Peer
• Delete a BGP Peer
• Add/Delete a BGP Policy

For information on BGP Policy CRD, please refer here.

An example of CRD is stored in manifest/crds. Setting up a BGP Peer as an example is as follows:

1. Pre-Processing (Register kube-loxilb CRDs with K8s).

First of all change the kube-loxilb.yaml arguments. It need to add - --enableBGPCRDs option.

        args:
            - --loxiURL=http://12.12.12.1:11111
            - --cidrPools=defaultPool=123.123.123.1/24
            #- --cidrPools=defaultPool=123.123.123.1/24,pool2=124.124.124.1/24,pool3=125.125.125.1/24
            #- --cidr6Pools=defaultPool==3ffe::1/96
            #- --monitor
            #- --setBGP=65100
            #- --extBGPPeers=50.50.50.1:65101,51.51.51.1:65102
            - --enableBGPCRDs
            #- --setRoles=0.0.0.0
            #- --setLBMode=1
            #- --setUniqueIP=false

And Apply lbpeercrd.yaml

kubectl apply -f manifest/crds/lbpeercrd.yaml

2. CRD definition

You need to create a yaml file that adds a peer for BGP. The example below is an example of creating a Peer with a RemoteAS number of Peer IP address 65123 at 123.123.123.2. Create a file named bgp-peer.yaml and add the contents below.

apiVersion: "bgppeer.loxilb.io/v1"
kind: BGPPeerService
metadata:
  name: bgp-peer-test
spec:
  ipAddress: 123.123.123.2
  remoteAs: 65123
  remotePort: 179

3. Apply CRD to add a new BGP Peer

kubectl apply -f bgp-peer.yaml

4. Verify the applied CRD

You can check it in two ways. The first one can be checked through loxicmd(in loxilb container), and the second one can be checked through kubectl.

# loxicmd
kubectl exec -it {loxilb} -n kube-system -- loxicmd get bgpneigh 
|      PEER      |  AS   |   UP/DOWN   |    STATE    | 
|----------------|-------|-------------|-------------|
| 123.123.123.2  | 65123 | never       | ACTIVE      |

# kubectl
kubectl get bgppeerservice
NAME            PEER            AS   
bgp-peer-test   123.123.123.2   65123