Awesome
Tower
Tower is a network tunnel used to proxy KubeSphere API requests to member clusters. Tower is built on top HTTP, secured via SSH. The main idea is stolen from Chisel(https://github.com/jpillora/chisel).
How does it work
Proxy <------> Agent |--> ks-apiserver
|--> kube-apiserver
In short, agents running in member cluster will connect proxy running in host cluster. After successfully handshaking, agents will establish a secured ssh connection with proxy, allowing traffic from host cluster to forward to local kubernetes apiserver and kubesphere apiserver.
How to build
Clone the repo and run make under tower. There will be two binaries generated under directory bin/
$ make
$ ls bin/
proxy agent
How to use it
- Install tower crd in host cluster
$ cd config/crd
$ kustomize build | kubectl create -f -
customresourcedefinition.apiextensions.k8s.io/clusters.cluster.kubesphere.io created
- Run proxy on host cluster, the publish-service-address needs to be a address accessible for all member clusters.
$ ./bin/proxy --ca-cert ./certs/ca.crt --ca-key ./certs/ca.key --host 0.0.0.0 --port 8080 --publish-service-address 192.168.100.3 --kubeconfig ~/.kube/config
I1108 09:57:50.145666 1 options.go:46] CA set to "/ca.crt".
I1108 09:57:50.145795 1 options.go:47] CA key file set to "/ca.key".
I1108 09:57:50.145804 1 options.go:48] Host set to 0.0.0.0
I1108 09:57:50.145811 1 options.go:49] Agent port set to 8080.
I1108 09:57:50.145823 1 options.go:50] Kubeconfig set to "".
I1108 09:57:50.145842 1 options.go:51] Leader election set to false
I1108 09:57:50.275180 1 proxy.go:234] Listening on 0.0.0.0:8080...
- Create a agent object in host cluster
$ cat agent.yaml
apiVersion: cluster.kubesphere.io/v1alpha1
kind: Cluster
metadata:
name: alpha
namespace: kubesphere-system
spec:
connection:
type: proxy
token: ""
joinFederation: true
$ kubectl -n kubesphere-system create -f agent.yaml
- Proxy server will generate token and update cluster object. Use following command to get the token.
$ kubectl get cluster alpha -o jsonpath='{.spec.connection.token}'
6888951db2eef4323fd4f84d05f490be86af102f43e47c3c58e6a1ddc81cc253
- Run agent in member cluster with token get from previous step
$ ./bin/agent --kubeconfig ~/.kube/config --token 6888951db2eef4323fd4f84d05f490be86af102f43e47c3c58e6a1ddc81cc253 --name alpha --v 4
I0325 10:25:40.729137 1 agent.go:210] Handshaking...
I0325 10:25:40.748094 1 agent.go:126] fingerprint7b:fa:2d:64:08:95:4d:d4:74:18:3e:78:39:03:76:ce
I0325 10:25:40.753106 1 agent.go:222] Sending config
I0325 10:25:40.757505 1 agent.go:231] Connected.
- Now your member cluster is connected to host cluster, you can access member cluster kube-apiserver or ks-apiserver in host cluster
$ kubectl -n kubesphere-system get cluster alpha -o jsonpath='{.spec.connection.kubeconfig}' | base64 -d > config_alpha
$ kubectl --kubeconfig config_alpha get node
NAME STATUS ROLES AGE VERSION
master Ready master 2d3h v1.17.3
node1 Ready worker 2d3h v1.17.3
node2 Ready worker 2d3h v1.17.3
node3 Ready worker 2d3h v1.17.3
- Use tower to make a member cluster kubeapi accessable to public
If you want to make you member cluster kubeapi accessable to public, create a cluster resource as follows:
apiVersion: cluster.kubesphere.io/v1alpha1
kind: Cluster
metadata:
name: kind-test
namespace: kubesphere-system
annotations:
tower.kubesphere.io/external-lb-service-annoations: '{"eip.porter.kubesphere.io/v1alpha2":"porter-bgp-eip","lb.kubesphere.io/v1alpha1":"porter","protocol.porter.kubesphere.io/v1alpha1":"bgp"}'
spec:
connection:
type: proxy
token: ""
joinFederation: true
externalKubeAPIEnabled: true
With externalKubeAPIEnabled=true and connection.type=proxy tower will create the serivce with LoadBlancer type, content in annotation with key tower.kubesphere.io/external-lb-service-annoations will be applied to the service anntations as k-v, so that your can control how the ccm process the service.