Awesome
k8s-infra
Quick start
You'll need a set of ip addresses to create the K8s cluster.
Optionally use the infra-provisioning to provision machines (eg. Equinix Metal)
Build the prerequisites
docker build -t crosscloudci/k8s-infra-deps:latest --file Dockerfile.deps .
Build the Docker container
docker build -t crosscloudci/k8s-infra:latest .
Generate the kube-spray configuration file and provision the K8s cluster
Note: You must have either 3 or more masters. It doesn't matter how many workers you have.
docker run -v $(pwd):/k8s-infra:latest -v -ti kubespray /bin/bash
Inside of the container:
./k8s-infra/bin/k8sinfra generate_config --release-type=stable --master-hosts "<your-ip-address>,<your-ip-address>,<your-ip-address>" --worker-hosts "<your-ip-address>,<your-ip-address>,<your-ip-address>" -o /tmp/test.yml; \
./k8sinfra provision --config-file '/tmp/test.yml'"
Save the resulting kubeconfig file to your local development machine.
Useful Developer Commands
Build deps for k8s-infra
docker build -t crosscloudci/k8s-infra-deps:latest --file Dockerfile.deps .
Build k8s-infra
docker build -t crosscloudci/k8s-infra:latest .
Optional: Push to dockerhub repository
docker push crosscloudci/k8s-infra-deps:latest
Optional: push of k8s-infra docker image
docker push crosscloudci/k8s-infra:latest
Test the docker image
docker run -ti crosscloudci/k8s-infra:latest
Test with port mapping
docker run -e <env var>=<env data> -ti crosscloudci/k8s-infra:latest -p 4001:4001
Get name of the container
docker ps
Optional: Bash prompt
docker exec -ti <name of container> /bin/bash
Get docker ip address
docker exec -ti <name of container> ifconfig
Test manually adding ip addresses
docker run -ti crosscloudci/k8s-infra:latest k8s-infra/bin/k8sinfra generate_config --master-hosts "1.1.1.1,2.2.2.2,3.3.3.3" --worker-hosts "3.3.3.3,4.4.4.4,5.5.5.5"
Test manually adding ip addresses with generated cluster yml and provision generation with head release type
docker run -ti crosscloudci/k8s-infra:latest /bin/bash -c "k8s-infra/bin/k8sinfra generate_config --release-type=head --master-hosts "1.1.1.1,2.2.2.2,3.3.3.3" --worker-hosts "3.3.3.3,4.4.4.4,5.5.5.5" -o /tmp/test.yml; \
k8s-infra/bin/k8sinfra provision --config-file '/tmp/test.yml' --dry-run"
Test manually adding ip addresses with generated cluster yml and provision generation with stable release type
docker run -ti crosscloudci/k8s-infra:latest /bin/bash -c "k8s-infra/bin/k8sinfra generate_config --release-type=stable --master-hosts "1.1.1.1,2.2.2.2,3.3.3.3" --worker-hosts "3.3.3.3,4.4.4.4,5.5.5.5" -o /tmp/test.yml; \
k8s-infra/bin/k8sinfra provision --config-file '/tmp/test.yml' --dry-run"
Testing with the gitlab integration
To test with the gitlab integration (to get the source IPs from gitlab instead of manually providing them), you can use one of the following two options:
Pre-req: Successful provision Packet machines using infra-provisioning Option 1: use existing pipeline
Go to https://gitlab.cncf.ci/cncf/infra-provisioning/pipelines Find a successful pipeline and open the release job. Under the artifacts section select browse, then download the nodes.env file under the top-level Terraform directory. example url path: https://gitlab.cncf.ci/cncf/infra-provisioning/-/jobs/168517/artifacts/browse/terraform/nodes.env Option 2: run new pipeline
Go to https://gitlab.cncf.ci/cncf/infra-provisioning/pipelines Select run pipeline, then create a pipeline against the production branch. Once the job has finished open the release job. Under the artifacts section select browse, then download the nodes.env file under the top-level Terraform directory. example url path: https://gitlab.cncf.ci/cncf/infra-provisioning/-/jobs/168517/artifacts/browse/terraform/nodes.env
Test gitlab integration
stable
docker run -ti crosscloudci/k8s-infra:latest k8s-infra/bin/k8sinfra generate_config --release-type=stable --infra-job=168517
head
docker run -ti crosscloudci/k8s-infra:latest k8s-infra/bin/k8sinfra generate_config --release-type=head --infra-job=168517
Saving to a file
docker run -ti crosscloudci/k8s-infra:latest k8s-infra/bin/k8sinfra generate_config --infra-job=168517 -o /tmp/cluster.yml
Error message if cluster.yml structure is not valid
docker run -ti crosscloudci/k8s-infra:latest k8s-infra/bin/k8sinfra provision --config-file=k8s-infra/example_hosts-invalid_syntax.yml
Error message if cluster.yml structure is not valid
docker run -ti crosscloudci/k8s-infra:latest k8s-infra/bin/k8sinfra provision --config-file=k8s-infra/example_hosts-invalid_structure.yml
TESTING
Develop with rspec with the installed kubespray dependencies by mapping the k8s-infra directory into the container and running rspec spec
docker run -v $(pwd):/k8s-infra -v /home/pair/.ssh/id_rsa:/root/.ssh/id_rsa -ti kubespray /bin/bash