Home

Awesome

Jenkins Blue Ocean Kubernetes Ubuntu 16.04

Quickly provision Jenkins blue ocean on kubernetes v1.9 with RBAC and with persistent configuration. Go from a simple 16.04 VM to a portable, scalable CI-CD pipeline on kubernetes with automated SSL provisioned and automatic building on push to github. The tutorial takes around 20 minutes. The Jenkins instance can be deleted and moved between clouds while retaining the job configuration via the excellent Jenkins Helm chart https://github.com/kubernetes/charts/tree/master/stable/jenkins.

The example shown will use a single Hetzner server ($0.04/hour), but this first step can skipped, and ssh access to an ubuntu 16.04 machine can be used instead.

The sample CI-CD pipeline and application uses Lachlan Evanson's excellent croc-hunter tutorial, which I would encourage you to go through after this. https://github.com/lachie83/croc-hunter

Inspiration was also taken from Nick Johnson's excellent upcoming book 'Continuous Deployment with Kubernetes' (https://withku.be/)

Checkout repository

git clone https://github.com/EamonKeane/jenkins-blue.git

cd jenkins-blue

Setup with Hetzner Cloud

  1. Register on Hetzner (https://www.hetzner.com/cloud)
  2. Get API token from dashboard
  3. Install hcloud cli: brew install hetznercloud/tap/hcloud (https://github.com/hetznercloud/cli)
  4. hcloud ssh-key create --name $KEY_NAME --public-key-from-file ~/.ssh/id_rsa.pub
  5. hcloud context create jenkins-blue-ocean. Enter token when prompted
  6. Note your ssh-key ID returned from: hcloud ssh-key list
SERVER_NAME=jenkins-blue-ocean # replace this with your preferred name

SSH_KEY=7170 #replace with your ssh-key id here

SERVER_TYPE=cx41 # Machine with 16GB of ram, 4 vCPU, 160 GB disk (no commitment, $25 per month, 80% less than t2.xlarge on EC2)

To install a single node kubeadm on hetzner run (this will take around 4 minutes): https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

./kubernetes-hetzner.sh --SERVER_NAME=$SERVER_NAME --ssh-key=$SSH_KEY --SERVER_TYPE=$SERVER_TYPE

Set the jenkins ip variable:

JENKINS_IP=$(hcloud server list | grep -E $SERVER_NAME | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b");echo $JENKINS_IP

With ssh access to an ubuntu 16.04 machine

SSH_USER=root

JENKINS_IP=00.00.00.00 #Enter your machine IP here

To install a single node kubernetes kubeadm cluster run (this will take around 4 minutes): (kubernetes install commands from: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/)

./kubernetes-ubuntu1604.sh --SSH_USER=$SSH_USER --JENKINS_IP=$JENKINS_IP

Create DNS A-record

JENKINS_URL=jenkins.mysite.io # replace with your jenkins url

CROC_HUNTER_URL=croc-hunter.mysite.io # replace with your croc-hunter url

watch -n 5 dig $JENKINS_URL


watch -n 5 dig $CROC_HUNTER_URL

Fork the croc-hunter repo with example application

This contains a lot of best practice and contains a Jenkinsfile which is required to demonstrate Blue Ocean functionality. Alternatively specify your own project which has a Jenkinsfile. https://github.com/lachie83/croc-hunter/

https://github.com/EamonKeane/croc-hunter

Returning to the shell, run the following commands:

ORGANISATION=EamonKeane # enter your organisation (github username)

cd ..

git clone https://github.com/$ORGANISATION/croc-hunter.git

cd croc-hunter

IMAGE_REPOSITORY_ORGANISATION=eamonkeane

IMAGE_REPOSITORY_URL=quay.io/eamonkeane/croc-hunter

Update the values for croc-hunter to match your values (brew install jq)

jq ".app.hostname = \"$CROC_HUNTER_URL\"" Jenkinsfile.json > tmp.json && mv tmp.json Jenkinsfile.json

jq ".container_repo.master_acct = \"$IMAGE_REPOSITORY_ORGANISATION\"" Jenkinsfile.json > tmp.json && mv tmp.json Jenkinsfile.json

jq ".container_repo.image = \"$IMAGE_REPOSITORY_URL\"" Jenkinsfile.json > tmp.json && mv tmp.json Jenkinsfile.json

git add -A; git commit -m "changed croc hunter url and image repo"; git push origin master

Install jenkins to configure jobs and retrieve secrets

Prerequisites:

Export the kubectl config copied from the kubeadm machine:

cd ../jenkins-blue
export KUBECONFIG=$PWD/admin.conf

Create kubernetes image pull secret for croc-hunter

DOCKER_SERVER=quay.io

DOCKER_USERNAME=eamonkeane+crochunter

DOCKER_PASSWORD=

DOCKER_EMAIL=. # This is not important and can be left as a dot

kubectl create namespace croc-hunter

kubectl create secret docker-registry croc-hunter-secrets --namespace=croc-hunter --docker-server=$DOCKER_SERVER --docker-username=$DOCKER_USERNAME --docker-password=$DOCKER_PASSWORD --docker-email=$DOCKER_EMAIL

Jenkins Installation and Configuration

Replace your jenkins url in the hostname, TLS secret name, and TLS secret sections of jenkins-values-initial.yaml and jenkins-values.yaml

sed -i '' -e "s/jenkins\.mysite\.io/$JENKINS_URL/g" jenkins-values.yaml

sed -i '' -e "s/jenkins\.mysite\.io/$JENKINS_URL/g" jenkins-values-initial.yaml

Initial temporary installation of jenkins. This takes approx 4 minutes. This also installs nginx-ingress (configured for bare metal) and cert-manager (configured to auto-provision SSL certs) :

./jenkins-initial-install.sh 

echo https://$JENKINS_URL

printf $(kubectl get secret --namespace jenkins jenkins-jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo
  1. Click on Credentials
  2. Click on Jenkins link
  3. Click on Global Credentials
  4. Click add credentials
  5. Select Username and password
  6. Enter $DOCKER_USERNAME and $DOCKER_PASSWORD as above
  7. Enter ID as quay_creds
  8. Enter description as your choice e.g. croc-hunter-quay-creds
  9. Press OK
  1. Click on Jenkins Blue Ocean in side bar on home page
  2. Click on Create Pipeline
  3. Click on Github
  4. Click on 'create an access key here'
  5. Login to Github, enter token name, click generate token, copy token to clipboard
  6. Paste token into jenkins and click connect
  7. Select organisation and croc-hunter repo

Add github webhook

AUTH_TOKEN= #put your github API token here

ORGANISATION=EamonKeane #replace this with your github username or organisation

REPOSITORY=croc-hunter #replace this with your github repo if not using croc-hunter

github-webhook/create-github-webhook.sh --AUTH_TOKEN=$AUTH_TOKEN --SERVICE_URL=$JENKINS_URL --ORGANISATION=$ORGANISATION --REPOSITORY=$REPOSITORY

Verify the application deploys

Click on the master branch on Jenkins blue ocean. After approximately 5 minutes, the application and test will have completed. The croc-hunter application will be available on https://$CROC_HUNTER_URL

Copy jenkins configuration

./copy-jenkins-config.sh

Persist Jenkins data in helm chart

  {{- $files := .Files }}
  {{- range tuple "blue-ocean-config.xml" }}
  {{ . }}: |-
    {{ $files.Get . }}
  {{- end }}


    mkdir -p /var/jenkins_home/users/admin/;
    cp -n /var/jenkins_config/blue_ocean_credentials.xml /var/jenkins_home/users/admin/config.xml;


echo "    croc-hunter: |-" >> jenkins-jobs.yaml

cat jenkins-jobs/croc-hunter/config.xml | sed 's/^/      /' >> jenkins-jobs.yaml

The jenkins-jobs.yaml should look like the below

Master:
  Jobs: |-
    croc-hunter: |-
      <?xml version='1.0' encoding='UTF-8'?>

Nuke the jenkins installation

helm del --purge jenkins

Install jenkins with values persisted

kubectl create -f kubernetes-yaml/jenkins-pv.yaml

kubectl create -f kubernetes-yaml/jenkins-pvc.yaml

kubectl get pvc -n jenkins


helm upgrade --install --namespace jenkins --wait --values jenkins-values.yaml --values jenkins-jobs.yaml jenkins jenkins/

Make a change to croc-hunter repository

cd ../croc-hunter

echo "change -- ignore" >> README.md

git add -A; git commit -m "made change to README.md"; git push origin master

Login to jenkins

printf $(kubectl get secret --namespace jenkins jenkins-jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo

Click on Jenkins blue ocean and you will now see master building and the croc-hunter app will update.

Tidying up

hcloud server delete $SERVER_NAME

Delete github API tokens and keys from bash history or from github.