Awesome
Sample Python KEDA Service Bus Scaler
Sample Code for Python based Kubernetes Jobs for Event-driven Autoscaling using KEDA Azure Service Bus Scaler.
Prerequisite
Setup
-
Create an Azure Service Bus Queue
-
Deploy KEDA 2.0+ in AKS Cluster
-
Get the Azure Service Bus connection string
-
Get base64 encoded values of -
- Connection String:
echo -n "<connection string>" | base64
- Queue Name:
echo -n "<queue name>" | base64
- Connection String:
-
Modify the following configuration files -
-
keda-config/jobs/secret.yaml: update
servicebus-queue
andservicebus-connectionstring
by the values from above commandsdata: servicebus-queue: <base64_encoded_servicebus_queue_name> servicebus-connectionstring: <base64_encoded_servicebus_connection_string>
-
keda-config/jobs/keda.yaml: update
servicebus_queue_name
andservicebus_namespace
with plain text values of respective queue name and namespace nametriggers: - type: azure-servicebus metadata: # Required: queueName OR topicName and subscriptionName queueName: <servicebus_queue_name> # Required: Define what Azure Service Bus to authenticate to with Managed Identity namespace: <servicebus_namespace>
-
-
Deploy the KEDA enabled Sample Kubernets Job (in
default
namespace)
kubectl apply -f keda-config/jobs/secret.yaml
kubectl apply -f keda-config/jobs/keda.yaml
- Verify deployment
~/sample-python-keda-service-bus-scaler(main*) » kubectl get ScaledJob
NAME MAX TRIGGERS AUTHENTICATION READY ACTIVE AGE
servicebus-queue-so-sample-job 100 azure-servicebus auth-service-bus-sample-job 1s
--------------------------------------------------------------------------------------------------------------
~/sample-python-keda-service-bus-scaler(main*) » kubectl get TriggerAuthentication
NAME PODIDENTITY SECRET ENV
auth-service-bus-sample-job sample-job-secret
Test
-
Populate message in Azure Service Bus Queue
-
Modify the following configuration file -
- test-message-creator/azure-service-bus.yaml: update
servicebus_queue_name
andservicebus_connection_string
with plain text values of respective queue name and connection string
- test-message-creator/azure-service-bus.yaml: update
-
Execute -
pip3 install -r requirements.txt cd test-message-creator && python3 send_message_queue.py
-
Expected result -
[07/Jun/2021 15:57:45] INFO - Start sending messages ......... [07/Jun/2021 15:57:53] INFO - Sent a list of 1000 messages ......... [07/Jun/2021 15:57:53] INFO - Done sending messages [07/Jun/2021 15:57:53] INFO - -----------------------
-
-
Monitor AKS Cluster and Azure Service Bus Queue
-
Monitor AKS Cluster and check if the Job (100) got created as soon as there are messages populated in Azure Service Bus Queue
~/sample-python-keda-service-bus-scaler(main*) » kubectl get pods --selector=app=sample-keda-job | grep -v '^NAME' | wc -l 100 -------------------------------------------------------------------------------------------------------------- ~/sample-python-keda-service-bus-scaler(main*) » kubectl get pods NAME READY STATUS RESTARTS AGE servicebus-queue-so-sample-job-26htm-hw7cm 0/1 Completed 0 2m21s servicebus-queue-so-sample-job-26pk7-8sw8g 0/1 Completed 0 2m52s ........ servicebus-queue-so-sample-job-6jp7k-q9tv7 0/1 Completed 0 2m41s servicebus-queue-so-sample-job-6qrjg-pfx5n 0/1 Completed 0 2m47s -------------------------------------------------------------------------------------------------------------- ~/sample-python-keda-service-bus-scaler(main*) » kubectl logs servicebus-queue-so-sample-job-xxfvj-l4t86 started receiving message ========================= Message in list 960 Message in list 961 Message in list 962 Message in list 963 Message in list 964 Message in list 965 Message in list 966 Message in list 967 Message in list 968 Message in list 969 ========================= completed receiving message
-
Check for Messages in Azure Service Bus Respective Queue
- Message Count
- Message Incoming / Outgoing
-
(Optional) Build
- The existing source is built and uploaded to registry -
docker.pkg.github.com/prabdeb/sample-python-keda-service-bus-scaler/consumer:latest
- However if there is any modifications in source code, the docker image can be build by command -
docker build -t sample-python-keda-service-bus-scaler:latest -f Dockerfile-job .
Cleaning up resources
-
Delete sample application
kubectl delete -f keda-config/jobs/keda.yaml kubectl delete -f keda-config/jobs/secret.yaml