Awesome
Azure Function (.NET Core) consuming Blobs scaled by KEDA
Following sample includes an Azure function in .Net core that triggers when new blobs added to Azure Blob Storage and scale via KEDA.
The sample blob consumer will recive a single blob at a time(per instance), and write the blob name and size into the log to simulate performing work then delete the blob from the container. When adding blobs, KEDA will drive the container to scale out according to the count of the blobs in given container.
Pre-requisites
- Docker installed
- Azure Function Core Tools v2
- Kubernetes cluster with KEDA v1.1+ installed
- Helm
Tutorial
1. Create a new directory for the function app
mkdir blob_consumer
cd blob_consumer
2. Initialize the directory for functions
func init . --docker
Select dotnet for function worker runtime
3. Add a new blob triggered function
func new
Select BlobTrigger for the template and enter blob_consumer for the function name
4. Create two Azure storage accounts.
We'll create two azure storage accounts. one for AzureWebJobsStorage and one for adding blobs for processing.
You can use the Azure CLI, the Azure cloud shell, or the Azure portal. The following is how you do it using Azure CLI.
<storage-name> would be replaced by two unique storage account names.
az group create -l westus -n hello-keda
az storage account create --sku Standard_LRS --location westus -g hello-keda -n <storage-name1>
az storage account create --sku Standard_LRS --location westus -g hello-keda -n <storage-name2>
5. Update the function metadata with two storage accounts connection strings.
Open the blob_consumer directory in an editor. We'll need to update the both connection strings info for the blob trigger.
Run the below command twice to copy both connection strings for two storage accounts. Replace the <storage-name> with the unique storage account names given for <storage-name1> and <storage-name2> in above step.
az storage account show-connection-string --name <storage-name> --query connectionString
Open local.settings.json which has the local debug connection string settings. Replace the {AzureWebJobsStorage} and {TEST_STORAGE_CONNECTION_STRING} with the above two connection string values:
local.settings.json
{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net;AccountName=mystorageaccount1;AccountKey=shhhh===",
"FUNCTIONS_WORKER_RUNTIME": "dotnet",
"TEST_STORAGE_CONNECTION_STRING": "DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net;AccountName=mystorageaccount2;AccountKey=shhhh===",
"BLOB_SUB_PATH": "blobsubpath/"
}
}
6. Debug and test the function locally (optional)
Start the function locally
func start
Create a text file with sample text for testing the blob processing.
Go to your Azure Storage account in the Azure Portal and open the Storage Explorer. Select the <storage-name1>
Blob container and upload the created text file for processing. ( Refer below image and note that this example use a sub path for detecting blobs hence "Upload to folder" has the value <blobsubpath>
)
You should see your function running locally.
[15/10/2021 10:45:01 PM] Executing 'blob_consumer' (Reason='New blob detected: container-name/blobsubpath/blob-name.txt', Id=a2c528a0-9b49-456d-9680-ec6973b300d7)
[15/10/2021 10:45:01 PM] C# Blob trigger function Processed blob
Name:blob-name.txt
Size: 14 Bytes
[15/10/2021 10:45:01 PM] Deleting blob blob-name.txt
[15/10/2021 10:45:01 PM] Executed 'blob_consumer' (Succeeded, Id=a2c528a0-9b49-456d-9680-ec6973b300d7)
7. Build and push the blob consumer container image
You need to build and push the blob_consumer
Docker image to a container registry before deploying it to the cluster. For example, to use Docker Hub:
export REGISTRY=slurplk
docker build -t blob-consumer .
docker push blob-consumer $REGISTRY/blob-consumer
8. Install KEDA
Follow the instructions to deploy KEDA in your cluster.
To confirm that KEDA has successfully installed you can run the following command and should see the following CRD.
kubectl get customresourcedefinition scaledobjects.keda.k8s.io
NAME CREATED AT
scaledobjects.keda.k8s.io 2020-09-15T01:00:59Z
9. Deploy Function App using Helm
First you need to update the deploy\blob-consumer\templates\secret.yaml
file and add the two blob storage connection strings. Replace the below <storage-account-connection-string> and <blob-processing-storage-account-connection-string> with the two storage account connection string values created in step 4.
data:
AzureWebJobsStorage: {{ <storage-account-connection-string> | b64enc }}
TEST_STORAGE_CONNECTION_STRING: {{ <blob-processing-storage-account-connection-string> | b64enc }}
Now deploy using Helm chart.
cd deploy
helm install blob-consumer --namespace default blob-consumer
10. Add a blob and validate the function app scales with KEDA
Initially after deploy should see 0 pods as the blob container is empty.
kubectl get deploy
Upload a file to the blob container (using the Storage Explorer shown in step 6 above). KEDA will detect the event and add a pod.
kubectl get pods -w
The blob file will be processed( this sample write the blob name and size into the log to simulate performing work then delete the blob from the container). You can validate the file was consumed by using kubectl logs on the activated pod. After all files are processed and the cooldown period has elapsed, the last pod should scale back down to zero.
Cleaning up resources
Delete the function deployment
helm del blob-consumer -n default
Delete two storage accounts
az storage account delete --name <storage-name1>
az storage account delete --name <storage-name2>
Uninstall KEDA
func kubernetes remove --namespace keda