Home

Awesome

services: service-fabric, container-service, kubernetes, cosmos-db, service-bus, application-insights, container-registry, dns, aks, key-vault platforms: docker, dotnet-core, aspnet-core author: paolosalvatori

Multi-Container Sample with Service Fabric and Kubernetes

This sample demonstrates how create a multi-container application using ASP.NET Core and Docker and deploy it on an Azure Service Fabric Linux cluster with the DNS Service or on an Azure Container Service Kubernetes cluster on Linux. The multi-container application adopts a microservices architecture. A microservices architecture consists of a collection of small, autonomous services. Each service is self-contained and should implement a single business capability. Microservices have the following characteristics:

For detailed guidance about building a microservices architecture on Azure, see Designing, building, and operating microservices on Azure.

This solution is covered in detail in the following e-books available on Microsoft Azure site:

Azure Service Fabric and ACS/Kubernetes Environments

This repository contains a sample multi-container application and the scripts to deploy it on the following environments:

Prerequisites for development machine

  1. Install Microsoft Visual Studio 2017 with .NET Core workload. For more information, see Visual Studio Tools for Docker.
  2. Install Docker for Windows and configure it to use Linux containers.
  3. Set up the developer environment
  4. Create a Service Fabric Linux cluster in Azure with a minimum of five nodes and the DNS Service enabled. The demo requires a cluster running on Azure. For the purposes of this sample, you can eventually use the party cluster.
  5. Clone or download this container solution into a directory on the local machine.
  6. Specify the value of the application parameters (see the Configuration section for more information).

Visual Studio Solution

The solution has the following solution folders:

Note: both the frontend (todoweb) and backend (todoapi) containerized services use the microsoft/aspnetcore:2.0 as base Docker image. For more information, see Official .NET Docker images

Service Fabric Architecture

The following picture shows the architecture of the Service Fabric application.

Architecture

Kubernetes Architecture

The following picture shows the architecture of the Kubernetes application running on Azure Container Service.

Architecture

Note: this diagram represents a simplification of the actual implementation of Kubernetes Services as it omits important entities such kube-proxy, Iptables, Service Types, etc. For more information, see Kubernetes Services.

Configuration

In ASP.NET Core, the configuration API provides a way of configuring an app based on a list of name-value pairs. Configuration is read at runtime from multiple sources. The name-value pairs can be grouped into a multi-level hierarchy. There are configuration providers for:

TodoApi Service Configuration

The following table contains the configuration of the TodoApi service defined in the appsettings.json file.

{
  "AzureKeyVault": {
  "Certificate": {
	"CertificateEnvironmentVariable": "",
	"KeyEnvironmentVariable": ""
  },
  "ClientId": "",
  "Name": ""
  },
  "RepositoryService": {
	  "CosmosDb": {
		  "EndpointUri": "",
		  "PrimaryKey": "",
		  "DatabaseName": "",
		  "CollectionName": ""
	  }
  },
  "NotificationService": {
	  "ServiceBus": {
		  "ConnectionString": "",
		  "QueueName": ""
	  }
  },
  "DataProtection": {
	"BlobStorage": {
	  "ConnectionString": "",
	  "ContainerName": ""
	}
  },
  "ApplicationInsights": {
	  "InstrumentationKey": ""
  },
  "Logging": {
	  "IncludeScopes": false,
	  "Debug": {
		  "LogLevel": {
			  "Default": "Information"
		  }
	  },
	  "Console": {
		  "LogLevel": {
			  "Default": "Information"
		  }
	  },
	  "EventSource": {
		  "LogLevel": {
			  "Default": "Warning"
		  }
	  },
	  "ApplicationInsights": {
		  "LogLevel": {
			  "Default": "Information"
		  }
	  }
  }
}

Notes:

TodoWeb Service Configuration

The following table contains the configuration of the TodoWeb service defined in the appsettings.json file.

{
  "AzureKeyVault": {
	"Certificate": {
	  "CertificateEnvironmentVariable": "",
	  "KeyEnvironmentVariable": ""
	},
	"ClientId": "",
	"Name": ""
  },
  "TodoApiService": {
	"EndpointUri": ""
  },
  "DataProtection": {
	"BlobStorage": {
	  "ConnectionString": "",
	  "ContainerName": ""
	}
  },
  "Logging": {
	"IncludeScopes": false,
	"LogLevel": {
	  "Default": "Warning"
	},
	"ApplicationInsights": {
	  "LogLevel": {
		"Default": "Information"
	  }
	}
  },
  "ApplicationInsights": {
	"InstrumentationKey": ""
  }
}

Notes:

How Configuration works in ASP.NET Core

The CreateDefaultBuilder extension method in an ASP.NET Core 2.x app adds configuration providers for reading JSON files and system configuration sources:

ASP.NET Core allows to use additional configuration providers to read settings from a heterogeneous range of repositories. Later in this document, we'll see how to:

Configuration consists of a hierarchical list of name-value pairs in which the nodes are separated by a colon. To retrieve a value, access the Configuration indexer with the corresponding item's key. For example, if you want to retrieve the value of the QueueName setting from the configuration of the todoapi service, you have to use the following format.

var queueName = Configuration["NotificationService:ServiceBus:QueueName"];

If you want to create an environment variable to provide a value for a setting defined in the appsettings.json file, you can replace : (colon) with __ (double underscore).

NotificationService__ServiceBus__QueueName=todoapi

The CreateDefaultBuilder helper method specifies environment variables last, so that the local environment can override anything set in deployed configuration files. This allows allows to define settings in the appsettings.json file, but leave their value empty, and specify their value using environment variables.

For more information on configuration, see the following resources:

How to define the Docker images and containers

Using Visual Studio Tools for Docker, I built an image based the on the microsoft/aspnetcore:2.0 standard image. Then, the tool creates a Dockerfile for both the frontend and backend service that you can later customize at will. The Dockerfile contains instructions for setting up the environment inside your container, loading the application you want to run, and mapping ports. The Dockerfile is the input to the docker build command, which creates the image. Below you can see the Dockerfile of the todoapi and todoweb services.

TodoApi Dockerfile

FROM microsoft/aspnetcore:2.0
ARG source
WORKDIR /app
EXPOSE 80
COPY ${source:-obj/Docker/publish} .
ENTRYPOINT ["dotnet", "TodoApi.dll"]

TodoWeb Dockerfile

FROM microsoft/aspnetcore:2.0
ARG source
WORKDIR /app
EXPOSE 80
COPY ${source:-obj/Docker/publish} .
ENTRYPOINT ["dotnet", "TodoWeb.dll"]

The Visual Studio Tools for Docker also creates the docker-compose.yml and docker-compose-override.yml files that you can use to test the application locally.

docker-compose.yml

version: '3'

services:
  todoapi:
	image: todoapi
	build:
	  context: ./TodoApi
	  dockerfile: Dockerfile

  todoweb:
	image: todoweb
	build:
	  context: ./TodoWeb
	  dockerfile: Dockerfile

docker-compose-override.yml

version: '3'

services:
  todoapi:
	environment:
	  - ASPNETCORE_ENVIRONMENT=Development
	  - RepositoryService__CosmosDb__EndpointUri=COSMOS_DB_ENDPOINT_URI
	  - RepositoryService__CosmosDb__PrimaryKey=COSMOS_DB_PRIMARY_KEY
	  - RepositoryService__CosmosDb__DatabaseName=TodoApiDb
	  - RepositoryService__CosmosDb__CollectionName=TodoApiCollection
	  - NotificationService__ServiceBus__ConnectionString=SERVICE_BUS_CONNECTION_STRING
	  - NotificationService__ServiceBus__QueueName=todoapi
	  - DataProtection__BlobStorage__ConnectionString=STORAGE_ACCOUNT_CONNECTION_STRING
	  - DataProtection__BlobStorage__ContainerName=todoapi
	  - ApplicationInsights__InstrumentationKey=APPLICATION_INSIGHTS_INSTRUMENTATION_KEY

	ports:
	  - "80"

  todoweb:
	environment:
	  - ASPNETCORE_ENVIRONMENT=Development
	  - TodoApiService__EndpointUri=todoapi
	  - DataProtection__BlobStorage__ConnectionString=STORAGE_ACCOUNT_CONNECTION_STRING
	  - DataProtection__BlobStorage__ContainerName=todoweb
	  - ApplicationInsights__InstrumentationKey=APPLICATION_INSIGHTS_INSTRUMENTATION_KEY
	ports:
	  - "80"

Configuration

Before debugging the application in Visual Studio, make the following changes to the docker-compose-override.yml file:

When using Azure Key Vault to store secrets, and Linux Containers to run the front-end and back-end services, you should use the following docker compose file to debug the application locally in Visual Studio. When deploying and running the application on AKS, where sensitive data is stored as secrets in the Kubernetes cluster, you can ignore this part.

docker-compose-override.yml

version: '3'

services:
  todoapi:
	environment:
	  - ASPNETCORE_ENVIRONMENT=Development
	  - NotificationService__ServiceBus__QueueName=todoapi
	  - DataProtection__BlobStorage__ContainerName=todoapi
	  - AzureKeyVault__Certificate__CertificateEnvironmentVariable=Certificates_TodoApiPkg_Code_TodoListCert_PEM
	  - AzureKeyVault__Certificate__KeyEnvironmentVariable=Certificates_TodoApiPkg_Code_TodoListCert_PrivateKey
	  - AzureKeyVault__ClientId=AZURE_AD_APPLICATION_ID
	  - AzureKeyVault__Name=AZURE_KEY_VAULT_NAME
	  - Certificates_TodoApiPkg_Code_TodoListCert_PEM=/pem/KeyVaultCertificate.pem
	  - Certificates_TodoApiPkg_Code_TodoListCert_PrivateKey=/pem/KeyVaultCertificate.key
	ports:
	  - "80"
	volumes:
	  - C:\Temp\Pem:/pem

  todoweb:
	environment:
	  - ASPNETCORE_ENVIRONMENT=Development
	  - TodoApiService__EndpointUri=todoapi
	  - DataProtection__BlobStorage__ContainerName=todoweb
	  - AzureKeyVault__Certificate__CertificateEnvironmentVariable=Certificates_TodoApiPkg_Code_TodoListCert_PEM
	  - AzureKeyVault__Certificate__KeyEnvironmentVariable=Certificates_TodoApiPkg_Code_TodoListCert_PrivateKey
	  - AzureKeyVault__ClientId=AZURE_AD_APPLICATION_ID
	  - AzureKeyVault__Name=AZURE_KEY_VAULT_NAME
	  - Certificates_TodoApiPkg_Code_TodoListCert_PEM=/pem/KeyVaultCertificate.pem
	  - Certificates_TodoApiPkg_Code_TodoListCert_PrivateKey=/pem/KeyVaultCertificate.key
	ports:
	  - "80"
	volumes:
	  - C:\Temp\Pem:/pem

Configuration

Before debugging the solution in Visual Studio, make sure to perform the following tasks:

Push Docker images to Docker Hub

You can execute the following command file to tag and register the images in your repository on Docker Hub. Make sure to replace the placeholder DOCKER_HUB_REPOSITORY with the name of your repository on Docker Hub before running the batch file.

push-images-to-docker-hub.cmd

REM login to docker hub
docker login -u DOCKER_HUB_REPOSITORY -p DOCKER_HUB_PASSWORD

REM tag the local todoapi:latest image with the name of the DOCKER_HUB_REPOSITORY
docker tag todoapi:latest DOCKER_HUB_REPOSITORY/todoapi:v1

REM push the image DOCKER_HUB_REPOSITORY/todoapi:latest to the DOCKER_HUB_REPOSITORY
docker push DOCKER_HUB_REPOSITORY/todoapi:v1

REM tag the local todoweb:latest image with the name of the DOCKER_HUB_REPOSITORY
docker tag todoweb:latest DOCKER_HUB_REPOSITORY/todoweb:v1

REM push the image DOCKER_HUB_REPOSITORY/todoweb:latest to the DOCKER_HUB_REPOSITORY 
docker push DOCKER_HUB_REPOSITORY/todoweb:v1

REM browse to https://hub.docker.com/r/DOCKER_HUB_REPOSITORY/
start chrome https://hub.docker.com/r/DOCKER_HUB_REPOSITORY/

Configuration

Before running the above script file, make the following changes:

Alternatively, you can register and deploy your images from the Azure Container Registry. Let's see how you can perform this task.

Create an Azure Container Registry

The Azure Container Registry is a managed Docker registry service based on the open-source Docker Registry 2.0. This repository can be used to store images for container deployments Aaure Container Service, Azure App Service, Azure Batch, Azure Service Fabric, and others. For more information, see Introduction to private Docker container registries in Azure. You can run the following script to create an Azure Container Registry. Make sure to replace the placeholder AZURE_CONTAINER_REGISTRY with the name of your Azure Container Registry before running the batch file.

create-azure-container-registry.cmd

REM Create a resource group for the Azure Container Registry
az group create --name ContainerRegistryResourceGroup --location westus2 --output jsonc

REM Create an Azure Container Registry. The name of the Container Registry must be unique
az acr create --resource-group ContainerRegistryResourceGroup --name AZURE_CONTAINER_REGISTRY --sku Basic --admin-enabled true

REM Login to the newly created Azure Container Registry
az acr login --name AZURE_CONTAINER_REGISTRY

Push Docker images to Azure Container Registry

You can execute the following command file to tag and register the images in your repository on Docker Hub. Make sure to replace the placeholder AZURE_CONTAINER_REGISTRY with the name of your Azure Container Registry before running the batch file.

push-images-to-azure-container-registry.cmd

REM Login to the newly created Azure Container Registry
call az acr login --name AZURE_CONTAINER_REGISTRY

REM Each container image needs to be tagged with the loginServer name of the registry. 
REM This tag is used for routing when pushing container images to an image registry.
REM Save the loginServer name to the AKS_CONTAINER_REGISTRY environment variable
for /f "delims=" %%a in ('call az acr list --resource-group ContainerRegistryResourceGroup --query "[].{acrLoginServer:loginServer}" --output tsv') do @set AKS_CONTAINER_REGISTRY=%%a

REM tag the local todoapi:latest image with the loginServer of the container registry
docker tag todoapi:latest %AKS_CONTAINER_REGISTRY%/todoapi:v1

REM publish <container registry>/todoapi:latest to the container registry on Azure
docker push %AKS_CONTAINER_REGISTRY%/todoapi:v1

REM tag the local todoweb:latest image with the loginServer of the container registry
docker tag todoweb:latest %AKS_CONTAINER_REGISTRY%/todoweb:v1

REM publish <container registry>/todoweb:latest to the container registry on Azure
docker push %AKS_CONTAINER_REGISTRY%/todoweb:v1

REM List images in the container registry on Azure
call az acr repository list --name AZURE_CONTAINER_REGISTRY --output table

Storing secret parameters in Azure Key Vault

If you plan to deploy the application to a Service Fabric Windows or Linux cluster in Azure, you should store sensitive data like connection strings, password, or instrumentation keys in Azure Key Vault. The frontend and backend services that compose the multi-container application in this sample are ASP.NET Core projects. ASP.NET Core supplies a configuration provider for Azure Key Vault in the Microsoft.Extensions.Configuration.AzureKeyVault NuGet package. This configuration provider allows an application to use the Application Id and Application Key of an Azure Active Directory Application to authenticate against Azure Key Vault as explained at Azure Key Vault configuration provider. However, the approach explained in the documentation requires to define the ClientId and ClientSecret in the service configuration, but this technique is not safe, because a malicious user could use and use these credentials to access secrets in Azure Key Vault. Another way to authenticate an Azure AD application is by using a Client ID and a Certificate instead of a Client ID and Client Secret. Following are the steps to use a Certificate in an Azure Web App:

Get or Create a Certificate

For our purposes, we will make a test certificate. Here are a couple of commands that you can use in a command prompt to create a certificate. Change directory to where you want the cert files created. Also, for the beginning and ending date of the certificate, use the current date plus 1 year.

makecert -sv KeyVaultCertificate.pvk -n "cn=TodoListApp" KeyVaultCertificate.cer -b 01/01/2018 -e 12/31/2030 -r
pvk2pfx -pvk KeyVaultCertificate.pvk -spc KeyVaultCertificate.cer -pfx KeyVaultCertificate.pfx -po trustno1

Make note of the password for the .pfx (in this example: trustno1). You will need it below. If you are planning to deploy the sample application to an Azure Service Fabric Windows cluster, the .pfx and .cer files created will suffice your needs. Instead, if you plan to deploy the application to an Azure Service Fabric Linux cluster, and you want to debug the frontend and backend service locally, you will need to create a .pem file and a .key file starting from the .pfx file using the following commands:

openssl pkcs12 -in KeyVaultCertificate.pfx -out KeyVaultCertificatePEM.pem -nodes -nokeys
openssl pkcs12 -in KeyVaultCertificate.pfx -out KeyVaultCertificatePEM.key -nodes -nocerts

Create a Key Vault using Azure CLI

In order to protect sensitive data from unauthorized users, you should store secrets in Key Vault. The following script can be used to create an Azure Key Vault:

CreateKeyVault.cmd

REM Create a Resource Group for Key Vault
call az group create --name TodoListKeyVaultResourceGroup --location WestEurope

REM Create Key Vault
call az keyvault create --name TodoListKeyVault --resource-group TodoListKeyVaultResourceGroup

Add secrets to Azure Key Vault using Azure CLI

To add sensitive configuration data to Azure Key Vault, you can use the following script:

AddSecretsToKeyVault.cmd

REM add Cosmos DB Endpoint URI secret to Key Vault
call az keyvault secret set --name RepositoryService--CosmosDb--EndpointUri --vault-name TodoListKeyVault  --value "COSMOS_DB_ENDPOINT_URI" --description "Cosmos DB endpoint URI"

REM add Cosmos DB Primary Key secret to Key Vault
call az keyvault secret set --name RepositoryService--CosmosDb--PrimaryKey --vault-name TodoListKeyVault  --value "COSMOS_DB_PRIMARY_KEY --description "Cosmos DB primary key"

REM add Cosmos DB Database Name secret to Key Vault
call az keyvault secret set --name RepositoryService--CosmosDb--DatabaseName --vault-name TodoListKeyVault  --value "COSMOS_DB_DATABASE_NAME" --description "Cosmos DB database name"

REM add Cosmos DB Collection Name secret to Key Vault
call az keyvault secret set --name RepositoryService--CosmosDb--CollectionName --vault-name TodoListKeyVault  --value "COSMOS_DB_COLLECTION_NAME" --description "Cosmos DB collection name"

REM add Service Bus Connection String secret to Key Vault
call az keyvault secret set --name NotificationService--ServiceBus--ConnectionString --vault-name TodoListKeyVault  --value "SERVICE_BUS_CONNECTION_STRING" --description "Service Bus connection string"

REM add Data Protection Blob Storage Connection String secret to Key Vault
call az keyvault secret set --name DataProtection--BlobStorage--ConnectionString --vault-name TodoListKeyVault  --value "STORAGE_ACCOUNT_CONNECTION_STRING" --description "Data Protection blob storage connection string"

REM add Application Insights Instrumentation Key secret to Key Vault
call az keyvault secret set --name ApplicationInsights--InstrumentationKey --vault-name TodoListKeyVault  --value "APPLICATION_INSIGHTS_INSTRUMENTATION_KEY" --description "Application Insights instrumentation key"

REM List secrets in Key Vault
call az keyvault secret list --vault-name TodoListKeyVault --output table

Configuration

Before running the above script, make the following changes:

Associate the certificate with an Azure AD application

The next step is to associate the certificate with an Azure AD Application. Presently, the Azure portal does not support this workflow; this can be completed through PowerShell. Run the following commands to associate the certificate with a new Azure AD application called ServiceFabricTodoListApp:

CreateAADApplication.ps1

# Login to Azure Resource Manager
Login-AzureRmAccount

# Select a default subscription for your current session in case your account has multiple Azure subscriptions
Get-AzureRmSubscription –SubscriptionName "SUBSCRIPTION-NAME" | Select-AzureRmSubscription

# Variables
$pfxFile = $PSScriptRoot + '\KeyVaultCertificate.cer'
$displayName = "ServiceFabricTodoListApp"
$appUrl = "http://ServiceFabricTodoListApp"a
$keyVaultName = "TodoListKeyVault"
$keyVaultResourceGroup = "TodoListKeyVaultResourceGroup"

# Get certificate from file
$x509 = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2
$x509.Import($pfxFile)

# Get certificate raw data in base64 format
$credValue = [System.Convert]::ToBase64String($x509.GetRawCertData())

# Create a new Azure AD application for TodoListApp
$adapp = New-AzureRmADApplication -DisplayName $displayName `
								  -HomePage $appUrl `
								  -IdentifierUris $appUrl `
								  -CertValue $credValue `
								  -StartDate $x509.NotBefore `
								  -EndDate $x509.NotAfter

# Create a new Azure AD service proncipal for the Azure AD TodoListApp application
$sp = New-AzureRmADServicePrincipal -ApplicationId $adapp.ApplicationId

# Grants permissions for a user, application, or security group to perform operations with a key vault.
Set-AzureRmKeyVaultAccessPolicy -VaultName $keyVaultName `
								-ResourceGroupName $keyVaultResourceGroup `
								-ServicePrincipalName $adapp.ApplicationId `
								-PermissionsToSecrets all

# get the thumbprint to use in your app settings
$x509.Thumbprint

# get the application id to use in your app settings
$adapp.ApplicationId

After you have run these commands, you can see the application in Azure AD. Make sure to take note of the certificate thumbprint and ApplicationId printed out by the script. To learn more about Azure AD Application Objects and ServicePrincipal Objects, see Application Objects and Service Principal Objects.

How Service Fabric passes a certificate to a container

Service Fabric provides a mechanism for services running inside a container to access a certificate that is installed on the nodes in a Windows or Linux cluster. You can secure your container services by specifying a certificate. The certificate information is provided in the application manifest under the ContainerHostPolicies tag as the following snippet shows:

<ContainerHostPolicies CodePackageRef="NodeContainerService.Code">
	<CertificateRef Name="MyCert1" X509StoreName="My" X509FindValue="[Thumbprint1]"/>
	<CertificateRef Name="MyCert2" X509FindValue="[Thumbprint2]"/>

For windows clusters, when starting the application, the runtime reads the certificates and generates a .pfx file and password for each certificate. This .pfx file and password file are accessible inside the container using the following environment variables:

For Linux clusters, the certificates as .pem files are simply copied over from the store specified by X509StoreName onto the container. The corresponding environment variables on Linux are:

Alternatively, if you already have the certificates in the required form and would simply want to access it inside the container, you can create a data package inside your app package and specify the following inside your application manifest:

<ContainerHostPolicies CodePackageRef="NodeContainerService.Code">
   <CertificateRef Name="MyCert1" DataPackageRef="[DataPackageName]" DataPackageVersion="[Version]" RelativePath="[Relative Path to certificate inside DataPackage]" Password="[password]" IsPasswordEncrypted="[true/false]"/>

For more information on how to configure certificates for a containerized service in Service Fabric, see Container Security. For more information on manage certificates used by a Service Fabric cluster in Azure, see Add or remove certificates for a Service Fabric cluster in Azure.

How to read the certificate from code and initialize Key Vault configuration provider

When deploying the application to a Service Fabric Linux cluster in Azure, you need to specify a certificate in the CertificateRef inside the ContainerHostPolicies of both the frontend and backend service, using one of the techniques described in the previous section. Service Fabric will copy the certificate files inside the container and will create two environment variables that will contain the path of:

Below you can see the code used by the Program class of the TodoApi service.

Program.cs

using System;
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;

namespace TodoApi
{
	public class Program
	{
		public static void Main(string[] args)
		{
			BuildWebHost(args).Run();
		}

		public static IWebHost BuildWebHost(string[] args)
		{
			var builder = WebHost.CreateDefaultBuilder(args)
				.UseApplicationInsights()
				.CaptureStartupErrors(true)
				.UseSetting(WebHostDefaults.DetailedErrorsKey, "true")
				.ConfigureAppConfiguration(ConfigConfiguration)
				.UseStartup<Startup>()
				.Build();

			return builder;
		}

		private static void ConfigConfiguration(WebHostBuilderContext webHostBuilderContext, IConfigurationBuilder configurationBuilder)
		{
			var configuration = configurationBuilder.Build();

			// Read the name of the environment variable set by Service Fabric that contain the location of the PEM file
			var certificateEnvironmentVariable = configuration["AzureKeyVault:Certificate:CertificateEnvironmentVariable"];
			if (string.IsNullOrWhiteSpace(certificateEnvironmentVariable))
			{
				return;
			}

			// Read the name of the environment variable set by Service Fabric that contain the location of the KEY file
			var keyEnvironmentVariable = configuration["AzureKeyVault:Certificate:KeyEnvironmentVariable"];
			if (string.IsNullOrWhiteSpace(keyEnvironmentVariable))
			{
				return;
			}

			// Read the client ID
			var clientId = configuration["AzureKeyVault:ClientId"];
			if (string.IsNullOrWhiteSpace(clientId))
			{
				return;
			}

			// Read the name of the Azure Key Vault
			var name = configuration["AzureKeyVault:Name"];
			if (string.IsNullOrWhiteSpace(name))
			{
				return;
			}

			// Read the location of the certificate file from the environment variable
			var certificateFilePath = Environment.GetEnvironmentVariable(certificateEnvironmentVariable);
			if (string.IsNullOrWhiteSpace(certificateFilePath))
			{
				return;
			}

			// Read the location of the key file from the environment variable
			var keyFilePath = Environment.GetEnvironmentVariable(keyEnvironmentVariable);
			if (string.IsNullOrWhiteSpace(keyFilePath))
			{
				return;
			}

			// Read the certificate used to authenticate against Azure Key Vault
			var certificate = Helpers.CertificateHelper.GetCertificateAsync(certificateFilePath, keyFilePath).Result;
			if (certificate == null)
			{
				return;
			}

			// Configure the application to read settings from Azure Key Vault
			configurationBuilder.AddAzureKeyVault($"https://{name}.vault.azure.net/",
												  clientId,
												  certificate);
		}
	}
}

Remarks

The ConfigureAppConfiguration method performs the following actiosn:

Below you can see the code of the CertificateHelper.GetCertificateAsync method:

public static async Task<X509Certificate2> GetCertificateAsync(string certificateFilePath, string keyFilePath)
{
	// Validate parameters
	if (string.IsNullOrEmpty(certificateFilePath))
	{
		throw new ArgumentException($"{nameof(certificateFilePath)} parameter cannot bu null or empty.", nameof(certificateFilePath));
	}

	if (string.IsNullOrEmpty(keyFilePath))
	{
		throw new ArgumentException($"{nameof(keyFilePath)} parameter cannot bu null or empty.", nameof(keyFilePath));
	}

	if (!File.Exists(certificateFilePath))
	{
		throw new FileNotFoundException($"{certificateFilePath} file not found.", certificateFilePath);
	}

	if (!File.Exists(keyFilePath))
	{
		throw new FileNotFoundException($"{keyFilePath} file not found.", keyFilePath);
	}

	if (Environment.OSVersion.Platform.ToString().ToLower().Contains("win"))
	{
		SetReadPermission(certificateFilePath);
		SetReadPermission(keyFilePath);
		var password = File.ReadAllLines(keyFilePath, Encoding.Default)[0];
		password = password.Replace("\0", string.Empty);
		var certificate = new X509Certificate2(certificateFilePath, password);
		return certificate;
	}
	else
	{
		var pemCertificate = await File.ReadAllTextAsync(certificateFilePath);
		var pemKey = await File.ReadAllTextAsync(keyFilePath);

		var certBuffer = GetBytesFromPem(pemCertificate, CertificateFileType.Certificate);
		var keyBuffer = GetBytesFromPem(pemKey, CertificateFileType.Pkcs8PrivateKey);

		var certificate = new X509Certificate2(certBuffer);
		var privateKey = DecodePrivateKeyInfo(keyBuffer);
		certificate = certificate.Copy​With​Private​Key(privateKey);
		return certificate;
	}
}

Note: the application uses the classes contained in the System.Security.Cryptography.OpenSsl NuGet package to read certificates in a Linux cluster.

Service Fabric Deployment with Application Manifest and Service Manifests

In the Visual Studio solution you can find three projects to deploy the multi-container application to an Azure Service Fabric Cluster:

Note: We recommend that you encrypt the repository password by using an encipherment certificate that's deployed to all nodes of the cluster. When Service Fabric deploys the service package to the cluster, the encipherment certificate is used to decrypt the cipher text. The Invoke-ServiceFabricEncryptText cmdlet is used to create the cipher text for the password, which is added to the ApplicationManifest.xml file.

Deploy the application to a Linux cluster: parameters defined in the application manifest

As mentioned above, the TodoAppForLinuxFromACR project allows you to specify secret parameters in clear-text in the Cloud.xml file only for testing purposes without the need to store them in Azure Key Vault. This technique should not be used to deploy an application to a production environment. Unathorized users could read sensitive data, like connection strings, keys and passwords, from the manifests stored in the source code repository (e.g. GitHub) or retrieve this data from the Service Fabric Explorer dashboard, once the application is deployed. This section shows the service manifests, application manifest and application parameters file contained in this project. Looking at the service manifests below, you can observe that all configuration data is passed to each service using environment variables.

TodoApi ServiceManifest.xml

<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="TodoApiPkg"
				 Version="1.0.0"
				 xmlns="http://schemas.microsoft.com/2011/01/fabric"
				 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
				 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ServiceTypes>
	<!-- This is the name of your ServiceType.
		 The UseImplicitHost attribute indicates this is a guest service. -->
	<StatelessServiceType ServiceTypeName="TodoApiType" UseImplicitHost="true" />
  </ServiceTypes>

  <!-- Code package is your service executable. -->
  <CodePackage Name="Code" Version="1.0.0">
	<EntryPoint>
	  <!-- Follow this link for more information about deploying Windows containers to Service Fabric: https://aka.ms/sfguestcontainers -->
	  <ContainerHost>
		<ImageName>AZURE_CONTAINER_REGISTRY_NAME.azurecr.io/todoapi:v1</ImageName>
	  </ContainerHost>
	</EntryPoint>
	<!-- Pass environment variables to your container: -->
	<EnvironmentVariables>
	  <EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value=""/>
	  <EnvironmentVariable Name="RepositoryService__CosmosDb__EndpointUri" Value=""/>
	  <EnvironmentVariable Name="RepositoryService__CosmosDb__PrimaryKey" Value=""/>
	  <EnvironmentVariable Name="RepositoryService__CosmosDb__DatabaseName" Value=""/>
	  <EnvironmentVariable Name="RepositoryService__CosmosDb__CollectionName" Value=""/>
	  <EnvironmentVariable Name="NotificationService__ServiceBus__ConnectionString" Value=""/>
	  <EnvironmentVariable Name="NotificationService__ServiceBus__QueueName" Value=""/>
	  <EnvironmentVariable Name="DataProtection__BlobStorage__ConnectionString" Value=""/>
	  <EnvironmentVariable Name="DataProtection__BlobStorage__ContainerName" Value=""/>
	  <EnvironmentVariable Name="ApplicationInsights__InstrumentationKey" Value=""/>
	</EnvironmentVariables>
  </CodePackage>

  <!-- Config package is the contents of the Config directoy under PackageRoot that contains an 
	   independently-updateable and versioned set of custom configuration settings for your service. -->
  <ConfigPackage Name="Config" Version="1.0.0" />

  <Resources>
	<Endpoints>
	  <!-- This endpoint is used by the communication listener to obtain the port on which to 
		   listen. Please note that if your service is partitioned, this port is shared with 
		   replicas of different partitions that are placed in your code. -->
	  <Endpoint Name="TodoApiEndpoint" Port="80" UriScheme="http" Protocol="http"/>
	</Endpoints>
  </Resources>
</ServiceManifest>

TodoWeb ServiceManifest.xml

<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="TodoWebPkg"
				 Version="1.0.0"
				 xmlns="http://schemas.microsoft.com/2011/01/fabric"
				 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
				 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ServiceTypes>
	<!-- This is the name of your ServiceType.
		 The UseImplicitHost attribute indicates this is a guest service. -->
	<StatelessServiceType ServiceTypeName="TodoWebType" UseImplicitHost="true" />
  </ServiceTypes>

  <!-- Code package is your service executable. -->
  <CodePackage Name="Code" Version="1.0.0">
	<EntryPoint>
	  <!-- Follow this link for more information about deploying Windows containers to Service Fabric: https://aka.ms/sfguestcontainers -->
	  <ContainerHost>
		<ImageName>AZURE_CONTAINER_REGISTRY_NAME.azurecr.io/todoweb:v1</ImageName>
	  </ContainerHost>
	</EntryPoint>
	<!-- Pass environment variables to your container: -->
	<EnvironmentVariables>
	  <EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value=""/>
	  <EnvironmentVariable Name="TodoApiService__EndpointUri" Value=""/>
	  <EnvironmentVariable Name="DataProtection__BlobStorage__ConnectionString" Value=""/>
	  <EnvironmentVariable Name="DataProtection__BlobStorage__ContainerName" Value=""/>
	  <EnvironmentVariable Name="ApplicationInsights__InstrumentationKey" Value=""/>
	</EnvironmentVariables>
  </CodePackage>

  <!-- Config package is the contents of the Config directoy under PackageRoot that contains an 
	   independently-updateable and versioned set of custom configuration settings for your service. -->
  <ConfigPackage Name="Config" Version="1.0.0" />

  <Resources>
	<Endpoints>
	  <!-- This endpoint is used by the communication listener to obtain the port on which to 
		   listen. Please note that if your service is partitioned, this port is shared with 
		   replicas of different partitions that are placed in your code. -->
	  <Endpoint Name="TodoWebEndpoint" Port="8080" UriScheme="http" Protocol="http"/>
	</Endpoints>
  </Resources>
</ServiceManifest>

Looking at the application manifest, you can observe that the certificate used by each service needs to be installed in the Azure Service Fabric Linux cluster before deploying the application. In fact, the certificate identified by its thumbprint needs to exist in the cluster nodes before Docker containers are created for each service.

ApplicationManifest.xml

<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest ApplicationTypeName="TodoAppType"
					 ApplicationTypeVersion="1.0.0"
					 xmlns="http://schemas.microsoft.com/2011/01/fabric"
					 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
					 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Parameters>
	<Parameter Name="ACR_Username" DefaultValue="" />
	<Parameter Name="ACR_Password" DefaultValue="" />
	<Parameter Name="TodoWeb_InstanceCount" DefaultValue="-1" />
	<Parameter Name="TodoWeb_ASPNETCORE_ENVIRONMENT" DefaultValue="Development"/>
	<Parameter Name="TodoWeb_TodoApiService__EndpointUri" DefaultValue=""/>
	<Parameter Name="TodoWeb_DataProtection__BlobStorage__ConnectionString" DefaultValue=""/>
	<Parameter Name="TodoWeb_DataProtection__BlobStorage__ContainerName" DefaultValue=""/>
	<Parameter Name="TodoWeb_ApplicationInsights__InstrumentationKey" DefaultValue=""/>
	<Parameter Name="TodoApi_InstanceCount" DefaultValue="-1" />
	<Parameter Name="TodoApi_ASPNETCORE_ENVIRONMENT" DefaultValue="Development"/>
	<Parameter Name="TodoApi_RepositoryService__CosmosDb__EndpointUri" DefaultValue=""/>
	<Parameter Name="TodoApi_RepositoryService__CosmosDb__PrimaryKey" DefaultValue=""/>
	<Parameter Name="TodoApi_RepositoryService__CosmosDb__DatabaseName" DefaultValue=""/>
	<Parameter Name="TodoApi_RepositoryService__CosmosDb__CollectionName" DefaultValue=""/>
	<Parameter Name="TodoApi_NotificationService__ServiceBus__ConnectionString" DefaultValue=""/>
	<Parameter Name="TodoApi_NotificationService__ServiceBus__QueueName" DefaultValue=""/>
	<Parameter Name="TodoApi_DataProtection__BlobStorage__ConnectionString" DefaultValue=""/>
	<Parameter Name="TodoApi_DataProtection__BlobStorage__ContainerName" DefaultValue=""/>
	<Parameter Name="TodoApi_ApplicationInsights__InstrumentationKey" DefaultValue=""/>
  </Parameters>
  <!-- Import the ServiceManifest from the ServicePackage. The ServiceManifestName and ServiceManifestVersion 
	   should match the Name and Version attributes of the ServiceManifest element defined in the 
	   ServiceManifest.xml file. -->
  <ServiceManifestImport>
	<ServiceManifestRef ServiceManifestName="TodoWebPkg" ServiceManifestVersion="1.0.0" />
	<ConfigOverrides />
	<EnvironmentOverrides CodePackageRef="Code">
	  <EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value="[TodoWeb_ASPNETCORE_ENVIRONMENT]"/>
	  <EnvironmentVariable Name="TodoApiService__EndpointUri" Value="[TodoWeb_TodoApiService__EndpointUri]"/>
	  <EnvironmentVariable Name="DataProtection__BlobStorage__ConnectionString" Value="[TodoWeb_DataProtection__BlobStorage__ConnectionString]"/>
	  <EnvironmentVariable Name="DataProtection__BlobStorage__ContainerName" Value="[TodoWeb_DataProtection__BlobStorage__ContainerName]"/>
	  <EnvironmentVariable Name="ApplicationInsights__InstrumentationKey" Value="[TodoWeb_ApplicationInsights__InstrumentationKey]"/>
	</EnvironmentOverrides>
	<Policies>
	  <ContainerHostPolicies CodePackageRef="Code">
		<RepositoryCredentials AccountName="[ACR_Username]" Password="[ACR_Password]" PasswordEncrypted="false"/>
		<PortBinding ContainerPort="80" EndpointRef="TodoWebEndpoint" />
	  </ContainerHostPolicies>
	</Policies>
  </ServiceManifestImport>
  <ServiceManifestImport>
	<ServiceManifestRef ServiceManifestName="TodoApiPkg" ServiceManifestVersion="1.0.0" />
	<ConfigOverrides />
	<EnvironmentOverrides CodePackageRef="Code">
	  <EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value="[TodoApi_ASPNETCORE_ENVIRONMENT]"/>
	  <EnvironmentVariable Name="RepositoryService__CosmosDb__EndpointUri" Value="[TodoApi_RepositoryService__CosmosDb__EndpointUri]"/>
	  <EnvironmentVariable Name="RepositoryService__CosmosDb__PrimaryKey" Value="[TodoApi_RepositoryService__CosmosDb__PrimaryKey]"/>
	  <EnvironmentVariable Name="RepositoryService__CosmosDb__DatabaseName" Value="[TodoApi_RepositoryService__CosmosDb__DatabaseName]"/>
	  <EnvironmentVariable Name="RepositoryService__CosmosDb__CollectionName" Value="[TodoApi_RepositoryService__CosmosDb__CollectionName]"/>
	  <EnvironmentVariable Name="NotificationService__ServiceBus__ConnectionString" Value="[TodoApi_NotificationService__ServiceBus__ConnectionString]"/>
	  <EnvironmentVariable Name="NotificationService__ServiceBus__QueueName" Value="[TodoApi_NotificationService__ServiceBus__QueueName]"/>
	  <EnvironmentVariable Name="DataProtection__BlobStorage__ConnectionString" Value="[TodApi_DataProtection__BlobStorage__ConnectionString]"/>
	  <EnvironmentVariable Name="DataProtection__BlobStorage__ContainerName" Value="[TodoApi_DataProtection__BlobStorage__ContainerName]"/>
	  <EnvironmentVariable Name="ApplicationInsights__InstrumentationKey" Value="[TodoApi_ApplicationInsights__InstrumentationKey]"/>
	</EnvironmentOverrides>
	<Policies>
	  <ContainerHostPolicies CodePackageRef="Code">
		<RepositoryCredentials AccountName="[ACR_Username]" Password="[ACR_Password]" PasswordEncrypted="false"/>
		<PortBinding ContainerPort="80" EndpointRef="TodoApiEndpoint" />
	  </ContainerHostPolicies>
	</Policies>
  </ServiceManifestImport>
  <DefaultServices>
	<!-- The section below creates instances of service types, when an instance of this 
		 application type is created. You can also create one or more instances of service type using the 
		 ServiceFabric PowerShell module.
		 
		 The attribute ServiceTypeName below must match the name defined in the imported ServiceManifest.xml file. -->
	<Service Name="TodoWeb" ServicePackageActivationMode="ExclusiveProcess" ServiceDnsName="todoweb.todoapp">
	  <StatelessService ServiceTypeName="TodoWebType" InstanceCount="[TodoWeb_InstanceCount]">
		<SingletonPartition />
	  </StatelessService>
	</Service>
	<Service Name="TodoApi" ServicePackageActivationMode="ExclusiveProcess" ServiceDnsName="todoapi.todoapp">
	  <StatelessService ServiceTypeName="TodoApiType" InstanceCount="[TodoApi_InstanceCount]">
		<SingletonPartition />
	  </StatelessService>
	</Service>
  </DefaultServices>
</ApplicationManifest>

Cloud.xml

<?xml version="1.0" encoding="utf-8"?>
<Application Name="fabric:/TodoApp" xmlns="http://schemas.microsoft.com/2011/01/fabric" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Parameters>
	<Parameter Name="ACR_Username" Value="ACR_USERNAME" />
	<Parameter Name="ACR_Password" Value="ACR_PASSWORD" />
	<Parameter Name="TodoWeb_InstanceCount" Value="-1" />
	<Parameter Name="TodoWeb_ASPNETCORE_ENVIRONMENT" Value="Development"/>
	<Parameter Name="TodoWeb_TodoApiService__EndpointUri" Value="todoapi.todoapp"/>
	<Parameter Name="TodoWeb_DataProtection__BlobStorage__ConnectionString" Value="STORAGE_ACCOUNT_CONNECTION_STRING"/>
	<Parameter Name="TodoWeb_DataProtection__BlobStorage__ContainerName" Value="todoweb"/>
	<Parameter Name="TodoWeb_ApplicationInsights__InstrumentationKey" Value="APPLICATION_INSIGHTS_INSTRUMENTATION_KEY"/>
	<Parameter Name="TodoApi_InstanceCount" Value="-1" />
	<Parameter Name="TodoApi_ASPNETCORE_ENVIRONMENT" Value="Development"/>
	<Parameter Name="TodoApi_RepositoryService__CosmosDb__EndpointUri" Value="COSMOS_DB_ENDPOINT_URI"/>
	<Parameter Name="TodoApi_RepositoryService__CosmosDb__PrimaryKey" Value="COSMOS_DB_PRIMARY_KEY"/>
	<Parameter Name="TodoApi_RepositoryService__CosmosDb__DatabaseName" Value="COSMOS_DB_DATABASE_NAME"/>
	<Parameter Name="TodoApi_RepositoryService__CosmosDb__CollectionName" Value="COSMOS_DB_COLLECTION_NAME"/>
	<Parameter Name="TodoApi_NotificationService__ServiceBus__ConnectionString" Value="SERVICE_BUS_CONNECTION_STRING"/>
	<Parameter Name="TodoApi_NotificationService__ServiceBus__QueueName" Value="todoapi"/>
	<Parameter Name="TodoApi_DataProtection__BlobStorage__ConnectionString" Value="STORAGE_ACCOUNT_CONNECTION_STRING"/>
	<Parameter Name="TodoApi_DataProtection__BlobStorage__ContainerName" Value="todoapi"/>
	<Parameter Name="TodoApi_ApplicationInsights__InstrumentationKey" Value="APPLICATION_INSIGHTS_INSTRUMENTATION_KEY"/>
  </Parameters>
</Application>

Configuration

Before deploying the application to your Azure Service Fabric Linux cluster, open the Cloud.xml file and make the following changes:

Then, open the ServiceManifest of both the TodoApi and TodoWeb services and make the following changes:

Observations

namespace TodoWeb.Services
{
	/// <summary>
	/// TodoApiService class
	/// </summary>
	public class TodoApiService : ITodoApiService
	{
		#region Private Constants
		private const string DefaultBaseAddress = "todoapi";
		...
		#endregion

		#region Private Instance Fields
		private readonly IOptions<TodoApiServiceOptions> _options;
		private readonly ILogger<TodoApiService> _logger;
		private readonly HttpClient _httpClient;
		#endregion

		#region Public Constructor
		/// <summary>
		/// Initializes a new instance of the TodoApiService class. 
		/// </summary>
		/// <param name="options"Service options.</param>
		/// <param name="logger">Logger.</param>
		public TodoApiService(IOptions<TodoApiServiceOptions> options, 
							  ILogger<TodoApiService> logger)
		{
			_options = options;
			_logger = logger;

			var endpoint = string.IsNullOrWhiteSpace(_options?.Value?.EndpointUri) ?
						   DefaultBaseAddress :
						   _options.Value.EndpointUri;
			_httpClient = new HttpClient
			{
				BaseAddress = new Uri($"http://{endpoint}")
			};
			_httpClient.DefaultRequestHeaders.Accept.Clear();
			_httpClient.DefaultRequestHeaders.Add("ContentType", "application/json");
			_httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

			_logger.LogInformation(LoggingEvents.Configuration, $"HttpClient.BaseAddress = {_httpClient.BaseAddress}");
		}
		#endregion
	...
	}
}

The following picture shows the multi-container application using the Service Fabric Explorer.

Manifests

Deploy the application to a Linux cluster: parameters stored in Azure Key Vault

The TodoAppForLinuxFromDockerHub project shows how to safely deploy a multi-container application to an Azure Service Fabric Linux cluster in a production environment. As a security best practice, you should never store sensitive configuration data in the application manifest, service manifest or application parameters file of a Service Fabric application. Unauthorized users could steal this data from the source code repository. This project makes use of a single Azure Key Vault repository for storing secrets. Key Vault is a cloud-hosted service for managing cryptographic keys and other secrets. On larger projects, you should use multiple vaults for different environments (development & test, quality assurance, performance testing, production) and grant permissions to these resources only to a restricted set of authorized developers and operators. This project requires that the following sensitive data are stored in Azure Key Vault:

Instead, the following parameters are defined in clear-text in the Cloud.xml file:

Looking at the service manifests below, you can observe that both the frontend and backend service read configuration data from environment variables.

TodoApi ServiceManifest.xml

<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="TodoApiPkg"
				 Version="1.0.0"
				 xmlns="http://schemas.microsoft.com/2011/01/fabric"
				 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
				 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ServiceTypes>
	<!-- This is the name of your ServiceType.
		 The UseImplicitHost attribute indicates this is a guest service. -->
	<StatelessServiceType ServiceTypeName="TodoApiType" UseImplicitHost="true" />
  </ServiceTypes>

  <!-- Code package is your service executable. -->
  <CodePackage Name="Code" Version="1.0.0">
	<EntryPoint>
	  <!-- Follow this link for more information about deploying Windows containers to Service Fabric: https://aka.ms/sfguestcontainers -->
	  <ContainerHost>
		<ImageName>DOCKER_HUB_REPOSITORY/todoapi:v1</ImageName>
	  </ContainerHost>
	</EntryPoint>
	<!-- Pass environment variables to your container: -->
	<EnvironmentVariables>
	  <EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value=""/>
	  <EnvironmentVariable Name="AzureKeyVault__Certificate__CertificateEnvironmentVariable" Value=""/>
	  <EnvironmentVariable Name="AzureKeyVault__Certificate__KeyEnvironmentVariable" Value=""/>
	  <EnvironmentVariable Name="AzureKeyVault__ClientId" Value=""/>
	  <EnvironmentVariable Name="AzureKeyVault__Name" Value=""/>
	  <EnvironmentVariable Name="NotificationService__ServiceBus__QueueName" Value=""/>
	  <EnvironmentVariable Name="DataProtection__BlobStorage__ContainerName" Value=""/>
	</EnvironmentVariables>
  </CodePackage>

  <!-- Config package is the contents of the Config directoy under PackageRoot that contains an 
	   independently-updateable and versioned set of custom configuration settings for your service. -->
  <ConfigPackage Name="Config" Version="1.0.0" />

  <Resources>
	<Endpoints>
	  <!-- This endpoint is used by the communication listener to obtain the port on which to 
		   listen. Please note that if your service is partitioned, this port is shared with 
		   replicas of different partitions that are placed in your code. -->
	  <Endpoint Name="TodoApiEndpoint" Port="80" UriScheme="http" Protocol="http"/>
	</Endpoints>
  </Resources>
</ServiceManifest>

TodoWeb ServiceManifest.xml

<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="TodoWebPkg"
				 Version="1.0.0"
				 xmlns="http://schemas.microsoft.com/2011/01/fabric"
				 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
				 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ServiceTypes>
	<!-- This is the name of your ServiceType.
		 The UseImplicitHost attribute indicates this is a guest service. -->
	<StatelessServiceType ServiceTypeName="TodoWebType" UseImplicitHost="true" />
  </ServiceTypes>

  <!-- Code package is your service executable. -->
  <CodePackage Name="Code" Version="1.0.0">
	<EntryPoint>
	  <!-- Follow this link for more information about deploying Windows containers to Service Fabric: https://aka.ms/sfguestcontainers -->
	  <ContainerHost>
		<ImageName>DOCKER_HUB_REPOSITORY/todoweb:v1</ImageName>
	  </ContainerHost>
	</EntryPoint>
	<!-- Pass environment variables to your container: -->
	<EnvironmentVariables>
	  <EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value=""/>
	  <EnvironmentVariable Name="AzureKeyVault__Certificate__CertificateEnvironmentVariable" Value=""/>
	  <EnvironmentVariable Name="AzureKeyVault__Certificate__KeyEnvironmentVariable" Value=""/>
	  <EnvironmentVariable Name="AzureKeyVault__ClientId" Value=""/>
	  <EnvironmentVariable Name="AzureKeyVault__Name" Value=""/>
	  <EnvironmentVariable Name="TodoApiService__EndpointUri" Value=""/>
	  <EnvironmentVariable Name="DataProtection__BlobStorage__ContainerName" Value=""/>
	</EnvironmentVariables>
  </CodePackage>

  <!-- Config package is the contents of the Config directoy under PackageRoot that contains an 
	   independently-updateable and versioned set of custom configuration settings for your service. -->
  <ConfigPackage Name="Config" Version="1.0.0" />

  <Resources>
	<Endpoints>
	  <!-- This endpoint is used by the communication listener to obtain the port on which to 
		   listen. Please note that if your service is partitioned, this port is shared with 
		   replicas of different partitions that are placed in your code. -->
	  <Endpoint Name="TodoWebEndpoint" Port="8080" UriScheme="http" Protocol="http"/>
	</Endpoints>
  </Resources>
</ServiceManifest>

Looking at the application manifest, you can observe that the certificate used by each service needs to be installed in the Azure Service Fabric Linux cluster before deploying the application. In fact, the certificate identified by its thumbprint needs to exist in the cluster nodes before Docker containers are created for each service. As an alternative, if the certificate used by the application to authenticate against Azure Key Vault is not already installed in the cluster, and you have the certificate in the required form (.pfx format for a Windows cluster, .pem and .key format for a Linux cluster),you can use a data package to pass the certificate to both the frontend and backend service. We'll look at this technique in the next section where we explain how to deploy the application to a Service Fabric Windows cluster in Azure.

ApplicationManifest.xml

<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest ApplicationTypeName="TodoAppType"
					 ApplicationTypeVersion="1.0.0"
					 xmlns="http://schemas.microsoft.com/2011/01/fabric"
					 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
					 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Parameters>
	<!-- Shared Parameters -->
	<Parameter Name="DockerHub_Username" DefaultValue="" />
	<Parameter Name="DockerHub_Password" DefaultValue="" />
	<Parameter Name="ASPNETCORE_ENVIRONMENT" DefaultValue=""/>
	<Parameter Name="Certificate_Thumbprint" DefaultValue="" />
	<Parameter Name="AzureKeyVault__ClientId" DefaultValue=""/>
	<Parameter Name="AzureKeyVault__Name" DefaultValue=""/>
	<!-- TodoWeb Parameters -->
	<Parameter Name="TodoWeb_InstanceCount" DefaultValue="-1" />
	<Parameter Name="TodoWeb_AzureKeyVault__Certificate__CertificateEnvironmentVariable" DefaultValue=""/>
	<Parameter Name="TodoWeb_AzureKeyVault__Certificate__KeyEnvironmentVariable" DefaultValue=""/>
	<Parameter Name="TodoWeb_TodoApiService__EndpointUri" DefaultValue=""/>
	<Parameter Name="TodoWeb_DataProtection__BlobStorage__ContainerName" DefaultValue=""/>
	<!-- TodoApi Parameters -->
	<Parameter Name="TodoApi_InstanceCount" DefaultValue="-1" />
	<Parameter Name="TodoApi_AzureKeyVault__Certificate__CertificateEnvironmentVariable" DefaultValue=""/>
	<Parameter Name="TodoApi_AzureKeyVault__Certificate__KeyEnvironmentVariable" DefaultValue=""/>
	<Parameter Name="TodoApi_NotificationService__ServiceBus__QueueName" DefaultValue=""/>
	<Parameter Name="TodoApi_DataProtection__BlobStorage__ContainerName" DefaultValue=""/>
  </Parameters>
  <!-- Import the ServiceManifest from the ServicePackage. The ServiceManifestName and ServiceManifestVersion 
	   should match the Name and Version attributes of the ServiceManifest element defined in the 
	   ServiceManifest.xml file. -->
  <ServiceManifestImport>
	<ServiceManifestRef ServiceManifestName="TodoWebPkg" ServiceManifestVersion="1.0.0" />
	<ConfigOverrides />
	<EnvironmentOverrides CodePackageRef="Code">
	  <EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value="[ASPNETCORE_ENVIRONMENT]"/>
	  <EnvironmentVariable Name="AzureKeyVault__Certificate__CertificateEnvironmentVariable" Value="[TodoWeb_AzureKeyVault__Certificate__CertificateEnvironmentVariable]"/>
	  <EnvironmentVariable Name="AzureKeyVault__Certificate__KeyEnvironmentVariable" Value="[TodoWeb_AzureKeyVault__Certificate__KeyEnvironmentVariable]"/>
	  <EnvironmentVariable Name="AzureKeyVault__ClientId" Value="[AzureKeyVault__ClientId]"/>
	  <EnvironmentVariable Name="AzureKeyVault__Name" Value="[AzureKeyVault__Name]"/>
	  <EnvironmentVariable Name="TodoApiService__EndpointUri" Value="[TodoWeb_TodoApiService__EndpointUri]"/>
	  <EnvironmentVariable Name="DataProtection__BlobStorage__ContainerName" Value="[TodoWeb_DataProtection__BlobStorage__ContainerName]"/>
	</EnvironmentOverrides>
	<Policies>
	  <ContainerHostPolicies CodePackageRef="Code">
		<RepositoryCredentials AccountName="[DockerHub_Username]" Password="[DockerHub_Password]" PasswordEncrypted="false"/>
		<PortBinding ContainerPort="80" EndpointRef="TodoWebEndpoint" />
		<CertificateRef Name="TodoListCert" X509FindValue="[Certificate_Thumbprint]"/>
	  </ContainerHostPolicies>
	</Policies>
  </ServiceManifestImport>
  <ServiceManifestImport>
	<ServiceManifestRef ServiceManifestName="TodoApiPkg" ServiceManifestVersion="1.0.0" />
	<ConfigOverrides />
	<EnvironmentOverrides CodePackageRef="Code">
	  <EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value="[ASPNETCORE_ENVIRONMENT]"/>
	  <EnvironmentVariable Name="AzureKeyVault__Certificate__CertificateEnvironmentVariable" Value="[TodoApi_AzureKeyVault__Certificate__CertificateEnvironmentVariable]"/>
	  <EnvironmentVariable Name="AzureKeyVault__Certificate__KeyEnvironmentVariable" Value="[TodoApi_AzureKeyVault__Certificate__KeyEnvironmentVariable]"/>
	  <EnvironmentVariable Name="AzureKeyVault__ClientId" Value="[AzureKeyVault__ClientId]"/>
	  <EnvironmentVariable Name="AzureKeyVault__Name" Value="[AzureKeyVault__Name]"/>
	  <EnvironmentVariable Name="NotificationService__ServiceBus__QueueName" Value="[TodoApi_NotificationService__ServiceBus__QueueName]"/>
	  <EnvironmentVariable Name="DataProtection__BlobStorage__ContainerName" Value="[TodoApi_DataProtection__BlobStorage__ContainerName]"/>
	</EnvironmentOverrides>
	<Policies>
	  <ContainerHostPolicies CodePackageRef="Code">
		<RepositoryCredentials AccountName="[DockerHub_Username]" Password="[DockerHub_Password]" PasswordEncrypted="false"/>
		<PortBinding ContainerPort="80" EndpointRef="TodoApiEndpoint" />
		<CertificateRef Name="TodoListCert" X509FindValue="[Certificate_Thumbprint]"/>
	  </ContainerHostPolicies>
	</Policies>
  </ServiceManifestImport>
  <DefaultServices>
	<!-- The section below creates instances of service types, when an instance of this 
		 application type is created. You can also create one or more instances of service type using the 
		 ServiceFabric PowerShell module.
		 
		 The attribute ServiceTypeName below must match the name defined in the imported ServiceManifest.xml file. -->
	<Service Name="TodoWeb" ServicePackageActivationMode="ExclusiveProcess" ServiceDnsName="todoweb.todoapp">
	  <StatelessService ServiceTypeName="TodoWebType" InstanceCount="[TodoWeb_InstanceCount]">
		<SingletonPartition />
	  </StatelessService>
	</Service>
	<Service Name="TodoApi" ServicePackageActivationMode="ExclusiveProcess" ServiceDnsName="todoapi.todoapp">
	  <StatelessService ServiceTypeName="TodoApiType" InstanceCount="[TodoApi_InstanceCount]">
		<SingletonPartition />
	  </StatelessService>
	</Service>
  </DefaultServices>
</ApplicationManifest>

Cloud.xml

<?xml version="1.0" encoding="utf-8"?>
<Application Name="fabric:/TodoApp" xmlns="http://schemas.microsoft.com/2011/01/fabric" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Parameters>
	<!-- Shared Parameters -->
	<Parameter Name="DockerHub_Username" Value="DOCKER_HUB_USERNAME" />
	<Parameter Name="DockerHub_Password" Value="DOCKER_HUB_PASSWORD" />
	<Parameter Name="ASPNETCORE_ENVIRONMENT" Value="Development"/>
	<Parameter Name="Certificate_Thumbprint" Value="CERTIFICATE_THUMBPRINT" />
	<Parameter Name="AzureKeyVault__ClientId" Value="AZURE_AD_APPLICATION_ID"/>
	<Parameter Name="AzureKeyVault__Name" Value="AZURE_KEY_VAULT_NAME"/>
	<!-- TodoWeb Parameters -->
	<Parameter Name="TodoWeb_InstanceCount" Value="-1" />
	<Parameter Name="TodoWeb_AzureKeyVault__Certificate__CertificateEnvironmentVariable" Value="Certificates_TodoWebPkg_Code_TodoListCert_PEM"/>
	<Parameter Name="TodoWeb_AzureKeyVault__Certificate__KeyEnvironmentVariable" Value="Certificates_TodoWebPkg_Code_TodoListCert_PrivateKey"/>
	<Parameter Name="TodoWeb_TodoApiService__EndpointUri" Value="todoapi.todoapp"/>
	<Parameter Name="TodoWeb_DataProtection__BlobStorage__ContainerName" Value="todoweb"/>
	<!-- TodoApi Parameters -->
	<Parameter Name="TodoApi_InstanceCount" Value="-1" />
	<Parameter Name="TodoWeb_AzureKeyVault__Certificate__CertificateEnvironmentVariable" Value="Certificates_TodoWebPkg_Code_TodoListCert_PEM"/>
	<Parameter Name="TodoWeb_AzureKeyVault__Certificate__KeyEnvironmentVariable" Value="Certificates_TodoWebPkg_Code_TodoListCert_PrivateKey"/>
	<Parameter Name="TodoApi_NotificationService__ServiceBus__QueueName" Value="todoapi"/>
	<Parameter Name="TodoApi_DataProtection__BlobStorage__ContainerName" Value="todoapi"/>
  </Parameters>
</Application>

Configuration

Before deploying the application to your Azure Service Fabric Linux cluster, open the Cloud.xml file and make the following changes:

Then, open the ServiceManifest of both the TodoApi and TodoWeb services and make the following changes:

Deploy the application to a Windows cluster: parameters stored in Azure Key Vault

The TodoAppForWindowsFromDockerHub project shows how to safely deploy a multi-container application to an Azure Service Fabric Windows cluster in a production environment. As mentioned in the previous section, you should never store sensitive configuration data in the application manifest, service manifest or application parameters file of a Service Fabric application. Unauthorized users could steal this data from the source code repository. This project makes use of a single Azure Key Vault repository for storing secrets. Key Vault is a cloud-hosted service for managing cryptographic keys and other secrets. On larger projects, you should use multiple vaults for different environments (development & test, quality assurance, performance testing, production) and grant permissions to these resources only to a restricted set of authorized developers and operators. This project requires that the following sensitive data are stored in Azure Key Vault:

Instead, the following parameters are defined in clear-text in the Cloud.xml file:

Looking at the service manifests below, you can observe that both the frontend and backend service read configuration data from environment variables, while the certificate used to authenticate against Azure Key Vault is contained in data package called Data.

Note: Docker containers offer isolation not virtualization. The operating system of the host machine and container image must be the same. You cannot use a Linux container on a Windows machine or a Windows container on a Linux machine. As a consequence, you cannot use a Docker image for Linux to deploy a Window container to a Service Fabric Windows cluster in Azure. Likewise, you cannot use a Docker image for Windows to deploy a Linux container to a Service Fabric Linux cluster in Azure. Hence, you need to build OS-specific images to deploy the multi-container application to a Service Fabric Windows or Linux cluster in Azure.

TodoApi ServiceManifest.xml

<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="TodoApiPkg"
				 Version="1.0.0"
				 xmlns="http://schemas.microsoft.com/2011/01/fabric"
				 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
				 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ServiceTypes>
	<!-- This is the name of your ServiceType.
		 The UseImplicitHost attribute indicates this is a guest service. -->
	<StatelessServiceType ServiceTypeName="TodoApiType" UseImplicitHost="true" />
  </ServiceTypes>

  <!-- Code package is your service executable. -->
  <CodePackage Name="Code" Version="1.0.0">
	<EntryPoint>
	  <!-- Follow this link for more information about deploying Windows containers to Service Fabric: https://aka.ms/sfguestcontainers -->
	  <ContainerHost>
		<ImageName>DOCKER_HUB_REPOSITORY/wintodoapi:v1</ImageName>
	  </ContainerHost>
	</EntryPoint>
	<!-- Pass environment variables to your container: -->
	<EnvironmentVariables>
	  <EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value=""/>
	  <EnvironmentVariable Name="AzureKeyVault__Certificate__CertificateEnvironmentVariable" Value=""/>
	  <EnvironmentVariable Name="AzureKeyVault__Certificate__KeyEnvironmentVariable" Value=""/>
	  <EnvironmentVariable Name="AzureKeyVault__ClientId" Value=""/>
	  <EnvironmentVariable Name="AzureKeyVault__Name" Value=""/>
	  <EnvironmentVariable Name="NotificationService__ServiceBus__QueueName" Value=""/>
	  <EnvironmentVariable Name="DataProtection__BlobStorage__ContainerName" Value=""/>
	</EnvironmentVariables>
  </CodePackage>

  <!-- Config package is the contents of the Config directoy under PackageRoot that contains an 
	   independently-updateable and versioned set of custom configuration settings for your service. -->
  <ConfigPackage Name="Config" Version="1.0.0" />

  <!-- Data Package defines the name of a folder which contains data files, in this case the certificate
	   used by the container to authenticate against Key Vault -->
  <DataPackage Name="Data" Version="1.0.0"/>

  <Resources>
	<Endpoints>
	  <!-- This endpoint is used by the communication listener to obtain the port on which to 
		   listen. Please note that if your service is partitioned, this port is shared with 
		   replicas of different partitions that are placed in your code. -->
	  <Endpoint Name="TodoApiEndpoint" Port="80" UriScheme="http" Protocol="http"/>
	</Endpoints>
  </Resources>
</ServiceManifest>

TodoWeb ServiceManifest.xml

<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="TodoWebPkg"
				 Version="1.0.0"
				 xmlns="http://schemas.microsoft.com/2011/01/fabric"
				 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
				 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ServiceTypes>
	<!-- This is the name of your ServiceType.
		 The UseImplicitHost attribute indicates this is a guest service. -->
	<StatelessServiceType ServiceTypeName="TodoWebType" UseImplicitHost="true" />
  </ServiceTypes>

  <!-- Code package is your service executable. -->
  <CodePackage Name="Code" Version="1.0.0">
	<EntryPoint>
	  <!-- Follow this link for more information about deploying Windows containers to Service Fabric: https://aka.ms/sfguestcontainers -->
	  <ContainerHost>
		<ImageName>DOCKER_HUB_REPOSITORY/wintodoweb:v1</ImageName>
	  </ContainerHost>
	</EntryPoint>
	<!-- Pass environment variables to your container: -->
	<EnvironmentVariables>
	  <EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value=""/>
	  <EnvironmentVariable Name="AzureKeyVault__Certificate__CertificateEnvironmentVariable" Value=""/>
	  <EnvironmentVariable Name="AzureKeyVault__Certificate__KeyEnvironmentVariable" Value=""/>
	  <EnvironmentVariable Name="AzureKeyVault__ClientId" Value=""/>
	  <EnvironmentVariable Name="AzureKeyVault__Name" Value=""/>
	  <EnvironmentVariable Name="TodoApiService__EndpointUri" Value=""/>
	  <EnvironmentVariable Name="DataProtection__BlobStorage__ContainerName" Value=""/>
	</EnvironmentVariables>
  </CodePackage>

  <!-- Config package is the contents of the Config directoy under PackageRoot that contains an 
	   independently-updateable and versioned set of custom configuration settings for your service. -->
  <ConfigPackage Name="Config" Version="1.0.0" />

  <!-- Data Package defines the name of a folder which contains data files, in this case the certificate
  used by the container to authenticate against Key Vault -->
  <DataPackage Name="Data" Version="1.0.0"/>

  <Resources>
	<Endpoints>
	  <!-- This endpoint is used by the communication listener to obtain the port on which to 
		   listen. Please note that if your service is partitioned, this port is shared with 
		   replicas of different partitions that are placed in your code. -->
	  <Endpoint Name="TodoWebEndpoint" Port="8080" UriScheme="http" Protocol="http"/>
	</Endpoints>
  </Resources>
</ServiceManifest>

Looking at the application manifest, you can observe the certificate used by the frontend and backend services to authenticate against Azure Key Vault is defined in a data package.

ApplicationManifest.xml

<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest ApplicationTypeName="TodoAppType"
					 ApplicationTypeVersion="1.0.0"
					 xmlns="http://schemas.microsoft.com/2011/01/fabric"
					 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
					 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Parameters>
	<!-- Shared Parameters -->
	<Parameter Name="DockerHub_Username" DefaultValue="" />
	<Parameter Name="DockerHub_Password" DefaultValue="" />
	<Parameter Name="ASPNETCORE_ENVIRONMENT" DefaultValue=""/>
	<Parameter Name="Certificate_Thumbprint" DefaultValue="" />
	<Parameter Name="AzureKeyVault__ClientId" DefaultValue=""/>
	<Parameter Name="AzureKeyVault__Name" DefaultValue=""/>
	<!-- TodoWeb Parameters -->
	<Parameter Name="TodoWeb_InstanceCount" DefaultValue="-1" />
	<Parameter Name="TodoWeb_AzureKeyVault__Certificate__CertificateEnvironmentVariable" DefaultValue=""/>
	<Parameter Name="TodoWeb_AzureKeyVault__Certificate__KeyEnvironmentVariable" DefaultValue=""/>
	<Parameter Name="TodoWeb_TodoApiService__EndpointUri" DefaultValue=""/>
	<Parameter Name="TodoWeb_DataProtection__BlobStorage__ContainerName" DefaultValue=""/>
	<!-- TodoApi Parameters -->
	<Parameter Name="TodoApi_InstanceCount" DefaultValue="-1" />
	<Parameter Name="TodoApi_AzureKeyVault__Certificate__CertificateEnvironmentVariable" DefaultValue=""/>
	<Parameter Name="TodoApi_AzureKeyVault__Certificate__KeyEnvironmentVariable" DefaultValue=""/>
	<Parameter Name="TodoApi_NotificationService__ServiceBus__QueueName" DefaultValue=""/>
	<Parameter Name="TodoApi_DataProtection__BlobStorage__ContainerName" DefaultValue=""/>
  </Parameters>
  <!-- Import the ServiceManifest from the ServicePackage. The ServiceManifestName and ServiceManifestVersion 
	   should match the Name and Version attributes of the ServiceManifest element defined in the 
	   ServiceManifest.xml file. -->
  <ServiceManifestImport>
	<ServiceManifestRef ServiceManifestName="TodoWebPkg" ServiceManifestVersion="1.0.0" />
	<ConfigOverrides />
	<EnvironmentOverrides CodePackageRef="Code">
	  <EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value="[ASPNETCORE_ENVIRONMENT]"/>
	  <EnvironmentVariable Name="AzureKeyVault__Certificate__CertificateEnvironmentVariable" Value="[TodoWeb_AzureKeyVault__Certificate__CertificateEnvironmentVariable]"/>
	  <EnvironmentVariable Name="AzureKeyVault__Certificate__KeyEnvironmentVariable" Value="[TodoWeb_AzureKeyVault__Certificate__KeyEnvironmentVariable]"/>
	  <EnvironmentVariable Name="AzureKeyVault__ClientId" Value="[AzureKeyVault__ClientId]"/>
	  <EnvironmentVariable Name="AzureKeyVault__Name" Value="[AzureKeyVault__Name]"/>
	  <EnvironmentVariable Name="TodoApiService__EndpointUri" Value="[TodoWeb_TodoApiService__EndpointUri]"/>
	  <EnvironmentVariable Name="DataProtection__BlobStorage__ContainerName" Value="[TodoWeb_DataProtection__BlobStorage__ContainerName]"/>
	</EnvironmentOverrides>
	<Policies>
	  <ContainerHostPolicies CodePackageRef="Code">
		<RepositoryCredentials AccountName="[DockerHub_Username]" Password="[DockerHub_Password]" PasswordEncrypted="false"/>
		<PortBinding ContainerPort="80" EndpointRef="TodoWebEndpoint" />
		<CertificateRef Name="TodoListCert" X509FindValue="[Certificate_Thumbprint]"/>
	  </ContainerHostPolicies>
	</Policies>
  </ServiceManifestImport>
  <ServiceManifestImport>
	<ServiceManifestRef ServiceManifestName="TodoApiPkg" ServiceManifestVersion="1.0.0" />
	<ConfigOverrides />
	<EnvironmentOverrides CodePackageRef="Code">
	  <EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value="[ASPNETCORE_ENVIRONMENT]"/>
	  <EnvironmentVariable Name="AzureKeyVault__Certificate__CertificateEnvironmentVariable" Value="[TodoApi_AzureKeyVault__Certificate__CertificateEnvironmentVariable]"/>
	  <EnvironmentVariable Name="AzureKeyVault__Certificate__KeyEnvironmentVariable" Value="[TodoApi_AzureKeyVault__Certificate__KeyEnvironmentVariable]"/>
	  <EnvironmentVariable Name="AzureKeyVault__ClientId" Value="[AzureKeyVault__ClientId]"/>
	  <EnvironmentVariable Name="AzureKeyVault__Name" Value="[AzureKeyVault__Name]"/>
	  <EnvironmentVariable Name="NotificationService__ServiceBus__QueueName" Value="[TodoApi_NotificationService__ServiceBus__QueueName]"/>
	  <EnvironmentVariable Name="DataProtection__BlobStorage__ContainerName" Value="[TodoApi_DataProtection__BlobStorage__ContainerName]"/>
	</EnvironmentOverrides>
	<Policies>
	  <ContainerHostPolicies CodePackageRef="Code">
		<RepositoryCredentials AccountName="[DockerHub_Username]" Password="[DockerHub_Password]" PasswordEncrypted="false"/>
		<PortBinding ContainerPort="80" EndpointRef="TodoApiEndpoint" />
		<CertificateRef Name="TodoListCert" X509FindValue="[Certificate_Thumbprint]"/>
	  </ContainerHostPolicies>
	</Policies>
  </ServiceManifestImport>
  <DefaultServices>
	<!-- The section below creates instances of service types, when an instance of this 
		 application type is created. You can also create one or more instances of service type using the 
		 ServiceFabric PowerShell module.
		 
		 The attribute ServiceTypeName below must match the name defined in the imported ServiceManifest.xml file. -->
	<Service Name="TodoWeb" ServicePackageActivationMode="ExclusiveProcess" ServiceDnsName="todoweb.todoapp">
	  <StatelessService ServiceTypeName="TodoWebType" InstanceCount="[TodoWeb_InstanceCount]">
		<SingletonPartition />
	  </StatelessService>
	</Service>
	<Service Name="TodoApi" ServicePackageActivationMode="ExclusiveProcess" ServiceDnsName="todoapi.todoapp">
	  <StatelessService ServiceTypeName="TodoApiType" InstanceCount="[TodoApi_InstanceCount]">
		<SingletonPartition />
	  </StatelessService>
	</Service>
  </DefaultServices>
</ApplicationManifest>

Cloud.xml

<?xml version="1.0" encoding="utf-8"?>
<Application Name="fabric:/TodoApp" xmlns="http://schemas.microsoft.com/2011/01/fabric" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Parameters>
	<!-- Shared Parameters -->
	<Parameter Name="DockerHub_Username" Value="DOCKER_HUB_USERNAME" />
	<Parameter Name="DockerHub_Password" Value="DOCKER_HUB_PASSWORD" />
	<Parameter Name="ASPNETCORE_ENVIRONMENT" Value="Development"/>
	<Parameter Name="AzureKeyVault__ClientId" Value="AZURE_AD_APPLICATION_ID"/>
	<Parameter Name="AzureKeyVault__Name" Value="AZURE_KEY_VAULT_NAME"/>
	<!-- TodoWeb Parameters -->
	<Parameter Name="TodoWeb_InstanceCount" Value="-1" />
	<Parameter Name="TodoWeb_AzureKeyVault__Certificate__CertificateEnvironmentVariable" Value="Certificates_TodoWebPkg_Code_TodoListCert_PEM"/>
	<Parameter Name="TodoWeb_AzureKeyVault__Certificate__KeyEnvironmentVariable" Value="Certificates_TodoWebPkg_Code_TodoListCert_PrivateKey"/>
	<Parameter Name="TodoWeb_TodoApiService__EndpointUri" Value="todoapi.todoapp"/>
	<Parameter Name="TodoWeb_DataProtection__BlobStorage__ContainerName" Value="todoweb"/>
	<!-- TodoApi Parameters -->
	<Parameter Name="TodoApi_InstanceCount" Value="-1" />
	<Parameter Name="TodoWeb_AzureKeyVault__Certificate__CertificateEnvironmentVariable" Value="Certificates_TodoWebPkg_Code_TodoListCert_PEM"/>
	<Parameter Name="TodoWeb_AzureKeyVault__Certificate__KeyEnvironmentVariable" Value="Certificates_TodoWebPkg_Code_TodoListCert_PrivateKey"/>
	<Parameter Name="TodoApi_NotificationService__ServiceBus__QueueName" Value="todoapi"/>
	<Parameter Name="TodoApi_DataProtection__BlobStorage__ContainerName" Value="todoapi"/>
  </Parameters>
</Application>

Configuration

Before deploying the application to your Azure Service Fabric Windows cluster, open the Cloud.xml file and make the following changes:

Then, open the ServiceManifest of both the TodoApi and TodoWeb services and make the following changes:

Service Fabric Deployment with Docker Compose

Docker uses the docker-compose.yml file for defining multi-container applications. To make it easy for developers familiar with Docker to orchestrate existing container applications on Service Fabric, now you can use docker compose to deploy your multi-container application to a Service Fabric cluster in Azure. For more information, see Docker Compose deployment support in Azure Service Fabric.

Note 1: this project allows you to specify secret parameters in clear-text in the .yaml file only for testing purposes without the need to store them in Azure Key Vault. We highly discourage to use this approach in a production environment and we strongly recommend you to store sensitive configuration data in Azure Key Vault.

Note 2: Service Fabric can accept version 3 and later of docker-compose.yml files.

Below you can see the batch script and PowerShell script used to deploy the multi-container application to a Service Fabric Linux cluster in Azure with DNS Service. You can pull Docker images from an Azure Container Registry or from a Docker Hub repository.

Pull images from Azure Container Service

To deploy the multi-container application pulling the Docker images from an Azure Container Registry you can use one of the following scripts:

servicefabric-docker-compose-from-azure-container-registry.yml

version: '3'

services:
  todoapi:
	image:AZURE_CONTAINER_REGISTRY_NAME.azurecr.io/todoapi:latest
	deploy:
	  mode: replicated
	  replicas: 5
	environment:
	  - ASPNETCORE_ENVIRONMENT=Development
	  - RepositoryService__CosmosDb__EndpointUri=COSMOS_DB_ENDPOINT_URI
	  - RepositoryService__CosmosDb__PrimaryKey=COSMOS_DB_PRIMARY_KEY
	  - RepositoryService__CosmosDb__DatabaseName=TodoApiDb
	  - RepositoryService__CosmosDb__CollectionName=TodoApiCollection
	  - DataProtection__BlobStorage__ConnectionString=STORAGE_ACCOUNT_CONNECTION_STRING
	  - DataProtection__BlobStorage__ContainerName=todoapi
	  - NotificationService__ServiceBus__ConnectionString=SERVICE_BUS_CONNECTION_STRING
	  - NotificationService__ServiceBus__QueueName=todoapi
	  - ApplicationInsights__InstrumentationKey=APPLICATION_INSIGHTS_INSTRUMENTATION_KEY
	ports:
	  - "8081:80"

  todoweb:
	image:AZURE_CONTAINER_REGISTRY_NAME.azurecr.io/todoweb:latest
	deploy:
	  mode: replicated
	  replicas: 5
	environment:
	  - ASPNETCORE_ENVIRONMENT=Development
	  - TodoApiService__EndpointUri=todoapi.todoapp:8081
	  - DataProtection__BlobStorage__ConnectionString=STORAGE_ACCOUNT_CONNECTION_STRING
	  - DataProtection__BlobStorage__ContainerName=todoweb
	  - ApplicationInsights__InstrumentationKey=APPLICATION_INSIGHTS_INSTRUMENTATION_KEY
	ports:
	  - "8082:80"

servicefabric-create-deployment-from-azure-container-registry.cmd

REM Create a Service Fabric Compose deployment from a docker-compose.yml file
sfctl compose create --name DockerComposeTodoApp --file-path servicefabric-docker-compose-from-azure-container-registry.yml --user AZURE_CONTAINER_REGISTRY_USERNAME --encrypted-pass AZURE_CONTAINER_REGISTRY_PASSWORD

servicefabric-create-deployment-from-azure-container-registry.ps1

# Copy the package and register application type of version 1.0
$connectionEndpoint = "SERVICE_FABRIC_NAME.SERVICE_FABRIC_LOCATION.cloudapp.azure.com:19000"
$dockerComposeFile = $PSScriptRoot + '\servicefabric-docker-compose-from-azure-container-registry.yml'


Connect-ServiceFabricCluster -ConnectionEndpoint $connectionEndpoint 

New-ServiceFabricComposeDeployment -DeploymentName DockerComposeTodoApp -Compose $dockerComposeFile -RegistryUserName AZURE_CONTAINER_REGISTRY_USERNAME -RegistryPassword AZURE_CONTAINER_REGISTRY_PASSWORD

Configuration

Before deploying the application to your Azure Service Fabric Linux cluster, open the batch and PowerShell scripts and make the following changes:

Then, open the YAML file and make the following changes:

You can retrieve the usename and password of your Azure Container Registry by running folloring command:

az acr credential show -n AZURE_CONTAINER_REGISTRY

as shown in the following picture:

Credentials

Pull images from Docker Hub

To deploy the multi-container application pulling the Docker images from an Docker Hub you can use the following scripts:

servicefabric-docker-compose-from-docker-hub.yml

version: '3'

services:
  todoapi:
	image: DOCKER_HUB_REPOSITORY/todoapi:latest
	deploy:
	  mode: replicated
	  replicas: 5
	environment:
	  - ASPNETCORE_ENVIRONMENT=Development
	  - RepositoryService__CosmosDb__EndpointUri=COSMOS_DB_ENDPOINT_URI
	  - RepositoryService__CosmosDb__PrimaryKey=COSMOS_DB_PRIMARY_KEY
	  - RepositoryService__CosmosDb__DatabaseName=TodoApiDb
	  - RepositoryService__CosmosDb__CollectionName=TodoApiCollection
	  - DataProtection__BlobStorage__ConnectionString=STORAGE_ACCOUNT_CONNECTION_STRING
	  - DataProtection__BlobStorage__ContainerName=todoapi
	  - NotificationService__ServiceBus__ConnectionString=SERVICE_BUS_CONNECTION_STRING
	  - NotificationService__ServiceBus__QueueName=todoapi
	  - ApplicationInsights__InstrumentationKey=APPLICATION_INSIGHTS_INSTRUMENTATION_KEY
	ports:
	  - "8081:80"

  todoweb:
	image: DOCKER_HUB_REPOSITORY/todoweb:latest
	deploy:
	  mode: replicated
	  replicas: 5
	environment:
	  - ASPNETCORE_ENVIRONMENT=Development
	  - TodoApiService__EndpointUri=todoapi.todoapp:8081
	  - DataProtection__BlobStorage__ConnectionString=STORAGE_ACCOUNT_CONNECTION_STRING
	  - DataProtection__BlobStorage__ContainerName=todoweb
	  - ApplicationInsights__InstrumentationKey=APPLICATION_INSIGHTS_INSTRUMENTATION_KEY
	ports:
	  - "8082:80"

servicefabric-create-deployment-from-docker-hub.cmd

REM Create a Service Fabric Compose deployment from a docker-compose.yml file
sfctl compose create --name DockerComposeTodoApp --file-path servicefabric-docker-compose-from-docker-hub.yml --user DOCKER_HUB_USERNAME --encrypted-pass DOCKER_HUB_PASSWORD

servicefabric-create-deployment-from-docker-hub.ps1

# Copy the package and register application type of version 1.0
$connectionEndpoint = "SERVICE_FABRIC_NAME.SERVICE_FABRIC_LOCATION.cloudapp.azure.com:19000"
$dockerComposeFile = $PSScriptRoot + '\servicefabric-docker-compose-from-docker-hub.yml'


Connect-ServiceFabricCluster -ConnectionEndpoint $connectionEndpoint 

New-ServiceFabricComposeDeployment -DeploymentName DockerComposeTodoApp -Compose $dockerComposeFile -RegistryUserName DOCKER_HUB_USERNAME -RegistryPassword DOCKER_HUB_PASSWORD

Configuration

Before deploying the application to your Azure Service Fabric Linux cluster, open the batch scripts and PowerShell scripts and make the following changes:

The open the YAML file and make the following changes:

Observations

The following picture shows the multi-container application using the Service Fabric Explorer.

Architecture

Deploy the multi-container on an Azure Container Service Kubernetes Cluster

First of all, you have to create an Azure Container Service Kubernetes cluster. At this regard, you can use one of the following options:

Create an ACS cluster

You can create an Azure Container Service Kubernetes cluster using the Azure CLI and then use the kubectl command line interface to run commands against the Kubernetes cluster in Azure. For more information, see Deploy a Kubernetes cluster in Azure Container Service.

The following batch script shows how you can create an Azure Container Service Kubernetes cluster.

create-kubernetes-acs-cluster.cmd

REM Create a resource group for Kubernetes
az group create --name AcsKubernetesResourceGroup --location WestEurope --tags orchestrator=kubernetes

REM Create a Kubernetes cluster using ACS
az acs create --orchestrator-type kubernetes --name AcsKubernetes --resource-group AcsKubernetesResourceGroup --generate-ssh-keys --output jsonc

REM Install kubectl on the local machine
az acs kubernetes install-cli

REM Get credentials to connect to Kubernetes cluster using kubectl
az acs kubernetes get-credentials --name AcsKubernetes --resource-group AcsKubernetesResourceGroup 

REM Browse to Kubernetes Web UI
az acs kubernetes browse --name AcsKubernetes --resource-group AcsKubernetesResourceGroup

The last command launches the Kubernetes web UI that allows to manage all the Kubernetes entities (services, pods, deployments, etc.).

Architecture

Create an AKS cluster

You can create a managed Kubernetes service using the Azure CLI and then use the kubectl command line interface to run commands against the Kubernetes cluster in Azure. For more information, see Deploy an Azure Container Service (AKS) cluster.

The following batch script shows how you can create an Azure Container Service Kubernetes cluster.

create-kubernetes-aks-cluster.cmd

REM Create a resource group for Kubernetes
az group create --name AksKubernetesResourceGroup --location westus2 --output jsonc

REM Create a Kubernetes cluster using ACS
az aks create --resource-group AksKubernetesResourceGroup --name AksKubernetes --agent-count 3 --generate-ssh-keys --output jsonc

REM Install kubectl on the local machine
az aks install-cli

REM Get credentials to connect to Kubernetes cluster using kubectl
az aks get-credentials --name AksKubernetes --resource-group AksKubernetesResourceGroup 

REM Browse to Kubernetes Web UI
az aks browse --name AksKubernetes --resource-group AksKubernetesResourceGroup

You can also create the cluster from the Azure Cloud Shell. In this case, you can skip the az aks install-cli command as the kubectl command line interface is already installed in your shell.

Use a ConfigMap in Kubernetes to define non-sensitive configuration data

ConfigMaps are entities that can be used in Kubernetes to decouple non-sensitive configuration data from images and templates used to deploy an application. In particular, you can create a ConfigMap object to specify application paremeters and then map the environment variables in the Pod specification to the keys defined the ConfigMap. For more information, see Configure a Pod to Use a ConfigMap.

The first step is to create a YAML file to define a ConfigMap. The multi-container sample uses a ConfigMap to define a value for the following parameter\environment variable pairs:

The following YAML file can be used to create a ConfigMap object named todolist-configmap that contains a value for the above parameters.

todolist-configmap.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: todolist-configmap
  namespace: default
data:
  aspNetCoreEnvironment: Development
  todoApiServiceBusQueueName: todoapi
  todoApiServiceEndpointUri: todoapi
  todoWebDataProtectionBlobStorageContainerName: todoweb
  todoApiDataProtectionBlobStorageContainerName: todoapi

The following command can be used to create the todolist-configmap object in the Kubernetes cluster.

create-todolist-configmap.cmd

kubectl create --filename todolist-configmap.yml --record

You can use one of the following commands to read the value of the keys from the todolist-configmap:

# Get todolist-configmap
 kubectl get configmaps todolist-configmap -o yaml

 # Describe todolist-configmap
kubectl describe configmap todolist-configmap

Use a Secret in Kubernetes to define sensitive configuration data

In Kubernetes, a Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in an image; putting it in a Secret object allows for more control over how it is used, and reduces the risk of accidental exposure.

Sensitive data like connection strings, passwords and keys may be put in the YAML files used to create deployments and pods. However, putting such information in a Secret object provides the following advantages:

Objects of type secret are intended to hold sensitive information, such as passwords, connection strings, OAuth tokens, and ssh keys. Putting sensitive data in a secret is safer and more flexible than putting it verbatim in a pod definition or in a docker image. For more information, see Secrets.

The first step is to create a YAML file to define a Secret. Each item in the file must be base64 encoded. The multi-container sample uses a Secret to define a value for the following parameter\environment variable pairs:

The following YAML file can be used to create a Secret object named todolist-secret that contains a value for the above parameters.

todolist-secret.yml

apiVersion: v1
kind: Secret
metadata:
  name: todolist-secret
type: Opaque
data:
  cosmosDbEndpointUri: BASE64-ENCODED-COSMOS-DB-ENDPOINT-URI
  cosmosDBPrimaryKey: BASE64-ENCODED-COSMOS-DB-PRIMARY-KEY
  cosmosDbDatabaseName: BASE64-ENCODED-COSMOS-DB-DATABASE-NAME
  cosmosDbCollectionName: BASE64-ENCODED-COSMOS-DB-COLLECTION-NAME
  serviceBusConnectionString: BASE64-ENCODED-SERVICE-BUS-CONNECTION-STRING
  dataProtectionBlobStorageConnectionString: BASE64-ENCODED-BLOB-STORAGE-CONNECTION-STRING
  applicationInsightsInstrumentationKey: BASE64-ENCODED-APP-INSIGHTS-INSTRUMENTATION-KEY

On Windows, you can use the following command to translate a value into a base64 format.

powershell "[convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes(\"value\"))"

In a Linux Bash Shell, you can use the base64 command line utility to encode a value to a base64 format.

 echo -n "value" | base64

The following command can be used to create the todolist-secret object in the Kubernetes cluster.

create-todolist-secret.cmd

kubectl create --filename todolist-secret.yml --record

You can use kubectl to read the value of a setting defined from a secret object. For example, you can use the following Bash command to read the value of cosmosDbCollectionName parameter.

# Get secret
 kubectl get secret todolist-secret -o jsonpath="{.data.cosmosDbCollectionName}" | base64 --decode; echo

Deploy the multi-container application to ACS\Kubernetes from a local machine

On Kubernetes the multi-container application is composed by two services, one of the frontend service and one for the backend service, and 5 pods for each service. Each pod contains just a container or the todoapi or todoweb ASP.NET Core apps. The Docker images can be pulled from an Azure Container Registry or from Docker Hub. The solution contains scripts and yaml files to accomplish both tasks, but for brevity, let's see how you can deploy the application pulling the Docker images from a Docker Hub repository. The following YAML file contains the definition for the necessary services and deployments.

todolist-deployments-and-services-from-docker-hub.yml

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: todoapi
  labels:
	app: todoapi
spec:
  replicas: 3
  selector:
	matchLabels:
	  app: todoapi
  strategy:
	rollingUpdate:
	  maxSurge: 1
	  maxUnavailable: 1
  minReadySeconds: 5
  template:
	metadata:
	  labels:
		app: todoapi
	spec:
	  containers:
	  - name: todoapi
		image: paolosalvatori/todoapi:v2
		ports:
		- containerPort: 80
		env:
		- name: ASPNETCORE_ENVIRONMENT
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: aspNetCoreEnvironment
		- name: RepositoryService__CosmosDb__EndpointUri
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: cosmosDbEndpointUri
		- name: RepositoryService__CosmosDb__PrimaryKey
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: cosmosDBPrimaryKey
		- name: RepositoryService__CosmosDb__DatabaseName
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: cosmosDbDatabaseName
		- name: RepositoryService__CosmosDb__CollectionName
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: cosmosDbCollectionName
		- name: NotificationService__ServiceBus__ConnectionString
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: serviceBusConnectionString
		- name: NotificationService__ServiceBus__QueueName
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: todoApiServiceBusQueueName
		- name: DataProtection__BlobStorage__ConnectionString
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: dataProtectionBlobStorageConnectionString
		- name: DataProtection__BlobStorage__ContainerName
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: todoApiDataProtectionBlobStorageContainerName
		- name: ApplicationInsights__InstrumentationKey
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: applicationInsightsInstrumentationKey
---
apiVersion: v1
kind: Service
metadata:
  name: todoapi
  labels:
	app: todoapi
spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
	port: 80
  selector:
	app: todoapi
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: todoweb
  labels:
	app: todoweb
spec:
  replicas: 3
  selector:
	matchLabels:
	  app: todoweb
  strategy:
	rollingUpdate:
	  maxSurge: 1
	  maxUnavailable: 1
  minReadySeconds: 5 
  template:
	metadata:
	  labels:
		app: todoweb
	spec:
	  containers:
	  - name: todoweb
		image: paolosalvatori/todoweb:v2
		ports:
		- containerPort: 80
		env:
		- name: ASPNETCORE_ENVIRONMENT
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: aspNetCoreEnvironment
		- name: TodoApiService__EndpointUri
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: todoApiServiceEndpointUri
		- name: DataProtection__BlobStorage__ConnectionString
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: dataProtectionBlobStorageConnectionString
		- name: DataProtection__BlobStorage__ContainerName
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: todoWebDataProtectionBlobStorageContainerName
		- name: ApplicationInsights__InstrumentationKey
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: applicationInsightsInstrumentationKey
---
apiVersion: v1
kind: Service
metadata:
  name: todoweb
  labels:
	app: todoweb
spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
	port: 80
  selector:
	app: todoweb

Configuration

To use a secret in an environment variable in a pod specification, you need to proceed as follows:

For more information, see Using Secrets as Environment Variables.

Before deploying the application to your Azure Container Service Kubernetes cluster, open the YAML file and make the following changes:

The following script can be used to deploy the services and deployments.

create-application-in-kubernetes-from-docker-hub.cmd

kubectl create --filename todolist-deployments-and-services-from-docker-hub.yml --record

You can use the kubectl command line interface to list the newly created services and deployments by running the following commands:

REM Get services  
kubectl get services 

REM Get deployments
kubectl get deployments

As shown in the following pictures, you can see the todoapi and todoweb services and the todoapi and todoweb deployments.

Kubectl

The spec of both the todoapi and todoweb services define LoadBalancer as service type. Therefore, when you run the script, the Azure Container Service Kubernetes cluster creates two Load Balancing Rules with the Public IP of both services.

As you can observe, the service of type LoadBalancer have both a Cluster IP, which is a virtual, internal IP used by the kube-proxy running on each node of the cluster. On the contrary, if the type of the service is defined as ClusterIP, no Public IP is exposed over the internet, and, as consequence, no load balancing rule is created when you create a service of type ClusterIP. More in general, Kubernetes ServiceTypes allow you to specify what kind of service you want. The default is ClusterIP.

In our example, if you want to provide the ability to call the REST services exposed by the todoapi service to external applications running outside of the Kubernetes cluster, and not only to the todoweb service running on the same cluster, you have to specify LoadBalancer as a service type.

Use the Azure Management Portal to look at the Frontend IP configuration of the Azure Load Balancer used by ACS in front of the Kubernetes cluster nodes.

Kubectl

  1. The first row contains an IP address which corresponds to the Public IP of the azure-vote-front service. This is a quickstart sample deployed on the same Kubernetes cluster. For more information, see Deploy Kubernetes cluster for Linux containers
  2. The second row contains an IP address which corresponds to the Public IP of the todoapi service.
  3. The third row contains an IP address which corresponds to the Public IP of the todoweb service.

Likewise, if you use the Azure Management Portal to look at the Load balancing rules defined on the Load Balancer used in front of the cluster nodes, you can note that there is a rule for each Public IP on the port 80, all sharing the same Backend Pool

Kubectl

if you use the Azure Management Portal to look at the Backend pools, you can see that in this topology, the Azure Container Service Kubernetes cluster uses a single backend pool composed of just 3 nodes.

Kubectl

You can retrieve information about cluster nodes also running the following command:

REM Get nodes  
kubectl get nodes

as shown in the following picture:

Kubectl

Instead, if you want to use the todoapi service only as a backend service from the todoweb service, and you don't want to expose it publicly, you can specify ClusterIP as its service type. Choosing this value makes the service only reachable from within the cluster.

For more information, see:

Assign a Custom DNS to the public IP of the frontend service

As you can see in the previous picture, I could use the public IP exposed by the frontend service, that is 13.93.46.58, to access the web UI. However, you can register a public domain using domain registrar like [GoDaddy][https://uk.godaddy.com/] and associate this public domain to the address exposed by the Kubernetes service using the Azure DNS service. In fact, the Azure DNS allows you to host a DNS zone and manage the DNS records for a domain in Azure. In order for DNS queries for a custom domain to reach Azure DNS, the domain has to be delegated to Azure DNS from the parent domain. Keep in mind Azure DNS is not the domain registrar.

For more information on how to create an Azure DNS service and use the Azure CLI to manage zones and records, please see the following resources:

For more information to delegate a public domain to Azure DNS, see the following article:

I personally proceeded as follows:

REM Create a resource group for the DNS Zone and records
az group create --name DnsResourceGroup --location westeurope

REM Create a DNS zone called babosbird.com in the resource group DnsResourceGroup 
az network dns zone create --name babosbird.com --resource-group DnsResourceGroup --tags environment=K8s type=DNS

REM Create an A record for the public ip exposed by the public load balancer created for the TodoApi service in Kubernetes
az network dns record-set a add-record --resource-group DnsResourceGroup --zone-name babosbird.com --record-set-name www --ipv4-address 13.93.46.58
az network dns record-set a add-record --resource-group DnsResourceGroup --zone-name babosbird.com --record-set-name * --ipv4-address 13.93.46.58

REM View records
az network dns record-set list --resource-group DnsResourceGroup --zone-name babosbird.com

REM View records records at the top of the zone.
az network dns record-set list --resource-group DnsResourceGroup --zone-name babosbird.com --query "[?name=='@' && ends_with(type, 'NS')]|[0]"

REM Retrieve the name servers for your zone. These name servers should be configured with 
REM the domain name registrar where you purchased the domain name.
az network dns zone list --resource-group DnsResourceGroup --query "[?name=='babosbird.com']|[0].nameServers" --output json

nslookup -type=A www.babosbird.com

Deploy the multi-container application to AKS from the Azure Cloud Shell

You can also deploy the multi-container application from the Azure Cloud Shell. The Azure Cloud Shell utilizes Azure File storage to persist files across sessions. In order to use a YAML file that defines the services and deployments of the multi-container application from the Azure Cloud Shell, you can run the following batch script from a command-prompt on your local machine using the Azure CLI. The batch file performs the follwoing steps:

REM Create a resource group for the storage account
az group create --name RESOURCE_GROUP --location LOCATION --output jsonc

REM Create a storage account for clouddrive
az storage account create --name STORAGE_ACCOUNT_NAME --resource-group RESOURCE_GROUP --sku Standard_RAGRS --location LOCATION

REM Create a file share in the storage account
az storage share create --name SHARE_NAME --account-name STORAGE_ACCOUNT_NAME --account-key STORAGE_ACCOUNT_PRIMARY_KEY --resource-group RESOURCE_GROUP 

REM upload the YAML containing the definition of services and deployments to the clouddrive from a command-prompt on the local machine
az storage file upload --account-name STORAGE_ACCOUNT_NAME --account-key STORAGE_ACCOUNT_PRIMARY_KEY --share-name SHARE_NAME --source PATH_TO_YAML_FILE

At this point, launch Azure Cloud Shell from the top navigation of the Azure portal, select Bash from the drop down list and execute the following commands:

REM Get credentials to connect to Kubernetes cluster using kubectl
az aks get-credentials --name AksKubernetes --resource-group AksKubernetesResourceGroup 

REM Mount the file share called shell in the storage account called babo
clouddrive mount --subscription SUBSCRIPTION_ID --resource-group RESOURCE_GROUP --storage-account **STORAGE_ACCOUNT_NAME** --file-share SHARE_NAME

REM Show information about the file system
df

REM CD the cloudrive
cd /usr/USR/clouddrive

Configuration

Before runnng the above script, make the following changes:

For more information on how to mount a File Share from an Azure Cloud Shell, see Persist files in Azure Cloud Shell.

You can also deploy the multi-container application using the Azure CLI on your local machine or from the Azure Cloud Shell.

Use a ConfigMap in Kubernetes to define non-sensitive configuration data

ConfigMaps are entities that can be used in Kubernetes to decouple non-sensitive configuration data from images and templates used to deploy an application. In particular, you can create a ConfigMap object to specify application paremeters and then map the environment variables in the Pod specification to the keys defined the ConfigMap. For more information, see Configure a Pod to Use a ConfigMap.

The first step is to create a YAML file to define a ConfigMap. The multi-container sample uses a ConfigMap to define a value for the following parameter\environment variable pairs:

The following YAML file can be used to create a ConfigMap object named todolist-configmap that contains a value for the above parameters.

todolist-configmap.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: todolist-configmap
  namespace: default
data:
  aspNetCoreEnvironment: Development
  todoApiServiceBusQueueName: todoapi
  todoApiServiceEndpointUri: todoapi
  todoWebDataProtectionBlobStorageContainerName: todoweb
  todoApiDataProtectionBlobStorageContainerName: todoapi

The following command can be used to create the todolist-configmap object in the Kubernetes cluster.

create-todolist-configmap.cmd

kubectl create --filename todolist-configmap.yml --record

You can use one of the following commands to read the value of the keys from the todolist-configmap:

# Get todolist-configmap
 kubectl get configmaps todolist-configmap -o yaml

 # Describe todolist-configmap
kubectl describe configmap todolist-configmap

Use a Secret in Kubernetes to define sensitive configuration data

In Kubernetes, a Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in an image; putting it in a Secret object allows for more control over how it is used, and reduces the risk of accidental exposure.

Sensitive data like connection strings, passwords and keys may be put in the YAML files used to create deployments and pods. However, putting such information in a Secret object provides the following advantages:

Objects of type secret are intended to hold sensitive information, such as passwords, connection strings, OAuth tokens, and ssh keys. Putting sensitive data in a secret is safer and more flexible than putting it verbatim in a pod definition or in a docker image. For more information, see Secrets.

The first step is to create a YAML file to define a Secret. Each item in the file must be base64 encoded. The multi-container sample uses a Secret to define a value for the following parameter\environment variable pairs:

The following YAML file can be used to create a Secret object named todolist-secret that contains a value for the above parameters.

todolist-secret.yml

apiVersion: v1
kind: Secret
metadata:
  name: todolist-secret
type: Opaque
data:
  cosmosDbEndpointUri: BASE64-ENCODED-COSMOS-DB-ENDPOINT-URI
  cosmosDBPrimaryKey: BASE64-ENCODED-COSMOS-DB-PRIMARY-KEY
  cosmosDbDatabaseName: BASE64-ENCODED-COSMOS-DB-DATABASE-NAME
  cosmosDbCollectionName: BASE64-ENCODED-COSMOS-DB-COLLECTION-NAME
  serviceBusConnectionString: BASE64-ENCODED-SERVICE-BUS-CONNECTION-STRING
  dataProtectionBlobStorageConnectionString: BASE64-ENCODED-BLOB-STORAGE-CONNECTION-STRING
  applicationInsightsInstrumentationKey: BASE64-ENCODED-APP-INSIGHTS-INSTRUMENTATION-KEY

On Windows, you can use the following command to translate a value into a base64 format.

powershell "[convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes(\"value\"))"

In a Linux Bash Shell, you can use the base64 command line utility to encode a value to a base64 format.

 echo -n "value" | base64

The following command can be used to create the todolist-secret object in the Kubernetes cluster.

create-todolist-secret.cmd

kubectl create --filename todolist-secret.yml --record

You can use kubectl to read the value of a setting defined from a secret object. For example, you can use the following Bash command to read the value of cosmosDbCollectionName parameter.

# Get secret
 kubectl get secret todolist-secret -o jsonpath="{.data.cosmosDbCollectionName}" | base64 --decode; echo

Deploy the application to Kubernetes

You can now deploy the application. This time use a YAML file configured to pull Docker images from an Azure Container Service repository.

todolist-deployments-and-services-from-azure-container-registry.yml

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: todoapi
  labels:
	app: todoapi
spec:
  replicas: 3
  selector:
	matchLabels:
	  app: todoapi
  strategy:
	rollingUpdate:
	  maxSurge: 1
	  maxUnavailable: 1
  minReadySeconds: 5
  template:
	metadata:
	  labels:
		app: todoapi
	spec:
	  containers:
	  - name: todoapi
		image: AZURE_CONTAINER_REGISTRY_NAME.azurecr.io/todoapi:v1
		ports:
		- containerPort: 80
		env:
		- name: ASPNETCORE_ENVIRONMENT
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: aspNetCoreEnvironment
		- name: RepositoryService__CosmosDb__EndpointUri
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: cosmosDbEndpointUri
		- name: RepositoryService__CosmosDb__PrimaryKey
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: cosmosDBPrimaryKey
		- name: RepositoryService__CosmosDb__DatabaseName
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: cosmosDbDatabaseName
		- name: RepositoryService__CosmosDb__CollectionName
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: cosmosDbCollectionName
		- name: NotificationService__ServiceBus__ConnectionString
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: serviceBusConnectionString
		- name: NotificationService__ServiceBus__QueueName
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: todoApiServiceBusQueueName
		- name: DataProtection__BlobStorage__ConnectionString
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: dataProtectionBlobStorageConnectionString
		- name: DataProtection__BlobStorage__ContainerName
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: todoApiDataProtectionBlobStorageContainerName
		- name: ApplicationInsights__InstrumentationKey
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: applicationInsightsInstrumentationKey
---
apiVersion: v1
kind: Service
metadata:
  name: todoapi
  labels:
	app: todoapi
spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
	port: 80
  selector:
	app: todoapi
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: todoweb
  labels:
	app: todoweb
spec:
  replicas: 3
  selector:
	matchLabels:
	  app: todoweb
  strategy:
	rollingUpdate:
	  maxSurge: 1
	  maxUnavailable: 1
  minReadySeconds: 5 
  template:
	metadata:
	  labels:
		app: todoweb
	spec:
	  containers:
	  - name: todoweb
		image: AZURE_CONTAINER_REGISTRY_NAME.azurecr.io/todoweb:v1
		ports:
		- containerPort: 80
		env:
		- name: ASPNETCORE_ENVIRONMENT
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: aspNetCoreEnvironment
		- name: TodoApiService__EndpointUri
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: todoApiServiceEndpointUri
		- name: DataProtection__BlobStorage__ConnectionString
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: dataProtectionBlobStorageConnectionString
		- name: DataProtection__BlobStorage__ContainerName
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: todoWebDataProtectionBlobStorageContainerName
		- name: ApplicationInsights__InstrumentationKey
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: applicationInsightsInstrumentationKey
---
apiVersion: v1
kind: Service
metadata:
  name: todoweb
  labels:
	app: todoweb
spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
	port: 80
  selector:
	app: todoweb

Configuration

Before deploying the application to your managed Kubernetes service, open the YAML file and make the following changes:

Finally, run the following command from the Azure Cloud Shell to deploy the multi-container application to your Kubernetes cluster.

kubectl create --filename todolist-deployments-and-services-from-azure-container-registry.yml --record

You can run the following command to display the EXTERNAL-IP of the todoweb frontend service.

kubectl get services

as shown in the following picture:

AksAzureCloudShell

Finally, to verify that the application works as expected we can browse to the EXTERNAL-IP of the todoweb frontend service, as shown in the following picture.

AksTodoWeb

Configure Nginx Ingress Controller for TLS termination on Kubernetes

So far, we have seen how to configure both the todoweb frontend service and todoapi backend service to expose each a public HTTP endpoint. Now, let's assume we want to expose only the todoweb frontend service and configure it to use an HTTPS endpoint instead of an HTTP endpoint. To implement TLS termination in our Kubernetes cluster, we'll use an Ingress object and the Nginx Ingress controller. This component is a daemon, deployed as a Kubernetes Pod, that watches the apiserver's /ingresses endpoint for updates to the Ingress resource. The Nginx Ingress controller can be used to implement patterns like path-based fanout, SSL passthrough, TLS termination, basic or digest http authentication. For more information, see Advanced Ingress Configuration

You can deploy the Nginx Ingress controller to your Kubernetes cluster in Azure by using the kubectl CLI or using Helm. Helm is a tool for managing Kubernetes charts. Charts are packages of pre-configured Kubernetes resources. You can use Helm to:

For more information, see the following resources:

You can use the following Bash script to install and initialize Helm. In particular, the helm init command is used to install Helm components in a Kubernetes cluster and make client-side configurations.

install-helm.sh

#!/bin/bash
# Install helm on the local machine

# Download Helm
curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh

# Make get_helm executable
$ chmod 700 get_helm.sh

# Execute get_helm.sh
$ ./get_helm.sh

# Initialize helm
helm init

For more information, see Use Helm with Azure Container Service (AKS).

Then, you can use the following Bash script to install the Nginx Ingress controller in your Kubernetes cluster.

install-nginx-ingress-controller.sh

# Installs nginx-ingress using helm 
helm install stable/nginx-ingress -n nginx-ingress

You can you can use the following Bash script to scale out the number of replicas used by the Nginx Ingress controller to from 1 to 3.

scale-nginx-ingress-controller-replicas.sh

# Scale the number of replicas of the nginx-ingress-controller to 3
kubectl scale deployment nginx-ingress-controller --replicas=3 

# Scale the number of replicas of the nginx-ingress-default-backend to 3
kubectl scale deployment nginx-ingress-default-backend --replicas=3

The next step is to create a secret containing the certificate and private key used for TLS termination. We can use openssl utility to create a test certificate. You can use the following Bash script to install openssl.

install-open-ssl.sh

# Install openssl utility 
sudo apt update && sudo apt install openssl

Use the following Bash script to create a self-signed certificate for testing.

create-certificate.sh

# Create a self-signed certificate. Note: private and public keys are saved locally
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginxsvc/O=nginxsvc"

You can now run the following Bash script to create a Secret in your Kubernetes cluster using the self-signed certificate and its private key. Note: in a production environment, you should replace the self-signed certificate with a valid certificate issued by a trusted certificate authority.

create-tls-secret.sh

# Use the private and public key to create a secret used for SSL termination
kubectl create secret tls tls-secret --key tls.key --cert tls.crt

Note: the data keys must be named tls.crt and tls.key.

As an alternative, you can also the following YAML file to define the tls-secret:

tls-secret.yml

apiVersion: v1
kind: Secret
type: kubernetes.io/tls
metadata:
  name: tls-secret
data:
  tls.crt: [BASE64_ENCODED_CERT]
  tls.key: [BASE64_ENCODED_KEY]

You can run the following Bash command to create the tls-secret in your Kubernetes cluster using the above YAML file.

# Create tls-secret using YAML file
kubectl create -f /mnt/c/[PATH-TO-YAML-FILE]/tls-secret.yml

In this tutorial, you have seen how to manually create and deploy a certificate for TLS termination to a Kubernetes cluster. However, you can leverage kube-lego for automatic certificate generation. The open-soutce tool kube-lego automatically acquires certificates for Kubernetes Ingress resources from Let's Encrypt that issues certificates for free. For more information on kube-lego, see Configure Https / TLS / SSL on Kubernetes with Kube-Lego hosted on Azure Container Service. For more information about Helm, see The Kubernetes Package Manager on GitHub. For more information about NGINX Ingress controller, see the Readme file for its deployment.

Now you are ready to install the multi-container application to Kubernetes. In this section we'll see how to use the kubectl CLI to create services and deployments using the definitions contained in a YAML file. In the next section, we'll see how to create a Helm chart and use it to deploy the application to Kubernetes.

todolist-deployments-and-services-from-docker-hub-ssl.yml

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: ssl-todoapi
  labels:
	app: ssl-todoapi
spec:
  replicas: 3
  selector:
	matchLabels:
	  app: ssl-todoapi
  strategy:
	rollingUpdate:
	  maxSurge: 1
	  maxUnavailable: 1
  minReadySeconds: 5
  template:
	metadata:
	  labels:
		app: ssl-todoapi
	spec:
	  containers:
	  - name: ssl-todoapi
		image: DOCKER_HUB_REPOSITORY/todoapi:latest
		ports:
		- containerPort: 80
		env:
		- name: ASPNETCORE_ENVIRONMENT
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: aspNetCoreEnvironment
		- name: RepositoryService__CosmosDb__EndpointUri
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: cosmosDbEndpointUri
		- name: RepositoryService__CosmosDb__PrimaryKey
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: cosmosDBPrimaryKey
		- name: RepositoryService__CosmosDb__DatabaseName
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: cosmosDbDatabaseName
		- name: RepositoryService__CosmosDb__CollectionName
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: cosmosDbCollectionName
		- name: NotificationService__ServiceBus__ConnectionString
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: serviceBusConnectionString
		- name: NotificationService__ServiceBus__QueueName
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: todoApiServiceBusQueueName
		- name: DataProtection__BlobStorage__ConnectionString
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: dataProtectionBlobStorageConnectionString
		- name: DataProtection__BlobStorage__ContainerName
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: todoApiDataProtectionBlobStorageContainerName
		- name: ApplicationInsights__InstrumentationKey
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: applicationInsightsInstrumentationKey
---
apiVersion: v1
kind: Service
metadata:
  name: ssl-todoapi
  labels:
	app: ssl-todoapi
spec:
  type: ClusterIP
  ports:
  - protocol: TCP
	port: 80
  selector:
	app: ssl-todoapi
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: ssl-todoweb
  labels:
	app: ssl-todoapi
spec:
  replicas: 3
  selector:
	matchLabels:
	  app: ssl-todoweb
  strategy:
	rollingUpdate:
	  maxSurge: 1
	  maxUnavailable: 1
  minReadySeconds: 5 
  template:
	metadata:
	  labels:
		app: ssl-todoweb
	spec:
	  containers:
	  - name: ssl-todoweb
		image: DOCKER_HUB_REPOSITORY/todoweb:v1
		ports:
		- containerPort: 80
		env:
		- name: ASPNETCORE_ENVIRONMENT
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: aspNetCoreEnvironment
		- name: TodoApiService__EndpointUri
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: todoApiServiceEndpointUri
		- name: DataProtection__BlobStorage__ConnectionString
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: dataProtectionBlobStorageConnectionString
		- name: DataProtection__BlobStorage__ContainerName
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: todoWebDataProtectionBlobStorageContainerName
		- name: ApplicationInsights__InstrumentationKey
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: applicationInsightsInstrumentationKey
---
apiVersion: v1
kind: Service
metadata:
  name: ssl-todoweb
  labels:
	app: ssl-todoweb
spec:
  type: ClusterIP
  ports:
  - protocol: TCP
	port: 80
  selector:
	app: ssl-todoweb

Configuration

Before deploying the application to your managed Kubernetes service, open the YAML file and make the following changes:

Observations

Reading the above YAML file you can note the following:

You can now run the following Bash script to deploy the ssl-todoapi and ssl-todoweb services to your Kubernetes cluster.

# Deploy the TodoList application using kubectl or helm
kubectl create -f /mnt/c/[PATH-TO-YAML-FILE]/todolist-deployments-and-services-from-docker-hub-ssl.yml --record

In order to expose the ssl-todoweb frontend service with a HTTPS public endpoint, you have to create an Ingress object in the same namespace as the tls-secret object.

nginx-ingress.yml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
spec:
  tls:
  - secretName: tls-secret
  backend:
	serviceName: ssl-todoweb

You can run the following Bash command to create the nginx-ingress in your Kubernetes cluster using the above YAML file.

create-nginx-ingress.sh

# Create nginx-ingress using a YAML file
kubectl create -f /mnt/c/[PATH-TO-YAML-FILE]/nginx-ingress.yml

We have configured the Nginx Ingress Controller to implement SSL termination and route traffic to our service ssl-todoweb via HTTP. You can run the following Bash command to display the EXTERNAL-IP exposed by the Nginx Ingress Controller service.

kubectl get service nginx-ingress-nginx-ingress-controller

as shown in the following picture:

AksAzureCloudShell

Finally, to verify that the application works as expected we can browse to the EXTERNAL-IP of the Nginx Ingress Controller frontend service, as shown in the following picture.

AksTodoWeb

Use Helm to package and deploy your application to Kubernetes

As mentioned above, Helm is a tool for managing Kubernetes charts. Charts are packages of pre-configured Kubernetes resources. You can use Helm to:

Helm uses a packaging format called charts. A chart is a collection of files that describe a related set of Kubernetes resources. Charts are created as files and structured in a particular directory tree, then they can be packaged into versioned archives to be deployed. For more information see Charts in the Helm documentation.

A single chart can be used to create the services and deployments that compose the todolist multi-container application. The chart is organized as a collection of files inside of a directory. The directory name is the name of the chart, so in our case is equal to todolist. The following picture shows the structure of the directory structure.

AksAzureCloudShell

Description

Below you can see the content of the main files used by the chart.

Chart.yaml

name: TodoList
version: 1.0.0
description: todolist multi-container app
keywords:
  - todoapi
home: https://github.com/paolosalvatori/service-fabric-acs-kubernetes-multi-container-app
sources:
  - https://github.com/paolosalvatori/service-fabric-acs-kubernetes-multi-container-app
maintainers: # (optional)
  - name: Paolo Salvatori
	email: paolos@microsoft.com
	url: https://github.com/paolosalvatori
engine: gotpl 
appVersion: 1.0.0

values.yaml

imageRegistry: "paolosalvatori"
frontendImage: "todoweb"
backendImage: "todoapi"
frontendTag: "v2"
backendTag: "v2"
frontend: "todoweb"
backend: "todoapi"
frontendServiceType: "LoadBalancer"
backendServiceType: "LoadBalancer"

templates/Deployment.yaml

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: {{.Values.backend}}
  labels:
	app: {{.Values.backend}}
spec:
  replicas: 3
  selector:
	matchLabels:
	  app: {{.Values.backend}}
  strategy:
	rollingUpdate:
	  maxSurge: 1
	  maxUnavailable: 1
  minReadySeconds: 5
  template:
	metadata:
	  labels:
		app: {{.Values.backend}}
	spec:
	  containers:
	  - name: {{.Values.backend}}
		image: {{.Values.imageRegistry}}/{{.Values.backendImage}}:{{.Values.backendTag}}
		ports:
		- containerPort: 80
		env:
		- name: ASPNETCORE_ENVIRONMENT
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: aspNetCoreEnvironment
		- name: RepositoryService__CosmosDb__EndpointUri
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: cosmosDbEndpointUri
		- name: RepositoryService__CosmosDb__PrimaryKey
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: cosmosDBPrimaryKey
		- name: RepositoryService__CosmosDb__DatabaseName
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: cosmosDbDatabaseName
		- name: RepositoryService__CosmosDb__CollectionName
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: cosmosDbCollectionName
		- name: NotificationService__ServiceBus__ConnectionString
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: serviceBusConnectionString
		- name: NotificationService__ServiceBus__QueueName
		  value: {{.Values.queueName}}
		- name: DataProtection__BlobStorage__ConnectionString
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: dataProtectionBlobStorageConnectionString
		- name: DataProtection__BlobStorage__ContainerName
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: todoApiDataProtectionBlobStorageContainerName
		- name: ApplicationInsights__InstrumentationKey
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: applicationInsightsInstrumentationKey
---
apiVersion: v1
kind: Service
metadata:
  name: {{.Values.backend}}
spec:
  type: {{.Values.backendServiceType}}
  ports:
  - protocol: TCP
	port: 80
  selector:
	app: {{.Values.backend}}
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: {{.Values.frontend}}
  labels:
	app: {{.Values.frontend}}
spec:
  replicas: 3
  selector:
	matchLabels:
	  app: {{.Values.frontend}}
  strategy:
	rollingUpdate:
	  maxSurge: 1
	  maxUnavailable: 1
  minReadySeconds: 5 
  template:
	metadata:
	  labels:
		app: {{.Values.frontend}}
	spec:
	  containers:
	  - name: {{.Values.frontend}}
		image: {{.Values.imageRegistry}}/{{.Values.frontendImage}}:{{.Values.frontendTag}}
		ports:
		- containerPort: 80
		env:
		- name: ASPNETCORE_ENVIRONMENT
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: aspNetCoreEnvironment
		- name: TodoApiService__EndpointUri
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: todoApiServiceEndpointUri
		- name: DataProtection__BlobStorage__ConnectionString
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: dataProtectionBlobStorageConnectionString
		- name: DataProtection__BlobStorage__ContainerName
		  valueFrom:
			configMapKeyRef:
			  name: todolist-configmap
			  key: todoWebDataProtectionBlobStorageContainerName
		- name: ApplicationInsights__InstrumentationKey
		  valueFrom:
			secretKeyRef:
				name: todolist-secret
				key: applicationInsightsInstrumentationKey
---
apiVersion: v1
kind: Service
metadata:
  name: {{.Values.frontend}}
spec:
  type: {{.Values.frontendServiceType}}
  ports:
  - protocol: TCP
	port: 80
  selector:
	app: {{.Values.frontend}}

Observations

Examining the files above, you can note the following:

You can run the following Bash script to package the chart. This operation creates a compressed TAR Archive file with the following format ChartName-ChartVersion.tgz in the current folder, in our case TodoList-1.0.0.tgz.

# Build package
helm package /mnt/c/[PATH-TO-CHART]/TodoList

Make sure to have created the todolist-config and todolist-secret in your Kubernetes cluster as explained previously. Now, if you want to install the todolist application, run the following Bash command:

# Install package
helm install --name todolist TodoList-1.0.0.tgz

Now, let's assume we want to deploy a version of the application that uses HTTPS instead of HTTP. we want to change the service type for both the todoapi and todoweb services from LoadBalancer to ClusterIP and we want to add the ssl- prefix to the name of the corresponding services and deployments. You can accomplish this task simply creating the following ssl-todolist-values.yaml file

frontend: "ssl-todoweb"
backend: "ssl-todoapi"
frontendServiceType: "ClusterIP"
backendServiceType: "ClusterIP"

and by running this Bash command:

# Install package for ssl-todolist
helm install --name ssl-todolist --values /mnt/c/[PATH-TO-CHART]/ssl-todolist-values.yaml TodoList-1.0.0.tgz

Services

This section provides a brief introduction of the services used by the solution.

Azure Service Fabric

Azure Service Fabric is a distributed systems platform that makes it easy to package, deploy, and manage scalable and reliable microservices and containers. Service Fabric also addresses the significant challenges in developing and managing cloud native applications. Developers and administrators can avoid complex infrastructure problems and focus on implementing mission-critical, demanding workloads that are scalable, reliable, and manageable. Service Fabric represents the next-generation platform for building and managing these enterprise-class, tier-1, cloud-scale applications running in containers. For more information, see Overview of Azure Service Fabric.

Kubernetes

Kubernetes is a powerful system for managing containerized applications in a clustered environment. It aims to provide better ways of managing related, distributed components across varied infrastructure. Kubernetes is an open-source platform designed to automate deploying, scaling, and operating application containers. With Kubernetes, you are able to quickly and efficiently respond to customer demand:

For more information, see What is Kubernetes?.

Azure Container Service for Kubernetes

Azure Container Service for Kubernetes makes it simple to create, configure, and manage a cluster of virtual machines that are preconfigured to run containerized applications. This enables you to use your existing skills, or draw upon a large and growing body of community expertise, to deploy and manage container-based applications on Microsoft Azure. For more information, see Azure Container Service for Kubernetes.

Managed Kubernetes Service

Azure Container Service (AKS) manages your hosted Kubernetes environment, making it quick and easy to deploy and manage containerized applications without container orchestration expertise. It also eliminates the burden of ongoing operations and maintenance by provisioning, upgrading, and scaling resources on demand, without taking your applications offline. For more information, see Azure Container Service (AKS).

Application Insights

Application Insights is an extensible Application Performance Management (APM) service for web developers on multiple platforms. Use it to monitor your live web application. It will automatically detect performance anomalies. It includes powerful analytics tools to help you diagnose issues and to understand what users actually do with your app. It's designed to help you continuously improve performance and usability. It works for apps on a wide variety of platforms including .NET, Node.js and J2EE, hosted on-premises or in the cloud. For more information, see What is Application Insights?.

Service Bus Messaging

Microsoft Azure Service Bus is a reliable message delivery service. The purpose of this service is to make communication easier. When two or more parties want to exchange data, they need a communication facilitator. Service Bus is a brokered, or third-party communication mechanism. Service Bus messaging with queues, topics, and subscriptions can be thought of as asynchronous, or "temporally decoupled." Producers (senders) and consumers (receivers) do not have to be online at the same time. The messaging infrastructure reliably stores messages in a "broker" (for example, a queue) until the consuming party is ready to receive them. For more information, see Service Bus Messaging.

Azure Cosmos DB

Azure Cosmos DB is Microsoft's globally distributed, multi-model database. With the click of a button, Azure Cosmos DB enables you to elastically and independently scale throughput and storage across any number of Azure's geographic regions. Azure Cosmos DB supports multiple data models and popular APIs for accessing and querying data. The sample uses the Document DB API which provides a schema-less JSON database engine with SQL querying capabilities. For more information, see About Azure Cosmos DB

Azure Key Vault

Azure Key Vault helps safeguard cryptographic keys and secrets used by cloud applications and services. By using Key Vault, you can encrypt keys and secrets (such as authentication keys, storage account keys, data encryption keys, .PFX files, and passwords) by using keys that are protected by hardware security modules (HSMs). For added assurance, you can import or generate keys in HSMs. If you choose to do this, Microsoft processes your keys in FIPS 140-2 Level 2 validated HSMs (hardware and firmware). Key Vault streamlines the key management process and enables you to maintain control of keys that access and encrypt your data. Developers can create keys for development and testing in minutes, and then seamlessly migrate them to production keys. Security administrators can grant (and revoke) permission to keys, as needed.For more information, see What is Azure Key Vault?

Azure Container Registry

Azure Container Registry is a managed Docker registry service based on the open-source Docker Registry 2.0. Create and maintain Azure container registries to store and manage your private Docker container images. Use container registries in Azure with your existing container development and deployment pipelines, and draw on the body of Docker community expertise. For more information, see Introduction to private Docker container registries in Azure.

Azure DNS Service

The Domain Name System, or DNS, is responsible for translating (or resolving) a website or service name to its IP address. Azure DNS is a hosting service for DNS domains, providing name resolution using Microsoft Azure infrastructure. By hosting your domains in Azure, you can manage your DNS records using the same credentials, APIs, tools, and billing as your other Azure services. For more information, see Azure DNS overview.

More information

The Service Fabric container documentation provides details on the container features and scenarios.

The following are other useful links which contain more in depth information

The Azure Container Service with Kubernetes Documentation provides details on the container features and scenarios.

The following are other useful links which contain more in depth information

For more information on Docker, see the following resources:

For more information on ASP.NET, see the following resources:

MSFT OSS Code Of Conduct Notice

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.