Awesome

Key Vault FlexVolume [DEPRECATED]

WARNING :warning: This solution has been deprecated. Continue to use this for kubernetes version 1.15. For 1.16+, please use Azure Key Vault Provider for Secret Store CSI Driver

This solution is no longer supported as 1.15 is out of support in AKS

Seamlessly integrate your key management systems with Kubernetes.

Secrets, keys, and certificates in a key management system become a volume accessible to pods. Once the volume is mounted, its data is available directly in the container filesystem for your application.

Getting Started
Detailed Use Cases
Design
About Key Vault
About Certificates
Contributing
Code of Conduct

Getting started

Supported Providers

Azure Key Vault

💡 NOTE: To enable encryption at rest of Kubernetes data in etcd, use the Kubernetes KMS plugin for Azure Key Vault.

Installing Key Vault FlexVolume

OPTION 1: New AKS Engine cluster

AKS Engine creates customized Kubernetes clusters on Azure.

Follow the AKS Engine add-on documentation to create a new Kubernetes cluster with Key Vault FlexVolume already deployed.

OPTION 2: Existing AKS cluster

Azure Kubernetes Service (AKS) creates managed, supported Kubernetes clusters on Azure.

Deploy Key Vault FlexVolume to your AKS cluster with this command:

kubectl create -f https://raw.githubusercontent.com/Azure/kubernetes-keyvault-flexvol/master/deployment/kv-flexvol-installer.yaml

To validate Key Vault FlexVolume is running as expected, run the following command:

kubectl get pods -n kv

The output should show keyvault-flexvolume pods running on each agent node:

NAME                        READY     STATUS    RESTARTS   AGE
keyvault-flexvolume-f7bx8   1/1       Running   0          3m
keyvault-flexvolume-rcxbl   1/1       Running   0          3m
keyvault-flexvolume-z6jm6   1/1       Running   0          3m

If using keyvault-flexvolume in a cluster with pod security policy enabled, create the following policy that enables the spec required for keyvault-flexvolume to work -

kubectl apply -f https://raw.githubusercontent.com/Azure/kubernetes-keyvault-flexvol/master/deployment/kv-flexvol-psp.yaml

Using Key Vault FlexVolume

Key Vault FlexVolume offers four modes for accessing a Key Vault instance: Service Principal, Pod Identity, [VMSS User Assigned Managed Identity], [VMSS System Assigned Managed Identity].

OPTION 1: Service Principal

Add your service principal credentials as Kubernetes secrets accessible by the Key Vault FlexVolume driver.

kubectl create secret generic kvcreds --from-literal clientid=<CLIENTID> --from-literal clientsecret=<CLIENTSECRET> --type=azure/kv

Ensure this service principal has all the required permissions to access content in your Key Vault instance. If not, run the following Azure CLI commands:

# [Required for version < v0.0.13] Assign Reader Role to the service principal for your keyvault
az role assignment create --role Reader --assignee <principalid> --scope /subscriptions/<subscriptionid>/resourcegroups/<resourcegroup>/providers/Microsoft.KeyVault/vaults/<keyvaultname>

# Assign key vault permissions to your service principal
az keyvault set-policy -n $KV_NAME --key-permissions get --spn <YOUR SPN CLIENT ID>
az keyvault set-policy -n $KV_NAME --secret-permissions get --spn <YOUR SPN CLIENT ID>
az keyvault set-policy -n $KV_NAME --certificate-permissions get --spn <YOUR SPN CLIENT ID>

Fill in the missing pieces in this deployment for your own deployment. Make sure to:

Reference the service principal Kubernetes secret created in the previous step
```
secretRef:
  name: kvcreds
```

Pass in properties for the Key Vault instance to the FlexVolume driver.

Name	Required	Description	Default Value
usepodidentity	no	specify access mode: use a service principal or pod identity or vm managed identity	"false"
usevmmanagedidentity	not required, available for version >= v0.0.15	specify access mode: use a service principal or pod identity or vm managed identity	"false"
vmmanagedidentityclientid	not required, available for version >= v0.0.15	If using a user assigned identity as the VM's managed identity, then specify the identity's client id. If empty, then defaults to use the system assigned identity on the VM	""
keyvaultname	yes	name of Key Vault instance	""
keyvaultobjectnames	yes	names of Key Vault objects to access	""
keyvaultobjectaliases	no	filenames to use when writing the objects	keyvaultobjectnames
keyvaultobjecttypes	yes	types of Key Vault objects: secret, key or cert	""
keyvaultobjectversions	no	versions of Key Vault objects, if not provided, will use latest	""
resourcegroup	required for version < v0.0.14	name of resource group containing Key Vault instance	""
subscriptionid	required for version < v0.0.14	name of subscription containing Key Vault instance	""
tenantid	yes	name of tenant containing Key Vault instance	""
cloudname	no	Name of the cloud environment, e.g. something like AzureChinaCloud, AzureGermanCloud. If not provided, the default public Azure cloud will be used	""
nmiport	not required, available for version >= v0.0.17	Port number of the NMI daemonset. If not provided, the default NMI port is used	"2579"

Multiple values in the keyvaultobjectnames, keyvaultobjecttypes and keyvaultobjectversions properties should be separated with semicolons (;).

Specify mount path of flexvolume to mount key vault objects

volumeMounts:
   - name: test
      mountPath: /kvmnt
      readOnly: true

Example of an nginx pod accessing a secret from a Key Vault instance:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-flex-kv
spec:
  containers:
  - name: nginx-flex-kv
    image: nginx
    volumeMounts:
    - name: test
      mountPath: /kvmnt
      readOnly: true
  volumes:
  - name: test
    flexVolume:
      driver: "azure/kv"
      secretRef:
        name: kvcreds                             # [OPTIONAL] not required if using Pod Identity
      options:
        usepodidentity: "false"                   # [OPTIONAL] if not provided, will default to "false"
        usevmmanagedidentity: "false"             # [OPTIONAL new in version >= v0.0.15] if not provided, will default to "false"
        vmmanagedidentityclientid: "clientid"     # [OPTIONAL new in version >= v0.0.15] use the client id to specify which user assigned managed identity to use, leave empty to use system assigned managed identity
        keyvaultname: "testkeyvault"              # [REQUIRED] the name of the KeyVault
        keyvaultobjectnames: "testsecret"         # [REQUIRED] list of KeyVault object names (semi-colon separated)
        keyvaultobjectaliases: "secret.json"      # [OPTIONAL] list of KeyVault object aliases
        keyvaultobjecttypes: secret               # [REQUIRED] list of KeyVault object types: secret, key, cert (semi-colon separated)
        keyvaultobjectversions: "testversion"     # [OPTIONAL] list of KeyVault object versions (semi-colon separated), will get latest if empty
        resourcegroup: "testresourcegroup"        # [REQUIRED for version < v0.0.14] the resource group of the KeyVault
        subscriptionid: "testsub"                 # [REQUIRED for version < v0.0.14] the subscription ID of the KeyVault
        tenantid: "testtenant"                    # [REQUIRED] the tenant ID of the KeyVault
        nmiport: "nmiportnumber"                  # [OPTIONAL new in version >= v0.0.17] port number of the NMI daemonset, will default to "2579"

Deploy your app

kubectl create -f deployment/nginx-flex-kv.yaml

Validate the pod has access to the secret from key vault:

kubectl exec -it nginx-flex-kv cat /kvmnt/testsecret
testvalue

OPTION 2: Pod identity

💡 The basic steps to configure AAD Pod Identity are reproduced here, but please refer to that project's README for more detail.

Install AAD Pod Identity

Run this command to create the aad-pod-identity deployment on an RBAC-enabled cluster:

kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml

Create an Azure Identity

Run this Azure CLI command, and take note of the clientId and id values it returns:
```
az identity create -g <resourcegroup> -n <name> -o json
```
Assign Cluster SPN Role

If the Service Principal used for the cluster was created separately (not automatically, as part of an AKS cluster's MC_ resource group), assign it the "Managed Identity Operator" role:
```
az role assignment create --role "Managed Identity Operator" --assignee <sp id> --scope <full id of the managed identity>
```

Assign Azure Identity Roles

Ensure that your Azure Identity has the role assignments required to see your Key Vault instance and to access its content. Run the following Azure CLI commands to assign these roles if needed:

# Assign Reader Role to new Identity for your Key Vault
az role assignment create --role Reader --assignee <principalid> --scope /subscriptions/<subscriptionid>/resourcegroups/<resourcegroup>/providers/Microsoft.KeyVault/vaults/<keyvaultname>

# set policy to access keys in your Key Vault
az keyvault set-policy -n $KV_NAME --key-permissions get --spn <YOUR AZURE USER IDENTITY CLIENT ID>
# set policy to access secrets in your Key Vault
az keyvault set-policy -n $KV_NAME --secret-permissions get --spn <YOUR AZURE USER IDENTITY CLIENT ID>
# set policy to access certs in your Key Vault
az keyvault set-policy -n $KV_NAME --certificate-permissions get --spn <YOUR AZURE USER IDENTITY CLIENT ID>

Install the Azure Identity

Save this Kubernetes manifest to a file named aadpodidentity.yaml:

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
  name: <a-idname>
spec:
  type: 0
  resourceID: /subscriptions/<subid>/resourcegroups/<resourcegroup>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<name>
  clientID: <clientId>

Replace the placeholders with your user identity values. Set type: 0 for user-assigned MSI.

Finally, save your changes to the file, then create the AzureIdentity resource in your cluster:

kubectl apply -f aadpodidentity.yaml

Install the Azure Identity Binding

Save this Kubernetes manifest to a file named aadpodidentitybinding.yaml:
```
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
  name: demo1-azure-identity-binding
spec:
  azureIdentity: <a-idname>
  selector: <label value to match>
```
Replace the placeholders with your values. Ensure that the AzureIdentity name matches the one in aadpodidentity.yaml.

Finally, save your changes to the file, then create the AzureIdentityBinding resource in your cluster:
```
kubectl apply -f aadpodidentitybinding.yaml
```
Update the Application Deployment

Your application manifest needs a couple of changes. Refer to the nginx-flex-kv-podid deployment as an example.

a. Include the aadpodidbinding label to match the selector value from the previous step:
```
metadata:
labels:
    aadpodidbinding: "NAME OF the AzureIdentityBinding SELECTOR"
```
b. Set usepodidentity to true:
```
usepodidentity: "true"
```

Deploy your app

kubectl create -f deployment/nginx-flex-kv-podidentity.yaml

Validate the pod can access the secret from Key Vault:

kubectl exec -it nginx-flex-kv-podid cat /kvmnt/testsecret
testvalue

NOTE: When using the Pod Identity option mode, there may be some delay in obtaining the objects from Key Vault. During pod creation time, AAD Pod Identity needs to create the AzureAssignedIdentity for the pod based on the AzureIdentity and AzureIdentityBinding and retrieve the token for Key Vault. It is possible for the pod volume mount to fail during this time. If it does, the kubelet will keep retrying until after the token retrieval is complete and the mount succeeds.

OPTION 3: VMSS User Assigned Managed Identity [New in version >= v0.0.15]

This option allows flexvol to use the user assigned managed identity on the k8s cluster VMSS directly.

Warning: As of today (2019/09), AKS does not preserve the user assigned identity on VMSS during upgrade. You will need to re-assign the managed identities to VMSS after an upgrade. The improved experience is planned.

Create Azure Managed Identity

az identity create -g <RESOURCE GROUP> -n <IDENTITY NAME>

Grant Azure Managed Identity KeyVault permissions

Ensure that your Azure Identity has the role assignments required to see your Key Vault instance and to access its content. Run the following Azure CLI commands to assign these roles if needed:

# set policy to access keys in your Key Vault
az keyvault set-policy -n $KV_NAME --key-permissions get --spn <YOUR AZURE MANAGED IDENTITY CLIENT ID>
# set policy to access secrets in your Key Vault
az keyvault set-policy -n $KV_NAME --secret-permissions get --spn <YOUR AZURE MANAGED IDENTITY CLIENT ID>
# set policy to access certs in your Key Vault
az keyvault set-policy -n $KV_NAME --certificate-permissions get --spn <YOUR AZURE MANAGED IDENTITY CLIENT ID>

Assign Azure Managed Identity to VMSS

az vmss identity assign -g <RESOURCE GROUP> -n <K8S-AGENT-POOL-VMSS> --identities <USER ASSIGNED IDENTITY RESOURCE ID>

Deploy your application. Specify usevmmanagedidentity to true and provide vmmanagedidentityclientid.

usevmmanagedidentity: "true"               # [OPTIONAL] if not provided, will default to "false"
vmmanagedidentityclientid: "clientid"      # [OPTIONAL] use the client id to specify which user assigned managed identity to use, leave empty to use system assigned managed identity

OPTION 4: VMSS System Assigned Managed Identity [New in version >= v0.0.15]

This option allows flexvol to use the system assigned managed identity on the k8s cluster VMSS directly.

Verify that the nodes have its own system assigned managed identity

az vmss identity show -g <resource group>  -n <vmss scalset name> -o yaml

The output should contain type: SystemAssigned.

Grant Azure Managed Identity KeyVault permissions

Ensure that the Azure system assigned Identity has the role assignments required to see your Key Vault instance and to access its content. Run the following Azure CLI commands to assign these roles if needed:

# set policy to access keys in your Key Vault
az keyvault set-policy -n $KV_NAME --key-permissions get --object-id <YOUR AZURE MANAGED IDENTITY OBJECT ID>
# set policy to access secrets in your Key Vault
az keyvault set-policy -n $KV_NAME --secret-permissions get --object-id <YOUR AZURE MANAGED IDENTITY OBJECT ID>
# set policy to access certs in your Key Vault
az keyvault set-policy -n $KV_NAME --certificate-permissions get --object-id <YOUR AZURE MANAGED IDENTITY OBJECT ID>

Deploy your application. Specify usevmmanagedidentity to true.

usevmmanagedidentity: "true"               # [OPTIONAL] if not provided, will default to "false"

Detailed use cases

Use Key Vault FlexVol to set up an SSL entrypoint with Istio
Use Key Vault FlexVol to set up an SSL entrypoint with Traefik

Design

To learn more about the design of Key Vault FlexVolume, see Concept.

About Key Vault

Key Vault FlexVolume interacts with Key Vault objects by using the Key Vault API.

Azure Key Vault has thorough documentation available to help clarify the difference between keys, secrets, and certificates.

About Certificates

It is important to understand how a certificate is structured in Key Vault.

As mentioned in Certificates are complex objects and Composition of a Certificate, Azure Key Vault (AKV) represents an X.509 certificate as three related resources:

an AKV-certificate
an AKV-key
an AKV-secret

All three will share the same name and the same version and can be fetched independently.

The AKV-certificate provides the public key and certificate metadata. Specifying cert in keyvaultobjecttypes will fetch the public key and certificate metadata.
The AKV-key provides the private key of the X.509 certificate. It can be useful for performing cryptographic operations such as signing if the corresponding certificate was marked as non-exportable. Specifying key in keyvaultobjecttypes will fetch the private key of the certificate if its policy allows for private key exporting.
The AKV-secret provides a way to export the full X.509 certificate, including its private key (if its policy allows for private key exporting). Specifying secret in keyvaultobjecttypes will fetch the base64-encoded certificate bundle.

Contributing

The Key Vault FlexVolume project welcomes contributions and suggestions. Please see CONTRIBUTING for details.

Code of conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information, see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.