Home

Awesome

Key Vault FlexVolume [DEPRECATED]

WARNING :warning: This solution has been deprecated. Continue to use this for kubernetes version 1.15. For 1.16+, please use Azure Key Vault Provider for Secret Store CSI Driver

This solution is no longer supported as 1.15 is out of support in AKS

Seamlessly integrate your key management systems with Kubernetes.

Secrets, keys, and certificates in a key management system become a volume accessible to pods. Once the volume is mounted, its data is available directly in the container filesystem for your application.

CircleCI

Contents

Getting started

Supported Providers

Installing Key Vault FlexVolume

OPTION 1: New AKS Engine cluster

AKS Engine creates customized Kubernetes clusters on Azure.

Follow the AKS Engine add-on documentation to create a new Kubernetes cluster with Key Vault FlexVolume already deployed.

OPTION 2: Existing AKS cluster

Azure Kubernetes Service (AKS) creates managed, supported Kubernetes clusters on Azure.

Deploy Key Vault FlexVolume to your AKS cluster with this command:

kubectl create -f https://raw.githubusercontent.com/Azure/kubernetes-keyvault-flexvol/master/deployment/kv-flexvol-installer.yaml

To validate Key Vault FlexVolume is running as expected, run the following command:

kubectl get pods -n kv

The output should show keyvault-flexvolume pods running on each agent node:

NAME                        READY     STATUS    RESTARTS   AGE
keyvault-flexvolume-f7bx8   1/1       Running   0          3m
keyvault-flexvolume-rcxbl   1/1       Running   0          3m
keyvault-flexvolume-z6jm6   1/1       Running   0          3m

If using keyvault-flexvolume in a cluster with pod security policy enabled, create the following policy that enables the spec required for keyvault-flexvolume to work -

kubectl apply -f https://raw.githubusercontent.com/Azure/kubernetes-keyvault-flexvol/master/deployment/kv-flexvol-psp.yaml

Using Key Vault FlexVolume

Key Vault FlexVolume offers four modes for accessing a Key Vault instance: Service Principal, Pod Identity, [VMSS User Assigned Managed Identity], [VMSS System Assigned Managed Identity].

OPTION 1: Service Principal

Add your service principal credentials as Kubernetes secrets accessible by the Key Vault FlexVolume driver.

kubectl create secret generic kvcreds --from-literal clientid=<CLIENTID> --from-literal clientsecret=<CLIENTSECRET> --type=azure/kv

Ensure this service principal has all the required permissions to access content in your Key Vault instance. If not, run the following Azure CLI commands:

# [Required for version < v0.0.13] Assign Reader Role to the service principal for your keyvault
az role assignment create --role Reader --assignee <principalid> --scope /subscriptions/<subscriptionid>/resourcegroups/<resourcegroup>/providers/Microsoft.KeyVault/vaults/<keyvaultname>

# Assign key vault permissions to your service principal
az keyvault set-policy -n $KV_NAME --key-permissions get --spn <YOUR SPN CLIENT ID>
az keyvault set-policy -n $KV_NAME --secret-permissions get --spn <YOUR SPN CLIENT ID>
az keyvault set-policy -n $KV_NAME --certificate-permissions get --spn <YOUR SPN CLIENT ID>

Fill in the missing pieces in this deployment for your own deployment. Make sure to:

  1. Reference the service principal Kubernetes secret created in the previous step

    secretRef:
      name: kvcreds
    
  2. Pass in properties for the Key Vault instance to the FlexVolume driver.

    NameRequiredDescriptionDefault Value
    usepodidentitynospecify access mode: use a service principal or pod identity or vm managed identity"false"
    usevmmanagedidentitynot required, available for version >= v0.0.15specify access mode: use a service principal or pod identity or vm managed identity"false"
    vmmanagedidentityclientidnot required, available for version >= v0.0.15If using a user assigned identity as the VM's managed identity, then specify the identity's client id. If empty, then defaults to use the system assigned identity on the VM""
    keyvaultnameyesname of Key Vault instance""
    keyvaultobjectnamesyesnames of Key Vault objects to access""
    keyvaultobjectaliasesnofilenames to use when writing the objectskeyvaultobjectnames
    keyvaultobjecttypesyestypes of Key Vault objects: secret, key or cert""
    keyvaultobjectversionsnoversions of Key Vault objects, if not provided, will use latest""
    resourcegrouprequired for version < v0.0.14name of resource group containing Key Vault instance""
    subscriptionidrequired for version < v0.0.14name of subscription containing Key Vault instance""
    tenantidyesname of tenant containing Key Vault instance""
    cloudnamenoName of the cloud environment, e.g. something like AzureChinaCloud, AzureGermanCloud. If not provided, the default public Azure cloud will be used""
    nmiportnot required, available for version >= v0.0.17Port number of the NMI daemonset. If not provided, the default NMI port is used"2579"

    Multiple values in the keyvaultobjectnames, keyvaultobjecttypes and keyvaultobjectversions properties should be separated with semicolons (;).

  3. Specify mount path of flexvolume to mount key vault objects

    volumeMounts:
       - name: test
          mountPath: /kvmnt
          readOnly: true
    

    Example of an nginx pod accessing a secret from a Key Vault instance:

    apiVersion: v1
    kind: Pod
    metadata:
      name: nginx-flex-kv
    spec:
      containers:
      - name: nginx-flex-kv
        image: nginx
        volumeMounts:
        - name: test
          mountPath: /kvmnt
          readOnly: true
      volumes:
      - name: test
        flexVolume:
          driver: "azure/kv"
          secretRef:
            name: kvcreds                             # [OPTIONAL] not required if using Pod Identity
          options:
            usepodidentity: "false"                   # [OPTIONAL] if not provided, will default to "false"
            usevmmanagedidentity: "false"             # [OPTIONAL new in version >= v0.0.15] if not provided, will default to "false"
            vmmanagedidentityclientid: "clientid"     # [OPTIONAL new in version >= v0.0.15] use the client id to specify which user assigned managed identity to use, leave empty to use system assigned managed identity
            keyvaultname: "testkeyvault"              # [REQUIRED] the name of the KeyVault
            keyvaultobjectnames: "testsecret"         # [REQUIRED] list of KeyVault object names (semi-colon separated)
            keyvaultobjectaliases: "secret.json"      # [OPTIONAL] list of KeyVault object aliases
            keyvaultobjecttypes: secret               # [REQUIRED] list of KeyVault object types: secret, key, cert (semi-colon separated)
            keyvaultobjectversions: "testversion"     # [OPTIONAL] list of KeyVault object versions (semi-colon separated), will get latest if empty
            resourcegroup: "testresourcegroup"        # [REQUIRED for version < v0.0.14] the resource group of the KeyVault
            subscriptionid: "testsub"                 # [REQUIRED for version < v0.0.14] the subscription ID of the KeyVault
            tenantid: "testtenant"                    # [REQUIRED] the tenant ID of the KeyVault
            nmiport: "nmiportnumber"                  # [OPTIONAL new in version >= v0.0.17] port number of the NMI daemonset, will default to "2579"
    

    Deploy your app

    kubectl create -f deployment/nginx-flex-kv.yaml
    

    Validate the pod has access to the secret from key vault:

    kubectl exec -it nginx-flex-kv cat /kvmnt/testsecret
    testvalue
    

OPTION 2: Pod identity

💡 The basic steps to configure AAD Pod Identity are reproduced here, but please refer to that project's README for more detail.

  1. Install AAD Pod Identity

    Run this command to create the aad-pod-identity deployment on an RBAC-enabled cluster:

    kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml
    
    
  2. Create an Azure Identity

    Run this Azure CLI command, and take note of the clientId and id values it returns:

    az identity create -g <resourcegroup> -n <name> -o json
    
  3. Assign Cluster SPN Role

    If the Service Principal used for the cluster was created separately (not automatically, as part of an AKS cluster's MC_ resource group), assign it the "Managed Identity Operator" role:

    az role assignment create --role "Managed Identity Operator" --assignee <sp id> --scope <full id of the managed identity>
    
  4. Assign Azure Identity Roles

    Ensure that your Azure Identity has the role assignments required to see your Key Vault instance and to access its content. Run the following Azure CLI commands to assign these roles if needed:

    # Assign Reader Role to new Identity for your Key Vault
    az role assignment create --role Reader --assignee <principalid> --scope /subscriptions/<subscriptionid>/resourcegroups/<resourcegroup>/providers/Microsoft.KeyVault/vaults/<keyvaultname>
    
    # set policy to access keys in your Key Vault
    az keyvault set-policy -n $KV_NAME --key-permissions get --spn <YOUR AZURE USER IDENTITY CLIENT ID>
    # set policy to access secrets in your Key Vault
    az keyvault set-policy -n $KV_NAME --secret-permissions get --spn <YOUR AZURE USER IDENTITY CLIENT ID>
    # set policy to access certs in your Key Vault
    az keyvault set-policy -n $KV_NAME --certificate-permissions get --spn <YOUR AZURE USER IDENTITY CLIENT ID>
    
  5. Install the Azure Identity

    Save this Kubernetes manifest to a file named aadpodidentity.yaml:

    apiVersion: "aadpodidentity.k8s.io/v1"
    kind: AzureIdentity
    metadata:
      name: <a-idname>
    spec:
      type: 0
      resourceID: /subscriptions/<subid>/resourcegroups/<resourcegroup>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<name>
      clientID: <clientId>
    

    Replace the placeholders with your user identity values. Set type: 0 for user-assigned MSI.

    Finally, save your changes to the file, then create the AzureIdentity resource in your cluster:

    kubectl apply -f aadpodidentity.yaml
    
  6. Install the Azure Identity Binding

    Save this Kubernetes manifest to a file named aadpodidentitybinding.yaml:

    apiVersion: "aadpodidentity.k8s.io/v1"
    kind: AzureIdentityBinding
    metadata:
      name: demo1-azure-identity-binding
    spec:
      azureIdentity: <a-idname>
      selector: <label value to match>
    

    Replace the placeholders with your values. Ensure that the AzureIdentity name matches the one in aadpodidentity.yaml.

    Finally, save your changes to the file, then create the AzureIdentityBinding resource in your cluster:

    kubectl apply -f aadpodidentitybinding.yaml
    
  7. Update the Application Deployment

    Your application manifest needs a couple of changes. Refer to the nginx-flex-kv-podid deployment as an example.

    a. Include the aadpodidbinding label to match the selector value from the previous step:

    metadata:
    labels:
        aadpodidbinding: "NAME OF the AzureIdentityBinding SELECTOR"
    

    b. Set usepodidentity to true:

    usepodidentity: "true"
    
  8. Deploy your app

    kubectl create -f deployment/nginx-flex-kv-podidentity.yaml
    
  9. Validate the pod can access the secret from Key Vault:

    kubectl exec -it nginx-flex-kv-podid cat /kvmnt/testsecret
    testvalue
    

NOTE: When using the Pod Identity option mode, there may be some delay in obtaining the objects from Key Vault. During pod creation time, AAD Pod Identity needs to create the AzureAssignedIdentity for the pod based on the AzureIdentity and AzureIdentityBinding and retrieve the token for Key Vault. It is possible for the pod volume mount to fail during this time. If it does, the kubelet will keep retrying until after the token retrieval is complete and the mount succeeds.

OPTION 3: VMSS User Assigned Managed Identity [New in version >= v0.0.15]

This option allows flexvol to use the user assigned managed identity on the k8s cluster VMSS directly.

Warning: As of today (2019/09), AKS does not preserve the user assigned identity on VMSS during upgrade. You will need to re-assign the managed identities to VMSS after an upgrade. The improved experience is planned.

  1. Create Azure Managed Identity
az identity create -g <RESOURCE GROUP> -n <IDENTITY NAME>
  1. Grant Azure Managed Identity KeyVault permissions

    Ensure that your Azure Identity has the role assignments required to see your Key Vault instance and to access its content. Run the following Azure CLI commands to assign these roles if needed:

    # set policy to access keys in your Key Vault
    az keyvault set-policy -n $KV_NAME --key-permissions get --spn <YOUR AZURE MANAGED IDENTITY CLIENT ID>
    # set policy to access secrets in your Key Vault
    az keyvault set-policy -n $KV_NAME --secret-permissions get --spn <YOUR AZURE MANAGED IDENTITY CLIENT ID>
    # set policy to access certs in your Key Vault
    az keyvault set-policy -n $KV_NAME --certificate-permissions get --spn <YOUR AZURE MANAGED IDENTITY CLIENT ID>
    
  2. Assign Azure Managed Identity to VMSS

az vmss identity assign -g <RESOURCE GROUP> -n <K8S-AGENT-POOL-VMSS> --identities <USER ASSIGNED IDENTITY RESOURCE ID>
  1. Deploy your application. Specify usevmmanagedidentity to true and provide vmmanagedidentityclientid.
usevmmanagedidentity: "true"               # [OPTIONAL] if not provided, will default to "false"
vmmanagedidentityclientid: "clientid"      # [OPTIONAL] use the client id to specify which user assigned managed identity to use, leave empty to use system assigned managed identity

OPTION 4: VMSS System Assigned Managed Identity [New in version >= v0.0.15]

This option allows flexvol to use the system assigned managed identity on the k8s cluster VMSS directly.

  1. Verify that the nodes have its own system assigned managed identity
az vmss identity show -g <resource group>  -n <vmss scalset name> -o yaml

The output should contain type: SystemAssigned.

  1. Grant Azure Managed Identity KeyVault permissions

    Ensure that the Azure system assigned Identity has the role assignments required to see your Key Vault instance and to access its content. Run the following Azure CLI commands to assign these roles if needed:

    # set policy to access keys in your Key Vault
    az keyvault set-policy -n $KV_NAME --key-permissions get --object-id <YOUR AZURE MANAGED IDENTITY OBJECT ID>
    # set policy to access secrets in your Key Vault
    az keyvault set-policy -n $KV_NAME --secret-permissions get --object-id <YOUR AZURE MANAGED IDENTITY OBJECT ID>
    # set policy to access certs in your Key Vault
    az keyvault set-policy -n $KV_NAME --certificate-permissions get --object-id <YOUR AZURE MANAGED IDENTITY OBJECT ID>
    
  2. Deploy your application. Specify usevmmanagedidentity to true.

usevmmanagedidentity: "true"               # [OPTIONAL] if not provided, will default to "false"

Detailed use cases

Design

To learn more about the design of Key Vault FlexVolume, see Concept.

About Key Vault

Key Vault FlexVolume interacts with Key Vault objects by using the Key Vault API.

Azure Key Vault has thorough documentation available to help clarify the difference between keys, secrets, and certificates.

About Certificates

It is important to understand how a certificate is structured in Key Vault.

As mentioned in Certificates are complex objects and Composition of a Certificate, Azure Key Vault (AKV) represents an X.509 certificate as three related resources:

All three will share the same name and the same version and can be fetched independently.

Contributing

The Key Vault FlexVolume project welcomes contributions and suggestions. Please see CONTRIBUTING for details.

Code of conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information, see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.