Integrate between Istio Ingress Gateway and Azure Key Vault to use stored certificates and secrets

We are using Azure Kubernetes Service for deploying our microservices. We are storing all our certificates and secrets in Azure Key Vault. We are also using Istio Service Mesh in our current architecture. We want to integrate between Istio Ingress Gateway to Azure Key Vault so that we could refer the 3rd party certificate stored in Azure Key Vault. We are also using CSI driver in our current architecture. Is that integration possible using CSI driver or any other mechanism? Can you please share any reference link?

Yes, it is possible using the Azure secrets store CSI driver:

You would install that driver, set your azure keyvault policies and then apply your SecretProviderClass to expose the keyvault cert as a secret in the cluster.

There is an additional step required (touched on here Using the Azure Key Vault Provider | Azure Key Vault Provider for Secrets Store CSI Driver) where you need to deploy a small workload which needs to stay running in the cluster which loads and mounts the csi driver secret as a volume (think busybox for a seperate deployment or just embed that mentioned section into an existing workload).

After that’s set up you can then inject your cert as secret for tls via the credentialName:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: my-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: ingress-cert-tls
    hosts:
    - "test.mysite.com"

Thanks for the response.
We want to use the secrets using CSI driver into the Istio Ingress gateway instead of using at the individual PODs using POD identity. Can we do that without mounting the secret using the CSI driver? Otherwise, if mounting is required how can I refer the secret volume mount from the Istio Ingress Gateway? What will be the Istio Ingress Gateway yaml file structure with CSI driver secret volume mount?

You don’t have to use POD identity. You can use the user-assigned managed identity option (which I’m using):

You can grab the existing service principal from the AKS cluster and then use that in the SecretProviderClass i.e.

$aksMId = az aks show -n $AksClusterName -g $AksResourceGroupName --query "identityProfile.kubeletidentity.clientId" -o tsv
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname
  namespace: istio-system
spec:
  provider: azure
  secretObjects:                       # secretObjects defines the desired state of synced K8s secret objects
  - secretName: ingress-tls-csi
    type: kubernetes.io/tls
    data:
    - objectName: "$KeyVaultCertName"
      key: tls.key
    - objectName: "$KeyVaultCertName"
      key: tls.crt
  parameters:
    keyvaultName: "$KeyVaultName"
    tenantId: "$tenantId"                   # the Azure Active Directory tenant ID of the KeyVault
    usePodIdentity: "false"                 # [OPTIONAL for Azure] if not provided, will default to "false"
    useVMManagedIdentity: "true"            # [OPTIONAL available for version > 0.0.4] if not provided, will default to "false"
    userAssignedIdentityID: "$aksMId"       # [OPTIONAL available for version > 0.0.4] Using ManagedIdentity i.e: "identityProfile.kubeletidentity.clientId". Use the client id to specify which user assigned managed identity to use. If using a user assigned identity as the VM's managed identity, then specify the identity's client id. If empty, then defaults to use the system assigned identity on the VM
    cloudName: ""                           # [OPTIONAL for Azure] if not provided, azure environment will default to AzurePublicCloud
    cloudEnvFileName: ""                    # [OPTIONAL available for version > 0.0.7] use to define path to file for populating azure environment
    objects: |
      array:
        - |
          objectName: $KeyVaultCertName
          objectType: secret

We are also using Azure Principal id as you are using. However, my question is on how to refer Volume Mount in Istio Ingress Gateway. Please see my question.

As mentioned in my first reply, the magic happens in a separate workload which loads in the azure secrets as a volume. I’m sure there’s a better way to do this but here’s a rough example that worked for me:

# Required in order to fetch and sync secrets from Azure Key Vault.
# NOTE: The k8 secrets will be deleted if the pods are removed.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-tls-loader
  namespace: istio-system  
  labels:
    app: busybox-tls-loader
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox-tls-loader
  template:
    metadata:
      labels:
        app: busybox-tls-loader
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      terminationGracePeriodSeconds: 0
      containers:
        - name: busybox-tls-loader
          image: k8s.gcr.io/e2e-test-images/busybox:1.29
          resources:
            requests:
              memory: "100Mi"
              cpu: "50m"
            limits:
              memory: "200Mi"
              cpu: "200m"
          command: ["/bin/sh", "-ec", "while :; do echo '.'; sleep 5 ; done"]
          volumeMounts:
          - name: secrets-store01-inline
            mountPath: "/mnt/secrets-store"
            readOnly: true
      volumes:
        - name: secrets-store01-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "azure-kvname"

Hello basaksom,

I published here an example on how to use Azure Key Vault together with Istio:

These 2 files should address your specific questions:

The solution in the tutorial does not require to use Kubernetes Secret, the certificates are mounted directly into the istiod and istio-ingressgateway pods using the csi driver.

Please try the tutorial and give me feedback if this was useful to you.

Thank you

Hello zioproto,

I am using your example and have the following question.

What would be the procedure when I will have to renew the certificate? Does it automatically renew when I import the new certificate to the keystore or will I have to redeploy istio?

Thanks in advance

Hello sercavila,
thanks for your question.
You don’t need to redeploy Istio

Please check the documentation about “autorotation”:

Please let me know if you find everything you need in the documentation. Thank you