We are using Azure Kubernetes Service for deploying our microservices. We are storing all our certificates and secrets in Azure Key Vault. We are also using Istio Service Mesh in our current architecture. We want to integrate between Istio Ingress Gateway to Azure Key Vault so that we could refer the 3rd party certificate stored in Azure Key Vault. We are also using CSI driver in our current architecture. Is that integration possible using CSI driver or any other mechanism? Can you please share any reference link?
Yes, it is possible using the Azure secrets store CSI driver:
You would install that driver, set your azure keyvault policies and then apply your SecretProviderClass to expose the keyvault cert as a secret in the cluster.
There is an additional step required (touched on here Using the Azure Key Vault Provider | Azure Key Vault Provider for Secrets Store CSI Driver) where you need to deploy a small workload which needs to stay running in the cluster which loads and mounts the csi driver secret as a volume (think busybox for a seperate deployment or just embed that mentioned section into an existing workload).
After that’s set up you can then inject your cert as secret for tls via the credentialName:
apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: my-gateway spec: selector: istio: ingressgateway servers: - port: number: 443 name: https protocol: HTTPS tls: mode: SIMPLE credentialName: ingress-cert-tls hosts: - "test.mysite.com"
Thanks for the response.
We want to use the secrets using CSI driver into the Istio Ingress gateway instead of using at the individual PODs using POD identity. Can we do that without mounting the secret using the CSI driver? Otherwise, if mounting is required how can I refer the secret volume mount from the Istio Ingress Gateway? What will be the Istio Ingress Gateway yaml file structure with CSI driver secret volume mount?
You don’t have to use POD identity. You can use the user-assigned managed identity option (which I’m using):
You can grab the existing service principal from the AKS cluster and then use that in the SecretProviderClass i.e.
$aksMId = az aks show -n $AksClusterName -g $AksResourceGroupName --query "identityProfile.kubeletidentity.clientId" -o tsv
apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: azure-kvname namespace: istio-system spec: provider: azure secretObjects: # secretObjects defines the desired state of synced K8s secret objects - secretName: ingress-tls-csi type: kubernetes.io/tls data: - objectName: "$KeyVaultCertName" key: tls.key - objectName: "$KeyVaultCertName" key: tls.crt parameters: keyvaultName: "$KeyVaultName" tenantId: "$tenantId" # the Azure Active Directory tenant ID of the KeyVault usePodIdentity: "false" # [OPTIONAL for Azure] if not provided, will default to "false" useVMManagedIdentity: "true" # [OPTIONAL available for version > 0.0.4] if not provided, will default to "false" userAssignedIdentityID: "$aksMId" # [OPTIONAL available for version > 0.0.4] Using ManagedIdentity i.e: "identityProfile.kubeletidentity.clientId". Use the client id to specify which user assigned managed identity to use. If using a user assigned identity as the VM's managed identity, then specify the identity's client id. If empty, then defaults to use the system assigned identity on the VM cloudName: "" # [OPTIONAL for Azure] if not provided, azure environment will default to AzurePublicCloud cloudEnvFileName: "" # [OPTIONAL available for version > 0.0.7] use to define path to file for populating azure environment objects: | array: - | objectName: $KeyVaultCertName objectType: secret
We are also using Azure Principal id as you are using. However, my question is on how to refer Volume Mount in Istio Ingress Gateway. Please see my question.
As mentioned in my first reply, the magic happens in a separate workload which loads in the azure secrets as a volume. I’m sure there’s a better way to do this but here’s a rough example that worked for me:
# Required in order to fetch and sync secrets from Azure Key Vault. # NOTE: The k8 secrets will be deleted if the pods are removed. apiVersion: apps/v1 kind: Deployment metadata: name: busybox-tls-loader namespace: istio-system labels: app: busybox-tls-loader spec: replicas: 1 selector: matchLabels: app: busybox-tls-loader template: metadata: labels: app: busybox-tls-loader annotations: sidecar.istio.io/inject: "false" spec: terminationGracePeriodSeconds: 0 containers: - name: busybox-tls-loader image: k8s.gcr.io/e2e-test-images/busybox:1.29 resources: requests: memory: "100Mi" cpu: "50m" limits: memory: "200Mi" cpu: "200m" command: ["/bin/sh", "-ec", "while :; do echo '.'; sleep 5 ; done"] volumeMounts: - name: secrets-store01-inline mountPath: "/mnt/secrets-store" readOnly: true volumes: - name: secrets-store01-inline csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "azure-kvname"
I published here an example on how to use Azure Key Vault together with Istio:
These 2 files should address your specific questions:
The solution in the tutorial does not require to use Kubernetes Secret, the certificates are mounted directly into the istiod and istio-ingressgateway pods using the csi driver.
Please try the tutorial and give me feedback if this was useful to you.