We are using Azure Kubernetes Service for deploying our microservices. We are storing all our certificates and secrets in Azure Key Vault. We are also using Istio Service Mesh in our current architecture. We want to integrate between Istio Ingress Gateway to Azure Key Vault so that we could refer the 3rd party certificate stored in Azure Key Vault. We are also using CSI driver in our current architecture. Is that integration possible using CSI driver or any other mechanism? Can you please share any reference link?
Yes, it is possible using the Azure secrets store CSI driver:
You would install that driver, set your azure keyvault policies and then apply your SecretProviderClass to expose the keyvault cert as a secret in the cluster.
There is an additional step required (touched on here Using the Azure Key Vault Provider | Azure Key Vault Provider for Secrets Store CSI Driver) where you need to deploy a small workload which needs to stay running in the cluster which loads and mounts the csi driver secret as a volume (think busybox for a seperate deployment or just embed that mentioned section into an existing workload).
After that’s set up you can then inject your cert as secret for tls via the credentialName:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: my-gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: ingress-cert-tls
hosts:
- "test.mysite.com"
Thanks for the response.
We want to use the secrets using CSI driver into the Istio Ingress gateway instead of using at the individual PODs using POD identity. Can we do that without mounting the secret using the CSI driver? Otherwise, if mounting is required how can I refer the secret volume mount from the Istio Ingress Gateway? What will be the Istio Ingress Gateway yaml file structure with CSI driver secret volume mount?
You don’t have to use POD identity. You can use the user-assigned managed identity option (which I’m using):
You can grab the existing service principal from the AKS cluster and then use that in the SecretProviderClass i.e.
$aksMId = az aks show -n $AksClusterName -g $AksResourceGroupName --query "identityProfile.kubeletidentity.clientId" -o tsv
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: azure-kvname
namespace: istio-system
spec:
provider: azure
secretObjects: # secretObjects defines the desired state of synced K8s secret objects
- secretName: ingress-tls-csi
type: kubernetes.io/tls
data:
- objectName: "$KeyVaultCertName"
key: tls.key
- objectName: "$KeyVaultCertName"
key: tls.crt
parameters:
keyvaultName: "$KeyVaultName"
tenantId: "$tenantId" # the Azure Active Directory tenant ID of the KeyVault
usePodIdentity: "false" # [OPTIONAL for Azure] if not provided, will default to "false"
useVMManagedIdentity: "true" # [OPTIONAL available for version > 0.0.4] if not provided, will default to "false"
userAssignedIdentityID: "$aksMId" # [OPTIONAL available for version > 0.0.4] Using ManagedIdentity i.e: "identityProfile.kubeletidentity.clientId". Use the client id to specify which user assigned managed identity to use. If using a user assigned identity as the VM's managed identity, then specify the identity's client id. If empty, then defaults to use the system assigned identity on the VM
cloudName: "" # [OPTIONAL for Azure] if not provided, azure environment will default to AzurePublicCloud
cloudEnvFileName: "" # [OPTIONAL available for version > 0.0.7] use to define path to file for populating azure environment
objects: |
array:
- |
objectName: $KeyVaultCertName
objectType: secret
We are also using Azure Principal id as you are using. However, my question is on how to refer Volume Mount in Istio Ingress Gateway. Please see my question.
As mentioned in my first reply, the magic happens in a separate workload which loads in the azure secrets as a volume. I’m sure there’s a better way to do this but here’s a rough example that worked for me:
# Required in order to fetch and sync secrets from Azure Key Vault.
# NOTE: The k8 secrets will be deleted if the pods are removed.
apiVersion: apps/v1
kind: Deployment
metadata:
name: busybox-tls-loader
namespace: istio-system
labels:
app: busybox-tls-loader
spec:
replicas: 1
selector:
matchLabels:
app: busybox-tls-loader
template:
metadata:
labels:
app: busybox-tls-loader
annotations:
sidecar.istio.io/inject: "false"
spec:
terminationGracePeriodSeconds: 0
containers:
- name: busybox-tls-loader
image: k8s.gcr.io/e2e-test-images/busybox:1.29
resources:
requests:
memory: "100Mi"
cpu: "50m"
limits:
memory: "200Mi"
cpu: "200m"
command: ["/bin/sh", "-ec", "while :; do echo '.'; sleep 5 ; done"]
volumeMounts:
- name: secrets-store01-inline
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: secrets-store01-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "azure-kvname"
Hello basaksom,
I published here an example on how to use Azure Key Vault together with Istio:
These 2 files should address your specific questions:
The solution in the tutorial does not require to use Kubernetes Secret, the certificates are mounted directly into the istiod and istio-ingressgateway pods using the csi driver.
Please try the tutorial and give me feedback if this was useful to you.
Thank you
Hello zioproto,
I am using your example and have the following question.
What would be the procedure when I will have to renew the certificate? Does it automatically renew when I import the new certificate to the keystore or will I have to redeploy istio?
Thanks in advance
Hello sercavila,
thanks for your question.
You don’t need to redeploy Istio
Please check the documentation about “autorotation”:
Please let me know if you find everything you need in the documentation. Thank you