TLS SDS/credentialName not working with Ingress Gateway

I have a workload running on a kubernetes cluster with Istio. I can use TLS with the one shared certificate, but I can’t get credentialName to work. Here are some relevant snippets from my Gateway

This works

  mode: SIMPLE
  privateKey: /etc/istio/ingressgateway-certs/tls.key
  serverCertificate: /etc/istio/ingressgateway-certs/tls.crt

This fails

  credentialName: api-termination-istio-tls-secret-autogen
  mode: SIMPLE

My Istio Ingress gateway is running in the namespace istio-system, and the secrets are in the same namespace (truncated output below).

[centos@k8s-master-0 ~]$ kubectl get pod -n istio-system
NAME                                      READY   STATUS      RESTARTS   AGE
istio-ingressgateway-7dd57888b8-68d4q     2/2     Running     0          16h

The above pod has the following description

[centos@k8s-master-0 ~]$ kubectl describe pod istio-ingressgateway-7dd57888b8-68d4q -n istio-system
Name:               istio-ingressgateway-7dd57888b8-68d4q
Namespace:          istio-system
Priority:           0
PriorityClassName:  <none>
Node:               high-memory-node-0/
Start Time:         Tue, 20 Aug 2019 20:28:40 +0000
Labels:             app=istio-ingressgateway
Annotations: false
Status:             Running
Controlled By:      ReplicaSet/istio-ingressgateway-7dd57888b8
    Container ID:  docker://7d4f6a94f7c2b24f3cb9734497e9ecb46085ee0c04d0f34eb1e0ef7761417386
    Image ID:      docker-pullable://istio/proxyv2@sha256:6db6c0ae3f65d02edc97e7900e816c1139b6269182ffb17fa3843f036d17b264
    Ports:         15020/TCP, 80/TCP, 443/TCP, 31400/TCP, 15029/TCP, 15030/TCP, 15031/TCP, 15032/TCP, 15443/TCP, 15090/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Tue, 20 Aug 2019 20:28:42 +0000
    Ready:          True
    Restart Count:  0
      cpu:     2
      memory:  1Gi
      cpu:      10m
      memory:   40Mi
    Readiness:  http-get http://:15020/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30
      NODE_NAME:                     (v1:spec.nodeName)
      POD_NAME:                     istio-ingressgateway-7dd57888b8-68d4q (
      POD_NAMESPACE:                istio-system (v1:metadata.namespace)
      INSTANCE_IP:                   (v1:status.podIP)
      HOST_IP:                       (v1:status.hostIP)
      ISTIO_META_POD_NAME:          istio-ingressgateway-7dd57888b8-68d4q (
      ISTIO_META_CONFIG_NAMESPACE:  istio-system (v1:metadata.namespace)
      ISTIO_META_ROUTER_MODE:       sni-dnat
      /etc/certs from istio-certs (ro)
      /etc/istio/ingressgateway-ca-certs from ingressgateway-ca-certs (ro)
      /etc/istio/ingressgateway-certs from ingressgateway-certs (ro)
      /var/run/secrets/ from istio-ingressgateway-service-account-token-rxrms (ro)
    Container ID:   docker://95f1a55d3fb83f0cfaba44951e2ef00ed479888cf29b0283c7c95b297abd917c
    Image ID:       docker-pullable://
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 20 Aug 2019 20:28:46 +0000
    Ready:          True
    Restart Count:  0
      cpu:     2
      memory:  1Gi
      cpu:     100m
      memory:  128Mi
      ENABLE_WORKLOAD_SDS:         false
      INGRESS_GATEWAY_NAMESPACE:   istio-system (v1:metadata.namespace)
      /var/run/ingress_gateway from ingressgatewaysdsudspath (rw)
      /var/run/secrets/ from istio-ingressgateway-service-account-token-rxrms (ro)
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio.istio-ingressgateway-service-account
    Optional:    true
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-certs
    Optional:    true
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-ca-certs
    Optional:    true
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-service-account-token-rxrms
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations: for 300s
        for 300s
Events:          <none>

And finally, a look at the secrets (truncated output):

[centos@k8s-master-0 ~]$ kubectl get secret -n istio-system
NAME                                                 TYPE                                  DATA   AGE
api-termination-istio-tls-secret-autogen                        2      22h
default-token-q4vwd                           3      13d
istio-ca-secret                                                  5      13d
istio-ingressgateway-certs                                      2      12d
istio-ingressgateway-service-account-token-rxrms   3      13d
istio.default                                               3      13d
istio.istio-ingressgateway-service-account                  3      13d

I created the secrets istio-ingressgateway-certs and api-termination-istio-tls-secret-autogen.

Log output from the SDS container is

[centos@k8s-master-0 ~]$ kubectl logs istio-ingressgateway-7dd57888b8-68d4q -n istio-system ingress-sds
2019-08-20T20:28:46.959055Z	info	ControlZ available at
2019-08-20T20:28:47.035608Z	warn	secretFetcherLog	failed load server cert/key pair from secret kiali: server cert or private key is empty
2019-08-20T20:28:47.064675Z	info	sdsServiceLog	SDS gRPC server for ingress gateway controller starts, listening on "/var/run/ingress_gateway/sds"

2019-08-20T20:28:47.064763Z	info	sdsServiceLog	Start SDS grpc server for ingress gateway proxy
2019-08-20T20:28:47.064893Z	info	citadel agent monitor has started.
2019-08-20T20:28:47.065525Z	info	monitor	Monitor server started.

The failed response looks like this

[centos@k8s-master-0 ~]$ curl -v
* About to connect() to port 443 (#0)
*   Trying
* Connected to ( port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS error -5938 (PR_END_OF_FILE_ERROR)
* Encountered end of file
* Closing connection 0
curl: (35) Encountered end of file

I’m not sure what the next steps are to troubleshoot this.

There are a few places to look.

First, port-forward the config_dump endpoint on the ingress-gateway with kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=istio-ingressgateway -o jsonpath='{.items[0]}') 15000:15000 & and then hit localhost:15000/config_dump.

Search the config_dump output for the secret name api-termination-istio-tls-secret-autogen. There should be a dynamic_active_listener configured with this sds_secret_config. For example, when I follow the Secure Ingress (SDS) tutorial, I end up with a dynamic_active_listener with the following sds config:

          "tls_certificate_sds_secret_configs": [
            "name": "httpbin-credential",
            "sds_config": {
             "api_config_source": {
              "api_type": "GRPC",
              "grpc_services": [
                "google_grpc": {
                 "target_uri": "unix:/var/run/ingress_gateway/sds",
                 "stat_prefix": "sdsstat"
             "initial_fetch_timeout": "0s"

If this is present, it means the ingress-gateway is configured to receive that secret from the embedded nodeagent, which is serving sds secrets at unix:/var/run/ingress_gateway/

In the current master branch, the nodeagent has a debug endpoint which shows all active sds connections, and the secrets being served. You could use the latest build, and port forward 8080:8080 on the ingress as well, and then hit http://localhost:8080/debug/sds/gateway. There should again be a secret with the same name. For example, when hitting this debug endpoint after running the secure ingress SDS task, I get

 	"clients": [
 			"connection_id": "router~",
 			"proxy": "router~",
 			"resource_name": "httpbin-credential",
 			"root_cert": "",
 			"created_time": "2019-08-21T17:07:55Z",
 			"expire_time": "2020-08-30T17:07:03Z"

Assuming everything seems in order from these previous steps, it could be a problem with the actual content of the secrets your supplying? Let me know if this helps at all.

When I view the contents of localhost:15000/config_dump, there is not a section for tls_certificate_sds_secret_configs. I do see that the directory is mounted and there’s something in that directory:

root@istio-ingressgateway-6845fd4977-hgthj:/var/run/ingress_gateway# ls -la
total 0
drwxrwxrwx. 2 root root 17 Aug 21 19:59 .
drwxr-xr-x. 1 root root 44 Aug 21 19:59 ..
srw-rw-rw-. 1 root root  0 Aug 21 19:59 sds

That doesn’t appear to be a file, so I’m not sure what it is. I don’t see any clients

root@istio-ingressgateway-6845fd4977-hgthj:/var/run/ingress_gateway# curl http://localhost:8080/debug/sds/gateway
 	"clients": []

Above I posted the output to get secrets. Here’s the content of the secret (obviously redacted).

[centos@k8s-master-0 ~]$ kubectl get secret api-termination-istio-tls-secret-autogen -o yaml -n istio-system
apiVersion: v1
  tls.crt: REDACTED
  tls.key: REDACTED
kind: Secret
  creationTimestamp: 2019-08-20T14:35:16Z
  name: api-termination-istio-tls-secret-autogen
  namespace: istio-system
  resourceVersion: "33667053"
  selfLink: /api/v1/namespaces/istio-system/secrets/api-termination-istio-tls-secret-autogen
  uid: ba7ce062-c357-11e9-b4a3-fa163ed0611a

When I look back at I notice that step #4 references " serverCertificate and privateKey should not be empty." Does that mean I should make the keys in my Secret serverCertificate and privateKey? When I try that, kubernetes complains that the keys need to be tls.crt and tls.key

I deleted the TLS Secret and created a new generic secret based on the tutorial, but I still don’t see the key/cert. The results are the same as above.

This is a Unix Domain Socket, which the ingress-gateway uses to retrieve secrets from the SDS service running on the embedded node agent. With SDS, the secrets never get mounted into the pod.

Have you tried making the keys of the secret key and cert exactly?

I made a mistake earlier, but the good news is that it’s working now. I combed through the deployment.yaml for the ingressgateway and found that it was missing the environment variable ISTIO_META_USER_SDS and one volumeMounts for the existing ingress-proxy container. I added those, but I forgot to change my Gateway back to referencing a credentialName instead of the file based approach.

I have confirmed that a generic Secret, as shown in the tutorial, and a tls Secret both work.

Thanks for all your help. It’s all working now.

Thank you for posting this issue and especially last comment. I was able to configure TLS for ingress gateway using credentialName/SDS. Someone who have problem with configuration TLS using credentialName need to check your env variable ISTIO_META_USER_SDS and volumeMounts for istio-proxy.

I am a little confuse about this. Also for me it works when using serverCertificate and privateKey but not when using credentialName.

Where should I add the env ISTIO_META_USER_SDS? On istio-ingressgateway workload? What should be the value of ISTIO_META_USER_SDS variable?
SDS_ENABLED which is also a variable of istio-ingressgateway by default has a value of false. Should I change it to true?

Thank you

1 Like

Even I have the same issue with Istio 1.6.8. Need to find a way to enable sds

1 Like

I appear to be having the same issue with 1.9.0. credentialName does not work

1 Like