TLS SDS/credentialName not working with Ingress Gateway

Security
1

I have a workload running on a kubernetes cluster with Istio. I can use TLS with the one shared certificate, but I can’t get credentialName to work. Here are some relevant snippets from my Gateway

This works

tls:
  mode: SIMPLE
  privateKey: /etc/istio/ingressgateway-certs/tls.key
  serverCertificate: /etc/istio/ingressgateway-certs/tls.crt

This fails

tls:
  credentialName: api-termination-istio-tls-secret-autogen
  mode: SIMPLE

My Istio Ingress gateway is running in the namespace istio-system, and the secrets are in the same namespace (truncated output below).

[centos@k8s-master-0 ~]$ kubectl get pod -n istio-system
NAME                                      READY   STATUS      RESTARTS   AGE
...
istio-ingressgateway-7dd57888b8-68d4q     2/2     Running     0          16h

The above pod has the following description

[centos@k8s-master-0 ~]$ kubectl describe pod istio-ingressgateway-7dd57888b8-68d4q -n istio-system
Name:               istio-ingressgateway-7dd57888b8-68d4q
Namespace:          istio-system
Priority:           0
PriorityClassName:  <none>
Node:               high-memory-node-0/10.0.40.13
Start Time:         Tue, 20 Aug 2019 20:28:40 +0000
Labels:             app=istio-ingressgateway
                    chart=gateways
                    heritage=Tiller
                    istio=ingressgateway
                    pod-template-hash=7dd57888b8
                    release=istio
Annotations:        sidecar.istio.io/inject: false
Status:             Running
IP:                 10.233.72.210
Controlled By:      ReplicaSet/istio-ingressgateway-7dd57888b8
Containers:
  istio-proxy:
    Container ID:  docker://7d4f6a94f7c2b24f3cb9734497e9ecb46085ee0c04d0f34eb1e0ef7761417386
    Image:         docker.io/istio/proxyv2:1.2.3
    Image ID:      docker-pullable://istio/proxyv2@sha256:6db6c0ae3f65d02edc97e7900e816c1139b6269182ffb17fa3843f036d17b264
    Ports:         15020/TCP, 80/TCP, 443/TCP, 31400/TCP, 15029/TCP, 15030/TCP, 15031/TCP, 15032/TCP, 15443/TCP, 15090/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      proxy
      router
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --log_output_level=default:info
      --drainDuration
      45s
      --parentShutdownDuration
      1m0s
      --connectTimeout
      10s
      --serviceCluster
      istio-ingressgateway
      --zipkinAddress
      zipkin:9411
      --proxyAdminPort
      15000
      --statusPort
      15020
      --controlPlaneAuthPolicy
      NONE
      --discoveryAddress
      istio-pilot:15010
    State:          Running
      Started:      Tue, 20 Aug 2019 20:28:42 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:      10m
      memory:   40Mi
    Readiness:  http-get http://:15020/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30
    Environment:
      NODE_NAME:                     (v1:spec.nodeName)
      POD_NAME:                     istio-ingressgateway-7dd57888b8-68d4q (v1:metadata.name)
      POD_NAMESPACE:                istio-system (v1:metadata.namespace)
      INSTANCE_IP:                   (v1:status.podIP)
      HOST_IP:                       (v1:status.hostIP)
      ISTIO_META_POD_NAME:          istio-ingressgateway-7dd57888b8-68d4q (v1:metadata.name)
      ISTIO_META_CONFIG_NAMESPACE:  istio-system (v1:metadata.namespace)
      ISTIO_META_ROUTER_MODE:       sni-dnat
    Mounts:
      /etc/certs from istio-certs (ro)
      /etc/istio/ingressgateway-ca-certs from ingressgateway-ca-certs (ro)
      /etc/istio/ingressgateway-certs from ingressgateway-certs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from istio-ingressgateway-service-account-token-rxrms (ro)
  ingress-sds:
    Container ID:   docker://95f1a55d3fb83f0cfaba44951e2ef00ed479888cf29b0283c7c95b297abd917c
    Image:          gcr.io/istio-release/node-agent-k8s:master-latest-daily
    Image ID:       docker-pullable://gcr.io/istio-release/node-agent-k8s@sha256:a290b89c16ee17f233405a64618d94af425bdce70d468960dcc10d1ea1f589dc
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 20 Aug 2019 20:28:46 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:     100m
      memory:  128Mi
    Environment:
      ENABLE_WORKLOAD_SDS:         false
      ENABLE_INGRESS_GATEWAY_SDS:  true
      INGRESS_GATEWAY_NAMESPACE:   istio-system (v1:metadata.namespace)
    Mounts:
      /var/run/ingress_gateway from ingressgatewaysdsudspath (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from istio-ingressgateway-service-account-token-rxrms (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  ingressgatewaysdsudspath:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  istio-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio.istio-ingressgateway-service-account
    Optional:    true
  ingressgateway-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-certs
    Optional:    true
  ingressgateway-ca-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-ca-certs
    Optional:    true
  istio-ingressgateway-service-account-token-rxrms:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-service-account-token-rxrms
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

And finally, a look at the secrets (truncated output):

[centos@k8s-master-0 ~]$ kubectl get secret -n istio-system
NAME                                                 TYPE                                  DATA   AGE
api-termination-istio-tls-secret-autogen             kubernetes.io/tls                     2      22h
default-token-q4vwd                                  kubernetes.io/service-account-token   3      13d
istio-ca-secret                                      istio.io/ca-root                      5      13d
istio-ingressgateway-certs                           kubernetes.io/tls                     2      12d
istio-ingressgateway-service-account-token-rxrms     kubernetes.io/service-account-token   3      13d
istio.default                                        istio.io/key-and-cert                 3      13d
istio.istio-ingressgateway-service-account           istio.io/key-and-cert                 3      13d

I created the secrets istio-ingressgateway-certs and api-termination-istio-tls-secret-autogen.

Log output from the SDS container is

[centos@k8s-master-0 ~]$ kubectl logs istio-ingressgateway-7dd57888b8-68d4q -n istio-system ingress-sds
2019-08-20T20:28:46.959055Z	info	ControlZ available at 127.0.0.1:9876
2019-08-20T20:28:47.035608Z	warn	secretFetcherLog	failed load server cert/key pair from secret kiali: server cert or private key is empty
2019-08-20T20:28:47.064675Z	info	sdsServiceLog	SDS gRPC server for ingress gateway controller starts, listening on "/var/run/ingress_gateway/sds"

2019-08-20T20:28:47.064763Z	info	sdsServiceLog	Start SDS grpc server for ingress gateway proxy
2019-08-20T20:28:47.064893Z	info	citadel agent monitor has started.
2019-08-20T20:28:47.065525Z	info	monitor	Monitor server started.

The failed response looks like this

[centos@k8s-master-0 ~]$ curl -v https://api-termination-istio.dc.k8s.com/loginproxy
* About to connect() to api-termination-istio.dc.k8s.com port 443 (#0)
*   Trying 192.168.24.239...
* Connected to api-termination-istio.dc.k8s.com (192.168.24.239) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS error -5938 (PR_END_OF_FILE_ERROR)
* Encountered end of file
* Closing connection 0
curl: (35) Encountered end of file

I’m not sure what the next steps are to troubleshoot this.

2

There are a few places to look.

First, port-forward the config_dump endpoint on the ingress-gateway with kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=istio-ingressgateway -o jsonpath='{.items[0].metadata.name}') 15000:15000 & and then hit localhost:15000/config_dump.

Search the config_dump output for the secret name api-termination-istio-tls-secret-autogen. There should be a dynamic_active_listener configured with this sds_secret_config. For example, when I follow the Secure Ingress (SDS) tutorial, I end up with a dynamic_active_listener with the following sds config:

          "tls_certificate_sds_secret_configs": [
           {
            "name": "httpbin-credential",
            "sds_config": {
             "api_config_source": {
              "api_type": "GRPC",
              "grpc_services": [
               {
                "google_grpc": {
                 "target_uri": "unix:/var/run/ingress_gateway/sds",
                 "stat_prefix": "sdsstat"
                }
               }
              ]
             },
             "initial_fetch_timeout": "0s"
            }
           }
          ]

If this is present, it means the ingress-gateway is configured to receive that secret from the embedded nodeagent, which is serving sds secrets at unix:/var/run/ingress_gateway/

In the current master branch, the nodeagent has a debug endpoint which shows all active sds connections, and the secrets being served. You could use the latest build, and port forward 8080:8080 on the ingress as well, and then hit http://localhost:8080/debug/sds/gateway. There should again be a secret with the same name. For example, when hitting this debug endpoint after running the secure ingress SDS task, I get

{
 	"clients": [
 		{
 			"connection_id": "router~10.4.3.252~istio-ingressgateway-699f69df6b-vv2f2.istio-system~istio-system.svc.cluster.local-1",
 			"proxy": "router~10.4.3.252~istio-ingressgateway-699f69df6b-vv2f2.istio-system~istio-system.svc.cluster.local",
 			"resource_name": "httpbin-credential",
 			"certificate_chain": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZVekNDQXp1Z0F3SUJBZ0lERUFJU01BMEdDU3FHU0liM0RRRUJDd1VBTUVveEN6QUpCZ05WQkFZVEFsVlQKTVE4d0RRWURWUVFJREFaRVpXNXBZV3d4RERBS0JnTlZCQW9NQTBScGN6RWNNQm9HQTFVRUF3d1RhSFIwY0dKcApiaTVsZUdGdGNHeGxMbU52YlRBZUZ3MHhPVEE0TWpFeE56QTNNRE5hRncweU1EQTRNekF4TnpBM01ETmFNR0F4CkN6QUpCZ05WQkFZVEFsVlRNUTh3RFFZRFZRUUlEQVpFWlc1cFlXd3hGREFTQmdOVkJBY01DMU53Y21sdVoyWnAKWld4a01Rd3dDZ1lEVlFRS0RBTkVhWE14SERBYUJnTlZCQU1NRTJoMGRIQmlhVzR1WlhoaGJYQnNaUzVqYjIwdwpnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFESDV0UERBcXZVa0N6TEVLUU5WRmJzCmwrMzRRMU1VUnNoMjB4cG16WGZZRk91K1R1N252USszRWhtT2swcS9YMEdyZHoyK09PMm9wYllSYy9jSE0rOE4KVlhMZnltQkJWZHZvZ0dsbHBFQUZ4NWVoYitDSFVwRDZnWUxwVjRKWGxwZXMzZk5YeXNIZm54UjRoblN3a25xWAppaCtqOVoxbHZIM1BwWFdvMzcvbEtPdmdXK3RpUER5VlNheFhmc2pEZWZWZGNIeVVKamt6a091cjFuekd3clR1ClM0RVdsTW1jaThmUmN3bS9qWjFNcTc4N2hsMXRGYnVSOVMrK0hrOU95TEZZQkd3czRwZ21MQkJjdkFpeUFVcXcKZi9mRFFJOUpVckRUc1dIcjBWeDdpcmdnQzJHWUhORmoycnQ4Vm9aU2FkL2JtQkFCTk9McWxMWlJZdDNuZ0syUApBZ01CQUFHamdnRXFNSUlCSmpBSkJnTlZIUk1FQWpBQU1CRUdDV0NHU0FHRytFSUJBUVFFQXdJR1FEQXpCZ2xnCmhrZ0JodmhDQVEwRUpoWWtUM0JsYmxOVFRDQkhaVzVsY21GMFpXUWdVMlZ5ZG1WeUlFTmxjblJwWm1sallYUmwKTUIwR0ExVWREZ1FXQkJTbmlLaFNUYnVFUWplSVAwWHRrQ3lCQzB0Q1N6Q0JqQVlEVlIwakJJR0VNSUdCZ0JRWgp6cndZdVEzdENUY0xCZTljNE1jQTB6aExnS0ZrcEdJd1lERUxNQWtHQTFVRUJoTUNWVk14RHpBTkJnTlZCQWdNCkJrUmxibWxoYkRFVU1CSUdBMVVFQnd3TFUzQnlhVzVuWm1sbGJHUXhEREFLQmdOVkJBb01BMFJwY3pFY01Cb0cKQTFVRUF3d1RhSFIwY0dKcGJpNWxlR0Z0Y0d4bExtTnZiWUlERUFJU01BNEdBMVVkRHdFQi93UUVBd0lGb0RBVApCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBVEFOQmdrcWhraUc5dzBCQVFzRkFBT0NBZ0VBSk9KQUw4OWYrUUlzCllhZnQwblkzTjdYai9xdlRsV1FkaU0vUCtKMXRMbjZGQk5Pc29UdE5NZGpSaGo2aTcxN1g5SFEvMXNHYjBrU2IKN0JMMVdKKzFlZFdDV2g1a1NnemJIL1FYaGlHcUoxcnU5UTk0KzhKcWJ2U3VOb2ltdCtPVStHNXRMUE5NeGIySQpVSlJxVlUzYURjbVJzaGhCSEhVZm11aG9DRE5PUjFWb09oYmZXVXlZdDVaUTQrOVlDR0c3RkFjMHFjWm9JbHEyCkVUM2pueG5wejc4L0NnQm16aXltWEI1dVJmSng1SXI4MGdvQXFIVUlUUlpFbVFmRlRwbmxuNmRYL1M2KzNLejIKY0lTcFg5c3pLUGgycm5HRHBta1F2M0RrL2QxTlJRYVVNck4yMUhGaTRKdjFoRjJtUHlZQzBsbm1HbzArTUVZQwp1bFlXcmloaFpDMzNMeTEvdkpCZkIxR1E0d09FK1dzMzBwT1kxd2wwRE5VcXJ0RkpSTDVrZW9tMVRtKzcrYzd2CldFOFRIUGpVdFFjRmFjenAzQ2M3Vzd6YlJJRk9LWHhFT3IzTkdGYVIxWmRxZk55eFZ6TmRuK3JhdGtnTWthNlgKOG5ML1FKSVlIQ0tjbWM0bXFtRHZ6dlhBUEdqZFlqb1JvOGpRbU5WKzV1VHpheUhKVTBzMldmNVlGU1VZQU5wSQpMazFNZy9IdnpVaTRrcFY0ckhKV3IzVDUzS3NTSzkyZlBubEhVZ0dCTzJQTDJHWER1a3pWWXpBMW9UWGM0UGRUCnJPOGg0d1BRSndSSUhZQ2YwelBGcUMrZ1BXb0tyQkxlbXN2VXM5clAyYnlyRDBaOVVUU0E4cWZQeitNbmhVamcKd3pnS3gwc3RFR3pKQm5scURER0Q2anJVU1FYcG5Rbz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=",
 			"root_cert": "",
 			"created_time": "2019-08-21T17:07:55Z",
 			"expire_time": "2020-08-30T17:07:03Z"
 		}
 	]
 }

Assuming everything seems in order from these previous steps, it could be a problem with the actual content of the secrets your supplying? Let me know if this helps at all.

3

When I view the contents of localhost:15000/config_dump, there is not a section for tls_certificate_sds_secret_configs. I do see that the directory is mounted and there’s something in that directory:

root@istio-ingressgateway-6845fd4977-hgthj:/var/run/ingress_gateway# ls -la
total 0
drwxrwxrwx. 2 root root 17 Aug 21 19:59 .
drwxr-xr-x. 1 root root 44 Aug 21 19:59 ..
srw-rw-rw-. 1 root root  0 Aug 21 19:59 sds

That doesn’t appear to be a file, so I’m not sure what it is. I don’t see any clients

root@istio-ingressgateway-6845fd4977-hgthj:/var/run/ingress_gateway# curl http://localhost:8080/debug/sds/gateway
{
 	"clients": []
}

Above I posted the output to get secrets. Here’s the content of the secret (obviously redacted).

[centos@k8s-master-0 ~]$ kubectl get secret api-termination-istio-tls-secret-autogen -o yaml -n istio-system
apiVersion: v1
data:
  tls.crt: REDACTED
  tls.key: REDACTED
kind: Secret
metadata:
  creationTimestamp: 2019-08-20T14:35:16Z
  name: api-termination-istio-tls-secret-autogen
  namespace: istio-system
  resourceVersion: "33667053"
  selfLink: /api/v1/namespaces/istio-system/secrets/api-termination-istio-tls-secret-autogen
  uid: ba7ce062-c357-11e9-b4a3-fa163ed0611a
type: kubernetes.io/tls

When I look back at https://istio.io/docs/tasks/traffic-management/ingress/secure-ingress-sds/#configure-a-tls-ingress-gateway-for-multiple-hosts I notice that step #4 references " serverCertificate and privateKey should not be empty." Does that mean I should make the keys in my Secret serverCertificate and privateKey? When I try that, kubernetes complains that the keys need to be tls.crt and tls.key…

4

I deleted the TLS Secret and created a new generic secret based on the tutorial, but I still don’t see the key/cert. The results are the same as above.

5

This is a Unix Domain Socket, which the ingress-gateway uses to retrieve secrets from the SDS service running on the embedded node agent. With SDS, the secrets never get mounted into the pod.

Have you tried making the keys of the secret key and cert exactly?

6

I made a mistake earlier, but the good news is that it’s working now. I combed through the deployment.yaml for the ingressgateway and found that it was missing the environment variable ISTIO_META_USER_SDS and one volumeMounts for the existing ingress-proxy container. I added those, but I forgot to change my Gateway back to referencing a credentialName instead of the file based approach.

I have confirmed that a generic Secret, as shown in the tutorial, and a tls Secret both work.

Thanks for all your help. It’s all working now.

7

Thank you for posting this issue and especially last comment. I was able to configure TLS for ingress gateway using credentialName/SDS. Someone who have problem with configuration TLS using credentialName need to check your env variable ISTIO_META_USER_SDS and volumeMounts for istio-proxy.

8

I am a little confuse about this. Also for me it works when using serverCertificate and privateKey but not when using credentialName.

Where should I add the env ISTIO_META_USER_SDS? On istio-ingressgateway workload? What should be the value of ISTIO_META_USER_SDS variable?
SDS_ENABLED which is also a variable of istio-ingressgateway by default has a value of false. Should I change it to true?

Thank you

1 Like
9

Even I have the same issue with Istio 1.6.8. Need to find a way to enable sds

1 Like
10

I appear to be having the same issue with 1.9.0. credentialName does not work

1 Like