Can't get Ingress gateway to start up when SDS is enabled

1

Hi folks,

I’ve tried this earlier and got the same result. I’m trying it again.

I enabled SDS for the Ingress gateway.

  1. istio-security-post-install-1.4.2 shows as 1/2 Running. Istio-init and kubectl completed. Istio-proxy is running. Is this normal? I’ve always thought that it was one of those install-time pods that would complete and would no longer be needed afterwards. So I’m confused as to why it’s still running. What is its purpose?

  2. The more important thing in this post:

istio-ingressgateway-794d7fc889-4mzs5 1/2 CreateContainerError 0 39m

It is ingress-sds that fails. But when I do “describe” or “logs”, that’s also all I see… “CreateContainerError”. How else can I troubleshoot this?

Here are the relevant sections in the “describe” output:

ingress-sds:
    Container ID:   
    Image:          my.dtr.here/istio/node-agent-k8s:1.4.2
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CreateContainerError
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:     100m
      memory:  128Mi
    Environment:
      ENABLE_WORKLOAD_SDS:         false
      ENABLE_INGRESS_GATEWAY_SDS:  true
      INGRESS_GATEWAY_NAMESPACE:   istio-system (v1:metadata.namespace)
    Mounts:
      /var/run/ingress_gateway from ingressgatewaysdsudspath (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from istio-ingressgateway-service-account-token-btk2r (ro)

…

Events:
  Type     Reason            Age                  From                               Message
  ----     ------            ----                 ----                               -------
  Normal   Scheduled         40m                  default-scheduler                  Successfully assigned istio-system/istio-ingressgateway-794d7fc889-4mzs5 to hostname.company.com
  Normal   Pulling           40m                  kubelet, hostname.company.com  Pulling image "nonprod.dtr.trusted.visa.com/istio/node-agent-k8s:1.4.2"
  Normal   Pulled            40m                  kubelet, hostname.company.com   Successfully pulled image "nonprod.dtr.trusted.visa.com/istio/node-agent-k8s:1.4.2"
  Normal   Pulled            40m                  kubelet, hostname.company.com   Container image "nonprod.dtr.trusted.visa.com/istio/proxyv2:1.4.2" already present on machine
  Normal   Created           40m                  kubelet, hostname.company.com   Created container istio-proxy
  Normal   Started           40m                  kubelet, hostname.company.com   Started container istio-proxy
  Warning  Unhealthy         40m (x6 over 40m)    kubelet, hostname.company.com   Readiness probe failed: Get http://192.168.125.223:15020/healthz/ready: dial tcp 192.168.125.223:15020: connect: connection refused
  Warning  Failed            40m (x4 over 40m)    kubelet, hostname.company.com   Error: Error response from daemon: No command specified
  Normal   Pulled            40m (x3 over 40m)    kubelet, hostname.company.com   Container image "my.dtr.here/istio/node-agent-k8s:1.4.2" already present on machine
  Warning  Unhealthy         40m (x2 over 40m)    kubelet, hostname.company.com  Readiness probe failed: HTTP probe failed with statuscode: 503
  Warning  DNSConfigForming  39s (x189 over 40m)  kubelet, hostname.company.com   Search Line limits were exceeded, some search paths have been omitted, the applied search line is: istio-system.svc.cluster.local svc.cluster.local cluster.local company.com usa.company.net trusted.company.com

This is all I see in the ingress-sds logs:

Error from server (BadRequest): container "ingress-sds" in pod "istio-ingressgateway-794d7fc889-4mzs5" is waiting to start: CreateContainerError

Anything obvious to you, folks?

Anywhere else I can look to get to the bottom of the failure?

Thanks,
jaid

2

Problem solved. Turns out I had a bad (but non-corrupt) copy of the node-agent-k8s image. It was missing the Entrypoint. I remember downloading it from the prerelease directory. I pulled a new one and it worked. And it’s no April Fools’ joke.