Ingressgateway not accepting connections, error:0900006e:PEM routines:OPENSSL_internal:NO_START_LINE


We are trying to deploy Istio 1.8.1 on OpenShift, not RHOSM but “vanilla” Istio through manifests generated with istioctl.

We are currently unable to get istio-ingressgateway to work. On start, it gives the following:

2023-07-24T11:45:11.410755Z     info    FLAG: --concurrency="0"
2023-07-24T11:45:11.410804Z     info    FLAG: --domain="istio-system.svc.cluster.local"
2023-07-24T11:45:11.410809Z     info    FLAG: --help="false"
2023-07-24T11:45:11.410812Z     info    FLAG: --log_as_json="false"
2023-07-24T11:45:11.410814Z     info    FLAG: --log_caller=""
2023-07-24T11:45:11.410816Z     info    FLAG: --log_output_level="default:info"
2023-07-24T11:45:11.410818Z     info    FLAG: --log_rotate=""
2023-07-24T11:45:11.410820Z     info    FLAG: --log_rotate_max_age="30"
2023-07-24T11:45:11.410822Z     info    FLAG: --log_rotate_max_backups="1000"
2023-07-24T11:45:11.410824Z     info    FLAG: --log_rotate_max_size="104857600"
2023-07-24T11:45:11.410826Z     info    FLAG: --log_stacktrace_level="default:none"
2023-07-24T11:45:11.410832Z     info    FLAG: --log_target="[stdout]"
2023-07-24T11:45:11.410835Z     info    FLAG: --meshConfig="./etc/istio/config/mesh"
2023-07-24T11:45:11.410837Z     info    FLAG: --outlierLogPath=""
2023-07-24T11:45:11.410839Z     info    FLAG: --profiling="true"
2023-07-24T11:45:11.410841Z     info    FLAG: --proxyComponentLogLevel="misc:error"
2023-07-24T11:45:11.410843Z     info    FLAG: --proxyLogLevel="warn"
2023-07-24T11:45:11.410846Z     info    FLAG: --s2a_enable_appengine_dialer="false"
2023-07-24T11:45:11.410848Z     info    FLAG: --s2a_timeout="3s"
2023-07-24T11:45:11.410850Z     info    FLAG: --serviceCluster="istio-proxy"
2023-07-24T11:45:11.410852Z     info    FLAG: --stsPort="0"
2023-07-24T11:45:11.410854Z     info    FLAG: --templateFile=""
2023-07-24T11:45:11.410856Z     info    FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2023-07-24T11:45:11.410858Z     info    FLAG: --vklog="0"
2023-07-24T11:45:11.410861Z     info    Version 1.18.0-697f4ff0fe6ee531bbd4f5f2a6b4b1f302c955a8-Clean
2023-07-24T11:45:11.423989Z     info    Maximum file descriptors (ulimit -n): 1048576
2023-07-24T11:45:11.424229Z     info    Proxy role      ips=[] type=router id=istio-ingressgateway-5f67854b64-6mdsv.istio-system domain=istio-system.svc.cluster.local
2023-07-24T11:45:11.424304Z     info    Apply mesh config from file defaultConfig:
  discoveryAddress: istiod.istio-system.svc:15012
  proxyMetadata: {}
      address: zipkin.istio-system:9411
  - prometheus
enablePrometheusMerge: true
rootNamespace: istio-system
trustDomain: cluster.local
2023-07-24T11:45:11.426032Z     info    cpu limit detected as 2, setting concurrency
2023-07-24T11:45:11.426268Z     info    Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
proxyAdminPort: 15000
serviceCluster: istio-proxy
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
    address: zipkin.istio-system:9411

2023-07-24T11:45:11.426283Z     info    JWT policy is third-party-jwt
2023-07-24T11:45:11.426287Z     info    using credential fetcher of JWT type in cluster.local trust domain
2023-07-24T11:45:11.526575Z     info    Workload SDS socket not found. Starting Istio SDS Server
2023-07-24T11:45:11.526575Z     info    Opening status port 15020
2023-07-24T11:45:11.526616Z     info    CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2023-07-24T11:45:11.526657Z     info    Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2023-07-24T11:45:11.543803Z     info    ads     All caches have been synced up in 136.986027ms, marking server ready
2023-07-24T11:45:11.544109Z     info    xdsproxy        Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2023-07-24T11:45:11.544165Z     info    sds     Starting SDS grpc server
2023-07-24T11:45:11.544404Z     info    starting Http service at
2023-07-24T11:45:11.545521Z     info    Pilot SAN: [istiod.istio-system.svc]
2023-07-24T11:45:11.546523Z     info    Starting proxy agent
2023-07-24T11:45:11.546582Z     info    starting
2023-07-24T11:45:11.546619Z     info    Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields --log-format %Y-%m-%dT%T.%fZ        %l      envoy %n %g:%#  %v      thread=%t -l warn --component-log-level misc:error --concurrency 2]
2023-07-24T11:45:11.603250Z     info    xdsproxy        connected to upstream XDS server: istiod.istio-system.svc:15012
2023-07-24T11:45:11.684382Z     info    ads     ADS: new connection for node:istio-ingressgateway-5f67854b64-6mdsv.istio-system-1
2023-07-24T11:45:11.684407Z     info    ads     ADS: new connection for node:istio-ingressgateway-5f67854b64-6mdsv.istio-system-2
2023-07-24T11:45:11.743707Z     info    cache   generated new workload certificate      latency=199.577937ms ttl=23h59m59.256304003s
2023-07-24T11:45:11.743800Z     info    cache   Root cert has changed, start rotating root cert
2023-07-24T11:45:11.743826Z     info    ads     XDS: Incremental Pushing:0 ConnectedEndpoints:2 Version:
2023-07-24T11:45:11.743880Z     info    cache   returned workload trust anchor from cache       ttl=23h59m59.256125855s
2023-07-24T11:45:11.743960Z     info    cache   returned workload trust anchor from cache       ttl=23h59m59.256043704s
2023-07-24T11:45:11.744193Z     info    cache   returned workload certificate from cache        ttl=23h59m59.255827087s
2023-07-24T11:45:11.744454Z     info    ads     SDS: PUSH request for node:istio-ingressgateway-5f67854b64-6mdsv.istio-system resources:1 size:4.0kB resource:default
2023-07-24T11:45:11.744464Z     info    ads     SDS: PUSH request for node:istio-ingressgateway-5f67854b64-6mdsv.istio-system resources:1 size:1.1kB resource:ROOTCA
2023-07-24T11:45:11.744533Z     info    cache   returned workload trust anchor from cache       ttl=23h59m59.255472373s
2023-07-24T11:45:11.799139Z     warning envoy config external/envoy/source/common/config/  gRPC config for rejected: Failed to load private key from <inline>, Cause: error:0900006e:PEM routines:OPENSSL_internal:NO_START_LINE   thread=20
2023-07-24T11:45:12.098307Z     info    Readiness succeeded in 695.967394ms
2023-07-24T11:45:12.098653Z     info    Envoy proxy is ready
2023-07-24T12:16:44.410953Z     info    xdsproxy        connected to upstream XDS server: istiod.istio-system.svc:15012

Trying to connect from the pod itself fails with “Connection reset by peer”:

$ oc exec istio-ingressgateway-5f67854b64-6mdsv -- curl https://localhost:8443
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to localhost:8443
command terminated with exit code 35

Using a passthrough Openshift Route we get the same result.

Currently all the Gateways are using the same secret, “router-certs”, which has been generated with cert manager and is correctly formed. We have created the istio-ingressgateway-certs secret also, and it is mounted in the Pod.

Can there be any other certificates involved? Do we have to mount the “router-certs” secret?

Thank you.

Solved. Although we have been unable to see what was wrong with it, regenerating the router-certs Secret fixed the problem.