Greetings.
We are trying to deploy Istio 1.8.1 on OpenShift, not RHOSM but “vanilla” Istio through manifests generated with istioctl.
We are currently unable to get istio-ingressgateway to work. On start, it gives the following:
2023-07-24T11:45:11.410755Z info FLAG: --concurrency="0"
2023-07-24T11:45:11.410804Z info FLAG: --domain="istio-system.svc.cluster.local"
2023-07-24T11:45:11.410809Z info FLAG: --help="false"
2023-07-24T11:45:11.410812Z info FLAG: --log_as_json="false"
2023-07-24T11:45:11.410814Z info FLAG: --log_caller=""
2023-07-24T11:45:11.410816Z info FLAG: --log_output_level="default:info"
2023-07-24T11:45:11.410818Z info FLAG: --log_rotate=""
2023-07-24T11:45:11.410820Z info FLAG: --log_rotate_max_age="30"
2023-07-24T11:45:11.410822Z info FLAG: --log_rotate_max_backups="1000"
2023-07-24T11:45:11.410824Z info FLAG: --log_rotate_max_size="104857600"
2023-07-24T11:45:11.410826Z info FLAG: --log_stacktrace_level="default:none"
2023-07-24T11:45:11.410832Z info FLAG: --log_target="[stdout]"
2023-07-24T11:45:11.410835Z info FLAG: --meshConfig="./etc/istio/config/mesh"
2023-07-24T11:45:11.410837Z info FLAG: --outlierLogPath=""
2023-07-24T11:45:11.410839Z info FLAG: --profiling="true"
2023-07-24T11:45:11.410841Z info FLAG: --proxyComponentLogLevel="misc:error"
2023-07-24T11:45:11.410843Z info FLAG: --proxyLogLevel="warn"
2023-07-24T11:45:11.410846Z info FLAG: --s2a_enable_appengine_dialer="false"
2023-07-24T11:45:11.410848Z info FLAG: --s2a_timeout="3s"
2023-07-24T11:45:11.410850Z info FLAG: --serviceCluster="istio-proxy"
2023-07-24T11:45:11.410852Z info FLAG: --stsPort="0"
2023-07-24T11:45:11.410854Z info FLAG: --templateFile=""
2023-07-24T11:45:11.410856Z info FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2023-07-24T11:45:11.410858Z info FLAG: --vklog="0"
2023-07-24T11:45:11.410861Z info Version 1.18.0-697f4ff0fe6ee531bbd4f5f2a6b4b1f302c955a8-Clean
2023-07-24T11:45:11.423989Z info Maximum file descriptors (ulimit -n): 1048576
2023-07-24T11:45:11.424229Z info Proxy role ips=[172.30.17.96] type=router id=istio-ingressgateway-5f67854b64-6mdsv.istio-system domain=istio-system.svc.cluster.local
2023-07-24T11:45:11.424304Z info Apply mesh config from file defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
proxyMetadata: {}
tracing:
zipkin:
address: zipkin.istio-system:9411
defaultProviders:
metrics:
- prometheus
enablePrometheusMerge: true
rootNamespace: istio-system
trustDomain: cluster.local
2023-07-24T11:45:11.426032Z info cpu limit detected as 2, setting concurrency
2023-07-24T11:45:11.426268Z info Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
proxyAdminPort: 15000
serviceCluster: istio-proxy
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
zipkin:
address: zipkin.istio-system:9411
2023-07-24T11:45:11.426283Z info JWT policy is third-party-jwt
2023-07-24T11:45:11.426287Z info using credential fetcher of JWT type in cluster.local trust domain
2023-07-24T11:45:11.526575Z info Workload SDS socket not found. Starting Istio SDS Server
2023-07-24T11:45:11.526575Z info Opening status port 15020
2023-07-24T11:45:11.526616Z info CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2023-07-24T11:45:11.526657Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2023-07-24T11:45:11.543803Z info ads All caches have been synced up in 136.986027ms, marking server ready
2023-07-24T11:45:11.544109Z info xdsproxy Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2023-07-24T11:45:11.544165Z info sds Starting SDS grpc server
2023-07-24T11:45:11.544404Z info starting Http service at 127.0.0.1:15004
2023-07-24T11:45:11.545521Z info Pilot SAN: [istiod.istio-system.svc]
2023-07-24T11:45:11.546523Z info Starting proxy agent
2023-07-24T11:45:11.546582Z info starting
2023-07-24T11:45:11.546619Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields --log-format %Y-%m-%dT%T.%fZ %l envoy %n %g:%# %v thread=%t -l warn --component-log-level misc:error --concurrency 2]
2023-07-24T11:45:11.603250Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2023-07-24T11:45:11.684382Z info ads ADS: new connection for node:istio-ingressgateway-5f67854b64-6mdsv.istio-system-1
2023-07-24T11:45:11.684407Z info ads ADS: new connection for node:istio-ingressgateway-5f67854b64-6mdsv.istio-system-2
2023-07-24T11:45:11.743707Z info cache generated new workload certificate latency=199.577937ms ttl=23h59m59.256304003s
2023-07-24T11:45:11.743800Z info cache Root cert has changed, start rotating root cert
2023-07-24T11:45:11.743826Z info ads XDS: Incremental Pushing:0 ConnectedEndpoints:2 Version:
2023-07-24T11:45:11.743880Z info cache returned workload trust anchor from cache ttl=23h59m59.256125855s
2023-07-24T11:45:11.743960Z info cache returned workload trust anchor from cache ttl=23h59m59.256043704s
2023-07-24T11:45:11.744193Z info cache returned workload certificate from cache ttl=23h59m59.255827087s
2023-07-24T11:45:11.744454Z info ads SDS: PUSH request for node:istio-ingressgateway-5f67854b64-6mdsv.istio-system resources:1 size:4.0kB resource:default
2023-07-24T11:45:11.744464Z info ads SDS: PUSH request for node:istio-ingressgateway-5f67854b64-6mdsv.istio-system resources:1 size:1.1kB resource:ROOTCA
2023-07-24T11:45:11.744533Z info cache returned workload trust anchor from cache ttl=23h59m59.255472373s
2023-07-24T11:45:11.799139Z warning envoy config external/envoy/source/common/config/grpc_subscription_impl.cc:128 gRPC config for type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret rejected: Failed to load private key from <inline>, Cause: error:0900006e:PEM routines:OPENSSL_internal:NO_START_LINE thread=20
2023-07-24T11:45:12.098307Z info Readiness succeeded in 695.967394ms
2023-07-24T11:45:12.098653Z info Envoy proxy is ready
2023-07-24T12:16:44.410953Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
Trying to connect from the pod itself fails with “Connection reset by peer”:
$ oc exec istio-ingressgateway-5f67854b64-6mdsv -- curl https://localhost:8443
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to localhost:8443
command terminated with exit code 35
Using a passthrough Openshift Route we get the same result.
Currently all the Gateways are using the same secret, “router-certs”, which has been generated with cert manager and is correctly formed. We have created the istio-ingressgateway-certs secret also, and it is mounted in the Pod.
Can there be any other certificates involved? Do we have to mount the “router-certs” secret?
Thank you.