I’ve followed the guide at Istio / Install Primary-Remote with two KinD clusters with MetalLB installed. When I do the step “ Configure cluster2 as a remote”, the istioctl install fails. The ingressgateway fails to become ready. The logs are full of complaints that the Pilot CA cannot be verified. This makes sense to me as there’s not a step where this is provided to the remote cluster. Is it supposed to be obtained by istiod CA management?
Going on to “Verify the installation”, the remote cluster can’t contact either the primary or remote service. The primary cluster only contacts the primary service.
Any ideas what I’ve messed up, or what to look for?
Thanks,
Dan.
I’ve gone on to experiment, by taking the CA from the primary node and overwriting the secret configuration in the remote mode. This causes the error message changes to authentication failure.
2021-04-26T14:10:09.951517Z info Proxy role ips=[10.244.0.23 fe80::c435:79ff:fe0f:4f01] type=sidecar id=sleep-557747455f-b5ptv.sample domain=sample.svc.cluster.local
2021-04-26T14:10:09.951549Z info JWT policy is third-party-jwt
2021-04-26T14:10:09.951565Z info Pilot SAN: [istiod-remote.istio-system.svc]
2021-04-26T14:10:09.951591Z info CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2021-04-26T14:10:09.951652Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2021-04-26T14:10:09.951771Z info citadelclient Citadel client using custom root cert: istiod.istio-system.svc:15012
2021-04-26T14:10:10.010113Z info ads All caches have been synced up in 73.0118ms, marking server ready
2021-04-26T14:10:10.011540Z info sds SDS server for workload certificates started, listening on “./etc/istio/proxy/SDS”
2021-04-26T14:10:10.011594Z info xdsproxy Initializing with upstream address “istiod-remote.istio-system.svc:15012” and cluster “remote”
2021-04-26T14:10:10.012190Z info sds Start SDS grpc server
2021-04-26T14:10:10.014222Z info Opening status port 15020
2021-04-26T14:10:10.014400Z info Received new config, creating new Envoy epoch 0
2021-04-26T14:10:10.014483Z info Epoch 0 starting
2021-04-26T14:10:10.012997Z info Starting proxy agent
2021-04-26T14:10:10.071723Z info Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster sleep.sample --service-node sidecar~10.244.0.23~sleep-557747455f-b5ptv.sample~sample.svc.cluster.local --local-address-ip-version v4 --bootstrap-version 3 --log-format %Y-%m-%dT%T.%fZ %l envoy %n %v -l warning --component-log-level misc:error --concurrency 2]
2021-04-26T14:10:10.145137Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-26T14:10:10.145203Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-26T14:10:10.145644Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-26T14:10:10.145703Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-26T14:10:10.221145Z info xdsproxy connected to upstream XDS server: istiod-remote.istio-system.svc:15012
2021-04-26T14:10:10.223686Z warn xdsproxy upstream terminated with unexpected error rpc error: code = Unauthenticated desc = authentication failure
2021-04-26T14:10:10.226579Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-26T14:10:10.324412Z info cache Root cert has changed, start rotating root cert
2021-04-26T14:10:10.324500Z info ads XDS: Incremental Pushing:0 ConnectedEndpoints:0 Version:
2021-04-26T14:10:10.324571Z info cache generated new workload certificate latency=313.1383ms ttl=23h59m59.6754551s
2021-04-26T14:10:10.395512Z info xdsproxy connected to upstream XDS server: istiod-remote.istio-system.svc:15012
2021-04-26T14:10:10.399037Z warn xdsproxy upstream terminated with unexpected error rpc error: code = Unauthenticated desc = authentication failure
2021-04-26T14:10:10.399575Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-26T14:10:10.466953Z info xdsproxy connected to upstream XDS server: istiod-remote.istio-system.svc:15012
2021-04-26T14:10:10.473444Z warn xdsproxy upstream terminated with unexpected error rpc error: code = Unauthenticated desc = authentication failure
2021-04-26T14:10:10.476690Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-26T14:10:12.142104Z info xdsproxy connected to upstream XDS server: istiod-remote.istio-system.svc:15012
2021-04-26T14:10:12.145329Z warn xdsproxy upstream terminated with unexpected error rpc error: code = Unauthenticated desc = authentication failure
2021-04-26T14:10:12.146144Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-26T14:10:14.750826Z info xdsproxy connected to upstream XDS server: istiod-remote.istio-system.svc:15012
2021-04-26T14:10:14.756147Z warn xdsproxy upstream terminated with unexpected error rpc error: code = Unauthenticated desc = authentication failure
2021-04-26T14:10:14.757044Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-26T14:10:19.977073Z info xdsproxy connected to upstream XDS server: istiod-remote.istio-system.svc:15012
2021-04-26T14:10:19.979307Z warn xdsproxy upstream terminated with unexpected error rpc error: code = Unauthenticated desc = authentication failure
2021-04-26T14:10:19.979307Z error xdsproxy upstream send error for type url type.googleapis.com/envoy.config.cluster.v3.Cluster: EOF
2021-04-26T14:10:19.979927Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-26T14:10:20.781374Z info xdsproxy connected to upstream XDS server: istiod-remote.istio-system.svc:15012
2021-04-26T14:10:20.786323Z warn xdsproxy upstream terminated with unexpected error rpc error: code = Unauthenticated desc = authentication failure
2021-04-26T14:10:20.787035Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-26T14:10:41.027085Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
I assume that there’s something else that needs to be set to generate valid workload certificates?
Is this something that’s supposed to happen automatically within Istio, or should there be a step when setting up a remote cluster where the relevant configuration is made prior to installing Istio?
Thanks,
Dan.
Hey, were you able to find the solution for authentication failure problem ?