I am using latest istio 1.10 and Openshift 4.6.28_1543.
Was following instructions for creating shared certs and then instructions for installation of the cluster itself. When enabling Endpoint Discovery was following recommendation https://github.com/istio/istio/pull/30565 to deal with more than one secret linked to the remote ServiceAccount.
After install is complete, istio-eastwestgateway never goes to ready state giving an error:
warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unauthenticated desc = request authenticate failure
2021-06-09T12:51:28.716446Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-06-09T12:51:28.717909Z warn xdsproxy upstream terminated with unexpected error rpc error: code = Unauthenticated desc = authentication failure
2021-06-09T12:51:28.718307Z warning envoy config StreamAggregatedResources gRPC config stream closed: 16, authentication failure
2021-06-09T12:51:39.580394Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-06-09T12:51:39.580976Z warn xdsproxy upstream terminated with unexpected error rpc error: code = Unauthenticated desc = authentication failure
2021-06-09T12:51:39.581235Z warning envoy config StreamAggregatedResources gRPC config stream closed: 16, authentication failure
2021-06-09T12:52:09.221937Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-06-09T12:52:09.222548Z warn xdsproxy upstream terminated with unexpected error rpc error: code = Unauthenticated desc = authentication failure
2021-06-09T12:52:09.222859Z warning envoy config StreamAggregatedResources gRPC config stream closed: 16, authentication failure
2021-06-09T12:52:14.051534Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
When I go to istiod log, I see the following:
2021-06-09T13:10:53.102750Z error watch error in cluster borisistio1: failed to list *v1.Namespace: Get "https://c104-e.ca-tor.containers.cloud.ibm.com:30042/api/v1/namespaces?fieldSelector=metadata.name%3Distio-system&limit=500&resourceVersion=0": x509: certificate signed by unknown authority
2021-06-09T13:11:04.080923Z error klog k8s.io/client-go@v0.20.5/tools/cache/reflector.go:167: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get "https://c104-e.ca-tor.containers.cloud.ibm.com:30042/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
2021-06-09T13:11:08.307736Z error ads Failed to authenticate client from 172.17.38.27:43732: Authenticator ClientCertAuthenticator: no verified chain is found; Authenticator KubeJWTAuthenticator: could not get cluster borisistio2's kube client
2021-06-09T13:11:10.063775Z error watch error in cluster borisistio1: failed to list *v1.Service: Get "https://c104-e.ca-tor.containers.cloud.ibm.com:30042/api/v1/services?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
2021-06-09T13:11:20.950034Z warn serverca Authentication failed for 172.17.38.27:49520: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: could not get cluster borisistio2's kube client.
2021-06-09T13:11:29.835597Z error watch error in cluster borisistio1: failed to list *v1.Pod: Get "https://c104-e.ca-tor.containers.cloud.ibm.com:30042/api/v1/pods?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
2021-06-09T13:11:30.369876Z error klog k8s.io/client-go@v0.20.5/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://c104-e.ca-tor.containers.cloud.ibm.com:30042/api/v1/secrets?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
2021-06-09T13:11:31.866537Z error watch error in cluster borisistio1: failed to list *v1.Namespace: Get "https://c104-e.ca-tor.containers.cloud.ibm.com:30042/api/v1/namespaces?fieldSelector=metadata.name%3Distio-system&limit=500&resourceVersion=0": x509: certificate signed by unknown authority
2021-06-09T13:11:33.359713Z error watch error in cluster borisistio1: failed to list *v1.Endpoints: Get "https://c104-e.ca-tor.containers.cloud.ibm.com:30042/api/v1/endpoints?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
2021-06-09T13:11:33.365392Z error ads Failed to authenticate client from 172.17.38.27:51456: Authenticator ClientCertAuthenticator: no verified chain is found; Authenticator KubeJWTAuthenticator: could not get cluster borisistio2's kube client
2021-06-09T13:11:42.506484Z error klog k8s.io/client-go@v0.20.5/tools/cache/reflector.go:167: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get "https://c104-e.ca-tor.containers.cloud.ibm.com:30042/api/v1/namespaces?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
2021-06-09T13:11:43.524527Z error watch error in cluster borisistio1: failed to list *v1.Node: Get "https://c104-e.ca-tor.containers.cloud.ibm.com:30042/api/v1/nodes?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
2021-06-09T13:11:55.400797Z error watch error in cluster borisistio1: failed to list *v1.Service: Get "https://c104-e.ca-tor.containers.cloud.ibm.com:30042/api/v1/services?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
2021-06-09T13:11:58.555091Z error ads Failed to authenticate client from 172.17.38.27:53482: Authenticator ClientCertAuthenticator: no verified chain is found; Authenticator KubeJWTAuthenticator: could not get cluster borisistio2's kube client
2021-06-09T13:12:01.020546Z error klog k8s.io/client-go@v0.20.5/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://c104-e.ca-tor.containers.cloud.ibm.com:30042/api/v1/secrets?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
2021-06-09T13:12:02.794998Z error ads Failed to authenticate client from 172.17.38.27:53558: Authenticator ClientCertAuthenticator: no verified chain is found; Authenticator KubeJWTAuthenticator: could not get cluster borisistio2's kube client
2021-06-09T13:12:11.546204Z error watch error in cluster borisistio1: failed to list *v1.Endpoints: Get "https://c104-e.ca-tor.containers.cloud.ibm.com:30042/api/v1/endpoints?limit=500&resourceVersion=0": x509: certificate signed by unknown authority
Any idea on what is happening?