I’m currently struggling with having a VM join a mesh.
In a standalone proof-of-concept cluster (created with kubeadm), everything worked out as expected (following the guide from the documentation.
When transporting that configuration into our production K8s setup, I have problems with the CSR for the VM. I tried to track down differences in the setup between the proof-of-concept and the regular setup, trief different Istio version (1.7.3, 1.7.6, 1.8.2) but to no avail. Both the logs from pilot-agent and envoy on the VM side and the Istiod-Logs seem not to provide any meaningful insights.
In 1.7.3, I couldn’t identify any logs being produced in Istiod. In 1.8.2, there are ADS logs and istioctl ps knows about the VM, but there is still a problem on the VM side.
I’d appreciate any hints as to where I could look into or how to debug further. Is this even a problem with signing the CSR in istiod or is it between agent and envoy on the VM?
There are the logs from pilot-agent/envoy on the VM
2021-01-20T11:47:16.762221Z info xdsproxy Envoy ADS stream established
2021-01-20T11:47:16.762400Z info xdsproxy connecting to upstream XDS server: istiod.istio-system.svc:15012
2021-01-20T11:47:16.862818Z debug sds sds/sdsservice.go:595 resource:ROOTCA add a new connection
2021-01-20T11:47:16.862871Z info sds sds/sdsservice.go:277 resource:ROOTCA new connection
2021-01-20T11:47:16.862943Z debug sds sds/sdsservice.go:326 resource:ROOTCA received SDS request from proxy "sidecar~10.90.41.175~pps-istiotest-fe-vm3.ams-meshexpansion-dev~ams-meshexpansion-dev.svc.c
luster.local", first request: true, version info "", error details <nil>
2021-01-20T11:47:16.862957Z info sds sds/sdsservice.go:342 Skipping waiting for gateway secret
2021-01-20T11:47:16.863248Z info cache adding watcher for file ./etc/certs/root-cert.pem
2021-01-20T11:47:16.863345Z info cache GenerateSecret from file ROOTCA
2021-01-20T11:47:16.864005Z info sds sds/sdsservice.go:637 resource:ROOTCA pushed root cert to proxy
2021-01-20T11:47:16.864041Z debug sds sds/sdsservice.go:638 resource:ROOTCA pushed root cert <redacted>
to proxy
2021-01-20T11:47:16.864323Z debug sds sds/sdsservice.go:595 resource:default add a new connection
2021-01-20T11:47:16.864339Z info sds sds/sdsservice.go:277 resource:default new connection
2021-01-20T11:47:16.864390Z debug sds sds/sdsservice.go:326 resource:default received SDS request from proxy "sidecar~10.90.41.175~pps-istiotest-fe-vm3.ams-meshexpansion-dev~ams-meshexpansion-dev.svc.
cluster.local", first request: true, version info "", error details <nil>
2021-01-20T11:47:16.864402Z info sds sds/sdsservice.go:342 Skipping waiting for gateway secret
2021-01-20T11:47:16.918770Z debug sds sds/sdsservice.go:320 resource:ROOTCA received SDS ACK from proxy "sidecar~10.90.41.175~pps-istiotest-fe-vm3.ams-meshexpansion-dev~ams-meshexpansion-dev.svc.clust
er.local", version info "2021-01-20 11:47:16.863002572 +0000 UTC m=+0.387582993", error details <nil>
2021-01-20T11:47:17.301050Z error cache cannot load key pair from /etc/certs: open /etc/certs/cert-chain.pem: no such file or directory
2021-01-20T11:47:17.303371Z error citadelclient citadel/client.go:101 Failed to create certificate: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR