Node_agent running on vm cannot connect to istio-citadel:8060

Hi,
I am trying to use mesh expansion feature.
On gke i have created a cluster with istio feature enabled.
These are the services which are created as a result of that.


kubectl get service -n istio-system
NAME                     TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                                                                                                                                      AGE
istio-citadel            ClusterIP      10.68.26.171   <none>          8060/TCP,15014/TCP                                                                                                                           83s
istio-galley             ClusterIP      10.68.30.127   <none>          443/TCP,15014/TCP,9901/TCP                                                                                                                   83s
istio-ingressgateway     LoadBalancer   10.68.23.213   34.82.102.180   15020:32626/TCP,80:32168/TCP,443:30721/TCP,31400:31806/TCP,15029:31936/TCP,15030:31974/TCP,15031:32403/TCP,15032:31062/TCP,15443:30434/TCP   83s
istio-pilot              ClusterIP      10.68.21.210   <none>          15010/TCP,15011/TCP,8080/TCP,15014/TCP                                                                                                       83s
istio-policy             ClusterIP      10.68.19.22    <none>          9091/TCP,15004/TCP,15014/TCP                                                                                                                 83s
istio-sidecar-injector   ClusterIP      10.68.23.80    <none>          443/TCP,15014/TCP                                                                                                                            83s
istio-telemetry          ClusterIP      10.68.19.246   <none>          9091/TCP,15004/TCP,15014/TCP,42422/TCP                                                                                                       83s
promsd                   ClusterIP      10.68.28.44    <none>          9090/TCP                                                                                                                                     82s

Above i see that istio-ingressgateway has a load balancer but port no 8060 is not open on this load balancer. However istio-citadel is listening on 8060. Am wording how mesh expansion will work in this case as node_agent is trying to connect to 34.82.102.180:8060 but 8060 is not open on the load balancer.


I am following Istioldie 1.1 / Mesh Expansion.


Below are the logs from node_agent running on the vm.

sudo node_agent
sudo: unable to resolve host ip-10-0-1-38
2021-01-05T06:44:18.819820Z info parsed scheme: “”
2021-01-05T06:44:18.820019Z info scheme “” not registered, fallback to default scheme
2021-01-05T06:44:18.820180Z info Starting Node Agent
2021-01-05T06:44:18.820315Z info Node Agent starts successfully.
2021-01-05T06:44:18.833250Z info ccResolverWrapper: sending new addresses to cc: [{istio-citadel:8060 0 }]
2021-01-05T06:44:18.833395Z info ClientConn switching balancer to “pick_first”
2021-01-05T06:44:18.833548Z info pickfirstBalancer: HandleSubConnStateChange: 0xc420210a30, CONNECTING
2021-01-05T06:44:18.851254Z info grpc: addrConn.createTransport failed to connect to {istio-citadel:8060 0 }. Err :connection error: desc = “transport: Error while dialing dial tcp 34.82.102.180:8060: connect: connection refused”. Reconnecting…
2021-01-05T06:44:18.871441Z info pickfirstBalancer: HandleSubConnStateChange: 0xc420210a30, TRANSIENT_FAILURE
2021-01-05T06:44:19.286842Z info Sending CSR (retrial #0) …
2021-01-05T06:44:19.287126Z error CSR signing failed: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = “transport: Error while dialing dial tcp 34.82.102.180:8060: connect: connection refused”. Will retry in 5s
2021-01-05T06:44:19.837294Z info pickfirstBalancer: HandleSubConnStateChange: 0xc420210a30, CONNECTING
2021-01-05T06:44:19.851044Z info grpc: addrConn.createTransport failed to connect to {istio-citadel:8060 0 }. Err :connection error: desc = “transport: Error while dialing dial tcp 34.82.102.180:8060: connect: connection refused”. Reconnecting…
2021-01-05T06:44:19.851155Z info pickfirstBalancer: HandleSubConnStateChange: 0xc420210a30, TRANSIENT_FAILURE
2021-01-05T06:44:21.328110Z info pickfirstBalancer: HandleSubConnStateChange: 0xc420210a30, CONNECTING
2021-01-05T06:44:21.342889Z info grpc: addrConn.createTransport failed to connect to {istio-citadel:8060 0 }. Err :connection error: desc = “transport: Error while dialing dial tcp 34.82.102.180:8060: connect: connection refused”. Reconnecting…
2021-01-05T06:44:21.343052Z info pickfirstBalancer: HandleSubConnStateChange: 0xc420210a30, TRANSIENT_FAILURE
2021-01-05T06:44:23.772050Z info pickfirstBalancer: HandleSubConnStateChange: 0xc420210a30, CONNECTING
2021-01-05T06:44:23.785851Z info grpc: addrConn.createTransport failed to connect to {istio-citadel:8060 0 }. Err :connection error: desc = “transport: Error while dialing dial tcp 34.82.102.180:8060: connect: connection refused”. Reconnecting…
2021-01-05T06:44:23.785912Z info pickfirstBalancer: HandleSubConnStateChange: 0xc420210a30, TRANSIENT_FAILURE
2021-01-05T06:44:24.560448Z info Sending CSR (retrial #1) …
2021-01-05T06:44:24.560708Z error CSR signing failed: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = “transport: Error while dialing dial tcp 34.82.102.180:8060: connect: connection refused”. Will retry in 10s
2021-01-05T06:44:27.827284Z info pickfirstBalancer: HandleSubConnStateChange: 0xc420210a30, CONNECTING
2021-01-05T06:44:27.841673Z info grpc: addrConn.createTransport failed to connect to {istio-citadel:8060 0 }. Err :connection error: desc = “transport: Error while dialing dial tcp 34.82.102.180:8060: connect: connection refused”. Reconnecting…
2021-01-05T06:44:27.841696Z info pickfirstBalancer: HandleSubConnStateChange: 0xc420210a30, TRANSIENT_FAILURE