Pods not created with headless service

Networking
1

I have implemented a a multi-cluster by following the primary-remote setup. The setup has one primary cluster and two remote clusters. when verifing the setup ( as done her) there is a problem initialising pods when deploying a application.
The problem:

 Warning   FailedCreate        replicaset/helloworld-v1-776f57d5f6   Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject/cluster/RemoteClusterName/net/network1?timeout=10s": cannot route to service with ClusterIP "None"

Has anybody experienced this problem before? i’m using the VMware Tanzu Kubernetes Grid to run my kubernetes environment.

2

I am currently running into the exact same issue across two KinD clusters.

The istiod service does have an properly configured endpoint inside the remote cluster:

$ kubectl -n istio-system get endpoints
NAME     ENDPOINTS             AGE
istiod   172.23.255.10:15012   33m

From the same remote cluster, I am able to resolve the IP address of the istiod service, and to reach the status endpoint of istiod over the primary cluster’s east-west gateway:

# curl -D- http://istiod.istio-system.svc.cluster.local:15021/healthz/ready
HTTP/1.1 200 OK
date: Thu, 06 Oct 2022 15:30:53 GMT
content-length: 0
x-envoy-upstream-service-time: 0
server: envoy

I’m using Istio’s default CA, which according to the Istio / Multicluster / Before you begin page should work just fine with a single primary:

If you are planning to deploy only one primary cluster (i.e., one of the Primary-Remote installations, below), you will only have a single CA (i.e., istiod on cluster1 ) issuing certificates for both clusters. In that case, you can skip the following CA certificate generation step and simply use the default self-signed CA for the installation.

3

@Noto I found the issue.

It seems like the multicluster tutorial is missing one essential piece of information: the configuration of the remote also needs to include multicluster.clusterName.

After performing the following change to the configuration of my remote(s), the istiod Service does acquire a clusterIP, and sidecar injection starts working as expected:

diff --git a/setup-clusters.sh b/setup-clusters.sh
index f366e56..d59f55f 100755
--- a/setup-clusters.sh
+++ b/setup-clusters.sh
@@ -400,6 +400,10 @@ function istio::deploy_remote {
        trustDomain: ${ISTIO_TRUST_DOMAIN}
      values:
        global:
+         multiCluster:
+           clusterName: ${cluster_name}
          remotePilotAddress: ${pilot_addr}
        istiodRemote:
          injectionPath: /inject/cluster/${cluster_name}/net/${ISTIO_NETWORK}

$ kubectl -n istio-system get service/istiod
NAME     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
istiod   ClusterIP   10.255.30.179   <none>        15012/TCP,443/TCP   4m5s

$ kubectl -n sample describe replicaset/helloworld-v1-fdb8c8c58
[...]
Events:
  Type    Reason            Age    From                   Message
  ----    ------            ----   ----                   -------
  Normal  SuccessfulCreate  4m24s  replicaset-controller  Created pod: helloworld-v1-fdb8c8c58-vmpq8
4

Note: this issue is specific to Istio pre 1.15. In fact, the docs for Istio 1.14 do mention these additional attributes.

Since Istio 1.15, istiod-less remotes is used by default, so they are no longer required.