Demystifying MeshNetworks when using Multi-Cluster deployments or VMs

1

Hola Community!

As the current docs stand, it’s difficult to understand how to use/configure MeshNetworks in IstioOperator for any type of installation, especially Multi-Cluster and VMs.

Do we have documentation that goes into more detail on how to use MeshNetworks.

What I would like to have answered is:

1. When must we specify MeshNetworks?
→ The reason for this question is that we can see in
Install Multi-Primary on different networks that to set up two clusters we don’t need to specify the networks in the clusters at all.

Which is the opposite of what you’d expect according to the MeshNetworks description:

MeshNetworks (config map) provides information about the set of networks inside a mesh and how to route to endpoints in each network.

Assumption: Can we skip specifying networks when we label the namespaces with topology.istio.io/network?

2. What’s the benefit of specifying NetworkEndpoints (of the MeshNetworks.Network) explicitly?

Assumption: That’s how the control plane decides how to configure workloads located in different networks.

Assumptions about the properties:

  • fromCidr: statically define workloads that are within a network. (i.e. 10.10.0.0/16 means that a workload with the IP 10.10.0.122 is considered to be in this network when created anywhere in the mesh)
  • fromRegistry: dynamically interrogate the registry to define in which network a workload is.

If the above assumptions are correct, it means that this is actually important only during workload bootstrapping. Is that correct?

3. What’s the purpose of the section in the IstioNetworkGateway (of the MeshNetworks.Network)?

Assumption: This is how Control Plane knows how to configure workloads in other networks how to reach services within it.

Are any of my assumptions correct? Are there docs that can help me dig the details of this?

Best,
Rinor

2

I put this visually to make it easier to talk about these concepts (and even easier to spot misunderstandings).

image
image987×530 70.1 KB

The configuration shown in the image is applied to the cluster west-cluster; elaborating the points from the image:

  1. The endpoints for the workloads on east-network are retrieved from the registry east-cluster (which has the prerequisite that the secret on how to query the east cluster is already in the istio-system namespace)
  2. To reach the workloads in the east-network network the gateway at the IP 58.23.129.75 will be used. To be more concrete, the CP uses this information to configure workloads in the west-network on how to route traffic to workloads in the east-network
  3. The west-network where the CP is itself installed queries the workload endpoints dynamically from the west-cluster registry (Do we need the secret for west-cluster kubeconfig?).
  4. To reach the workloads on the same network we are pointing to the Ingress Controller istio-ingressgateway ? I picked this example from the docs, and am not sure why is this the case?

Why do we need 4? And did I understand the other points (1-3) correctly?

3

tl;dr MeshNetworks is a legacy piece of configuration that is only needed for rare/advanced cases.

  1. When must we specify MeshNetworks?

When istiod can’t infer the details in your next two assumptions.

Gateways:

  • Infer based on the spec.externalIPs or status.loadBalancer.Ingress field on a k8s Service that has the topology.istio.io/network label.

  • We also support NodePort type services, setting port mappings appropriately and finding the Node IP automatically. This should respond to Node reassignment/changes dynamically but we only have unit tests for this scenario.

  • For Endpoints, we infer based on the system namespace having topology.istio.io/network set, but meshNetworks...fromRegistry can override that.

    • Really, the only time you would need this is to specify endpoints by cidr, which is especially rare.
1 Like