Multi Cluster Load Balancing Weight Constraint Error

1

I’m trying to set up a multi-network multi-primary mesh in EKS between two regions. Per Multi-Cluster/Multi-Network - Cannot use a hostname-based gateway for east-west traffic · Issue #29359 · istio/istio · GitHub, I am setting up EIPs strapped onto my EW NLB and notifying Istio about them via the istio cm. When I add workloads following the multi cluster verify runbook though, envoy gets very mad at the config Istio is sending it:

2021-09-23T13:08:01.396294Z warn    ads ADS:EDS: ACK ERROR sleep-557747455f-8mtxg.sample-105 Internal:Proto constraint validation failed (ClusterLoadAssignmentValidationError.Endpoints[0]: embedded message failed validation | caused by LocalityLbEndpointsValidationError.LbEndpoints[1]: embedded message failed validation | caused by LbEndpointValidationError.LoadBalancingWeight: value must be greater than or equal to 1): cluster_name: "outbound|80||sleep.sample.svc.cluster.local"
endpoints {
  locality {
    region: "us-east-1"
    zone: "us-east-1a"
  }
  lb_endpoints {
    endpoint {
      address {
        socket_address {
          address: "US-EAST-1A-IP"
          port_value: 15443
        }
      }
    }
    metadata {
      filter_metadata {
        key: "envoy.transport_socket_match"
        value {
          fields {
            key: "tlsMode"
            value {
              string_value: "istio"
            }
          }
        }
      }
      filter_metadata {
        key: "istio"
        value {
          fields {
            key: "workload"
            value {
              string_value: ";;;;us-west-2-cluster"
            }
          }
        }
      }
    }
    load_balancing_weight {
      value: 1
    }
  }
  lb_endpoints {
    endpoint {
      address {
        socket_address {
          address: "US-EAST-1B-IP"
          port_value: 15443
        }
      }
    }
    metadata {
      filter_metadata {
        key: "envoy.transport_socket_match"
        value {
          fields {
            key: "tlsMode"
            value {
              string_value: "istio"
            }
          }
        }
      }
      filter_metadata {
        key: "istio"
        value {
          fields {
            key: "workload"
            value {
              string_value: ";;;;us-west-2-cluster"
            }
          }
        }
      }
    }
    load_balancing_weight {
    }
  }
...

Clearly, the load balancing weight is in fact omitted – but I haven’t done anything here other than add the EKS workaround to multi-primary multi-network vanilla stuff.

My config map fwiw:

kubectl get cm -n istio-system istio -ojsonpath='{.data.meshNetworks}' | sed -f sedscript
networks:
  vpc-0295cb78f0beb68e6:
    endpoints:
    - fromRegistry: us-east-1-cluster
    gateways:
    - address: US-EAST-1A-IP
      port: 15443
    - address: US-EAST-1B-IP
      port: 15443
    - address: US-EAST-1C-IP
      port: 15443
  vpc-0384b42480762c56a:
    endpoints:
    - fromRegistry: us-west-2-cluster
    gateways:
    - address: US-WEST-2A-IP
      port: 15443
    - address: US-WEST-2B-IP
      port: 15443
    - address: US-WEST-2C-IP
      port: 15443

Given the relatively few changes I’ve made, it seems like this might be a bug in Istio?

istio 1.11.2
k8s 1.20

2

Hi @jwilner,

I was fighting the same issue. As you linked there are suggestions to add both networks into the config map. Eventually I only added the network definition of the other cluster only, and it started working properly.

e.g. Config map in cluster 1 contains network endpoints of cluster 2 only.

If you cluster communication works correctly, then you should see the ip addresses of your easwestgateway loadbalancer listed in the proxy config endpoints of each service. You can check that with e.g:

istioctl --context $CLUSTER_1 proxy-config endpoint "$(kubectl --context $CLUSTER_1 get pod -l app=helloworld -o jsonpath='{.items[0].metadata.name}')" | grep helloworld

The output will be something like this:

100.96.33.1:5000                 HEALTHY     OK                outbound|5000||helloworld.default.svc.cluster.local
US-WEST-2A-IP:15443        HEALTHY     OK                outbound|5000||helloworld.default.svc.cluster.local
US-WEST-2B-IP:15443        HEALTHY     OK                outbound|5000||helloworld.default.svc.cluster.local
US-WEST-2C-IP:15443        HEALTHY     OK                outbound|5000||helloworld.default.svc.cluster.local