Need help understanding load-balancing between clusters

Networking
1

Dear all,

I think I’m experiencing the same issue as described here:

The setup:

DC1 and DC2 are both running Ingress and East/West-Gateways with Istio 1.8.1

SmartDNS is enabled (at least I hope so):
kubectl -n istio-system get configmaps istio -o yaml

    apiVersion: v1
    data:
      mesh: |-
        defaultConfig:
          discoveryAddress: istiod.istio-system.svc:15012
          meshId: mesh1
          proxyMetadata:
            DNS_AGENT: ""
            ISTIO_META_DNS_CAPTURE: "true"
            ISTIO_META_PROXY_XDS_VIA_AGENT: "true"   
            (...)

DC1 doesn’t run any pod in question, only a Gateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kuard-gateway
spec:
  selector:
    istio: ingressgateway 
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kuard
spec:
  hosts:
  - "*"
  gateways:
  - kuard-gateway
  http:
  - route:
    - destination:
        host: kuard
        port:
          number: 8080

DC2 (as a “Worker” cluster) is running the same gateway, service plus the deployment:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kuard-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kuard
spec:
  hosts:
  - "*"
  gateways:
  - kuard-gateway
  http:
  - route:
    - destination:
        host: kuard
        port:
          number: 8080
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: kuard
  name: kuard
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: kuard
  sessionAffinity: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kuard
  labels:
    app: kuard
spec:
  replicas: 3
  selector:
    matchLabels:
      app: kuard
  template:
    metadata:
      labels:
        app: kuard
    spec:
      containers:
      - name: kuard
        image: gcr.io/kuar-demo/kuard-amd64:1
        imagePullPolicy: IfNotPresent #Always
        ports:
        - containerPort: 8080

Now accessing the DC2’s IP in the browser seems to load-balance nicely between the three pods. But accessing DC1’s IP in the browser always returns a single’s pod response.

Any help is greatly appreciated!

2

Hey.

The issue is that the cross-network-gateway (aka the east-west gateway) is in TLS passthrough mode. it only load-balances TCP connections and not HTTP requests. Most clients keep the connections alive for a while, that’s why you end-up with all the requests going to the same pod. I hope that make sense?

The way I solved that in the past was to convert my east-west gateway into a L7 HTTP gateway. It’s a bit slower, but at least you get proper load-balancing.

1 Like
3

woah, awesome! Thank you… it’s REALLY driving me nuts… but now I do (at least partly) understand the issue.

4

@chaudyg would you mind sharing how you converted the load balancers?