Binding VirtualService to gateway in remote cluster with split-horizon EDS?

Networking
1

We’ve recently done a redeployment of Istio 1.1 across our clusters, configured according to the Split-horizon EDS topology documented here. The contents of the two relevant clusters are distributed like this:

cluster-1

  • Istio control plane in istio-system namespace.
  • cert-manager installation in the cert-manager namespace.
  • Central gateway definition in istio-system (due to limitations regarding resolving the secrets produced by the cert-manager auto-renewal).

cluster-2

  • Istio Citadel and sidecar injector as proscribed by the documentation.
  • Deployment and service for application named birdcage in production namespace.
  • VirtualService definition in production namespace to add routing for production/birdcage to the gateway in cluster-1.

My expectation here was that when I created the VirtualService in cluster-2, the Istio control plane in cluster-1 (which was configured at installation to have access to the K8S API of cluster-2) would be informed. It would then attempt to serve the resource request in the local (cluster-1), resolving the gateway in cluster-1/istio-system. The definitions for the gateway defined in cluster-1 and the virtual service defined in cluster-2 are reproduced below:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
    name: birdcage
    namespace: production
spec:
    hosts:
    - myapp.test.mydomain.io
    gateways:
    - central-gateway.istio-system.svc.cluster.local
    http:
    - match:
        - uri:
            prefix: /login
      route:
      - destination
            host: birdcage.production.svc.cluster.local

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
    name: central-gateway
    namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*.test.mydomain.io`
    - test.mydomain.io
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpRedirect: true
  - hosts:
    - '*.test.mydomain.io'
    - test.mydomain.io
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt

Edit 3/28/2019: Rewrote question for added clarity.

2

As a follow-up to this, I have redeployed the VirtualService into cluster-1 in the above scenario and it appears to work. So, as I had assumed, it appears that VirtualService definitions in remote clusters when using split-horizon are simply not recognized? Perhaps I am misunderstanding the intent here, I apologize if so. Could someone clarify if this is intended? I was under the impression that configuring the control plane in cluster-1 to point to the K8S API of cluster-2 was to enable tracking resources like VirtualService in the remote?

3

The virtualservice created in cluster 2 has no effect, because pilot do watch only service/pods/endpoints from remote clusters, but only config rules like DR VS Gateway Sidecar from local/primary cluster

2 Likes
4

Are there any workarounds for this for teams looking to allow self-service routing on deployment? We have a number of remotes, each delegated to a particular team to suit their needs but each has services that need to be exposed through a public gateway which we maintain in cluster-1. We have considered something like combining Federation with split-horizon EDS to achieve this, would this seem reasonable?

Thank you for your reply! I swear the most challenging part of Istio so far has been communicating with other deployers when we have issues. Wish there were some more immediate, chat-based community to discuss the small stuff.

5

I think the only resolution for you is to watch all clusters.But istio doesn’t support now. But maybe deploy galley for each cluster can help. I haven’t tried, but it seems possible.

6

You can confirm with config wg

7

I have discovered the source of my issue and wonder if you can suggest a workaround? It appears that because the gateway hosting my application domain is created in cluster-1, that same gateway configuration is replicated to the ingress of cluster-2 where the cluster-2 ingress fails to configure the listener because it does not have access to the locally-created certificates. Is there a way to prevent this specific gateway from being distributed to the secondary cluster?

To clarify, traffic enters through the gateway in cluster-1 with an indicated host name of test.mydomain.io and is then forwarded to the ingress of cluster-2 on port 443 but because the gateway definition has also been propagated to cluster-2, port 443 has not been configured because Envoy failed to locate the requisite certificates.

8

Currently, istio propagate configurations to all sidecars and gateways of all clusters.
You said you use the split-horizon eds, it is sni based routing. You donot have to specify custom cert i think.

9

ThisGatewayconfigures 443 port to pass incoming traffic through to the target service specified in a request’s SNI header

If you specify tls certificates in gateway, I think for other plain http or tcp protocol service, you can not access across clusters.

10

arent you better off to simply have multiple independent istios, and forward traffic from ingress gateway in main cluster to ingress gateway of backend clusters? You simply need service entries in the main clusters

11

Yes, but this is the crux of the problem. Our goal is to allow independent teams to deploy into their own clusters and have the Istio control plane observe that a VirtualService definition has been created in their remote and replicate it locally (within the cluster hosting the control plane). This is sort of an expected behaviour for a mesh, I would assume.

We ended up simply using Kubernetes Federation v2 to replicate the VirtualService CRD into the control plane cluster where it could be picked up by Pilot.

As Federation v2 becomes more stable, is Istio looking to integrate with their replication patterns and the existing federation controllers? It was very refreshing to have it just work out of the box in a way that was well-documented and expected. Their implementation of federated services was much easier to debug/understand than our experience trying to get Istio to resolve services across clusters. If I need to manually create ServiceEntry/VirtualService definitions in every cluster that has to resolve a remote service, what is really being “discovered”?