How can you use a delegate VirtualService to route both internal and external traffic?

Networking
1

We’re using Istio 1.6.14. I have a microservice that is used for both internal (in-cluster) and external (via an Istio ingress gateway) calls. Let’s say the microservice exposes /operation so if you call http://mysvc.myns.svc.cluster.local/operation you’ll get a response.

When accessed from the outside, I need to do some rewriting. It comes in with an extra prefix, like http://api.mydomain.com/api/operation so the path /api/operation needs to be rewritten. Internal calls don’t require this rewrite.

I am trying to do canary deployments and allow for VirtualService routing across three different deployments - a stable version (the current); a baseline version (a “control group” version of the stable for statistics comparison); and a canary version (the new version). I want to be able to change the traffic routing percentages in one place and have it work for both internal and external communications.

I though a VirtualService delegate would be the right way to go, so I created this:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: load-balancer
  namespace: myns
spec:
  http:
    - route:
        - destination:
            host: mysvc-stable
          weight: 50
        - destination:
            host: mysvc-baseline
          weight: 25
        - destination:
            host: mysvc-canary
          weight: 25

I then set up my external traffic routing VirtualService pointing at my gateway. Note the hosts is the IP address of the inbound request because we’re going through an Apigee mTLS connection so the DNS resolves to Apigee, not to the Istio ingress. But this works just fine.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: external-traffic
  namespace: myns
spec:
  gateways:
    - istio-system/apigee-mtls
  hosts:
    - 1.2.3.4
  http:
    - delegate:
        name: load-balancer
        namespace: myns
      match:
        - uri:
            prefix: /api/operation
      rewrite:
        uri: /operation

With just these two VirtualService in place, I can access my service from the outside and the traffic routing works perfectly. So far, so good.

The problem is that if I access internal to the cluster then I’m not getting traffic routing yet. I need a VirtualService for the inside, too. But that’s where things fall down. I tried putting this:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: internal-traffic
  namespace: myns
spec:
  gateways:
    - mesh
  hosts:
    - mysvc
  http:
    - delegate:
        name: load-balancer
        namespace: myns

But I got the error Error from server: error when creating "vs.yaml": admission webhook "validation.istio.io" denied the request: configuration is invalid: http delegate only applies to gateway

I did find this issue in the Istio repo where some folks talked about the same error but I wasn’t sure if this is the same situation or something else.

I then tried to change the delegate to have the internal routing…

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: load-balancer
  namespace: myns
spec:
  gateways:
    - mesh
  hosts:
    - mysvc
  http:
    - route:
        - destination:
            host: mysvc-stable
          weight: 50
        - destination:
            host: mysvc-baseline
          weight: 25
        - destination:
            host: mysvc-canary
          weight: 25

…but as soon as that gets deployed, internal traffic gets routed right but external traffic no longer comes through at all. I get 404 Not Found.

I can’t delegate something on the mesh gateway, but I also can’t delegate something from an ingress gateway to a VirtualService with a named host.

Am I missing something?

2

I have also tried skipping delegates and routing from my external traffic VirtualService to my internal traffic VirtualService like this:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: load-balancer
  namespace: myns
spec:
  hosts:
    - mysvc
  http:
    - route:
        - destination:
            host: mysvc-stable
          weight: 50
        - destination:
            host: mysvc-baseline
          weight: 25
        - destination:
            host: mysvc-canary
          weight: 25
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: external-traffic
  namespace: myns
spec:
  gateways:
    - istio-system/apigee-mtls
  hosts:
    - 1.2.3.4
  http:
    - route:
        - destination:
            host: mysvc
      match:
        - uri:
            prefix: /api/operation
      rewrite:
        uri: /operation

However, it doesn’t appear I can route one VirtualService to another, so when the external-traffic VirtualService sees the host mysvc it goes straight to the Kubernetes Service and doesn’t use any of the load-balancer configuration.

3

I never did figure this out. However, I did figure out a workaround.

First, in Apigee:

  • Set a header that indicates the internal service host I’m routing to. So requests for /api/operation I know should go to mysvc.myns.svc.cluster.local, so I set Service-Host: mysvc.myns.svc.cluster.local.
  • Do all the path rewrite and modification in Apigee so I don’t need to do that in Istio.

At this point, when someone requests endpoint.mydomain.com/api/operation, it will get forwarded to 1.2.3.4/operation and will have a header Service-Host: mysvc.myns.svc.cluster.local.

In Istio, apply an EnvoyFilter on the ingress that’s tied to Istio. This filter will look for Service-Host and, if present, update the Host header to be that value.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: propagate-host-header-from-apigee
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
      app: istio-ingressgateway
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.http_connection_manager"
            subFilter:
              # istio.metadata_exchange is the first filter in the connection
              # manager, at least in Istio 1.6.14.
              name: "istio.metadata_exchange"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.lua
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
            inline_code: |
              function envoy_on_request(request_handle)
                local service_host = request_handle:headers():get("service-host")
                if service_host ~= nil then
                  request_handle:headers():replace("host", service_host)
                end
              end

Now when Istio gets a request on that gateway, it’ll replace the Host header before Istio starts figuring out how to route the traffic.

Finally, I have a single VirtualService that handles all the traffic routing. No delegate required.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: mysvc
  namespace: myns
spec:
  gateways:
    - istio-system/apigee-mtls
    - mesh
  hosts:
    - mysvc
  http:
    - route:
        - destination:
            host: mysvc-stable
          weight: 50
        - destination:
            host: mysvc-baseline
          weight: 25
        - destination:
            host: mysvc-canary
          weight: 25

This will route both the internal and the external traffic using the same VirtualService because it will see it all as one big traffic flow based on Host header.

Obviously whether this is right for you is entirely up to you, like if you need/want to treat internal and external traffic differently with respect to routing or authentication or whatever.

4

Apologies for reviving an old thread. I came across this when I was trying to figure out delegate myself.

It appears that rewrite doesn’t do anything with delegated match.

In the delegated VS, you need to use the same match that is defined delegator. slash and everything.

Not from docs or anything. It’s what I found from experimenting.