Failure when two k8s `Ingress`es with not the same domain are configured to use the same ingress gateway

Hello,

I have a working setup that allows me to use OAuth2 to authenticate a user before allowing him access to protected resources.
However, this setup only fully works when I use a Gateway and a VirtualService, not when using k8s Ingress with a custom Ingress Class.

First, let me describe my setup.

  • I’m using Istio 1.8.2 (but same behavior was observed w/ 1.7.5 and 1.8.0), installed via the Istio Operator;
  • I’m acting on the default ingress-gateway, w/ the default settings;
  • I have installed a RequestAuthentication, an AuthorizationPolicy and an EnvoyFilter with the following manifests:
# See https://www.blog.jetstack.io/blog/istio-oidc/
---
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: istio-ingressgateway
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  jwtRules:
    # From https://accounts.google.com/.well-known/openid-configuration
    - issuer: https://accounts.google.com
      jwksUri: https://www.googleapis.com/oauth2/v3/certs
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: istio-ingressgateway
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  rules:
    - from:
        - source:
            requestPrincipals: ["*"]
---
# https://www.paraesthesia.com/archive/2020/09/03/setting-up-oauth2-proxy-with-istio/
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: istio-ingressgateway
spec:
  workloadSelector:
    labels:
      app: istio-ingressgateway
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: envoy.http_connection_manager
              subFilter:
                # This filter exists as we create it above using the RequestAuthentication object (IIUC)
                name: envoy.filters.http.jwt_authn
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.ext_authz
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
            # The RequestAuthentication + AuthorizationPolicy will deny or allow
            # requests correctly even if the oauth2 proxy is down
            failure_mode_allow: true
            http_service:
              server_uri:
                # URIs here should be to the oauth2-proxy service inside your
                # cluster, in the namespace where it was deployed. The port
                # in that 'cluster' line should also match up.
                uri: "http://oauth2-proxy.oauth2-proxy.svc.cluster.local:4180"
                cluster: "outbound|4180||oauth2-proxy.oauth2-proxy.svc.cluster.local"
                timeout: 15s
              authorization_request:
                allowed_headers:
                  patterns:
                    - exact: cookie
                      ignore_case: true
              authorization_response:
                allowed_upstream_headers:
                  patterns:
                    - exact: authorization
                      ignore_case: true

My problem, specifically occurs when two (or more) k8s Ingresses using two different domains (or more) are setup.
(First, a note, AFAIK it is not possible to specify the Ingress Gateway that will handle the Ingress, so both Ingresses are handled by the same Ingress Gateway; the default one (istio-ingressgateway).)

The behavior is as follow (tested both w/ Safari and Firefox, and even Chromium):

  • Load first URL, redirect is done correctly;
  • Load second URL, using another browser, still ok;
  • Load second URL w/ same browser as first one in another tab (not long after having loaded first URL) -> we get the following response: RBAC: access denied w/ HTTP code 403. Note the EnvoyFilter is not even reached; the RequestAuthentication or AuthorizationPolicy deny the request directly AFAICT.

So I have three questions from here:

  • I know k8s Ingress are not recommended, however are they still officially supported, and should my setup work?
  • Do anybody knows what the hell is happening and how to fix it? I can use VirtualServices and Gateways to setup the IAP, but when using apps that are available on Helm, it’s easier to use their template and just say ingress.enabled: true and ingress.ingressClassName: istio;
  • Last question, not fully related: Is it possible to set the Ingress Gateway that will handle the k8s Ingress?

My thoughts:

  • For the first and third questions, I don’t think changing the Ingress Gateway is possible, and that is IMHO such a big omission that either I missed something big, or specifying an Ingress this way is close to unsupported (maybe it was supported as a proof of concept, idk);
  • For the second question, I think the browser keeps a connection open to the server, and when we load the page on the second domain, the same connection is used, which confuses the hell out of Istio because the domain is not the same. No idea if this theory is right though…

I’m pretty sure I’ve hit an Istio bug, but I’m posting here first to get some feedback regarding this issue.

Thanks for the help!

EDIT: Here are some logs from the ingress gateway pods. As one can see, among other, on the request that fails the requested server name is not the same as the request authority:

Request that works: "GET /alerts HTTP/2" 200 - "-" 0 186731 23 19 "10.132.0.71" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15" "28a22fb3-7faa-4022-abd2-ea5ef1b8cd15"   "prometheus.mydomain.com" "10.52.6.23:9090" outbound|9090||kube-prom-stack-kube-prome-prometheus.monitoring.svc.cluster.local 10.52.6.21:35994 10.52.6.21:8443 10.132.0.71:17848 prometheus.mydomain.com -
Request that fails: "GET /       HTTP/2" 403 - "-" 0 19     0  -  "10.132.0.71" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15" "5c3c3f8d-c621-4292-8410-284875312e14" "alertmanager.mydomain.com" "-"               -                                                                                 -                10.52.6.21:8443 10.132.0.71:17848 prometheus.mydomain.com -

Log format:

[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% "%UPSTREAM_TRANSPORT_FAILURE_REASON%" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n

Seems to be tracked here if I’m not mistaken Istio does not adhere to HTTP/2 RFC 7540 · Issue #13589 · istio/istio · GitHub (found from 404 NR when using browser on multiple ingress gateways · Issue #9429 · istio/istio · GitHub).