I have a working setup that allows me to use OAuth2 to authenticate a user before allowing him access to protected resources.
However, this setup only fully works when I use a
Gateway and a
VirtualService, not when using k8s
Ingress with a custom Ingress Class.
First, let me describe my setup.
- I’m using Istio 1.8.2 (but same behavior was observed w/ 1.7.5 and 1.8.0), installed via the Istio Operator;
- I’m acting on the default
ingress-gateway, w/ the default settings;
- I have installed a
EnvoyFilterwith the following manifests:
# See https://www.blog.jetstack.io/blog/istio-oidc/ --- apiVersion: security.istio.io/v1beta1 kind: RequestAuthentication metadata: name: istio-ingressgateway spec: selector: matchLabels: app: istio-ingressgateway jwtRules: # From https://accounts.google.com/.well-known/openid-configuration - issuer: https://accounts.google.com jwksUri: https://www.googleapis.com/oauth2/v3/certs --- apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: istio-ingressgateway spec: selector: matchLabels: app: istio-ingressgateway rules: - from: - source: requestPrincipals: ["*"] --- # https://www.paraesthesia.com/archive/2020/09/03/setting-up-oauth2-proxy-with-istio/ apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: istio-ingressgateway spec: workloadSelector: labels: app: istio-ingressgateway configPatches: - applyTo: HTTP_FILTER match: context: GATEWAY listener: filterChain: filter: name: envoy.http_connection_manager subFilter: # This filter exists as we create it above using the RequestAuthentication object (IIUC) name: envoy.filters.http.jwt_authn patch: operation: INSERT_BEFORE value: name: envoy.filters.http.ext_authz typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz # The RequestAuthentication + AuthorizationPolicy will deny or allow # requests correctly even if the oauth2 proxy is down failure_mode_allow: true http_service: server_uri: # URIs here should be to the oauth2-proxy service inside your # cluster, in the namespace where it was deployed. The port # in that 'cluster' line should also match up. uri: "http://oauth2-proxy.oauth2-proxy.svc.cluster.local:4180" cluster: "outbound|4180||oauth2-proxy.oauth2-proxy.svc.cluster.local" timeout: 15s authorization_request: allowed_headers: patterns: - exact: cookie ignore_case: true authorization_response: allowed_upstream_headers: patterns: - exact: authorization ignore_case: true
My problem, specifically occurs when two (or more) k8s
Ingresses using two different domains (or more) are setup.
(First, a note, AFAIK it is not possible to specify the
Ingress Gateway that will handle the
Ingress, so both
Ingresses are handled by the same
Ingress Gateway; the default one (
The behavior is as follow (tested both w/ Safari and Firefox, and even Chromium):
- Load first URL, redirect is done correctly;
- Load second URL, using another browser, still ok;
- Load second URL w/ same browser as first one in another tab (not long after having loaded first URL) -> we get the following response:
RBAC: access deniedw/ HTTP code 403. Note the
EnvoyFilteris not even reached; the
AuthorizationPolicydeny the request directly AFAICT.
So I have three questions from here:
- I know k8s Ingress are not recommended, however are they still officially supported, and should my setup work?
- Do anybody knows what the hell is happening and how to fix it? I can use
Gateways to setup the IAP, but when using apps that are available on Helm, it’s easier to use their template and just say
- Last question, not fully related: Is it possible to set the Ingress Gateway that will handle the k8s
- For the first and third questions, I don’t think changing the Ingress Gateway is possible, and that is IMHO such a big omission that either I missed something big, or specifying an
Ingressthis way is close to unsupported (maybe it was supported as a proof of concept, idk);
- For the second question, I think the browser keeps a connection open to the server, and when we load the page on the second domain, the same connection is used, which confuses the hell out of Istio because the domain is not the same. No idea if this theory is right though…
I’m pretty sure I’ve hit an Istio bug, but I’m posting here first to get some feedback regarding this issue.
Thanks for the help!
EDIT: Here are some logs from the ingress gateway pods. As one can see, among other, on the request that fails the requested server name is not the same as the request authority:
Request that works: "GET /alerts HTTP/2" 200 - "-" 0 186731 23 19 "10.132.0.71" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15" "28a22fb3-7faa-4022-abd2-ea5ef1b8cd15" "prometheus.mydomain.com" "10.52.6.23:9090" outbound|9090||kube-prom-stack-kube-prome-prometheus.monitoring.svc.cluster.local 10.52.6.21:35994 10.52.6.21:8443 10.132.0.71:17848 prometheus.mydomain.com - Request that fails: "GET / HTTP/2" 403 - "-" 0 19 0 - "10.132.0.71" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15" "5c3c3f8d-c621-4292-8410-284875312e14" "alertmanager.mydomain.com" "-" - - 10.52.6.21:8443 10.132.0.71:17848 prometheus.mydomain.com -
[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% "%UPSTREAM_TRANSPORT_FAILURE_REASON%" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n