Hello,
I have a working setup that allows me to use OAuth2 to authenticate a user before allowing him access to protected resources.
However, this setup only fully works when I use a Gateway and a VirtualService, not when using k8s Ingress with a custom Ingress Class.
First, let me describe my setup.
- I’m using Istio 1.8.2 (but same behavior was observed w/ 1.7.5 and 1.8.0), installed via the Istio Operator;
- I’m acting on the default
ingress-gateway, w/ the default settings; - I have installed a
RequestAuthentication, anAuthorizationPolicyand anEnvoyFilterwith the following manifests:
# See https://www.blog.jetstack.io/blog/istio-oidc/
---
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: istio-ingressgateway
spec:
selector:
matchLabels:
app: istio-ingressgateway
jwtRules:
# From https://accounts.google.com/.well-known/openid-configuration
- issuer: https://accounts.google.com
jwksUri: https://www.googleapis.com/oauth2/v3/certs
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: istio-ingressgateway
spec:
selector:
matchLabels:
app: istio-ingressgateway
rules:
- from:
- source:
requestPrincipals: ["*"]
---
# https://www.paraesthesia.com/archive/2020/09/03/setting-up-oauth2-proxy-with-istio/
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: istio-ingressgateway
spec:
workloadSelector:
labels:
app: istio-ingressgateway
configPatches:
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: envoy.http_connection_manager
subFilter:
# This filter exists as we create it above using the RequestAuthentication object (IIUC)
name: envoy.filters.http.jwt_authn
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.ext_authz
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
# The RequestAuthentication + AuthorizationPolicy will deny or allow
# requests correctly even if the oauth2 proxy is down
failure_mode_allow: true
http_service:
server_uri:
# URIs here should be to the oauth2-proxy service inside your
# cluster, in the namespace where it was deployed. The port
# in that 'cluster' line should also match up.
uri: "http://oauth2-proxy.oauth2-proxy.svc.cluster.local:4180"
cluster: "outbound|4180||oauth2-proxy.oauth2-proxy.svc.cluster.local"
timeout: 15s
authorization_request:
allowed_headers:
patterns:
- exact: cookie
ignore_case: true
authorization_response:
allowed_upstream_headers:
patterns:
- exact: authorization
ignore_case: true
My problem, specifically occurs when two (or more) k8s Ingresses using two different domains (or more) are setup.
(First, a note, AFAIK it is not possible to specify the Ingress Gateway that will handle the Ingress, so both Ingresses are handled by the same Ingress Gateway; the default one (istio-ingressgateway).)
The behavior is as follow (tested both w/ Safari and Firefox, and even Chromium):
- Load first URL, redirect is done correctly;
- Load second URL, using another browser, still ok;
- Load second URL w/ same browser as first one in another tab (not long after having loaded first URL) -> we get the following response:
RBAC: access deniedw/ HTTP code 403. Note theEnvoyFilteris not even reached; theRequestAuthenticationorAuthorizationPolicydeny the request directly AFAICT.
So I have three questions from here:
- I know k8s Ingress are not recommended, however are they still officially supported, and should my setup work?
- Do anybody knows what the hell is happening and how to fix it? I can use
VirtualServices andGateways to setup the IAP, but when using apps that are available on Helm, it’s easier to use their template and just sayingress.enabled: trueandingress.ingressClassName: istio; - Last question, not fully related: Is it possible to set the Ingress Gateway that will handle the k8s
Ingress?
My thoughts:
- For the first and third questions, I don’t think changing the Ingress Gateway is possible, and that is IMHO such a big omission that either I missed something big, or specifying an
Ingressthis way is close to unsupported (maybe it was supported as a proof of concept, idk); - For the second question, I think the browser keeps a connection open to the server, and when we load the page on the second domain, the same connection is used, which confuses the hell out of Istio because the domain is not the same. No idea if this theory is right though…
I’m pretty sure I’ve hit an Istio bug, but I’m posting here first to get some feedback regarding this issue.
Thanks for the help!
EDIT: Here are some logs from the ingress gateway pods. As one can see, among other, on the request that fails the requested server name is not the same as the request authority:
Request that works: "GET /alerts HTTP/2" 200 - "-" 0 186731 23 19 "10.132.0.71" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15" "28a22fb3-7faa-4022-abd2-ea5ef1b8cd15" "prometheus.mydomain.com" "10.52.6.23:9090" outbound|9090||kube-prom-stack-kube-prome-prometheus.monitoring.svc.cluster.local 10.52.6.21:35994 10.52.6.21:8443 10.132.0.71:17848 prometheus.mydomain.com -
Request that fails: "GET / HTTP/2" 403 - "-" 0 19 0 - "10.132.0.71" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15" "5c3c3f8d-c621-4292-8410-284875312e14" "alertmanager.mydomain.com" "-" - - 10.52.6.21:8443 10.132.0.71:17848 prometheus.mydomain.com -
Log format:
[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% "%UPSTREAM_TRANSPORT_FAILURE_REASON%" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n