Istio 1.8.2. RBAC: access denied. JWT is valid

Security
1

$ istioctl version --remote

client version: 1.8.2
control plane version: 1.8.2
data plane version: 1.8.2 (9 proxies)

$ kubectl version --short

Client Version: v1.18.10
Server Version: v1.18.10

Istio was installed using Helm.

$ helm version --short

v3.3.1+g249e521

my JWT:

{
  ...
  "iss": "https://test.qq/auth/realms/master",
  ...
}

$ cat test_auth_pol.yaml

---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: jwt-allow-platform-admin
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: front-admin
  action: ALLOW
  rules:
  - when:
    - key: request.auth.claims[iss]
      values: ["*"]

$ kubectl apply -f test_auth_pol.yaml

$ kubectl logs istiod-c578b5969-hfhlt -n istio-system -f|grep authorization

2021-01-30T05:21:11.629653Z	debug	authorization	builder/builder.go:141	rule ns[istio-system]-policy[jwt-allow-platform-admin]-rule[0] generated policy: permissions:{and_rules:{rules:{any:true}}}  principals:{and_ids:{ids:{or_ids:{ids:{metadata:{filter:"istio_authn"  path:{key:"request.auth.claims"}  path:{key:"iss"}  value:{list_match:{one_of:{string_match:{safe_regex:{google_re2:{}  regex:".+"}}}}}}}}}}}

$ kubectl exec front-admin-5c6c5bb7c5-pcs8g -c istio-proxy -n cp-front-admin – pilot-agent request GET config_dump|less

                  "principals": [
                   {
                    "and_ids": {
                     "ids": [
                      {
                       "or_ids": {
                        "ids": [
                         {
                          "metadata": {
                           "filter": "istio_authn",
                           "path": [
                            {
                             "key": "request.auth.claims"
                            },
                            {
                             "key": "iss"
                            }
                           ],
                           "value": {
                            "list_match": {
                             "one_of": {
                              "string_match": {
                               "safe_regex": {
                                "google_re2": {},
                                "regex": ".+"
                               }
                              }
                             }
                            }
                           }
                          }
                         }
                        ]
                       }
                      }
                     ]
                    }
                   }
                  ]

But after successful authentication in keycloak, I get “RBAC: access denied”.
Everything goes fine if I go to other services in the service mesh.
I’ve tried several variations of request.auth.claims, but none of them work. I took the data from the valid JWT. The data was not nested.

$ kubectl logs front-admin-5c6c5bb7c5-pcs8g -c istio-proxy -n cp-front-admin -f

2021-01-30T04:30:50.854804Z	debug	envoy rbac	enforced denied

Any ideas on how can I fix this behavior?

2

What is your RequestAuthentication for the JWT issuer? and could you add the full Envoy debug log, it should tell which step goes wrong.

3

RequestAuthentication:

---
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: jwt-keycloak
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  jwtRules:
  - issuer: "https://test.qq/auth/realms/master"
    jwksUri: "https://test.qq/auth/realms/master/protocol/openid-connect/certs"
    forwardOriginalToken: true

Envoy’s debug logs:

2021-02-03T08:06:49.222111Z	debug	envoy rbac	checking request: requestedServerName: outbound_.80_._.front-admin.cp-front-admin.svc.stage.cp.local, sourceIP: ip:39288, directRemoteIP: ip:39288, remoteIP: ip:0,localAddress: ip:80, ssl: uriSanPeerCertificate: spiffe://stage.cp.local/ns/istio-system/sa/istio-ingressgateway-service-account, dnsSanPeerCertificate: , subjectPeerCertificate: , headers: ':authority', 'console-stage.cp.4cloud.cf'
':path', '/'
':method', 'GET'
'pragma', 'no-cache'
'cache-control', 'no-cache'
'sec-ch-ua', '"Chromium";v="88", "Google Chrome";v="88", ";Not A Brand";v="99"'
'sec-ch-ua-mobile', '?0'
'upgrade-insecure-requests', '1'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36'
'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
'sec-fetch-site', 'same-site'
'sec-fetch-mode', 'navigate'
'sec-fetch-user', '?1'
'sec-fetch-dest', 'document'
'accept-encoding', 'gzip, deflate, br'
'accept-language', 'ru-RU,ru;q=0.9'
'cookie', '__Host-authservice-session-id-cookie=<valid_cookie>'
'x-forwarded-proto', 'http'
'x-forwarded-for', 'ip,ip'
'x-envoy-external-address', 'ip'
'x-request-id', '5d741ab5-073c-415e-b028-7340787714b0'
'authorization', 'Bearer <valid_token>'
'x-envoy-attempt-count', '1'
'x-b3-traceid', '5fc27a0883646dbc6a2b7f70d429a78c'
'x-b3-spanid', '6a2b7f70d429a78c'
'x-b3-sampled', '0'
'x-envoy-original-path', '/admin'
'content-length', '0'
'x-forwarded-client-cert', 'By=spiffe://stage.cp.local/ns/cp-front-admin/sa/default;Hash=hash;Subject="";URI=spiffe://stage.cp.local/ns/istio-system/sa/istio-ingressgateway-service-account'
, dynamicMetadata: filter_metadata {
  key: "istio_authn"
  value {
    fields {
      key: "request.auth.principal"
      value {
        string_value: "stage.cp.local/ns/istio-system/sa/istio-ingressgateway-service-account"
      }
    }
    fields {
      key: "source.namespace"
      value {
        string_value: "istio-system"
      }
    }
    fields {
      key: "source.principal"
      value {
        string_value: "stage.cp.local/ns/istio-system/sa/istio-ingressgateway-service-account"
      }
    }
    fields {
      key: "source.user"
      value {
        string_value: "stage.cp.local/ns/istio-system/sa/istio-ingressgateway-service-account"
      }
    }
  }
}

2021-02-03T08:06:49.222167Z	debug	envoy rbac	enforced denied, matched policy none
4

looks like your JWT token is not verified at all, does https://test.qq/auth/realms/master use a self-signed cert? You could check the istiod log to verify if it is able to fetch the jwksUri.

5

Thank you for your answer.
Issuer certificate issued by Let’s Encrypt. The test.qq domain is not real, it has been modified.

Last time it did not work because RequestAuthentication was always at the ingressgateway level, and the rule was at the application level.
Now I tried to remove everything related to authentication (works (starts up without authentication)), and add RequestAuthentication to the application level - RequestAuthentication does not work at the application level, but it works if i set RequestAuthentication only at the gateway level. I still can’t figure out why = (

RequestAuthentication at the ingressgateway level can be seen in the comment above.
RequestAuthentication at the app level can be seen below.

---
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: jwt-keycloak
  namespace: cp-testcicd
spec:
  jwtRules:
  - issuer: "https://test.qq/auth/realms/master"
    jwksUri: "https://test.qq/auth/realms/master/protocol/openid-connect/certs"
    forwardOriginalToken: true

Any ideas on how can I get RequestAuthentication to work at the application level?

6

It works if I set RequestAuthentication to both levels (application and ingressgateway).
AuthorizationPolicy also works at the application level.
Thank you for help :slight_smile:

7

How did you define both levels? In the same yml file or in two different?
Can you output the yml file?
Thanks in advance :slight_smile: Dimi