RBAC returns either 403 or 302 for each route randomly

I am setting up Istio to do RBAC with keycloak as our auth server. I have set, as far as I can tell, the required policy, clusterRbacConfig, ServiceRoleBindings and ServiceRoles however any request to the backend always returns either a 403 or 302 with no response data, the same route and token flips between the two on refresh. I have included both config files for istio (I use one file to configure the services deployments and ingress, and another to do security) as well as a decoded JWT being sent. I have expunged all data specific to our application and replaced it with placeholder data (so you can see where settings / names match). requests are made to http://:443/backend/<PATH 1> as an example. I am using istio V1.1.1 on the cluster and istiocli for windows V1.1.7.

Any help or advice would be greatly appreciated,

security-config yaml:
apiVersion: “authentication.istio.io/v1alpha1
kind: “Policy”
metadata:
name: “require-mtls-jwt”
spec:
targets:

  • name: backend
    peers:
  • mtls: {}
    origins:
  • jwt:
    issuer: “”
    jwksUri: “”
    trigger_rules:
    • excluded_paths:
      • exact: /sign-up
        principalBinding: USE_ORIGIN

apiVersion: “rbac.istio.io/v1alpha1
kind: ClusterRbacConfig
metadata:
name: default
spec:
mode: ‘ON_WITH_INCLUSION’
inclusion:
services: [“backend”]

apiVersion: “rbac.istio.io/v1alpha1
kind: ServiceRoleBinding
metadata:
name: bind-permission-1
namespace: istio-development
spec:
subjects:

  • user: “*”
    properties:
    request.auth.claims[roles]: “<PERMISSION 1>”
    roleRef:
    kind: ServiceRole
    name: “permission-1”

apiVersion: “rbac.istio.io/v1alpha1
kind: ServiceRoleBinding
metadata:
name: bind-permission-2
namespace: istio-development
spec:
subjects:

  • user: “*”
    properties:
    request.auth.claims[roles]: “<PERMISSION 2>”
    roleRef:
    kind: ServiceRole
    name: “permission-2”

apiVersion: “rbac.istio.io/v1alpha1
kind: ServiceRoleBinding
metadata:
name: bind-permission-3
namespace: istio-development
spec:
subjects:

  • user: “*”
    properties:
    request.auth.claims[roles]: “<PERMISSION 3>”
    roleRef:
    kind: ServiceRole
    name: “permission-3”

apiVersion: “rbac.istio.io/v1alpha1
kind: ServiceRoleBinding
metadata:
name: all-users-services-binding
namespace: istio-development
spec:
subjects:

  • user: “*”
    roleRef:
    kind: ServiceRole
    name: “open-resource”

apiVersion: “rbac.istio.io/v1alpha1
kind: ServiceRole
metadata:
name: permission-1
namespace: istio-development
spec:
rules:

  • services: [“backend.istio-development.svc.cluster.local”]
    methods: [“POST”]
    paths: ["/path1", “/path2”]

apiVersion: “rbac.istio.io/v1alpha1
kind: ServiceRole
metadata:
name: permission-2
namespace: istio-development
spec:
rules:

  • services: [“backend.istio-development.svc.cluster.local”]
    methods: [“POST”]
    paths: ["/path3"]

apiVersion: “rbac.istio.io/v1alpha1
kind: ServiceRole
metadata:
name: permission-3
namespace: istio-development
spec:
rules:

  • services: [“backend.istio-development.svc.cluster.local”]
    methods: [“POST”]
    paths: ["/path4"]

apiVersion: “rbac.istio.io/v1alpha1
kind: ServiceRole
metadata:
name: open-resource
namespace: istio-development
spec:
rules:

  • services: [“frontend.istio-development.svc.cluster.local”]

core config yaml:

Copyright 2017 Istio Authors

Licensed under the Apache License, Version 2.0 (the “License”);

##################################################################################################

backend

##################################################################################################

apiVersion: v1
kind: Service
metadata:
name: backend
labels:
app: backend
service: backend
spec:
ports:

  • port: 8737
    name: http
    selector:
    app: backend

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: backend-v1
labels:
app: backend
version: v1
spec:
replicas: 1
template:
metadata:
labels:
app: backend
version: v1
spec:
containers:
- name: backend
image:
imagePullPolicy: Always
env:
- name:
value:
ports:
- containerPort: 8737

apiVersion: v1
kind: Service
metadata:
name: frontend
labels:
app: frontend
service: frontend
spec:
ports:

  • port: 3000
    name: http
    selector:
    app: frontend

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: frontend-v1
labels:
app: frontend
version: v1
spec:
replicas: 1
template:
metadata:
labels:
app: frontend
version: v1
spec:
containers:
- name: frontend
image:
imagePullPolicy: Always
ports:
- containerPort: 3000

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: frontend-gateway
spec:
selector:
istio: ingressgateway # use Istio default gateway implementation
servers:

  • port:
    number: 443
    name: http
    protocol: HTTP
    hosts:
    • “*”

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: frontend
spec:
hosts:

  • “*”
    gateways:
  • frontend-gateway
    http:
  • match:
    • uri:
      exact: “/”
    • uri:
      prefix: “/example-path”
      route:
    • destination:
      port:
      number: 3000
      host: frontend

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: backend
spec:
hosts:

  • “*”
    gateways:
  • frontend-gateway
    http:
  • match:
    • uri:
      prefix: “/backend/”
      rewrite:
      uri: “/”
      route:
    • destination:
      port:
      number: 8737
      host: backend

JWT:
{
“alg”: “RS256”,
“typ”: “JWT”,
“kid”: “”
}
{
“jti”: “bd72df7a-d2b9-41e3-8f5f-b940129033bb”,
“exp”: 1559055836,
“nbf”: 0,
“iat”: 1559054936,
“iss”: “”,
“aud”: “account”,
“sub”: “fbfe5148-99ee-4355-9559-92bf0571408e”,
“typ”: “Bearer”,
“azp”: “backend”,
“nonce”: “fea1abd1-dad4-43fa-9fda-a9fc8a5f3099”,
“auth_time”: 1559054166,
“session_state”: “1f87022f-2b4d-48be-b90a-cc8dc6dc05da”,
“acr”: “0”,
“allowed-origins”: [
",
"http://:443/

],
“realm_access”: {
“roles”: [
“<PERMISSION 1>”,
“<PERMISSION 2>”,
“<PERMISSION 3>”
]
},
“resource_access”: {
“account”: {
“roles”: [
“manage-account”,
“manage-account-links”,
“view-profile”
]
}
},
“scope”: “openid profile email”,
“email_verified”: true,
“roles”: [
“<PERMISSION 1>”,
“<PERMISSION 2>”,
“<PERMISSION 3>”
],
“name”: “”,
“preferred_username”: “”,
“given_name”: “”,
“family_name”: “”,
“email”: “”
}

Looking at your ClusterRbacConfig, would you mind to use the full service name? (e.g. backend.istio-development.svc.cluster.local)

Not at all, I have made that change though there is no change in the responses from istio.
Thank you for the advice.

Small update, I have removed the ‘mtls: {}’ as it was causing consistant 503s, I am back to sometimes 403 sometimes 302s.