503 error with requestPrincipals in AuthorizationPlicy (istio 1.5.2)

Security
1

Hello, I am having a problem, and I was someone had some insights on it

Basically, I have the 2 following Security Rules :

apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
  name: "jwt-setup"
  namespace: dev
spec:
  selector:
    matchLabels:
      k8s-app: myapp-server
  jwtRules:
  - issuer: "mycompany"
    jwksuri: "mycompany/jwts.json"
    forwardOriginalToken: true
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: myapp-policy
 namespace: dev
spec:
 action: ALLOW
 selector:
   matchLabels:
     k8s-app: myapp-server
 rules:
 - from:
   - source:
       requestPrincipals: ["*"]

As soon as I add those 2 in my setup, every single one of my calls end up with a 503: upstream connect error or disconnect/reset before headers. reset reason: connection termination

If I simply change : requestPrincipals: ["*"] to principals: ["*"], everything goes back to accept all connections

Basically, everywhere I looked, it seems like this should work, but simply doesn’t …

I was wondering what exactly is happening here, am I missing some configuration?
Is there any way to debug what’s happening so I can see what happens to my call?

Some extra information :

  • dev.mycompany points to the ingress-gateway in istio-system
  • Install was done using istioctl manifest apply --set profile=demo
  • I have tried adding the previous security calls directly on istio-ingressgateway instead of myapp, it works there, but allows all traffic, ignoring my JWT config altogether

Virtual Service :

kind: VirtualService
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: myapp-virtual-service
  namespace: dev
spec:
  hosts:
    - dev.mycompany
  gateways:
    - myapp-gateway
  http:
    - match:
        - method:
            exact: POST
          uri:
            prefix: /api
      name: myapp-server
      rewrite:
        uri: /graphql
      route:
        - destination:
            host: myapp-service
            port:
              number: 5000

Gateway :

kind: Gateway
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: myapp-gateway
  namespace: dev
spec:
  servers:
    - hosts:
        - dev.mycompany
      port:
        name: http
        number: 80
        protocol: HTTP
    - hosts:
        - dev.mycompany
      port:
        name: https
        number: 443
        protocol: HTTP
  selector:
    istio: ingressgateway

Let me know if you need more information

2

cc @YangminZhu, @liminwang

3

It does not look like authorization issue. Can you check if the jwksUri (“mycompany/jwts.json”) is reachable from inside cluster?

4

Yep, I have gone through all istio pods and my app pods and curl on that URI, every single node can access it

I’m also trying with it (or with jwks directly) and I am getting exactly the same problem

Example of my jwks
https://raw.githubusercontent.com/sbolduc-reactive/test_files/master/jwts.json

Also, another question would be, is there a way to put Istio in trace/debug mode so I can see the calls it does and maybe debug it from there?

5

Hi @sbolduc

AFAIK Authorization policy doen’t give 503 error
requestPrincipal & principal are different things.
see here https://istio.io/docs/reference/config/security/authorization-policy/#Source

Istio constructs the requestPrincipal by combining the iss and sub of the JWT token with a / separator. see this task.

May be this will help you

6

I understand that, but requestPrincipas should give me a 401 unauthorized if I had a token problem, not a 503

Which is why I am wondering if there is some way to debug the pathways that Istio use. If so, I can try to debug, and will probably notice something completely unrelated (as it happen with these kinds of bugs :joy:)

Thanks