Jwt Authentication for a specific service results in "upstream connect error or disconnect/reset before headers"

[chaturvedia]

I have a sample helloweb service deployed in default namespace that I can access through ingress-gateway

If I set a jwtPolicy for the whole ingress-gateway I can see the JWT validation happens fine and with a valid Jwt token I can get access to this service.

Policy (Global JWT Policy works)

apiVersion: “authentication.istio.io/v1alpha1”
kind: “Policy”
metadata:
name: “ingressgateway”
namespace: istio-system
spec:
targets:

• name: istio-ingressgateway
  origins:
• jwt:
  issuer: “service-gateway”
  jwksUri: “https://raw.githubusercontent.com/chaturvedia/jwkstest/master/jwks.json”
  principalBinding: USE_ORIGIN

If however I try to set a Policy for just this service I get “upstream connect error or disconnect/reset before headers”

Policy for just that service like below (does not work)

apiVersion: “authentication.istio.io/v1alpha1”
kind: “Policy”
metadata:
name: “helloweb-policy”
spec:
targets:

• name: helloweb-backend
  ports:
  • number: 8080
    origins:
• jwt:
  issuer: “service-gateway”
  jwksUri: “https://raw.githubusercontent.com/chaturvedia/jwkstest/master/jwks.json”
  principalBinding: USE_ORIGIN

Looking at the container logs of istio-proxy for helloweb-backend does not show any details.

[YangminZhu]

What’s your TLS setting for helloweb-backend? Did you enable mTLS in the auth policy by default? If that’s the case, you need to modify your “helloweb-policy” to include the mTLS setting as well (or you can modify the destination rule to exclude the helloweb-backend from mTLS)

cc @diemtvu

[pbhoj]

I am working with @chaturvedia. mTLS is enabled. However, when we add mTLS setting to the helloweb-policy (as shown below), all requests go through, even without a JWT token.

apiVersion: “authentication.istio.io/v1alpha1”
kind: “Policy”
metadata:
name: “helloweb-policy”
spec:
targets:

• name: “helloweb-backend”
  ports:
  • number: 8080
    peers:
• mtls: {}
  origins:
• jwt:
  issuer: “service-gateway”
  jwksUri: “https://raw.githubusercontent.com/chaturvedia/jwkstest/master/jwks.json”
  principalBinding: USE_ORIGIN

[chaturvedia]

Update on this issue. Service port needs name to be specified as per Istio guidelines. Once the service port name is added jwt auth works fine

[divya21raj]

I’ve been stuck on a similar issue for quite some time… Could you please share your final file? It would help a lot. Thanks!

[YangminZhu]

I think @chaturvedia solved this by naming the service port according to Istio guidelines, could you double check if this applies to you? And you can also check your authentication policy to make sure it has both mTLS and JWT spec.

[divya21raj]

Here’s my auth policy ->

apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
name: auth-policy
spec:
targets:

• name: sa-web-app
  peers:
• mtls: {}
  origins:
• jwt:
  issuer: “<my_issuer>”
  jwksUri: “https://raw.githubusercontent.com/divya21raj/Istio-Powered-Sentiment-Analysis/master/pub_key.json”
  principalBinding: USE_ORIGIN

Without the mtls bit, I get the same error with reason being connection reset. With it, I get connection termination.

I don’t understand where service port comes into play.

[divya21raj]

Correction - Without the mtls bit, I get connection faliure, not connection reset…

[diemtvu]

@chaturvedia

olicy for the whole ingress-gateway I can see the JWT validation happens fine and with a valid Jwt token I can get access to this servic

Can you send us both the destination rules and policy in your system? Does it work if you keep the policy but remove the origins (jwt) part, or remove the policy completely?

My theory is the destination rule may have incorrect TLS settings.