Local JWKS HTTP service

I am having an issue with getting an authorization policy to work when it uses a JWKS served by an HTTP service in the mesh.

I set up a HTTP server in a service named jwws. I am able to reach it through the ingress using a virtual service that specifies

    - destination:
        host: jwks.default.svc.cluster.local
          number: 80

In my authorization policy if I specify
jwksUri: "http://jwks.default.svc.cluster.local/api-gateway.json"
requests get a 401 status returned.

If I change the URL to one external to the mesh, requests get a normal response. The content is the same in both the internal and external JWKS. Why would the internal address not be working?

Could you share your authorization, routing and deployment configuration?

It’s the pilot to fetch the public key from the jwksUri, so make sure your jwks.default.svc.cluster.local is accessible from pilot.

Were you able to solve this issue?

I gave up and developed a solution that did not involves an external JWKS. I am still concerned about how that will work when we get around to locking down egress.

For the initial issue, did you specify any JWT authentication policy for jwks.default.svc.cluster.local? The 401 could be returned because pilot doesn’t attach the JWT token to the request.

My scenario is a little different. I do not get any response from the gateway, it looks like the envoy proxy is crashing.



Proxy log:

Pilot log:

If try getting through the gateway without a Authorization header I get a 401 unauthorized response, as it should be, which probably means the problem is when istio JWT Authentication try to get the public key to validate the token.

I tried it with a service managed by istio and with a vanilla kubernetes service without the sidecar, same problem.

I’m using istio 1.2.2 in kubernetes by windows’ docker client. Helm default installation. And I’m having the same issure.
I configured a Policy like this:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
  name: "jwt-policy"
  namespace: team-b
  - name: istio-test-server
  - jwt:
      - "https://localhost/"
      issuer: "localhost"
      jwksUri: "http://jwt-service.platform.svc.cluster.local:8080/jwks"
      - excluded_paths:
        - exact: /version
  principalBinding: USE_ORIGIN
  - mtls: {}

The related logs in istio-pilot’s discovery container are:

2019-07-11T02:36:31.638699Z     error   model   Failed to fetch public key from "http://jwt-service.platform.svc.cluster.local:8080/jwks": Get http://jwt-service.platform.svc.cluster.local:8080/jwks: read tcp> read: connection reset by peer
2019-07-11T02:36:31.638734Z     warn    Failed to fetch jwt public key from "http://jwt-service.platform.svc.cluster.local:8080/jwks"

I exec into the istio-pilot container and find that I cannot curl to any services managed by istio. Like this:

curl http://jwt-service.platform.svc.cluster.local:8080/jwks

In that container, curl returns:

curl: (56) Recv failure: Connection reset by peer

Then I tried:

dig jwt-service.platform

And returns the correct IP, but still, curl to that IP returns the same curl: (56) Recv failure: Connection reset by peer.

I believe I have configured the rbac, destinationrules and policies correctly because I can curl that service from other service deployed by myself in cluster this way.

I wonder if jwksUri must be an external url address? How to get the local internal jwks server working?

The error read: connection reset by peer usually means it’s rejected by TLS due to incorrect configuration. Could you copy the destination rule, authentication policy and the service definition for jwt-service.platform?

Did you enable mTLS when installing Istio? please make sure your destination rule is configured to use mTLS for the jwt-service.platform.

It’s currently not possible to host JWKS on a server inside the mesh if you’re not using permissive mode. The reason is that the discovery container inside the istio-pilot pod retrieves the JWKS using a standard HTTP(S) client and does not go through an Envoy sidecar.

What I did was host the JWKS in a separate namespace outside the mesh.

Thanks for the remind @dgn.

@agasbzj sorry I forgot that the discovery container doesn’t go through the envoy, so you actually need to make sure you have an authentication policy that disables mTLS for the jwt-service.platform.svc.cluster.local.

iIm pretty new to istio, i’m experimenting the same issue.

How can I get istio-pilot logs to check if the error is the same?