Local JWKS HTTP service

1

I am having an issue with getting an authorization policy to work when it uses a JWKS served by an HTTP service in the mesh.

I set up a HTTP server in a service named jwws. I am able to reach it through the ingress using a virtual service that specifies

    route:
    - destination:
        host: jwks.default.svc.cluster.local
        port:
          number: 80

In my authorization policy if I specify
jwksUri: "http://jwks.default.svc.cluster.local/api-gateway.json"
requests get a 401 status returned.

If I change the URL to one external to the mesh, requests get a normal response. The content is the same in both the internal and external JWKS. Why would the internal address not be working?

2

Could you share your authorization, routing and deployment configuration?

3

It’s the pilot to fetch the public key from the jwksUri, so make sure your jwks.default.svc.cluster.local is accessible from pilot.

4

Were you able to solve this issue?

5

I gave up and developed a solution that did not involves an external JWKS. I am still concerned about how that will work when we get around to locking down egress.

6

For the initial issue, did you specify any JWT authentication policy for jwks.default.svc.cluster.local? The 401 could be returned because pilot doesn’t attach the JWT token to the request.

7

My scenario is a little different. I do not get any response from the gateway, it looks like the envoy proxy is crashing.

Configuration:

image
image.png838×173 5.53 KB

Response:
image
image.png1602×529 35.1 KB

Proxy log:
image
image.jpg1570×610 684 KB

Pilot log:
image
image.jpg1885×293 202 KB

If try getting through the gateway without a Authorization header I get a 401 unauthorized response, as it should be, which probably means the problem is when istio JWT Authentication try to get the public key to validate the token.

I tried it with a service managed by istio and with a vanilla kubernetes service without the sidecar, same problem.

8

Hi:
I’m using istio 1.2.2 in kubernetes by windows’ docker client. Helm default installation. And I’m having the same issure.
I configured a Policy like this:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "jwt-policy"
  namespace: team-b
spec:
  targets:
  - name: istio-test-server
  origins:
  - jwt:
      audiences:
      - "https://localhost/"
      issuer: "localhost"
      jwksUri: "http://jwt-service.platform.svc.cluster.local:8080/jwks"
      trigger_rules:
      - excluded_paths:
        - exact: /version
  principalBinding: USE_ORIGIN
  peers:
  - mtls: {}

The related logs in istio-pilot’s discovery container are:

2019-07-11T02:36:31.638699Z     error   model   Failed to fetch public key from "http://jwt-service.platform.svc.cluster.local:8080/jwks": Get http://jwt-service.platform.svc.cluster.local:8080/jwks: read tcp 10.1.0.200:35974->10.105.89.144:8080: read: connection reset by peer
2019-07-11T02:36:31.638734Z     warn    Failed to fetch jwt public key from "http://jwt-service.platform.svc.cluster.local:8080/jwks"

I exec into the istio-pilot container and find that I cannot curl to any services managed by istio. Like this:

curl http://jwt-service.platform.svc.cluster.local:8080/jwks

In that container, curl returns:

curl: (56) Recv failure: Connection reset by peer

Then I tried:

dig jwt-service.platform

And returns the correct IP, but still, curl to that IP returns the same curl: (56) Recv failure: Connection reset by peer.

I believe I have configured the rbac, destinationrules and policies correctly because I can curl that service from other service deployed by myself in cluster this way.

I wonder if jwksUri must be an external url address? How to get the local internal jwks server working?

9

The error read: connection reset by peer usually means it’s rejected by TLS due to incorrect configuration. Could you copy the destination rule, authentication policy and the service definition for jwt-service.platform?

Did you enable mTLS when installing Istio? please make sure your destination rule is configured to use mTLS for the jwt-service.platform.

10

It’s currently not possible to host JWKS on a server inside the mesh if you’re not using permissive mode. The reason is that the discovery container inside the istio-pilot pod retrieves the JWKS using a standard HTTP(S) client and does not go through an Envoy sidecar.

What I did was host the JWKS in a separate namespace outside the mesh.

11

Thanks for the remind @dgn.

@agasbzj sorry I forgot that the discovery container doesn’t go through the envoy, so you actually need to make sure you have an authentication policy that disables mTLS for the jwt-service.platform.svc.cluster.local.

12

Hi,
iIm pretty new to istio, i’m experimenting the same issue.

How can I get istio-pilot logs to check if the error is the same?

Best,
A