How to tell pilot to fetch JWKS certs using mTLS?

Security
1

Hi,

I have an IDP (keycloak) running in an istio-enabled namespace. The service has a policy like this:

apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  name: keycloak-jwt-policy
  namespace: public
spec:
  peers:
  - mtls:
      mode: STRICT
  targets:
  - name: keycloak
    ports:
    - containerPort: 8080
      name: http

The cluster-default DestinationRule is:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: default
  namespace: istio-system
spec:
  host: '*.local'
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

Istio is installed using helm and has controlPlaneSecurityEnabled: true
However I’m seeing a lot of errors from discovery when fetching JWKS certs from keycloak
Failed to fetch public key from ... which comes from this file: https://github.com/istio/istio/blob/f2222fe2c3c1543cb90562fa03d7887191b37844/pilot/pkg/model/jwks_resolver.go#L242

My hunch is that Pilot is not using mTLS to fetch the JWKS certs - and indeed if I exec into the pilot pod I’m not able to fetch the certificates.
If I use mTLS: PERMISSIVE for keycloak then Pilot can fetch the certificates, but I would rather avoid this.

How should my DestinationRule, Policy and MeshPolicy to instruct Pilot to use mTLS?

2

I noticed the same issue three months ago and could not find a solution yet. As far as I know, fetching JWKS certs using mTLS is not supported yet.

3

I talked with some folks and confirmed that we don’t go through sidecar by default when Pilot is fetching the Jwt public key, so then obviously mTLS fetching a IdP within cluster wont work…

And In future release, Pilot will use kube apiserver signed DNS cert. won’t have a sidecar.

I also remeber some customers are requiring to use customized key/cert when fetching public key (their IdP maybe is a normal https, with cert signed by internal CA). Maybe we can have some discussion/proposal around how to make this cutomizable…

/fyi @YangminZhu @diemtvu

4

From my perspective it would be very nice to make fetching of JWT public keys customizable.
My use-case is the following:

  • Istio ingress gateway with mTLS to downstream services
  • IdP issuing JWTs running mTLS, but without JWT checking, as this would cause a chicken-and-egg problem.
  • All services running mTLS and JWT checking from the IdP
  • controlPlaneSecurityEnabled: true

So the only thing in this setup that doesn’t support mTLS is Pilot. It seems un-intiutive that the IdP needs to run in Permissive mTLS mode instead of Strict.

Until we have a proposal for the above I think it should be mentioned in the documentation regarding JWT Policies that the JWKS URI cannot use mTLS.
It took quite some debugging to find out that mTLS was the problem :slight_smile:

5

@incfly Is there any update on mTLS for Pilot when fetching the JWT public key or atleast support for customized key/cert in environments where the cert is signed by an internal CA?

1 Like
6

Running into this issue too. Did you find a solution? Thanks