JWT first party/third party

1

Hi,

I can’t figure out a way to have istio working with third party JWT. Best I’ve got so is to be able to make pods to start, but then istiod is logging the following:

{"level":"warn","time":"2022-05-17T06:43:34.443111Z","msg":"Received token with aud [\"https://kubernetes.default.svc.cluster.local\" \"istio-ca\"], but expected 'kubernetes.default.svc'. BoundServiceAccountTokenVolume, default in Kubernetes 1.21+, is not compatible with first-party-jwt"}

Comments in the source code give more background to it, but even then, i’m unable to understand what “a good” configuration should look like.

Could please anyone point me to the right direction and tell me what I am missing or what’s wrong?

ADITIONAL INFO

The tokens mounted in istio-proxy containers look like this:

$ kubectl --context k8slab -n test-jwt exec ratings-v1-b6994bb9-2pz29 -c istio-proxy -- cat /var/run/secrets/kubernetes.io/serviceaccount/token
{
  "aud": [
    "https://kubernetes.default.svc.cluster.local"
  ],
  "exp": 1684305330,
  "iat": 1652769330,
  "iss": "https://kubernetes.default.svc.cluster.local",
  "kubernetes.io": {
    "namespace": "test-jwt",
    "pod": {
      "name": "ratings-v1-b6994bb9-2pz29",
      "uid": "5693c9af-4fb8-47a3-8481-d953e0ad852c"
    },
    "serviceaccount": {
      "name": "bookinfo-ratings",
      "uid": "f4808b19-e6f1-4b2a-a84e-5ee57fb3fe76"
    },
    "warnafter": 1652772937
  },
  "nbf": 1652769330,
  "sub": "system:serviceaccount:test-jwt:bookinfo-ratings"
}

Our cluster supports third-party:

$ kubectl get --raw /api/v1 | jq '.resources[] | select(.name | index("serviceaccounts/token"))'
{
  "name": "serviceaccounts/token",
  "singularName": "",
  "namespaced": true,
  "group": "authentication.k8s.io",
  "version": "v1",
  "kind": "TokenRequest",
  "verbs": [
    "create"
  ]
}

Master api-server relevant configuration looks like this:

$ cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep "service-account"
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key

Relevant jwt configuration of istiod-1-13-2 deployment is:

        env:
        - name: REVISION
          value: 1-13-2
        - name: JWT_POLICY
          value: third-party-jwt
        - name: TOKEN_AUDIENCES
          value: https://kubernetes.default.svc.cluster.local,istio-ca
        - name: PILOT_CERT_PROVIDER

Our kubernetes cluster version is:

$ kubectl version --short
Client Version: v1.21.12
Server Version: v1.21.11
2

Just in case someone else faces this. Problem was triggered by us, by manually changing the envs of the istiod deployment to start using “third-party jwt policy”. Istio needs more things to be modified for this to work as expected, and that something is the mounting of specific istio tokens in gateways and in istio-proxy (via mutatingwebhook). You should be able to tell what was missing by just running istioctl manifest generate based on a “first-party” and then a second one with a “third-party” and compare the differences.