End user authentication using JWT throws 401 unauthorized for few attempts before being successful

Our kubernetes cluster is running Istio 1.4.0. End user authentication is setup as below

kind: Policy
apiVersion: authentication.istio.io/v1alpha1
metadata:
  name: k8s-auth-policy
  namespace: products
spec:
  targets:
    - name: products-svc
  origins:
    - jwt:
        issuer: 'https://sts.windows.net/<tenentid>/'
        jwksUri: >-
          https://login.microsoftonline.com/<tenentid>/discovery/v2.0/keys
        triggerRules:
          - excludedPaths:
              - prefix: /swagger/

Client obtain the auth token from auth provider and its issued by is same as https://sts.windows.net/<tenentid>/ first few calls after obtaining new jwt token fails with message
Origin authentication failed. and after a while all the calls succeeds. There are 10 istio sidecars running for the service products-svc.

Any reason why this is happening?

Reference resources