GRPC Auth problem with GCP IAP

1

Hi all,

Working with Google Cloud and Kubeflow, I would like to establish a GRPC connection with a pod in my cluster.
My current setup includes a istio-ingressgateway configured with Cloud Identity-Aware Proxy as authorization layer.
I already have in place a virtual service to redirect my traffic from the ingress-gateway to the GRPC pod port.

When I local port-forward to my ingress-gateway http2 port, I am unable to send requests getting the following error:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAUTHENTICATED
        details = "Origin authentication failed"
        debug_error_string = "{"created":"@1587401119.875413000","description":"Error received from peer ipv6:[::1]:8080","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Origin authentication failed","grpc_status":16}"

From the error code I can notice that is a problem in authenticating my GRPC Channel.

The problem looks even stranger when I look to my ingress-gateway logs, where I can see the following for my failed request:

"POST /seldon.protos.Seldon/Predict HTTP/2" 200 - "-" 0 0 0 - "10.52.7.24" "grpc-python/1.27.2 grpc-c/9.0.0 (osx; chttp2; guantao)"

which in the logs looks successful.
My current security policy is:

apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"authentication.istio.io/v1alpha1","kind":"Policy","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"iap-ingress","app.kubernetes.io/instance":"iap-ingress-v1.0.0","app.kubernetes.io/managed-by":"kfctl","app.kubernetes.io/name":"iap-ingress","app.kubernetes.io/part-of":"kubeflow","app.kubernetes.io/version":"v1.0.0","kustomize.component":"iap-ingress"},"name":"ingress-jwt","namespace":"istio-system"},"spec":{"origins":[{"jwt":{"audiences":["TO_BE_PATCHED"],"issuer":"https://cloud.google.com/iap","jwksUri":"https://www.gstatic.com/iap/verify/public_key-jwk","jwtHeaders":["x-goog-iap-jwt-assertion"],"trigger_rules":[{"excluded_paths":[{"exact":"/healthz/ready"},{"prefix":"/.well-known/acme-challenge"}]}]}}],"principalBinding":"USE_ORIGIN","targets":[{"name":"istio-ingressgateway","ports":[{"number":80}]}]}}
  creationTimestamp: "2020-04-18T09:43:46Z"
  generation: 2
  labels:
    app.kubernetes.io/component: iap-ingress
    app.kubernetes.io/instance: iap-ingress-v1.0.0
    app.kubernetes.io/managed-by: kfctl
    app.kubernetes.io/name: iap-ingress
    app.kubernetes.io/part-of: kubeflow
    app.kubernetes.io/version: v1.0.0
    kustomize.component: iap-ingress
  name: ingress-jwt
  namespace: istio-system
  resourceVersion: "8170"
  selfLink: /apis/authentication.istio.io/v1alpha1/namespaces/istio-system/policies/ingress-jwt
  uid: *********
spec:
  origins:
  - jwt:
      audiences:
      - /projects/2********/global/backendServices/
      issuer: https://cloud.google.com/iap
      jwksUri: https://www.gstatic.com/iap/verify/public_key-jwk
      jwtHeaders:
      - x-goog-iap-jwt-assertion
      trigger_rules:
      - excluded_paths:
        - exact: /healthz/ready
        - prefix: /.well-known/acme-challenge
  principalBinding: USE_ORIGIN
  targets:
  - name: istio-ingressgateway
    ports:
    - number: 80

Using REST I was able to authenticate successfully my requests, adding a token to my request following what described here
I’ve tried various ways to authenticate my requests in GRPC with no success:

Is there any example in creating a secure channel with Cloud IAP?

Many Thanks