Exposing gRPC service externally results in Access Denied errors

1

Kubernetes 1.19 and Istio 1.7.8 (We still require Mixer), using the AWS Load Balancer controller

Ingress Gateway Service config:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
    service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  - service.k8s.aws/resources
  labels:
    app: istio-ingressgateway
    install.operator.istio.io/owning-resource: mtribes
    install.operator.istio.io/owning-resource-namespace: istio-system
    istio: ingressgateway
    istio.io/rev: default
    operator.istio.io/component: IngressGateways
    operator.istio.io/managed: Reconcile
    operator.istio.io/version: 1.7.8
  name: istio-ingressgateway
  namespace: istio-system
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: status-port
    nodePort: 30051
    port: 15021
    protocol: TCP
    targetPort: 15021
  - name: http2
    nodePort: 30551
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    nodePort: 31051
    port: 443
    protocol: TCP
    targetPort: 8443
  - name: tls
    nodePort: 31551
    port: 15443
    protocol: TCP
    targetPort: 15443
  - name: grpc-test
    nodePort: 32051
    port: 7766
    protocol: TCP
    targetPort: 7766
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  sessionAffinity: None
  type: LoadBalancer

We use this for local development of services so they can contact a dev environment on gRPC. The http logs of the ingress gateway show ‘200 POST’, everything looks fine the requests stop there. Using something like Bloom we see the following response:

Response :{  “error”: “7 PERMISSION_DENIED: RBAC: access denied”}

I can not find any config for this, how can i get envoy to allow it (or is it something else)?

There is an active authorization policy on this environment which whitelist IPS to hosts. I did try removing that policy quickly to see if we still received this error and we do. So it is not related. All http traffic works fine (even the gRPC traffic, as mentioned we see the 200 POST in our logs).

2

cc @YangminZhu for RBAC access denied.

3

We worked this out. We added to the top of our auth policy:

  - from:
    - source:
        ipBlocks:
        - 1.1.1.1/32
        - 2.2.2.2/32
    to:
    - operation:
        hosts:
        - hub.example.dev:7766
        methods:
        - POST
        ports:
        - "7766"

And this handles the new port / grpc traffic over the http traffic.