gRPC Validation loophole? [solved]

Security
1

When using the gRPC validation features within the Gateway it appears that incorrectly formatted JWT headers are ignored these are then allowed to flow into the services. In the following case we have a poorly formatted value getting past the gateway.

Is this a problem.

Client View

grpc_cli call $CLUSTER_INGRESS dev.cognizant_ai.experiment.Service.Get "uid: '1'" --metadata authorization:"Bearer 1"
connecting to aa39e9512911911e9b04006555a6c797-546098767.us-west-2.elb.amazonaws.com:80
Sending client initial metadata:
authorization : Bearer 1
Received trailing metadata from server:
date : Mon, 17 Jun 2019 17:48:34 GMT
server : istio-envoy
x-envoy-upstream-service-time : 1
Rpc failed with status code 16, error message: square/go-jose: compact JWS format must have three parts token="...arer 1" stack="[auth.go:136 auth.go:192 experimentsrv.pb.go:416 server.go:900 server.go:1122 server.go:617]"

Application service view

metadata.MD{"x-b3-traceid":[]string{"be8d87a1d4059fab20fa7f1b6100d40f"}, "grpc-accept-encoding":[]string{"identity,deflate,gzip"}, "accept-encoding":[]string{"identity,gzip"}, "x-envoy-external-address":[]string{"100.96.1.1"}, "x-forwarded-client-cert":[]string{"By=spiffe://cluster.local/ns/default/sa/default;Hash=d84eb4500c490da5b446c8d8624a3a871f7806d2edff227274077ea8d69156ec;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"}, "authorization":[]string{"Bearer 1"}, "x-forwarded-for":[]string{"100.96.1.1"}, "x-request-id":[]string{"63dc3828-6529-43e5-98ab-6e9d9ef9aa86"}, ":authority":[]string{"aa39e9512911911e9b04006555a6c797-546098767.us-west-2.elb.amazonaws.com:80"}, "x-forwarded-proto":[]string{"http"}, "x-b3-spanid":[]string{"a44b08f39f64c5a7"}, "x-b3-sampled":[]string{"0"}, "user-agent":[]string{"grpc-c++/1.21.2 grpc-c/7.0.0 (linux; chttp2; gandalf)"}, "x-b3-parentspanid":[]string{"20fa7f1b6100d40f"}}

JWT values that are empty or missing are rejected at the gateway.

grpc_cli call $CLUSTER_INGRESS dev.cognizant_ai.experiment.Service.Get "uid: '1'"                                    
connecting to aa39e9512911911e9b04006555a6c797-546098767.us-west-2.elb.amazonaws.com:80
Received trailing metadata from server:
date : Mon, 17 Jun 2019 17:49:16 GMT
server : istio-envoy
x-envoy-upstream-service-time : 0
Rpc failed with status code 16, error message: invalid security token

Correctly formatted Bearer values appear to be tested and if invalid the requests never terminate on the service.

grpc_cli call $CLUSTER_INGRESS dev.cognizant_ai.experiment.Service.Get "uid: '1'"                                    
connecting to aa39e9512911911e9b04006555a6c797-546098767.us-west-2.elb.amazonaws.com:80
Received trailing metadata from server:
date : Mon, 17 Jun 2019 17:49:16 GMT
server : istio-envoy
x-envoy-upstream-service-time : 0
Rpc failed with status code 16, error message: invalid security token
2

Can you clarify the configuration you use? Do you mean using Istio gateway API to enable this enforcement? Afaik, we don’t have such validation option in gateway API. May be I misunderstand something?

Thanks.

3

Here is the general YAML setup for using the gateway to validate the JWT. I believe that the gateway is doing something as it rejects empty tokens. There are also unit tests for validation however I dont know if these are for internal usage of JWT. c.f. https://istio.io/docs/ops/security/end-user-auth/

apiVersion: rbac.istio.io/v1alpha1
kind: ClusterRbacConfig
metadata:
  name: default
spec:
  mode: 'ON_WITH_INCLUSION'
  inclusion:
    namespaces: ["default"]
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: ingress-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP2
    hosts:
    - "*"
#
# Authentication is still an alpha feature so retain the service side validation and make
# mesh authentication optional as it is not yet stable
#
---
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  name: experiment-auth-policy
spec:
  targets:
  - name: ingress-gateway
  peers:
  - mtls: {}
  origins:
  - jwt:
      issuer: "https://cognizant-ai.auth0.com/"
      jwksUri: "https://cognizant-ai.auth0.com/.well-known/jwks.json"
      audiences:
        - "http://api.cognizant-ai.dev/experimentsrv"
  principalBinding: USE_ORIGIN
---
apiVersion: rbac.istio.io/v1alpha1
kind: ServiceRole
metadata:
  name: service-user
spec:
  rules:
  - services: ["*"]
    paths: ["*"]
    methods: ["*"]
---
apiVersion: rbac.istio.io/v1alpha1
kind: ServiceRoleBinding
metadata:
  name: all-authenticated-users
spec:
  subjects:
  - properties:
      source.principal: "*"
#      request.auth.claims[scope]: "all:experiments"
  roleRef:
    kind: ServiceRole
    name: service-user
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grpc-experiment-service
spec:
  gateways:
  - ingress-gateway
  hosts:
  - "*"
  http:
  - match:
    - uri:
        prefix: "/dev.cognizant_ai.experiment.Service/"
    - uri:
       prefix: "/grpc.reflection.v1alpha.ServerReflection/"
    route:
    - destination:
        host: experiment.default.svc.cluster.local
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: experiment
spec:
  host: experiment
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
---
apiVersion: v1
kind: Service
metadata:
  name: experiment
  labels:
    app: experiment
spec:
  ports:
  - port: 30001
    name: grpc-exp
    targetPort: 30001
  selector:
    app: experiment
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: experiment-v1
  labels:
    version: v1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: experiment
        version: v1
    spec:
      containers:
      - name: experiment
        {{if .duat.awsecr}}
        image: {{.duat.awsecr}}/platform-services/{{.duat.module}}:{{.duat.version}}
        {{else}}
        image: localhost:32000/platform-services/{{.duat.module}}:{{.duat.version}}
        {{end}}
        imagePullPolicy: Always
        resources:
          requests:
            memory: "2048Mi"
            cpu: "100m"
          limits:
            memory: "2048Mi"
            cpu: "100m"
        ports:
        - containerPort: 30001
          name: grpc-exp
        env:
        - name: "LOGXI_FORMAT"
          value: "happy,maxcol=1024"
        - name: "LOGXI"
          value: "*=TRC"
        - name: "IP_PORT"
          value: ":30001,0.0.0.0:30001"
        - name: "PGHOST"
          valueFrom:
            secretKeyRef:
              name: postgres
              key: host
        - name: "PGPORT"
          valueFrom:
            secretKeyRef:
              name: postgres
              key: port
        - name: "PGDATABASE"
          valueFrom:
            secretKeyRef:
              name: postgres
              key: database
        - name: "PGUSER"
          valueFrom:
            secretKeyRef:
              name: postgres
              key: username
        - name: "PGPASSWORD"
          valueFrom:
            secretKeyRef:
              name: postgres
              key: password
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: auth0-egress
spec:
  hosts:
  - "cognizant-ai.auth0.com"
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: psql-egress
spec:
  hosts:
    - "{{ expandenv "$PGHOST" }}"
  ports:
    - name: psql
      number: 5432
      protocol: tcp
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
 name: psql-egress
spec:
 host: "{{ expandenv "$PGHOST" }}"
 trafficPolicy:
   tls:
     mode: DISABLE
4

Firstly, I noticed that your policy is applied on target name ingress-gateway. I assumed you use the standard Istio installation, then this is probably not what you want. The name should be the name of the ingressgateway service, i.e istio-ingressgateway. In other words, your policy may not be applied on any service yet. To confirm, you may try to check ingress config dump, and looking for JWT configuration in there (expected none with the current setting)

kubectl -n istio-system exec  $(kubectl get pod -lapp=istio-ingressgateway -o jsonpath={.items..metadata.name} -n istio-system) -- curl localhost:15000/config_dump

While we on this, you may want to drop the peers (mtls), given traffic to ingress most likely from outside the mesh and doesn’t have mTLS support. If you need TLS, check out the TLS setting in the Gateway API.

5

Hi I modified my yaml with these changes with no effect, I dont see any mention in the config_dump of any of the items in the Policy. It appears the policy is getting applied within kubernetes but not going any further.

Name:         experiment-auth-policy
Namespace:    istio-system
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"authentication.istio.io/v1alpha1","kind":"Policy","metadata":{"annotations":{},"name":"experiment-auth-policy","namespace":...
API Version:  authentication.istio.io/v1alpha1
Kind:         Policy
Metadata:
  Creation Timestamp:  2019-06-22T07:00:34Z
  Generation:          1
  Resource Version:    13593
  Self Link:           /apis/authentication.istio.io/v1alpha1/namespaces/istio-system/policies/experiment-auth-policy
  UID:                 6e39fac5-94bb-11e9-92d3-064fcdbb8a36
Spec:
  Origins:
    Jwt:
      Audiences:
        http://api.cognizant-ai.dev/experimentsrv
      Issuer:    https://cognizant-ai.auth0.com/
      Jwks Uri:  https://cognizant-ai.auth0.com/.well-known/jwks.json
  Peers:
    Mtls:
  Principal Binding:  USE_ORIGIN
  Targets:
    Name:       istio-ingressgateway
    Namespace:  istio-system
Events:         <none>

Yaml policy

apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  name: experiment-auth-policy
  namespace: istio-system
spec:
  targets:
  - name: istio-ingressgateway
    namespace: istio-system
  peers:
  - mtls: {}
  origins:
  - jwt:
      issuer: "https://cognizant-ai.auth0.com/"
      jwksUri: "https://cognizant-ai.auth0.com/.well-known/jwks.json"
      audiences:
        - "http://api.cognizant-ai.dev/experimentsrv"
  principalBinding: USE_ORIGIN
6

Target shouldn’t have namespace (it uses the same namespace of the CR). Actually, webhook validate should reject this configuration. Can you run kubectl get policy experiment-auth-policy -n istio-system -o yaml and verify that it is the same as what you enter. It is a bug if the system accept the configuration above (but not the same as this one).

7

Hi, Yes it was the same

apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"authentication.istio.io/v1alpha1","kind":"Policy","metadata":{"annotations":{},"name":"experiment-auth-policy","namespace":"istio-system"},"spec":{"origins":[{"jwt":{"audiences":["http://api.cognizant-ai.dev/experimentsrv"],"issuer":"https://cognizant-ai.auth0.com/","jwksUri":"https://cognizant-ai.auth0.com/.well-known/jwks.json"}}],"peers":[{"mtls":{}}],"principalBinding":"USE_ORIGIN","targets":[{"name":"istio-ingressgateway","namespace":"istio-system"}]}}
  creationTimestamp: "2019-06-24T05:50:37Z"
  generation: 1
  name: experiment-auth-policy
  namespace: istio-system
  resourceVersion: "1291"
  selfLink: /apis/authentication.istio.io/v1alpha1/namespaces/istio-system/policies/experiment-auth-policy
  uid: fdd87926-9643-11e9-958c-06dd7915cd8a
spec:
  origins:
  - jwt:
      audiences:
      - http://api.cognizant-ai.dev/experimentsrv
      issuer: https://cognizant-ai.auth0.com/
      jwksUri: https://cognizant-ai.auth0.com/.well-known/jwks.json
  peers:
  - mtls: {}
  principalBinding: USE_ORIGIN
  targets:
  - name: istio-ingressgateway
    namespace: istio-system
8

Fixing the namespace issue appears to now authenticate the connections and block on failures. I will repost information to my original message about not being able to authenticate now this is understood. Thanks