Authentication policy for JWT appears to be ignored (1.1.8) [solved]

I am trying to configure Auth0 based authentication and it is being ignored. Instead of the proxy handling the validation of the authorization header, it is being passed through to the service without the Auth0 validation occuring. I test this by using an invalid bearer value and the gRPC request is not getting refused, instead the request is being passed through to my service and then it sees the invalid Bearer value.

Use Case is that I am wanting to implement a machine-to-machine application that requires the remote client to fetching a token without expecting the Istio services doing redirections to present a login exchange to get this done.

The test is done using the following:


$ grpc_cli call $CLUSTER_INGRESS dev.cognizant_ai.experiment.Service.Get "uid: '1'" --metadata authorization:"Bearer x"`
connecting to a0029d7d38dfa11e9906d061fe987042-2141910118.us-west-2.elb.amazonaws.com:80
Sending client initial metadata:
authorization : Bearer x
Received trailing metadata from server:
date : Fri, 14 Jun 2019 04:36:34 GMT
server : istio-envoy
x-envoy-upstream-service-time : 1
Rpc failed with status code 16, error message: square/go-jose: compact JWS format must have three parts token="...arer x" stack="[auth.go:138 auth.go:191 experimentsrv.pb.go:416 server.go:900 server.go:1122 server.go:617]"

The error result is coming from the Go application not the proxy. I would like the proxy to be validating this.

Many thanks to anyone who can help.

When looking at the proxy log the enforced allow is present as follows:


[2019-06-14 04:18:30.758][26][debug][rbac] [external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:64] checking request: remoteAddress: 100.96.1.9:57938, localAddress: 100.96.1.28:30001, ssl: uriSanPeerCertificate: spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account, subjectPeerCertificate: , headers: ':scheme', 'https'
':method', 'POST'
':authority', 'a0029d7d38dfa11e9906d061fe987042-2141910118.us-west-2.elb.amazonaws.com:80'
':path', '/dev.cognizant_ai.experiment.Service/Get'
'authorization', 'Bearer x'
'te', 'trailers'
'content-type', 'application/grpc'
'user-agent', 'grpc-c++/1.21.2 grpc-c/7.0.0 (linux; chttp2; gandalf)'
'grpc-accept-encoding', 'identity,deflate,gzip'
'accept-encoding', 'identity,gzip'
'x-forwarded-for', '100.96.1.1'
'x-forwarded-proto', 'http'
'x-envoy-external-address', '100.96.1.1'
'x-request-id', '483e6da1-dfae-4113-9663-5faf626fccf6'
'x-istio-attributes', 'CkIKGGRlc3RpbmF0aW9uLnNlcnZpY2UuaG9zdBImEiRleHBlcmltZW50LmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwKQAoXZGVzdGluYXRpb24uc2VydmljZS51aWQSJRIjaXN0aW86Ly9kZWZhdWx0L3NlcnZpY2VzL2V4cGVyaW1lbnQKKAoYZGVzdGluYXRpb24uc2VydmljZS5uYW1lEgwSCmV4cGVyaW1lbnQKKgodZGVzdGluYXRpb24uc2VydmljZS5uYW1lc3BhY2USCRIHZGVmYXVsdApPCgpzb3VyY2UudWlkEkESP2t1YmVybmV0ZXM6Ly9pc3Rpby1pbmdyZXNzZ2F0ZXdheS01NDQ5N2I1ODQ5LWtqc3B2LmlzdGlvLXN5c3RlbQ=='
'x-b3-traceid', '93f63f1e4a3da7e6584795e02e5e4c0b'
'x-b3-spanid', '584795e02e5e4c0b'
'x-b3-sampled', '0'
'x-forwarded-client-cert', 'By=spiffe://cluster.local/ns/default/sa/default;Hash=951bd7368db7099fbcdd9bcaf6137d3fc6276c142c2efa991f8584e61a31af99;Subject="";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account'
, dynamicMetadata: filter_metadata {
  key: "istio_authn"
  value {
    fields {
      key: "request.auth.principal"
      value {
        string_value: "cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
      }
    }
    fields {
      key: "source.namespace"
      value {
        string_value: "istio-system"
      }
    }
    fields {
      key: "source.principal"
      value {
        string_value: "cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
      }
    }
    fields {
      key: "source.user"
      value {
        string_value: "cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
      }
    }
  }
}

[2019-06-14 04:18:30.758][26][debug][rbac] [external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:104] enforced allowed
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: ingress-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP2
    hosts:
    - "*"
---
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  name: experiment-auth-policy
spec:
  targets:
  - name: experiment
  peers:
  - mtls:
  origins:
  - jwt:
      issuer: "https://cognizant-ai.auth0.com/"
      jwksUri: "https://cognizant-ai.auth0.com/.well-known/jwks.json"
      audiences:
        - "http://api.cognizant-ai.dev/experimentsrv"
  originIsOptional: false
  principalBinding: USE_ORIGIN
---
apiVersion: rbac.istio.io/v1alpha1
kind: ClusterRbacConfig
metadata:
  name: default
spec:
  mode: 'ON_WITH_INCLUSION'
  inclusion:
    namespaces: ["default"]
---
apiVersion: rbac.istio.io/v1alpha1
kind: ServiceRole
metadata:
  name: service-user
spec:
  rules:
  - services: ["*"]
    paths: ["*"]
    methods: ["*"]
---
apiVersion: rbac.istio.io/v1alpha1
kind: ServiceRoleBinding
metadata:
  name: all-authenticated-users
spec:
  subjects:
  - properties:
      source.principal: "*"
  roleRef:
    kind: ServiceRole
    name: service-user
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grpc-experiment-service
spec:
  gateways:
  - ingress-gateway
  hosts:
  - "*"
  http:
  - match:
    - uri:
        prefix: "/dev.cognizant_ai.experiment.Service/"
    - uri:
       prefix: "/grpc.reflection.v1alpha.ServerReflection/"
    route:
    - destination:
        host: experiment.default.svc.cluster.local
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: experiment
spec:
  host: experiment
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
---
apiVersion: v1
kind: Service
metadata:
  name: experiment
  labels:
    app: experiment
spec:
  ports:
  - port: 30001
    name: grpc-exp
    targetPort: 30001
  selector:
    app: experiment
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: experiment-v1
  labels:
    version: v1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: experiment
        version: v1
    spec:
      containers:
      - name: experiment
        {{if .duat.awsecr}}
        image: {{.duat.awsecr}}/platform-services/{{.duat.module}}:{{.duat.version}}
        {{else}}
        image: localhost:32000/platform-services/{{.duat.module}}:{{.duat.version}}
        {{end}}
        imagePullPolicy: Always
        resources:
          requests:
            memory: "2048Mi"
            cpu: "100m"
          limits:
            memory: "2048Mi"
            cpu: "100m"
        ports:
        - containerPort: 30001
          name: grpc-exp
        env:
        - name: "LOGXI_FORMAT"
          value: "happy,maxcol=1024"
        - name: "LOGXI"
          value: "*=TRC"
        - name: "IP_PORT"
          value: ":30001,0.0.0.0:30001"
        - name: "PGHOST"
          valueFrom:
            secretKeyRef:
              name: postgres
              key: host
        - name: "PGPORT"
          valueFrom:
            secretKeyRef:
              name: postgres
              key: port
        - name: "PGDATABASE"
          valueFrom:
            secretKeyRef:
              name: postgres
              key: database
        - name: "PGUSER"
          valueFrom:
            secretKeyRef:
              name: postgres
              key: username
        - name: "PGPASSWORD"
          valueFrom:
            secretKeyRef:
              name: postgres
              key: password
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: auth0-egress
spec:
  hosts:
  - "cognizant-ai.auth0.com"
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: psql-egress
spec:
  hosts:
    - "{{ expandenv "$PGHOST" }}"
  ports:
    - name: psql
      number: 5432
      protocol: tcp
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
 name: psql-egress
spec:
 host: "{{ expandenv "$PGHOST" }}"
 trafficPolicy:
   tls:
     mode: DISABLE

The problem in this case was a yaml file format issue that failed one or two components and silently ignored.

So I guess it solved the problem after you fixed the yaml file format issue? Just wondering is the file format related to Istio?

@KarlMutch could you provide the incorrect yaml policy and the correct one? It’s strange because if there was an formatting issue, the kubectl apply should fail because it will error out when converting yaml to protobuf.

@YangminZhu correct me if I’m wrong.

The authentication issues I am having have changed now the yaml is fixed and I’ll post a separate message about those.

The yaml issue has to do with multi resource separators. If these separators have trailing white space characters followed by dashes etc then the separator is ignored. This is occurring inside the kubectl rather than during istioctl injection.

When the separator is ignored then two resources will run together when kubectl runs and we loose the second yaml specified resource.