I need help getting logging working for failed requests to Auth0 for validation of Bearer tokens so I can determine the source/cause of auth failures.
Instructions provided to enable Authentication logging result in no output related to authentication or RBAC failures. c.f. https://istio.io/help/ops/security/debugging-authorization/, when they occur. I have run up the web page for enabling rbac logging and ‘crickets’.
grpc_cli ls $CLUSTER_INGRESS dev.cognizant_ai.experiment.Service -l --metadata Authorization:"Bearer ${AUTH0_TOKEN}"
Received an error when querying services endpoint.
ServerReflectionInfo rpc failed. Error code: 16, message: Origin authentication failed., debug info: {"created":"@1560631126.228257924","description":"Error received from peer ipv4:52.25.151.192:80","file":"src/core/lib/surface/call.cc","file_line":1046,"grpc_message":"Origin authentication failed.","grpc_status":16}
If I remove the Policy and let the authorization flow to my Go based service I can do my own validation successfully. But I really would like the mesh to do it for me.
The discovery container in the pilot shows the filter being built.
GC forced
gc 69 @4214.934s 0%: 0.082+1.0+0.15 ms clock, 1.3+0/3.6/8.0+2.4 ms cpu, 14->14->8 MB, 16 MB goal, 16 P
2019-06-15T20:33:03.967703Z debug rbac authz/rbac.go:346 building http filter config for {experiment.default.svc.cluster.local map[pod-template-hash:353666256 version:v1 app:experiment] map[destination.name:experiment destination.namespace:default destination.user:default]}
2019-06-15T20:33:03.967745Z debug rbac authz/rbac.go:478 checking role service-user
2019-06-15T20:33:03.967755Z debug rbac authz/rbac.go:482 rules[0] matched
2019-06-15T20:33:03.967894Z debug rbac authz/rbac.go:448 generated http filter config: {policies:<key:"service-user" value:<permissions:<and_rules:<rules:<or_rules:<rules:<header: > > > rules:<or_rules:<rules:<header: > > > > > principals:<and_ids:<ids:<metadata:<filter:"istio_authn" path: path: value:<list_match:<one_of:<string_match: > > > > > ids:<metadata:<filter:"istio_authn" path: value:<string_match: > > > > > > > {} [] 0}
2019-06-15T20:33:03.968730Z info rbac authz/rbac.go:349 built http filter config for experiment.default.svc.cluster.local
gc 70 @4318.917s 0%: 0.018+1.2+0.13 ms clock, 0.29+0.19/4.0/8.9+2.1 ms cpu, 16->16->8 MB, 17 MB goal, 16 P
gc 71 @4318.957s 0%: 0.006+1.2+0.13 ms clock, 0.10+0.13/4.3/7.6+2.1 ms cpu, 16->17->9 MB, 17 MB goal, 16 P
2019-06-15T20:33:04.022223Z info http: multiple response.WriteHeader calls
scvg28: inuse: 16, idle: 5, sys: 22, released: 0, consumed: 22 (MB)
gc 72 @4423.497s 0%: 0.018+1.1+0.13 ms clock, 0.29+0.14/3.7/9.1+2.1 ms cpu, 17->17->8 MB, 18 MB goal, 16 P
scvg29: inuse: 17, idle: 4, sys: 22, released: 0, consumed: 22 (MB)
The service proxy log is silent, it also does not appear to print information related to loading the policy which might be the best clue I have. The istio-policy, containers make no mention of the request when it fails. The galley log has no errors in it either.
Using istioctl to output the configuration differences shows that there are none and the proxies are what is expected. The same also for proxy-status.
The configuration is as follows
apiVersion: rbac.istio.io/v1alpha1
kind: ClusterRbacConfig
metadata:
name: default
spec:
mode: 'ON_WITH_INCLUSION'
inclusion:
namespaces: ["default"]
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: ingress-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP2
hosts:
- "*"
#
# Authentication is still an alpha feature so retain the service side validation and make
# mesh authentication optional as it is not yet stable
#
---
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
name: experiment-auth-policy
spec:
targets:
- name: experiment
peers:
- mtls: {}
origins:
- jwt:
issuer: "https://cognizant-ai.auth0.com/"
jwksUri: "https://cognizant-ai.auth0.com/.well-known/jwks.json"
audiences:
- "http://api.cognizant-ai.dev/experimentsrv"
principalBinding: USE_ORIGIN
---
apiVersion: rbac.istio.io/v1alpha1
kind: ServiceRole
metadata:
name: service-user
spec:
rules:
- services: ["*"]
paths: ["*"]
methods: ["*"]
---
apiVersion: rbac.istio.io/v1alpha1
kind: ServiceRoleBinding
metadata:
name: all-authenticated-users
spec:
subjects:
- properties:
source.principal: "*"
request.auth.claims[scope]: "all:experiments"
roleRef:
kind: ServiceRole
name: service-user
---
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: grpc-experiment-service
spec:
gateways:
- ingress-gateway
hosts:
- "*"
http:
- match:
- uri:
prefix: "/dev.cognizant_ai.experiment.Service/"
- uri:
prefix: "/grpc.reflection.v1alpha.ServerReflection/"
route:
- destination:
host: experiment.default.svc.cluster.local
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: experiment
spec:
host: experiment
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
---
apiVersion: v1
kind: Service
metadata:
name: experiment
labels:
app: experiment
spec:
ports:
- port: 30001
name: grpc-exp
targetPort: 30001
selector:
app: experiment
---