Unable to apply auth to service accounts

#1

Hi,

I installed Istio 1.1.2 in GKE cluster 1.12. It’s a new install.
I configured 2 clusters in multicluster configuration, one cluster with master control plane and second has minimul istio configuration.
Service discover works ok between clusters ( I can curl from pods across clusters ).

Then I want to test authorization, and it’s not fully working ( on single and multi cluster ) when I try to apply an authorization to specific service account.

what am I missing again… ? :frowning:

My goal is to authorize only deployment a to talk to service b-svc
Authorization is enabled, and I get access denied it’s all fine.

I can also authorize a to all others with user: "*"
I can then curl from all deployment pods to b-svc

example :

apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRole
metadata:
  name: b
  namespace: default
spec:
  rules:
  - services: ["b-svc.default.svc.cluster.local"]
    methods: ["GET"]
---
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRoleBinding
metadata:
  name: a-b
  namespace: default
spec:
  subjects:
  - user: "*"
  roleRef:
    kind: ServiceRole
    name: b

but authorization is not working ( I still get access denied when trying to curl from pod a to service b-svc ) when I implement the user field with either :
user: “a-sa”
or
user: “cluster.local/ns/default/sa/a-sa”

Example :

apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRole
metadata:
  name: b
  namespace: default
spec:
  rules:
  - services: ["b-svc.default.svc.cluster.local"]
    methods: ["GET"]
---
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRoleBinding
metadata:
  name: a-b
  namespace: default
spec:
  subjects:
  - user: "cluster.local/ns/default/sa/a-sa"
  roleRef:
    kind: ServiceRole
    name: b

Any ideas ?

I checked the dumps of proxy in a and b


a doesn’t get updated
b got updated with :

 "name": "envoy.filters.http.rbac",
             "config": {
              "rules": {
               "policies": {
                "b": {
                 "permissions": [
                  {
                   "and_rules": {
                    "rules": [
                     {
                      "or_rules": {
                       "rules": [
                        {
                         "header": {
                          "exact_match": "GET",
                          "name": ":method"

My services :

 francois_nature@cloudshell:~/myscripts/istio/abcd (still-freedom-234411)$ cat a-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: a-svc
spec:
  type: ClusterIP
  ports:
  - port: 80
    name: http
  selector:
    name: a
francois_nature@cloudshell:~/myscripts/istio/abcd (still-freedom-234411)$ cat b-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: b-svc
spec:
  type: ClusterIP
  ports:
  - port: 80
    name: http
  selector:
    name: b

roles and bindings :

apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRole
metadata:
  name: b
  namespace: default
spec:
  rules:
  - services: ["b-svc.default.svc.cluster.local"]
    methods: ["GET"]
---
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRoleBinding
metadata:
  name: a-b
  namespace: default
spec:
  subjects:
  - user: "cluster.local/ns/default/sa/a-sa"
  roleRef:
    kind: ServiceRole
    name: b

deployment and service account

francois_nature@cloudshell:~/myscripts/istio/abcd (still-freedom-234411)$ cat a-p.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: a-sa
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: a
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: a
    spec:
      serviceAccountName: a-sa
      containers:
      - name: a
        image: fnature/ngnix-curl:1
        ports:
         - containerPort: 80
           name: http
        volumeMounts:
          - name: html
            mountPath: /usr/share/nginx/html/
      volumes:
        - name: html
          configMap:
            name: res-clust1
            items:
              - key: message
                path: index.html
#2

I deploy the bookinfo example
and enable auth with same example

I have the same issue :frowning:
example
RBAC will work with binding with
- user: "*"
but not with
- user: "cluster.local/ns/default/sa/bookinfo-productpage"

#3

it appears that Pilot accepts the policy according to Pilot logs…

2019-04-19T17:39:45.893540Z     debug   rbac    generated http filter config: {policies:<key:"details-reviews-viewer" value:<permissions:<and_rules:<rules:<or_rules:<rules:<header:<name:":method" exact_match:"GET" > > > > > > principals:<and_ids:<ids:<metadata:<filter:"istio_authn" path:<key:"source.principal" > value:<string_match:<exact:"cluster.local/ns/default/sa/bookinfo-productpage" > > > > > > > >  <nil> {} [] 0}
2019-04-19T17:39:45.894535Z     info    rbac    built http filter config for details.default.svc.cluster.local

But issue is that
Pilot doesn’t Distributes Policies to Proxies Correctly :frowning:

dump of productpage shows that it hasn’t applied the policy…

"name": "envoy.filters.http.rbac",
         "config": {
          "rules": {
           "policies": {
            "productpage-viewer": {
             "permissions": [
              {
               "and_rules": {
                "rules": [
                 {
                  "or_rules": {
                   "rules": [
                    {
                     "header": {
                      "name": ":method",
                      "exact_match": "GET"
                     }
                    }
                   ]
                  }
                 }

How to troubleshoot Pilot further ?

#4

The envoy dump of productpage is not complete, it’s missing the principal part, could you upload all of them?

Also, if you’re following https://istio.io/help/ops/security/debugging-authorization/?_ga=2.257754635.120888657.1555353243-244168650.1519165063#ensure-proxies-enforce-policies-correctly, could you get the envoy logs as well? (with rbac logging level set to debug).

#5

How do I upload files here ?

#6

thank you YangminZhu
it’s there
https://drive.google.com/open?id=1xsbuPTDZnVkBLP7_VjoOapComwpshJCr

#7

Here is what I’ve done :

I create a single cluster and install istio the same way as before
I have the same issue

I notice :

uses Helm install and differ from https://istio.io/docs/setup/kubernetes/install/kubernetes/

Also I don’t have all istio-system pods described in https://istio.io/docs/setup/kubernetes/install/kubernetes/

so I create istio in new cluster

with istio steps from

( it uses helm )

and enable permissive TLS from

then I have the same issue

so I create new cluster

istio with strict TLS

this time I follow all steps from

and…

this works finally

conclusion : steps to install istio with multicluster as described are not sufficient for RBAC to work.

#8

well another problem I couldn’t resolve.

I did install master control cluster like

and the remote cluster like

and now it’s the remote cluster that won’t work
I get following errors from pods running on it
root@c-6998565984-knqgs:/# curl c-svc
UNAVAILABLE:upstream connect error or disconnect/reset before headersroot@c-6998565984-knqgs:/# ^C

I’ll forget this setup and concentrate on multiple control planes…

#9

I posted in

#10

Hi @YangminZhu, do you have any idea about this issue ? I’m pretty much stuck. I can’t have multicluster and RBAC working :frowning:

#11

@fnature

Looking at the productpage envoy log:

[2019-04-19 19:24:30.633][25][debug][rbac] [external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:62] checking request: remoteAddress: 10.8.3.3:43430, localAddress: 10.8.3.8:9080, ssl: none, headers: ':authority', '35.185.80.155'
':path', '/productpage'
':method', 'GET'
'upgrade-insecure-requests', '1'
'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1 Safari/605.1.15'
'accept-language', 'en-ie'
'accept-encoding', 'gzip, deflate'
'x-forwarded-for', '10.8.3.1'
'x-forwarded-proto', 'http'
'x-request-id', '617d2aca-81a5-44d9-b497-20a244254c2f'
'x-istio-attributes', 'CikKGGRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZRINEgtwcm9kdWN0cGFnZQoqCh1kZXN0aW5hdGlvbi5zZXJ2aWNlLm5hbWVzcGFjZRIJEgdkZWZhdWx0Ck8KCnNvdXJjZS51aWQSQRI/a3ViZXJuZXRlczovL2lzdGlvLWluZ3Jlc3NnYXRld2F5LTVmNTc3YmJiY2QtdnR6aHYuaXN0aW8tc3lzdGVtCkMKGGRlc3RpbmF0aW9uLnNlcnZpY2UuaG9zdBInEiVwcm9kdWN0cGFnZS5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsCkEKF2Rlc3RpbmF0aW9uLnNlcnZpY2UudWlkEiYSJGlzdGlvOi8vZGVmYXVsdC9zZXJ2aWNlcy9wcm9kdWN0cGFnZQ=='
'x-b3-traceid', '3d0c7f124c6d548da7bcef2242219ff5'
'x-b3-spanid', 'a7bcef2242219ff5'
'x-b3-sampled', '0'
'content-length', '0'
'x-envoy-internal', 'true'
, dynamicMetadata: filter_metadata {
  key: "istio_authn"
  value {
  }
}

Note the ssl: none in the log which means the request is plain-text instead of TLS.

I assume the request is made from another cluster to this productpage service, if you want to use the user field in the ServiceRoleBinding, you need to make sure the request is made based on mTLS, otherwise the productpage service cannot extract the identity from the peer certificate (there is no such certificate in this case).

#12

thank you @YangminZhu it works after I apply mTLS
is it possible to apply layer7 policy in istio without mTLS and encryption ?

#13

Yes, you can still use authorization even without mTLS. But keep in mind mTLS not only provide the encryption but also provide the identity (by x509 certificate), without mTLS, you won’t be able to authorize the request based on the peer’s certificate which has the service account of the source.

Fields not depending on the mTLS can still be used in authorization, for example, the source ip.

#14

ok @YangminZhu !
…I’m still on my journey trying to make RBAC working across multiple clusters.

Issue is now, is as soon as I enable mTLS, I can’t curl across clusters.
When mTLS is configured it works ok.

I enabled mTLS with :

apiVersion: "authentication.istio.io/v1alpha1"
kind: "MeshPolicy"
metadata:
  name: "default"
spec:
  peers:
  - mtls: {}

and

apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  host: "*.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

result is
istioctl authn tls-check a-54d8fb594f-lbztn b-svc.default.svc.cluster.local
HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
b-svc.default.svc.cluster.local:80 OK mTLS mTLS default/ default/istio-system

I can’t curl anymore across clusters
I have
root@a-54d8fb594f-lbztn:/# curl b-svc
curl: (56) Recv failure: Connection reset by peer

The multicluster was installed same as

#15

so it looks as if mTLS is not supported in multicluster with single control plane with VPN ?

in servicerolebinding, is it not just possible to apply the service role to a deployment ?
all doc refers to binding the service role to a service account.
applying to a deployment would not require mTLS then… ?