End User Auth for external use and MTLS for internal service-service communication?

Security
1

I’m trying to setup a namespace such that any services exposed through an ingress gateway/virtual service require end user JWT authentication, but the same service when accessed from another internal service will use normal mTLS authentication.

Here is my general setup:

Gateway and Virtual Service

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: gateway
spec:
  selector:
    istio: ingressgateway 
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "{{domain}}"
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: "{{certificate-secret-name}}" 
    hosts:
    - "{{domain}}"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ingress
spec:
  hosts:
  - "{{domain}}"
  gateways:
  - gateway
  http:
  - match:
    - uri:
        prefix: /auth/
    - uri:
        prefix: /auth
    rewrite:
      uri: "/"
    route:
    - destination:
        host: auth
        port:
          number: 8000
  - match:
    - uri:
        prefix: /api/
    - uri:
        prefix: /api
    route:
    - destination:
        host: "api"
        port:
          number: 8000

Auth Policy

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "auth"
spec:
  targets:
  - name: "api"
  peers:
  - mtls:
  origins:
  - jwt:
      issuer: "{{domain}}"
      jwksUri: "https://{{domain}}/.well-known/jwks.json"
  principalBinding: USE_ORIGIN

With this setup external JWT auth works. Any incoming request to the ‘api’ service requires a JWT.
The problem I’m hitting, which I thought would be solved by the ‘peers’ section in the auth policy is that it looks like JWT auth is also required when another service (lets call that service ‘limits’) tries to make a request to the ‘api’ service.

This is the response:

server returned error: HTTP/1.1 401 Unauthorized

If I remove the auth policy, communication is allowed again.

I have to assume that a typical use to end user authentication is for requests that originate outside of the mesh and that the same services that ingress will communicate with should also be accessible by other internal services that live within the mesh without requiring end user authentication.

Is there an additional auth policy I need to add to inform the mesh to not require end user authentication for internal service-service communication?

2

This is an interesting use case. As far as I know, we don’t support external for jwt+mTLS while internal for mTLS only.

The reason is for external -> ingress -> serviceA communication, we don’t distinguish ingress from other internal services (i.e., internal serviceB -> serviceA). As a result, serviceA just simply checks jwt and peer cert given your auth policy.

@YangminZhu please correct me if i am wrong. And do you think it is something we can support and is there any workaround to unblock rob@ ?

3

Sorry, I may have been a little unclear. I’m not looking for the combination of the two. I want either supported depending on use case.

Case #1
Traffic is coming from an end user outside of the mesh. In this case I want the origin JWT authentication applied.
(end user) => (gateway) => (api) (in this case JWT should be used)

Case #2
Traffic is coming from a service within the mesh. In this case I want mtls to be applied (but not JWT).
(limits) => (api) (in this case mtls should be used)

4

One thing you could do is to apply the JWT policy on (gateway) and apply the mTLS policy on (api). This might be a little different from what you might be thinking but I think it effectively have the same outcome.

If you want to apply the JWT policy on (api) for some reason, you may need to look into some workaround as it’s not directly supported in today’s API. For example, you maybe use it with RBAC to require JWT for requests from service account other than ingressgateway.

5

Thanks @YangminZhu. I can give that a try. One clarification here, when you say the gateway, do you mean something:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "auth"
spec:
  targets:
  - name: "gateway"
  peers:
  - mtls:
  origins:
  - jwt:
      issuer: "{{domain}}"
      jwksUri: "https://{{domain}}/.well-known/jwks.json"
  principalBinding: USE_ORIGIN

Where ‘gateway’ is the name of the actual Gateway resource. Or something like:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "auth"
spec:
  targets:
  - name: "istio-ingressgateway"
  peers:
  - mtls:
  origins:
  - jwt:
      issuer: "{{domain}}"
      jwksUri: "https://{{domain}}/.well-known/jwks.json"
  principalBinding: USE_ORIGIN

Where the target is the service name of the istio gateway in the istio-system namespace?

6

The target name should be "istio-ingressgateway" which is the service name of the ingress gateway. You should also set the namespace to “istio-system”:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "auth"
  namespace: "istio-system
spec:
  targets:
  - name: "istio-ingressgateway"
  peers:
  - mtls:
  origins:
  - jwt:
      issuer: "{{domain}}"
      jwksUri: "https://{{domain}}/.well-known/jwks.json"
  principalBinding: USE_ORIGIN
7

@YangminZhu yes that does work. However, there is one more wrinkle in my use case. Each namespace also has its own JWT signing key.

Adding the origin authentication does to the ingress gateway does work for a single namespace (I have a namespace per tenant). Once I had in more than one namespace, the authentication fails since there are multiple signing keys.

Is it possible to have ingress gateways outside if the istio-system namespace and alongside each tenant namespace?

8

You can add as many origins as you need, with different issuers. If any validate than the request is validated.

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "auth"
  namespace: "istio-system
spec:
  targets:
  - name: "istio-ingressgateway"
  peers:
  - mtls:
  origins:
  - jwt:
      issuer: "{{tenant1}}"
      jwksUri: "https://{{tenant1}}/.well-known/jwks.json"
  - jwt: # same certs, different issuer
      issuer: "{{tenant1}}.local" 
      jwksUri: "https://{{tenant1}}/.well-known/jwks.json"
  - jwt: # different certs and issuer
      issuer: "{{tenant2}}"
      jwksUri: "https://{{tenant2}}/.well-known/jwks.json"
  principalBinding: USE_ORIGIN
9

That makes sense, and should have been obvious to me. I’ll give that a try, thanks for your help!

10

@rjbez17 Thank you for providing the example, that’s very nice!

@rob Please let us know if you need any help.

11

@YangminZhu the Istio example End-user authentication is showing the JWT policy on the service httpbin, are you saying the JWT policy won’t work if traffic is coming from inside the mesh?

cat <<EOF | kubectl apply -n foo -f -
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "jwt-example"
spec:
  targets:
  - name: httpbin
  origins:
  - jwt:
      issuer: "testing@secure.istio.io"
      jwksUri: "https://raw.githubusercontent.com/istio/istio/release-1.2/security/tools/jwt/samples/jwks.json"
  principalBinding: USE_ORIGIN
EOF
12

@dwctua From what I’ve been trying to do, if JWT origin auth is set on a service, then any traffic (whether its coming from the outside into the gateway and through to that service or if its coming from anther internal service) will require JWT auth.

The example given above about switching JWT auth to the ingressgateway service/pod does work. I have verified that. In that case external traffic coming into the gateway will require JWT auth but none of the internal services will. The only piece I still need to try is to have multiple JWKS entries (from what the filter does I suspect this will work) but I’m not sure i’ll get to verifying that will work until later today or Monday.

13

Thanks for clarification @rob! I misread your examples. Personally I would prefer to have a consistent authentication regardless where is the traffic coming from.

14

Hello everyone.

I understood @rob needs as I have the same need: let the user access, with JWT, the services, while letting kube services call between each other without JWT.

I think new Istio gives the possibility of making that, according to the documentation, but I’d like that this could be confirmed by someone, as I’m going to try it only in a few days.


This doc link shows that authentication is now divided into Peer (so intended to do local services communication) and Request (for end user). This did not exist a year ago, so, for sure, the workaround you used could be replaced (we hope).
15

would love to know if with latest istio version (RequestAuthenticaiton + PeerAuthentication) we get this solved without a workaround.

as we are trying to achieve the exact same thing

10x

1 Like
16

This can be easily supported with the latest policy, you just need:

  1. Apply RequestAuthentication + AuthoziationPolicy on ingress gateway so that all traffic entering the mesh will need a valid JWT token;
apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
  name: "ingress-jwt"
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  jwtRules:
  - issuer: "testing@secure.istio.io"
    jwksUri: "https://raw.githubusercontent.com/istio/istio/release-1.7/security/tools/jwt/samples/jwks.json"
---
apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
  name: "ingress-jwt"
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  action: ALLOW
  rules:
  - from:
    - source:
        requestPrincipals: ["*"]
  1. Apply PeerAuthentication in mesh level so that all traffic within the mesh will require the mTLS.
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  mtls:
    mode: STRICT
1 Like
17

great that exactly what i did.
the only difference is that i didnt set the requestAuthentication on the ingress itself, but on the namespace itself, that’s why i needed to add the namespaces source on the authorization policy.

based on your suggestion it means that no matter to which app, whenever request came from outside the mesh, it will require jwt, but all communication inside the mesh, will use the mTLS withouth needing the token.
make sense.

10x

18

Hello,

I tried what you explained, with the following configurations:

Gateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: test-gateway
  namespace: dev
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"

RequestAuthentication:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: test-reqauth
  namespace: dev
spec:
  jwtRules:
  - issuer: "http://keycloak/auth/realms/realm"
    jwksUri: "http://keycloak/auth/realms/realm/protocol/openid-connect/certs"

PeerAuthentication:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: test-peerauth
  namespace: dev
spec:
  mtls:
    mode: STRICT

Deployment 1 (+ vs, svc, authorization policy)

apiVersion: v1
kind: Service
metadata:
  name: kc-istio-integr
  namespace: dev
  labels:
    app: kc-istio-integr
    service: kc-istio-integr
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: kc-istio-integr
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: kc-istio-integr
  namespace: dev
spec:
  hosts:
  - "*"
  gateways:
  - test-gateway
  http:
    - match:
      - uri:
          prefix: /uno
      route:
        - destination:
            port:
              number: 8080
            host: kc-istio-integr.dev.svc.cluster.local
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kc-istio-integr
  namespace: dev
  labels:
    app: kc-istio-integr
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kc-istio-integr
      version: v1
  template:
    metadata:
      labels:
        app: kc-istio-integr
        version: v1
    spec:
      containers:
      - name: kc-istio-integr
        image: image:1.0
        imagePullPolicy: IfNotPresent
        env:
         - name: SERVICE.URL
           value: "http://kc-istio-integr-2:8080"
        ports:
        - containerPort: 8080
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: kc-istio-integr-authpolicy
 namespace: dev
spec:
 selector:
   matchLabels:
     app: kc-istio-integr
 rules:
 - to:
   - operation:
       methods: ["OPTIONS"]
       paths: ["*"]
 - to:
   - operation:
       methods: ["GET"]
       paths: ["/uno*"]
   when:
   - key: request.auth.claims[realm_access][roles]
     values: ["READ"]
 - to:
   - operation:
       methods: ["POST", "PUT", "DELETE", "PATCH"]
       paths: ["/uno*"]
   when:
   - key: request.auth.claims[realm_access][roles]
     values: ["WRITE"]

Deployment 2 (+ vs, svc, authorization policy)

apiVersion: v1
kind: Service
metadata:
  name: kc-istio-integr-2
  namespace: dev
  labels:
    app: kc-istio-integr-2
    service: kc-istio-integr-2
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: kc-istio-integr-2
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: kc-istio-integr-2
  namespace: dev
spec:
  hosts:
  - "*"
  gateways:
  - test-gateway
  http:
    - match:
      - uri:
          prefix: /due
      route:
        - destination:
            port:
              number: 8080
            host: kc-istio-integr-2.dev.svc.cluster.local
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kc-istio-integr-2
  namespace: dev
  labels:
    app: kc-istio-integr-2
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kc-istio-integr-2
      version: v1
  template:
    metadata:
      labels:
        app: kc-istio-integr-2
        version: v1
    spec:
      containers:
      - name: kc-istio-integr-2
        image: image-2:1.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: kc-istio-integr-2-authpolicy
 namespace: dev
spec:
 selector:
   matchLabels:
     app: kc-istio-integr-2
 rules:
 - to:
   - operation:
       methods: ["OPTIONS"]
       paths: ["*"]
 - to:
   - operation:
       methods: ["GET"]
       paths: ["/due*"]
   when:
   - key: request.auth.claims[realm_access][roles]
     values: ["READ"]
 - to:
   - operation:
       methods: ["POST", "PUT", "DELETE", "PATCH"]
       paths: ["/due*"]
   when:
   - key: request.auth.claims[realm_access][roles]
     values: ["WRITE"]

From outside the cluster, I reach every http method with a valid JWT.
However, I have got an http method, on the 1st service, which tries to reach the 2nd service with an http request to
http://kc-istio-integr-2:8080/due/internaltest.
this should use an internal service-to-service communication granted by the PeerAuthentication - so it should work without a JWT, since the flow is cluster-internal - but it encounters 403 RBAC.

Any hint regarding that?
Maybe it is a not-handled mechanism, or bugged PeerAuthentication?
Documentation is very poor on that and I am not able to understand.

Thank you very much :slight_smile: