End User Auth for external use and MTLS for internal service-service communication?

I’m trying to setup a namespace such that any services exposed through an ingress gateway/virtual service require end user JWT authentication, but the same service when accessed from another internal service will use normal mTLS authentication.

Here is my general setup:

Gateway and Virtual Service

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: gateway
spec:
  selector:
    istio: ingressgateway 
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "{{domain}}"
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: "{{certificate-secret-name}}" 
    hosts:
    - "{{domain}}"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ingress
spec:
  hosts:
  - "{{domain}}"
  gateways:
  - gateway
  http:
  - match:
    - uri:
        prefix: /auth/
    - uri:
        prefix: /auth
    rewrite:
      uri: "/"
    route:
    - destination:
        host: auth
        port:
          number: 8000
  - match:
    - uri:
        prefix: /api/
    - uri:
        prefix: /api
    route:
    - destination:
        host: "api"
        port:
          number: 8000

Auth Policy

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "auth"
spec:
  targets:
  - name: "api"
  peers:
  - mtls:
  origins:
  - jwt:
      issuer: "{{domain}}"
      jwksUri: "https://{{domain}}/.well-known/jwks.json"
  principalBinding: USE_ORIGIN

With this setup external JWT auth works. Any incoming request to the ‘api’ service requires a JWT.
The problem I’m hitting, which I thought would be solved by the ‘peers’ section in the auth policy is that it looks like JWT auth is also required when another service (lets call that service ‘limits’) tries to make a request to the ‘api’ service.

This is the response:

server returned error: HTTP/1.1 401 Unauthorized

If I remove the auth policy, communication is allowed again.

I have to assume that a typical use to end user authentication is for requests that originate outside of the mesh and that the same services that ingress will communicate with should also be accessible by other internal services that live within the mesh without requiring end user authentication.

Is there an additional auth policy I need to add to inform the mesh to not require end user authentication for internal service-service communication?

This is an interesting use case. As far as I know, we don’t support external for jwt+mTLS while internal for mTLS only.

The reason is for external -> ingress -> serviceA communication, we don’t distinguish ingress from other internal services (i.e., internal serviceB -> serviceA). As a result, serviceA just simply checks jwt and peer cert given your auth policy.

@YangminZhu please correct me if i am wrong. And do you think it is something we can support and is there any workaround to unblock rob@ ?

Sorry, I may have been a little unclear. I’m not looking for the combination of the two. I want either supported depending on use case.

Case #1
Traffic is coming from an end user outside of the mesh. In this case I want the origin JWT authentication applied.
(end user) => (gateway) => (api) (in this case JWT should be used)

Case #2
Traffic is coming from a service within the mesh. In this case I want mtls to be applied (but not JWT).
(limits) => (api) (in this case mtls should be used)

One thing you could do is to apply the JWT policy on (gateway) and apply the mTLS policy on (api). This might be a little different from what you might be thinking but I think it effectively have the same outcome.

If you want to apply the JWT policy on (api) for some reason, you may need to look into some workaround as it’s not directly supported in today’s API. For example, you maybe use it with RBAC to require JWT for requests from service account other than ingressgateway.

Thanks @YangminZhu. I can give that a try. One clarification here, when you say the gateway, do you mean something:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "auth"
spec:
  targets:
  - name: "gateway"
  peers:
  - mtls:
  origins:
  - jwt:
      issuer: "{{domain}}"
      jwksUri: "https://{{domain}}/.well-known/jwks.json"
  principalBinding: USE_ORIGIN

Where ‘gateway’ is the name of the actual Gateway resource. Or something like:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "auth"
spec:
  targets:
  - name: "istio-ingressgateway"
  peers:
  - mtls:
  origins:
  - jwt:
      issuer: "{{domain}}"
      jwksUri: "https://{{domain}}/.well-known/jwks.json"
  principalBinding: USE_ORIGIN

Where the target is the service name of the istio gateway in the istio-system namespace?

The target name should be "istio-ingressgateway" which is the service name of the ingress gateway. You should also set the namespace to “istio-system”:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "auth"
  namespace: "istio-system
spec:
  targets:
  - name: "istio-ingressgateway"
  peers:
  - mtls:
  origins:
  - jwt:
      issuer: "{{domain}}"
      jwksUri: "https://{{domain}}/.well-known/jwks.json"
  principalBinding: USE_ORIGIN

@YangminZhu yes that does work. However, there is one more wrinkle in my use case. Each namespace also has its own JWT signing key.

Adding the origin authentication does to the ingress gateway does work for a single namespace (I have a namespace per tenant). Once I had in more than one namespace, the authentication fails since there are multiple signing keys.

Is it possible to have ingress gateways outside if the istio-system namespace and alongside each tenant namespace?

You can add as many origins as you need, with different issuers. If any validate than the request is validated.

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "auth"
  namespace: "istio-system
spec:
  targets:
  - name: "istio-ingressgateway"
  peers:
  - mtls:
  origins:
  - jwt:
      issuer: "{{tenant1}}"
      jwksUri: "https://{{tenant1}}/.well-known/jwks.json"
  - jwt: # same certs, different issuer
      issuer: "{{tenant1}}.local" 
      jwksUri: "https://{{tenant1}}/.well-known/jwks.json"
  - jwt: # different certs and issuer
      issuer: "{{tenant2}}"
      jwksUri: "https://{{tenant2}}/.well-known/jwks.json"
  principalBinding: USE_ORIGIN

That makes sense, and should have been obvious to me. I’ll give that a try, thanks for your help!

@rjbez17 Thank you for providing the example, that’s very nice!

@rob Please let us know if you need any help.

@YangminZhu the Istio example End-user authentication is showing the JWT policy on the service httpbin, are you saying the JWT policy won’t work if traffic is coming from inside the mesh?

cat <<EOF | kubectl apply -n foo -f -
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "jwt-example"
spec:
  targets:
  - name: httpbin
  origins:
  - jwt:
      issuer: "testing@secure.istio.io"
      jwksUri: "https://raw.githubusercontent.com/istio/istio/release-1.2/security/tools/jwt/samples/jwks.json"
  principalBinding: USE_ORIGIN
EOF

@dwctua From what I’ve been trying to do, if JWT origin auth is set on a service, then any traffic (whether its coming from the outside into the gateway and through to that service or if its coming from anther internal service) will require JWT auth.

The example given above about switching JWT auth to the ingressgateway service/pod does work. I have verified that. In that case external traffic coming into the gateway will require JWT auth but none of the internal services will. The only piece I still need to try is to have multiple JWKS entries (from what the filter does I suspect this will work) but I’m not sure i’ll get to verifying that will work until later today or Monday.

Thanks for clarification @rob! I misread your examples. Personally I would prefer to have a consistent authentication regardless where is the traffic coming from.