How to deny/allow connections to external services by source namespace?

Hi Experts!

Can you help me configure this use case, please?

I have two teams (dev1 and dev2) with pods in two namespaces (dev1 and dev2).
dev1 team should only have access to external service dev1.example.com and
dev2 team should only have access to dev2.example.com.
I want to only use the Istio mesh configuration to solve this and not use any kind of firewall features of the underlying platform/infra.

I started off by…

  • setting REGISTRY_ONLY to disallow all mesh egress traffic.
  • I then added ServiceEntry for all external services, e.g. dev1. example.com and dev2. example. com.
  • I then tried to apply AuthorizationPolicy to deny access to the specific external services depending on the source ns (dev1 or dev2).
  • I tried this with, and also without using, egress gateway, but pods in both dev1 and dev2 namespaces can still access all external services. I only want pods in dev1 ns to access dev1. example. com and pods in dev2 ns to access dev2. example. com.

Any guidance/pointer on how this can be achieved is much appreciated.

Thank you!

I have tried with the following config on a fresh install of istio 1.10 on kind.
The test is below the config:

The config

apiVersion: v1
kind: Namespace
metadata:
  name: dev1
  labels:
    istio-injection: enabled 
---
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: bbc
  namespace: dev1
spec:
  hosts:
  - www.bbc.com
  ports:
  - number: 80
    name: http-port
    protocol: HTTP
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: istio-egressgateway
  namespace: dev1
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - www.bbc.com
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: egressgateway-for-bbc
  namespace: dev1
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
  - name: bbc

---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: direct-bbc-through-egress-gateway
  namespace: dev1
spec:
  hosts:
  - www.bbc.com
  gateways:
  - istio-egressgateway
  - mesh
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        subset: bbc
        port:
          number: 80
      weight: 100
  - match:
    - gateways:
      - istio-egressgateway
      port: 80
    route:
    - destination:
        host: www.bbc.com
        port:
          number: 80
      weight: 100
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-bbc
  namespace: dev1
spec:
  action: DENY
  rules:
  - from:
    - source:
        namespaces: ["dev1"]
    to:
    - operation:
        hosts: ["www.bbc.com"]

How I test:

$ k create deployment fedora-dev1 -n dev1 --image=fedora -- sleep 999999

$ k get po -n dev1
NAME                           READY   STATUS    RESTARTS   AGE
fedora-dev1-6447bc7cd4-77rm2   2/2     Running   0          12m

# This should fail 
$ k exec fedora-dev1-6447bc7cd4-77rm2 -n dev1 -c fedora -- curl -o /dev/null -s -w "%{http_code}\n" -IL www.bbc.com
200

# If I remove the SE, it fails as expected
$ k delete serviceentries -n dev1 --all 

$ k exec fedora-dev1-6447bc7cd4-77rm2 -n dev1 -c fedora -- curl -o /dev/null -s -w "%{http_code}\n" -IL www.bbc.com
503

# Add back the SE
$ k create -f se.yaml 
serviceentry.networking.istio.io/bbc created

$ k exec fedora-dev1-6447bc7cd4-77rm2 -n dev1 -c fedora -- curl -o /dev/null -s -w "%{http_code}\n" -IL www.bbc.com
200

The egress GW log shows:

$ kubectl logs -l istio=egressgateway -c istio-proxy -n istio-system -f
[2021-06-09T08:20:21.646Z] "HEAD / HTTP/2" 503 NC cluster_not_found - "-" 0 0 0 - "10.244.0.8" "curl/7.76.1" "30fe7eaa-e3f0-9bd9-8e75-58158344dec2" "www.bbc.com" "-" - - 10.244.0.7:8080 10.244.0.8:33992 - -
[2021-06-09T08:20:25.345Z] "HEAD / HTTP/2" 503 NC cluster_not_found - "-" 0 0 0 - "10.244.0.8" "curl/7.76.1" "9308b6fd-440f-9144-902d-a936bcab147b" "www.bbc.com" "-" - - 10.244.0.7:8080 10.244.0.8:33992 - -
[2021-06-09T08:20:25.362Z] "HEAD / HTTP/2" 503 NC cluster_not_found - "-" 0 0 0 - "10.244.0.8" "curl/7.76.1" "9308b6fd-440f-9144-902d-a936bcab147b" "www.bbc.com" "-" - - 10.244.0.7:8080 10.244.0.8:33992 - -
[2021-06-09T08:20:25.408Z] "HEAD / HTTP/2" 503 NC cluster_not_found - "-" 0 0 0 - "10.244.0.8" "curl/7.76.1" "9308b6fd-440f-9144-902d-a936bcab147b" "www.bbc.com" "-" - - 10.244.0.7:8080 10.244.0.8:33992 - -
[2021-06-09T08:20:34.505Z] "HEAD / HTTP/2" 301 - via_upstream - "-" 0 0 552 552 "10.244.0.8" "curl/7.76.1" "26d66211-1b4d-9c4d-bb88-924e933ecdc6" "www.bbc.com" "212.58.237.248:80" outbound|80||www.bbc.com 10.244.0.7:44122 10.244.0.7:8080 10.244.0.8:33992 - -
[2021-06-09T08:23:11.616Z] "HEAD / HTTP/2" 301 - via_upstream - "-" 0 0 582 581 "10.244.0.8" "curl/7.76.1" "85a7c314-4638-9845-baae-b54e4ae8dd3d" "www.bbc.com" "212.58.233.248:80" outbound|80||www.bbc.com 10.244.0.7:46716 10.244.0.7:8080 10.244.0.8:33992 - -
[2021-06-09T08:23:31.099Z] "HEAD / HTTP/2" 503 NC cluster_not_found - "-" 0 0 0 - "10.244.0.8" "curl/7.76.1" "8cbec0a8-379f-9ab1-9333-ae6ac59426f1" "www.bbc.com" "-" - - 10.244.0.7:8080 10.244.0.8:34062 - -
[2021-06-09T08:23:31.112Z] "HEAD / HTTP/2" 503 NC cluster_not_found - "-" 0 0 0 - "10.244.0.8" "curl/7.76.1" "8cbec0a8-379f-9ab1-9333-ae6ac59426f1" "www.bbc.com" "-" - - 10.244.0.7:8080 10.244.0.8:34062 - -
[2021-06-09T08:23:31.153Z] "HEAD / HTTP/2" 503 NC cluster_not_found - "-" 0 0 0 - "10.244.0.8" "curl/7.76.1" "8cbec0a8-379f-9ab1-9333-ae6ac59426f1" "www.bbc.com" "-" - - 10.244.0.7:8080 10.244.0.8:34062 - -
[2021-06-09T08:23:43.365Z] "HEAD / HTTP/2" 301 - via_upstream - "-" 0 0 543 542 "10.244.0.8" "curl/7.76.1" "e347c911-ced8-9299-8f68-5d3813ca1e6f" "www.bbc.com" "212.58.233.248:80" outbound|80||www.bbc.com 10.244.0.7:47160 10.244.0.7:8080 10.244.0.8:33992 - -

Try to set ServiceEntry.ExportTo, which allows you to specify which namespaces the service can be exported to. AuthorizationPolicy does not work, it only works for application with sidecar.

Your authorization policy needs to apply at Egress Gateway workload which I’m assuming is installed in the istio-system namespace. You can use the corresponding egress GW workload selector and deploy the AuthZ policy in istio-system to enforce this behavior.

Basicallly. this should be the flow:

app (namespace: dev10 —> sidecar → (Authz policy checks) Egress GW —> BBC

Thank you @nrjpoddar and @hzxuzhonghu, that really helped and I got the simple use-case working using the examples, but, when I want to deny access based on ns using “source:” it does not work…

This is what I’m after:

app (namespace: dev1) —> sidecar --> (DENY  from dev1 ns) Egress GW —> CNN
app (namespace: dev2) —> sidecar --> (ALLOW from dev2 ns) Egress GW —> CNN

The following, 1st authorization policy in “istio-system” ns works fine (without “source:” defined).

1st policy (blocks the request as expected):

kind: AuthorizationPolicy
metadata:
  name: deny-cnn-politics-egress
spec:
  selector:
    matchLabels:
      istio: egressgateway
  action: DENY
  rules:
  - to:
    - operation:
        hosts: ["edition.cnn.com"]
        paths: ["/politics"]
kubectl exec "$SOURCE_POD" -c sleep -- curl -sS -o /dev/null -D - http://edition.cnn.com/politics
HTTP/1.1 403 Forbidden
...

This Authz Pol (which uses “default” ns as source) does not work:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-cnn-politics-egress
spec:
  selector:
    matchLabels:
      istio: egressgateway
  action: DENY
  rules:
  - from:
    - source:
        namespaces: ["default"]
    to:
    - operation:
        hosts: ["edition.cnn.com"]
        paths: ["/politics"]
kubectl exec "$SOURCE_POD" -c sleep -- curl -sS -o /dev/null -D - http://edition.cnn.com/politics
HTTP/1.1 301 Moved Permanently
...

Any ideas why http://edition.cnn.com/politics can still be reached using the 2nd policy although the 1st policy can block the request, as expected?

I read that mTLS mutual auth needs to be configured. I have tested the bookinfo app and verified using Kiali that all traffic between bookinfo services is secured with mTLS (lock icon). When I view the sleep pod and the egress gw in Kiali, it shows no lock icon. I would expect mTLS to be in use for AuthZ rbac to work. I assume I need to activate mTLS between the sleep pod and the egress gw. How can I get that to work, please?

I tried to add the following to the “egressgateway-for-cnn” DR, but I get 503s and the error “upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER”

trafficPolicy:
  tls:
   mode: ISTIO_MUTUAL

Any more help much appreciated.

Here are all my settings, which are from the examples:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: cnn
  namespace: default
spec:
  hosts:
  - edition.cnn.com
  ports:
  - name: http-port
    number: 80
    protocol: HTTP
  - name: https
    number: 443
    protocol: HTTPS
  resolution: DNS
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: istio-egressgateway
  namespace: default
spec:
  selector:
    istio: egressgateway
  servers:
  - hosts:
    - edition.cnn.com
    port:
      name: http
      number: 80
      protocol: HTTP
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: egressgateway-for-cnn
  namespace: default
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
  - name: cnn
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: direct-cnn-through-egress-gateway
  namespace: default
spec:
  gateways:
  - istio-egressgateway
  - mesh
  hosts:
  - edition.cnn.com
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        port:
          number: 80
        subset: cnn
      weight: 100
  - match:
    - gateways:
      - istio-egressgateway
      port: 80
    route:
    - destination:
        host: edition.cnn.com
        port:
          number: 80
      weight: 100
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-cnn-politics-egress
  namespace: istio-system
spec:
  action: DENY
  rules:
  - from:
    - source:
        namespaces:
        - default
    to:
    - operation:
        hosts:
        - edition.cnn.com
        paths:
        - /politics
  selector:
    matchLabels:
      istio: egressgateway

Hi @nrjpoddar and @hzxuzhonghu

I now realize I was going in the wrong direction and have got it working without using AuthorizationPolicy (as per your suggestion). I had to be careful trying different combinations of ExportTo. I can now add or remove the ServiceEntry in the source namespace to allow or deny access to the external service. If the SE is not present, the client receives a 503 Service unavailable. All traffic is routed through the egress gw.

Here is the config:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: cnn
spec:
  hosts:
  - edition.cnn.com
  exportTo:
  - "."
  - "istio-system"
  ports:
  - number: 80
    name: http-port
    protocol: HTTP
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - edition.cnn.com
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-for-cnn
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  exportTo:
  - "."
  subsets:
  - name: cnn
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-cnn-through-egress-gateway
spec:
  hosts:
  - edition.cnn.com
  exportTo:
  - "."
  - "istio-system"
  gateways:
  - istio-egressgateway
  - mesh
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        subset: cnn
        port:
          number: 80
      weight: 100
  - match:
    - gateways:
      - istio-egressgateway
      port: 80
    route:
    - destination:
        host: edition.cnn.com
        port:
          number: 80
      weight: 100