Istio Ingress IP whitelisting

Networking
1

Hello guys,

I would like to allow access to my K8S cluster only from some set of IPs. So I’ve implemented the approach discussed here https://istio.io/docs/tasks/policy-enforcement/denial-and-list/#ip-based-whitelists-or-blacklists.

The problem I am facing is that origin.ip value does not contain the original IP address of visitor but load balancer IP. When I switch Istio ingress gateway externalTrafficPolicy to Local, correct origin.ip is propagated.

But when externalTrafficPolicy is set to Local network routing stop working with error upstream connect error or disconnect/reset before headers. reset reason: connection termination.

I am on Istio 1.1.8.

Could anyone advise?

1 Like
2

Did you figure out how to solve this?

3

Sadly nope. But did not tried with newer version of Istio.

4

@YangminZhu Could you please shed some light on this? white-listing is a very common practice, but seems Istio doesn’t have any support on this. is there any work around for time being?

5

I tested on Istio 1.2 with a RBAC policy for IP whitelisting on Ingress gateway:

apiVersion: rbac.istio.io/v1alpha1
kind: ClusterRbacConfig
metadata:
  name: default
spec:
  mode: "ON"
---
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRole
metadata:
  name: istio-ingressgateway
  namespace: istio-system
spec:
  rules:
  - services: ["istio-ingressgateway.istio-system.svc.cluster.local"]
    methods: ["*"]
---
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRoleBinding
metadata:
  name: istio-ingressgateway-binding
  namespace: istio-system
spec:
  subjects:
  - properties:
      source.ip: 1.2.3.0/24
  - properties:
      source.ip: 5.6.7.8
  roleRef:
    kind: ServiceRole
    name: "istio-ingressgateway"

and

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gateway
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
  namespace: default
spec:
  gateways:
  - httpbin-gateway
  hosts:
  - '*'
  http:
  - route:
    - destination:
        host: httpbin
        port:
          number: 8000

I also set externalTrafficPolicy to Local in order to expose the real client IP.

This works well for me and the ingress gateway is able to forward the request to my httpbin backend in the cluster.

The error upstream connect error or disconnect/reset before headers. reset reason: connection termination. seems like some TLS connection issue.

@jaygridley, did you apply any TLS setting on the ingress gateway or any Authentication policy in the cluster? It might be also worth to try it again in Istio 1.2.

6

Thanks for looking into this. Your solution is interesting. Since you’re also using externalTrafficPolicy to Local to get the real client IP. What are the recommendation/best practice for this, using RBAC or white-list policy assume both works?

7

I’m not familiar withexternalTrafficPolicy and not sure about the best practice here, you might want to check with someone from the networking, cc @rshriram @Costin_Manolache1

I think both RBAC and white-list policy should just work in this case. Note I would recommend to use RBAC in this case as it’s enforced right in the envoy process which is much faster as it doesn’t send the check to a remote server.

8

Makes sense. I think this could be a rule of thumb. i.e. envoy first.

9

I’ve tried RBAC approach with externalTrafficPolicy set to Local, but source.ip seems to contain IP chain, e.g. 172.29.3.75,164.38.33.33 so my whitelist does not match. Can Envoy filter this out and populate source.ip with real remote IP?

10

I’ve checked IP whistelisting via Istio 1.2.2 and policies and still the same issue as described in my original post.

It seems like applying the policy breaks communication inside the cluster as the error is reported from multiple istio-proxies.

I have mTLS enabled cluster wide.

11

Because your source.ip has mutliple ip addresses you can use regex. If you want to.

12

I have used request.headers["x-envoy-external-address"] in handler to retrieve the client IP together with externalTrafficPolicy set to Local

13

@romachalm.

I am trying exactly the same thing. I have the following config as a starter

apiVersion: config.istio.io/v1alpha2
kind: handler
metadata:
  name: blacklistip
spec:
  compiledAdapter: listchecker

  params:
    providerUrl: http://urlist.com/ip.txt
    overrides: ["0.0.0.0.25"]
    blacklist: true
    entryType: IP_ADDRESSES
---
apiVersion: config.istio.io/v1alpha2
kind: instance
metadata:
  name: sourceip-blacklist
spec:
  compiledTemplate: listentry
  params:
    value: source.ip | ip("0.0.0.0")
---
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
  name: checkip-whitelist
spec:
  match: source.labels["istio"] == "ingressgateway"
  actions:
  - handler: blacklistip
    instances: [ sourceip-blacklist ]

What parameter do i need to set in the handler? Could you help me out. I have already set externalTrafficPolicy to Local for the istio ingressgateway service.

@jaygridley. Did you get this running? Could you share me the config please if possible?

Regards,
Kevin

15

@kevin_fernandes,

The problem for external IPs is that source.ip contains the IP of the ingress gateway pod, and not the real IP. When the request passes though the ingress gateway, a header x-envoy-external-address is added to the request. It is not injected when the request comes from the internal network though. So you have to filter in the match to avoid some nil error

My configuration is the following trying to map yours :

apiVersion: config.istio.io/v1alpha2
kind: handler
metadata:
  name: blacklistip
  namespace: istio-system
spec:
  compiledAdapter: listchecker
  params:
    blacklist: false
    entryType: IP_ADDRESSES
    overrides:
    - 0.0.0.25/8
---
apiVersion: config.istio.io/v1alpha2
kind: instance
metadata:
  name: sourceip-blacklist
  namespace: istio-system
spec:
  compiledTemplate: listentry
  params:
    value: request.headers["x-envoy-external-address"] | "0.0.0.0"
---
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
  name: sourceip-blacklist
  namespace: istio-system
spec:
  actions:
  - handler: blacklistip
    instances:
    - sourceip
  match: (source.labels["istio"] | "") == "ingressgateway"  && (request.headers["x-envoy-external-address"] | "") !=""

Also, the external traffic policy has to be set for ingress-gaterway. Using helm, I set it with :

--set gateways.istio-ingressgateway.externalTrafficPolicy=Local

Hope this helps

16

As Mixer has been deprecated in 1.5, please consider using authorization policy to allow/deny requests. If you have more concerns with new API, please file an issue.

1 Like