Istio Ingress IP whitelisting

Hello guys,

I would like to allow access to my K8S cluster only from some set of IPs. So I’ve implemented the approach discussed here https://istio.io/docs/tasks/policy-enforcement/denial-and-list/#ip-based-whitelists-or-blacklists.

The problem I am facing is that origin.ip value does not contain the original IP address of visitor but load balancer IP. When I switch Istio ingress gateway externalTrafficPolicy to Local, correct origin.ip is propagated.

But when externalTrafficPolicy is set to Local network routing stop working with error upstream connect error or disconnect/reset before headers. reset reason: connection termination.

I am on Istio 1.1.8.

Could anyone advise?

Did you figure out how to solve this?

Sadly nope. But did not tried with newer version of Istio.

@YangminZhu Could you please shed some light on this? white-listing is a very common practice, but seems Istio doesn’t have any support on this. is there any work around for time being?

I tested on Istio 1.2 with a RBAC policy for IP whitelisting on Ingress gateway:

apiVersion: rbac.istio.io/v1alpha1
kind: ClusterRbacConfig
metadata:
  name: default
spec:
  mode: "ON"
---
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRole
metadata:
  name: istio-ingressgateway
  namespace: istio-system
spec:
  rules:
  - services: ["istio-ingressgateway.istio-system.svc.cluster.local"]
    methods: ["*"]
---
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRoleBinding
metadata:
  name: istio-ingressgateway-binding
  namespace: istio-system
spec:
  subjects:
  - properties:
      source.ip: 1.2.3.0/24
  - properties:
      source.ip: 5.6.7.8
  roleRef:
    kind: ServiceRole
    name: "istio-ingressgateway"

and

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gateway
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
  namespace: default
spec:
  gateways:
  - httpbin-gateway
  hosts:
  - '*'
  http:
  - route:
    - destination:
        host: httpbin
        port:
          number: 8000

I also set externalTrafficPolicy to Local in order to expose the real client IP.

This works well for me and the ingress gateway is able to forward the request to my httpbin backend in the cluster.

The error upstream connect error or disconnect/reset before headers. reset reason: connection termination. seems like some TLS connection issue.

@jaygridley, did you apply any TLS setting on the ingress gateway or any Authentication policy in the cluster? It might be also worth to try it again in Istio 1.2.

Thanks for looking into this. Your solution is interesting. Since you’re also using externalTrafficPolicy to Local to get the real client IP. What are the recommendation/best practice for this, using RBAC or white-list policy assume both works?

I’m not familiar withexternalTrafficPolicy and not sure about the best practice here, you might want to check with someone from the networking, cc @rshriram @Costin_Manolache1

I think both RBAC and white-list policy should just work in this case. Note I would recommend to use RBAC in this case as it’s enforced right in the envoy process which is much faster as it doesn’t send the check to a remote server.

Makes sense. I think this could be a rule of thumb. i.e. envoy first.

I’ve tried RBAC approach with externalTrafficPolicy set to Local, but source.ip seems to contain IP chain, e.g. 172.29.3.75,164.38.33.33 so my whitelist does not match. Can Envoy filter this out and populate source.ip with real remote IP?

I’ve checked IP whistelisting via Istio 1.2.2 and policies and still the same issue as described in my original post.

It seems like applying the policy breaks communication inside the cluster as the error is reported from multiple istio-proxies.

I have mTLS enabled cluster wide.

Because your source.ip has mutliple ip addresses you can use regex. If you want to.

I have used request.headers["x-envoy-external-address"] in handler to retrieve the client IP together with externalTrafficPolicy set to Local