Forward source IP of HTTPS request

raj · April 22, 2020, 12:52pm

Is it possible to forward the client source IP to the container pod that is serving the request?
I have the pod and service serving the HTTPS request on port 9443. istio-ingressgateway has externalTrafficPolicy: Local
The forwarded headers are not added by proxy or ingress gateway. Is it due to HTTPS request or the filters are not in the correct namespace (I tried moving them to local namespace instead of istio-system, but that didn’t help)?

Service:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: controller-auth
  name: controller-auth-service
  namespace: controller
spec:
  ports:
  - name: https-auth
    port: 9443
    protocol: TCP
    targetPort: 9443
  selector:
    app: controller-auth
  type: ClusterIP


apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: controller-auth-gw
  namespace: controller
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: https
      number: 9443
      protocol: HTTPS
    tls:
      httpsRedirect: true

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: controller-auth-vs
  namespace: controller
spec:
  gateways:
  - controller-auth-gw
  hosts:
  - "*"
  tls:
  - match:
    - port: 9443
      sniHosts:
      - '*'
    route:
    - destination:
        host: controller-auth-service.controller.svc.cluster.local
        port:
          number: 9443




apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: auth-lua
  namespace: controller
spec:
  workloadSelector:
    labels:
      app: controller-auth
  configPatches:
    # The first patch adds the lua filter to the listener/http connection manager
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        portNumber: 9443
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
            subFilter:
              name: "envoy.router"
    patch:
      operation: INSERT_BEFORE
      value: # lua filter specification
       name: envoy.lua
       config:
         inlineCode: |
           function envoy_on_request(request_handle)
             local xff_header = request_handle:headers():get("X-Forwarded-For")
             local first_ip = string.gmatch(xff_header, "(%d+.%d+.%d+.%d+)")();
             first_ip = string.gsub(first_ip, ",", "")
             request_handle:headers():add("X-Custom-User-IP", first_ip);
             request_handle:headers():add("X-Test-User-IP", "1.1.1.1");
           end

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: xff-trust-hops
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: ANY
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager"
          use_remote_address: true
          xff_num_trusted_hops: 1

istioctl version
client version: 1.5.1
control plane version: 1.5.1
data plane version: 1.5.1 (8 proxies)

kubectl version --short
Client Version: v1.17.2
Server Version: v1.16.6

erSitzt · April 22, 2020, 3:39pm

Its very hard to read your yaml if you dont post it as preformatted text !

You can see the source ip at the point where SSL is terminated. As long as its still encrypted you cant modify any http headers, because you are not dealing with http
If your clients access any of your services via a proxy you can only see the ip of the proxy as the source ip, as long as you dont use the PROXY-protocol (https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) and even then all proxies in the proxy chain need to support it… i think…

I might be wrong about several things, but thats what i experienced and read on the internet…

raj · April 23, 2020, 5:08am

@erSitzt Thanks for your quick reply. I tried, formatting the yaml, but noticed that HTML is replacing the “-” with bullet points!!! Yes, without indentation, it is hard to read.
Isn’t the SSL terminated at container (where the certificates are present)? If not, it must be worker node host (or loadbalancer) in which case can’t the ingress gateway or envoy proxy add the header as it is post SSL termination?
I tried with interceptionMode as TPROXY instead of REDIRECT, still I see 127.0.0.1 as source IP at envoy proxy:
[2020-04-22T12:43:39.905Z] “- - -” 0 - “-” “-” 4686 2877 10759 - “-” “-” “-” “-” “127.0.0.1:9443” inbound|9443|https-auth|controller-auth-service.controller.svc.cluster.local 127.0.0.1:51364 10.42.3.83:9443 10.42.2.80:54614 outbound_.9443_._.controller-auth-service.controller.svc.cluster.local -

Is there a way I can specify the certificates in the gateway, add the source IP in the header and forward it as HTTP to application container?

Or is there any other way that the application would get to know of the source IP for an HTTPS request?

Thanks again
Raj

erSitzt · April 23, 2020, 6:41am

pre-formatted text

raj · April 23, 2020, 6:51am

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: controller-auth-gw
  namespace: controller
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: https
      number: 9443
      protocol: HTTPS
    tls:
      httpsRedirect: true



apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: controller-auth-vs
  namespace: controller
spec:
  gateways:
  - controller-auth-gw
  hosts:
  - '*'
  tls:
  - match:
    - port: 9443
      sniHosts:
      - '*'
    route:
    - destination:
        host: controller-auth-service.controller.svc.cluster.local
        port:
          number: 9443


apiVersion: v1
kind: Service
metadata:
  labels:
    app: controller-auth
  name: controller-auth-service
  namespace: controller
spec:
  ports:
  - name: https-auth
    port: 9443
    protocol: TCP
    targetPort: 9443
  selector:
    app: controller-auth
  type: ClusterIP

erSitzt · April 23, 2020, 6:52am

… and you might want to look at externalTrafficPolicy theres an active thread about the x-forwarded-for headers

erSitzt · April 23, 2020, 6:56am

But still… that wont help if there are any proxies between your client and your ingress-gateway… i think

raj · April 23, 2020, 7:16am

Yes, if there are proxies, I may still receive the proxy IP. But, at present I’m receiving 127.0.0.1 for all the requests, I would prefer an option to know the IP that ingress-gateway is receiving the request from.

raj · April 23, 2020, 7:29am

Thanks, I did look and posted the query there too. I have set the externalTrafficPolicy to Local
Actually, I do not prefer the option of setting externalTrafficPolicy to local, but it is better than not receiving the client IP details.

erdun · April 23, 2020, 8:00am

When proto is HTTP, there has filter envoy.http_connection_manager, HTTP listener filters

HTTPS listener filters

Your EnvoyFilter is only work on HTTP port not work HPPTS port

raj · April 23, 2020, 10:45am

Thanks @erdun, awesome info. It really makes sense why my filter wasn’t working. From where can I get/view this filter_chain hierarchy? Also, does a similar strategy (like adding code to envoy_on_request(request_handle)) work for the network.wasm or tcp_proxy. I guess, tcp_proxy may not get to know of the application/HTTP headers. So, is there any sample envoy filters for these filter_chains?
I would appreciate any pointer in the right direction.

Thanks again for the great help already done.

erdun · April 24, 2020, 2:36am

Filter_chain is envoy’s config, get envoy config

kubectl exec -ti ${podname} -c istio-proxy curl localhost:15000/config_dump > /tmp/config

All envoy filters
https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/filter/filter

The most convenient way to get source IP is setting use_remote_address to true in HTTP connection manager, or you can write wasm plugins, I haven’t found other ways, still trying to figure it out.

thiago_leite · July 29, 2020, 1:44pm

Hi all!

I have a simular issue and I need to preserve the source ip in x-forwarded-for:

I’ve just upgrade from 1.4.5(helm) to 1.6.5 (istioctl + istio operator) and I’m using NodePort 31380 for my ingress gateway and it is running as “deployment”:

–> ingress gateway service:
apiVersion: v1 kind: Service metadata: name: istio-ingressgateway namespace: istio-system spec: clusterIP: 10.43.34.60 externalTrafficPolicy: Cluster ports: - name: status-port nodePort: 30365 port: 15021 protocol: TCP targetPort: 15021 - name: http2 nodePort: 31380 port: 80 protocol: TCP targetPort: 8080 - name: https nodePort: 31390 port: 443 protocol: TCP targetPort: 8443 - name: tls nodePort: 31839 port: 15443 protocol: TCP targetPort: 15443 selector: app: istio-ingressgateway istio: ingressgateway sessionAffinity: None type: NodePort

–> ingress gateway deployment:
apiVersion: apps/v1 kind: Deployment metadata: annotations: labels: app: istio-ingressgateway spec: progressDeadlineSeconds: 600 replicas: 2 revisionHistoryLimit: 10 selector: matchLabels: app: istio-ingressgateway istio: ingressgateway strategy: rollingUpdate: maxSurge: 100% maxUnavailable: 25% type: RollingUpdate ..

I’m using AWS NLB with 100 backend nodes, I have a listener in port 80/443 and forward to port 31380 k8s worker nodes.

So because I’m using ingress gateway as deployment(I didnt see an option to deploy it daemonset) I cant use use “externalTrafficPolicy”: “Local”. If I’m not wrong, If I setup like that “externalTrafficPolicy”=“Local” the NLB will endup send my requests to a backend server that there isnt a ingress gateway pod running and the package will dropped.

In summary:

Looks like I need to deploy my ingress gateway as daemonset, then I’ll be able to use “externalTrafficPolicy”: “Local” and preserver my source ip address, but I dont have this option in istioctl/operator.

are there other options to preserve the source ip with nodeport + aws nlb ?

please advise

Thanks !

Topic		Replies	Views
TLS termination at edge Envoy (with nginx pod)	0	617	February 26, 2019
Can't access my nginx pod from an external IP Networking	0	252	November 12, 2022
Source IP from TCP gateway Networking	0	464	June 26, 2020
Original source ip address of client	0	1005	February 15, 2023
Ingress Gateway - Decrypt then send to a POD listening on https Security	3	1421	August 31, 2021

Forward source IP of HTTPS request

Related topics