TPROXY isn't applied at the ingress gateway if configured in the default mesh options

I configured my service mesh using the Istio CNI and set the default mesh options to use TPROXY by default.

spec:
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
    defaultConfig:
      interceptionMode: TPROXY

However my pods still are receiving packets with a source ip that matches that of the istio ingress gateway. I tried instead using a proxy config annotation on the ingress gateway deployment, but that didn’t change anything either.

According to the gateway log pods, TPROXY is indeed set. I’m confused… why am I getting packet redirects?

2022-03-11T17:43:44.935171Z     info    FLAG: --concurrency="0"
2022-03-11T17:43:44.935321Z     info    FLAG: --domain="istio-system.svc.cluster.local"
2022-03-11T17:43:44.935370Z     info    FLAG: --help="false"
2022-03-11T17:43:44.935376Z     info    FLAG: --log_as_json="false"
2022-03-11T17:43:44.935378Z     info    FLAG: --log_caller=""
2022-03-11T17:43:44.935381Z     info    FLAG: --log_output_level="default:info"
2022-03-11T17:43:44.935384Z     info    FLAG: --log_rotate=""
2022-03-11T17:43:44.935387Z     info    FLAG: --log_rotate_max_age="30"
2022-03-11T17:43:44.935390Z     info    FLAG: --log_rotate_max_backups="1000"
2022-03-11T17:43:44.935394Z     info    FLAG: --log_rotate_max_size="104857600"
2022-03-11T17:43:44.935396Z     info    FLAG: --log_stacktrace_level="default:none"
2022-03-11T17:43:44.935405Z     info    FLAG: --log_target="[stdout]"
2022-03-11T17:43:44.935409Z     info    FLAG: --meshConfig="./etc/istio/config/mesh"
2022-03-11T17:43:44.935412Z     info    FLAG: --outlierLogPath=""
2022-03-11T17:43:44.935415Z     info    FLAG: --proxyComponentLogLevel="misc:error"
2022-03-11T17:43:44.935418Z     info    FLAG: --proxyLogLevel="warning"
2022-03-11T17:43:44.935421Z     info    FLAG: --serviceCluster="istio-proxy"
2022-03-11T17:43:44.935424Z     info    FLAG: --stsPort="0"
2022-03-11T17:43:44.935427Z     info    FLAG: --templateFile=""
2022-03-11T17:43:44.935429Z     info    FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2022-03-11T17:43:44.935434Z     info    Version 1.11.7-bdacf35447b2d9903e092acfaf12098b997b2780-Clean
2022-03-11T17:43:44.935944Z     info    Proxy role      ips=[10.42.1.5 fe80::c489:baff:fe87:b2f3] type=router id=istio-ingressgateway-5cdf79b54f-nfg8m.istio-system domain=istio-system.svc.cluster.local
2022-03-11T17:43:44.936183Z     info    Apply mesh config from file accessLogFile: /dev/stdout
defaultConfig:
  discoveryAddress: istiod.istio-system.svc:15012
  interceptionMode: TPROXY
  proxyMetadata: {}
  tracing:
    zipkin:
      address: zipkin.istio-system:9411
enablePrometheusMerge: true
enableTracing: true
rootNamespace: istio-system
trustDomain: cluster.local
2022-03-11T17:43:44.938064Z     info    Effective config: binaryPath: /usr/local/bin/envoy
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
interceptionMode: TPROXY
parentShutdownDuration: 60s
proxyAdminPort: 15000
proxyMetadata: {}
serviceCluster: istio-proxy
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
  zipkin:
    address: zipkin.istio-system:9411

2022-03-11T17:43:44.938084Z     info    JWT policy is third-party-jwt
2022-03-11T17:43:44.940306Z     info    CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2022-03-11T17:43:44.940336Z     info    Opening status port 15020
2022-03-11T17:43:44.940395Z     info    Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2022-03-11T17:43:44.940569Z     info    citadelclient   Citadel client using custom root cert: istiod.istio-system.svc:15012
2022-03-11T17:43:45.011754Z     info    ads     All caches have been synced up in 84.436356ms, marking server ready
2022-03-11T17:43:45.014464Z     info    sds     SDS server for workload certificates started, listening on "etc/istio/proxy/SDS"
2022-03-11T17:43:45.014493Z     info    xdsproxy        Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2022-03-11T17:43:45.014583Z     info    sds     Starting SDS grpc server
2022-03-11T17:43:45.016246Z     info    Pilot SAN: [istiod.istio-system.svc]
2022-03-11T17:43:45.016318Z     info    starting Http service at 127.0.0.1:15004
2022-03-11T17:43:45.020714Z     info    Starting proxy agent
2022-03-11T17:43:45.020749Z     info    Epoch 0 starting
2022-03-11T17:43:45.020770Z     info    Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --drain-strategy immediate --parent-shutdown-time-s 60 --local-address-ip-version v4 --bootstrap-version 3 --file-flush-interval-msec 1000 --disable-hot-restart --log-format %Y-%m-%dT%T.%fZ    %l      envoy %n        %v -l warning --component-log-level misc:error]
2022-03-11T17:43:45.143586Z     info    cache   generated new workload certificate      latency=131.34561ms ttl=23h59m59.856427308s
2022-03-11T17:43:45.143656Z     info    cache   Root cert has changed, start rotating root cert
2022-03-11T17:43:45.143687Z     info    ads     XDS: Incremental Pushing:0 ConnectedEndpoints:0 Version:
2022-03-11T17:43:45.143797Z     info    cache   returned workload trust anchor from cache       ttl=23h59m59.856208089s
2022-03-11T17:43:45.177342Z     info    xdsproxy        connected to upstream XDS server: istiod.istio-system.svc:15012
2022-03-11T17:43:45.194107Z     info    ads     ADS: new connection for node:istio-ingressgateway-5cdf79b54f-nfg8m.istio-system-2
2022-03-11T17:43:45.194122Z     info    ads     ADS: new connection for node:istio-ingressgateway-5cdf79b54f-nfg8m.istio-system-1
2022-03-11T17:43:45.194206Z     info    cache   returned workload certificate from cache        ttl=23h59m59.805801149s
2022-03-11T17:43:45.194206Z     info    cache   returned workload trust anchor from cache       ttl=23h59m59.805798485s
2022-03-11T17:43:45.194475Z     info    ads     SDS: PUSH request for node:istio-ingressgateway-5cdf79b54f-nfg8m.istio-system resources:1 size:4.0kB resource:default
2022-03-11T17:43:45.194647Z     info    ads     SDS: PUSH request for node:istio-ingressgateway-5cdf79b54f-nfg8m.istio-system resources:1 size:1.1kB resource:ROOTCA
2022-03-11T17:43:46.472794Z     info    Initialization took 1.543587582s
2022-03-11T17:43:46.472811Z     info    Envoy proxy is ready
2022-03-11T18:13:09.297457Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 14, closing transport due to: connection error: desc = "error reading from server: EOF", received prior goaway: code: NO_ERROR, debug data:

2022-03-11T18:13:09.728853Z     info    xdsproxy        connected to upstream XDS server: istiod.istio-system.svc:15012
[2022-03-11T18:13:34.540Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 16 14 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "087da8b4-72fc-4f2a-8230-f529518e892f" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:35.873Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 1 1 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "4fb8f1c3-6e74-4365-b5cb-7168cacb51fe" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:35.965Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "db9483fa-96f1-4474-80b7-9ff83243689a" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:36.998Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "7090895e-1810-4eed-a444-3652c657f1c8" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:37.055Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "ec5482a5-9fad-476d-92cf-61be5114deb4" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:37.575Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 1 1 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "357a89e9-a2b8-403e-a832-96ba1cd228ed" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:37.633Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "81a8f84f-91e1-4ff8-9532-5bcae1f1d132" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:37.754Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "4a545ed8-4d15-4224-85cc-4e9ce1966d35" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:37.824Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "92c0f175-cfc1-4a8f-933f-e1ccf0f8cb5e" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:37.924Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "87f1a42a-cb75-434a-9eff-3e944ffdc4e3" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:37.977Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "9457dc4c-007f-42c9-98f3-9dcbf6b1101c" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:38.097Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "bd5fa328-4af2-4ffe-92ab-a9c900508d21" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:38.150Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "43b0fbf2-5587-4a08-a5f4-32e5697d94b1" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:38.288Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "8d5c8204-9e90-941c-b0da-7913c952b3a3" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:38.345Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "a17ea846-9b46-4894-9955-00291dec4b1f" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:38.466Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "7f219cc4-1954-464e-9329-1ed37a6222e3" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:38.519Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "178c6919-8066-4b01-a98f-bee6adb71232" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:38.679Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "e7fd14ac-e853-4ba0-a019-d75c619b512b" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:38.729Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "83477217-5551-4cc2-8245-8b2ce2ee85a7" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:38.872Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 2 2 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "85a8adf5-76f0-4aa0-af5d-e2b950b3cf33" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:38.928Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "0f068dbc-95ae-4c42-ab28-27b2b32490c4" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:39.063Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 1 1 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "b7c6b99b-6615-4ec7-bda6-675f0b8afb59" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:39.117Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "4baf6186-39e0-4b8c-92bb-33f27f2b3039" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:39.730Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "c77b3593-44c1-4d8b-b790-f9c7fdaa1842" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:39.790Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "226018e9-d175-4152-a76a-467917a5e74a" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:39.882Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "3f421004-e75f-4075-a577-a49cdfa4a99b" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:39.939Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "ff9afea4-b82b-4928-911f-f56686ec2846" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:40.441Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "b7926a0e-ab75-4e83-9343-ad59a918a0e5" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:40.497Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "bad8db62-f7c9-4ad3-878c-2fe9ebf883e5" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:40.611Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "62101620-bad2-472d-a0ad-d73d1309bbdf" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:40.671Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "98f233c1-25f9-4e82-b3e2-ac6a926a3859" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:40.799Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "05698261-2ed0-49db-92e9-1a5536bab11a" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:40.854Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "53935e5f-9bfc-4c8c-9dc4-2e66765b53d7" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:40.999Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "38085986-f5ab-4608-929a-47f061793adb" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:41.049Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "a971832a-fe5b-4605-8daf-073879ea9ddb" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:41.197Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "3c222195-763e-4385-b360-47b29d5be018" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:41.275Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "2d67f9a0-72d9-49af-8eb9-fefa3b44e475" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:41.388Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "08372bb3-d195-4d2b-b75a-38d966dfb50e" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:41.445Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "3a975f34-b456-4804-853a-a6869892d548" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:41.595Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "7c4ae6c7-4195-4ef6-8c04-e0d0910b99d4" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:41.646Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "44b74983-7bfc-485e-aec5-63116bfe4550" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:41.816Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 1 1 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "5c2527fc-7148-4d3a-9082-4a9a92e2c1c5" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:41.873Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 2 2 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "5bb33402-cc0c-462e-a951-1ff5113aeec7" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:42.005Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 2 2 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "a67e7cf1-6122-488f-a14e-e948fcb3541f" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:42.090Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 0 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "1488104e-61d7-4579-a252-1903ece3db47" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:42.227Z] "GET / HTTP/2" 200 - via_upstream - "-" 0 2608 2 2 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "603e8ef2-821b-4a2f-b2b4-d69cdb1328ca" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default
[2022-03-11T18:13:42.284Z] "GET /favicon.ico HTTP/2" 200 - via_upstream - "-" 0 2527 1 0 "10.55.0.42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36" "5aaa3e2b-e397-4ad1-8ce0-983c454826da" "asdf.localhost:9443" "10.42.1.10:80" outbound|8080||sourceip-test.default.svc.cluster.local 10.42.1.5:47016 10.42.1.5:8443 10.55.0.42:40108 asdf.localhost default

EDIT: I think I misunderstood the ingress gateway. It is impossible to preserve the source IP (without HTTP header injection) through the ingress gateway, or really any load balancer