Nginx Proxy Pass to Istio Ingress Gateway 404

Hello,

I have a nginx proxy which does a proxy_pass to a istio service.

server {                                                                                                                                                                                                                                      
  listen 3900;                                                                                                                                                                                                                                
  client_max_body_size 0;                                                                                                                                                                                                                     
  index index.html;                                                                                                                                                                                                                           
  server_name localhost;                                                                                                                                                                                                                      
  root /var/www/public;                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
  location /api-url {                                                                                                                                                                                                                         
    rewrite ^/api-url/(.*)$ /$1  break;                                                                                                                                                                                                       
    proxy_pass https://myapp.com;                                                                                                                                                                                          
    proxy_http_version 1.1;                                                                                                                                                                                                                   
    proxy_set_header Upgrade $http_upgrade;                                                                                                                                                                                                   
    proxy_set_header Connection 'upgrade';                                                                                                                                                                                                    
    proxy_set_header Host $proxy_host;                                                                                                                                                                                                        
    proxy_cache_bypass $http_upgrade;                                                                                                                                                                                                         
  }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
} 

In Istio I have a virtual service for myapp.com.

I get a 404 through the proxy_pass but a 200 via curl.

curl to nginx “http://localhost:3900/api-url/api/v1/completions?prefix=&field=sessionTags”
Returns a 404
Istio ingressgateway access log for this call:
[2019-11-07T21:22:13.404Z] "GET /api/v1/completions?prefix=&field=sessionTags HTTP/1.1" 404 - "-" "-" 0 0 0 - "10.42.192.0" "curl/7.66.0" "7ec136be-2e62-414f-8cd1-2c2ddfb7b4fd" "myapp.com" "-" - - 10.42.248.13:443 10.42.192.0:57128 -

curl to mimic nginx rewrite: "curl “https://myapp.com/api/v1/completions?prefix=&field=sessionTags”
Returns a 200
Istio ingressgateway access log for this call:
[2019-11-07T21:25:09.553Z] "GET /api/v1/completions?prefix=&field=sessionTags HTTP/1.1" 200 - "-" "-" 0 2 363 362 "10.42.48.0" "curl/7.66.0" "069d6d0a-940a-4bf2-869c-83bdc5a1ce44" "myapp.com" "10.42.140.3:8080" outbound|8080||myapp.mynamespace.svc.cluster.local - 10.42.248.13:443 10.42.48.0:13004 myapp.com

I can’t figure out what the issue is with the nginx proxy_pass module.

2 Likes

I tried to accomplish same task in apache and it worked as expected.

<VirtualHost *:3900>
        ServerName localhost

        SSLProxyEngine on
        <Location /api-url>
          RewriteEngine on
          RewriteRule "^/api-url/(.*)$ /$1"
          ProxyPass "https://myapp.com"
        </Location>

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

</VirtualHost>

I would prefer to use nginx though. If I can’t figure it out soon I will have to do a packet capture to compare requests.

Here are some istio proxy trace logs to compare nginx vs apache:
Nginx:

[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1877826] completed header: key=Host value=myapp.com
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1877826] completed header: key=Connection value=Keep-Alive
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1877826] completed header: key=X-Forwarded-For value=127.0.0.1
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1877826] completed header: key=X-Forwarded-Host value=localhost:3800
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1877826] completed header: key=X-Forwarded-Server value=localhost
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1877826] completed header: key=User-Agent value=curl/7.58.0
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:445] [C1877826] headers complete
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1877826] completed header: key=Accept value=*/*
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:466] [C1877826] message complete
[2019-11-11 18:24:27.295][31][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:583] [C1877826][S7885769596347870016] request headers complete (end_stream=true):
':authority', 'myapp.com'
':path', '/'
':method', 'GET'
'connection', 'Keep-Alive'
'x-forwarded-for', '127.0.0.1'
'x-forwarded-host', 'localhost:3800'
'x-forwarded-server', 'localhost'
'user-agent', 'curl/7.58.0'
'accept', '*/*'

[2019-11-11 18:24:27.295][31][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1062] [C1877826][S7885769596347870016] request end stream
[2019-11-11 18:24:27.295][31][debug][filter] [src/envoy/http/mixer/filter.cc:47] Called Mixer::Filter : Filter
[2019-11-11 18:24:27.295][31][debug][filter] [src/envoy/http/mixer/filter.cc:154] Called Mixer::Filter : setDecoderFilterCallbacks
[2019-11-11 18:24:27.295][31][debug][filter] [src/envoy/http/mixer/filter.cc:70] Called Mixer::Filter : decodeHeaders
[2019-11-11 18:24:27.295][31][debug][filter] [./src/envoy/utils/header_update.h:46] Mixer forward attributes set: Ck8KCnNvdXJjZS51aWQSQRI/a3ViZXJuZXRlczovL2lzdGlvLWluZ3Jlc3NnYXRld2F5LTViNzdkZDU5ODktc2x6bWouaXN0aW8tc3lzdGVt
[2019-11-11 18:24:27.295][31][debug][filter] [src/envoy/http/mixer/filter.cc:162] Called Mixer::Filter : check complete OK
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:833] [C1877826][S7885769596347870016] decode headers called: filter=0x6539770 status=0
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:833] [C1877826][S7885769596347870016] decode headers called: filter=0x64142d0 status=0
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:833] [C1877826][S7885769596347870016] decode headers called: filter=0x533a910 status=0
[2019-11-11 18:24:27.295][31][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1137] [C1877826][S7885769596347870016] Sending local reply with details direct_response
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1224] [C1877826][S7885769596347870016] encode headers called: filter=0x6d061e0 status=0
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1224] [C1877826][S7885769596347870016] encode headers called: filter=0x4aff950 status=0
[2019-11-11 18:24:27.295][31][debug][filter] [src/envoy/http/mixer/filter.cc:141] Called Mixer::Filter : encodeHeaders 2
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1224] [C1877826][S7885769596347870016] encode headers called: filter=0x6d06140 status=0
[2019-11-11 18:24:27.295][31][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1329] [C1877826][S7885769596347870016] encoding headers via codec (end_stream=true):
':status', '404'
'location', 'https://myapp.com/'
'date', 'Mon, 11 Nov 2019 18:24:27 GMT'
'server', 'istio-envoy'

Apache:

[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1422263] completed header: key=Host value=myapp.com
[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1422263] completed header: key=User-Agent value=curl/7.58.0
[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1422263] completed header: key=Accept value=*/*
[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1422263] completed header: key=X-Forwarded-For value=127.0.0.1
[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1422263] completed header: key=X-Forwarded-Host value=localhost:3900
[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1422263] completed header: key=X-Forwarded-Server value=localhost
[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:445] [C1422263] headers complete
[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1422263] completed header: key=Connection value=Keep-Alive
[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:466] [C1422263] message complete
[2019-11-11 18:08:39.250][32][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:583] [C1422263][S16574152508580966862] request headers complete (end_stream=true):
':authority', 'myapp.com'
':path', '/'
':method', 'GET'
'user-agent', 'curl/7.58.0'
'accept', '*/*'
'x-forwarded-for', '127.0.0.1'
'x-forwarded-host', 'localhost:3900'
'x-forwarded-server', 'localhost'
'connection', 'Keep-Alive'

[2019-11-11 18:08:39.250][32][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1062] [C1422263][S16574152508580966862] request end stream
[2019-11-11 18:08:39.250][32][debug][filter] [src/envoy/http/mixer/filter.cc:47] Called Mixer::Filter : Filter
[2019-11-11 18:08:39.250][32][debug][filter] [src/envoy/http/mixer/filter.cc:154] Called Mixer::Filter : setDecoderFilterCallbacks
[2019-11-11 18:08:39.250][32][debug][filter] [src/envoy/http/mixer/filter.cc:70] Called Mixer::Filter : decodeHeaders
[2019-11-11 18:08:39.251][32][debug][filter] [./src/envoy/utils/header_update.h:46] Mixer forward attributes set: CisKGGRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZRIPEg1iZG1zLWVkZ2UtYXBpCiwKHWRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZXNwYWNlEgsSCWJkbXMtZWRnZQpP
Cgpzb3VyY2UudWlkEkESP2t1YmVybmV0ZXM6Ly9pc3Rpby1pbmdyZXNzZ2F0ZXdheS01Yjc3ZGQ1OTg5LWtoazlrLmlzdGlvLXN5c3RlbQpHChhkZXN0aW5hdGlvbi5zZXJ2aWNlLmhvc3QSKxIpYmRtcy1lZGdlLWFwaS5iZG1zLWVkZ2Uuc3ZjLmNsdXN0ZXIubG9jYWwKRQoXZGVzdGluYXRpb24uc2VydmljZS51aW
QSKhIoaXN0aW86Ly9iZG1zLWVkZ2Uvc2VydmljZXMvYmRtcy1lZGdlLWFwaQ==
[2019-11-11 18:08:39.251][32][debug][filter] [src/envoy/http/mixer/filter.cc:162] Called Mixer::Filter : check complete OK
[2019-11-11 18:08:39.251][32][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:833] [C1422263][S16574152508580966862] decode headers called: filter=0x5465d60 status=0
[2019-11-11 18:08:39.251][32][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:833] [C1422263][S16574152508580966862] decode headers called: filter=0x6dea5a0 status=0
[2019-11-11 18:08:39.251][32][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:833] [C1422263][S16574152508580966862] decode headers called: filter=0x65bd950 status=0
[2019-11-11 18:08:39.251][32][debug][router] [external/envoy/source/common/router/router.cc:332] [C1422263][S16574152508580966862] cluster 'outbound|8080||bdms-edge-api.bdms-edge.svc.cluster.local' match for URL '/'
[2019-11-11 18:08:39.251][32][debug][router] [external/envoy/source/common/router/router.cc:393] [C1422263][S16574152508580966862] router decoding headers:
':authority', 'myapp.com'
':path', '/'
':method', 'GET'
':scheme', 'http'
'user-agent', 'curl/7.58.0'
'accept', '*/*'
'x-forwarded-for', '127.0.0.1,10.42.64.0'
'x-forwarded-host', 'localhost:3900'
'x-forwarded-server', 'localhost'
'x-forwarded-proto', 'https'
'x-envoy-external-address', '10.42.64.0'
'x-request-id', '3f45d1e9-a14e-4850-9717-6da90842e039'
'x-envoy-decorator-operation', 'bdms-edge-api.bdms-edge.svc.cluster.local:8080/*'
'x-istio-attributes', 'CisKGGRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZRIPEg1iZG1zLWVkZ2UtYXBpCiwKHWRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZXNwYWNlEgsSCWJkbXMtZWRnZQpPCgpzb3VyY2UudWlkEkESP2t1YmVybmV0ZXM6Ly9pc3Rpby1pbmdyZXNzZ2F0ZXdheS01Yjc3ZGQ1OTg5LWtoazlrLml
zdGlvLXN5c3RlbQpHChhkZXN0aW5hdGlvbi5zZXJ2aWNlLmhvc3QSKxIpYmRtcy1lZGdlLWFwaS5iZG1zLWVkZ2Uuc3ZjLmNsdXN0ZXIubG9jYWwKRQoXZGVzdGluYXRpb24uc2VydmljZS51aWQSKhIoaXN0aW86Ly9iZG1zLWVkZ2Uvc2VydmljZXMvYmRtcy1lZGdlLWFwaQ=='
'x-b3-traceid', '793d1e20ca1c6a1814516068b41add2e'
'x-b3-spanid', '14516068b41add2e'
'x-b3-sampled', '0'

It looks like something is going on in /src/envoy/utils/header_update.h.

I made sure the headers were exactly the same despite the x-forwarded headers.

I resolved it with this minimal nginx:

  location /api-url {                                                                                                                                                                                                                         
    rewrite ^/api-url/(.*)$ /$1  break;                                                                                                                                                                                                       
    proxy_pass https://myapp.com;                                                                                                                                                                                          
    proxy_http_version 1.1;               
    proxy_ssl_server_name on;                                                                                                                                                                                                                                                                                                                                                                                                           
  } 

They key was enabling proxy_ssl_server_name. Not sure why this isn’t enabled by default.

1 Like

I just ran into this exact issue, and adding proxy_ssl_server_name fixed my broken attempts at using nginx as a proxy between services in two kubernetes clusters.

I did stumble upon one clue that hints at this solution in the envoy access logs on the ingress gateways. A particular field named REQESTED_SERVER_NAME was always empty in the logs for my failed requests, the 404s. When I curled the same virtual service directly REQUESTED_SERVER_NAME would list the target virtual service host.

This GH discussion in the envoy code ties REQUESTED_SERVER_NAME back to SNI, and as soon as enabled proxy_ssl_server_name in nginx config I saw REQUESTED_SERVER_NAME list the correct host in the access logs of my now successful proxied requests.

1 Like

I also ran in this issue, but adding proxy_ssl_server_name didn’t work for me. In my scenario, my nginx is receiving http connections from an AWS LoadBalancer where the SSL is terminated.

My solution to this problem was to add the Host header containing the proxy_pass hostname to the location configuration.

proxy_set_header Host "http://myapp.com/";
proxy_pass http://myapp.com/;

Only then my requests were correctly answered by the proxy_pass endpoint.

Most likely, apache’s request add the Host header and nginx doesn’t.

1 Like

Couldn’t edit the above answer, so here’s the fix on the Host line:

proxy_set_header Host “myapp.com”;

1 Like

I also encountered this problem, try to delete the “proxy_set_header Host” setting and restore it to normal

My final workaround for this wasn’t great, but at least stopped with errors based on header modifications: I simply set the port name for the destination service as tcp-, forcing the protocol identification.

Unfortunatelly, I lost the instant request status visibility on kiali but at least I could add the gateway and the destination service to the mesh.