Nginx Proxy Pass to Istio Ingress Gateway 404

Hello,

I have a nginx proxy which does a proxy_pass to a istio service.

server {                                                                                                                                                                                                                                      
  listen 3900;                                                                                                                                                                                                                                
  client_max_body_size 0;                                                                                                                                                                                                                     
  index index.html;                                                                                                                                                                                                                           
  server_name localhost;                                                                                                                                                                                                                      
  root /var/www/public;                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
  location /api-url {                                                                                                                                                                                                                         
    rewrite ^/api-url/(.*)$ /$1  break;                                                                                                                                                                                                       
    proxy_pass https://myapp.com;                                                                                                                                                                                          
    proxy_http_version 1.1;                                                                                                                                                                                                                   
    proxy_set_header Upgrade $http_upgrade;                                                                                                                                                                                                   
    proxy_set_header Connection 'upgrade';                                                                                                                                                                                                    
    proxy_set_header Host $proxy_host;                                                                                                                                                                                                        
    proxy_cache_bypass $http_upgrade;                                                                                                                                                                                                         
  }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
} 

In Istio I have a virtual service for myapp.com.

I get a 404 through the proxy_pass but a 200 via curl.

curl to nginx “http://localhost:3900/api-url/api/v1/completions?prefix=&field=sessionTags
Returns a 404
Istio ingressgateway access log for this call:
[2019-11-07T21:22:13.404Z] "GET /api/v1/completions?prefix=&field=sessionTags HTTP/1.1" 404 - "-" "-" 0 0 0 - "10.42.192.0" "curl/7.66.0" "7ec136be-2e62-414f-8cd1-2c2ddfb7b4fd" "myapp.com" "-" - - 10.42.248.13:443 10.42.192.0:57128 -

curl to mimic nginx rewrite: "curl “https://myapp.com/api/v1/completions?prefix=&field=sessionTags
Returns a 200
Istio ingressgateway access log for this call:
[2019-11-07T21:25:09.553Z] "GET /api/v1/completions?prefix=&field=sessionTags HTTP/1.1" 200 - "-" "-" 0 2 363 362 "10.42.48.0" "curl/7.66.0" "069d6d0a-940a-4bf2-869c-83bdc5a1ce44" "myapp.com" "10.42.140.3:8080" outbound|8080||myapp.mynamespace.svc.cluster.local - 10.42.248.13:443 10.42.48.0:13004 myapp.com

I can’t figure out what the issue is with the nginx proxy_pass module.

2 Likes

I tried to accomplish same task in apache and it worked as expected.

<VirtualHost *:3900>
        ServerName localhost

        SSLProxyEngine on
        <Location /api-url>
          RewriteEngine on
          RewriteRule "^/api-url/(.*)$ /$1"
          ProxyPass "https://myapp.com"
        </Location>

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

</VirtualHost>

I would prefer to use nginx though. If I can’t figure it out soon I will have to do a packet capture to compare requests.

Here are some istio proxy trace logs to compare nginx vs apache:
Nginx:

[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1877826] completed header: key=Host value=myapp.com
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1877826] completed header: key=Connection value=Keep-Alive
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1877826] completed header: key=X-Forwarded-For value=127.0.0.1
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1877826] completed header: key=X-Forwarded-Host value=localhost:3800
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1877826] completed header: key=X-Forwarded-Server value=localhost
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1877826] completed header: key=User-Agent value=curl/7.58.0
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:445] [C1877826] headers complete
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1877826] completed header: key=Accept value=*/*
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:466] [C1877826] message complete
[2019-11-11 18:24:27.295][31][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:583] [C1877826][S7885769596347870016] request headers complete (end_stream=true):
':authority', 'myapp.com'
':path', '/'
':method', 'GET'
'connection', 'Keep-Alive'
'x-forwarded-for', '127.0.0.1'
'x-forwarded-host', 'localhost:3800'
'x-forwarded-server', 'localhost'
'user-agent', 'curl/7.58.0'
'accept', '*/*'

[2019-11-11 18:24:27.295][31][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1062] [C1877826][S7885769596347870016] request end stream
[2019-11-11 18:24:27.295][31][debug][filter] [src/envoy/http/mixer/filter.cc:47] Called Mixer::Filter : Filter
[2019-11-11 18:24:27.295][31][debug][filter] [src/envoy/http/mixer/filter.cc:154] Called Mixer::Filter : setDecoderFilterCallbacks
[2019-11-11 18:24:27.295][31][debug][filter] [src/envoy/http/mixer/filter.cc:70] Called Mixer::Filter : decodeHeaders
[2019-11-11 18:24:27.295][31][debug][filter] [./src/envoy/utils/header_update.h:46] Mixer forward attributes set: Ck8KCnNvdXJjZS51aWQSQRI/a3ViZXJuZXRlczovL2lzdGlvLWluZ3Jlc3NnYXRld2F5LTViNzdkZDU5ODktc2x6bWouaXN0aW8tc3lzdGVt
[2019-11-11 18:24:27.295][31][debug][filter] [src/envoy/http/mixer/filter.cc:162] Called Mixer::Filter : check complete OK
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:833] [C1877826][S7885769596347870016] decode headers called: filter=0x6539770 status=0
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:833] [C1877826][S7885769596347870016] decode headers called: filter=0x64142d0 status=0
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:833] [C1877826][S7885769596347870016] decode headers called: filter=0x533a910 status=0
[2019-11-11 18:24:27.295][31][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1137] [C1877826][S7885769596347870016] Sending local reply with details direct_response
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1224] [C1877826][S7885769596347870016] encode headers called: filter=0x6d061e0 status=0
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1224] [C1877826][S7885769596347870016] encode headers called: filter=0x4aff950 status=0
[2019-11-11 18:24:27.295][31][debug][filter] [src/envoy/http/mixer/filter.cc:141] Called Mixer::Filter : encodeHeaders 2
[2019-11-11 18:24:27.295][31][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1224] [C1877826][S7885769596347870016] encode headers called: filter=0x6d06140 status=0
[2019-11-11 18:24:27.295][31][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1329] [C1877826][S7885769596347870016] encoding headers via codec (end_stream=true):
':status', '404'
'location', 'https://myapp.com/'
'date', 'Mon, 11 Nov 2019 18:24:27 GMT'
'server', 'istio-envoy'

Apache:

[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1422263] completed header: key=Host value=myapp.com
[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1422263] completed header: key=User-Agent value=curl/7.58.0
[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1422263] completed header: key=Accept value=*/*
[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1422263] completed header: key=X-Forwarded-For value=127.0.0.1
[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1422263] completed header: key=X-Forwarded-Host value=localhost:3900
[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1422263] completed header: key=X-Forwarded-Server value=localhost
[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:445] [C1422263] headers complete
[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:334] [C1422263] completed header: key=Connection value=Keep-Alive
[2019-11-11 18:08:39.250][32][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:466] [C1422263] message complete
[2019-11-11 18:08:39.250][32][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:583] [C1422263][S16574152508580966862] request headers complete (end_stream=true):
':authority', 'myapp.com'
':path', '/'
':method', 'GET'
'user-agent', 'curl/7.58.0'
'accept', '*/*'
'x-forwarded-for', '127.0.0.1'
'x-forwarded-host', 'localhost:3900'
'x-forwarded-server', 'localhost'
'connection', 'Keep-Alive'

[2019-11-11 18:08:39.250][32][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1062] [C1422263][S16574152508580966862] request end stream
[2019-11-11 18:08:39.250][32][debug][filter] [src/envoy/http/mixer/filter.cc:47] Called Mixer::Filter : Filter
[2019-11-11 18:08:39.250][32][debug][filter] [src/envoy/http/mixer/filter.cc:154] Called Mixer::Filter : setDecoderFilterCallbacks
[2019-11-11 18:08:39.250][32][debug][filter] [src/envoy/http/mixer/filter.cc:70] Called Mixer::Filter : decodeHeaders
[2019-11-11 18:08:39.251][32][debug][filter] [./src/envoy/utils/header_update.h:46] Mixer forward attributes set: CisKGGRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZRIPEg1iZG1zLWVkZ2UtYXBpCiwKHWRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZXNwYWNlEgsSCWJkbXMtZWRnZQpP
Cgpzb3VyY2UudWlkEkESP2t1YmVybmV0ZXM6Ly9pc3Rpby1pbmdyZXNzZ2F0ZXdheS01Yjc3ZGQ1OTg5LWtoazlrLmlzdGlvLXN5c3RlbQpHChhkZXN0aW5hdGlvbi5zZXJ2aWNlLmhvc3QSKxIpYmRtcy1lZGdlLWFwaS5iZG1zLWVkZ2Uuc3ZjLmNsdXN0ZXIubG9jYWwKRQoXZGVzdGluYXRpb24uc2VydmljZS51aW
QSKhIoaXN0aW86Ly9iZG1zLWVkZ2Uvc2VydmljZXMvYmRtcy1lZGdlLWFwaQ==
[2019-11-11 18:08:39.251][32][debug][filter] [src/envoy/http/mixer/filter.cc:162] Called Mixer::Filter : check complete OK
[2019-11-11 18:08:39.251][32][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:833] [C1422263][S16574152508580966862] decode headers called: filter=0x5465d60 status=0
[2019-11-11 18:08:39.251][32][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:833] [C1422263][S16574152508580966862] decode headers called: filter=0x6dea5a0 status=0
[2019-11-11 18:08:39.251][32][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:833] [C1422263][S16574152508580966862] decode headers called: filter=0x65bd950 status=0
[2019-11-11 18:08:39.251][32][debug][router] [external/envoy/source/common/router/router.cc:332] [C1422263][S16574152508580966862] cluster 'outbound|8080||bdms-edge-api.bdms-edge.svc.cluster.local' match for URL '/'
[2019-11-11 18:08:39.251][32][debug][router] [external/envoy/source/common/router/router.cc:393] [C1422263][S16574152508580966862] router decoding headers:
':authority', 'myapp.com'
':path', '/'
':method', 'GET'
':scheme', 'http'
'user-agent', 'curl/7.58.0'
'accept', '*/*'
'x-forwarded-for', '127.0.0.1,10.42.64.0'
'x-forwarded-host', 'localhost:3900'
'x-forwarded-server', 'localhost'
'x-forwarded-proto', 'https'
'x-envoy-external-address', '10.42.64.0'
'x-request-id', '3f45d1e9-a14e-4850-9717-6da90842e039'
'x-envoy-decorator-operation', 'bdms-edge-api.bdms-edge.svc.cluster.local:8080/*'
'x-istio-attributes', 'CisKGGRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZRIPEg1iZG1zLWVkZ2UtYXBpCiwKHWRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZXNwYWNlEgsSCWJkbXMtZWRnZQpPCgpzb3VyY2UudWlkEkESP2t1YmVybmV0ZXM6Ly9pc3Rpby1pbmdyZXNzZ2F0ZXdheS01Yjc3ZGQ1OTg5LWtoazlrLml
zdGlvLXN5c3RlbQpHChhkZXN0aW5hdGlvbi5zZXJ2aWNlLmhvc3QSKxIpYmRtcy1lZGdlLWFwaS5iZG1zLWVkZ2Uuc3ZjLmNsdXN0ZXIubG9jYWwKRQoXZGVzdGluYXRpb24uc2VydmljZS51aWQSKhIoaXN0aW86Ly9iZG1zLWVkZ2Uvc2VydmljZXMvYmRtcy1lZGdlLWFwaQ=='
'x-b3-traceid', '793d1e20ca1c6a1814516068b41add2e'
'x-b3-spanid', '14516068b41add2e'
'x-b3-sampled', '0'

It looks like something is going on in /src/envoy/utils/header_update.h.

I made sure the headers were exactly the same despite the x-forwarded headers.

I resolved it with this minimal nginx:

  location /api-url {                                                                                                                                                                                                                         
    rewrite ^/api-url/(.*)$ /$1  break;                                                                                                                                                                                                       
    proxy_pass https://myapp.com;                                                                                                                                                                                          
    proxy_http_version 1.1;               
    proxy_ssl_server_name on;                                                                                                                                                                                                                                                                                                                                                                                                           
  } 

They key was enabling proxy_ssl_server_name. Not sure why this isn’t enabled by default.