HTTP2 requests upstream forwarding modified to HTTP1.1 by istio-proxy sidecar

We have development cluster deployed with istio 1.1.11, and all the outbound traffic from applications are rerouted via istio-proxy sidecars. We observed that the HTTP2 requests with prior knowledge on port 8080 are being forwarded as HTTP1.1 requests instead of HTTP2.

Tested running the same traffic on some random ports (e.g port 15021) and on app containers with no proxy side cars, it seem to run fine(HTTP2 requests are not modified to HTTP1.1).

Note: There are no virtual services or policies configured for this traffic.

outboundTrafficPolicy: ALLOW_ANY (istio/templates/configmap.yaml)
traffic.sidecar.istio.io/includeOutboundIPRanges: "*" (istio/templates/sidecar-injector-configmap.yaml)

Expected behavior
The HTTP2 requests with prior knowledge should be forwarded as HTTP/2 requests.

curl response logs:

Port 8080 -

[root@smfcc-0-209-0-dbg-696dcbd484-76ch4 /]# curl -H “Content-Type: application/json” -X PUT “http:/10.71.33.249:8080/nudm-uecm/v1/imsi-456123000000586/registrations/smf-registrations/15” -d ‘{“dnn”:“dnn1.att”,“pduSessionId”:15,“plmnId”:{“mcc”:“456”,“mnc”:“123”},“singleNssai”:{“sst”:1},“smfInstanceId”:“46bb3328-41da-4662-8523-e1b6b84ee19a”,“supportedFeatures”:“1”}’ --http2-prior-knowledge -vvv

  • Unwillingly accepted illegal URL using 1 slash!
  • Trying 10.71.33.249…
  • TCP_NODELAY set
  • Connected to 10.71.33.249 (10.71.33.249) port 8080 (#0)
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x558c27e0e440)

PUT /nudm-uecm/v1/imsi-456123000000586/registrations/smf-registrations/15 HTTP/2
Host: 10.71.33.249:8080
User-Agent: curl/7.59.0
Accept: /
Content-Type: application/json
Content-Length: 174

  • We are completely uploaded and fine
  • Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
    < HTTP/2 503
    < content-length: 95
    < content-type: text/plain
    < date: Mon, 05 Aug 2019 14:50:32 GMT
    < server: envoy
    <
  • Connection #0 to host 10.71.33.249 left intact
    upstream connect error or disconnect/reset before headers. reset reason: connection termination[root@smfcc-0-209-0-dbg-696dcbd484-76ch4 /]#
    [root@smfcc-0-209-0-dbg-696dcbd484-76ch4 /]#

Port 15021 -

[root@smfcc-0-209-0-dbg-696dcbd484-76ch4 /]# curl -H “Content-Type: application/json” -X PUT “http:/10.71.33.249:15021/nudm-uecm/v1/imsi-456123000000586/registrations/smf-registrations/15” -d ‘{“dnn”:“dnn1.att”,“pduSessionId”:15,“plmnId”:{“mcc”:“456”,“mnc”:“123”},“singleNssai”:{“sst”:1},“smfInstanceId”:“46bb3328-41da-4662-8523-e1b6b84ee19a”,“supportedFeatures”:“1”}’ --http2-prior-knowledge -vvv

  • Unwillingly accepted illegal URL using 1 slash!
  • Trying 10.71.33.249…
  • TCP_NODELAY set
  • Connected to 10.71.33.249 (10.71.33.249) port 15021 (#0)
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x55d03fe6c440)

PUT /nudm-uecm/v1/imsi-456123000000586/registrations/smf-registrations/15 HTTP/2
Host: 10.71.33.249:15021
User-Agent: curl/7.59.0
Accept: /
Content-Type: application/json
Content-Length: 174

  • We are completely uploaded and fine
  • Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
    < HTTP/2 200
    < content-type: application/json
    < content-length: 174
    < server: asyncio-h2
    <
  • Connection #0 to host 10.71.33.249 left intact
    {“dnn”:“dnn1.att”,“pduSessionId”:15,“plmnId”:{“mcc”:“456”,“mnc”:“123”},“singleNssai”:{“sst”:1},“smfInstanceId”:“46bb3328-41da-4662-8523-e1b6b84ee19a”,“supportedFeatures”:“1”}[root@smfcc-0-209-0-dbg-696dcbd484-76ch4 /]#
    [root@smfcc-0-209-0-dbg-696dcbd484-76ch4 /]#

Added pcap to the HTTP2 request on port 8080 with prior knowledge is being forwarded as HTTP1.1 by istio-proxy. · Issue #16059 · istio/istio · GitHub.

/cc @vadimeisenbergibm @spikecurtis
We would really appreciate it if you could either share some insight regarding this or cc the right people who would be able to answer this question.

@howardjohn asked a question regarding the issue https://github.com/istio/istio/issues/16059#issuecomment-518725876

The solution is : change the HTTPS to HTTP2 in ServiceEntry, and then use DestinationRule with TLS configuration, with this configuration, the external protocol will be HTTP2+TLS. We met the similar issue before and fixed it by using this solution.
But please note, this solution will meet another istio bug (https://github.com/istio/istio/issues/16458), so other communication (appl +sidecar, but appl use https directly (instead of using sidecar for TLS origination) + same port (declared in ServiceEntry) will fail.