Strange Issue with downloading file, using 'TCP' it works, using 'HTTP' breaks

Hi,
We are using Istio 1.4.1.
We create a service
“spec”: {
“ports”: [
{
“name”: “http”,
“protocol”: “TCP”,
“port”: 9609,
“targetPort”: 9000
}
],

We curl with
curl -v GET "http://fileservice.msdemo.svc.cluster.local:9609/file/o/81c54c13-f409-4060-825b-da10a7821f5d

it breaks with a 503 UC
[2020-03-05T19:15:51.896Z] “GET /file/o/81c54c13-f409-4060-825b-da10a7821f5d HTTP/1.1” 503 UC “-” “-” 0 95 11 - “-” “curl/7.52.1” “8ed89a2f-e1b1-948e-8542-eccde2302f20” “fileservice.msdemo.svc.cluster.local:9609” “127.0.0.1:9000” inbound|9609|http|fileservice.msdemo.svc.cluster.local - 10.244.2.15:9000 10.244.2.15:43992 - default

If we change the service spec to
“ports”: [
{
“name”: “tcp”,
“protocol”: “TCP”,
“port”: 9609,
“targetPort”: 9000
}
],

then it works and the same CURL now gets
< HTTP/1.1 200 OK
< transfer-encoding: chunked
< Content-Length: 3505
< Content-Disposition: attachment; filename=“download.jpg”; filename*=“UTF-8’'download.jpg”
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: Csrf-Token, X-Auth-Token, X-Requested-With, Accept, Content-Type, Authorization, Content-Length, Cache-Control, Pragma, If-Modified-Since
< Access-Control-Allow-Methods: HEAD,GET,POST,PUT,PATCH,DELETE
< Access-Control-Allow-Credentials: true
< Content-Type: image/jpeg
< Date: Thu, 05 Mar 2020 19:27:20 GMT

The best guess right now is that ‘http’ will trigger Istio to validate more things, such as the headers?
Does anyone know what’s wrong here? Thanks

Answering my own question here. I was able to set debug mode to Envoy proxy by doing a POST to the Envoy admin interface. Then I found the issue of
[Envoy (Epoch 0)] [2020-03-07 00:19:15.374][22][debug][client] [external/envoy/source/common/http/codec_client.cc:124] [C5668] protocol error: http/1.1 protocol error: HPE_UNEXPECTED_CONTENT_LENGTH
[Envoy (Epoch 0)] [2020-03-07 00:19:15.374][22][debug][connection] [external/envoy/source/common/network/connection_impl.cc:104] [C5668] closing data_to_write=0 type=1
[Envoy (Epoch 0)] [2020-03-07 00:19:15.374][22][debug][connection] [external/envoy/source/common/network/connection_impl.cc:193] [C5668] closing socket: 1
[Envoy (Epoch 0)] [2020-03-07 00:19:15.374][22][debug][client] [external/envoy/source/common/http/codec_client.cc:88] [C5668] disconnect. resetting 1 pending requests
[Envoy (Epoch 0)] [2020-03-07 00:19:15.374][22][debug][client] [external/envoy/source/common/http/codec_client.cc:111] [C5668] request reset
[Envoy (Epoch 0)] [2020-03-07 00:19:15.374][22][debug][router] [external/envoy/source/common/router/router.cc:911] [C5665][S11149092392049592664] upstream reset: reset reason connection termination

This is an issue with the ‘chunked’ encoding and ‘content-length’. I removed the ‘content-length’ header, then it is working.