"unsupported transfer encoding" error in istio proxy

1

Hello,

We are running into a problem with Istio 1.5.0.
We are sending a POST request to a container running in a pod. Let’s say this is app1.
This triggers a POST request from app1 to app2. App2 is running in another container in a different pod. App2 sends the POST response back to app1, and app1 sends a POST response back to the user.
Both pods have the istio-proxy sidecars running for these applications.

The istio-proxy of app1 is generating an unsupported transfer encoding error when receiving the POST response from app1:
[Envoy (Epoch 0)] [2020-04-06 19:27:00.513][24][debug][client] [external/envoy/source/common/http/codec_client.cc:127] [C64] protocol error: http/1.1 protocol error: unsupported transfer encoding

These are the response headers:
HTTP/1.1 201
Set-Cookie: XSRF-TOKEN=; Max-Age=0; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/
Set-Cookie: XSRF-TOKEN=f24efe82-c999-473a-8d50-081553c3d225; Path=/
content-disposition: inline;filename=f.txt
date: Mon, 06 Apr 2020 19:53:54 GMT
x-envoy-upstream-service-time: 12
server: envoy
transfer-encoding: chunked
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
Strict-Transport-Security: max-age=31536000 ; includeSubDomains
X-Frame-Options: DENY
Content-Type: application/json
Transfer-Encoding: chunked

{“id”:“5e8b88d3e6ba3d0001bc7205”,“title”:“test1”,“entrypoint”:“kvk-rpt-p-k-p-20191211”,“kvknummer”:“30217488”,“currentStartDate”:“2019-01-01”,“currentEndDate”:“2019-12-31”,“previousStartDate”:“2018-01-01”,“previousEndDate”:“2018-12-31”,“created”:“2020-04-06T19:53:55Z”,“updated”:“2020-04-06T19:53:55Z”,“duplicable”:false}

This relates to Envoy version 1.13:
http: blocks unsupported transfer-encodings. Can be reverted temporarily by setting runtime feature envoy.reloadable_features.reject_unsupported_transfer_encodings to false

That workaround works for me, but I’d rather have a definitive solution.
Does envoy not support Transfer-Encoding: chunked at all?
Or does this happen because this header is present twice?

Thanks in advance for your help.
Jasper Misset

1 Like
2

HI @jmisset
Would it be possible to share the complete workaround?

Many thanks

3

Hi mvicedo,

The workaround is running the following command:

kubectl exec podname -c istio-proxy – curl -X POST http://localhost:15000/runtime_modify?envoy.reloadable_features.reject_unsupported_transfer_encodings=true

PS - We have changed our container images, instead of running on Apache Tomcat, they now use Undertow. There the problem does not exist (the header only exists once) so this seems like an Apache Tomcat issue.

1 Like
4

Just for the next person that comes along, the workaround is to set the feature flag to false not true

brockmills@MacBook-Pro Tmp % kubectl exec mypod-f0eabb92-66d98c79fb-56gmk -c istio-proxy -- curl -XPOST -s -o /dev/null http://localhost:15000/runtime_modify\?envoy.reloadable_features.reject_unsupported_transfer_encodings=false
1 Like
5

Oops, my bad. Thanks for the correction!

We switched from tomcat to undertow for our application containers to get rid of this problem completely.

6

It’s not seem to be an error of Tomcat (or Jetty). The presence of header Transfer-Encoding and transfer-encoding (no matter the value of both) is killing to Istio proxy. Big question: ¿Why Istio can’t process this scenario? If the HTTP RFC said that any header present is case-insensitive ¿Why Istio handle that as case-sensitive?

Please checkout this answers: Are HTTP headers case-sensitive? - Stack Overflow, just for example:

They are not case sensitive. In fact NodeJS web server explicitly converts them to lower-case, before making them available in the request object.

7

It is not the casing of the header that is causing issues, but the fact that the headers are duplicate.

In the scenario we are using:

We are sending a POST request to a container running in a pod. Let’s say this is app1.
This triggers a POST request from app1 to app2. App2 is running in another container in a different pod. App2 sends the POST response back to app1, and app1 sends a POST response back to the user.
Both pods have the istio-proxy sidecars running for these applications.

We noticed that when using Tomcat as a webcontainer for our docker images, the response header Transfer-Encoding: chunked is added by both app2 when sending the response back to app1, and by app1 when sending the response back to the user. But the header from app2 is not removed in this last step and is still present. I would expect that the tomcat in app1 recognizes that it should not sent this header (originating from app2) back to the user. It should drop this header as it was only valid for the HTTP response from app2 to app1.

When using Undertow (at least for app1) we no longer see the duplicate header.

I hope this makes sense, I find it a bit hard to explain :slight_smile:

Regards,
Jasper

8

Again, tell me if I’m wrong: the HTTP protocol permit multiple headers, including the Transfer-Encoding header. The fact is Tomcat is sending twice this header (and others) but some servers, as Node for example, has same behaviour.

I can reproduce the same error puting an Apache server or Nginx server sending, for every request, this headers responses: Transfer-Encoding: chunked, TransFer-Encoding: chunked, TRANSFER-ENCODING: chunked and Isio will crash, but that headers are respecting the HTTP Protocol.

I understand the scenario with Tomcat and the workaround changing to Undertow. But, my question is not trivial becouse this workaround is very expensive for my team (More than 1500 artifacts deployed with Tomcat)

The question here is if Istio is respecting the HTTP protocol or not. If is this case, ¿at some version of Istio this will be fixed?

9

The header ‘Transfer-Encoding’ may appear more than once, however, the value ‘chunked’ cannot be present more than once. Different values (e.g. 1x gzip, 1x chunked) is allowed.

From the RFC:

A sender MUST NOT apply chunked more than once to a message body (i.e., chunking an
already chunked message is not allowed).

So I believe that Envoy Proxy (and istio by extension) is respecting the HTTP protocol. However I feel a workaround (like ignoring more than 1 chunked headers or something similar) would be nice.

10

Excellent. You are right, the protocol is respected.
Thanks Jasper, I will talk about it with the team

Best regards

11

You’re welcome.
Let me know if you find any solution!

1 Like
12

At all cases the solution is the same: changing the Web Server to Undertow, Jetty has the Tomcat behavior.

Our approach is avoid return the app B response directly, I mean
public ResponseEntity callToBService() {
return appBClient.call();
}

to:
public ResponseEntity callToBService() {
ResponseEntity result = appBClient.call();

    return ResponseEntity.ok().body(result.getBody()); // The headers are present once
}