I have an Angular UI that communicates to a backend service called loan-spa.
It’s grpc-web, so the request goes through an Istio/Envoy proxy. That part works fine.
When the loan-spa service calls loan-svc via grpc, it gets a:
io.grpc.StatusException: UNAVAILABLE: Network closed for unknown reason
Looking at the sidecar log, it appears the root problem is:
invalid frame: Invalid HTTP header field was received on stream
The only thing that appears questionable in the outbound header (to me) is ‘TE: trailers’, though the spec says it’s OK. (https://tools.ietf.org/html/rfc7540#page-9)
When I run the applications outside of K8S with an envoy proxy all is well. But, I have been
unsuccessful getting this to work with K8S/Istio.
istioctl version
client version: 1.6.8
pilot version: 1.6.8
cluster-local-gateway version:
data plane version: 1.6.8 (14 proxies)
Client (loan-spa) calling loan-svc
20:14:47.258 [DefaultDispatcher-worker-1] INFO c.s.loan.LoanService$Companion - Calling loan-svc.processApplication
20:14:47.289 [grpc-nio-worker-ELG-5-2] DEBUG io.grpc.netty.NettyClientHandler - [id: 0x5fe41263, L:/10.42.2.92:48888 - R:/10.42.0.78:9092] OUTBOUND SETTINGS: ack=false settings={ENABLE_PUSH=0, MAX_CONCURRENT_STREAMS=0, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192}
20:14:47.290 [grpc-nio-worker-ELG-5-2] DEBUG io.grpc.netty.NettyClientHandler - [id: 0x5fe41263, L:/10.42.2.92:48888 - R:/10.42.0.78:9092] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041
20:14:47.292 [grpc-nio-worker-ELG-5-2] DEBUG io.grpc.netty.NettyClientHandler - [id: 0x5fe41263, L:/10.42.2.92:48888 - R:/10.42.0.78:9092] INBOUND SETTINGS: ack=false settings={HEADER_TABLE_SIZE=4096, MAX_CONCURRENT_STREAMS=2147483647, INITIAL_WINDOW_SIZE=268435456,=0}
20:14:47.292 [grpc-nio-worker-ELG-5-2] DEBUG io.grpc.netty.NettyClientHandler - [id: 0x5fe41263, L:/10.42.2.92:48888 - R:/10.42.0.78:9092] OUTBOUND SETTINGS: ack=true
20:14:47.293 [grpc-nio-worker-ELG-5-2] DEBUG io.grpc.netty.NettyClientHandler - [id: 0x5fe41263, L:/10.42.2.92:48888 - R:/10.42.0.78:9092] INBOUND SETTINGS: ack=true
20:14:47.294 [grpc-nio-worker-ELG-5-2] DEBUG io.grpc.netty.NettyClientHandler - [id: 0x5fe41263, L:/10.42.2.92:48888 - R:/10.42.0.78:9092] INBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=268369921
20:14:47.295 [grpc-nio-worker-ELG-5-2] DEBUG io.grpc.netty.NettyClientHandler - [id: 0x5fe41263, L:/10.42.2.92:48888 - R:/10.42.0.78:9092] OUTBOUND HEADERS: streamId=3 headers=GrpcHttp2OutboundHeaders[:authority: //loan-svc, :path: /com.sumglobal.loan.Loan/ProcessApplication, :method: POST, :scheme: http, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.30.2, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false
20:14:47.297 [grpc-nio-worker-ELG-5-2] DEBUG io.grpc.netty.NettyClientHandler - [id: 0x5fe41263, L:/10.42.2.92:48888 - R:/10.42.0.78:9092] OUTBOUND DATA: streamId=3 padding=0 endStream=true length=72 bytes=000000004308ae11100a1a093336206d6f6e746873220452454e5428c0bb013202414c3a087665726966696564420b6372656469745f636172644d3333dd415d...
20:14:48.304 [DefaultDispatcher-worker-1] ERROR c.s.l.LoanSpaServiceEndpoint$Companion - io.grpc.StatusException: UNAVAILABLE: Network closed for unknown reason
loan-spa sidecar (istio-proxy):
2020-09-02T20:14:47.253109Z debug envoy http [external/envoy/source/common/http/conn_manager_impl.cc:1710] [C36495][S2188793821905667711] encoding headers via codec (end_stream=false):
':status', '200'
'content-type', 'application/grpc-web-text+proto'
'grpc-encoding', 'identity'
'grpc-accept-encoding', 'gzip'
'x-envoy-upstream-service-time', '8'
'x-envoy-peer-metadata', 'ChoKCkNMVVNURVJfSUQSDBoKS3ViZXJuZXRlcwo2CgxJTlNUQU5DRV9JUFMSJhokMTAuNDIuMi45MixmZTgwOjo1OGE5Ojc5ZmY6ZmU5YjpkNTVlCuUBCgZMQUJFTFMS2gEq1wEKEQoDYXBwEgoaCGxvYW4tc3BhChkKDGlzdGlvLmlvL3JldhIJGgdkZWZhdWx0CiEKEXBvZC10ZW1wbGF0ZS1oYXNoEgwaCjc0ZjljOTZkYmIKJAoZc2VjdXJpdHkuaXN0aW8uaW8vdGxzTW9kZRIHGgVpc3RpbwotCh9zZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1uYW1lEgoaCGxvYW4tc3BhCi8KI3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLXJldmlzaW9uEggaBmxhdGVzdAoaCgdNRVNIX0lEEg8aDWNsdXN0ZXIubG9jYWwKIwoETkFNRRIbGhlsb2FuLXNwYS03NGY5Yzk2ZGJiLXNyNnZzChMKCU5BTUVTUEFDRRIGGgRsb2FuCkkKBU9XTkVSEkAaPmt1YmVybmV0ZXM6Ly9hcGlzL2FwcHMvdjEvbmFtZXNwYWNlcy9sb2FuL2RlcGxveW1lbnRzL2xvYW4tc3BhChwKD1NFUlZJQ0VfQUNDT1VOVBIJGgdkZWZhdWx0ChsKDVdPUktMT0FEX05BTUUSChoIbG9hbi1zcGE='
'x-envoy-peer-metadata-id', 'sidecar~10.42.2.92~loan-spa-74f9c96dbb-sr6vs.loan~loan.svc.cluster.local'
'date', 'Wed, 02 Sep 2020 20:14:47 GMT'
'server', 'istio-envoy'
2020-09-02T20:14:47.291644Z debug envoy http2 [external/envoy/source/common/http/http2/codec_impl.cc:1012] [C36499] updating connection-level initial window size to 268435456
2020-09-02T20:14:47.297973Z debug envoy http [external/envoy/source/common/http/conn_manager_impl.cc:268] [C36499] new stream
2020-09-02T20:14:47.298058Z debug envoy http2 [external/envoy/source/common/http/http2/codec_impl.cc:717] [C36499] invalid frame: Invalid HTTP header field was received on stream 3
2020-09-02T20:14:47.298128Z debug envoy http [external/envoy/source/common/http/conn_manager_impl.cc:284] [C36499] dispatch error: The user callback function failed
2020-09-02T20:14:47.298140Z debug envoy http [external/envoy/source/common/http/conn_manager_impl.cc:1940] [C36499][S9392232610876426736] stream reset