Upstream connect error or disconnect/reset before headers. reset reason: connection termination

1

I am trying to expose the default Jaeger client via the istio ingress gateway and am getting an error I cannot understand. With curl it seems to work normally.

$ curl https://jaeger.foo.com/ -I
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
date: Fri, 15 Nov 2019 12:20:52 GMT
x-envoy-upstream-service-time: 0
server: istio-envoy
transfer-encoding: chunked

However in the browser I am seeing 503 with the error ‘upstream connect error or disconnect/reset before headers. reset reason: connection termination’

I am a little unsure how to go about debugging this. Trying working out how to enable access logs for envoy is really confusing me - … ???

Cheers,

Andrew

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: jaeger-gateway
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "jaeger.foo.com"
    tls:
      httpsRedirect: true # sends 301 redirect for http requests
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - "jaeger.foo.com"
    tls:
      mode: SIMPLE
      # these keys have to exist in a secret called 'istio-ingressgateway-certs' in the istio-system namespace.
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt


apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: jaeger-query
spec:
  hosts:
  - "jaeger.foo.com"
  gateways:
  - jaeger-gateway
  http:
  - match:
    - uri:
        prefix: /
    - uri:
        prefix: /ping
    route:
    - destination:
        port:
          number: 16686
        host: jaeger-query-ext


apiVersion: v1
kind: Service
metadata:
  name: jaeger-query-ext
  namespace: istio-system
  annotations:
  labels:
    app: jaeger
    jaeger-infra: jaeger-service
    chart: tracing
    heritage: Tiller
    release: istio
spec:
  ports:
    - name: query-http
      port: 16686
      protocol: TCP
      targetPort: 16686
  selector:
    app: jaeger

$ kubectl version
Client Version: version.Info{Major:“1”, Minor:“13+”, GitVersion:“v1.13.11-dispatcher”, GitCommit:“2e298c7e992f83f47af60cf4830b11c7370f6668”, GitTreeState:“clean”, BuildDate:“2019-09-19T22:26:40Z”, GoVersion:“go1.11.13”, Compiler:“gc”, Platform:“darwin/amd64”}
Server Version: version.Info{Major:“1”, Minor:“13+”, GitVersion:“v1.13.11-gke.14”, GitCommit:“56d89863d1033f9668ddd6e1c1aea81cd846ef88”, GitTreeState:“clean”, BuildDate:“2019-11-07T19:12:22Z”, GoVersion:“go1.12.11b4”, Compiler:“gc”, Platform:“linux/amd64”}

$ istioctl version
client version: 1.4.0-beta.5
control plane version: 1.4.0-beta.5
Error: 1 error occurred:
* error execing into istio-pilot-5db477d8d7-gsdmn discovery container: unable to upgrade connection: container not found (“discovery”)

2

I have discovered that when making the request with http1.1 it works but http2 is broken.

# curl --http1.1  -I https://jaeger.foo.com/
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
date: Sat, 16 Nov 2019 17:10:56 GMT
x-envoy-upstream-service-time: 0
server: istio-envoy
transfer-encoding: chunked

# curl --http2  -I https://jaeger.foo.com/
HTTP/2 503 
content-length: 95
content-type: text/plain
date: Sat, 16 Nov 2019 17:11:01 GMT
server: istio-envoy
1 Like
3

I’m experience the exact same issue. @mooperd have you been able to debug this further?

4

It’s a bug in Envoy. It doesn’t support/screws up Application-Layer Protocol Negotiation(ALPN) If you have a http1.1 service in the backend and a client sending Upgrade: h2c - Envoy will upgrade the connection to http2 and then force the backend to use http2 even if its not supported.


1 Like
5

do you think it’s a good idea to downgrade the istio version until this is patched? hmm. it has been a difficult upgrade :stuck_out_tongue:

6

Hmm, it seems that this has been a problem in Envoy for a while. Did this start happening since you upgraded? What is the version that you’re upgrading from?

7

i can confirm that with a downgrade to istio v1.3.5 everything works as expected again. I was trying an upgrade to v1.4 so it was using whatever Envoy that version of Istio’s mesh uses. Thanks for tracing down this issue, very interesting bug because around half of my services were still working (probably because they supported HTTP2).

Drove me crazy because HTTP still worked, and I thought it was a mismanaged cert or an issue with istio’s SDS service.

8

Hi.

Check by changing the port name of the jaeger service from ‘query-http’ to ‘http-query’.
It seems that from istio 1.4, when the naming convention is not followed it uses the same protocol as in input so http2 if you use https to access the gateway.
When specifying http-query, envoy will know that the backend it http1.1 and not http2 and will sned the request accordingly.

Regards
Aurelien

9

The issue still persists with Istio 1.4.1 , HTTPS fails with ‘upstream error’ whereas HTTP goes through. Everything works well with 1.3.5 and 1.3.6 . @mooperd have you found some solution ?

10

Workaround : changing port name to http-query fixed the issue

image
image796×248 37.2 KB

1 Like
12

I do not think this is a workaround. This is a recommended configuration and is documented in the latest Istio docs. https://istio.io/docs/ops/configuration/traffic-management/protocol-selection/

2 Likes
14

Hi, the message is bit misleading.

In my case Upstream connect error or disconnect/reset before headers. reset reason: connection termination was returned in case when containerPort in POD is set to incorrect value.

You can simulate this by adjusting incorrect value within httpbin pod

apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 8080 # would be 80 viz below
  selector:
    app: httpbin
---
....
      containers:
        - name: httpbin
          image: docker.io/kennethreitz/httpbin
          imagePullPolicy: IfNotPresent
          ports:
          - containerPort: 8080 #httpbin is running on 80
15

Hi,

i confirm this. The bug is not a bug but a behavior, which is expected, when the configuration is not
clean and ISTIO works more restricted. I like that.

Well, we worked hard to fix the issue and solved it with the forum thread. So, naming the ports in the
service definition right fix your issue.
In our case switched the ISTIO framework the http1.1 traffic automatically to http2 and the service ot
disrupted.

Best regards,
Jan

16

We are seeing this same problem with 1.4.6 and we have the port named correctly. We are using https and would like to force http1.1 using browser. Is that possible?

17

The right protocol must be selected

in my case

ports:
- port: 9080
  targetPort: 1900
  name: http-web

http-web being the protocol.

List of supported protocols can be found in the link

2 Likes
18

Worked like a charm :smiley: