Upstream connect error or disconnect/reset before headers. reset reason: connection termination

I am trying to expose the default Jaeger client via the istio ingress gateway and am getting an error I cannot understand. With curl it seems to work normally.

$ curl https://jaeger.foo.com/ -I
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
date: Fri, 15 Nov 2019 12:20:52 GMT
x-envoy-upstream-service-time: 0
server: istio-envoy
transfer-encoding: chunked

However in the browser I am seeing 503 with the error ‘upstream connect error or disconnect/reset before headers. reset reason: connection termination’

I am a little unsure how to go about debugging this. Trying working out how to enable access logs for envoy is really confusing me - … ???

Cheers,

Andrew

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: jaeger-gateway
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "jaeger.foo.com"
    tls:
      httpsRedirect: true # sends 301 redirect for http requests
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - "jaeger.foo.com"
    tls:
      mode: SIMPLE
      # these keys have to exist in a secret called 'istio-ingressgateway-certs' in the istio-system namespace.
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt


apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: jaeger-query
spec:
  hosts:
  - "jaeger.foo.com"
  gateways:
  - jaeger-gateway
  http:
  - match:
    - uri:
        prefix: /
    - uri:
        prefix: /ping
    route:
    - destination:
        port:
          number: 16686
        host: jaeger-query-ext


apiVersion: v1
kind: Service
metadata:
  name: jaeger-query-ext
  namespace: istio-system
  annotations:
  labels:
    app: jaeger
    jaeger-infra: jaeger-service
    chart: tracing
    heritage: Tiller
    release: istio
spec:
  ports:
    - name: query-http
      port: 16686
      protocol: TCP
      targetPort: 16686
  selector:
    app: jaeger

$ kubectl version
Client Version: version.Info{Major:“1”, Minor:“13+”, GitVersion:“v1.13.11-dispatcher”, GitCommit:“2e298c7e992f83f47af60cf4830b11c7370f6668”, GitTreeState:“clean”, BuildDate:“2019-09-19T22:26:40Z”, GoVersion:“go1.11.13”, Compiler:“gc”, Platform:“darwin/amd64”}
Server Version: version.Info{Major:“1”, Minor:“13+”, GitVersion:“v1.13.11-gke.14”, GitCommit:“56d89863d1033f9668ddd6e1c1aea81cd846ef88”, GitTreeState:“clean”, BuildDate:“2019-11-07T19:12:22Z”, GoVersion:“go1.12.11b4”, Compiler:“gc”, Platform:“linux/amd64”}

$ istioctl version
client version: 1.4.0-beta.5
control plane version: 1.4.0-beta.5
Error: 1 error occurred:
* error execing into istio-pilot-5db477d8d7-gsdmn discovery container: unable to upgrade connection: container not found (“discovery”)

I have discovered that when making the request with http1.1 it works but http2 is broken.

# curl --http1.1  -I https://jaeger.foo.com/
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
date: Sat, 16 Nov 2019 17:10:56 GMT
x-envoy-upstream-service-time: 0
server: istio-envoy
transfer-encoding: chunked

# curl --http2  -I https://jaeger.foo.com/
HTTP/2 503 
content-length: 95
content-type: text/plain
date: Sat, 16 Nov 2019 17:11:01 GMT
server: istio-envoy

I’m experience the exact same issue. @mooperd have you been able to debug this further?

It’s a bug in Envoy. It doesn’t support/screws up Application-Layer Protocol Negotiation(ALPN) If you have a http1.1 service in the backend and a client sending Upgrade: h2c - Envoy will upgrade the connection to http2 and then force the backend to use http2 even if its not supported.


1 Like

do you think it’s a good idea to downgrade the istio version until this is patched? hmm. it has been a difficult upgrade :stuck_out_tongue:

Hmm, it seems that this has been a problem in Envoy for a while. Did this start happening since you upgraded? What is the version that you’re upgrading from?

i can confirm that with a downgrade to istio v1.3.5 everything works as expected again. I was trying an upgrade to v1.4 so it was using whatever Envoy that version of Istio’s mesh uses. Thanks for tracing down this issue, very interesting bug because around half of my services were still working (probably because they supported HTTP2).

Drove me crazy because HTTP still worked, and I thought it was a mismanaged cert or an issue with istio’s SDS service.