Websocket failing through istio gateway. content-length: 0 injected into request

I am attempting to debug an issue where a websocket over wss:// is failing right away. I have stood up a simple example of a Python Websockets Server. The client is als from the example.

My setup is as follows

internet --> ambassador(handles auth) --> istio-ingressgateway.istio-system:80 --> gateway + virtual service --> pod

The websocket connection is over wss://[host...] and Ambassador is doing the TLS termination.

The server error is

websocket-test websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason
istio-proxy [2021-09-30T03:04:30.045Z] "- - -" 0 - - - "-" 4389 247 4 - "-" "-" "-" "-" "127.0.0.1:8000" inbound|8000|| 127.0.0.1:55930 10.63.29.46:8000 10.63.29.108:45766 outbound_.8000_._.websocket-test.websocket-test.svc.cluster.local -  

When I look at the server logs I do notice that the content length header is injected with 0.

[21-09-30T03:04:30][websockets.server][DEBUG] server < GET / HTTP/1.1
[21-09-30T03:04:30][websockets.server][DEBUG] server < Headers([('host', *****'), ('upgrade', 'websocket'), ('connection', 'Upgrade'), ('user-agent', 'Python/3.6 websockets/9.1'), ('content-length', '0')
[21-09-30T03:04:30][websockets.server][DEBUG] server > HTTP/1.1 101 Switching Protocols
[21-09-30T03:04:30][websockets.server][DEBUG] server > Headers([('Upgrade', 'websocket'), ('Connection', 'Upgrade'), ('Sec-WebSocket-Accept', 'jR+h/aFmr9DHEPeLneURZJgqNRQ='), ('Sec-WebSocket-Extensions', 'permessage-deflate'), ('Date', 'Thu, 30 Sep 2021 03:04:30 GMT'), ('Server', 'Python/3.6 websockets/9.1')])   

I’m wondering if it’s this error: Content-length: 0 header inserted in Websocket connection upgrade response · Issue #25814 · istio/istio · GitHub

This is how I’ve setup the test

apiVersion: v1
kind: Service
metadata:
  name: websocket-test
  labels:
    run: websocket-test
spec:
  ports:
  - port: 8000
    protocol: TCP
  selector:
    app: websocket-test
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: websocket-test
spec:
  gateways:
  -  websocket-test
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /websocket
    rewrite:
      uri: /
    route:
    - destination:
        host: websocket-test.websocket-test.svc.cluster.local
        port:
          number: 8000
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: websocket-test
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: web-websocket
      number: 80
      protocol: HTTP
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: allow-all  # A proper policy is needed here, this is here to ensure RBAC isn't the cause
spec:
 rules:
 - {}
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: websocket-test
spec:
  dnsNames:
    - **********
  secretName: websocket-test-tls-cert
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
---
apiVersion: getambassador.io/v2
kind: Host
metadata:
  name: websocket-test
  annotations:
    external-dns.ambassador-service: ambassador-public.ambassador-public
spec:
  ambassador_id: "public"
  hostname: *************************
  acmeProvider:
    authority: none
  tlsSecret:
    name: websocket-test-tls-cert
  tls:
    alpn_protocols: h2,http/1.1
---
kind: Mapping
apiVersion: getambassador.io/v2
metadata:
  name: websocket-test
spec:
  ambassador_id: "public"
  grpc: false
  prefix: /
  host: *****************
  service: istio-ingressgateway.istio-system:80
  allow_upgrade:
    - websocket

I’m on the following version of Istio

                    "ISTIO_PROXY_SHA": "istio-proxy:172db4cfdf037bc8cf1613969f94c25b6198bc4f",
                    "ISTIO_VERSION": "1.9.6",

Other things I’ve tried where the websocket test works:

  • Websocket directly behind Ambassador. It works just fine over wss://[insert host].
  • Kubectl port-forward the istio-ingressgateway to local laptop and connect to ws://localhost:8000.

The Github issues seem to suggest that upgrading to Istio 1.10 fix this. It is not clear or is something else wrong here?