Websocket failing through istio gateway. content-length: 0 injected into request

Networking
1

I am attempting to debug an issue where a websocket over wss:// is failing right away. I have stood up a simple example of a Python Websockets Server. The client is als from the example.

My setup is as follows

internet --> ambassador(handles auth) --> istio-ingressgateway.istio-system:80 --> gateway + virtual service --> pod

The websocket connection is over wss://[host...] and Ambassador is doing the TLS termination.

The server error is

websocket-test websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason
istio-proxy [2021-09-30T03:04:30.045Z] "- - -" 0 - - - "-" 4389 247 4 - "-" "-" "-" "-" "127.0.0.1:8000" inbound|8000|| 127.0.0.1:55930 10.63.29.46:8000 10.63.29.108:45766 outbound_.8000_._.websocket-test.websocket-test.svc.cluster.local -

When I look at the server logs I do notice that the content length header is injected with 0.

[21-09-30T03:04:30][websockets.server][DEBUG] server < GET / HTTP/1.1
[21-09-30T03:04:30][websockets.server][DEBUG] server < Headers([('host', *****'), ('upgrade', 'websocket'), ('connection', 'Upgrade'), ('user-agent', 'Python/3.6 websockets/9.1'), ('content-length', '0')
[21-09-30T03:04:30][websockets.server][DEBUG] server > HTTP/1.1 101 Switching Protocols
[21-09-30T03:04:30][websockets.server][DEBUG] server > Headers([('Upgrade', 'websocket'), ('Connection', 'Upgrade'), ('Sec-WebSocket-Accept', 'jR+h/aFmr9DHEPeLneURZJgqNRQ='), ('Sec-WebSocket-Extensions', 'permessage-deflate'), ('Date', 'Thu, 30 Sep 2021 03:04:30 GMT'), ('Server', 'Python/3.6 websockets/9.1')])

I’m wondering if it’s this error: Content-length: 0 header inserted in Websocket connection upgrade response · Issue #25814 · istio/istio · GitHub

This is how I’ve setup the test

apiVersion: v1
kind: Service
metadata:
  name: websocket-test
  labels:
    run: websocket-test
spec:
  ports:
  - port: 8000
    protocol: TCP
  selector:
    app: websocket-test
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: websocket-test
spec:
  gateways:
  -  websocket-test
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /websocket
    rewrite:
      uri: /
    route:
    - destination:
        host: websocket-test.websocket-test.svc.cluster.local
        port:
          number: 8000
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: websocket-test
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: web-websocket
      number: 80
      protocol: HTTP
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: allow-all  # A proper policy is needed here, this is here to ensure RBAC isn't the cause
spec:
 rules:
 - {}
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: websocket-test
spec:
  dnsNames:
    - **********
  secretName: websocket-test-tls-cert
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
---
apiVersion: getambassador.io/v2
kind: Host
metadata:
  name: websocket-test
  annotations:
    external-dns.ambassador-service: ambassador-public.ambassador-public
spec:
  ambassador_id: "public"
  hostname: *************************
  acmeProvider:
    authority: none
  tlsSecret:
    name: websocket-test-tls-cert
  tls:
    alpn_protocols: h2,http/1.1
---
kind: Mapping
apiVersion: getambassador.io/v2
metadata:
  name: websocket-test
spec:
  ambassador_id: "public"
  grpc: false
  prefix: /
  host: *****************
  service: istio-ingressgateway.istio-system:80
  allow_upgrade:
    - websocket

I’m on the following version of Istio

                    "ISTIO_PROXY_SHA": "istio-proxy:172db4cfdf037bc8cf1613969f94c25b6198bc4f",
                    "ISTIO_VERSION": "1.9.6",

Other things I’ve tried where the websocket test works:

  • Websocket directly behind Ambassador. It works just fine over wss://[insert host].
  • Kubectl port-forward the istio-ingressgateway to local laptop and connect to ws://localhost:8000.

The Github issues seem to suggest that upgrading to Istio 1.10 fix this. It is not clear or is something else wrong here?

2

So digging into this has yielded that when Ambassador routes traffic to the istio gateway or directly to a pod websockets fail. Ambassador is routing the traffic to the istio gateway using HTTP and it seems that ambassador is closing the connection for unknown reasons because the logs by the gateway and the proxy pod say the client is disconnecting.

The client in this case is likely ambassador.

I’m going to try to integrate ambassador and Isto more deeply using this guide: https://www.getambassador.io/docs/emissary/latest/howtos/istio/