Istio multi-network multi-mesh hub-and-spoke with no listening ports on spokes / passive / reverse tunnel

I’m trying to expose a specific service on a widely distributed multi-cloud set of kube clusters to a central control cluster. The goal is for the central control cluster to be able to discover, enumerate, and make outbound connections to the spoke services without having any public-reachable IPs or listening ports on the spokes.

Istio’s support for multi-network clusters with mesh federation looks to be ideal for the job, except that it requires mutual reachability over the public Internet. For a variety of reasons (compliance box-ticking, spokes behind NAT, etc) I need to have all the spokes initiate connections to the hub/control, and have all hub-to-spoke connections flow over a tunnel initiated by the spokes.

The desired model is similar to that used by Teleport - where the Teleport agents make persistent outbound connections to the Teleport Proxy, and the Teleport Proxy uses those to reverse-tunnel traffic back through the agents to the destination services. Unfortunately it looks like Teleport is unsuitable for this job in other ways - mainly due to overheads,

As far as I can tell the Istio Gateway and Istio Proxy do not offer any support for reverse tunnels.

If I could configure an Istio Proxy on the spoke networks to make outbound connections to the Istio Gateway or Proxy on the hub to establish reverse tunnels that Istio would use to route requests from hub services to exposed spoke services, that’d be ideal. But it doesn’t seem to be possible.

Anyone else had similar issues? Were you able to solve this with Istio? Or some other tool? I’ve been very surprised by how limited the options for this seemingly-obvious problem seem to be.

I posted a related question at Stack Overflow (question number 77277028, apparently I’m not allowed to link to it directly for some reason)

My findings so far suggest that Istio is not capable of satisfying this use case. It’d need support in the Proxy or Ingress Gateway and in the Egress Gateway for negotiating a reverse tunnel. The actual reverse tunnel is possible using HTTP/2 as demonstrated by GitHub - hotnops/gtunnel: A robust tunelling solution written in golang but it looks like it probably has some high round-trip overheads due to the need for command-and-control round-trips for each new tunneled connection. Still probably better than the mess of using SSH with tcp-in-tcp tunnels.

This topic is temporarily closed for at least 1 hour due to a large number of community flags.

This topic was automatically opened after 26 hours.

There’s a kube gateway-sig discussion thread relevant to this too: Can the Gateway API express passive or reverse-tunnel routes? (For hub-spoke east-west traffic w/o mutual reachability) · kubernetes-sigs/gateway-api · Discussion #2490 · GitHub