I have a problem and a proposed solution. I would appreciate any input that folks have on this.
Problem:
We have multiple instances of our software running on a single cluster, each in its own namespace. Each namespace has an istio gateway exposed through a NodePort service. External devices need to be able to connect to these services and they need to be routed to a particular namespace, and for various reasons, we can’t configure the devices to open connections on the ports that each gateway exposes. We need a way to map the ports that devices connect on to the gateway’s NodePorts. The devices are using raw TCP and UDP, so we can’t route by hostname.
Example:
On a single worker node at 1.2.3.4,
Device A connects:
1.2.3.4:9100 -> 1.2.3.4:32301 -> pod X in namespace A
Device B connects:
1.2.3.4:9100 -> 1.2.3.4:32302 -> pod X in namespace B
Obviously there’s an ambiguity here – we need a unique combination of IP & Port, and since we can’t change the port, we need a unique IP.
Proposed solution:
Configure a VIP for each namespace on exactly one worker node, and map ports for that VIP to a set of NodePorts for that namespace. A custom VIP controller in each namespace would ensure that exactly one worker node has that namespace’s VIP (to avoid an ARP collision) and that the iptables rules correctly map ports on that VIP to NodePorts on the same namespace’s istio gateway.
Example:
On a single worker node,
1.1.1.1 and 2.2.2.2 are externally addressable VIPs on the same worker node (1.2.3.4)
Device A connects:
1.1.1.1:9100 -> iptables routes to -> 1.2.3.4:32301 -> pod X in namespace A
Device B connects
2.2.2.2:9100 -> iptables routes to -> 1.2.3.4:32302 -> pod X in namespace B
Open questions:
Is there a more idiomatic way to handle this in istio? Could istio perform the functions that the proposed VIP controller performs? I suspect not, but I want to make sure we’re not reinventing the wheel here.