Hybrid Cloud Networking

We are designing our platform to be compatible with different cloud providers (GCloud, OCI, Azure etc). Our platform is built using micro-services and we have our 30 of them. We recently began migration to Kubernetes. Our problem is the following, some of our services are to be run in both our’s and customer’s infrastructure. For example, if we have three services X, Y & Z in our platform. If X is deployed on customer infrastructure and Y and Z on our infrastructure, how do I best set up networking amongst X, Y & Z? Currently we have had some success with HashiCorp’s Consul using it as a key-value store. Basically X will query Consul for Y’s endpoint. I am not sure if this is the best possible solution. I wonder if this problem is solvable through other tools such as KubeDNS or a distributed key-value store more cleanly: maybe run etcd node on customer’s infrastructure and our’s? Another complication that might arise is that Y and Z will not be in the same Kubernetes namespace on our infrastructure. For instance Y might be a web server serving static files. Where as Z might be a service we have deployed on behalf of the customer in our infrastructure and all three X, Y and Z must be able to talk to each other. Currently we are using Istio for service mesh (and TLS offloading). Is there a solution that leverages Istio in this case?