I was reviewing the multi-cluster survey and I realized there are probably others doing the same thing we are. I’d love to hear more about the architecture, configuration, and implementations that others are using to build their mesh with many clusters each with isolated control planes.
I’ll start:
At Shopify we run lots of GKE clusters. Some are very large and some are small. Some share a flat network but many do not. We’re using isolated control planes so that the mesh is not dependent on any one cluster to own the control plane. Our services may run in any cluster (or multiple clusters), and which cluster is not a permanent thing and not known by the service owners. With the number of services and clusters we need the ability to converge all this configuration automatically. We’re discovering we need a lot of glue to put this together.
This is very interesting and something we are also exploring, testing & working on making it easy for users to adopt. Can you elaborate a little more on your use case, specifically:
“Our services may run in any cluster (or multiple clusters), and which cluster is not a permanent thing and not known by the service owner” ->
Does this mean if services need to talk to each other they might end up in different clusters and you want to use Istio to transparently setup networking or you have scheduling magic to co-locate services which talk to each other on the same cluster?
If it’s the former what are the reasons for choosing this architecture over traditional multi-cluster with HA configured where often services communicating each other end up on the same cluster.
Configuration convergence - > Are you referring to the additional setup around creating ServiceEntries for accessing services in remote clusters, cert management, etc. as listed here. I’m curious if there’s extra configuration overhead that’s specific to your environment which might be interesting to know.
The former. We make no guarantees to our services regarding co-locating with any other service. We explicitly instruct them to assume they are not on the same cluster. This means today nearly all services are using full Ingress path to route to another service.
For various reasons we have multiple “tiers” that are isolated. And within that we have outgrown a single cluster in some cases so we have multiple clusters for each tier for each region. (We have lots of services.) Services need the ability to communicate across regions and tiers and clusters. Any given service might need to talk to another service and shouldn’t need to know where it’s located (at that moment). We’re aiming to have the service mesh solve this automatically.
Yes, mostly we’re automating the creation of ServiceEntries, VirtualServices, DestinationRules, etc as service owners won’t interact with Istio directly. Each service will each define the services it needs to communicate with (egress) in a manifest. Our automation will consume this, build a model of the world, and generate the proper configuration in each cluster.
The other thing specific to our environment is that we’re using our own (public) domains instead of *.svc.cluster.local or *.global domains which will simplify our transition onto the mesh.
Thanks for sharing this. If I understand correctly in this scheme all the services deployed in any clusters will need to have unique non-conflicting names. How do you manage that? All via DNS names on your public domain?
This architecture works quite well with the Istio Multi control plane, multi cluster design as it also requires non-conflicting services naming across clusters.
You mentioned that developers provide egress for their application in a manifest and automation takes it from there, is this a home grown mechanism with custom schema? Are you also building out Istio policies (Service to service Authz) based on this schema?
Mostly yes. Most services today exist in one place (possibly with a standby in another cluster/region). A few are in multiple places and have unique names or use local names. Nearly all services are via DNS names on a public domain. In the future we hope to migrate to more copies of every service and the mesh will know how to route. The service manifest + current location information will inform this routing.
Yes home grown and custom schema, but the schema could be generalized and we’re considering building the automation in open source.
