I am trying to figure out a good Istio based build/deploy pipeline for an application. The application supports a number of distinct environments (e.g., pre-staging, staging, and production, etc.), as well as feature branch deployments (e.g., feature-jp12, bugfix-dp34, etc.). The environments will be routed based on the hostname, and feature branches based on a header value.
I’ve found a bunch of documentation and blog posts about how to set up Istio’s VirtualService, DestinationRule, and so on, but I’m struggling with how to actually implement it in a CI/CD pipeline.
The application is built from source code, and the image name and tag combine to specify the environment and feature branch (if any). This information is plugged into a kustomize pipeline to correctly build all the definitions.
The problem that I’m having is that I want builds to be independent. So when a branch is built, it produces a new k8s Deployment, and it should then be deployed to the k8s cluster. The problem is that other branches have been built, so there is already a VirtualService/DestinationRule for the application in the cluster! I can write code or use kustomize to get the currently running VritualService/DestinationRule and update it to add the routing to the Deployment that was just built and deployed, but this has a number of problems.
First, there is a timing problem, as there is no global lock, so two deployments could occur at the same time, with one deployment overwriting the changes in the VirtualService that the other made. I can work around this by building a lock in the CI/CD system, so this seems solvable. Are there better approaches?
If I go with the approach above, I have the problem of garbage collection. I don’t want every version ever deployed to be on the cluster, so I need some way to clean up the VirtualService and DestinationRules. In a perfect world, I would delete the Deployment and the parts of the VirtualService/DestinationRule that pointed to that deployment would be removed. However, it is my understanding that this isn’t possible.
What are other people doing in this space? Is the solution to either hand edit/maintain the VirtualService/DestinationRule, adding and removing versions of the application? Or more custom tooling to do this automatically? I feel like I must be missing something.