I am working to introduce Terraform in the company for automatically managing the cloud environment.
The challenge is on the one side, that we are living in the IBM managed cloud and we want to use the
VPC and managed K8S cluster there.
Well, this runs fine now and as expected, but, I would like to install some packages via HELM into it like
the package of ISTIO.
I thought about using HELM to prevent a call to kubectl from Terraform as shell command to deploy the
yaml files I previously generated by manually by helm template. After some searching around via
Google gives me the impression, that this will not work.
So, question around, does anyone get a ISTIO deployment work with Terraform (11 or 12)?
While it is strongly recommended that you install via istioctl (or if you are willing to use alpha software, the istio operator https://istio.io/about/feature-stages/#core), there are ways to install helm charts via Terraform. However, doing so comes with a good deal of operational risk, especially if the deployment method you choose is using the Helm Tiller. If you really do need to install via helm in terraform, please try using helm template instead to dump the chart as rendered YAML, and then using kubectl apply in a script to install the generated templates. It is much safer than using normal Helm 2 when imbedded in Terraform. (Note: this may not apply to Helm 3, but it was released only five days ago and I have not evaluated it at all, nor do I know if it is even compatible with Helm 2 charts.)
To give some more background, if you haven’t delved very deep in Helm: the Helm Tiller is a pod that runs inside your cluster with very elevated permissions and keeps track of the state of charts you have deployed using configmaps or secrets it creates in the namespace. This internal state does not always accurately reflect the actual state of the resources in the cluster, and also tends not to play nice with the external state expected by Terraform. If you use one of the more deeply integrated tools (e.g. the Terraform Helm provider), you can run into situations where Terraform or Helm decide that the best way to get your chart into the target state is to delete it and recreate it, which on something that controls inter-pod communication in the cluster will cause serious issues. Rendering out to yaml via the Helm template command and running a kubectl apply means that all the logic regarding reonciling cluster state with the target state is handled by the kubernetes cluster, which is really where it needs to happen.
I’d also be cautious of mixing your infrastructure deployment terraform code with the code used to deploy things on said infrastructure. It may be fixed in Terraform 12, but in Terraform 11 it was a bit tricky to write modules such that they would wait for the kubernetes cluster to be ready to accept kubectl commands before trying and failing to run the kubectl apply script. Working around this issue is doable, but it may be better to separate out the cluster creation code from the cluster bootstrapping code.
If you want to continue down this path, I can give you some additional resources. Just let me know a bit more about what you are trying to accomplish so I can make sure they are what you need.
Our goal is to achieve IaC for all resources, we create and deploy in our cloud. We decided to use IBM as our cloud provider and I we feel comfortable to live there. Anyway it came to my mind to have code, which represents our created managed services (like VPC, K8S cluster, DBaaS,…) for dev and production. I decided to use terraform for that because of the two reasons:
even we run Terraform 11 against IBM (since the provider is not ported to 12), we have a code based declaration about our needed cloud services
after reading blog post series from Gruntworks [1] and their book [2], I am able to have a nice isolated and modulised representation of our infrastructure and cluster resources.
Now I specified for me and my team, that a ready VPC for us includes
VPC with network configuration
K8S cluster spread about availability zones inside the VPC
ISTIO as service mesh inside a K8S cluster
The last step would be to include an ISTIO deployment with Terraform. I tried Helm and Kubernetes provider but the ways did not satisfy me. .-)
Before starting with Terraform I installed ISTIO by helm template, got two yaml files (istio-init and istio) and deployed them via kubectl apply and I was happy. I did not try an installation with istioctl.
At the end would it be great to have IaC managing the deployment of all our needed things and use it in our job scheduler, like Jenkins, for testing our automatically deployment for new environments without doing much steps manually. .-)
And now I look for a way to get this achieved with ISTIO, but perhaps am I on the wrong way?
OK, if all you are planning to do is run a small set of initial bootstrapping scripts and commands, this is what I’d try:
Set up and test your Terraform pipeline in whatever job scheduler you wanted it to run in, make sure it provisions your k8s cluster
Add a local-exec provisioner ( https://www.terraform.io/docs/provisioners/local-exec.html) to the k8s provider call (or a properly configured null provider entry), and have it execute a setup script that you write which executes all the commands you want. Add a sleep-wait loop at the start of the script that will wait for the k8s control plane to respond, then execute the provisioning code once starts accepting traffic.
Adjust the request timeouts of Terraform so that the script can spin-wait long enough for the control plane to come up without timing out. I don’t recall exactly how to do this, but I think it is a flag you can pass to Terraform when it runs.
That should get istio running on your cluster. Take care to make sure that the bootstrap script you write is idempotent, since Terraform may run it multiple times if you make adjustments to the kubernetes cluster config down the line. Also keep in mind that this steps outside of the typical declarative nature of Terraform, so there won’t be any checks that the configuration is up to date on subsequent runs if you modify the bootstrap scripts at all.
I’ve not tested this exact process, but it is pretty similar to some things I have done previously. Hopefully it points you in the right direction. Let me know if you need any clarification.