Separate helm releases for Istio and gateways

1

Hello. I’ve run into an issue attempting to use a separate helm release for the Istio installation and then separate helm releases for each gateway. I’m finding that when I go to create a gateway using the example gateway-specific values file[1], it attempts to create several resources that already exist because of the original Istio helm installation. This causes “helm install” to fail.

Our basic install procedure is:

GIT_PATH=`git rev-parse --show-toplevel`
helm install ${GIT_PATH}/istio-1.1.5/install/kubernetes/helm/istio --name m-prototype-ilbgateway --namespace m-prototype-ilbgateway -f gke2-services-2-m-prototype-ilbgateway.yaml

The values file looks like:

# Common settings.
global:
  # Omit the istio-sidecar-injector configmap when generate a
  # standalone gateway. Gateways may be created in namespaces other
  # than `istio-system` and we don't want to re-create the injector
  # configmap in those.
  omitSidecarInjectorConfigMap: true

  # Istio control plane namespace: This specifies where the Istio control
  # plane was installed earlier.  Modify this if you installed the control
  # plane in a different namespace than istio-system.
  istioNamespace: istio-system

  proxy:
    # Sets the destination Statsd in envoy (the value of the "--statsdUdpAddress" proxy argument
    # would be <host>:<port>).
    # Disabled by default.
    # The istio-statsd-prom-bridge is deprecated and should not be used moving forward.
    envoyStatsd:
      # If enabled is set to true, host and port must also be provided. Istio no longer provides a statsd collector.
      enabled: false
      host: # example: statsd-svc.istio-system
      port: # example: 9125


#
# Gateways Configuration
# By default (if enabled) a pair of Ingress and Egress Gateways will be created for the mesh.
# You can add more gateways in addition to the defaults but make sure those are uniquely named
# and that NodePorts are not conflicting.
# Disable specifc gateway by setting the `enabled` to false.
#
gateways:
  enabled: true

  # Explicitly disable the default ingress gateways
  istio-ingressgateway:
    enabled: false

  istio-egressgateway:
    enabled: false

  istio-ilbgateway:
    enabled: false

  m-prototype-ilbgateway:
    enabled: true
    labels:
      app: m-prototype-ilbgateway
      istio: m-prototype-ilbgateway

The duplicate resources are:

ClusterRole / istio-reader
ClusterRoleBinding / istio-multi

I believe these are both cluster wide objects, so even moving the gateway helm release into its own namespace does not resolve the conflict.

Ultimately if you delete the failed install of the gateway, you have to do it with --purge, which ends up removing the ClusterRole and ClusterRoleBinding… which is bad. However, once you do that, you can technically reinstall the gateway as a helm release and it will work. The problem now becomes that this helm release now owns those resources. So if you go to remove this gateway, you could accidentally break important Istio stuff.

Any ideas on whether this is a bug, intended behavior, or some other work around is expected?

Thanks for your help.

[1] - https://github.com/istio/istio/blob/9b6d31b74d1c0cc9358cc82d395b53f71393326b/install/kubernetes/helm/istio/example-values/values-istio-gateways.yaml

2

ClusterRole and ClusterRoleBindings were recently removed from the gateway charts because they were not needed: https://github.com/istio/istio/pull/13292

At present, we only have Role and RoleBindings which are namespaced and probably shouldn’t be a problem with your use case: https://github.com/istio/istio/tree/master/install/kubernetes/helm/istio/charts/gateways/templates

3

Thank you, @skydoctor! It looks like I can backport that to 1.1.5 without issue until we are able to get on the 1.2.0 train.

4

I did a bit more testing based upon removing the two files mentioned in the PR you linked. Unfortunately that didn’t address the issue.

I’m by no means a helm aficionado, but now I’m thinking we are using the wrong helm install chart. Our understanding was that we were supposed to target ${GIT_PATH}/istio-1.1.5/install/kubernetes/helm/istio as the chart when creating gateways. Would it make more sense to not try and feed the gateway install through the main helm chart? More specifically, are we supposed to be targeting the subchart directly?

Thanks for your help. I truly appreciate it!

5

Ok – more testing on invoking the subchart directly. It looks like the subchart depends on a global section to look for imagePullSecrets under it. Even if you ensure you have a global section, other parts of the chart crash as it expects every key in the configuration file to be a gateway configuration stanza.

So my takeaway is that currently you can’t invoke the gateway subchart directly.

6

@mhite I use this black magic to install only gateways -

remove templates for gateways temporarily

mkdir tmp
mv install/kubernetes/helm/istio/charts/gateways/templates/* tmp/
helm template install/kubernetes/helm/istio/ --namespace istio-system --name istio > controlPlane.yaml
mv tmp/* install/kubernetes/helm/istio/charts/gateways/templates/
rm -r tmp/

generate yaml with everything

helm template $ISTIO_PATH/install/kubernetes/helm/istio/ --namespace istio-system --name istio --values $VALUES_PATH > fullPlane.yaml

subtract the two yamls to get only gateway config

diff -U $(wc -l < fullPlane.yaml) fullPlane.yaml controlPlane.yaml | sed -n ‘s/^-//p’ | tail -n +2 > dataPlane.yaml