Separate helm releases for Istio and gateways

Hello. I’ve run into an issue attempting to use a separate helm release for the Istio installation and then separate helm releases for each gateway. I’m finding that when I go to create a gateway using the example gateway-specific values file[1], it attempts to create several resources that already exist because of the original Istio helm installation. This causes “helm install” to fail.

Our basic install procedure is:

GIT_PATH=`git rev-parse --show-toplevel`
helm install ${GIT_PATH}/istio-1.1.5/install/kubernetes/helm/istio --name m-prototype-ilbgateway --namespace m-prototype-ilbgateway -f gke2-services-2-m-prototype-ilbgateway.yaml

The values file looks like:

# Common settings.
global:
  # Omit the istio-sidecar-injector configmap when generate a
  # standalone gateway. Gateways may be created in namespaces other
  # than `istio-system` and we don't want to re-create the injector
  # configmap in those.
  omitSidecarInjectorConfigMap: true

  # Istio control plane namespace: This specifies where the Istio control
  # plane was installed earlier.  Modify this if you installed the control
  # plane in a different namespace than istio-system.
  istioNamespace: istio-system

  proxy:
    # Sets the destination Statsd in envoy (the value of the "--statsdUdpAddress" proxy argument
    # would be <host>:<port>).
    # Disabled by default.
    # The istio-statsd-prom-bridge is deprecated and should not be used moving forward.
    envoyStatsd:
      # If enabled is set to true, host and port must also be provided. Istio no longer provides a statsd collector.
      enabled: false
      host: # example: statsd-svc.istio-system
      port: # example: 9125


#
# Gateways Configuration
# By default (if enabled) a pair of Ingress and Egress Gateways will be created for the mesh.
# You can add more gateways in addition to the defaults but make sure those are uniquely named
# and that NodePorts are not conflicting.
# Disable specifc gateway by setting the `enabled` to false.
#
gateways:
  enabled: true

  # Explicitly disable the default ingress gateways
  istio-ingressgateway:
    enabled: false

  istio-egressgateway:
    enabled: false

  istio-ilbgateway:
    enabled: false

  m-prototype-ilbgateway:
    enabled: true
    labels:
      app: m-prototype-ilbgateway
      istio: m-prototype-ilbgateway

The duplicate resources are:

ClusterRole / istio-reader
ClusterRoleBinding / istio-multi

I believe these are both cluster wide objects, so even moving the gateway helm release into its own namespace does not resolve the conflict.

Ultimately if you delete the failed install of the gateway, you have to do it with --purge, which ends up removing the ClusterRole and ClusterRoleBinding… which is bad. However, once you do that, you can technically reinstall the gateway as a helm release and it will work. The problem now becomes that this helm release now owns those resources. So if you go to remove this gateway, you could accidentally break important Istio stuff.

Any ideas on whether this is a bug, intended behavior, or some other work around is expected?

Thanks for your help.

[1] - https://github.com/istio/istio/blob/9b6d31b74d1c0cc9358cc82d395b53f71393326b/install/kubernetes/helm/istio/example-values/values-istio-gateways.yaml

ClusterRole and ClusterRoleBindings were recently removed from the gateway charts because they were not needed: https://github.com/istio/istio/pull/13292

At present, we only have Role and RoleBindings which are namespaced and probably shouldn’t be a problem with your use case: https://github.com/istio/istio/tree/master/install/kubernetes/helm/istio/charts/gateways/templates

Thank you, @skydoctor! It looks like I can backport that to 1.1.5 without issue until we are able to get on the 1.2.0 train.