Ingress-gateway in customized namescpace still connect istiod after install canary revision

Config
1

hello,

i’m doing Istio upgrading from 1.4.7(helm)–> 1.5.8(helm) --> 1.6.7 in my local VM
it was successfully upgrade from 1.4.7 to 1.5.8 using helm upgrade command, the customized ingress-gateway was in a namespace called pep-internal and works well after upgrade.

when doing upgrade from 1.5.8 to 1.6.7 follwing https://istio.io/latest/docs/setup/upgrade/ with command istioctl install --set revision=canary -f ingress.yaml the new ingress pod in pep-internal is not ready,

[root@localhost ~]# kubectl get po --namespace=pep-internal-ingress
NAME                            READY   STATUS    RESTARTS   AGE
pep-internal-6f4c5667cd-pwr8t   0/1     Running   0          23m
pep-internal-86b688f95c-jq2pq   1/1     Running   0          8d

logs of the pod

2020-08-18T08:57:39.959957Z	info	Proxy role: &model.Proxy{ClusterID:"", Type:"router", IPAddresses:[]string{"10.96.0.43"}, ID:"pep-internal-6f4c5667cd-pwr8t.pep-internal-ingress", Locality:(*envoy_api_v2_cor
e.Locality)(nil), DNSDomain:"pep-internal-ingress.svc.cluster.local", ConfigNamespace:"", Metadata:(*model.NodeMetadata)(nil), SidecarScope:(*model.SidecarScope)(nil), PrevSidecarScope:(*model.SidecarScope)(nil), M
ergedGateway:(*model.MergedGateway)(nil), ServiceInstances:[]*model.ServiceInstance(nil), IstioVersion:(*model.IstioVersion)(nil), ipv6Support:false, ipv4Support:false, GlobalUnicastIP:"", XdsResourceGenerator:mode
l.XdsResourceGenerator(nil), Active:map[string]*model.WatchedResource(nil)}
2020-08-18T08:57:39.959962Z	info	JWT policy is first-party-jwt
2020-08-18T08:57:39.959987Z	info	PilotSAN []string{"istiod.istio-system.svc"}
2020-08-18T08:57:39.959991Z	info	MixerSAN []string{"spiffe://cluster.local/ns/istio-system/sa/istio-mixer-service-account"}
2020-08-18T08:57:39.993585Z	info	serverOptions.CAEndpoint == istiod-canary.istio-system.svc:15012
2020-08-18T08:57:39.993602Z	info	Using user-configured CA istiod-canary.istio-system.svc:15012
2020-08-18T08:57:39.993605Z	info	istiod uses self-issued certificate
2020-08-18T09:21:19.343437Z	warning	envoy config	[bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2020-08-18T09:21:20.164006Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-08-18T09:21:22.163067Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-08-18T09:21:24.163527Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected

the pilot address of the new pod is not right PilotSAN []string{"istiod.istio-system.svc"}, because Istio was upgraded to 1.5.8 with helm, so there’s no istiod in my cluster, should the new ingress pod in pep-internal connect to PilotSAN []string{"istiod-canary.istio-system.svc"} ?

the default ingress gateway pod in namespace istio-system using the PilotSAN []string{"istiod-canary.istio-system.svc"}, so how can i config the new pod in my customized namespace pep-internal to use istiod-canary?

content of ingress.yaml 

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: "/dev/stdout"
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
      - name: pep-internal
        namespace: pep-internal-ingress
        label:
          istio: ingressgateway-pep-internal
        enabled: true
        k8s:
          hpaSpec:
            maxReplicas: 5
            minReplicas: 1
          resources:
            requests:
              cpu: 200m

  values:
    global:
      istioNamespace: istio-system
      logging:
        level: "default:debug"
2

Did you find what was the root cause? Same problem here with an egressgateway living outside of istio-system namespace

3

I encountered the same, and was able to find a solution so I thought I should share it here.

The ingress and egress gateway deployments loads the configmap istio-<revision> that is optinal, eg:

- configMap:
    defaultMode: 420
    name: istio-canary
    optional: true

This config map only exists in the system namespace (istio-system) and instructs istio-proxy to use a different discovery address. This config map is not available in other namespaces. That’s why gateways running in custom namespaces default to istiod.istio-system.svc. The workaround is to supply discoveryAddress through the PROXY_CONFIG env in the gateway. eg:

- name: PROXY_CONFIG
  value: |
    {"discoveryAddress":"istiod-canary.istio-system.svc:15012","proxyMetadata":{"DNS_AGENT":""}}

This applies to both helm based and istioctl based installations.

1 Like
4

Awesome. Thanks for sharing the workaround.
The problem is also present in latest 1.8.3 release and can be fixed with your fix.