I actually attempted a Canary on an existing cluster running 1.4.9 with version 1.6.10. I basically rewrote the helm config into an istio-operator config using an identical configuration for the ingress-gateways etc. I then deployed the canary, which replaces the gateways with the 1.6.10 gateway and deploys the canary of istio-d. After that I relabeled all of my namespaces with the canary revision tag and restarted all of my pods. Everything seemed great and tested ok so I manually removed all of my 1.4.9 assets and the helm secrets in the namespace. Everything still seemed great, until a deployment the next day pushed out new virtual-service config the following day, then I lost all ingress to one of my gateways. I realized my gateways didn’t have SDS in the 1.6.10 gateways and I couldn’t find anyway to mirror that configuration in 1.6, so I ended up pulling everything and rolling back and have been stuck on 1.4.9 since. This is the error my gateway that basically was unable to talk to any services under it.
|2020-09-28T17:44:25.996687Z|warning|envoy config|[bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamSecrets gRPC config stream closed: 16, request authenticate failure|
|---|---|---|---|
|2020-09-28T17:44:28.115769Z|info|sds|resource:default new connection|
|2020-09-28T17:44:28.115925Z|info|sds|Skipping waiting for ingress gateway secret|
|2020-09-28T17:44:28.426937Z|error|citadelclient|Failed to create certificate: rpc error: code = Unauthenticated desc = request authenticate failure|
|2020-09-28T17:44:28.426986Z|error|cache|resource:default request:4efd50bb-5705-44a2-81e0-4bd71ce0cd13 CSR hit non-retryable error (HTTP code: 0). Error: rpc error: code = Unauthenticated desc = request authenticate failure|
|2020-09-28T17:44:28.427020Z|error|cache|resource:default failed to generate secret for proxy: rpc error: code = Unauthenticated desc = request authenticate failure|
|2020-09-28T17:44:28.427040Z|error|sds|resource:default Close connection. Failed to get secret for proxy "router~172.16.10.235~istio-internal-ingressgateway-5b69f8cb69-xvmd8.istio-system~istio-system.svc.cluster.local" from secret cache: rpc error: code = Unauthenticated desc = request authenticate failure|
|2020-09-28T17:44:28.427155Z|info|sds|resource:default connection is terminated: rpc error: code = Canceled desc = context canceled|
|2020-09-28T17:44:28.427409Z|warning|envoy config|[bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamSecrets gRPC config stream closed: 16, request authenticate failure|