Hi,
I had running cluster with kubernetes 1.18 and Istio 1.8.2. I upgraded the cluster to 1.19 and Istio to 1.9.2.
I have installed Istio using the operator and did a in-place upgrade. The control plane seems to have upgraded successfully - Istiod is showing healthy and the status of istiooperator is HEALTHY as well. But as the application pods are restarted, none of the proxies comes up healthy. I see the following error in the proxy logs
2021-06-17T18:30:12.737997Z critical envoy main error initializing configuration 'etc/istio/proxy/envoy-rev0.json': The v2 xDS major version is deprecated and disabled by default. Support for v2 will be removed from Envoy at the start of Q1 2021. You may make use of v2 in Q4 2020 by following the advice in https://www.envoyproxy.io/docs/envoy/latest/faq/api/transition. (envoy.config.trace.v2.DynamicOtConfig)
The v2 xDS major version is deprecated and disabled by default. Support for v2 will be removed from Envoy at the start of Q1 2021. You may make use of v2 in Q4 2020 by following the advice in https://www.envoyproxy.io/docs/envoy/latest/faq/api/transition. (envoy.config.trace.v2.DynamicOtConfig)
2021-06-17T18:30:12.741052Z error Epoch 0 exited with error: exit status 1
I am aware of the deprecation notice from Istio release notes, but I am not using any custom EnvoyFilters in the cluster.
k get Envoyfilter -A
NAMESPACE NAME AGE
adp-istio-system metadata-exchange-1.8 7h57m
adp-istio-system metadata-exchange-1.9 8h
adp-istio-system stats-filter-1.8 7h57m
adp-istio-system stats-filter-1.9 8h
adp-istio-system tcp-metadata-exchange-1.8 7h57m
adp-istio-system tcp-metadata-exchange-1.9 8h
adp-istio-system tcp-stats-filter-1.8 7h57m
adp-istio-system tcp-stats-filter-1.9 8h
I also took a dump of “/etc/istio/proxy/envoy-rev0.json” from the proxy and check it for v2 filters, but didn’t find any.
kubectl exec istio-ingressgateway-65bc44fb98-p8s4z -- cat /etc/istio/proxy/envoy-rev0.json > envoy-rev0.json
grep "@type" envoy-rev0.json
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
"@type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
I can’t find where the envoy is finding v2 xDS, any pointers are appreciated.
PS: Our system is fully automated, so the upgrades I mentioned here is done in a Jenkins pipeline. And we have run multiple upgrades successfully. This is the first failure, so the bug is probably not consistent.