Istio Upgrade Error 1.14.1

1

Hi
When I upgrade Istio from 1.11.0 to 1.14.1
Istio operator shows status as “RECONCILING”
Istio operator log is looping into the below error

Can some body guide me how we can fix the above issues (details are given below)

  • NextGen-ADA-Istio % ~/Downloads/istio-1.14.1/bin/istioctl version
    client version: 1.14.1
    control plane version: 1.14.1
    data plane version: 1.14.1 (16 proxies)

  • Istio operator shows status as “RECONCILING”
    NextGen-ADA-Istio % k get istiooperators.install.istio.io -A
    NAMESPACE NAME REVISION STATUS AGE
    istio-system ada-istiocontrolplane RECONCILING 141m

  • istioctl analyze is giving validation succeded
    NextGen-ADA-Istio % ~/Downloads/istio-1.14.1/bin/istioctl analyze
    :heavy_check_mark: No validation issues found when analyzing namespace: istio-operator.

  • Istio operator log is looping into the below error

      2022-08-31T20:55:19.880875Z	info	kube	Starting Pilot K8S CRD controller
  2022-08-31T20:55:19.880983Z	info	kube	Pilot K8S CRD controller synced 119µs
  2022-08-31T20:55:19.989339Z	info	installer	Processing resources from manifest: IstiodRemote for CR ada-istiocontrolplane-istio-system-IstiodRemote-https://172.16.0.1:443
  2022-08-31T20:55:19.989367Z	info	installer	EgressGateways is waiting on dependency...
  2022-08-31T20:55:19.989385Z	info	installer	Generated manifest objects are the same as cached for component IstiodRemote.
  2022-08-31T20:55:19.989372Z	info	installer	IngressGateways is waiting on dependency...
  2022-08-31T20:55:19.989397Z	info	installer	Cni is waiting on dependency...
  2022-08-31T20:55:19.989344Z	info	installer	Processing resources from manifest: Base for CR ada-istiocontrolplane-istio-system-Base-https://172.16.0.1:443
  2022-08-31T20:55:19.989395Z	info	installer	Pilot is waiting on dependency...
  2022-08-31T20:55:20.581132Z	info	installer	Generated manifest objects are the same as cached for component Base.
  2022-08-31T20:55:20.581190Z	info	installer	Unblocking dependency Pilot.
  2022-08-31T20:55:20.581211Z	info	installer	Dependency for Pilot has completed, proceeding.
  2022-08-31T20:55:20.581242Z	info	installer	Processing resources from manifest: Pilot for CR ada-istiocontrolplane-istio-system-Pilot-https://172.16.0.1:443
  2022-08-31T20:55:20.774697Z	info	installer	Generated manifest objects are the same as cached for component Pilot.
  2022-08-31T20:55:20.774769Z	info	installer	Unblocking dependency Cni.
  2022-08-31T20:55:20.774804Z	info	installer	Unblocking dependency IngressGateways.
  2022-08-31T20:55:20.774809Z	info	installer	Unblocking dependency EgressGateways.
  2022-08-31T20:55:20.774843Z	info	installer	Dependency for EgressGateways has completed, proceeding.
  2022-08-31T20:55:20.774873Z	info	installer	Processing resources from manifest: EgressGateways for CR ada-istiocontrolplane-istio-system-EgressGateways-https://172.16.0.1:443
  2022-08-31T20:55:20.774886Z	info	installer	Generated manifest objects are the same as cached for component EgressGateways.
  2022-08-31T20:55:20.774894Z	info	installer	Dependency for IngressGateways has completed, proceeding.
  2022-08-31T20:55:20.774910Z	info	installer	Processing resources from manifest: IngressGateways for CR ada-istiocontrolplane-istio-system-IngressGateways-https://172.16.0.1:443
  2022-08-31T20:55:20.774901Z	info	installer	Dependency for Cni has completed, proceeding.
  2022-08-31T20:55:20.774964Z	info	installer	Processing resources from manifest: Cni for CR ada-istiocontrolplane-istio-system-Cni-https://172.16.0.1:443
  2022-08-31T20:55:20.774985Z	info	installer	Generated manifest objects are the same as cached for component Cni.
  2022-08-31T20:55:20.781076Z	info	installer	Generated manifest objects are the same as cached for component IngressGateways.
  - Pruning removed resources
  2022-08-31T20:55:52.087921Z	info	installer	Reconciling IstioOperator
  2022-08-31T20:55:52.467235Z	info	installer	Updating IstioOperator
  2022-08-31T20:55:52.467386Z	info	installer	Detecting third-party JWT support
  2022-08-31T20:55:53.677880Z	info	installer	Applying Kubernetes overlay:
  - kind: PodDisruptionBudget
    name: istiod
    patches:
    - path: spec.minAvailable
      value: 1
  
  
  2022-08-31T20:55:54.072617Z	info	installer	Applying Kubernetes overlay:
  - kind: PodDisruptionBudget
    name: istio-ingressgateway
    patches:
    - path: spec.minAvailable
      value: 3
  
  
  2022-08-31T20:55:54.479135Z	error	analysis	error setting up error handling for kube crdclient: 2 errors occurred:
        	* informer has already started
        	* informer has already started
2

We are also seeing this error on AKS when using Istio 1.14.3 and the operator-based deployment approach. The operator is OOM killed every 5-9 days and the IstioOperator CR is in a “Reconciling” state.

The error is also present in version 1.13.2, but here the operator is able to reconcile the CR and is not OOM killed.

We have checked our configuration for obvious mistakes, but the errors are still present even with a “clean” default profile CR.

3

So is there any way to fix this , and can we push this to production with this error if we are not seeing any other impacts