Istio-1-16-1-installation-failed-for-ingress-and-egress-gateway

getting below error:**

[root@itfopay-prod3-master1 wipuxadm]# istioctl install --set profile=demo
This will install the Istio 1.16.1 demo profile with ["Istio core" "Istiod" "Ingress gateways" "Egress gateways"] components into the cluster. Proceed? (y/N) y
✔ Istio core installed
✔ Istiod installed
✘ Egress gateways encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the conditionsystem/istio-ingressgateway
  Deployment/istio-system/istio-egressgateway (containers with unready status: [istio-proxy])
✘ Ingress gateways encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the condition
  Deployment/istio-system/istio-ingressgateway (containers with unready status: [istio-proxy])
- Pruning removed resources

I checked the Ingress logs and getting:

[root@itfopay-prod3-master1 wipuxadm]# k get po -n istio-system
NAME                                    READY   STATUS    RESTARTS   AGE
istio-egressgateway-7475c75b68-qjdv9    0/1     Running   0          10m
istio-ingressgateway-6688c7f65d-9hk2q   0/1     Running   0          10m
istiod-885df7bc9-m5zrb                  1/1     Running   0          10m

Ingress Pod log:

[root@itfopay-prod3-master1 wipuxadm]# k logs -f -n istio-system istio-ingressgateway-6688c7f65d-9hk2q
2023-01-21T09:50:19.722804Z     info    FLAG: --concurrency="0"
2023-01-21T09:50:19.722836Z     info    FLAG: --domain="istio-system.svc.cluster.local"
2023-01-21T09:50:19.722843Z     info    FLAG: --help="false"
2023-01-21T09:50:19.722846Z     info    FLAG: --log_as_json="false"
2023-01-21T09:50:19.722849Z     info    FLAG: --log_caller=""
2023-01-21T09:50:19.722851Z     info    FLAG: --log_output_level="default:info"
2023-01-21T09:50:19.722854Z     info    FLAG: --log_rotate=""
2023-01-21T09:50:19.722857Z     info    FLAG: --log_rotate_max_age="30"
2023-01-21T09:50:19.722860Z     info    FLAG: --log_rotate_max_backups="1000"
2023-01-21T09:50:19.722862Z     info    FLAG: --log_rotate_max_size="104857600"
2023-01-21T09:50:19.722865Z     info    FLAG: --log_stacktrace_level="default:none"
2023-01-21T09:50:19.722871Z     info    FLAG: --log_target="[stdout]"
2023-01-21T09:50:19.722874Z     info    FLAG: --meshConfig="./etc/istio/config/mesh"
2023-01-21T09:50:19.722877Z     info    FLAG: --outlierLogPath=""
2023-01-21T09:50:19.722880Z     info    FLAG: --proxyComponentLogLevel="misc:error"
2023-01-21T09:50:19.722882Z     info    FLAG: --proxyLogLevel="warning"
2023-01-21T09:50:19.722885Z     info    FLAG: --serviceCluster="istio-proxy"
2023-01-21T09:50:19.722889Z     info    FLAG: --stsPort="0"
2023-01-21T09:50:19.722891Z     info    FLAG: --templateFile=""
2023-01-21T09:50:19.722896Z     info    FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2023-01-21T09:50:19.722904Z     info    FLAG: --vklog="0"
2023-01-21T09:50:19.722909Z     info    Version 1.16.1-f6d7bf648e571a6a523210d97bde8b489250354b-Clean
2023-01-21T09:50:19.727010Z     info    Maximum file descriptors (ulimit -n): 1048576
2023-01-21T09:50:19.727162Z     info    Proxy role      ips=[10.244.3.90] type=router id=istio-ingressgateway-6688c7f65d-9hk2q.istio-system domain=istio-system.svc.cluster.local
2023-01-21T09:50:19.727233Z     info    Apply mesh config from file accessLogFile: /dev/stdout
defaultConfig:
  discoveryAddress: istiod.istio-system.svc:15012
  proxyMetadata: {}
  tracing:
    zipkin:
      address: zipkin.istio-system:9411
enablePrometheusMerge: true
extensionProviders:
- envoyOtelAls:
    port: 4317
    service: opentelemetry-collector.istio-system.svc.cluster.local
  name: otel
rootNamespace: istio-system
trustDomain: cluster.local
2023-01-21T09:50:19.732271Z     info    Effective config: binaryPath: /usr/local/bin/envoy
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: istio-proxy
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
  zipkin:
    address: zipkin.istio-system:9411

2023-01-21T09:50:19.732321Z     info    JWT policy is third-party-jwt
2023-01-21T09:50:19.732327Z     info    using credential fetcher of JWT type in cluster.local trust domain
2023-01-21T09:50:21.732984Z     info    Workload SDS socket not found. Starting Istio SDS Server
2023-01-21T09:50:21.733019Z     info    CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2023-01-21T09:50:21.733009Z     info    Opening status port 15020
2023-01-21T09:50:21.733066Z     info    Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2023-01-21T09:50:21.733190Z     info    citadelclient   Citadel client using custom root cert: var/run/secrets/istio/root-cert.pem
2023-01-21T09:50:21.752589Z     info    ads     All caches have been synced up in 2.033712613s, marking server ready
2023-01-21T09:50:21.753009Z     info    xdsproxy        Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2023-01-21T09:50:21.753526Z     info    sds     Starting SDS grpc server
2023-01-21T09:50:21.755731Z     info    starting Http service at 127.0.0.1:15004
2023-01-21T09:50:21.757373Z     info    Pilot SAN: [istiod.istio-system.svc]
2023-01-21T09:50:21.759150Z     info    Starting proxy agent
2023-01-21T09:50:21.759203Z     info    starting
2023-01-21T09:50:21.759262Z     info    Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --parent-shutdown-time-s 60 --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --log-format %Y-%m-%dT%T.%fZ  %l      envoy %n        %v -l warning --component-log-level misc:error]

2023-01-21T09:57:35.899133Z     warn    sds     failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.90:47480->10.96.0.10:53: i/o timeout"
2023-01-21T09:57:47.742912Z     warning envoy config    StreamAggregatedResources gRPC config stream to xds-grpc closed since 425s ago: 14, connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2023-01-21T09:58:24.903150Z     warning envoy config    StreamAggregatedResources gRPC config stream to xds-grpc closed since 463s ago: 14, connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2023-01-21T09:58:54.914879Z     warn    ca      ca request failed, starting attempt 1 in 109.434019ms
2023-01-21T09:58:54.915166Z     warning envoy config    StreamAggregatedResources gRPC config stream to xds-grpc closed since 493s ago: 14, connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.90:32922->10.96.0.10:53: i/o timeout"
2023-01-21T09:58:55.025581Z     warn    ca      ca request failed, starting attempt 2 in 196.977492ms
2023-01-21T09:58:55.223194Z     warn    ca      ca request failed, starting attempt 3 in 395.689636ms
2023-01-21T09:58:55.620020Z     warn    ca      ca request failed, starting attempt 4 in 739.027151ms
2023-01-21T09:58:56.360201Z     warn    sds     failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.90:32922->10.96.0.10:53: i/o timeout"
2023-01-21T09:59:44.014538Z     warning envoy config    StreamAggregatedResources gRPC config stream to xds-grpc closed since 542s ago: 14, connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2023-01-21T10:00:15.658480Z     warn    ca      ca request failed, starting attempt 1 in 108.563067ms
2023-01-21T10:00:15.768101Z     warn    ca      ca request failed, starting attempt 2 in 213.415014ms
2023-01-21T10:00:15.982769Z     warn    ca      ca request failed, starting attempt 3 in 390.712645ms
2023-01-21T10:00:16.374599Z     warn    ca      ca request failed, starting attempt 4 in 807.591954ms
2023-01-21T10:00:17.182782Z     warn    sds     failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.90:32922->10.96.0.10:53: i/o timeout"
2023-01-21T10:00:19.121357Z     warning envoy config    StreamAggregatedResources gRPC config stream to xds-grpc closed since 577s ago: 14, connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2023-01-21T10:00:51.189645Z     warning envoy config    StreamAggregatedResources gRPC config stream to xds-grpc closed since 609s ago: 14, connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2023-01-21T10:01:37.903048Z     warning envoy config    StreamAggregatedResources gRPC config stream to xds-grpc closed since 656s ago: 14, connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc: i/o timeout"

I am also currently getting same issue for 1.11.5…

I am also facing the same issue with v1.15.6, looks like its due to support for new arm arch.
Not sure how we can pass amd64 to istio/proxyv2:1.15.6 docker image.

See logs of my failed pod:
kc -n istio-system logs istio-egressgateway-ssss.xxxx
exec /usr/local/bin/pilot-agent: exec format error