Istio-1.5.4 installation on K3 cluster (single Node) --- MountVolume.SetUp failed for volume in Prometheus & istio-ingressgateway Pods

Hello All,

I am keep on trying to install the Istio on a K3-cluster. But I am failing in doing it due to the below errors. Where as I was successfully implemented it on a GKE cluster many times.
Posting only the Events of the two pods (prometheus & ingressgaetway):

Prometheus Pod:

Events:
Type Reason Age From Message


Normal Scheduled default-scheduler Successfully assigned istio-system/prometheus-5c4c88f684-4gxrx to ser05dvvm490
Warning FailedMount 41m kubelet, ser05dvvm490 MountVolume.SetUp failed for volume “prometheus-token-x9bwv” : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 41m kubelet, ser05dvvm490 MountVolume.SetUp failed for volume “config-volume” : failed to sync configmap cache: timed out waiting for the condition
Normal Started 41m (x2 over 41m) kubelet, ser05dvvm490 Started container istio-proxy
Normal Pulled 41m (x3 over 41m) kubelet, ser05dvvm490 Container image “docker.io/prom/prometheus:v2.15.1” already present on machine
Normal Created 41m (x3 over 41m) kubelet, ser05dvvm490 Created container prometheus
Normal Started 41m (x3 over 41m) kubelet, ser05dvvm490 Started container prometheus
Normal Pulled 41m (x3 over 41m) kubelet, ser05dvvm490 Container image “docker.io/istio/proxyv2:1.5.0” already present on machine
Normal Created 41m (x3 over 41m) kubelet, ser05dvvm490 Created container istio-proxy
Warning BackOff 26m (x71 over 41m) kubelet, ser05dvvm490 Back-off restarting failed container
Warning BackOff 86s (x207 over 41m) kubelet, ser05dvvm490 Back-off restarting failed container

Istio-ingressgateway Pod:

Events:
Type Reason Age From Message


Normal Scheduled default-scheduler Successfully assigned istio-system/istio-ingressgateway-cb45b475f-2lqqz to ser05dvvm490
Warning FailedMount 46m kubelet, ser05dvvm490 MountVolume.SetUp failed for volume “istiod-ca-cert” : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 46m kubelet, ser05dvvm490 MountVolume.SetUp failed for volume “ingressgateway-ca-certs” : failed to sync secret cache: timed out waiting for the condition
Normal Pulled 45m (x5 over 46m) kubelet, ser05dvvm490 Container image “docker.io/istio/proxyv2:1.5.0” already present on machine
Normal Created 45m (x5 over 46m) kubelet, ser05dvvm490 Created container istio-proxy
Normal Started 45m (x5 over 46m) kubelet, ser05dvvm490 Started container istio-proxy
Warning BackOff 109s (x207 over 46m) kubelet, ser05dvvm490 Back-off restarting failed container

Please help me to resolve this issue.

Thanks,
Anil

Hello Team,

The Istio pods status is shown below in addition to my above issue:
[admin@… ] $ kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istio-init-crd-mixer-1.5.4-ftb9m 0/1 Completed 0 3h9m
istio-init-crd-all-1.5.4-2dn8q 0/1 Completed 0 3h9m
istio-tracing-dd8696d45-rd79m 1/1 Running 0 87m
istiod-5fd8499848-x4np9 1/1 Running 0 87m
grafana-7c9d9d488c-r9j9k 1/1 Running 0 87m
kiali-5d5c78d58d-wdt2t 1/1 Running 0 87m
istio-ingressgateway-cb45b475f-2lqqz 0/1 CrashLoopBackOff 21 87m
prometheus-5c4c88f684-4gxrx 1/2 CrashLoopBackOff 43 87m

Hi All,

Adding to the above, I am attaching the logs below:

kubectl logs prometheus-5c4c88f684-h5p9f -n istio-system

error: a container name must be specified for pod prometheus-5c4c88f684-h5p9f, choose one of: [prometheus istio-proxy]
[admin@SER05DVVM490 istio]$ kubectl logs prometheus-5c4c88f684-h5p9f -n istio-system -c prometheus
level=warn ts=2020-05-21T12:26:59.085Z caller=main.go:283 deprecation_notice="‘storage.tsdb.retention’ flag is deprecated use ‘storage.tsdb.retention.time’ instead."
level=info ts=2020-05-21T12:26:59.085Z caller=main.go:330 msg=“Starting Prometheus” version="(version=2.15.1, branch=HEAD, revision=8744510c6391d3ef46d8294a7e1f46e57407ab13)"
level=info ts=2020-05-21T12:26:59.085Z caller=main.go:331 build_context="(go=go1.13.5, user=root@4b1e33c71b9d, date=20191225-01:04:15)"
level=info ts=2020-05-21T12:26:59.085Z caller=main.go:332 host_details="(Linux 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 13:49:54 UTC 2020 x86_64 prometheus-5c4c88f684-h5p9f (none))"
level=info ts=2020-05-21T12:26:59.085Z caller=main.go:333 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2020-05-21T12:26:59.085Z caller=main.go:334 vm_limits="(soft=unlimited, hard=unlimited)"
level=error ts=2020-05-21T12:26:59.085Z caller=query_logger.go:107 component=activeQueryTracker msg=“Failed to create directory for logging active queries”
level=error ts=2020-05-21T12:26:59.085Z caller=query_logger.go:85 component=activeQueryTracker msg=“Error opening query log file” file=data/queries.active err=“open data/queries.active: no such file or directory”
panic: Unable to create mmap-ed active query log

goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x24dda5b, 0x5, 0x14, 0x2c62100, 0xc000537740, 0x2c62100)
/app/promql/query_logger.go:115 +0x48c
main.main()
/app/cmd/prometheus/main.go:362 +0x5229

Istio-ingress gateway pod logs

2020-05-21T12:32:15.061769Z info parsed scheme: “”
2020-05-21T12:32:15.061782Z info scheme “” not registered, fallback to default scheme
2020-05-21T12:32:15.061802Z info ccResolverWrapper: sending update to cc: {[{istio-pilot.istio-system.svc:15012 0 }] }
2020-05-21T12:32:15.061813Z info ClientConn switching balancer to “pick_first”
2020-05-21T12:32:15.061992Z info pickfirstBalancer: HandleSubConnStateChange: 0xc00033da80, {CONNECTING }
2020-05-21T12:32:15.071243Z info pickfirstBalancer: HandleSubConnStateChange: 0xc00033da80, {READY }
2020-05-21T12:32:15.105421Z info Starting gateway SDS
2020-05-21T12:32:15.121099Z warn secretfetcher failed load server cert/key pair from secret kiali: server cert or private key is empty
2020-05-21T12:32:15.208867Z info sds SDS gRPC server for workload UDS starts, listening on “/etc/istio/proxy/SDS”

2020-05-21T12:32:15.209013Z info sds Start SDS grpc server
2020-05-21T12:32:15.209093Z info sds SDS gRPC server for ingress gateway controller starts, listening on “/var/run/ingress_gateway/sds”

2020-05-21T12:32:15.209123Z info PilotSAN string{“istio-pilot.istio-system.svc”}
2020-05-21T12:32:15.209137Z info Starting proxy agent
2020-05-21T12:32:15.209136Z info sds Start SDS grpc server for ingress gateway proxy
2020-05-21T12:32:15.209171Z info Opening status port 15020

2020-05-21T12:32:15.209177Z info Received new config, creating new Envoy epoch 0
2020-05-21T12:32:15.209381Z info Epoch 0 starting
2020-05-21T12:32:16.246437Z info Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~10.42.0.11~istio-ingressgateway-cb45b475f-h6xtk.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --log-format [Envoy (Epoch 0)] [%Y-%m-%d %T.%e][%t][%l][%n] %v -l warning --component-log-level misc:error]
[2020-05-21 12:32:16.278][23][critical][assert] [external/envoy/source/server/hot_restart_impl.cc:45] panic: cannot open shared memory region /envoy_shared_memory_0 check user permissions. Error: Permission denied
[2020-05-21 12:32:16.278][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:83] Caught Aborted, suspect faulting address 0x17
[2020-05-21 12:32:16.278][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:70] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-05-21 12:32:16.278][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:71] Envoy version: 73f240a29bece92a8882a36893ccce07b4a54664/1.13.1-dev/Clean/RELEASE/BoringSSL
[2020-05-21 12:32:16.278][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #0: __restore_rt [0x7f397e5b6890]
[2020-05-21 12:32:16.300][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #1: Envoy::Server::HotRestartImpl::HotRestartImpl() [0x55e337281018]
[2020-05-21 12:32:16.321][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #2: Envoy::MainCommonBase::MainCommonBase() [0x55e335f7a700]
[2020-05-21 12:32:16.339][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #3: Envoy::MainCommon::MainCommon() [0x55e335f7b0da]
[2020-05-21 12:32:16.358][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #4: main [0x55e335f79d63]
[2020-05-21 12:32:16.358][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #5: __libc_start_main [0x7f397e1d4b97]
2020-05-21T12:32:16.934540Z error Epoch 0 exited with error: signal: aborted (core dumped)
2020-05-21T12:32:16.934958Z info No more active epochs, terminating