Hello All,
I am keep on trying to install the Istio on a K3-cluster. But I am failing in doing it due to the below errors. Where as I was successfully implemented it on a GKE cluster many times.
Posting only the Events of the two pods (prometheus & ingressgaetway):
Prometheus Pod:
Events:
Type Reason Age From Message
Normal Scheduled default-scheduler Successfully assigned istio-system/prometheus-5c4c88f684-4gxrx to ser05dvvm490
Warning FailedMount 41m kubelet, ser05dvvm490 MountVolume.SetUp failed for volume “prometheus-token-x9bwv” : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 41m kubelet, ser05dvvm490 MountVolume.SetUp failed for volume “config-volume” : failed to sync configmap cache: timed out waiting for the condition
Normal Started 41m (x2 over 41m) kubelet, ser05dvvm490 Started container istio-proxy
Normal Pulled 41m (x3 over 41m) kubelet, ser05dvvm490 Container image “docker.io/prom/prometheus:v2.15.1” already present on machine
Normal Created 41m (x3 over 41m) kubelet, ser05dvvm490 Created container prometheus
Normal Started 41m (x3 over 41m) kubelet, ser05dvvm490 Started container prometheus
Normal Pulled 41m (x3 over 41m) kubelet, ser05dvvm490 Container image “docker.io/istio/proxyv2:1.5.0” already present on machine
Normal Created 41m (x3 over 41m) kubelet, ser05dvvm490 Created container istio-proxy
Warning BackOff 26m (x71 over 41m) kubelet, ser05dvvm490 Back-off restarting failed container
Warning BackOff 86s (x207 over 41m) kubelet, ser05dvvm490 Back-off restarting failed container
Istio-ingressgateway Pod:
Events:
Type Reason Age From Message
Normal Scheduled default-scheduler Successfully assigned istio-system/istio-ingressgateway-cb45b475f-2lqqz to ser05dvvm490
Warning FailedMount 46m kubelet, ser05dvvm490 MountVolume.SetUp failed for volume “istiod-ca-cert” : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 46m kubelet, ser05dvvm490 MountVolume.SetUp failed for volume “ingressgateway-ca-certs” : failed to sync secret cache: timed out waiting for the condition
Normal Pulled 45m (x5 over 46m) kubelet, ser05dvvm490 Container image “docker.io/istio/proxyv2:1.5.0” already present on machine
Normal Created 45m (x5 over 46m) kubelet, ser05dvvm490 Created container istio-proxy
Normal Started 45m (x5 over 46m) kubelet, ser05dvvm490 Started container istio-proxy
Warning BackOff 109s (x207 over 46m) kubelet, ser05dvvm490 Back-off restarting failed container
Please help me to resolve this issue.
Thanks,
Anil
Hello Team,
The Istio pods status is shown below in addition to my above issue:
[admin@… ] $ kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istio-init-crd-mixer-1.5.4-ftb9m 0/1 Completed 0 3h9m
istio-init-crd-all-1.5.4-2dn8q 0/1 Completed 0 3h9m
istio-tracing-dd8696d45-rd79m 1/1 Running 0 87m
istiod-5fd8499848-x4np9 1/1 Running 0 87m
grafana-7c9d9d488c-r9j9k 1/1 Running 0 87m
kiali-5d5c78d58d-wdt2t 1/1 Running 0 87m
istio-ingressgateway-cb45b475f-2lqqz 0/1 CrashLoopBackOff 21 87m
prometheus-5c4c88f684-4gxrx 1/2 CrashLoopBackOff 43 87m
Hi All,
Adding to the above, I am attaching the logs below:
kubectl logs prometheus-5c4c88f684-h5p9f -n istio-system
error: a container name must be specified for pod prometheus-5c4c88f684-h5p9f, choose one of: [prometheus istio-proxy]
[admin@SER05DVVM490 istio]$ kubectl logs prometheus-5c4c88f684-h5p9f -n istio-system -c prometheus
level=warn ts=2020-05-21T12:26:59.085Z caller=main.go:283 deprecation_notice="‘storage.tsdb.retention’ flag is deprecated use ‘storage.tsdb.retention.time’ instead."
level=info ts=2020-05-21T12:26:59.085Z caller=main.go:330 msg=“Starting Prometheus” version="(version=2.15.1, branch=HEAD, revision=8744510c6391d3ef46d8294a7e1f46e57407ab13)"
level=info ts=2020-05-21T12:26:59.085Z caller=main.go:331 build_context="(go=go1.13.5, user=root@4b1e33c71b9d, date=20191225-01:04:15)"
level=info ts=2020-05-21T12:26:59.085Z caller=main.go:332 host_details="(Linux 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 13:49:54 UTC 2020 x86_64 prometheus-5c4c88f684-h5p9f (none))"
level=info ts=2020-05-21T12:26:59.085Z caller=main.go:333 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2020-05-21T12:26:59.085Z caller=main.go:334 vm_limits="(soft=unlimited, hard=unlimited)"
level=error ts=2020-05-21T12:26:59.085Z caller=query_logger.go:107 component=activeQueryTracker msg=“Failed to create directory for logging active queries”
level=error ts=2020-05-21T12:26:59.085Z caller=query_logger.go:85 component=activeQueryTracker msg=“Error opening query log file” file=data/queries.active err=“open data/queries.active: no such file or directory”
panic: Unable to create mmap-ed active query log
goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x24dda5b, 0x5, 0x14, 0x2c62100, 0xc000537740, 0x2c62100)
/app/promql/query_logger.go:115 +0x48c
main.main()
/app/cmd/prometheus/main.go:362 +0x5229
Istio-ingress gateway pod logs
2020-05-21T12:32:15.061769Z info parsed scheme: “”
2020-05-21T12:32:15.061782Z info scheme “” not registered, fallback to default scheme
2020-05-21T12:32:15.061802Z info ccResolverWrapper: sending update to cc: {[{istio-pilot.istio-system.svc:15012 0 }] }
2020-05-21T12:32:15.061813Z info ClientConn switching balancer to “pick_first”
2020-05-21T12:32:15.061992Z info pickfirstBalancer: HandleSubConnStateChange: 0xc00033da80, {CONNECTING }
2020-05-21T12:32:15.071243Z info pickfirstBalancer: HandleSubConnStateChange: 0xc00033da80, {READY }
2020-05-21T12:32:15.105421Z info Starting gateway SDS
2020-05-21T12:32:15.121099Z warn secretfetcher failed load server cert/key pair from secret kiali: server cert or private key is empty
2020-05-21T12:32:15.208867Z info sds SDS gRPC server for workload UDS starts, listening on “/etc/istio/proxy/SDS”
2020-05-21T12:32:15.209013Z info sds Start SDS grpc server
2020-05-21T12:32:15.209093Z info sds SDS gRPC server for ingress gateway controller starts, listening on “/var/run/ingress_gateway/sds”
2020-05-21T12:32:15.209123Z info PilotSAN string{“istio-pilot.istio-system.svc”}
2020-05-21T12:32:15.209137Z info Starting proxy agent
2020-05-21T12:32:15.209136Z info sds Start SDS grpc server for ingress gateway proxy
2020-05-21T12:32:15.209171Z info Opening status port 15020
2020-05-21T12:32:15.209177Z info Received new config, creating new Envoy epoch 0
2020-05-21T12:32:15.209381Z info Epoch 0 starting
2020-05-21T12:32:16.246437Z info Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~10.42.0.11~istio-ingressgateway-cb45b475f-h6xtk.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --log-format [Envoy (Epoch 0)] [%Y-%m-%d %T.%e][%t][%l][%n] %v -l warning --component-log-level misc:error]
[2020-05-21 12:32:16.278][23][critical][assert] [external/envoy/source/server/hot_restart_impl.cc:45] panic: cannot open shared memory region /envoy_shared_memory_0 check user permissions. Error: Permission denied
[2020-05-21 12:32:16.278][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:83] Caught Aborted, suspect faulting address 0x17
[2020-05-21 12:32:16.278][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:70] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-05-21 12:32:16.278][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:71] Envoy version: 73f240a29bece92a8882a36893ccce07b4a54664/1.13.1-dev/Clean/RELEASE/BoringSSL
[2020-05-21 12:32:16.278][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #0: __restore_rt [0x7f397e5b6890]
[2020-05-21 12:32:16.300][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #1: Envoy::Server::HotRestartImpl::HotRestartImpl() [0x55e337281018]
[2020-05-21 12:32:16.321][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #2: Envoy::MainCommonBase::MainCommonBase() [0x55e335f7a700]
[2020-05-21 12:32:16.339][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #3: Envoy::MainCommon::MainCommon() [0x55e335f7b0da]
[2020-05-21 12:32:16.358][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #4: main [0x55e335f79d63]
[2020-05-21 12:32:16.358][23][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #5: __libc_start_main [0x7f397e1d4b97]
2020-05-21T12:32:16.934540Z error Epoch 0 exited with error: signal: aborted (core dumped)
2020-05-21T12:32:16.934958Z info No more active epochs, terminating
Hi All,
Forgot to update it. The issue is resolved.
Initially I thought the error is due to Istio configuration settings in the values.yaml and digging a deep dive into Istio. It also helped me to understand more towards Istio service mesh.
But the issue is at the OS level, as I am using a CentOS VM on which I installed K3s-Cluster.
So, here we need to change the mode of selinux from “Enforcing” to “Permissive” and then the issue is resolved and all the Pods are up and running fine.
One more important information, as I am referring to the Istio Platform Setup page, there is no K3s-Cluster. Please have a look into it and kindly add K3s-Cluster as well.
Hopefully, I am believing that the Istio developers will add the K3s as well into the Istio platform setup page on how to deploy Istio into it.
K3s is now the CNCF approved one.
https://landscape.cncf.io/selected=k3s
Best Regards,
Anil Kumar Koduri
kumar.k.anil@gmail.com