(I am running on bare-metal/VM Kubernetes v1.15)
Deploying Istio with the standard helm charts, all services work correctly + are successfully deployed in the kubernetes dashboard, with proxyv2 sidecars on the mixer and pilot pods in istio system.
When deploying a business application to another namespace (labelled correctly with istio-injection: enabled), the pods within this namespace successfully deploy, but without the sidecar.
However, the sidecar-injector is labelling each pod with the annotation:
“sidecar.istio.io/status: {“version”:“e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855”,“initContainers”:null,“containers”:null,“volumes”:null,“imagePullSecrets”:null}”.
This suggests to me that the mutatingwebhook is successfully identifying the label and attempting to inject the sidecar into the pod, however it has not got the correct configuration stored (expect non-null values in annotation?)
kubectl describe mutatingwebhookconfiguration istio-sidecar-injector
Name: istio-sidecar-injector
Namespace:
Labels: app=sidecarInjectorWebhook
chart=sidecarInjectorWebhook
heritage=Tiller
release=istio
Annotations:
API Version: admissionregistration.k8s.io/v1beta1
Kind: MutatingWebhookConfiguration
Metadata:
Creation Timestamp: 2019-09-04T08:46:19Z
Generation: 2
Resource Version: 1236807
Self Link: /apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations/istio-sidecar-injector
UID: bdf9cd18-bd29-4643-b1b0-5623bdbcd965
Webhooks:
Admission Review Versions:
v1beta1
Client Config:
Ca Bundle:
Service:
Name: istio-sidecar-injector
Namespace: istio-system
Path: /inject
Port: 443
Failure Policy: Fail
Match Policy: Exact
Name: sidecar-injector.istio.io
Namespace Selector:
Match Labels:
Istio - Injection: enabled
Object Selector:
Reinvocation Policy: Never
Rules:
API Groups:
API Versions:
v1
Operations:
CREATE
Resources:
pods
Scope: *
Side Effects: Unknown
Timeout Seconds: 30
Events:
kubectl describe pod business-app-microservice-1 -n myspace
Name: business-app-microservice-1
Namespace: myspace
Priority: 0
PriorityClassName:
Node: node-11/
Start Time: Wed, 04 Sep 2019 10:07:51 +0100
Labels: app=business-app-microservice
pod-template-hash=84d6744b5d
Annotations: sidecar.istio.io/status:
{“version”:“e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855”,“initContainers”:null,“containers”:null,“volumes”:null,"imag…
Status: Running
IP:
Controlled By: ReplicaSet/business-app-microservice
Containers:
aor-gateway:
Container ID:
Image: <privaterepo/image>
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
State: Running
Started: Wed, 04 Sep 2019 10:08:12 +0100
Ready: True
Restart Count: 0
Limits:
memory: 500Mi
Requests:
memory: 500Mi
Liveness: http-get http://:8000/actuator/health delay=60s timeout=30s period=30s #success=1 #failure=3
Readiness: http-get http://:8000/actuator/health delay=30s timeout=30s period=30s #success=1 #failure=10
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from helm-release-token-k4g58 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
helm-release-token-k4g58:
Type: Secret (a volume populated by a Secret)
SecretName: helm-release-token-k4g58
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
Normal Scheduled 27m default-scheduler Successfully assigned myspace/business-app to node-11
Normal Pulling 27m kubelet, node-11 Pulling image “business-image”
Normal Pulled 27m kubelet, node-11 Successfully pulled image “business-image”
Normal Created 27m kubelet, node-11 Created container business-app
Normal Started 27m kubelet, node-11 Started container business-app
Would expect to see additional containers for istio-init / istioproxyv2 in the pod.
kubectl describe pod istio-sidecar-injector -n istio-system
Name: istio-sidecar-injector
Namespace: istio-system
Priority: 0
PriorityClassName:
Node: node-11/
Start Time: Wed, 04 Sep 2019 09:46:20 +0100
Labels: app=sidecarInjectorWebhook
chart=sidecarInjectorWebhook
heritage=Tiller
istio=sidecar-injector
pod-template-hash=79b6dfd884
release=istio
Annotations: sidecar.istio.io/inject: false
Status: Running
IP:
Controlled By: ReplicaSet/istio-sidecar-injector
Containers:
sidecar-injector-webhook:
Container ID: hidden
Image: hidden
Image ID: hidden
Port:
Host Port:
Args:
–caCertFile=/etc/istio/certs/root-cert.pem
–tlsCertFile=/etc/istio/certs/cert-chain.pem
–tlsKeyFile=/etc/istio/certs/key.pem
–injectConfig=/etc/istio/inject/config
–meshConfig=/etc/istio/config/mesh
–healthCheckInterval=2s
–healthCheckFile=/health
State: Running
Started: Wed, 04 Sep 2019 09:46:29 +0100
Ready: True
Restart Count: 0
Requests:
cpu: 10m
Liveness: exec [/usr/local/bin/sidecar-injector probe --probe-path=/health --interval=4s] delay=4s timeout=1s period=4s #success=1 #failure=3
Readiness: exec [/usr/local/bin/sidecar-injector probe --probe-path=/health --interval=4s] delay=4s timeout=1s period=4s #success=1 #failure=3
Environment:
Mounts:
/etc/istio/certs from certs (ro)
/etc/istio/config from config-volume (ro)
/etc/istio/inject from inject-config (ro)
/var/run/secrets/kubernetes.io/serviceaccount from istio-sidecar-injector-service-account-token-79h8k (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: istio
Optional: false
certs:
Type: Secret (a volume populated by a Secret)
SecretName: istio.istio-sidecar-injector-service-account
Optional: false
inject-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: istio-sidecar-injector
Optional: false
istio-sidecar-injector-service-account-token:
Type: Secret (a volume populated by a Secret)
SecretName: istio-sidecar-injector-service-account-token
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
Normal Scheduled 60m default-scheduler Successfully assigned istio-system/istio-sidecar-injector to node-11
Warning FailedMount 60m (x4 over 60m) kubelet, node-11 MountVolume.SetUp failed for volume “certs” : secret “istio.istio-sidecar-injector-service-account” not found
Normal Pulled 60m kubelet, node-11 Container image “istio/sidecar_injector:1.2.3” already present on machine
Normal Created 60m kubelet, node-11 Created container sidecar-injector-webhook
Normal Started 60m kubelet, node-11 Started container sidecar-injector-webhook
I have tried purging istio from the cluster and reinstalling from scratch, however still hit the same error.
Pretty lost for what to try next if you can give any suggestions for additional troubleshooting?