Hi,
zipkin tracing was working fine after upgrade 1.8.x to 1.11.3 (in k8s 1.20.11) , few weeks later, i realised that no tracing in jaeger (1.23.0) for the majority of services, also for some services it’s working for one pod only from 2 of the same service
I have no errors in proxy sidecar , a tcpdump in proxy sidecar doesn’t show any call from envoy to the collector jaeger
I checked the conf between envoy of working and not working one, there is no difference, the only difference i found it’s in logs
The envoy where the tracing works i have this log
2022-04-22T17:40:53.427806Z info envoy config loading tracing configuration
2022-04-22T17:40:53.427813Z info envoy config validating default server-wide tracing driver: envoy.tracers.zipkin
2022-04-22T17:40:53.428166Z info envoy config loading 0 static secret(s)
2022-04-22T17:40:53.428173Z info envoy config loading 5 cluster(s)
2022-04-22T17:40:53.622185Z info envoy config loading 2 listener(s)
2022-04-22T17:40:53.626314Z info envoy config loading stats configuration
2022-04-22T17:40:53.627700Z info envoy main starting main dispatch loop
2022-04-22T17:40:53.628759Z info envoy runtime RTDS has finished initialization
2022-04-22T17:40:53.628769Z info envoy upstream cm init: initializing cds
2022-04-22T17:40:54.041529Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
logs of envoy with tracing not working:
2022-04-12T12:01:21.672451Z info envoy config loading tracing configuration
2022-04-12T12:01:21.672465Z info envoy config loading 0 static secret(s)
2022-04-12T12:01:21.672468Z info envoy config loading 5 cluster(s)
2022-04-12T12:01:21.692575Z info envoy config loading 2 listener(s)
2022-04-12T12:01:21.770175Z info envoy tracing instantiating a new tracer: envoy.dynamic.ot
2022-04-12T12:01:22.346236Z info envoy config loading stats configuration
2022-04-12T12:01:22.347604Z info envoy main starting main dispatch loop
2022-04-12T12:01:22.348537Z info envoy runtime RTDS has finished initialization
2022-04-12T12:01:22.348549Z info envoy upstream cm init: initializing cds
2022-04-12T12:01:22.353492Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
It looks like sometimes, the envoy is not aware about the conf i did about zipkin so , it is instantiating a new tracer: envoy.dynamic.ot ?
I have also in istiod logs a lot of warn and error
2022-04-23T07:54:38.725639Z warn Secure Webhook Server is not ready: Get "https://:15017/httpsReady": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-04-23T07:55:17.125820Z error klog Failed to update lock: Operation cannot be fulfilled on configmaps "istio-namespace-controller-election": the object has been modified; please apply your changes to the latest version and try again
2022-04-23T07:55:20.027770Z error klog Failed to update lock: Operation cannot be fulfilled on configmaps "istio-leader": the object has been modified; please apply your changes to the latest version and try again
2022-04-23T07:55:24.627735Z warn kube couldn't find "istio-system/istiod-7456447bfc-67tsj" in informer index
2022-04-23T07:55:24.627824Z warn kube couldn't find "istio-system/jaeger-query-7d55d49794-cphxj" in informer index
2022-04-23T07:55:24.627835Z warn kube couldn't find "services-qa-int/postgres-bookmark-744d6f974d-g9hkh" in informer index
2022-04-23T07:55:24.627852Z warn kube couldn't find "istio-system/jaeger-collector-5b8698b9fc-6hqkk" in informer index
2022-04-23T07:55:24.627855Z warn kube
the conf of istioOperator:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
meshConfig:
enableTracing: true
defaultConfig:
tracing:
zipkin:
address: jaeger-collector.istio-system:9411
components:
ingressGateways:
- name: istio-ingressgateway
enabled: true
k8s:
service:
type: NodePort
ports:
- port: 443
targetPort: 8443
name: https
nodePort: 31000
- port: 15021
targetPort: 15021
name: status-port
- port: 80
targetPort: 8080
name: http2
nodePort: 32080
- port: 15443
targetPort: 15443
name: tls
resources:
limits:
memory: 1Gi
cpu: 200m
requests:
memory: 75Mi
cpu: 15m
egressGateways:
- name: istio-egressgateway
enabled: false
pilot:
enabled: true
k8s:
resources:
limits:
memory: 1Gi
cpu: 200m
requests:
memory: 300Mi
cpu: 50m
replicaCount: 2
values:
global:
jwtPolicy: first-party-jwt
logging:
level: "default:info"
proxy:
logLevel: info
imagePullPolicy: "IfNotPresent"
pilot:
autoscaleEnabled: false
traceSampling: 1
jwksResolverExtraRootCA: |
-----BEGIN CERTIFICATE-----
XXXXXXXXXXXXXXXXXXXXXX
-----END CERTIFICATE-----
gateways:
istio-egressgateway:
autoscaleEnabled: false
istio-ingressgateway:
autoscaleEnabled: false
Thank you for your help