I have a Istio on GKE setup with Grafana added on after install, but I’m unable to see any metrics outside the istio-system namespace in Prometheus/Grafana. According to the troubleshooting guide, this appears to be because I’m completely missing a istio-mesh target. (I’m rather mystified by how this happened, since everything else for Istio appears to be in place, but that’s another story.)
My question: how exactly do I go about updating this config in a persistent way? It’s not editable in the UI and if I update the prometheus configmap with kubectl, my changes appear to get silently nuked – the changes last for a few seconds, and then the resourceVersion gets bumped up and my additions disappear.
$ kubectl apply -f prom-configmap.yaml --namespace=istio-system $ kubectl get configmap prometheus --namespace=istio-system -o yaml >prom-configmap2.yaml
At this point the changes are registered with the timestamp set and resourceVersion ticking up:
$ diff prom-configmap.yaml prom-configmap2.yaml | more 121c121 < --- > > creationTimestamp: "2019-02-15T06:32:49Z" > resourceVersion: "955954" > uid: 83994e96-30eb-11e9-a9e9-42010a940019
But when I try again a few seconds later, my additions have disappeared:
$ kubectl get configmap prometheus --namespace=istio-system -o yaml >prom-configmap3.yaml $ diff prom-configmap3.yaml prom-configmap2.yaml < resourceVersion: "955978" --- > resourceVersion: "955954" 107a108,121 > - job_name: 'istio-mesh' > # Override the global default and scrape targets from this job every 5 seconds. > scrape_interval: 5s > # metrics_path defaults to '/metrics' > # # scheme defaults to 'http'. > static_configs: > - targets: ['istio-mixer.istio-system:42422'] > > kubernetes_sd_configs: > - role: endpoints > namespaces: > names: > - istio-system