I am currently in the process of rolling out Istio (v1.0.6) for a new set of services. With each deployment I am creating a new k8s service, and updating the Istio VirtualService to point to the new service when ready to shift the traffic across, all of this works fine. My concern is that the Istio shipped Prometheus instance continues to expose metrics for the old, no longer in use services. Ideally I would like to stop reporting the old data once the old service has been removed, or some fixed period afterwards.
The handler has a metrics expiry policy. This means that Mixer will stop holding on to a metric after a certain period (so that it won’t be picked up in prometheus scrapes). That is one part of the solution.
EDIT: I think i have found the prometheus handler configuration so will investigate either modifying the default or creating an additional config.istio.io/prometheus config to eliminate storing the version information.
Hi Douglas,
Thanks for the reply. I think I have ruled out prometheus being an issue, as I can see a varying number of istio_requests_total metrics returned from the different istio-telemetry pods, with more metrics being returned from longer running pods.
This leads me to believe that either the expiry is not operating as it should (I don’t know where I ought to be looking for this configuration) or the old deployment / services are still configured somewhere in istio / envoy, so aren’t ageing the statistics.
I have tried looking at the cluster configs, but really don’t know what I am looking for, other than old release names which I haven’t found yet.
Any additional hints / places to look for configuration errors etc would be greatly appreciated.