Best Practice for Prometheus/Grafana?

I have an umbrella Helm chart that installs some cluster-level services for my company, two of which are Prometheus and Grafana, which are installed via the prometheus-operator chart. Is it better to use the ones that come with Istio? If so, do the Istio versions also support things like loading dashboards from configmaps, running Grafana as stateless, and the like?

Thanks for any advice you can offer!

You can run istio’s prometheus and a separate grafana. Though with prometheus operator now supported you might want to run that separately too. I feel the prometheus and especially grafana from the istio chart are great while getting started, but don’t always have all the features you might need long term. That’s just my 2 cents.

Can you point me to any docs on this? I’d love to be able to send Istio metrics and dashboards to my existing prometheus-operator. Thanks!

with istio defaults installation of prom and grafana it comes without any persistance envoloved. We need to use persistance to persist the metrics. Same goes with jaegar tracing. So its better to have your own monitoring and metric stack. You can use the same queries as provided by istio, just connect prometheus to scrape those endpoints

The installation yamls for generating prometheus-operator resources for Istio are available in the installer repo: https://github.com/istio/installer/tree/master/istio-telemetry/prometheus-operator.

Using those should get you what you desire.

Doug, can we add some docs ? Would it be possible to get them to work with kustomize and as ‘a-la-carte’ options ?

@Costin_Manolache1 we definitely can/should add more docs. if you point me to similar docs for other bits of the installer that cover kustomize, etc., i’ll happily create a matching set for prometheus-operator install. I’m also looking for a place to put something on using the integration on the website. Suggestions are welcome.