Hi,
I would like to monitor the Istio metrics with the usage of Prometheus Operator, but it seems to cause the memory leak. I am using the semi-official PodMonitor and ServiceMonitor taken from Istio Github:
When I create those ServiceMonitor and PodMonitor, the memory usage starts to grow slowly and in an hour it goes from around 10GB up to 30GB. This causes Prometheus Operator to be killed with Out Of Memory exception. I have spotted that the number of the time series grows constantly as well after enabling Istio monitoring.
Do you have any suggestions on what can be the root cause of the issue and how to solve it?
Thank you! Any help appreciated!