Where is Telemetry data stored?

#1

Hello everyone,

All the telemetry data for the calls that is being transferred from envoy proxies to mixer to backends. Where is the data getting stored ?

Whats the max amount of data it can store ? Does this gets deleted after-few days as per some rule ?

#2

Using a standard Istio installation, the data is stored in Prometheus (look in the istio-system namespace - you will see the Prometheus pod in there).

The default retention period is 15 days. Telemetry older than that will be purged by Prometheus. The Prometheus default is 15 days as you see documented here, and I don’t think Istio changes that default.

#3

BTW: I’m not sure Istio keeps Prometheus’ default of 15 days. It might be even shorter. I do not know for sure.

#4

OK, curiosity got me - so I hunted this down. Istio sets the Prometheus retention period to 6 hours. Here’s where it sets the Prometheus cmdline option:

and here is where the value of that is set to 6 hours:

(side note to Istio helm chart maintainers - from the prom docs, storage.tsdb.retention has been deprecated in favor of storage.tsdb.retention.time - see https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects )

#5

For the record, the new storage.tsdb.retention.time flag is only available since Prometheus v2.7 and the deprecated flag (eg storage.tsdb.retention) will be kept until Prometheus 3.x (which shouldn’t happen any time soon).

#6

Thats perfect.

Thanks a lot for this @jmazzitelli .

By any chance do you know what all config. files need to be changed, what all new CRDs or kubernetes native objects need to be deployed ?

#7

@Sourabh_Wadhwa if you’re on Istio 1.0, you would need to edit directly the Prometheus deployment resource and change the args key. For the upcoming 1.1, Helm exposes this parameter as @jmazzitelli already mentioned.

#8

Hi,

How to change storage path in prometheus.

Ref: https://github.com/istio/istio/issues/11974

Regards,
Sathish