Where is Telemetry data stored?

Sourabh_Wadhwa · February 22, 2019, 12:53am

Hello everyone,

All the telemetry data for the calls that is being transferred from envoy proxies to mixer to backends. Where is the data getting stored ?

Whats the max amount of data it can store ? Does this gets deleted after-few days as per some rule ?

jmazzitelli · February 22, 2019, 1:18am

Using a standard Istio installation, the data is stored in Prometheus (look in the istio-system namespace - you will see the Prometheus pod in there).

The default retention period is 15 days. Telemetry older than that will be purged by Prometheus. The Prometheus default is 15 days as you see documented here, and I don’t think Istio changes that default.

jmazzitelli · February 22, 2019, 1:23am

BTW: I’m not sure Istio keeps Prometheus’ default of 15 days. It might be even shorter. I do not know for sure.

jmazzitelli · February 22, 2019, 1:37am

OK, curiosity got me - so I hunted this down. Istio sets the Prometheus retention period to 6 hours. Here’s where it sets the Prometheus cmdline option:

github.com

istio/istio/blob/fcd35891f2bd99a996c59e07ee187b179c83f04d/install/kubernetes/helm/istio/charts/prometheus/templates/deployment.yaml#L47


  imagePullPolicy: {{ .Values.global.imagePullPolicy }}
  volumeMounts:
    - mountPath: /etc/istio-certs
      name: istio-certs
{{- end }}
containers:
  - name: prometheus
    image: "{{ .Values.hub }}/prometheus:{{ .Values.tag }}"
    imagePullPolicy: {{ .Values.global.imagePullPolicy }}
    args:
      - '--storage.tsdb.retention={{ .Values.retention }}'
      - '--config.file=/etc/prometheus/prometheus.yml'
    ports:
      - containerPort: 9090
        name: http
    livenessProbe:
      httpGet:
        path: /-/healthy
        port: 9090
    readinessProbe:
      httpGet:

and here is where the value of that is set to 6 hours:

github.com

istio/istio/blob/fcd35891f2bd99a996c59e07ee187b179c83f04d/install/kubernetes/helm/istio/charts/prometheus/values.yaml#L8


#
# addon prometheus configuration
#
enabled: true
replicaCount: 1
hub: docker.io/prom
tag: v2.3.1
retention: 6h
nodeSelector: {}


# Controls the frequency of prometheus scraping
scrapeInterval: 15s


contextPath: /prometheus


ingress:
enabled: false
## Used to create an Ingress record.

(side note to Istio helm chart maintainers - from the prom docs, storage.tsdb.retention has been deprecated in favor of storage.tsdb.retention.time - see https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects )

Simon_Pasquier · February 22, 2019, 9:29am

For the record, the new storage.tsdb.retention.time flag is only available since Prometheus v2.7 and the deprecated flag (eg storage.tsdb.retention) will be kept until Prometheus 3.x (which shouldn’t happen any time soon).

Sourabh_Wadhwa · February 26, 2019, 8:31pm

Thats perfect.

Thanks a lot for this @jmazzitelli .

By any chance do you know what all config. files need to be changed, what all new CRDs or kubernetes native objects need to be deployed ?

Simon_Pasquier · February 27, 2019, 9:43am

@Sourabh_Wadhwa if you’re on Istio 1.0, you would need to edit directly the Prometheus deployment resource and change the args key. For the upcoming 1.1, Helm exposes this parameter as @jmazzitelli already mentioned.

sathishkumar · April 9, 2019, 6:50am

Hi,

How to change storage path in prometheus.

Ref: https://github.com/istio/istio/issues/11974

Regards,
Sathish

Topic		Replies	Views
Prometheus Stats Expiry	3	1080	April 1, 2019
Where can I find the data of Prometheus?	2	361	January 19, 2022
Metric expiry in TelemetryV2 proxies Policies and Telemetry	0	491	December 1, 2020
Issue with Prometheus and Kiali Policies and Telemetry	0	193	December 1, 2023
BYO Prometheus with mTLS Policies and Telemetry	18	5820	December 21, 2020

Where is Telemetry data stored?

Related Topics