Some envoy_cluster_upstream_rq_* counters were not available on prometheus query

Policies and Telemetry
1

we are using Istio 1.1.4 for trial but did not deploy mixer as only focus on traffic control part in gray upgrade. we used official /tools/perf/load/ tool to test the istio.

  1. In our service deploy config yaml, enabled the inbound/outbound metrics and reporting to prometheus.
  template:
    metadata:
      annotations:
            prometheus.io/scheme: http
            prometheus.io/scrape: "true"  sidecar.istio.io/statsInclusionPrefixes:cluster.inbound,cluster.outbound,listener,cluster,cluster_manager,listener_manager,http_mixer_filter,tcp_mixer_filter,server,cluster.xds-grpc
      creationTimestamp: null
      labels:
        app: svc00-0
        role: service
        version: v1
  1. for prometheus config map, commented our following drop rule so prometheus can gather all inbound/outbound counters.
  #metric_relabel_configs:
  # Exclude some of the envoy metrics that have massive cardinality
  # This list may need to be pruned further moving forward, as informed
  # by performance and scalability testing.
  #- source_labels: [ cluster_name ]
  #  regex: '(outbound|inbound|prometheus_stats).*'
  #  action: drop
  #- source_labels: [ tcp_prefix ]
  #  regex: '(outbound|inbound|prometheus_stats).*'
  #  action: drop
  #- source_labels: [ listener_address ]
  #  regex: '(.+)'
  #  action: drop
  #- source_labels: [ http_conn_manager_listener_prefix ]
  #  regex: '(.+)'
  #  action: drop
  #- source_labels: [ http_conn_manager_prefix ]
  #  regex: '(.+)'
  #  action: drop
  #- source_labels: [ __name__ ]
  #  regex: 'envoy_tls.*'
  #  action: drop
  #- source_labels: [ __name__ ]
  #  regex: 'envoy_tcp_downstream.*'
  #  action: drop
  #- source_labels: [ __name__ ]
  #  regex: 'envoy_http_(stats|admin).*'
  #  action: drop
  #- source_labels: [ __name__ ]
  #  regex: 'envoy_cluster_(lb|retry|bind|internal|max|original).*'
  #  action: drop
  1. on Service pod, can get inbound stats via 15000 ports:

$ kubectl exec -it svc00-0v1-987cb7585-8t946 -n service-graph00 -c istio-proxy curl “localhost:15000/stats”

and here is part of metrics, upstream_rq_200/2xx/total are there.

cluster.inbound|8080|http-web|svc00-0.service-graph00.svc.cluster.local.upstream_rq_200: 4647791
cluster.inbound|8080|http-web|svc00-0.service-graph00.svc.cluster.local.upstream_rq_2xx: 4647791
cluster.inbound|8080|http-web|svc00-0.service-graph00.svc.cluster.local.upstream_rq_total: 4755544

  1. but on prometheus UI, can only get envoy_cluster_upstream_rq_total stats data, but can not query upstream_rq_200 or 2XX data.

can any one share light on this? not sure if its our config issue or istio’s intend only to report all upgream counters to prometheus. thanks for help.