Grafana Pilot Dashboard xDS metrics missing

Hello,
I am looking at the default Istio Pilot Dashboard under the xDS section.All the panels in that section have no information.

Then I checked our external prometheus and noticed we dont have any metrics for:

envoy_cluster_update_attempt{}

We use a Service Monitor for prometheus as such:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istio-monitor
  namespace: monitoring
  prometheus: kube-prometheus
  labels:
    monitoring: services
    prometheus: kube-prometheus
spec:
  selector:
    matchExpressions:
      - {key: istio, operator: In, values: [pilot, mixer, citadel, galley, ingressgateway]}
  namespaceSelector:
    matchNames:
      - istio-system
  jobLabel: istio
  endpoints:
  - port: prometheus
    interval: 15s
  - port: http-monitoring
    interval: 15s

However Im not seeing any envoy_cluster_* metrics. How can I enable that using a service monitor

If you are looking for prometheus operator config for Istio, I suggest looking in the new installer repo. There is a set of configs for ServiceMonitors available.

In particular, you will likely be interested in: https://github.com/istio/installer/blob/2fce227d672c9149cca30db7dbfff3cd003964e2/istio-telemetry/prometheus-operator/templates/servicemonitors.yaml#L40-L70

@douglas-reid awesome thanks man