Istio Ingress Gateway Inbound Metrics Missing

Policies and Telemetry Kiali
1

With Nginx Ingress Controller we are able to collect metrics such as Controller Request Volume, Response Status Codes overall for the Nginx Ingress Controller. I am trying to either setup a similar dashboard or see relevant metrics in Istio.

However, I dont see metrics where the destination workload is istio-ingressgateway

ie:

istio_requests_total{destination_workload_namespace=~"istio-system", destination_workload="istio-ingressgateway"}

returns no results.

Should I be seeing results for this query and have something misconfigured, or is this not possible with Istio.
Or how would I set up these metrics?

2

Kind of answering my own question.

I found in Istio by default logs mostly client side load balancing specific stats such as these:

cluster_manager
listener_manager
http_mixer_filter
tcp_mixer_filter
server
cluster.xds-grpc

to see server stats you will need to add these

downstream_rq_1xx,downstream_rq_2xx,downstream_rq_3xx,downstream_rq_4xx,downstream_rq_5xx,downstream_rq_time,downstream_cx_tx_bytes_total,downstream_cx_rx_bytes_total,downstream_cx_total

To do so update the following environment variable on ingress-gateway

containers:
      - name: istio-proxy
        env:
        - name: ISTIO_METAJSON_STATS
          value: |
            {"sidecar.istio.io/statsInclusionSuffixes": "downstream_rq_1xx" , "downstream_rq_2xx" ,"downstream_rq_3xx","downstream_rq_4xx","downstream_rq_5xx","downstream_rq_time","downstream_cx_tx_bytes_total","downstream_cx_rx_bytes_total",downstream_cx_total"}

Then you could create a pod monitor to scrape those metrics (if you have your own prometheus)

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: envoy-stats
  namespace: monitoring
  prometheus: kube-prometheus
  labels:
    monitoring: services
    prometheus: kube-prometheus
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  namespaceSelector:
    matchNames:
      - istio-system
  podMetricsEndpoints:
    - interval: 30s
      port: http-envoy-prom

reference: https://github.com/istio/istio/wiki/Envoy-native-telemetry

3

@crhuber inbound traffic is not seen even after adding the given env variable. I am using Istio 1.8 currently. Is there any other way to get inbound metrics?