Prometheus request.duration collection on external HTTPS URL?

in istio 1.9 its enough to annotate your pod with prometheus.io/scrape: 'true'
to allow istio to inject Prometheus port (15020) and path to your pod.
Then Prometheus will start collecting the data.
however, if you app POD access external web site (https://cnn.com) which is defined as a ServiceEntry EXTERNAL_MESH entity (egress rule is REGISTRY_ONLY) . The pod is not using istio-egressgateway to route all outgoing traffic.
for cnn I don’t see any istio_request_duration_milliseconds histogram
I do see the request duration to inner K8s services that are in my mesh.
is that’s by design ?
or is that because metrics will not be collected for HTTPS ? sinceI read in istio docs that those metrics are collected only for TCP and HTTP/2 . nothing about TLS/HTTPS.

@tomhawk ,

There are two main aspects:

1) Prometheus + Istio design

When you use the annotation prometheus.io/scrape, you tell prometheus that it can discover that POD and, from time to time, perform a GET request on the metrics endpoint. This endpoint, in a mesh, is served by Envoy as a default. So, Envoy provides all metrics (inclusing istio_request_duration_milliseconds) that is record when intercepting every request.

If cnn.com provides a metrics endpoint in Prometheus format (which I think is not the case), you can configure Prometheus to “crawl” that endpoint from time to time and then, you’ll have that metric (if available). If you want to know more about how Prometheus work, you can look at Overview | Prometheus.

2) HTTP

For HTTPS, the challenge indeed exists. The point is that HTTP follows a Request/Response model over TCP and your client may want to leverage open TCP connections for issuing more than one HTTP interaction. Because of that, if you application connects over HTTPS, Envoy will only intercept the TCP connection and will not be able to look inside the packages sent to determine the response time.

A workaround for that is to use TLS origination: Istio / Egress Gateways with TLS Origination (File Mount), This way, your application opens a HTTP interaction with Envoy which, in turn, connects to CNN over TLS. This way, Envoy will have access to the payloads and will be able to calculate response times and check HTTP status codes.

Hope it helps!

@Leandro_Rodrigues_de thanks for the response. However, I don’t want Prometheus to scrape CNN.com . I just want it to report on my app POD: histograms for getting a response times from CNN.com.
But from your answer I start to think maybe Prometheus can collect metrics on “endpoints” that are different than scrapping PODs ?

now: lets assume I will drive all PODs external request to the egress GW (Which will do the TLS origination): will istio report histograms based on URL when I scrape the istio egress GW pod ?

Short answer: yes. You can also do this without Egress Gateway, making TLS origination in every POD. See this link for that. The service name will be the one you choose in the Service Entry you create for CNN.

This way Istio will provide you with de response times histogram for external requests to the desired services. This is what I meant in the HTTP section from my previous answer. Of course, this will be a client’s perspective, which I think address your needs!