Monitoring Envoy with DataDog Autodiscovery

1

I’m trying to get the DataDog Envoy integration working with Istio.

To get it to work, I’m using the Autodiscovery feature.
We have a custom sidecar injector, which is putting the metadata.annotations on for us.
The DD agent (which is a daemon) is set up to check http://$(HOST):15000/stats

apiVersion: v1
kind: Pod
metadata:
  annotations:
    ad.datadoghq.com/istio-proxy.check_names: '["envoy"]'
    ad.datadoghq.com/istio-proxy.init_configs: '[{}]'
    ad.datadoghq.com/istio-proxy.instances: '[{"stats_url":"http://%%host%%:15000/stats?usedonly","cache_metrics":true,"verify_ssl":false,"skip_proxy":true,"timeout":10}]'
...

When I connect to the DataDog Agent pod and run the Envoy check, I cannot connect to the stats endpoint:

# agent check envoy
Error: (envoy.py:73) | Error accessing Envoy endpoint `http://10.52.23.200:15000/stats?usedonly`

Then:

# curl -v http://10.52.23.200:15000/stats       
*   Trying 10.52.23.200...
* TCP_NODELAY set
* Connected to 10.52.23.200 (10.52.23.200) port 15000 (#0)
> GET /stats HTTP/1.1
> Host: 10.52.23.200:15000
> User-Agent: curl/7.59.0
> Accept: */*
> 
* Empty reply from server
* Connection #0 to host 10.52.23.200 left intact
curl: (52) Empty reply from server

How do I allow the Agent to talk to the stats endpoint?

Bonus points if we can do this over TLS/mTLS.

Thanks,
-mk

2 Likes
2

The iptables output, for reference:

web-6d96bfc47f-2h2bj /home/michaelkipper # iptables -t nat --list
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
RETURN     tcp  --  anywhere             anywhere             tcp dpt:8888
ISTIO_INBOUND  tcp  --  anywhere             anywhere            

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ISTIO_OUTPUT  tcp  --  anywhere             anywhere            

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain ISTIO_INBOUND (1 references)
target     prot opt source               destination         
ISTIO_IN_REDIRECT  tcp  --  anywhere             anywhere             tcp dpt:15000
ISTIO_IN_REDIRECT  tcp  --  anywhere             anywhere             tcp dpt:15090
ISTIO_IN_REDIRECT  tcp  --  anywhere             anywhere             tcp dpt:http-alt
ISTIO_IN_REDIRECT  tcp  --  anywhere             anywhere             tcp dpt:8888

Chain ISTIO_IN_REDIRECT (4 references)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             anywhere             redir ports 15001

Chain ISTIO_OUTPUT (1 references)
target     prot opt source               destination         
ISTIO_REDIRECT  all  --  anywhere            !localhost           
RETURN     all  --  anywhere             anywhere             owner UID match 1337
RETURN     all  --  anywhere             anywhere             owner GID match 1337
RETURN     all  --  anywhere             localhost           
ISTIO_REDIRECT  all  --  anywhere             anywhere            

Chain ISTIO_REDIRECT (2 references)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             anywhere             redir ports 15001
3

@michaelkipper - Did you manage to figure this out in the end? Currently running into the same problem trying to implement datadog monitoring for one of our microservices.

root@datadog-agent-5dcxj:/# curl -vvv http://10.1.32.113:15000/stats?usedonly
  • Trying 10.1.32.113:15000…
  • TCP_NODELAY set
  • connect to 10.1.32.113 port 15000 failed: Connection refused
  • Failed to connect to 10.1.32.113 port 15000: Connection refused
  • Closing connection 0
    curl: (7) Failed to connect to 10.1.32.113 port 15000: Connection refused
4

So far i’ve discovered following things:

  1. Istio does not expose port 15000 in envoy proxy by default,
  2. Istio chart does not support exposing additional ports (can’t fix above),
  3. Istio exposes port 15090 which can only access http://<pod-ip>:15090/stats/prometheus
  4. Datadog’s Envoy integration cannot use Prometheus endpoint

currently looking for ways to get those metrics out

5

I’m running into this issue as well. Did we every find a fix for it?

6

This seems to be close:

      annotations:
        ad.datadoghq.com/istio-proxy.check_names: '["openmetrics"]'
        ad.datadoghq.com/istio-proxy.init_configs: '[{}]'
        ad.datadoghq.com/istio-proxy.instances: '[{"prometheus_url":"http://%%host%%:15090/stats/prometheus", "namespace": "dev-dashboard", "metrics": [{"istio_requests_total":"istio_requests_total"}]}]'

Still trying to hammer some stuff out, and have engaged Datadog support. From what I can tell, Istio’s prometheus is using an envoy-stats check to discover and collect metrics from the istio-proxy directly. Trying to figure out how to replicate that with Datadog is a little harder than it should be.

7

I made DD autodicovery work once with Istio 1.5, but then envoy was outputting some of the metrics out of order, which made the Datadog agent ignore most of the metrics. Datadog Support wasn’t helpful in my case, I ended up switching my setup to use Prometheus instead.

I’m interested to see if this is now working OK.

8

I’m currently fighting with DD support now about this. It’s…frustrating. I may wind up trying to scrape from istio’s prometheus using federation.

9

Hi @donnyv12 … I have recently setup Datadog integration through the Istio, Envoy and Prometheus plugins (take your pick). I have send you an email on your email address for an introduction.

10

I’m having the same problem. How did you solve it?

11

@donnyv12 Did you manage to figure it out? I got them working by using openmetrics

          "ad.datadoghq.com/istio-proxy.check_names": "[\"envoy\"]",
          "ad.datadoghq.com/istio-proxy.init_configs": "[{}]",
          "ad.datadoghq.com/istio-proxy.instances": "[{\"openmetrics_endpoint\": \"http://%%host%%:15090/stats/prometheus\", \"histogram_buckets_as_distributions\": \"true\", \"log_requests\": \"true\"}]",
          "ad.datadoghq.com/istio-proxy.logs": "[{\"source\":\"envoy\",\"service\":\"tms\"}]"