Monitoring Envoy with DataDog Autodiscovery


#1

I’m trying to get the DataDog Envoy integration working with Istio.

To get it to work, I’m using the Autodiscovery feature.
We have a custom sidecar injector, which is putting the metadata.annotations on for us.
The DD agent (which is a daemon) is set up to check http://$(HOST):15000/stats

apiVersion: v1
kind: Pod
metadata:
  annotations:
    ad.datadoghq.com/istio-proxy.check_names: '["envoy"]'
    ad.datadoghq.com/istio-proxy.init_configs: '[{}]'
    ad.datadoghq.com/istio-proxy.instances: '[{"stats_url":"http://%%host%%:15000/stats?usedonly","cache_metrics":true,"verify_ssl":false,"skip_proxy":true,"timeout":10}]'
...

When I connect to the DataDog Agent pod and run the Envoy check, I cannot connect to the stats endpoint:

# agent check envoy
Error: (envoy.py:73) | Error accessing Envoy endpoint `http://10.52.23.200:15000/stats?usedonly`

Then:

# curl -v http://10.52.23.200:15000/stats       
*   Trying 10.52.23.200...
* TCP_NODELAY set
* Connected to 10.52.23.200 (10.52.23.200) port 15000 (#0)
> GET /stats HTTP/1.1
> Host: 10.52.23.200:15000
> User-Agent: curl/7.59.0
> Accept: */*
> 
* Empty reply from server
* Connection #0 to host 10.52.23.200 left intact
curl: (52) Empty reply from server

How do I allow the Agent to talk to the stats endpoint?

Bonus points if we can do this over TLS/mTLS.

Thanks,
-mk


#2

The iptables output, for reference:

web-6d96bfc47f-2h2bj /home/michaelkipper # iptables -t nat --list
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
RETURN     tcp  --  anywhere             anywhere             tcp dpt:8888
ISTIO_INBOUND  tcp  --  anywhere             anywhere            

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ISTIO_OUTPUT  tcp  --  anywhere             anywhere            

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain ISTIO_INBOUND (1 references)
target     prot opt source               destination         
ISTIO_IN_REDIRECT  tcp  --  anywhere             anywhere             tcp dpt:15000
ISTIO_IN_REDIRECT  tcp  --  anywhere             anywhere             tcp dpt:15090
ISTIO_IN_REDIRECT  tcp  --  anywhere             anywhere             tcp dpt:http-alt
ISTIO_IN_REDIRECT  tcp  --  anywhere             anywhere             tcp dpt:8888

Chain ISTIO_IN_REDIRECT (4 references)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             anywhere             redir ports 15001

Chain ISTIO_OUTPUT (1 references)
target     prot opt source               destination         
ISTIO_REDIRECT  all  --  anywhere            !localhost           
RETURN     all  --  anywhere             anywhere             owner UID match 1337
RETURN     all  --  anywhere             anywhere             owner GID match 1337
RETURN     all  --  anywhere             localhost           
ISTIO_REDIRECT  all  --  anywhere             anywhere            

Chain ISTIO_REDIRECT (2 references)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             anywhere             redir ports 15001