Telemetry information not captured

Policies and Telemetry
1

59533602-a29c0400-8eb1-11e9-8d04-55cb25a52e9b.jpg1593×1475 143 KB

I have the above setup functional where the traffic flows from the istio ingress gateway to nginx contoller and finally to the speedtest service, however, I do not see the required telemetry information captured for traffic between the nginx ingress controller pod and the speedtest service.

Both the nginx controller and the speedtest are part of the service mesh. Pods of both these services have sidecar injected in them.

The traffic flows perfectly fine from the nginx ingress controller svc to the speedtest pod.

nginx ingress controller svc —> nginx-ingress pod —> speedtest-svc —> speedtestpod

However, no telemetry data is getting collected for the link between nginx ingress pod --> speedtest-svc

Here are the manifests used

virtual-service


apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx-virtual-service
  namespace: default
spec:
  gateways:
  - nginx-gateway
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: nginx-ingress-controller
        port:
          number: 80
    timeout: 86400s

nginx ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/service-upstream: "true"
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/use-regex: "true"
  labels:
    app: vbergbauer-speedtest
  name: speedtest-ingress
  namespace: default
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: speedtest-vbergbauer-speedtest
          servicePort: 8903
        path: /(speedtest)/?(.*)

Any hints on why the telemetry information is not being captured for the hop between the nginx ingress controller pod and the speedtest-svc

I did not explicitly open any http-ports for the speedtest service. Could this be the reason for the metrics not being captured ?

Istio version: 1.1.7

2

This is likely the issue. Please see Named Service Ports section from: Istio / Application Requirements.

3

Thanks. I did rename the service exposed to port http-speedtestport but it did not help .

So should the http-speedtest port also be included in the istio-ingress-gateway ports? Does it also need to opened via a gateway and a virtual service ?

4

what happens when you curl the speedtest service directly?

do you see istio_requests_total{reporter="destination"} metrics for direct calls to the service?

5

I try to connect directly to the service and see what metrics were generated

I only see the metrics from the service to the istio-telemetry, but do not find anything related to direct calls to the service

istio_requests_total{connection_security_policy="none",destination_app="telemetry",
destination_principal="unknown",
destination_service="istio-telemetry.istio- system.svc.cluster.local",
destination_service_name="istio-telemetry",
destination_service_namespace="istio-system",destination_version="unknown",
destination_workload="istio-telemetry",
destination_workload_namespace="istio-system",instance="ip",
job="istiomesh",permissive_response_code="none",
permissive_response_policyid="none",reporter="destination",request_protocol="grpc",
response_code="200",response_flags="-",
source_app="vbergbauer-speedtest",source_principal="unknown",
source_version="unknown",
source_workload="speedtest-vbergbauer-speedtest",source_workload_namespace="default"}
6

The ones you are seeing are only from the server side for istio-telemetry.

Can you confirm that your endpoints are indeed sitting behind a proxy and that the proxy is in fact handling traffic? A get pods in that namespace should help.

Please also post your service spec and any logs from the proxy that you can see.