Labelling behaves differently when port name has different prefix

Policies and Telemetry
1

We are experiencing an odd behaviour when the port name of a deployment and service is prefixed tcp- compared to http-.

If we set the port name to be tcp (see below) we do not see any labelling in ISTIO Prometheus i.e under istio_requests_total, and it looks like all pods match against the ingressgateway rather than their own pod label with many unknown values.

container:
ports:
    - containerPort: 8443
      name: tcp-8443
      protocol: TCP
service:
ports:
  - name: tcp-8443
    port: 8443
    protocol: TCP
    targetPort: tcp-8443

Snippet from istio prometheus with incorrect labelling:

istio_requests_total{connection_security_policy="unknown",container="istio-proxy",destination_app="unknown",destination_canonical_revision="latest",destination_canonical_service="unknown",destination_principal="spiffe://cluster.local/ns/identity/sa/identity",destination_service="id-account-auth.identity.svc.cluster.local",destination_service_name="id-account-auth.identity.svc.cluster.local",destination_service_namespace="unknown",destination_version="unknown",destination_workload="unknown",destination_workload_namespace="unknown",endpoint="15090",instance="100.64.xxx.xxx:15090",job="istio-ingressgateway",namespace="istio-system",pod="istio-ingressgateway-78c5ccc6fc-8t2sf",**pod_name="istio-ingressgateway-78c5ccc6fc-**8t2sf",reporter="source",request_protocol="http",response_code="0",response_flags="DC",service="istio-ingressgateway",source_app="istio-ingressgateway",source_canonical_revision="latest",source_canonical_service="istio-ingressgateway",source_principal="spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account",source_version="unknown",source_workload="istio-ingressgateway",source_workload_namespace="istio-system"} | 8363

However, if we simply change the port name to have http- prefix (see below) we can see the correct labelling in istio prometheus:

container:
ports:
    - containerPort: 8443
      name: http-8443
      protocol: TCP
service:
ports:
  - name: http-8443
    port: 8443
    protocol: TCP
    targetPort: http-8443

Snippet from istio prometheus with correct labelling.

istio_requests_total{connection_security_policy="mutual_tls",container="istio-proxy",destination_app="id-account-auth",destination_canonical_revision="19b8d28",destination_canonical_service="id-account-auth",destination_principal="spiffe://cluster.local/ns/identity-stg/sa/identity",destination_service="id-account-auth.identity-stg.svc.cluster.local",destination_service_name="id-account-auth",destination_service_namespace="identity-stg",destination_version="19b8d28",destination_workload="id-account-auth",destination_workload_namespace="identity-stg",endpoint="15090",instance="100.64.xxx.xxx:15090",job="id-account-auth",namespace="identity-stg",pod="id-account-auth-5965ff5c4d-m2gwp",**pod_name="id-account-auth-5965ff5c4d-m2gwp",**reporter="destination",request_protocol="http",response_code="200",response_flags="-",service="id-account-auth",source_app="istio-ingressgateway",source_canonical_revision="latest",source_canonical_service="istio-ingressgateway",source_principal="spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account",source_version="unknown",source_workload="istio-ingressgateway",source_workload_namespace="istio-system"}
2

See https://preliminary.istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/

3

Thanks for the link. We are aware that traffic behaviour is different but it’s not obvious anywhere in the documentation that the labelling in istio_requests_total would not be the same for the different connection types i.e. lots of unknown when tcp prefix is used.