Removing the x-envoy-peer-metadata header

Config
1

Background

So we had an interesting issue related to #17635. The number of unique series coming from istio was growing very fast. I started looking into it and the metrics had the label egressor-xxxxxx. At first I was very confused since we didn’t have such a deployment but after a while we realised that it was due to headers sent by an external party. Specifically x-envoy-peer-metadata with workload_name changing over time.

Trying to solve it

We want to drop the x-envoy-peer-metadata header for external traffic but AWS ALB or WAF does not support this. We are not using Ingress Gateway. I have seen some solutions for dropping headers for incoming/outgoing traffic in istio using Envoyfilters and Virtualservices. Virtual services was not going to work here so I started testing using EnvoyFilters.

I have tried using Lua to modify the headers which worked with other headers but not the x-envoy-peer headers.

I have tried using the ROUTE_CONFIGURATION both attempts seems to work with some headers but not x-envoy-peer headers.

My last attempt was trying to configure the early_header_mutation_extensions from the http_connection_manager since the description seems to be exactly what I want. Drop headers before they are processed by telemetry. But I cannot get it to show up in my envoy config. Some variations of it seems to not be rejected but when it is not rejected nothing happens.

Current iteration of the EnvoyFilter:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: request-remove-headers
  namespace: jx-staging
spec:
  workloadSelector:
    labels:
      app: appname
  configPatches:
  - applyTo: HTTP_FILTER
    match: 
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: INSERT_BEFORE
      value:
        config_discovery:
          config_source:
            ads: {}
          type_urls: ["type.googleapis.com/envoy.extensions.http.early_header_mutation.header_mutation.v3.HeaderMutation"]
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          early_header_mutation_extensions:
          - name: http.early_header_mutation.header_mutation
            typed_config:
              "@type": "type.googleapis.com/envoy.extensions.http.early_header_mutation.header_mutation.v3.HeaderMutation"
              mutations:
              - remove: "x-envoy-peer-metadata"

Questions

  • Does anyone know how to configure early_header_mutation_extensions using a EnvoyFilter?
  • Can you remove these headers using a ingress gateway?
  • Any other ways for removing these headers?