Envoy Stripping Custom Headers

Policies and Telemetry
1

We’re running into an issue where if we configure our kubernetes service for HTTP, Envoy will begin stripping our custom headers. As soon as we move it back to a Layer4 proxy (changing the service name prefix to something else), our headers pass into the mesh correctly. I’ve traced the network and looked into the logs - whenever Envoy determines that it needs to add the x-request-id and all the external tracing, is when it drops our headers.

The documentation for Istio/Envoy states that it’ll sanitize headers with the x- prefix, but doesn’t say anything about removing custom headers. Is this expected or is there an easy way to keep Envoy as a layer 7 proxy while being able to maintain our headers?

Our headers are prefixed with ‘wt-’ which has no value and we could change to something else if it would help.

2

I’m using Istio 1.1 prerelease. I deployed the httpbin sample:

kubectl apply -f samples/httpbin/httpbin.yaml
kubectl apply -f samples/httpbin/httpbin-gateway.yaml

I did curl -H "wt-abc: def" -H "wt-123: 456" <my-public-ip>:31380/headers and verified that headers beginning with wt- are making it through:

{
  "headers": {
    "Accept": "*/*", 
   ...
    "Wt-123": "456", 
    "Wt-Abc": "def", 
    "X-B3-Parentspanid": "a3ee2ed5318e6539", 
    ...
  }
}
3

Thanks, Ed! I should’ve mentioned we’re running Istio 1.0.5 - I haven’t run a test against a plain vanilla httpbin sample for our version. I’ll do that now and compare to our prod environment.

4

Hi,

My query is simialr to what was asked by OP.

I am using Istio 1.0.6. We have one of the headers “x-correlation-id”. I want to get this header in Jaeger. How is that possible ? Will it work in 1.1 ?

Also, If I want to have this header “x-correlation-id” as one of the prometheus metric, how is it possible ??

5

Can you lease send me the URL where this is documented ? I couldnt find it anywhere

6

Hey Sourabh - This is the link I was referring to: https://www.envoyproxy.io/docs/envoy/latest/configuration/http_conn_man/header_sanitizing

7

Header sanitation issues aside: If you want to use direct integration (envoy → jaeger) for tracing, you will need to provide updates to the Envoy tracing functionality itself to customize. I’m not sure what the best way would be to accomplish that. If you want to use Mixer to generate your tracespans before sending to Jaeger, you can update your template to include a label with an expression that looks something like request.headers["x-correlation-id"].

You can add headers to prometheus metrics by altering the instances involved (and the prom configuration) just as for the tracespan template. However, adding correlation ids to metrics isn’t advised at any scale, due to cardinality concerns. Please see: Metric and label naming | Prometheus

CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values

8

Hello Douglas,

Thanks for your reply.

What do you mean by when you say “update your template”. Which template are we talking about here. Is there any reference somewhere which I can use to do my desired configuration.

9

The following config artifacts:

2 Likes
10

Hello @douglas-reid,

Thanks for the reply. I can see a lot of CRDs there in the config. yaml.

to change data what jaeger is receiving, what all CRDs I have to modify ?

11

@Sourabh_Wadhwa the links i sent were links to metrics definitions that show exactly which pieces of config you would need to modify to add labels to metrics exposed via Prometheus. There is nothing there about changing the data jaeger is receiving.

12

I tried fetching custom header in Prometheus with the changes you suggested:

I added following line in “metric” named “requestcount”:

correlation_id: request.headers[“x-correlation-id”] | “unknown”

I also added the name to "handler named "prometheus:

metrics:
- name: requests_total
  instance_name: requestcount.metric.{{ .Release.Namespace }}
  kind: COUNTER
  label_names:
  - reporter
  - correlation_id

I can see the metric “correlation_id” appearing in prometheus but not able to see any value to it. Its all “unknown”

13

I looked at the metrics collected by prometheus today and I am now able to get the values of custom headers in prometheus.

Thanks @douglas-reid

1 Like