When implementing request tracing in our clusters we noticed that x-request-ids generated by nginx get overwritten by the Envoy proxy.
Our internal nginx gateway(outside the k8s cluster) proxies requests to Istio’s 31380 port with all the request headers, the x-forwarded-for with nginx’s ip appended and a generated x-request-id by nginx gateway.
Having checked Envoy’s header sanitizing we are trying to modify the proxy configuration for our Istio ingress and egress gateways so requests coming from our nginx gateway are recognised as internal requests and nginx, Istio and Envoy use the same x-request-id for a request and can be traced easily. Once we get them right, we’ll move onto the sidecar proxies.
After having a play with EnvoyFilters we can see that the specified filter gets appended to existing ones and can be seen with:
istioctl proxy-config listeners istio-ingressgateway-6755b9bbf6-mx6f8 -n istio-system -ojson
When issuing the command we see that another filter is created next to the default http_connection_manager.
We can also see that the config option use_remote_address for envoy in the ingress gateway is set to true by Istio.(couldn’t find anything in the repo that’d set it though)
Is there a way to amend the ingress gateway proxy’s http_connection_manager filter config without creating a new one with the same name with custom k8s objects provided by Istio or is there only a more messy way to modify the envoy config? We’ve found the envoy-rev0.json under /etc/istio/proxy inside the ingress gateway container, but it looks a bit different to the output of the istioctl proxy-config listeners command so we suppose envoy gets its configuration from somewhere else (too?).
Apply the following manifest to modify the envoy config used by the ingress gateway:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: gateway-filter
namespace: istio-system
spec:
workloadLabels:
app: istio-ingressgateway
filters:
- filterName: envoy.http_connection_manager
filterType: NETWORK
listenerMatch:
portNumber: 80
listenerType: GATEWAY
filterConfig:
stat_prefix: "0.0.0.0_80"
generate_request_id: false
use_remote_address: false
rds:
config_source:
ads: {}
route_config_name: http.80
use the following to see the listeners:
$ istioctl proxy-config listeners istio-ingressgateway-6755b9bbf6-mx6f8 -n istio-system -ojson
[
{
"name": "0.0.0.0_80",
"address": {
"socketAddress": {
"address": "0.0.0.0",
"portValue": 80
}
},
"filterChains": [
{
"filters": [
{
"name": "envoy.http_connection_manager",
"config": {
"generate_request_id": false,
"rds": {
"config_source": {
"ads": {}
},
"route_config_name": "http.80"
},
"stat_prefix": "0.0.0.0_80",
"use_remote_address": false
}
},
{
"name": "envoy.http_connection_manager",
"config": {
"access_log": [
{
"config": {
"format": "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%%PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS%\n",
"path": "/dev/stdout"
},
"name": "envoy.file_access_log"
}
],
"generate_request_id": true,
"http_filters": [
{
"config": {
"default_destination_service": "default",
"forward_attributes": {
"attributes": {
"source.uid": {
"string_value": "kubernetes://istio-ingressgateway-6755b9bbf6-mx6f8.istio-system"
}
}
},
"mixer_attributes": {
"attributes": {
"context.reporter.kind": {
"string_value": "outbound"
},
"context.reporter.uid": {
"string_value": "kubernetes://istio-ingressgateway-6755b9bbf6-mx6f8.istio-system"
},
"source.namespace": {
"string_value": "istio-system"
},
"source.uid": {
"string_value": "kubernetes://istio-ingressgateway-6755b9bbf6-mx6f8.istio-system"
}
}
},
"service_configs": {
"default": {}
},
"transport": {
"attributes_for_mixer_proxy": {
"attributes": {
"source.uid": {
"string_value": "kubernetes://istio-ingressgateway-6755b9bbf6-mx6f8.istio-system"
}
}
},
"check_cluster": "outbound|9091||istio-policy.istio-system.svc.cluster.local",
"network_fail_policy": {
"policy": "FAIL_CLOSE"
},
"report_cluster": "outbound|9091||istio-telemetry.istio-system.svc.cluster.local"
}
},
"name": "mixer"
},
{
"name": "envoy.cors"
},
{
"name": "envoy.fault"
},
{
"name": "envoy.router"
}
],
"rds": {
"config_source": {
"ads": {}
},
"route_config_name": "http.80"
},
"stat_prefix": "0.0.0.0_80",
"stream_idle_timeout": "0.000s",
"tracing": {
"client_sampling": {
"value": 100
},
"operation_name": "EGRESS",
"overall_sampling": {
"value": 100
},
"random_sampling": {
"value": 5
}
},
"upgrade_configs": [
{
"upgrade_type": "websocket"
}
],
"use_remote_address": true
}
}
]
}
]
}
]
Check the proxy-status and see its output:
$ istioctl proxy-status istio-ingressgateway-6755b9bbf6-mx6f8.istio-system
Stderr when execute [/usr/local/bin/pilot-discovery request GET /debug/config_dump?proxyID=istio-ingressgateway-6755b9bbf6-mx6f8.istio-system ]: gc 1 @0.327s 0%: 0.033+1.7+1.6 ms clock, 0.13+0/1.5/1.3+6.6 ms cpu, 4->4->1 MB, 5 MB goal, 4 P
gc 2 @0.340s 1%: 0.008+1.7+1.8 ms clock, 0.035+0.11/1.3/1.5+7.4 ms cpu, 4->4->2 MB, 5 MB goal, 4 P
Clusters Match
Listeners Match
Routes Match
The pilot’s discovery container doesn’t log out any warnings.
Observe that requests to your cluster hang and report a stream timeout after a while.
$ istioctl version
Version: 1.0.5
GitRevision: c1707e45e71c75d74bf3a5dec8c7086f32f32fad
User: root@6f6ea1061f2b
Hub: docker.io/istio
GolangVersion: go1.10.4
BuildStatus: Clean
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:39:04Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:31:33Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Envoy proxy configuration should be easily configurable via Istio objects. Requests coming from an internal nginx gateway outside the k8s cluster could be recognised as internal requests so x-request-ids don’t get overwritten.