Expected Behavior: apply the patch in the EnvoyFilter and have envoy run the code in the lua filter
Actual Behavior: patch applied, but the code in the lua filter either does not run or somehow has no effect on logs or headers
Steps to reproduce:
- install istio: https://istio.io/docs/setup/install/helm/#tabset-docs-setup-install-helm-1-3-tab (via Helm with Mutual TLS Enabled)
- create the EnvoyFilter (below)
- send a request to an exposed service in your kube cluster
- observe no logs containing “wtf” or “blahblahblah” for the proxy container
- observe the presence of the “x-b3-sampled” header (dump headers to logs from your application)
- iterate on the lua code in the filter, observe that syntax issues are observable in istio-proxy logs, observe that no valid change to header manipulation has any effect on the headers, and no valid change to logging statements has any effect on the logs from the proxy container.
I’ve created an EnvoyFilter like this:
kind: EnvoyFilter
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"networking.istio.io/v1alpha3","kind":"EnvoyFilter","metadata":{"annotations":{},"name":"legacy-mtls-headers","namespace":"sandbox-willmilton"},"spec":{"configPatches":[{"applyTo":"HTTP_FILTER","match":{"context":"SIDECAR_INBOUND","listener":{"filterChain":{"filter":{"name":"envoy.http_connection_manager","subFilter":{"name":"envoy.router"}}},"portNumber":80}},"patch":{"operation":"INSERT_BEFORE","value":{"name":"envoy.lua","typed_config":{"@type":"type.googleapis.com/envoy.config.filter.http.lua.v2.Lua","inline_code":"function envoy_on_request(request_handle)\n print(\"blahblahblahblah\")\n request_handle:headers():remove(\"x-b3-sampled\")\nend\nfunction envoy_on_response(response_handle)\n response_handle:logErr(\"wtf\")\nend\n"}}}}]}}
creationTimestamp: "2020-05-08T17:51:00Z"
generation: 22
name: legacy-mtls-headers
namespace: sandbox-willmilton
resourceVersion: "147545807"
selfLink: /apis/networking.istio.io/v1alpha3/namespaces/sandbox-willmilton/envoyfilters/legacy-mtls-headers
uid: 7a942d60-9154-11ea-ae51-42010a8a017f
spec:
configPatches:
- applyTo: HTTP_FILTER
match:
context: SIDECAR_INBOUND
listener:
filterChain:
filter:
name: envoy.http_connection_manager
subFilter:
name: envoy.router
portNumber: 80
patch:
operation: INSERT_BEFORE
value:
name: envoy.lua
typed_config:
'@type': type.googleapis.com/envoy.config.filter.http.lua.v2.Lua
inline_code: |
function envoy_on_request(request_handle)
print("blahblahblahblah")
request_handle:headers():remove("x-b3-sampled")
end
function envoy_on_response(response_handle)
response_handle:logErr("wtf")
end
… and it’s in the listener for the workload I’m interested in:
{
"name": "10.4.3.56_80",
"address": {
"socketAddress": {
"address": "10.4.3.56",
"portValue": 80
}
},
"filterChains": [
{
"filters": [
{
"name": "envoy.http_connection_manager",
"typedConfig": {
"@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
"statPrefix": "inbound_10.4.3.56_80",
"routeConfig": {
"name": "inbound|80|http-app|valhalla.sandbox-willmilton.svc.cluster.local",
"virtualHosts": [
{
"name": "inbound|http|80",
"domains": [
"*"
],
"routes": [
{
"name": "default",
"match": {
"prefix": "/"
},
"route": {
"cluster": "inbound|80|http-app|valhalla.sandbox-willmilton.svc.cluster.local",
"timeout": "0s",
"maxGrpcTimeout": "0s"
},
"decorator": {
"operation": "valhalla.sandbox-willmilton.svc.cluster.local:80/*"
},
"typedPerFilterConfig": {
"mixer": {
"@type": "type.googleapis.com/istio.mixer.v1.config.client.ServiceConfig",
"mixerAttributes": {
"attributes": {
"destination.service.host": {
"stringValue": "valhalla.sandbox-willmilton.svc.cluster.local"
},
"destination.service.name": {
"stringValue": "valhalla"
},
"destination.service.namespace": {
"stringValue": "sandbox-willmilton"
},
"destination.service.uid": {
"stringValue": "istio://sandbox-willmilton/services/valhalla"
}
}
}
}
}
}
]
}
],
"validateClusters": false
},
"httpFilters": [
{
"name": "istio_authn",
"typedConfig": {
"@type": "type.googleapis.com/istio.envoy.config.filter.http.authn.v2alpha1.FilterConfig",
"policy": {
"peers": [
{
"mtls": {}
}
]
}
}
},
{
"name": "mixer",
"typedConfig": {
"@type": "type.googleapis.com/istio.mixer.v1.config.client.HttpClientConfig",
"transport": {
"networkFailPolicy": {
"policy": "FAIL_CLOSE",
"baseRetryWait": "0.080s",
"maxRetryWait": "1s"
},
"checkCluster": "outbound|15004||istio-policy.istio-system.svc.cluster.local",
"reportCluster": "outbound|15004||istio-telemetry.istio-system.svc.cluster.local",
"reportBatchMaxEntries": 100,
"reportBatchMaxTime": "1s"
},
"serviceConfigs": {
"default": {}
},
"defaultDestinationService": "default",
"mixerAttributes": {
"attributes": {
"context.proxy_version": {
"stringValue": "1.5.1"
},
"context.reporter.kind": {
"stringValue": "inbound"
},
"context.reporter.uid": {
"stringValue": "kubernetes://valhalla-5699495c78-t5vx8.sandbox-willmilton"
},
"destination.ip": {
"bytesValue": "AAAAAAAAAAAAAP//CgQDOA=="
},
"destination.namespace": {
"stringValue": "sandbox-willmilton"
},
"destination.port": {
"int64Value": "80"
},
"destination.uid": {
"stringValue": "kubernetes://valhalla-5699495c78-t5vx8.sandbox-willmilton"
}
}
}
}
},
{
"name": "envoy.cors"
},
{
"name": "envoy.fault"
},
{
"name": "envoy.lua",
"typedConfig": {
"@type": "type.googleapis.com/envoy.config.filter.http.lua.v2.Lua",
"inlineCode": "function envoy_on_request(request_handle)\n print(\"blahblahblahblah\")\n request_handle:headers():remove(\"x-b3-sampled\")\nend\nfunction envoy_on_response(response_handle)\n response_handle:logErr(\"wtf\")\nend\n"
}
},
{
"name": "envoy.router"
}
],
"tracing": {
"clientSampling": {
"value": 100
},
"randomSampling": {
"value": 100
},
"overallSampling": {
"value": 100
}
},
"serverName": "istio-envoy",
"streamIdleTimeout": "0s",
"accessLog": [
{
"name": "envoy.file_access_log",
"typedConfig": {
"@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
"path": "/dev/stdout",
"format": "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"
}
}
],
"useRemoteAddress": false,
"generateRequestId": true,
"forwardClientCertDetails": "APPEND_FORWARD",
"setCurrentClientCertDetails": {
"subject": true,
"dns": true,
"uri": true
},
"upgradeConfigs": [
{
"upgradeType": "websocket"
}
],
"normalizePath": true
}
}
],
"transportSocket": {
"name": "envoy.transport_sockets.tls",
"typedConfig": {
"@type": "type.googleapis.com/envoy.api.v2.auth.DownstreamTlsContext",
"commonTlsContext": {
"tlsCertificates": [
{
"certificateChain": {
"filename": "/etc/certs/cert-chain.pem"
},
"privateKey": {
"filename": "[redacted]"
}
}
],
"validationContext": {
"trustedCa": {
"filename": "/etc/certs/root-cert.pem"
}
},
"alpnProtocols": [
"istio-peer-exchange",
"h2",
"http/1.1"
]
},
"requireClientCertificate": true
}
}
}
],
"deprecatedV1": {
"bindToPort": false
},
"listenerFilters": [
{
"name": "envoy.listener.tls_inspector"
}
],
"listenerFiltersTimeout": "0.100s",
"continueOnListenerFiltersTimeout": true,
"trafficDirection": "INBOUND"
}
… but no headers are modified, and the logging statement has no effect. I even checked the envoy config_dump directly, and it’s the same.
Is there some mechanism that precludes the lua filter from running, or that reverts headers after running it? Are headers perhaps getting passed through before my filter runs?
I installed istio with the directions here: https://istio.io/docs/setup/install/helm/#tabset-docs-setup-install-helm-1-3-tab (Mutual TLS Enabled). Is it possible that this setup configures mixer and the use of the mixer filters in some way that is incompatible with lua filters? I’ve tried putting the lua filter at every possible position in the filter chain: if I try to use INSERT_FIRST then the filter is not present, and if I try to use INSERT_AFTER with subfilter envoy.router, I get an error saying that envoy.router must be the last filter. Aside from those impossible positions of the lua filter, I’ve tried every position, and the behavior is the same: no logs, no change to headers.
The request is getting to the application I’m running, and I’m dumping all the headers with it.