EnvoyFilter for access_log

Config
1

Hi,

i want to filter out all 2xx responses from the access log.

I have oriented myself on the examples on: https://github.com/istio/istio/wiki/EnvoyFilter-Samples#tracing-and-access-logging

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: filter-status-code
spec:
  configPatches:
    - applyTo: NETWORK_FILTER
      match:
	context: ANY
	listener:
	  filterChain:
	    filter:
	      name: "envoy.http_connection_manager"
      patch:
	operation: MERGE
	value:
	  config:
	    access_log:
	      - name: envoy.file_access_log
	        typed_config:
	          "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog"
	          path: /dev/stdout
	          format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"
	        filter:
	          status_code_filter:
	            comparison:
	              op: GE
	              value:
	                runtime_key: "400"

Before applying EnvoyFilter:

"access_log": [
  {
   "name": "envoy.file_access_log",
   "typed_config": {
    "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
    "path": "/dev/stdout",
    "format": "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"
   }
  }
]

After applying EnvoyFilter:

"access_log": [
  {
   "name": "envoy.file_access_log",
   "typed_config": {
    "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
    "path": "/dev/stdout",
    "format": "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"
   }
  },
  {
   "name": "envoy.file_access_log",
   "filter": {
    "status_code_filter": {
     "comparison": {
      "op": "GE",
      "value": {
       "runtime_key": "400"
      }
     }
    }
   },
   "typed_config": {
    "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
    "path": "/dev/stdout",
    "format": "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"
   }
  }
]

I expected to get only one access_log entry, because i am using the MERGE operation . But the generated configuration contains two access_log entries.

Can anyone suggest what am I be doing wrong.

2

It is working as per the spec, in that arrays are merged [protobuf merge semantics - where arrays are appended to]. That said, it brings up a deficiency in the API - the need to replace fields in an array field or replace an entire array field. Could you file an issue in istio/api to enhance the API ?

3

Has there been any progress in this area?

4

In case anyone still lands here, the Telemetry configuration resolves this situation. Istio / Telemetry

CEL expression for selecting when requests/connections should be logged.

Examples:

response.code >= 400
connection.mtls && request.url_path.contains('v1beta3')
5

How can I specify log format with Telemetry API?