Adding custom response headers using Istio's (1.6.0) envoy lua filter

I am running Istio 1.6.0. I wanted to add some custom headers to all the outbound responses originating from my service. So I was trying to use lua envoyfilter to achieve that. However, I don’t see my proxy getting properly configured.

The envoy filter config that I’m trying to use is

kind: EnvoyFilter
metadata:
  name: lua-filter
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
            subFilter:
              name: "envoy.router"
    patch:
      operation: INSERT_BEFORE
      value:
       name: envoy.lua
       typed_config:
         "@type": "type.googleapis.com/envoy.config.filter.http.lua.v2.Lua"
         inlineCode: |
            function envoy_on_response(response_handle)
                response_handle:logInfo(" ========= XXXXX ========== ""); 
                response_handle:headers():add("X-User-Header", "worked");
            end

I do have my ingress-gateway pods running in the istio-system namespace

❯ kgp -l istio=ingressgateway -n istio-system
NAME                              READY   STATUS    RESTARTS   AGE
ingress-gateway-b4b5cffc9-wz75r   1/1     Running   0          3d12h
ingress-gateway-b4b5cffc9-znx9b   1/1     Running   0          28h

I was hoping that I would see X-User-Header when I curl for my service. Unfortunately, I’m not seeing any custom headers.

I tried checking the proxy-configs of the ingress-gateway pod in the istio-system, and I don’t see the envoy.lua configured at all. I’m not sure whether I’m debugging it correctly.

 istioctl proxy-config listener ingress-gateway-b4b5cffc9-wz75r.istio-system  -n istio-system --port 443 -o json | grep "name"
        "name": "0.0.0.0_443",
                        "name": "istio.stats",
                        "name": "envoy.tcp_proxy",
                        "name": "istio.stats",
                        "name": "envoy.tcp_proxy",
                "name": "envoy.listener.tls_inspector",

Please let me know what is that I’m missing or incorrectly configured. Any advice on how to debug further also would be really helpful.

Thank you so much.

I’m not really sure which log level your ingress is running, but I would definitely check its log to make sure its running your filter… I see you have a logInfo, if that’s the case, try logCritial or other levels, and make sure the line is getting printed on your log to make sure your filter is being run.

Thanks for the suggestion. Moved the log from info to warn. Still no luck. Any advise/suggestion to debug further?

I would change to logError, and I would uninstall it, use kubectl logs -f on istiod, and insert it in parallel to see if anything is showing up on istiod logs while the filter is being installed.
First thing is to make sure you’re filter is really getting installed.
Second thing: make sure your “test call” is really going through the gateway and matching the your configuration. I’ve done this in the past but it was a different version, and the matching was a little bit different, but in my case I had to specify the port as well: