EnvoyFilter with negative port match or mixing tcp_proxy filter with differenet ports

Networking
1

I have a service making outbound calls to another service with some strange TCP timeout behavior.

Someone (not me) worked around this with the following EnvoyFilter

  - applyTo: NETWORK_FILTER
    match:
      context: SIDECAR_OUTBOUND
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.tcp_proxy
    patch:
      operation: MERGE
      value:
        name: envoy.filters.network.tcp_proxy
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          idle_timeout: 30s
          max_connect_attempts: 3

But that idle_timeout is also causing problems for some other TCP connections such as connection to redis.

How can I give a list of ports for which this idle_timeout:30s will not take effect, or stack multiple configs with different values?

I have tried this but it does not work, the port 6379 does not show up in envoy config dump when I run istioctl dashboard envoy.

  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: SIDECAR_OUTBOUND
      listener:
        portNumber: 6379
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.network.tcp_proxy
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          idle_timeout: 4320h
          max_connect_attempts: 3
  - applyTo: NETWORK_FILTER
    match:
      context: SIDECAR_OUTBOUND
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.tcp_proxy
    patch:
      operation: MERGE
      value:
        name: envoy.filters.network.tcp_proxy
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          idle_timeout: 30s
          max_connect_attempts: 3