Istio Envoy (EnvoyFilter) skipping ext_authz filter (Istio 1.5.5)

Networking
1

Hi, I am trying to implement ext_authz filter via EnvoyFilter in Istio 1.5.5. The configuration is already in the envoy (checking via envoy sidecar dashboard)

*Istio 1.5.5
*Sidecar Envoy (check is 1.13.1-dev)

"name": "envoy.http_connection_manager",
               "typed_config": {
                "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
                "stat_prefix": "inbound_10.14.51.153_5000",
                "route_config": {
                 "name": "inbound|5000|http|helloworld.s-aws-istio-testing.svc.cluster.local",
                 "virtual_hosts": [
                  {
                   "name": "inbound|http|5000",
                   "domains": [
                    "*"
                   ],
                   "routes": [
                    {
                     "match": {
                      "prefix": "/"
                     },
                     "route": {
                      "cluster": "inbound|5000|http|helloworld.s-aws-istio-testing.svc.cluster.local",
                      "timeout": "0s",
                      "max_grpc_timeout": "0s"
                     },
                     "decorator": {
                      "operation": "helloworld.s-aws-istio-testing.svc.cluster.local:5000/*"
                     },
                     "name": "default"
                    }
                   ]
                  }
                 ],
                 "validate_clusters": false
                },
                "http_filters": [
                 {
                  "name": "envoy.ext_authz",
                  "typed_config": {
                   "@type": "type.googleapis.com/envoy.config.filter.http.ext_authz.v2.ExtAuthz",
                   "grpc_service": {
                    "envoy_grpc": {
                     "cluster_name": "patched.grpc-server.s-aws-istio-testing.svc.cluster.local"
                    },
                    "timeout": "0.500s"
                   }
                  }

but the log from the sidecar informs that it’s skipping the ext_authz filter (I activate debug mode on filter).

[Envoy (Epoch 0)] [2020-07-12 15:19:27.093][34][debug][filter] [src/envoy/http/authn/http_filter.cc:49] Called AuthenticationFilter : onDestroy

[Envoy (Epoch 0)] [2020-07-12 15:21:15.194][32][debug][filter] [external/envoy/source/extensions/filters/listener/original_dst/original_dst.cc:18] original_dst: New connection accepted
    [Envoy (Epoch 0)] [2020-07-12 15:21:15.194][32][debug][filter] [external/envoy/source/extensions/filters/listener/tls_inspector/tls_inspector.cc:78] tls inspector: new connection accepted
    [Envoy (Epoch 0)] [2020-07-12 15:21:15.195][32][debug][filter] [src/envoy/http/authn/http_filter.cc:55] AuthenticationFilter::decodeHeaders with config
    policy {
      peers {
        mtls {
          mode: PERMISSIVE
        }
      }
    }

[Envoy (Epoch 0)] [2020-07-12 15:21:15.195][32][debug][filter] [src/envoy/http/authn/authenticator_base.cc:98] [C53] validateX509 mode PERMISSIVE: ssl=false, has_user=false
[Envoy (Epoch 0)] [2020-07-12 15:21:15.195][32][debug][filter] [src/envoy/http/authn/filter_context.cc:43] Payload has not peer authentication data
[Envoy (Epoch 0)] [2020-07-12 15:21:15.195][32][debug][filter] [src/envoy/http/authn/filter_context.cc:61] Set principal from peer:
[Envoy (Epoch 0)] [2020-07-12 15:21:15.195][32][debug][filter] [src/envoy/http/authn/origin_authenticator.cc:106] Origin authenticator succeeded
[Envoy (Epoch 0)] [2020-07-12 15:21:15.195][32][debug][filter] [src/envoy/http/authn/http_filter.cc:88] Saved Dynamic Metadata:

[Envoy (Epoch 0)] [2020-07-12 15:21:15.469][32][debug][filter] [src/envoy/http/authn/http_filter.cc:49] Called AuthenticationFilter : onDestroyThe configuration

There is no packet flowing to the grpc-server service(external authorization service) when I try to sniff the helloworld sidecar.

There is the manifest that I used to create the EnvoyFilter

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: helloworld-ef
  namespace: s-aws-istio-testing
spec:
  workloadSelector:
    labels:
      app: helloworld
      version: v1
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        portNumber: 5000
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: INSERT_BEFORE
      value:
        name: "envoy.ext_authz"
        typed_config:
          "@type": type.googleapis.com/envoy.config.filter.http.ext_authz.v2.ExtAuthz
          failure_mode_allow: false
          grpc_service:
            timeout: 0.5s
            envoy_grpc:
              cluster_name: patched.grpc-server.s-aws-istio-testing.svc.cluster.local
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ext-authz-filter-cluster-patch
  namespace: s-aws-istio-testing
spec:
  workloadLabels:
    app: helloworld
    version: v1
  configPatches:
  - applyTo: CLUSTER
    match:
      cluster:
        service: grpc-server.s-aws-istio-testing.svc.cluster.local
    patch:
      operation: MERGE
      value:
        name: "patched.grpc-server.s-aws-istio-testing.svc.cluster.local"

Anyone know why the filter is not working?

2

@Zufar_Dhiyaulhaq did you get this to work?

3

yes, the problem is in the envoyFilter. you can follow something like this

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: <envoy-filter-name>-ef
  namespace: <namespace>
spec:
  workloadSelector:
    labels:
      <key>: <value>
      <key>: <value>
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
            subFilter:
              name: "envoy.router"
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.ext_authz
        typed_config:
          "@type": type.googleapis.com/envoy.config.filter.http.ext_authz.v2.ExtAuthz
          failure_mode_allow: <false|true>
          grpc_service:
            envoy_grpc:
              cluster_name: patched.<external-authz-service>.<external-authz-namespace>.svc.cluster.local
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: <envoy-filter-name>-patch-ef
  namespace: <namespace>
spec:
  workloadLabels:
      <key>: <value>
      <key>: <value>
  configPatches:
  - applyTo: CLUSTER
    match:
      cluster:
        service: <external-authz-service>.<external-authz-namespace>.svc.cluster.local
    patch:
      operation: MERGE
      value:
        name: "patched.<external-authz-service>.<external-authz-namespace>.svc.cluster.local"
4

Thanks :).

Be aware that workloadLabels: is no longer supported

it should be something like

  workloadSelector:
    labels:
      istio: ingressgateway

Or

  workloadSelector:
    labels:
      app: istio-ingressgateway
1 Like
5

start with what version Jacob?

6

As far as I can tell since 1.2: Istioldie 1.2 / Envoy Filter
1.3 has no reference of that workloadLabels.
Istioldie 1.3 / Envoy Filter

It seemed to work using workloadLabels except it then didn’t match anything and applied to all sidecars and gateways in the cluster, so my ingressgateway send it to my ext_authz service sidecar which then also tried to find some where to send it, causing it to fail with No Route.

7

First and foremost, thanks! I finally got this working due to your post. What I would like to know (if you can answer that of course), is why do we need the second filter patching the clusters name? For the sake of it, I even renamed the cluster to the exact same name it already had, and it worked. Whereas, if I don’t add the second filter with the MERGE operation, auth requests aren’t routed to my auth service.

I don’t understand this. If the cluster exists and has already been discovered by EDS and shows up in the various services, then why don’t requests get routed to it (nothing shows up in proxy logs). But, if I path the name, use even the same name it already had, then requests are correctly routed there.

Would like to understand this in all honesty. Just baffled by this behaviour.

8

You need it because the | character is invalid in the host header, see more here https://github.com/istio/istio/issues/21841

By the way I’m working on a design to provide first-class support for ext_authz in Istio, the goal is to really improve the user experience that you never need to worry about these tricky internal tech details, and also don’t need to use the EnvoyFilter CRD (hard to use and troubleshoot). You can take a preview of the design doc (still work-in-progress) here: https://docs.google.com/document/d/1V4mCQCw7mlGp0zSQQXYoBdbKMDnkPOjeyUb85U07iSI/edit#

3 Likes
9

Brilliant! Thanks for explaining. Hadn’t seen that issue yet, but makes perfect sense!