Istio JWT authentication does not seem to be working

Hello,

I am trying to configure JWT authentication on an istio-ingress gateway.

Here is how my config looks like:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: my-authentication
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  jwtRules:
  - issuer: "https://auth.dev.my.com"
    jwks: "{\"keys\":[{\"kty\":\"EC\",\"crv\":\"P-521\",\"kid\":\"d73c9314db8467a44a47c8492832e4eecfe1f05a\",\"x\":\"AHjB1n3AJ28NTI_sd-d2WS3HqY62tyyf2WQdmxJ25cQ_FjSuoi3OZ2iUFnIb_Io0iLpfdzK1pWXOG3wFIy8PCxE8\",\"y\":\"Aa9sQ-fElPyNdYWxszkUFIs0s5Cr-E2nDAb-I0UCM8Iw7vocN5tWSqZggiSN_Gjw5kykdjpHCjlZwvQRToU2Sl_z\"},{\"kty\":\"EC\",\"crv\":\"P-521\",\"kid\":\"a442711b9e2c722ed0c3d7daf0a04fd36961ef5f\",\"x\":\"ADptQRR6Bq0yWh3jxskEGRzY6-gBde0PbXwlN74zDpxX5EcX1gIKYUfpvacI05pEaazujh7WNU_XyGvwbJ_7XwSa\",\"y\":\"AQLhtYL_rnOjoVlxRq4H-lPFR8HokJsJ5q9iYTwD8HL4Dm25S52MyNE694yj_RmOPi59vH6aYjaPD-UTWMWRp0Z8\"}]}"
    outputPayloadToHeader: X-My-Auth-Payload
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: my-authentication-default
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  action: ALLOW
  rules:
  - when:
    - key: request.auth.claims[iss]
      values:
        - https://auth.dev.my.com

I am making a request with a valid JWT in access_token http-only cookie which is transformed into an Authorization header by the following EnvoyFilter:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: my-auth-token
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
        portNumber: 8443
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.lua
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
          inlineCode: |
            function stringSplit(inputstr, sep)
              if sep == nil then
                sep = "%s"
              end
              local t={}
              for str in string.gmatch(inputstr, "([^"..sep.."]+)") do
                table.insert(t, str)
              end
              return t
            end

            function envoy_on_request(handle) 
              headers = handle:headers()

              path = headers:get(":path")
              if path == "/health" or path == "/metrics" then
                return
              end

              cookieString = headers:get("cookie")
              if cookieString ~= nil then
                splitCookieString = stringSplit(cookieString, ";")
                
                jwt = nil
                for i, cookieItem in ipairs(splitCookieString) do
                  if string.find(cookieItem, "access_token") ~= nil then
                    jwt = string.gsub(cookieItem, "access_token=", "")
                  end
                end

                if jwt ~= nil then
                  token = string.gsub(jwt, "^ ", "")
                  headers:replace("Authorization", "Bearer: "..token)
                end
              end
            end

However, for every request, I keep getting 403 Forbidden with the following in the body:

RBAC: access denied

I am not able to find any logs why this is happening.
Only potential clue I did find is this error message in the discovery container of istiod:

error	authorization	skipped rule ns[istio-system]-policy[my-authentication-default]-rule[0]: request.auth.claims[iss] must not be used in TCP

I have spent quite a few hours on this. Would really appreciate any help I can get here.

Thanks!

some quick thing to check, make sure the Lua filter is inserted before the istio_authn and Envoy jwt filter to make sure the JWT token will be validated by Istio.

Also it will be very useful to get the Envoy log and config dump to help the debug, see Istio / Security Problems

@YangminZhu how can I find out if Lua filter is inserted before all the other filters?
Also, where can I get Envoy log from?

Thanks a lot for your help :slight_smile:

You can follow the link in my last reply to get the envoy config dump, it can also tell if your current EnvoyFilter inserts the Lua filter before others, I’m not sure exactly how to do but I think the EnvoyFilter has a field to specify the other filter name to insert before.

Thanks @YangminZhu !

I just verified that the Lua filter to transform Cookie to Authorization header is inserted before all the other filters.

Here is the exact order:

- envoy.filters.http.lua # the one transforming Cookie to Authorization header
- istio.metadata_exchange
- envoy.filters.http.jwt_authn
- istio_authn
- envoy.filters.http.rbac
- envoy.filters.http.cors
- envoy.filters.http.fault
- istio.stats
- envoy.filters.http.router

Here is the relevant portion from the istio-proxy dump of the istio ingressgateway:

{
  "name": "0.0.0.0_8443",
  "active_state": {
  "version_info": "2021-01-15T23:07:54Z/1",
  "listener": {
    "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
    "name": "0.0.0.0_8443",
    "address": {
    "socket_address": {
      "address": "0.0.0.0",
      "port_value": 8443
    }
    },
    "filter_chains": [
    {
      "filter_chain_match": {
      "server_names": [
        "*.dev.my.com",
        "*.preprod.my.com",
        "*.preview.my.com",
        "dev.my.com",
        "preprod.my.com",
        "preview.my.com"
      ]
      },
      "filters": [
      {
        "name": "envoy.filters.network.http_connection_manager",
        "typed_config": {
        "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
        "stat_prefix": "outbound_0.0.0.0_8443",
        "rds": {
          "config_source": {
          "ads": {},
          "resource_api_version": "V3"
          },
          "route_config_name": "https.443.https.my-gateway.istio-system"
        },
        "http_filters": [
          {
          "name": "envoy.filters.http.lua",
          "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua",
            "inline_code": "function stringSplit(inputstr, sep)\n  if sep == nil then\n    sep = \"%s\"\n  end\n  local t={}\n  for str in string.gmatch(inputstr, \"([^\"..sep..\"]+)\") do\n    table.insert(t, str)\n  end\n  return t\nend\n\nfunction envoy_on_request(handle) \n  headers = handle:headers()\n\n  path = headers:get(\":path\")\n  if path == \"/health\" or path == \"/metrics\" then\n    return\n  end\n\n  cookieString = headers:get(\"cookie\")\n  if cookieString ~= nil then\n    splitCookieString = stringSplit(cookieString, \";\")\n    \n    jwt = nil\n    for i, cookieItem in ipairs(splitCookieString) do\n      if string.find(cookieItem, \"access_token\") ~= nil then\n        jwt = string.gsub(cookieItem, \"access_token=\", \"\")\n      end\n    end\n\n    if jwt ~= nil then\n      token = string.gsub(jwt, \"^ \", \"\")\n      headers:replace(\"Authorization\", \"Bearer: \"..token)\n    end\n  end\nend\n"
          }
          },
          {
          "name": "istio.metadata_exchange",
          "typed_config": {
            "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
            "type_url": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
            "value": {
            "config": {
              "vm_config": {
              "runtime": "envoy.wasm.runtime.null",
              "code": {
                "local": {
                "inline_string": "envoy.wasm.metadata_exchange"
                }
              }
              },
              "configuration": {
              "@type": "type.googleapis.com/google.protobuf.StringValue",
              "value": "{}\n"
              }
            }
            }
          }
          },
          {
          "name": "envoy.filters.http.jwt_authn",
          "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication",
            "providers": {
            "origins-0": {
              "issuer": "https://auth.dev.my.com",
              "local_jwks": {
              "inline_string": "{\"keys\":[{\"kty\":\"EC\",\"crv\":\"P-521\",\"kid\":\"d73c9314db8467a44a47c8492832e4eecfe1f05a\",\"x\":\"AHjB1n3AJ28NTI_sd-d2WS3HqY62tyyf2WQdmxJ25cQ_FjSuoi3OZ2iUFnIb_Io0iLpfdzK1pWXOG3wFIy8PCxE8\",\"y\":\"Aa9sQ-fElPyNdYWxszkUFIs0s5Cr-E2nDAb-I0UCM8Iw7vocN5tWSqZggiSN_Gjw5kykdjpHCjlZwvQRToU2Sl_z\"},{\"kty\":\"EC\",\"crv\":\"P-521\",\"kid\":\"a442711b9e2c722ed0c3d7daf0a04fd36961ef5f\",\"x\":\"ADptQRR6Bq0yWh3jxskEGRzY6-gBde0PbXwlN74zDpxX5EcX1gIKYUfpvacI05pEaazujh7WNU_XyGvwbJ_7XwSa\",\"y\":\"AQLhtYL_rnOjoVlxRq4H-lPFR8HokJsJ5q9iYTwD8HL4Dm25S52MyNE694yj_RmOPi59vH6aYjaPD-UTWMWRp0Z8\"}]}"
              },
              "forward_payload_header": "X-My-Auth-Payload",
              "payload_in_metadata": "https://auth.dev.my.com"
            }
            },
            "rules": [
            {
              "match": {
              "prefix": "/"
              },
              "requires": {
              "requires_any": {
                "requirements": [
                {
                  "provider_name": "origins-0"
                },
                {
                  "allow_missing": {}
                }
                ]
              }
              }
            }
            ]
          }
          },
          {
          "name": "istio_authn",
          "typed_config": {
            "@type": "type.googleapis.com/istio.envoy.config.filter.http.authn.v2alpha1.FilterConfig",
            "policy": {
            "origins": [
              {
              "jwt": {
                "issuer": "https://auth.dev.my.com"
              }
              }
            ],
            "origin_is_optional": true,
            "principal_binding": "USE_ORIGIN"
            },
            "skip_validate_trust_domain": true
          }
          },
          {
          "name": "envoy.filters.http.rbac",
          "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBAC",
            "rules": {
            "policies": {
              "ns[istio-system]-policy[my-authentication-default]-rule[0]": {
              "permissions": [
                {
                "and_rules": {
                  "rules": [
                  {
                    "any": true
                  }
                  ]
                }
                }
              ],
              "principals": [
                {
                "and_ids": {
                  "ids": [
                  {
                    "or_ids": {
                    "ids": [
                      {
                      "metadata": {
                        "filter": "istio_authn",
                        "path": [
                        {
                          "key": "request.auth.claims"
                        },
                        {
                          "key": "iss"
                        }
                        ],
                        "value": {
                        "list_match": {
                          "one_of": {
                          "string_match": {
                            "exact": "https://auth.dev.my.com"
                          }
                          }
                        }
                        }
                      }
                      }
                    ]
                    }
                  }
                  ]
                }
                }
              ]
              }
            }
            }
          }
          },
          {
          "name": "envoy.filters.http.cors",
          "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors"
          }
          },
          {
          "name": "envoy.filters.http.fault",
          "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
          }
          },
          {
          "name": "istio.stats",
          "typed_config": {
            "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
            "type_url": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
            "value": {
            "config": {
              "root_id": "stats_outbound",
              "vm_config": {
              "vm_id": "stats_outbound",
              "runtime": "envoy.wasm.runtime.null",
              "code": {
                "local": {
                "inline_string": "envoy.wasm.stats"
                }
              }
              },
              "configuration": {
              "@type": "type.googleapis.com/google.protobuf.StringValue",
              "value": "{\n  \"debug\": \"false\",\n  \"stat_prefix\": \"istio\",\n  \"disable_host_header_fallback\": true\n}\n"
              }
            }
            }
          }
          },
          {
          "name": "envoy.filters.http.router",
          "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
          }
          }
        ],
        "tracing": {
          "client_sampling": {
          "value": 100
          },
          "random_sampling": {
          "value": 1
          },
          "overall_sampling": {
          "value": 100
          }
        },
        "http_protocol_options": {},
        "server_name": "istio-envoy",
        "use_remote_address": true,
        "generate_request_id": true,
        "forward_client_cert_details": "SANITIZE_SET",
        "set_current_client_cert_details": {
          "subject": true,
          "cert": true,
          "dns": true,
          "uri": true
        },
        "upgrade_configs": [
          {
          "upgrade_type": "websocket"
          }
        ],
        "stream_idle_timeout": "0s",
        "normalize_path": true
        }
      }
      ],
      "transport_socket": {
      "name": "envoy.transport_sockets.tls",
      "typed_config": {
        "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
        "common_tls_context": {
        "alpn_protocols": [
          "h2",
          "http/1.1"
        ],
        "tls_certificate_sds_secret_configs": [
          {
          "name": "my-com-tls",
          "sds_config": {
            "api_config_source": {
            "api_type": "GRPC",
            "grpc_services": [
              {
              "google_grpc": {
                "target_uri": "unix:./var/run/ingress_gateway/sds",
                "stat_prefix": "sdsstat"
              }
              }
            ],
            "transport_api_version": "V3"
            },
            "resource_api_version": "V3"
          }
          }
        ]
        },
        "require_client_certificate": false
      }
      }
    }
    ],
    "listener_filters": [
    {
      "name": "envoy.filters.listener.tls_inspector",
      "typed_config": {
      "@type": "type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector"
      }
    }
    ],
    "traffic_direction": "OUTBOUND"
  },
  "last_updated": "2021-01-16T04:23:19.773Z"
  }
}

The filter config looks correct to me, the authz is also configured correctly. Could you include the debug logging in the Envoy, I wonder could it be due to the authorization header somehow not generated correctly?

You can follow this Istio / Security Problems to enable the debug logging.

@YangminZhu

Here are the logs after enabling debugging. It certainly appears to be ordering issue.
However, I am really confused about this now since lua filter is registered as the first filter but does not seem to execute in that order.

1/25/2021 10:36:53 PM ':authority', 'api.dev.my.com'
1/25/2021 10:36:53 PM ':path', '/api/endpoint'
1/25/2021 10:36:53 PM ':method', 'GET'
1/25/2021 10:36:53 PM 'user-agent', 'PostmanRuntime/7.26.8'
1/25/2021 10:36:53 PM 'accept', '*/*'
1/25/2021 10:36:53 PM 'postman-token', 'e0935561-8a9d-41cb-b933-bb1783f11dd5'
1/25/2021 10:36:53 PM 'accept-encoding', 'gzip, deflate, br'
1/25/2021 10:36:53 PM 'connection', 'keep-alive'
1/25/2021 10:36:53 PM 'cookie', 'access_token=my_valid_jwt_token'
1/25/2021 10:36:53 PM 
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410668Z	debug	envoy http	[C1208248][S8236170745172492049] request end stream
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410716Z	debug	envoy jwt	Called Filter : setDecoderFilterCallbacks
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410825Z	debug	envoy lua	coroutine finished
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410854Z	debug	envoy jwt	Called Filter : decodeHeaders
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410861Z	debug	envoy jwt	Prefix requirement '/' matched.
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410870Z	debug	envoy jwt	extract authorizationBearer
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410877Z	debug	envoy jwt	origins-0: JWT authentication starts (allow_failed=false), tokens size=0
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410881Z	debug	envoy jwt	origins-0: JWT token verification completed with: Jwt is missing
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410887Z	debug	envoy jwt	Called AllowMissingVerifierImpl.verify : verify
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410892Z	debug	envoy jwt	extract authorizationBearer
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410896Z	debug	envoy jwt	_IS_ALLOW_MISSING_: JWT authentication starts (allow_failed=false), tokens size=0
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410900Z	debug	envoy jwt	_IS_ALLOW_MISSING_: JWT token verification completed with: Jwt is missing
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410904Z	debug	envoy jwt	Called Filter : check complete OK
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410934Z	debug	envoy filter	AuthenticationFilter::decodeHeaders with config
1/25/2021 10:36:53 PM policy {
1/25/2021 10:36:53 PM   origins {
1/25/2021 10:36:53 PM     jwt {
1/25/2021 10:36:53 PM       issuer: "https://auth.dev.my.com"
1/25/2021 10:36:53 PM     }
1/25/2021 10:36:53 PM   }
1/25/2021 10:36:53 PM   origin_is_optional: true
1/25/2021 10:36:53 PM   principal_binding: USE_ORIGIN
1/25/2021 10:36:53 PM }
1/25/2021 10:36:53 PM skip_validate_trust_domain: true
1/25/2021 10:36:53 PM 
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410947Z	debug	envoy filter	No method defined. Skip source authentication.
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410955Z	debug	envoy filter	Validating request path /api/endpoint for jwt issuer: "https://auth.dev.my.com"
1/25/2021 10:36:53 PM 
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410959Z	debug	envoy filter	No dynamic_metadata found for filter envoy.filters.http.jwt_authn
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410963Z	debug	envoy filter	No dynamic_metadata found for filter jwt-auth
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410965Z	debug	envoy filter	Origin authenticator failed
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.410972Z	debug	envoy filter	Saved Dynamic Metadata:
1/25/2021 10:36:53 PM 
1/25/2021 10:36:53 PM 2021-01-26T06:36:53.411019Z	debug	envoy rbac	checking request: requestedServerName: api.dev.my.com, sourceIP: 10.10.32.4:62363, directRemoteIP: 10.10.32.4:62363, remoteIP: 10.10.32.4:62363,localAddress: 10.10.34.209:8443, ssl: uriSanPeerCertificate: , dnsSanPeerCertificate: , subjectPeerCertificate: , headers: ':authority', 'api.dev.my.com'
1/25/2021 10:36:53 PM ':path', '/api/endpoint'
1/25/2021 10:36:53 PM ':method', 'GET'
1/25/2021 10:36:53 PM 'user-agent', 'PostmanRuntime/7.26.8'
1/25/2021 10:36:53 PM 'accept', '*/*'
1/25/2021 10:36:53 PM 'postman-token', 'e0935561-8a9d-41cb-b933-bb1783f11dd5'
1/25/2021 10:36:53 PM 'accept-encoding', 'gzip, deflate, br'
1/25/2021 10:36:53 PM 'cookie', 'access_token=my_valid_jwt_token'
1/25/2021 10:36:53 PM 'x-forwarded-for', '10.10.32.4'
1/25/2021 10:36:53 PM 'x-forwarded-proto', 'https'
1/25/2021 10:36:53 PM 'x-envoy-internal', 'true'
1/25/2021 10:36:53 PM 'x-request-id', '907506c1-4fe7-4585-a744-88343c564af4'
1/25/2021 10:36:53 PM 'x-envoy-decorator-operation', 'my-pod.my-ns.svc.cluster.local:8080/api/endpoint/*'
1/25/2021 10:36:53 PM 'authorization', 'Bearer: my_valid_jwt_token'
1/25/2021 10:36:53 PM 'x-envoy-peer-metadata', 'envoy_peer_metadata'
1/25/2021 10:36:53 PM 'x-envoy-peer-metadata-id', 'router~10.10.34.209~istio-ingressgateway-b755f965d-nsdt2.istio-system~istio-system.svc.cluster.local'
1/25/2021 10:36:53 PM , dynamicMetadata: filter_metadata {
1/25/2021 10:36:53 PM   key: "istio_authn"
1/25/2021 10:36:53 PM   value {
1/25/2021 10:36:53 PM   }
1/25/2021 10:36:53 PM }

Solved in Istio JWT authentication fails even for valid JWT tokens · Issue #30140 · istio/istio · GitHub.