JWKS Parse Error

jammerful · October 18, 2019, 10:43pm

Hello,

I’m trying to upgrade from 1.2.2 to a new version, preferably to 1.3.3. However, when I try to upgrade, I’m seeing an error on the istio proxy sidecar starting properly due to a mysterious JWKS parse error. I’m seeing this error on versions 1.2.7 and 1.3.3:

[2019-10-18 22:15:55.856][33][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, no healthy upstream
[2019-10-18 22:15:55.856][33][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:50] Unable to establish new stream
[2019-10-18 22:15:56.947][33][warning][config] [external/envoy/source/common/config/grpc_mux_subscription_impl.cc:81] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) 10.110.123.108_8443: Issuer 'https://pingfederate.mycorp.io:5555' in jwt_authn config has invalid local jwks: Jwks EC [x] or [y] field is missing or has a parse error., virtualInbound: Issuer 'https://pingfederate.mycorp.io:5555' in jwt_authn config has invalid local jwks: Jwks EC [x] or [y] field is missing or has a parse error.

I checked the JWKS and all keys with kty = “EC” have both an “x” and “y” field set. Beyond this my OIDC provider is PingFederate which I doubt it serving up an invalid JWKS, and this JWKS works on 1.2.2.

Thanks for any help

jammerful · October 29, 2019, 2:51pm

If anyone could help me out here, I’d appreciate it. Right now I’m unable to upgrade.

JimmyChen · December 3, 2019, 10:14pm

If you rollback to 1.2.2, does the error still exist? @YangminZhu

YangminZhu · December 3, 2019, 10:47pm

Istio 1.3 changes to use the upstream Envoy JWT filter which is following the JWKS standard more strictly and this is possibly causing the issue.

Could you share the jwks to reproduce the issue?

jammerful · December 3, 2019, 11:11pm

Yes if I rollback to 1.2.2 I don’t have an issue. @YangminZhu

I can share a redacted version of the JWKS, is that okay? Note this is from PingFederate, which is a pretty standard enterprise IdP. Thank you for the help

jammerful · December 4, 2019, 7:02pm

@YangminZhu Here is the redacted JWKS, I’ll work on getting approval to send the full JWKS to you. It would preferable if I could send you personal message/email instead of posting the full one on here. Please let me know if you need any additional information, alternatively I can help reproduce the issue myself.

  {
  "keys": [
    {
      "kty": "RSA",
      "kid": "REDACTED",
      "use": "sig",
      "n": "REDACTED",
      "e": "REDACTED",
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y": "REDACTED",
      "crv": "P-521"
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y": "REDACTED",
      "crv": "P-384"
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y": "REDACTED",
      "crv": "P-256"
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y": "REDACTED",
      "crv": "P-256"
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y": "REDACTED",
      "crv": "P-521"
    },
    {
      "kty": "RSA",
      "kid": "REDACTED",
      "use": "sig",
      "n": "REDACTED",
      "e": "REDACTED"
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y": "REDACTED",
      "crv": "P-384"
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y":"REDACTED",
      "crv": "P-521"
    },
    {
      "kty": "RSA",
      "kid": "REDACTED",
      "use": "sig",
      "n": "REDACTED",
      "e": "REDACTED"
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y": "REDACTED",
      "crv": "P-384"
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y": "REDACTED",
      "crv": "P-256"
    }
  ]
}

Michael_Whittlestone · December 5, 2019, 12:05pm

Having same problem with Istio and PingFederate. Have found out that copying the JWKS to a new location without the EC’s works fine (at least for a few hours until the keys rotate). Our IdTokens are RSA signed

jammerful · December 5, 2019, 8:12pm

@YangminZhu Here is the full JWKS

{"keys":[{"kty":"EC","kid":"Ozxgl1WsWyQBF1lcbYyjYl","use":"sig","x":"Vq3vKJCeJyihZIauouQp3eFmuYLCsEcID_sGigAx2gs","y":"3UaPoda-HCrJeU4i960qFHMschTdmqjkitiIyYc5svA","crv":"P-256"},{"kty":"EC","kid":"RBsu7DAEUcrgx4X91TVrsY","use":"sig","x":"dQ_SlIctw4nWZEJ2rasiyrps7jXuxr1E81zw4X-aaY1LLRhcaUxpInytXeZK5mOm","y":"xrhQlZQZeDWm7VRHThevLTKqOkqfNwTmHL7P_f23BPn8SLlXd9p1jS4LzL0KK0rL","crv":"P-384"},{"kty":"RSA","kid":"X3oArm2sGh5pO0jWl5O41C","use":"sig","n":"lOTXC2Pfi0dajzDAOERQLTtT0_GbjAFaNflzV-0tWxban1CPEN0n5UG5z-c0KxKY6fhZshQ1Opr3VbQmE1MGSeYf3qEUD4Th3ZubVV_2Yhuio-UZXllz1EGgAh7sC9TzQi84jIYj_mhdno4l6Y3FVlvM6VtNYRGPDNgjRoVfd63vocXAqgUok6LpEcL9MbrvmK-hFPVNX7euGN_xm_qZM5-JrJMnKz6shnjrul7yZ-ZExzfFx_LSBqum-fkKv2FfEoJqyzyVlbabmDUZ81B9ZP0nfaP3e-IRSQECuXf52PfHqEgZbrax8hKAfpZKJX643gxSCnBWdmNO-BFSlho4pw","e":"AQAB"},{"kty":"EC","kid":"sxG_WeuLxIKXoVit-8vyQf","use":"sig","x":"AG3w2vYgVbn4E27rkxZPUVrzLWhMctY5GOP6xygLLFwNRaoOx2gnlQPwAsEXHxz80u5lfmOms0pJSjuDrNqs5pB4","y":"Ad0K-hbFmTVj3nMOw7jAdl21dlU35pG1g7h_Tswr0VYfxqg4ubIPyXrrtmlKH8q3c2Gqgq77Uq12qfcDE8zF2a4v","crv":"P-521"},{"kty":"EC","kid":"7uLnfLOhXPmOZ2BUaeUue-","use":"sig","x":"fR71QOze0q-0uDBuOyTdJANOA0Kz_WT0ykuuuLH94uCJzD13B5OBB1y4jtOvIYvX","y":"WwjH9z9kzSsGZmkl5BvXhyCO8udP9nNFuZM4WugT7C-kD0UNqojpCEkigWorRlin","crv":"P-384"},{"kty":"EC","kid":"FWwgHZozeidVHog8YEjhT-","use":"sig","x":"C2U_Vuv_t3VEOv-UaprJTir3L_SWq_gjwVSUcLTq5Bc","y":"U9F6c5qlnRLSR_KRiCyAzRk7YUpXrXbqsUJYYlTPw7A","crv":"P-256"},{"kty":"RSA","kid":"aNspNiifg5EiN-R1R-RO4O","use":"sig","n":"klTzMXRM83AWKoKRZQwI5XOdJ0tB0jr1ieo_uLT_W6w_hNY8jbFzN1xwxjyUZFBHQVDU2PlyjRZDKQ5tIRs05__DwCvYdDiN0i5f5ChpB66UkAUn1IuuGUQYP7OHVSdeDCFCq3q8UGrl2FCFRxyL9k9hu0VX7Pj06SNGr74_hbCXF6VJmcgDmoXrTXEC5hSBciauGmAxlIbhDbDx-3QHrPJ5P4OLCwH0kfW2RB50O6DMZmKhXeSqE0WiI9KbMz_Mq-TIMiPNkRC1Wsdv4sKoYHf1t6sdp6n9qconQRGwGZNaWcKk6nno-26a_8CxGAuHh8BIFIvFj5AohFFXu2-slw","e":"AQAB"},{"kty":"EC","kid":"ikc6qm-M_topQoGzydo8zI","use":"sig","x":"AE8C5SEixbzl9ez1NPsYOehpnPohIAMxucjZjw2E9aVCBdGo7t2hKv2Aa7ql4zpeTucrDP3ZyUKK6-D3m3C10ojx","y":"AXNjApqwEKFjFmR4bbxAf_8nS74zDuOtXsq1PnoU58ZcHON1ZBUOEXY0Y6IPw1Q8ngaLHg30I300L1ZL0aIX4ygv","crv":"P-521"},{"kty":"EC","kid":"403dlMPeSjSNp4cQ0GBSLL","use":"sig","x":"AI4FHJCcr7fMCb2BfCj9l6bptD513AVVSbOLP2wPM0Aq6CKOEWlN5Yc8qW1Z8NFOPkjeBsCagV02qNJFxH0oi9em","y":"AK3nxsAk3aMOwqIGU-9xXW2I_wwkRxbTR5cPPceWEzpvUHXqyfOXihVYPTJkbaYggyOWjlpw5RYJZGBTLkFUzdal","crv":"P-521"},{"kty":"EC","kid":"IcGz3uUD_EfbMQWQ-6SrmT","use":"sig","x":"10bDbsUH92XHCizkNtBzumyzro8aZypDTqG6ob1fMXk","y":"zfradNJdJ739STTO9vQmOtMl3r7XlnX2SNEusGsFtz0","crv":"P-256"},{"kty":"EC","kid":"oTrCYqttZBLTIv4R5lrfiZ","use":"sig","x":"kOxZSdBd2DPGjuf0lrV30Bc8LCj2EBMfmluPc3sV44fnsZWtTnQz4pCcDjz2hzOY","y":"587zK_ggKmzpYaKz8AaBKERnAsF0AfLjIo2dAu0BdCO0FzSJGyV_cDpNmAdx8ah4","crv":"P-384"},{"kty":"RSA","kid":"vty_MATEPV9warjejj_hef","use":"sig","n":"6TDzjPXHfjDygJ3wa-0BB4m028hatl-PnfT5BEFcIkkWhGRswQSpiGCGjb3DXQ4LxAwZ_XM5RuBYMOMZw9qEU01lhQycqYVOub7R0lli2oDETW4pOATa6JW7QyyXcbbcnYxqj6qfwKb_XfXBDfLpwT8K1_ylJxAymV1ZgfeXDexGBr7d4fLNzgGV7CjZcYmMftn3CktKrA2vy8fLQ2wfVQgfo1J9UqCmLoo6sorW2Sn23Vsx4sTN8OkcrYzpd8_0Gj0X4jXsnk05rfvAWfn8iBHxe6ERBuVh48SWkGjpRnLLoyFmZFg0d1aDlskWp2rYu0VeeUAfKDAeGAU3_1ihuQ","e":"AQAB"}]}

YangminZhu · December 6, 2019, 1:52am

Thank you for the information, I will test it to try to reproduce the issue.

@Michael_Whittlestone what do you mean for without the EC? Do you mean remove the "kty":"EC" in the jwks?

jammerful · December 6, 2019, 4:14am

@YangminZhu Yes, I think if you remove the EC keys it will go away given the code below. I can’t test as I ran into this issue a while ago, and it takes a while to repro for me. Looking into the issue more, I think the jwt verify library needs to be updated in Envoy, it doesn’t have support for EC-384 and EC-512 keys and that is causing it the JWKS parse error. See the following links:

Where envoy is throwing the first part of the error:

github.com

envoyproxy/envoy/blob/88fe0c9310d67fd4c8797dc37278641dd9415fd0/source/extensions/filters/http/jwt_authn/filter_factory.cc#L33


* Validate inline jwks, make sure they are the valid
*/
void validateJwtConfig(const JwtAuthentication& proto_config, Api::Api& api) {
 for (const auto& it : proto_config.providers()) {
   const auto& provider = it.second;
   const auto inline_jwks = Config::DataSource::read(provider.local_jwks(), true, api);
   if (!inline_jwks.empty()) {
     auto jwks_obj = Jwks::createFrom(inline_jwks, Jwks::JWKS);
     if (jwks_obj->getStatus() != Status::Ok) {
       throw EnvoyException(fmt::format(
           "Issuer '{}' in jwt_authn config has invalid local jwks: {}", provider.issuer(),
           ::google::jwt_verify::getStatusString(jwks_obj->getStatus())));
     }
   }
 }
}


} // namespace


Http::FilterFactoryCb
FilterFactory::createFilterFactoryFromProtoTyped(const JwtAuthentication& proto_config,

Where the second part of the error message is coming from: https://github.com/google/jwt_verify_lib/blob/9f10e2d60d42edeb6662e185707a7d6a4ebc5604/src/status.cc#L72
Where the second part of the error maybe coming from: https://github.com/google/jwt_verify_lib/blob/9f10e2d60d42edeb6662e185707a7d6a4ebc5604/src/jwks.cc#L90
Adds EC-384 and EC-512 support: https://github.com/google/jwt_verify_lib/commit/4d9461344d9768c3cd05f8d90d4518b83d113f9d
Looks like its already fixed in envoy: https://github.com/envoyproxy/envoy/commit/b9faab6747a9a884cc4518b8f0e574acd763c3cf

YangminZhu · December 6, 2019, 8:39pm

Thanks for the investigation, please see my reply in https://github.com/istio/istio/issues/19424#issuecomment-562729218

jammerful · December 13, 2019, 11:23pm

@Michael_Whittlestone Where you able to resolve the issue? Did you just remove the EC keys from PingFederate’s JWKS? I tried moving to 1.3.6 and 1.4.2 and still facing the same or similar issues.

jammerful · December 17, 2019, 1:12am

@YangminZhu @JimmyChen Any help will be greatly appreciated. See my cooment on the GitHub ticket: https://github.com/istio/istio/issues/19424

YangminZhu · January 7, 2020, 12:43am

Note, this issue is fixed in 1.4.3 release.

liquid · February 4, 2020, 2:02pm

Can confirm this works fine with Istio 1.4.3 and PingFederate JWKS.

jammerful · February 4, 2020, 6:35pm

@YangminZhu Thank you for fixing this, it was rolled out to our production environment already

Topic		Replies	Views
Error adding/updating listener(s) virtualInbound: Provider 'origins-0' in jwt_authn config has invalid local jwks: Jwks RSA [n] or [e] field is missing or has a parse error openshift	3	1951	April 12, 2023
gRPC config stream closed: 14, no healthy upstream	0	4183	May 6, 2020
Incorrect envoyproxy listener merging Networking	0	628	May 19, 2023
Getting errors from envoy proxy with istio 1.9.1 version on GRPC web service	0	1070	April 29, 2021
Istio 1.6.3: ingressgateway error "Proto constraint validation failed" Security	3	3287	July 20, 2020

JWKS Parse Error

Related Topics