JWKS Parse Error

Hello,

I’m trying to upgrade from 1.2.2 to a new version, preferably to 1.3.3. However, when I try to upgrade, I’m seeing an error on the istio proxy sidecar starting properly due to a mysterious JWKS parse error. I’m seeing this error on versions 1.2.7 and 1.3.3:

[2019-10-18 22:15:55.856][33][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, no healthy upstream
[2019-10-18 22:15:55.856][33][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:50] Unable to establish new stream
[2019-10-18 22:15:56.947][33][warning][config] [external/envoy/source/common/config/grpc_mux_subscription_impl.cc:81] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) 10.110.123.108_8443: Issuer 'https://pingfederate.mycorp.io:5555' in jwt_authn config has invalid local jwks: Jwks EC [x] or [y] field is missing or has a parse error., virtualInbound: Issuer 'https://pingfederate.mycorp.io:5555' in jwt_authn config has invalid local jwks: Jwks EC [x] or [y] field is missing or has a parse error.

I checked the JWKS and all keys with kty = “EC” have both an “x” and “y” field set. Beyond this my OIDC provider is PingFederate which I doubt it serving up an invalid JWKS, and this JWKS works on 1.2.2.

Thanks for any help

If anyone could help me out here, I’d appreciate it. Right now I’m unable to upgrade.

If you rollback to 1.2.2, does the error still exist? @YangminZhu

Istio 1.3 changes to use the upstream Envoy JWT filter which is following the JWKS standard more strictly and this is possibly causing the issue.

Could you share the jwks to reproduce the issue?

Yes if I rollback to 1.2.2 I don’t have an issue. @YangminZhu

I can share a redacted version of the JWKS, is that okay? Note this is from PingFederate, which is a pretty standard enterprise IdP. Thank you for the help :slight_smile:

@YangminZhu Here is the redacted JWKS, I’ll work on getting approval to send the full JWKS to you. It would preferable if I could send you personal message/email instead of posting the full one on here. Please let me know if you need any additional information, alternatively I can help reproduce the issue myself.

  {
  "keys": [
    {
      "kty": "RSA",
      "kid": "REDACTED",
      "use": "sig",
      "n": "REDACTED",
      "e": "REDACTED",
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y": "REDACTED",
      "crv": "P-521"
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y": "REDACTED",
      "crv": "P-384"
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y": "REDACTED",
      "crv": "P-256"
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y": "REDACTED",
      "crv": "P-256"
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y": "REDACTED",
      "crv": "P-521"
    },
    {
      "kty": "RSA",
      "kid": "REDACTED",
      "use": "sig",
      "n": "REDACTED",
      "e": "REDACTED"
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y": "REDACTED",
      "crv": "P-384"
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y":"REDACTED",
      "crv": "P-521"
    },
    {
      "kty": "RSA",
      "kid": "REDACTED",
      "use": "sig",
      "n": "REDACTED",
      "e": "REDACTED"
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y": "REDACTED",
      "crv": "P-384"
    },
    {
      "kty": "EC",
      "kid": "REDACTED",
      "use": "sig",
      "x": "REDACTED",
      "y": "REDACTED",
      "crv": "P-256"
    }
  ]
}

Having same problem with Istio and PingFederate. Have found out that copying the JWKS to a new location without the EC’s works fine (at least for a few hours until the keys rotate). Our IdTokens are RSA signed

@YangminZhu Here is the full JWKS

{"keys":[{"kty":"EC","kid":"Ozxgl1WsWyQBF1lcbYyjYl","use":"sig","x":"Vq3vKJCeJyihZIauouQp3eFmuYLCsEcID_sGigAx2gs","y":"3UaPoda-HCrJeU4i960qFHMschTdmqjkitiIyYc5svA","crv":"P-256"},{"kty":"EC","kid":"RBsu7DAEUcrgx4X91TVrsY","use":"sig","x":"dQ_SlIctw4nWZEJ2rasiyrps7jXuxr1E81zw4X-aaY1LLRhcaUxpInytXeZK5mOm","y":"xrhQlZQZeDWm7VRHThevLTKqOkqfNwTmHL7P_f23BPn8SLlXd9p1jS4LzL0KK0rL","crv":"P-384"},{"kty":"RSA","kid":"X3oArm2sGh5pO0jWl5O41C","use":"sig","n":"lOTXC2Pfi0dajzDAOERQLTtT0_GbjAFaNflzV-0tWxban1CPEN0n5UG5z-c0KxKY6fhZshQ1Opr3VbQmE1MGSeYf3qEUD4Th3ZubVV_2Yhuio-UZXllz1EGgAh7sC9TzQi84jIYj_mhdno4l6Y3FVlvM6VtNYRGPDNgjRoVfd63vocXAqgUok6LpEcL9MbrvmK-hFPVNX7euGN_xm_qZM5-JrJMnKz6shnjrul7yZ-ZExzfFx_LSBqum-fkKv2FfEoJqyzyVlbabmDUZ81B9ZP0nfaP3e-IRSQECuXf52PfHqEgZbrax8hKAfpZKJX643gxSCnBWdmNO-BFSlho4pw","e":"AQAB"},{"kty":"EC","kid":"sxG_WeuLxIKXoVit-8vyQf","use":"sig","x":"AG3w2vYgVbn4E27rkxZPUVrzLWhMctY5GOP6xygLLFwNRaoOx2gnlQPwAsEXHxz80u5lfmOms0pJSjuDrNqs5pB4","y":"Ad0K-hbFmTVj3nMOw7jAdl21dlU35pG1g7h_Tswr0VYfxqg4ubIPyXrrtmlKH8q3c2Gqgq77Uq12qfcDE8zF2a4v","crv":"P-521"},{"kty":"EC","kid":"7uLnfLOhXPmOZ2BUaeUue-","use":"sig","x":"fR71QOze0q-0uDBuOyTdJANOA0Kz_WT0ykuuuLH94uCJzD13B5OBB1y4jtOvIYvX","y":"WwjH9z9kzSsGZmkl5BvXhyCO8udP9nNFuZM4WugT7C-kD0UNqojpCEkigWorRlin","crv":"P-384"},{"kty":"EC","kid":"FWwgHZozeidVHog8YEjhT-","use":"sig","x":"C2U_Vuv_t3VEOv-UaprJTir3L_SWq_gjwVSUcLTq5Bc","y":"U9F6c5qlnRLSR_KRiCyAzRk7YUpXrXbqsUJYYlTPw7A","crv":"P-256"},{"kty":"RSA","kid":"aNspNiifg5EiN-R1R-RO4O","use":"sig","n":"klTzMXRM83AWKoKRZQwI5XOdJ0tB0jr1ieo_uLT_W6w_hNY8jbFzN1xwxjyUZFBHQVDU2PlyjRZDKQ5tIRs05__DwCvYdDiN0i5f5ChpB66UkAUn1IuuGUQYP7OHVSdeDCFCq3q8UGrl2FCFRxyL9k9hu0VX7Pj06SNGr74_hbCXF6VJmcgDmoXrTXEC5hSBciauGmAxlIbhDbDx-3QHrPJ5P4OLCwH0kfW2RB50O6DMZmKhXeSqE0WiI9KbMz_Mq-TIMiPNkRC1Wsdv4sKoYHf1t6sdp6n9qconQRGwGZNaWcKk6nno-26a_8CxGAuHh8BIFIvFj5AohFFXu2-slw","e":"AQAB"},{"kty":"EC","kid":"ikc6qm-M_topQoGzydo8zI","use":"sig","x":"AE8C5SEixbzl9ez1NPsYOehpnPohIAMxucjZjw2E9aVCBdGo7t2hKv2Aa7ql4zpeTucrDP3ZyUKK6-D3m3C10ojx","y":"AXNjApqwEKFjFmR4bbxAf_8nS74zDuOtXsq1PnoU58ZcHON1ZBUOEXY0Y6IPw1Q8ngaLHg30I300L1ZL0aIX4ygv","crv":"P-521"},{"kty":"EC","kid":"403dlMPeSjSNp4cQ0GBSLL","use":"sig","x":"AI4FHJCcr7fMCb2BfCj9l6bptD513AVVSbOLP2wPM0Aq6CKOEWlN5Yc8qW1Z8NFOPkjeBsCagV02qNJFxH0oi9em","y":"AK3nxsAk3aMOwqIGU-9xXW2I_wwkRxbTR5cPPceWEzpvUHXqyfOXihVYPTJkbaYggyOWjlpw5RYJZGBTLkFUzdal","crv":"P-521"},{"kty":"EC","kid":"IcGz3uUD_EfbMQWQ-6SrmT","use":"sig","x":"10bDbsUH92XHCizkNtBzumyzro8aZypDTqG6ob1fMXk","y":"zfradNJdJ739STTO9vQmOtMl3r7XlnX2SNEusGsFtz0","crv":"P-256"},{"kty":"EC","kid":"oTrCYqttZBLTIv4R5lrfiZ","use":"sig","x":"kOxZSdBd2DPGjuf0lrV30Bc8LCj2EBMfmluPc3sV44fnsZWtTnQz4pCcDjz2hzOY","y":"587zK_ggKmzpYaKz8AaBKERnAsF0AfLjIo2dAu0BdCO0FzSJGyV_cDpNmAdx8ah4","crv":"P-384"},{"kty":"RSA","kid":"vty_MATEPV9warjejj_hef","use":"sig","n":"6TDzjPXHfjDygJ3wa-0BB4m028hatl-PnfT5BEFcIkkWhGRswQSpiGCGjb3DXQ4LxAwZ_XM5RuBYMOMZw9qEU01lhQycqYVOub7R0lli2oDETW4pOATa6JW7QyyXcbbcnYxqj6qfwKb_XfXBDfLpwT8K1_ylJxAymV1ZgfeXDexGBr7d4fLNzgGV7CjZcYmMftn3CktKrA2vy8fLQ2wfVQgfo1J9UqCmLoo6sorW2Sn23Vsx4sTN8OkcrYzpd8_0Gj0X4jXsnk05rfvAWfn8iBHxe6ERBuVh48SWkGjpRnLLoyFmZFg0d1aDlskWp2rYu0VeeUAfKDAeGAU3_1ihuQ","e":"AQAB"}]}

Thank you for the information, I will test it to try to reproduce the issue.

@Michael_Whittlestone what do you mean for without the EC? Do you mean remove the "kty":"EC" in the jwks?

@YangminZhu Yes, I think if you remove the EC keys it will go away given the code below. I can’t test as I ran into this issue a while ago, and it takes a while to repro for me. Looking into the issue more, I think the jwt verify library needs to be updated in Envoy, it doesn’t have support for EC-384 and EC-512 keys and that is causing it the JWKS parse error. See the following links:

Where envoy is throwing the first part of the error:

Where the second part of the error message is coming from: https://github.com/google/jwt_verify_lib/blob/9f10e2d60d42edeb6662e185707a7d6a4ebc5604/src/status.cc#L72
Where the second part of the error maybe coming from: https://github.com/google/jwt_verify_lib/blob/9f10e2d60d42edeb6662e185707a7d6a4ebc5604/src/jwks.cc#L90
Adds EC-384 and EC-512 support: https://github.com/google/jwt_verify_lib/commit/4d9461344d9768c3cd05f8d90d4518b83d113f9d
Looks like its already fixed in envoy: https://github.com/envoyproxy/envoy/commit/b9faab6747a9a884cc4518b8f0e574acd763c3cf

Thanks for the investigation, please see my reply in https://github.com/istio/istio/issues/19424#issuecomment-562729218

@Michael_Whittlestone Where you able to resolve the issue? Did you just remove the EC keys from PingFederate’s JWKS? I tried moving to 1.3.6 and 1.4.2 and still facing the same or similar issues.

@YangminZhu @JimmyChen Any help will be greatly appreciated. See my cooment on the GitHub ticket: https://github.com/istio/istio/issues/19424

Note, this issue is fixed in 1.4.3 release.

Can confirm this works fine with Istio 1.4.3 and PingFederate JWKS.

@YangminZhu Thank you for fixing this, it was rolled out to our production environment already :slight_smile: