Istio Security Update - CVE 2019-12995

Announcements
1

Personal comment from duderino: I’m sorry for the timing of this one. We received a vulnerability report made publicly last Sunday 6/23, we tried to get a fix out earlier this week, but sadly we finally have one at an exceedingly awkward time. We do not want to announce things like these late on Fridays. Many people worked hard on this one. I’m sorry we couldn’t get this to you sooner.

A bug in Istio’s JWT validation filter causes Istio Proxy to crash in certain cases when the request contains a malformed JWT token. The bug was discovered and reported by a user on GitHub on June 23, 2019.

This bug affects all versions of Istio that are using the JWT authentication policy.

The symptoms of the bug is an HTTP 503 error seen by the client, and

Epoch 0 terminated with an error: signal: segmentation fault (core dumped)

in the Envoy logs.

The Envoy crash can be triggered using a malformed JWT without a valid signature, and on any URI being accessed regardless of the trigger_rules in the JWT specification. Thus, this bug makes the Istio proxy vulnerable to a potential DoS attack.

This vulnerability is referred to as CVE 2019-12995

Affected Istio releases

The following Istio releases are vulnerable:

  • 1.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.0.6, 1.0.7, 1.0.8
  • 1.1, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.1.6, 1.1.7, 1.1.8, 1.1.9
  • 1.2, 1.2.1

Impact score

Overall CVSS score: 7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H/E:F/RL:O/RC:C

Vulnerability impact and Detection

The Istio proxy is vulnerable if the following two conditions are satisfied:

  • A JWT authentication policy is applied to it.
  • The JWT issuer (specified by jwksUri) uses RSA algorithm for signature verification (Note the bug is just triggered in this condition and have nothing to do with the RSA algorithm)

If JWT policy is applied to the Istio ingress gateway, please be aware that any external user who has access to the ingress gateway could crash it with a single HTTP request.

If JWT policy is applied to the sidecar only, please keep in mind it might still be vulnerable. For example, the Istio ingress gateway might forward the JWT token to the sidecar which could be a malformed JWT token that crashes the sidecar.

A vulnerable Istio proxy will crash on an HTTP request with a malformed JWT token. When Istio proxy crashes, all existing connections will be disconnected immediately. The pilot-agent will restart the crashed Istio proxy automatically and it may take a few seconds to a few minutes for the restart. pilot-agent will stop restarting the Istio proxy after it crashed more than 10 times. In this case, Kubernetes will redeploy the pod (including the workload behind Istio proxy).

To detect if there is any JWT authentication policy applied in your cluster, run the following command which print either of the following output:

  • Found JWT in authentication policy, YOU ARE AFFECTED
  • Did NOT find JWT in authentication policy, YOU ARE NOT AFFECTED

$ cat <<‘EOF’ | bash -
set -e
set -u
set -o pipefail

red=tput setaf 1
green=tput setaf 2
reset=tput sgr0

echo “Checking authentication policy…”

JWKS_URI=()
JWKS_URI+=($(kubectl get policy --all-namespaces -o jsonpath=‘{range .items[]}{.spec.origins[].jwt.jwksUri}{" "}{end}’))
JWKS_URI+=($(kubectl get meshpolicy --all-namespaces -o jsonpath=‘{range .items[]}{.spec.origins[].jwt.jwksUri}{" "}{end}’))
if [ “${#JWKS_URI[@]}” != 0 ]; then
echo “${red}Found JWT in authentication policy, YOU ARE AFFECTED${reset}”
exit 1
fi

echo “${green}Did NOT find JWT in authentication policy, YOU ARE NOT AFFECTED${reset}”
EOF

Mitigations

This bug is fixed in the following versions of Istio:

  • For Istio 1.0.x deployments: update to a minimum version of Istio 1.0.9
  • For Istio 1.1.x deployments: update to a minimum version of Istio 1.1.10
  • For Istio 1.2.x deployments: update to a minimum version of Istio 1.2.2

If you cannot immediately upgrade to one of these releases, you have the additional option of injecting a Lua filter into older versions of Istio. This filter has been verified to work with Istio versions 1.1.9, 1.0.8, 1.0.6, and 1.1.3.

The Lua filter is injected before the Istio jwt-auth filter.
If a JWT token is presented on an http request, the Lua filter will check if the JWT token header contains alg:ES256. If the filter finds such a JWT token, the request is rejected.

To install the Lua filter, please invoke the following commands:

git clone git@github.com:istio/tools.git
cd tools/examples/luacheck/
./setup.sh

The setup script uses helm template to produce an envoyFilter resource that deploys to gateways. You may change the listener type to ANY to also apply it to sidecars. You should only do this if you enforce JWT policies on sidecars and sidecars receive direct traffic from the outside.

Credit

The Istio team would like to thank Divya Raj for the original bug report.

We’d like to remind our community to follow the vulnerability reporting process described at https://istio.io/about/security-vulnerabilities/ to report any bug that can result in a security vulnerability

2 Likes