Istio 1.6.3 - RequestAuthentication don't work with Keycloak

Hi all,
I’m trying to enabe user authentication in istio 1.6.3 using a keycloak server.
Testing my framework with the demo token from the documentation, it works. But with the Keycloak token the authorization fails.

Here the step I followed

RequestAuthentication with demo token - File h-test-ingress.yaml

apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
  name: "h-ingress-jwt"
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  jwtRules:
  - issuer: "testing@secure.istio.io"
    jwksUri: "https://raw.githubusercontent.com/istio/istio/release-1.6/security/tools/jwt/samples/jwks.json"

---

Apply the file an test without token, wrong token and correct token

export TOKEN=$(curl https://raw.githubusercontent.com/istio/istio/release-1.6/security/tools/jwt/samples/demo.jwt -s)
sysop@hdev:~/software/hproject$ export TOKEN=$(curl https://raw.githubusercontent.com/istio/istio/release-1.6/security/tools/jwt/samples/demo.jwt -s)
sysop@hdev:~/software/hproject$ echo -e "\n\nWITHOUT token"

WITHOUT token
sysop@hdev:~/software/hproject$ curl --insecure https://www.h.net -s -o /dev/null -w "%{http_code}\n"
200
sysop@hdev:~/software/hproject$ echo -e "\n\nwith INVALID token"

with INVALID token
sysop@hdev:~/software/hproject$ curl --insecure --header "Authorization: Bearer pippo" https://www.h.net -s -o /dev/null -w "%{http_code}\n"
401
sysop@hdev:~/software/hproject$ echo -e "\n\nWITH token"

WITH token
sysop@hdev:~/software/hproject$ curl --insecure --header "Authorization: Bearer $TOKEN" https://www.h.net -s -o /dev/null -w "%{http_code}\n"
200
sysop@hdev:~/software/hproject

The access with token gives correctly 200

RequestAuthentication with keycloak token - File h-keycloak2-ingress.yaml

apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
  name: "h-ingress-jwt"
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  jwtRules:
  - issuer: "http://192.168.202.21:8000/auth/realms/istio"
    jwksUri: "http://192.168.202.21:8000/auth/realms/istio/protocol/openid-connect/certs"

---

Apply the file
Execute a login to Keycloak with user test (verify login works) and get the access token

sysop@hdev:~/software/hproject$ curl --insecure -L -X POST ‘https://192.168.202.21:8443/auth/realms/istio/protocol/openid-connect/token
-H ‘Content-Type: application/x-www-form-urlencoded’
–data-urlencode ‘client_id=cistio’
–data-urlencode ‘grant_type=password’
–data-urlencode ‘client_secret=d74ae23c-4b7b-4f88-8876-4a0456e6f832’
–data-urlencode ‘scope=openid’
–data-urlencode ‘username=test’
–data-urlencode ‘password=aaaaaaaa’ | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 204 100 72 100 132 529 970 --:–:-- --:–:-- --:–:-- 1500
{
“error”: “invalid_grant”,
“error_description”: “Invalid user credentials”
}
sysop@hdev:~/software/hproject$ curl --insecure -L -X POST ‘https://192.168.202.21:8443/auth/realms/istio/protocol/openid-connect/token
-H ‘Content-Type: application/x-www-form-urlencoded’
–data-urlencode ‘client_id=cistio’
–data-urlencode ‘grant_type=password’
–data-urlencode ‘client_secret=d74ae23c-4b7b-4f88-8876-4a0456e6f832’
–data-urlencode ‘scope=openid’
–data-urlencode ‘username=test’
–data-urlencode ‘password=test’ | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3332 100 3204 100 128 24646 984 --:–:-- --:–:-- --:–:-- 25630
{
“access_token”: “eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJtS3hqTHFWVnFTSThZRzJWUFZBZEhmQkljN24zZ0NXS1UyMTZGSTU1NGN3In0.eyJleHAiOjE1OTQxMzc0MDksImlhdCI6MTU5NDEzNzEwOSwianRpIjoiNTdiMjY5OTAtNWQzYi00YzRlLThhYTYtMGQ1ZjdiZWFhOTA4IiwiaXNzIjoiaHR0cHM6Ly8xOTIuMTY4LjIwMi4yMTo4NDQzL2F1dGgvcmVhbG1zL2lzdGlvIiwiYXVkIjoiYWNjb3VudCIsInN1YiI6IjdmNzdiOWI0LThjNzgtNDM4MS04NTIzLTAwZDc5MTgzNzZhZiIsInR5cCI6IkJlYXJlciIsImF6cCI6ImNpc3RpbyIsInNlc3Npb25fc3RhdGUiOiJkZGNmNmQwMy04MzQ3LTQ5NmMtYWJjMS1jNzNmMjJhYWJiMjAiLCJhY3IiOiIxIiwiYWxsb3dlZC1vcmlnaW5zIjpbIiR7YXV0aEJhc2VVcmx9Il0sInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJlZGl0b3IiLCJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJvcGVuaWQgcHJvZmlsZSBlbWFpbCIsImVtYWlsX3ZlcmlmaWVkIjpmYWxzZSwibmFtZSI6InRlc3QgdGVzdCIsInByZWZlcnJlZF91c2VybmFtZSI6InRlc3QiLCJnaXZlbl9uYW1lIjoidGVzdCIsImZhbWlseV9uYW1lIjoidGVzdCIsImVtYWlsIjoidGVzdEBoLm5ldCJ9.hRqRUlaMDXdyRyup5Yl81qgOwW9jknyYoa4Gatzi1XhjDJYEw2k6Ajn-w8MSjTR87LneZP_u1Sba4s6cEsJVwHvU5ofku5ZkM8AK-vtBbPj6056i-OHGcIOg69Sinv_AgeHA_Xtxqv-Jl1a-BaLnlk_gOFfOTSmJTkfzVvTTmeagOAwYt8BVC9ePV26DYEaR0qcz1vTaHpN3BwDh0rqpsyHDHn9T7WDw8fxuJ8MbHcnUsGP5Y6PaKXQV4rpAqzu6wDC7_mbtxvCGolJ3uwzYsoenDWfhjCnRdumO8vSXnJA39jRPrbCHQonXB_WCOLlcr5PWO4d0yVd8437d0–eBA”,
“expires_in”: 300,
“refresh_expires_in”: 1800,
“refresh_token”: “eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI5MTA3YTUwMy1kMjNhLTRhYjctODc0OS01NmMyMDY4OWY3NzAifQ.eyJleHAiOjE1OTQxMzg5MDksImlhdCI6MTU5NDEzNzEwOSwianRpIjoiZmUyNGE3MmMtMTI4Ni00MjhkLWJmOGMtYTA5NWJjZGM4NDRmIiwiaXNzIjoiaHR0cHM6Ly8xOTIuMTY4LjIwMi4yMTo4NDQzL2F1dGgvcmVhbG1zL2lzdGlvIiwiYXVkIjoiaHR0cHM6Ly8xOTIuMTY4LjIwMi4yMTo4NDQzL2F1dGgvcmVhbG1zL2lzdGlvIiwic3ViIjoiN2Y3N2I5YjQtOGM3OC00MzgxLTg1MjMtMDBkNzkxODM3NmFmIiwidHlwIjoiUmVmcmVzaCIsImF6cCI6ImNpc3RpbyIsInNlc3Npb25fc3RhdGUiOiJkZGNmNmQwMy04MzQ3LTQ5NmMtYWJjMS1jNzNmMjJhYWJiMjAiLCJzY29wZSI6Im9wZW5pZCBwcm9maWxlIGVtYWlsIn0.aEz82LpzZvps13Gmx-cIM6H68WTv3GxHMsy2nxgLBUc”,
“token_type”: “bearer”,
“id_token”: “eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJtS3hqTHFWVnFTSThZRzJWUFZBZEhmQkljN24zZ0NXS1UyMTZGSTU1NGN3In0.eyJleHAiOjE1OTQxMzc0MDksImlhdCI6MTU5NDEzNzEwOSwiYXV0aF90aW1lIjowLCJqdGkiOiJjZWNjZjZkOS1hYmQ3LTQyNmYtYWIyYy02MWJkNmRhMzVkZjAiLCJpc3MiOiJodHRwczovLzE5Mi4xNjguMjAyLjIxOjg0NDMvYXV0aC9yZWFsbXMvaXN0aW8iLCJhdWQiOiJjaXN0aW8iLCJzdWIiOiI3Zjc3YjliNC04Yzc4LTQzODEtODUyMy0wMGQ3OTE4Mzc2YWYiLCJ0eXAiOiJJRCIsImF6cCI6ImNpc3RpbyIsInNlc3Npb25fc3RhdGUiOiJkZGNmNmQwMy04MzQ3LTQ5NmMtYWJjMS1jNzNmMjJhYWJiMjAiLCJhY3IiOiIxIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJuYW1lIjoidGVzdCB0ZXN0IiwicHJlZmVycmVkX3VzZXJuYW1lIjoidGVzdCIsImdpdmVuX25hbWUiOiJ0ZXN0IiwiZmFtaWx5X25hbWUiOiJ0ZXN0IiwiZW1haWwiOiJ0ZXN0QGgubmV0In0.iySr1nUyvuKp5vS11lWgqkXWZFjrpwbOxQR2vApMy8oq4AZ9vexraSXuS4OJFt7pmWsCvGJbcMo1dByV3d9ZslswCeorMTCEQu9Cy1TvmVM6zvWWR-r-MtGcK3RJ12fKRgYF0tfK_JTVgYHBu_bbhiXup3NellBtprF8Sh8KIsKcnVRpZPqihOoBIJrK3tUY4ErKk5WFV-evprKMAi_yaKGXgMhrOtpwrY9iygcSDfRmTVIqJJ0OkqQo9OA8H209_rV8PpL5zwz55LUlNkfqDIbS-c–Vt2lZlJ5zd4QJGLqyX5l_TNfmJp2JZMvBz3fnxnWosz5L5AIIXUkdbtRew”,
“not-before-policy”: 0,
“session_state”: “ddcf6d03-8347-496c-abc1-c73f22aabb20”,
“scope”: “openid profile email”
}
sysop@hdev:~/software/hproject$

Set the token with the last generated access token

sysop@hdev:~/software/hproject$ export TOKEN="eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJtS3hqTHFWVnFTSThZRzJWUFZBZEhmQkljN24zZ0NXS1UyMTZGSTU1NGN3In0.eyJleHAiOjE1OTQxMzc0MDksImlhdCI6MTU5NDEzNzEwOSwianRpIjoiNTdiMjY5OTAtNWQzYi00YzRlLThhYTYtMGQ1ZjdiZWFhOTA4IiwiaXNzIjoiaHR0cHM6Ly8xOTIuMTY4LjIwMi4yMTo4NDQzL2F1dGgvcmVhbG1zL2lzdGlvIiwiYXVkIjoiYWNjb3VudCIsInN1YiI6IjdmNzdiOWI0LThjNzgtNDM4MS04NTIzLTAwZDc5MTgzNzZhZiIsInR5cCI6IkJ

    ...

BaLnlk_gOFfOTSmJTkfzVvTTmeagOAwYt8BVC9ePV26DYEaR0qcz1vTaHpN3BwDh0rqpsyHDHn9T7WDw8fxuJ8MbHcnUsGP5Y6PaKXQV4rpAqzu6wDC7_mbtxvCGolJ3uwzYsoenDWfhjCnRdumO8vSXnJA39jRPrbCHQonXB_WCOLlcr5PWO4d0yVd8437d0--eBA"
sysop@hdev:~/software/hproject$ 

Test the access

sysop@hdev:~/software/hproject$ echo -e "\n\nWITHOUT token"

WITHOUT token
sysop@hdev:~/software/hproject$ curl --insecure https://www.h.net -s -o /dev/null -w "%{http_code}\n"
200
sysop@hdev:~/software/hproject$ echo -e "\n\nwith INVALID token"

with INVALID token
sysop@hdev:~/software/hproject$ curl --insecure --header "Authorization: Bearer pippo" https://www.h.net -s -o /dev/null -w "%{http_code}\n"
401
sysop@hdev:~/software/hproject$ echo -e "\n\nWITH token"

WITH token
sysop@hdev:~/software/hproject$ curl --insecure --header "Authorization: Bearer $TOKEN" https://www.h.net -s -o /dev/null -w "%{http_code}\n"
401
sysop@hdev:~/software/hproject$ 

The token is rejected and I receive a 401 code.

My framework:

  • 4 KVM virtual machines with Ubuntu 20.04
  • K3S kubernetes distribution (1 master, 3 workers)
  • istio 1.6.3 (upgraded from 1.6.1 upgraded from 1.6.0)
  • keykloak server installed in the same cluster using the helm chart codecentric/keycloak setting values to expose the service using loadbalancer and port 8000 for http and 8443 for https
  • a keycloak realm called istio, a client called cistio and a user called test
  • the service at https://www.h.net is an httpbin application
  • the https certificates are self-signed

What I’m doing wrong?
There is any working example of RequestAuthorization using keycloak?

Looking at ingressgateway logs I see these errors:

2020-07-09T09:36:50.669693Z     warn    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 7 successful, 0 rejected; lds updates: 0 successful, 7 rejected
2020-07-09T09:36:52.669736Z     warn    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 7 successful, 0 rejected; lds updates: 0 successful, 7 rejected
2020-07-09T09:36:54.670022Z     warn    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 7 successful, 0 rejected; lds updates: 0 successful, 7 rejected
2020-07-09T09:36:55.068148Z     warning envoy config    [external/envoy/source/common/config/grpc_subscription_impl.cc:101] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) 0.0.0.0_8443: Proto constraint validation failed (JwtAuthenticationValidationError.Providers[key]: ["embedded message failed validation"] | caused by JwtProviderValidationError.LocalJwks: ["embedded message failed validation"] | caused by DataSourceValidationError.InlineString: ["value length must be at least " '\x01' " bytes"]): providers {
  key: "origins-0"
  value {
    issuer: "http://192.168.202.21:8000/auth/realms/istio"
    local_jwks {
      inline_string: ""
    }
    payload_in_metadata: "http://192.168.202.21:8000/auth/realms/istio"
  }
}
rules {
  match {
    prefix: "/"
  }
  requires {
    requires_any {
      requirements {
        provider_name: "origins-0"
      }
      requirements {
        allow_missing {
        }
      }
    }
  }
}

0.0.0.0_8080: Proto constraint validation failed (JwtAuthenticationValidationError.Providers[key]: ["embedded message failed validation"] | caused by JwtProviderValidationError.LocalJwks: ["embedded message failed validation"] | caused by DataSourceValidationError.InlineString: ["value length must be at least " '\x01' " bytes"]): providers {
  key: "origins-0"
  value {
    issuer: "http://192.168.202.21:8000/auth/realms/istio"
    local_jwks {
      inline_string: ""
    }
    payload_in_metadata: "http://192.168.202.21:8000/auth/realms/istio"
  }
}
rules {
  match {
    prefix: "/"
  }
  requires {
    requires_any {
      requirements {
        provider_name: "origins-0"
      }
      requirements {
        allow_missing {
        }
      }
    }
  }
}


2020-07-09T09:36:55.706732Z     warning envoy config    [external/envoy/source/common/config/grpc_subscription_impl.cc:101] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) 0.0.0.0_8443: Proto constraint validation failed (JwtAuthenticationValidationError.Providers[key]: ["embedded message failed validation"] | caused by JwtProviderValidationError.LocalJwks: ["embedded message failed validation"] | caused by DataSourceValidationError.InlineString: ["value length must be at least " '\x01' " bytes"]): providers {
  key: "origins-0"
  value {
    issuer: "http://192.168.202.21:8000/auth/realms/istio"
    local_jwks {
      inline_string: ""
    }
    payload_in_metadata: "http://192.168.202.21:8000/auth/realms/istio"
  }
}
rules {
  match {
    prefix: "/"
  }
  requires {
    requires_any {
      requirements {
        provider_name: "origins-0"
      }
      requirements {
        allow_missing {
        }
      }
    }
  }
}

0.0.0.0_8080: Proto constraint validation failed (JwtAuthenticationValidationError.Providers[key]: ["embedded message failed validation"] | caused by JwtProviderValidationError.LocalJwks: ["embedded message failed validation"] | caused by DataSourceValidationError.InlineString: ["value length must be at least " '\x01' " bytes"]): providers {
  key: "origins-0"
  value {
    issuer: "http://192.168.202.21:8000/auth/realms/istio"
    local_jwks {
      inline_string: ""
    }
    payload_in_metadata: "http://192.168.202.21:8000/auth/realms/istio"
  }
}
rules {
  match {
    prefix: "/"
  }
  requires {
    requires_any {
      requirements {
        provider_name: "origins-0"
      }
      requirements {
        allow_missing {
        }
      }
    }
  }
}


2020-07-09T09:36:56.669676Z     warn    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 9 successful, 0 rejected; lds updates: 0 successful, 9 rejected
2020-07-09T09:36:58.669808Z     warn    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 9 successful, 0 rejected; lds updates: 0 successful, 9 rejected
2020-07-09T09:36:59.466385Z     warning envoy config    [external/envoy/source/common/config/grpc_subscription_impl.cc:101] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) 0.0.0.0_8443: Proto constraint validation failed (JwtAuthenticationValidationError.Providers[key]: ["embedded message failed validation"] | caused by JwtProviderValidationError.LocalJwks: ["embedded message failed validation"] | caused by DataSourceValidationError.InlineString: ["value length must be at least " '\x01' " bytes"]): providers {
  key: "origins-0"
  value {
    issuer: "http://192.168.202.21:8000/auth/realms/istio"
    local_jwks {
      inline_string: ""
    }
    payload_in_metadata: "http://192.168.202.21:8000/auth/realms/istio"
  }
}
rules {
  match {
    prefix: "/"
  }
  requires {
    requires_any {
      requirements {
        provider_name: "origins-0"
      }
      requirements {
        allow_missing {
        }
      }
    }
  }
}

0.0.0.0_8080: Proto constraint validation failed (JwtAuthenticationValidationError.Providers[key]: ["embedded message failed validation"] | caused by JwtProviderValidationError.LocalJwks: ["embedded message failed validation"] | caused by DataSourceValidationError.InlineString: ["value length must be at least " '\x01' " bytes"]): providers {
  key: "origins-0"
  value {
    issuer: "http://192.168.202.21:8000/auth/realms/istio"
    local_jwks {
      inline_string: ""
    }
    payload_in_metadata: "http://192.168.202.21:8000/auth/realms/istio"
  }
}
rules {
  match {
    prefix: "/"
  }
  requires {
    requires_any {
      requirements {
        provider_name: "origins-0"
      }
      requirements {
        allow_missing {
        }
      }
    }
  }
}


2020-07-09T09:37:00.669911Z     warn    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 11 successful, 0 rejected; lds updates: 0 successful, 10 rejected
2020-07-09T09:37:01.818665Z     warning envoy config    [external/envoy/source/common/config/grpc_subscription_impl.cc:101] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) 0.0.0.0_8443: Proto constraint validation failed (JwtAuthenticationValidationError.Providers[key]: ["embedded message failed validation"] | caused by JwtProviderValidationError.LocalJwks: ["embedded message failed validation"] | caused by DataSourceValidationError.InlineString: ["value length must be at least " '\x01' " bytes"]): providers {
  key: "origins-0"
  value {
    issuer: "http://192.168.202.21:8000/auth/realms/istio"
    local_jwks {
      inline_string: ""
    }
    payload_in_metadata: "http://192.168.202.21:8000/auth/realms/istio"
  }
}
rules {
  match {
    prefix: "/"
  }
  requires {
    requires_any {
      requirements {
        provider_name: "origins-0"
      }
      requirements {
        allow_missing {
        }
      }
    }
  }
}

0.0.0.0_8080: Proto constraint validation failed (JwtAuthenticationValidationError.Providers[key]: ["embedded message failed validation"] | caused by JwtProviderValidationError.LocalJwks: ["embedded message failed validation"] | caused by DataSourceValidationError.InlineString: ["value length must be at least " '\x01' " bytes"]): providers {
  key: "origins-0"
  value {
    issuer: "http://192.168.202.21:8000/auth/realms/istio"
    local_jwks {
      inline_string: ""
    }
    payload_in_metadata: "http://192.168.202.21:8000/auth/realms/istio"
  }
}
rules {
  match {
    prefix: "/"
  }
  requires {
    requires_any {
      requirements {
        provider_name: "origins-0"
      }
      requirements {
        allow_missing {
        }
      }
    }
  }
}


2020-07-09T09:37:02.669666Z     warn    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 11 successful, 0 rejected; lds updates: 0 successful, 11 rejected
2020-07-09T09:37:04.669697Z     warn    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 11 successful, 0 rejected; lds updates: 0 successful, 11 rejected
2020-07-09T09:37:06.669999Z     warn    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 11 successful, 0 rejected; lds updates: 0 successful, 11 rejected
2020-07-09T09:37:08.669745Z     warn    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 11 successful, 0 rejected; lds updates: 0 successful, 11 rejected
2020-07-09T09:37:10.675882Z     warn    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 11 successful, 0 rejected; lds updates: 0 successful, 11 rejected

What is going wrong?

Thread closed - Changing title in another thread