Issue integrating ELK with keycloak on istio-enabled setup, externally accessed via istio ingressgateway

Networking
1

We have integrated ELK (elasticsearch & kibana) with openid-based authentication via keycloak in kubernetes.

The high-level flow of interactions among the 3 (elasticsearch, kibana, keycloak) in-general is as follows:

  1. Kibana service start up
    During kibana service startup, kibana pod interacts with keycloak to check its availability (kibana-keycloak backend interaction).

  2. Kibana login
    i. User hits kibana access url which redirects to keycloak authentication page
    (Kibana-keycloak frontend interaction)
    ii. Keycloak authenticates the user and returns a jwt token to kibana.
    iii. Kibana sends this token in the backend to elasticsearch requesting for access. Elasticsearch sends this token to keycloak for token introspection. (Elasticsearch-keycloak backend interaction)

Key points:

  • There is only one parameter in kibana configuration to specify keycloak openid metadata url. The same parameter is used for both frontend & backend interactions.
  • The keycloak openid metadata url configured in elasticsearch should have the same hostname as configured in kibana. If they are different, token introspection would fail due to issuer hostname mismatch.

Issue:
When ELK and keycloak are running in istio-enabled k8s cluster, kibana & keycloak UI are independently (without ELK & keycloak integration) accessible on browser via istio ingressgateway loadbalancer.
Keycloak is externally accesible at https://{keycloak-fqdn}:{nodeport} , while kibana is externally accesible at https://{kibana-node-fqdn}:{nodeport}/kibana. (Required gateways and virtualservices created accordingly)

  1. To integrate ELK with keycloak, if the externally-accesible keycloak url with nodeport is configured in ELK, the internal communication between kibana & keycloak fails because the external keycloak fqdn is not resolvable from within the k8s cluster.

  2. If the internally accesible keycloak url (keycloak k8s service with service port) is configured in ELK, the internal communication between kibana & keycloak would succeed, but on accessing kibana in the browser, it would not be able to redirect to keycloak as the browser cannot resolve the keycloak k8s service and also the k8s service port would be unknown.

Expectation:
Is there a solution to overcome this problem?

Version Details:
Istio: 1.5.2
ELK: 7.0.1
Keycloak: 10.0.1
Kubernetes: 1.17.4

2

The problem as you state it is

“There is only one parameter in kibana configuration to specify keycloak openid metadata url. The same parameter is used for both frontend & backend interactions.”

Ideally separate user and service account settings are required as they have to point to different endpoints.

We deploy our java applications in their own servlet containers and use apache Shiro servlet filters to intercept the inbound user requests. Inspection of the http headers is done and redirection to Keycloak occurs if JWT’s are missing, invalid or expired.

This would allow you to use the Kibana setting to point to the backend Keycloak service using an internal service name reference resolvable within the cluster.

Hope that helps.