Hello istio community!
I am currently trying to integrate istio with the Jupyter enterprise-gateway application (LINK).
However, whenever I enable istio sidecar injection for the namespace the application is running in, the pods of enterprise-gateway fail to communicate with each other.
What is Jupyter enterprise-gateway?
The enterprise-gateway is kind of an operator that spawns kernel pod’s with python jupyter runtime environments for interactive use. The communication between the enterprise-gateway operator and the kernel pod’s is done using python websockets.
Here is a diagram of the enterprise-gateway communication:
and here is the github issue I have open with them:
where I also outline my hunch where the communication is stuck.
I tried to debug into the network communication using kiali but it is not really helpful because most of the traffic is routed through kubernetes.
I am currently stuck at enterprise-gateway starting the kernel pod. The kernel pod sends its connection information to enterprise-gateway here (enterprise_gateway/launch_ipykernel.py at c3650d0b0e46cbc481e47df04892986701894bd1 · jupyter-server/enterprise_gateway · GitHub) via Pod ip 10.100.xxx.xxx and port 8877 using python socket. I replaced the Pod ip with the service fqdn but it is still not working.
I connected a shell to the kernel pod and was able to directly connect to the enterprise-gateway pod via pod and cluster ip:
jovyan@guest-4d933a6c-9fb8-4b6d-ad55-ec71832677e2:~$ nc -v 172.20.111.3 8877 Connection to 172.20.111.3 8877 port [tcp/*] succeeded! ^C jovyan@guest-4d933a6c-9fb8-4b6d-ad55-ec71832677e2:~$ nc -v 10.100.34.44 8877 Connection to 10.100.34.44 8877 port [tcp/*] succeeded! ^C
Can anyone please point me in the right direction here?
What other information can I provide?