Enable istio for Jupyter enterprise-gateway

thesse-neoloan · October 28, 2022, 11:56am

Hello istio community!

I am currently trying to integrate istio with the Jupyter enterprise-gateway application (LINK).
However, whenever I enable istio sidecar injection for the namespace the application is running in, the pods of enterprise-gateway fail to communicate with each other.

What is Jupyter enterprise-gateway?
The enterprise-gateway is kind of an operator that spawns kernel pod’s with python jupyter runtime environments for interactive use. The communication between the enterprise-gateway operator and the kernel pod’s is done using python websockets.

Here is a diagram of the enterprise-gateway communication:
https://jupyter-enterprise-gateway.readthedocs.io/en/latest/contributors/sequence-diagrams.html

and here is the github issue I have open with them:

github.com/jupyter-server/enterprise_gateway

enterprise-gateway does not connect to k8s kernel when istio is configured

opened 03:31PM - 07 Oct 22 UTC

tahesse

kubernetes configuration

Hello… enterprise-gateway team! I checked all other issues in the jupyter-server organization and googled a lot without luck yet. ## Description I am trying to run enterprise-gateway in my kubernetes cluster to be able to run remote kernels. I installed enterprise-gateway via helm with the chart from this repository. I extended the helm chart with my own helm chart which does install a namespace with istio labels before installing enterprise-gateway. My `values-override.yaml` looks as follows: ```yaml # https://jupyter-enterprise-gateway.readthedocs.io/en/latest/operators/deploy-kubernetes.html#helm-configuration-parameters enterprise-gateway: # Enterprise Gateway image name and tag to use. image: elyra/enterprise-gateway:dev # Enterprise Gateway image pull policy. imagePullPolicy: IfNotPresent # Log output level. logLevel: DEBUG # Whether to mirror working directories. mirrorWorkingDirs: false # Optional authorization token passed in all requests (see --EnterpriseGatewayApp.auth_token) authToken: "" # K8s Enterprise Gateway Service service: type: "ClusterIP" ports: # The primary port on which Enterprise Gateway is servicing requests. - name: "http" port: 8888 targetPort: 8888 # The port on which Enterprise Gateway will receive kernel connection info responses. - name: "http-response" port: 8877 targetPort: 8877 # https://github.com/jupyter-server/enterprise_gateway/blob/main/etc/kubernetes/helm/enterprise-gateway/templates/ingress.yaml ingress: enabled: false kernel: # Kernel cluster role created by this chart. clusterRole: kernel-controller # Will start kernels in the same namespace as EG if True. shareGatewayNamespace: false # Timeout for kernel launching in seconds. launchTimeout: 120 # Timeout for an idle kernel before its culled in seconds. Default is 1 hour. cullIdleTimeout: 3600 # List of kernel names that are available for use. allowedKernels: - python_kubernetes - spark_python_operator # - r_kubernetes # - python_tf_kubernetes # - python_tf_gpu_kubernetes # - scala_kubernetes # - spark_r_kubernetes # - spark_python_kubernetes # - spark_scala_kubernetes # Default kernel name should be something from the whitelist defaultKernelName: spark_python_operator # Kernel Image Puller (daemonset) # https://github.com/jupyter-server/enterprise_gateway/blob/main/etc/kubernetes/helm/enterprise-gateway/templates/daemonset.yaml kip: enabled: false ``` For testing purposes, I ran kubefwd for all namespaces to be able to communicate with the enterprise-gateway service. I can successfully call the enterprise-gateway REST endpoints from the CLI, e.g. `curl http://enterprise-gateway.ns-jupyter:8888/api/kernelspecs | jq` yields <details> <summary>this JSON response.</summary> <pre> { "default": "spark_python_operator", "kernelspecs": { "python_kubernetes": { "name": "python_kubernetes", "spec": { "argv": [ "python", "/usr/local/share/jupyter/kernels/python_kubernetes/scripts/launch_kubernetes.py", "--RemoteProcessProxy.kernel-id", "{kernel_id}", "--RemoteProcessProxy.port-range", "{port_range}", "--RemoteProcessProxy.response-address", "{response_address}", "--RemoteProcessProxy.public-key", "{public_key}" ], "env": {}, "display_name": "Python on Kubernetes", "language": "python", "interrupt_mode": "signal", "metadata": { "process_proxy": { "class_name": "enterprise_gateway.services.processproxies.k8s.KubernetesProcessProxy", "config": { "image_name": "elyra/kernel-py:dev" } }, "debugger": true } }, "resources": { "logo-64x64": "/kernelspecs/python_kubernetes/logo-64x64.png" } }, "spark_python_operator": { "name": "spark_python_operator", "spec": { "argv": [ "python", "/usr/local/share/jupyter/kernels/spark_python_operator/scripts/launch_custom_resource.py", "--RemoteProcessProxy.kernel-id", "{kernel_id}", "--RemoteProcessProxy.port-range", "{port_range}", "--RemoteProcessProxy.response-address", "{response_address}", "--RemoteProcessProxy.public-key", "{public_key}" ], "env": {}, "display_name": "Spark Operator (Python)", "language": "python", "interrupt_mode": "signal", "metadata": { "process_proxy": { "class_name": "enterprise_gateway.services.processproxies.spark_operator.SparkOperatorProcessProxy", "config": { "image_name": "elyra/kernel-spark-py:dev", "executor_image_name": "elyra/kernel-spark-py:dev" } } } }, "resources": { "logo-64x64": "/kernelspecs/spark_python_operator/logo-64x64.png" } } } } </pre> </details> However, when I try to spawn a remote kernel via CLI ```shell curl -X POST -i 'http://enterprise-gateway.ns-jupyter:8888/api/kernels' --data '{ "name": "python_kubernetes", "env": { "KERNEL_USERNAME": "jovyan" }}' ``` or via jupyter lab ```shell python3 -m jupyterlab --debug \ --gateway-url=http://enterprise-gateway.ns-jupyter:8888 \ --GatewayClient.http_user=guest \ --GatewayClient.http_pwd=guest-password \ --GatewayClient.request_timeout=240.0 \ --GatewayClient.connect_timeout=240.0 ``` it spawns the remote kernel <details><summary>remote kernel logs</summary><pre>/usr/local/bin/bootstrap-kernel.sh env: SHELL=/bin/bash KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_SERVICE_PORT=443 KERNEL_NAME=python_kubernetes HOSTNAME=jovyan-8fb19249-6000-4847-9f2e-cdd3e70d7eb8 LANGUAGE=en_US.UTF-8 KERNEL_SPARK_CONTEXT_INIT_MODE=none KERNEL_ID=8fb19249-6000-4847-9f2e-cdd3e70d7eb8 NB_UID=1000 PWD=/home/jovyan RESPONSE_ADDRESS=10.100.44.42:8877 MINICONDA_MD5=87e77f097f6ebb5127c77662dfc3165e HOME=/home/jovyan LANG=en_US.UTF-8 KUBERNETES_PORT_443_TCP=tcp://172.20.77.1:443 NB_GID=100 XDG_CACHE_HOME=/home/jovyan/.cache/ SHLVL=0 CONDA_DIR=/opt/conda MINICONDA_VERSION=4.8.2 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_ADDR=172.20.77.1 PORT_RANGE=0..0 KERNEL_USERNAME=jovyan KERNEL_LANGUAGE=python CONDA_VERSION=4.8.2 NB_USER=jovyan KUBERNETES_SERVICE_HOST=172.20.77.1 LC_ALL=en_US.UTF-8 KUBERNETES_PORT=tcp://172.20.77.1:443 KUBERNETES_PORT_443_TCP_PORT=443 PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/conda/bin PUBLIC_KEY=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDEfiWkzCCMl/VFI8J2042RvWh13bSihVo+xp6HQnnQ8YWO5MsyW/nelzcMa2eBJWB+Yg/IQ/0q6BRog7oqDpUNbUxwGSzU3TyBYeRQCtXynR/EjFNyswE6gQrg15GbFxwmz4nfMkKXtlpItLrslcUqVY+wlUd+sdbJe9YMLp3REwIDAQAB DEBIAN_FRONTEND=noninteractive KERNEL_NAMESPACE=jovyan-8fb19249-6000-4847-9f2e-cdd3e70d7eb8 _=/usr/bin/env + python /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py --kernel-id 8fb19249-6000-4847-9f2e-cdd3e70d7eb8 --port-range 0..0 --response-address 10.100.44.42:8877 --public-key MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDEfiWkzCCMl/VFI8J2042RvWh13bSihVo+xp6HQnnQ8YWO5MsyW/nelzcMa2eBJWB+Yg/IQ/0q6BRog7oqDpUNbUxwGSzU3TyBYeRQCtXynR/EjFNyswE6gQrg15GbFxwmz4nfMkKXtlpItLrslcUqVY+wlUd+sdbJe9YMLp3REwIDAQAB --spark-context-initialization-mode none [D 2022-10-07 15:09:13,424.424 launch_ipykernel] Using connection file '/tmp/kernel-8fb19249-6000-4847-9f2e-cdd3e70d7eb8_3kfui2q3.json'. [I 2022-10-07 15:09:13,426.426 launch_ipykernel] Signal socket bound to host: 0.0.0.0, port: 59305 [D 2022-10-07 15:09:13,427.427 launch_ipykernel] JSON Payload 'b'{"shell_port": 44887, "iopub_port": 38565, "stdin_port": 60485, "control_port": 34723, "hb_port": 59659, "ip": "0.0.0.0", "key": "f50f4038-8f8e-40eb-8a70-20d0fa8469db", "transport": "tcp", "signature_scheme": "hmac-sha256", "kernel_name": "", "pid": 9, "pgid": 7, "comm_port": 59305, "kernel_id": "8fb19249-6000-4847-9f2e-cdd3e70d7eb8"}' [D 2022-10-07 15:09:13,441.441 launch_ipykernel] Encrypted Payload 'b'eyJ2ZXJzaW9uIjogMSwgImtleSI6ICJUbUlTbkxheGNiMVE0c2Uwc2Zka09zalhGOWJhdXROTmtidHNvbVdlUzBFN3M4Zk1Jcm9idFgxaXBSTlVTS1ZBNklXd0ZDeCtVT0JDNkR0aXlYR1BxN3RqTkpZTVVrYlpFaUEyZVhCUGN3bjhneW5adHF2ZmVTWTZkMThJL1k0MkRVN0lmblcwU3B3T0VIbFVZdE5VYVdLR2o5VGxlMXVIQ3R2SktVd1M2enc9IiwgImNvbm5faW5mbyI6ICJiWjNnOStkT3psZHRoTEJ3cGVwcGR1bnVCZ3UwdDhiMWVOR3hGRGprTFpuUitKb044YVFrb0V1NEFSenhRMDlDamZQektzTFNHUjQ3bnNTeUEzOGwyTklNMUY2cTE3LzJXSjVWNHJ3bk5KaGs5N3Vtdlo4ZHg0blc4dlpua2hrT3lvVnE3RExrdlUrb0Q4dHl2aDNTWHpGa3EyTTVaZHFRZXFSMHpvWTMvcWhKT2MzSDY0ckJwcHhaTzZJcTF6WDJEd2tZVk9oU3BuL1IrLzRvRGxoNWNYck55QXFBeUhPNUZGaXJjYUdWZmVPRlFzUXFKZ2VaTVdKaHFyazE4aFI2Ylo2b3dtVGovQ3JtRXN1YUIzaXRVZnU5VjJIcGluaWVLbzdaZ2t4dVF2dEgrcHJ2dzR6bkdObldOVEhrQUh6MUw2QTJkakVpRlhHNGYvbzdIc1lpb0IwSUZKVGk1elpzMmMvU00yQmlvZW5Vd1JSeFhoelZid01pdEV2L0s2KzVEdzI4MHgwajdHc3Q5OTF0eHR5VXNQUC9Qa1ZweGRuMjJYTGxmMG9sZVhlVG5DSTd5ZkZCLzJBM2dhZENZN3ZxNVpTRXlDdWt1QVppTjIxdzFiZVZKQT09In0</pre></details> but enterprise-gateway is unable to connect to the remote kernel as can be seen from the [enterprise-gateway pod logs](https://gist.github.com/tahesse/cd6709f682befa11ef0f9bc53767baca#file-enterprise-gateway-pod-log-txt). ## Reproduce 1. Deploy the helm chart with the provided `values-override.yaml` from the previous section. 2. Invoke the CLI or jupyter lab command from the <i>Description</i> section and start a `'python_kubernetes'` kernel. ## Expected behavior I would expect that enterprise-gateway receives the connection information from the remote kernel so that enterprise-gateway does not timeout. ## Context ```shell $ python3 -m jupyter --version Selected Jupyter core packages... IPython : 8.5.0 ipykernel : 6.16.0 ipywidgets : 8.0.2 jupyter_client : 7.3.5 jupyter_core : 4.11.1 jupyter_server : 1.19.1 jupyterlab : 3.4.8 nbclient : 0.6.8 nbconvert : 7.1.0 nbformat : 5.6.1 notebook : 6.4.12 qtconsole : 5.3.2 traitlets : 5.4.0 ```  <details><summary>Command Line Output</summary> <pre> [D 2022-10-07 17:22:23.377 ServerApp] Searching ['/Users/tahesse/Code/fincloud-ml/infra/jupyter-k8s/jupyter', '/Users/tahesse/.jupyter', '/Users/tahesse/Library/Python/3.9/etc/jupyter', '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files [D 2022-10-07 17:22:23.377 ServerApp] Looking for jupyter_config in /etc/jupyter [D 2022-10-07 17:22:23.377 ServerApp] Looking for jupyter_config in /usr/local/etc/jupyter [D 2022-10-07 17:22:23.377 ServerApp] Looking for jupyter_config in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter [D 2022-10-07 17:22:23.377 ServerApp] Looking for jupyter_config in /Users/tahesse/Library/Python/3.9/etc/jupyter [D 2022-10-07 17:22:23.377 ServerApp] Looking for jupyter_config in /Users/tahesse/.jupyter [D 2022-10-07 17:22:23.377 ServerApp] Looking for jupyter_config in /Users/tahesse/Code/fincloud-ml/infra/jupyter-k8s/jupyter [D 2022-10-07 17:22:23.378 ServerApp] Looking for jupyter_server_config in /etc/jupyter [D 2022-10-07 17:22:23.378 ServerApp] Looking for jupyter_server_config in /usr/local/etc/jupyter [D 2022-10-07 17:22:23.378 ServerApp] Looking for jupyter_server_config in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter [D 2022-10-07 17:22:23.378 ServerApp] Looking for jupyter_server_config in /Users/tahesse/Library/Python/3.9/etc/jupyter [D 2022-10-07 17:22:23.378 ServerApp] Looking for jupyter_server_config in /Users/tahesse/.jupyter [D 2022-10-07 17:22:23.378 ServerApp] Looking for jupyter_server_config in /Users/tahesse/Code/fincloud-ml/infra/jupyter-k8s/jupyter [D 2022-10-07 17:22:23.380 ServerApp] Paths used for configuration of jupyter_server_config: /etc/jupyter/jupyter_server_config.json [D 2022-10-07 17:22:23.380 ServerApp] Paths used for configuration of jupyter_server_config: /usr/local/etc/jupyter/jupyter_server_config.json [D 2022-10-07 17:22:23.380 ServerApp] Paths used for configuration of jupyter_server_config: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/jupyter_server_config.json [D 2022-10-07 17:22:23.381 ServerApp] Paths used for configuration of jupyter_server_config: /Users/tahesse/Library/Python/3.9/etc/jupyter/jupyter_server_config.d/jupyterlab.json /Users/tahesse/Library/Python/3.9/etc/jupyter/jupyter_server_config.d/nbclassic.json /Users/tahesse/Library/Python/3.9/etc/jupyter/jupyter_server_config.d/notebook_shim.json /Users/tahesse/Library/Python/3.9/etc/jupyter/jupyter_server_config.json [D 2022-10-07 17:22:23.394 ServerApp] Paths used for configuration of jupyter_server_config: /Users/tahesse/.jupyter/jupyter_server_config.json [D 2022-10-07 17:22:23.394 ServerApp] Paths used for configuration of jupyter_server_config: /Users/tahesse/Code/fincloud-ml/infra/jupyter-k8s/jupyter/jupyter_server_config.json [D 2022-10-07 17:22:23.425 LabApp] Config changed: {'NotebookApp': {}, 'ServerApp': {'log_level': 'DEBUG', 'jpserver_extensions': <LazyConfigValue {'update': {'nbclassic': True, 'notebook_shim': True}}>}, 'GatewayClient': {'url': 'http://enterprise-gateway.ns-jupyter:8888', 'http_user': 'guest', 'http_pwd': 'guest-password', 'request_timeout': 240.0, 'connect_timeout': 240.0}, 'ExtensionApp': {'log_level': 'DEBUG'}} [I 2022-10-07 17:22:23.426 ServerApp] jupyterlab | extension was successfully linked. [D 2022-10-07 17:22:23.433 NotebookApp] Config changed: {'NotebookApp': {}, 'ServerApp': {'log_level': 'DEBUG', 'jpserver_extensions': <LazyConfigValue value={'jupyterlab': True, 'nbclassic': True, 'notebook_shim': True}>}, 'GatewayClient': {'url': 'http://enterprise-gateway.ns-jupyter:8888', 'http_user': 'guest', 'http_pwd': 'guest-password', 'request_timeout': 240.0, 'connect_timeout': 240.0}, 'ExtensionApp': {'log_level': 'DEBUG'}} [I 2022-10-07 17:22:23.433 ServerApp] nbclassic | extension was successfully linked. [D 2022-10-07 17:22:23.655 ServerApp] Paths used for configuration of jupyter_notebook_config: /Users/tahesse/.jupyter/jupyter_notebook_config.json [D 2022-10-07 17:22:23.655 ServerApp] Paths used for configuration of jupyter_notebook_config: /etc/jupyter/jupyter_notebook_config.json [D 2022-10-07 17:22:23.655 ServerApp] Paths used for configuration of jupyter_notebook_config: /usr/local/etc/jupyter/jupyter_notebook_config.json [D 2022-10-07 17:22:23.656 ServerApp] Paths used for configuration of jupyter_notebook_config: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/jupyter_notebook_config.json [D 2022-10-07 17:22:23.656 ServerApp] Paths used for configuration of jupyter_notebook_config: /Users/tahesse/Library/Python/3.9/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json /Users/tahesse/Library/Python/3.9/etc/jupyter/jupyter_notebook_config.json [D 2022-10-07 17:22:23.666 ServerApp] Paths used for configuration of jupyter_notebook_config: /Users/tahesse/.jupyter/jupyter_notebook_config.json [I 2022-10-07 17:22:23.666 ServerApp] notebook_shim | extension was successfully linked. [D 2022-10-07 17:22:23.667 ServerApp] Config changed: {'GatewayClient': {'url': 'http://enterprise-gateway.ns-jupyter:8888', 'http_user': 'guest', 'http_pwd': 'guest-password', 'request_timeout': 240.0, 'connect_timeout': 240.0}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'NotebookApp': {}, 'ServerApp': {'log_level': 'DEBUG', 'jpserver_extensions': <LazyConfigValue value={'jupyterlab': True, 'nbclassic': True, 'notebook_shim': True}>}} [D 2022-10-07 17:22:23.667 ServerApp] Raising open file limit: soft 256->4096; hard 9223372036854775807->9223372036854775807 [I 2022-10-07 17:22:23.753 ServerApp] notebook_shim | extension was successfully loaded. [I 2022-10-07 17:22:23.754 LabApp] JupyterLab extension loaded from /Users/tahesse/Library/Python/3.9/lib/python/site-packages/jupyterlab [I 2022-10-07 17:22:23.754 LabApp] JupyterLab application directory is /Users/tahesse/Library/Python/3.9/share/jupyter/lab [I 2022-10-07 17:22:23.757 ServerApp] jupyterlab | extension was successfully loaded. [I 2022-10-07 17:22:23.802 ServerApp] nbclassic | extension was successfully loaded. [I 2022-10-07 17:22:23.803 ServerApp] The port 8888 is already in use, trying another port. [I 2022-10-07 17:22:23.804 ServerApp] Serving notebooks from local directory: /Users/tahesse/Code/fincloud-ml/infra/jupyter-k8s/jupyter [I 2022-10-07 17:22:23.804 ServerApp] Jupyter Server 1.19.1 is running at: [I 2022-10-07 17:22:23.804 ServerApp] http://localhost:8889/lab?token=f8eca77899eac00ea665cfc352b982b793d5d882e1641bdf [I 2022-10-07 17:22:23.804 ServerApp] or http://127.0.0.1:8889/lab?token=f8eca77899eac00ea665cfc352b982b793d5d882e1641bdf [I 2022-10-07 17:22:23.804 ServerApp] Kernels will be managed by the Gateway server running at: [I 2022-10-07 17:22:23.804 ServerApp] http://enterprise-gateway.ns-jupyter:8888 [I 2022-10-07 17:22:23.804 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 2022-10-07 17:22:23.818 ServerApp] To access the server, open this file in a browser: file:///Users/tahesse/Library/Jupyter/runtime/jpserver-90417-open.html Or copy and paste one of these URLs: http://localhost:8889/lab?token=f8eca77899eac00ea665cfc352b982b793d5d882e1641bdf or http://127.0.0.1:8889/lab?token=f8eca77899eac00ea665cfc352b982b793d5d882e1641bdf [D 2022-10-07 17:22:25.395 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:25.404 ServerApp] Paths used for configuration of page_config: /etc/jupyter/labconfig/page_config.json [D 2022-10-07 17:22:25.405 ServerApp] Paths used for configuration of page_config: /usr/local/etc/jupyter/labconfig/page_config.json [D 2022-10-07 17:22:25.406 ServerApp] Paths used for configuration of page_config: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/page_config.json [D 2022-10-07 17:22:25.406 ServerApp] Paths used for configuration of page_config: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/page_config.json [D 2022-10-07 17:22:25.406 ServerApp] Paths used for configuration of page_config: /Users/tahesse/.jupyter/labconfig/page_config.json [D 2022-10-07 17:22:25.464 LabApp] Using contents: services/contents [D 2022-10-07 17:22:25.485 LabApp] 200 GET /lab?token=f8eca77899eac00ea665cfc352b982b793d5d882e1641bdf (127.0.0.1) 93.99ms [D 2022-10-07 17:22:25.508 ServerApp] Path main.191f751bb4b62e169926.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/main.191f751bb4b62e169926.js [D 2022-10-07 17:22:25.542 ServerApp] 200 GET /static/lab/main.191f751bb4b62e169926.js?v=191f751bb4b62e169926 (127.0.0.1) 34.56ms [D 2022-10-07 17:22:25.559 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/remoteEntry.1116b848a6f7126bf3e9.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/remoteEntry.1116b848a6f7126bf3e9.js [D 2022-10-07 17:22:25.578 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/remoteEntry.1116b848a6f7126bf3e9.js (127.0.0.1) 19.93ms [D 2022-10-07 17:22:25.580 ServerApp] Path jupyterlab_pygments/static/remoteEntry.aa1060b2d1221f8e5688.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/jupyterlab_pygments/static/remoteEntry.aa1060b2d1221f8e5688.js [D 2022-10-07 17:22:25.584 ServerApp] 200 GET /lab/extensions/jupyterlab_pygments/static/remoteEntry.aa1060b2d1221f8e5688.js (127.0.0.1) 25.10ms [D 2022-10-07 17:22:25.586 ServerApp] Path favicons/favicon.ico served from /Users/tahesse/Library/Python/3.9/lib/python/site-packages/jupyter_server/static/favicons/favicon.ico [D 2022-10-07 17:22:25.594 ServerApp] 200 GET /static/favicons/favicon.ico (127.0.0.1) 8.39ms [D 2022-10-07 17:22:25.606 ServerApp] Path 4155.784ca1752696680bf373.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/4155.784ca1752696680bf373.js [D 2022-10-07 17:22:25.615 ServerApp] 200 GET /static/lab/4155.784ca1752696680bf373.js?v=784ca1752696680bf373 (127.0.0.1) 20.06ms [D 2022-10-07 17:22:25.615 ServerApp] Path 2719.3fe9e6fa24f270d8b8d1.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/2719.3fe9e6fa24f270d8b8d1.js [D 2022-10-07 17:22:25.657 ServerApp] 200 GET /static/lab/2719.3fe9e6fa24f270d8b8d1.js?v=3fe9e6fa24f270d8b8d1 (127.0.0.1) 61.41ms [D 2022-10-07 17:22:25.658 ServerApp] Path 9473.098fe5e3f0187b3dc1fe.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/9473.098fe5e3f0187b3dc1fe.js [D 2022-10-07 17:22:25.709 ServerApp] 200 GET /static/lab/9473.098fe5e3f0187b3dc1fe.js?v=098fe5e3f0187b3dc1fe (127.0.0.1) 112.91ms [D 2022-10-07 17:22:25.710 ServerApp] Path 6700.3c95b4eec62b695c6f74.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/6700.3c95b4eec62b695c6f74.js [D 2022-10-07 17:22:25.721 ServerApp] 200 GET /static/lab/6700.3c95b4eec62b695c6f74.js?v=3c95b4eec62b695c6f74 (127.0.0.1) 116.92ms [D 2022-10-07 17:22:25.722 ServerApp] Path 1036.b407e23ec5843f2b574d.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/1036.b407e23ec5843f2b574d.js [D 2022-10-07 17:22:25.759 ServerApp] 200 GET /static/lab/1036.b407e23ec5843f2b574d.js?v=b407e23ec5843f2b574d (127.0.0.1) 153.75ms [D 2022-10-07 17:22:25.761 ServerApp] Path jlab_core.716b79be8019e5bdc801.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/jlab_core.716b79be8019e5bdc801.js [D 2022-10-07 17:22:25.860 ServerApp] 200 GET /static/lab/jlab_core.716b79be8019e5bdc801.js?v=716b79be8019e5bdc801 (127.0.0.1) 244.69ms [D 2022-10-07 17:22:25.864 ServerApp] Path 1033.890eeae47460e241b1e9.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/1033.890eeae47460e241b1e9.js [D 2022-10-07 17:22:25.890 ServerApp] 200 GET /static/lab/1033.890eeae47460e241b1e9.js?v=890eeae47460e241b1e9 (127.0.0.1) 28.88ms [D 2022-10-07 17:22:25.892 ServerApp] Path 4570.53adcb6f69939da383ff.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/4570.53adcb6f69939da383ff.js [D 2022-10-07 17:22:25.906 ServerApp] 200 GET /static/lab/4570.53adcb6f69939da383ff.js?v=53adcb6f69939da383ff (127.0.0.1) 44.50ms [D 2022-10-07 17:22:25.906 ServerApp] Path 807.96166378e2efe232d81e.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/807.96166378e2efe232d81e.js [D 2022-10-07 17:22:25.978 ServerApp] 200 GET /static/lab/807.96166378e2efe232d81e.js?v=96166378e2efe232d81e (127.0.0.1) 115.60ms [D 2022-10-07 17:22:25.978 ServerApp] Path 7294.f71c2889fedcd71bd1ee.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/7294.f71c2889fedcd71bd1ee.js [D 2022-10-07 17:22:25.984 ServerApp] 200 GET /static/lab/7294.f71c2889fedcd71bd1ee.js?v=f71c2889fedcd71bd1ee (127.0.0.1) 121.53ms [D 2022-10-07 17:22:25.985 ServerApp] Path 3935.4159b022aa6d82e44127.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/3935.4159b022aa6d82e44127.js [D 2022-10-07 17:22:26.032 ServerApp] 200 GET /static/lab/3935.4159b022aa6d82e44127.js?v=4159b022aa6d82e44127 (127.0.0.1) 168.47ms [D 2022-10-07 17:22:26.032 ServerApp] Path 5096.96faf5ddac346a9ad9c3.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/5096.96faf5ddac346a9ad9c3.js [D 2022-10-07 17:22:26.056 ServerApp] 200 GET /static/lab/5096.96faf5ddac346a9ad9c3.js?v=96faf5ddac346a9ad9c3 (127.0.0.1) 165.94ms [D 2022-10-07 17:22:26.063 ServerApp] Path 1142.074d125bb59f5a332666.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/1142.074d125bb59f5a332666.js [D 2022-10-07 17:22:26.072 ServerApp] 200 GET /static/lab/1142.074d125bb59f5a332666.js?v=074d125bb59f5a332666 (127.0.0.1) 13.72ms [D 2022-10-07 17:22:26.074 ServerApp] Path 7730.7e3a9fb140d2d55a51fc.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/7730.7e3a9fb140d2d55a51fc.js [D 2022-10-07 17:22:26.076 ServerApp] 200 GET /static/lab/7730.7e3a9fb140d2d55a51fc.js?v=7e3a9fb140d2d55a51fc (127.0.0.1) 17.94ms [D 2022-10-07 17:22:26.077 ServerApp] Path 3308.3a94151c0d57440646ab.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/3308.3a94151c0d57440646ab.js [D 2022-10-07 17:22:26.087 ServerApp] 200 GET /static/lab/3308.3a94151c0d57440646ab.js?v=3a94151c0d57440646ab (127.0.0.1) 27.34ms [D 2022-10-07 17:22:26.087 ServerApp] Path 4631.96a143e70f005fef7b59.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/4631.96a143e70f005fef7b59.js [D 2022-10-07 17:22:26.097 ServerApp] 200 GET /static/lab/4631.96a143e70f005fef7b59.js?v=96a143e70f005fef7b59 (127.0.0.1) 35.20ms [D 2022-10-07 17:22:26.097 ServerApp] Path 7755.d506a1d9dadf30b1e490.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/7755.d506a1d9dadf30b1e490.js [D 2022-10-07 17:22:26.104 ServerApp] 200 GET /static/lab/7755.d506a1d9dadf30b1e490.js?v=d506a1d9dadf30b1e490 (127.0.0.1) 41.44ms [D 2022-10-07 17:22:26.104 ServerApp] Path 6080.39703b8cc7bd284ce1bd.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/6080.39703b8cc7bd284ce1bd.js [D 2022-10-07 17:22:26.131 ServerApp] 200 GET /static/lab/6080.39703b8cc7bd284ce1bd.js?v=39703b8cc7bd284ce1bd (127.0.0.1) 57.94ms [D 2022-10-07 17:22:26.136 ServerApp] Path 2040.6489997c35a5d7b0b100.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/2040.6489997c35a5d7b0b100.js [D 2022-10-07 17:22:26.148 ServerApp] 200 GET /static/lab/2040.6489997c35a5d7b0b100.js?v=6489997c35a5d7b0b100 (127.0.0.1) 16.17ms [D 2022-10-07 17:22:26.150 ServerApp] Path 1057.a0608544097dd22a589e.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/1057.a0608544097dd22a589e.js [D 2022-10-07 17:22:26.154 ServerApp] 200 GET /static/lab/1057.a0608544097dd22a589e.js?v=a0608544097dd22a589e (127.0.0.1) 19.07ms [D 2022-10-07 17:22:26.154 ServerApp] Path 3502.c915d993b864739407e3.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/3502.c915d993b864739407e3.js [D 2022-10-07 17:22:26.171 ServerApp] 200 GET /static/lab/3502.c915d993b864739407e3.js?v=c915d993b864739407e3 (127.0.0.1) 35.42ms [D 2022-10-07 17:22:26.172 ServerApp] Path 3029.8636761a48c8422ce37f.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/3029.8636761a48c8422ce37f.js [D 2022-10-07 17:22:26.181 ServerApp] 200 GET /static/lab/3029.8636761a48c8422ce37f.js?v=8636761a48c8422ce37f (127.0.0.1) 45.05ms [D 2022-10-07 17:22:26.181 ServerApp] Path 3219.8a3701725aaa457179f6.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/3219.8a3701725aaa457179f6.js [D 2022-10-07 17:22:26.187 ServerApp] 200 GET /static/lab/3219.8a3701725aaa457179f6.js?v=8a3701725aaa457179f6 (127.0.0.1) 50.77ms [D 2022-10-07 17:22:26.187 ServerApp] Path 4151.fd7ca59b64f0adc97032.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/4151.fd7ca59b64f0adc97032.js [D 2022-10-07 17:22:26.191 ServerApp] 200 GET /static/lab/4151.fd7ca59b64f0adc97032.js?v=fd7ca59b64f0adc97032 (127.0.0.1) 41.75ms [D 2022-10-07 17:22:26.192 ServerApp] Path 1249.047c76b5ea96a41605db.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/1249.047c76b5ea96a41605db.js [D 2022-10-07 17:22:26.202 ServerApp] 200 GET /static/lab/1249.047c76b5ea96a41605db.js?v=047c76b5ea96a41605db (127.0.0.1) 10.68ms [D 2022-10-07 17:22:26.204 ServerApp] Path 1358.9ab4f57e8227ce75f427.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/1358.9ab4f57e8227ce75f427.js [D 2022-10-07 17:22:26.213 ServerApp] 200 GET /static/lab/1358.9ab4f57e8227ce75f427.js?v=9ab4f57e8227ce75f427 (127.0.0.1) 21.61ms [D 2022-10-07 17:22:26.214 ServerApp] Path 4429.c4f083ef6b6e29345fd4.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/4429.c4f083ef6b6e29345fd4.js [D 2022-10-07 17:22:26.219 ServerApp] 200 GET /static/lab/4429.c4f083ef6b6e29345fd4.js?v=c4f083ef6b6e29345fd4 (127.0.0.1) 27.66ms [D 2022-10-07 17:22:26.220 ServerApp] Path 466.239a0c009f33f1cb7cab.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/466.239a0c009f33f1cb7cab.js [D 2022-10-07 17:22:26.226 ServerApp] 200 GET /static/lab/466.239a0c009f33f1cb7cab.js?v=239a0c009f33f1cb7cab (127.0.0.1) 33.70ms [D 2022-10-07 17:22:26.226 ServerApp] Path 3496.ecb0e7fcc54191234ae6.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/3496.ecb0e7fcc54191234ae6.js [D 2022-10-07 17:22:26.230 ServerApp] 200 GET /static/lab/3496.ecb0e7fcc54191234ae6.js?v=ecb0e7fcc54191234ae6 (127.0.0.1) 37.58ms [D 2022-10-07 17:22:26.230 ServerApp] Path 126.1e2fb998804b27c72a3e.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/126.1e2fb998804b27c72a3e.js [D 2022-10-07 17:22:26.244 ServerApp] 200 GET /static/lab/126.1e2fb998804b27c72a3e.js?v=1e2fb998804b27c72a3e (127.0.0.1) 40.89ms [D 2022-10-07 17:22:26.246 ServerApp] Path 911.0c08f040896753efc653.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/911.0c08f040896753efc653.js [D 2022-10-07 17:22:26.252 ServerApp] 200 GET /static/lab/911.0c08f040896753efc653.js?v=0c08f040896753efc653 (127.0.0.1) 7.23ms [D 2022-10-07 17:22:26.252 ServerApp] Path 714.ff3007d0b781b5538482.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/714.ff3007d0b781b5538482.js [D 2022-10-07 17:22:26.301 ServerApp] 200 GET /static/lab/714.ff3007d0b781b5538482.js?v=ff3007d0b781b5538482 (127.0.0.1) 55.57ms [D 2022-10-07 17:22:26.301 ServerApp] Path 7796.098f091848a062457e84.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/7796.098f091848a062457e84.js [D 2022-10-07 17:22:26.308 ServerApp] 200 GET /static/lab/7796.098f091848a062457e84.js?v=098f091848a062457e84 (127.0.0.1) 62.26ms [D 2022-10-07 17:22:26.364 ServerApp] Path jupyterlab_pygments/static/568.1e2faa2ba0bbe59c4780.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/jupyterlab_pygments/static/568.1e2faa2ba0bbe59c4780.js [D 2022-10-07 17:22:26.366 ServerApp] 200 GET /lab/extensions/jupyterlab_pygments/static/568.1e2faa2ba0bbe59c4780.js?v=1e2faa2ba0bbe59c4780 (127.0.0.1) 2.50ms [D 2022-10-07 17:22:26.368 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/150.3e1e5adfd821b9b96340.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/150.3e1e5adfd821b9b96340.js [D 2022-10-07 17:22:26.375 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/150.3e1e5adfd821b9b96340.js?v=3e1e5adfd821b9b96340 (127.0.0.1) 8.76ms [D 2022-10-07 17:22:26.376 ServerApp] Path jupyterlab_pygments/static/747.8eb3ddccc7ec4987bff9.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/jupyterlab_pygments/static/747.8eb3ddccc7ec4987bff9.js [D 2022-10-07 17:22:26.388 ServerApp] 200 GET /lab/extensions/jupyterlab_pygments/static/747.8eb3ddccc7ec4987bff9.js?v=8eb3ddccc7ec4987bff9 (127.0.0.1) 20.69ms [D 2022-10-07 17:22:26.391 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/644.11f638668109f1562dd1.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/644.11f638668109f1562dd1.js [D 2022-10-07 17:22:26.405 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/644.11f638668109f1562dd1.js?v=11f638668109f1562dd1 (127.0.0.1) 16.39ms [D 2022-10-07 17:22:26.405 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/291.f707f721e4b3a5489ee0.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/291.f707f721e4b3a5489ee0.js [D 2022-10-07 17:22:26.430 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/291.f707f721e4b3a5489ee0.js?v=f707f721e4b3a5489ee0 (127.0.0.1) 41.09ms [D 2022-10-07 17:22:26.431 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/595.e3c9c115ecf5763f080b.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/595.e3c9c115ecf5763f080b.js [D 2022-10-07 17:22:26.446 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/595.e3c9c115ecf5763f080b.js?v=e3c9c115ecf5763f080b (127.0.0.1) 56.37ms [D 2022-10-07 17:22:26.447 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/113.dd66397047ecb9a605cf.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/113.dd66397047ecb9a605cf.js [D 2022-10-07 17:22:26.469 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/113.dd66397047ecb9a605cf.js?v=dd66397047ecb9a605cf (127.0.0.1) 79.03ms [D 2022-10-07 17:22:26.471 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/699.d004d23061da64f55a9a.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/699.d004d23061da64f55a9a.js [D 2022-10-07 17:22:26.480 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/699.d004d23061da64f55a9a.js?v=d004d23061da64f55a9a (127.0.0.1) 10.04ms [D 2022-10-07 17:22:26.480 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/965.8b60ee098074f7ed452d.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/965.8b60ee098074f7ed452d.js [D 2022-10-07 17:22:26.482 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/965.8b60ee098074f7ed452d.js?v=8b60ee098074f7ed452d (127.0.0.1) 11.65ms [D 2022-10-07 17:22:26.483 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/134.bcbea9feb6e7c4da7530.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/134.bcbea9feb6e7c4da7530.js [D 2022-10-07 17:22:26.493 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/134.bcbea9feb6e7c4da7530.js?v=bcbea9feb6e7c4da7530 (127.0.0.1) 10.99ms [D 2022-10-07 17:22:26.510 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:26.511 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:26.511 ServerApp] Paths used for configuration of default_setting_overrides: /etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:26.512 ServerApp] Paths used for configuration of default_setting_overrides: /usr/local/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:26.512 ServerApp] Paths used for configuration of default_setting_overrides: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:26.512 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:26.512 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/.jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:26.512 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:26.513 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:26.513 ServerApp] Request list kernel specs at: http://enterprise-gateway.ns-jupyter:8888/api/kernelspecs [D 2022-10-07 17:22:26.844 LabApp] 200 GET /lab/api/settings?1665156146509 (127.0.0.1) 333.67ms [D 2022-10-07 17:22:26.845 ServerApp] Request list kernels: http://enterprise-gateway.ns-jupyter:8888/api/kernels [D 2022-10-07 17:22:26.847 ServerApp] 200 GET /api/sessions?1665156146510 (127.0.0.1) 334.97ms [D 2022-10-07 17:22:26.854 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:26.857 ServerApp] 200 GET /api/terminals?1665156146510 (127.0.0.1) 3.08ms [D 2022-10-07 17:22:26.998 ServerApp] 200 GET /api/kernels?1665156146510 (127.0.0.1) 485.42ms [I 2022-10-07 17:22:27.015 ServerApp] Default kernel name on Gateway server (spark_python_operator) differs from Notebook server (python3). Updating to Gateway server's value. [D 2022-10-07 17:22:27.016 ServerApp] 200 GET /api/kernelspecs?1665156146507 (127.0.0.1) 506.02ms [D 2022-10-07 17:22:27.019 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:27.020 ServerApp] Request list kernel specs at: http://enterprise-gateway.ns-jupyter:8888/api/kernelspecs [D 2022-10-07 17:22:27.196 ServerApp] 200 GET /api/kernelspecs?1665156147018 (127.0.0.1) 176.54ms [D 2022-10-07 17:22:27.908 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:27.913 ServerApp] Paths used for configuration of default_setting_overrides: /etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:27.914 ServerApp] Paths used for configuration of default_setting_overrides: /usr/local/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:27.915 ServerApp] Paths used for configuration of default_setting_overrides: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:27.916 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:27.916 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/.jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.082 ServerApp] 200 GET /lab/api/translations/en?1665156147899 (127.0.0.1) 174.89ms [D 2022-10-07 17:22:28.106 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:28.121 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:28.133 ServerApp] 200 GET /lab/api/listings/@jupyterlab/extensionmanager-extension/listings.json?1665156148109 (127.0.0.1) 17.92ms [D 2022-10-07 17:22:28.134 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:28.136 ServerApp] 200 GET /api/contents?content=1&1665156148112 (127.0.0.1) 2.67ms [D 2022-10-07 17:22:28.137 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:28.137 ServerApp] Paths used for configuration of default_setting_overrides: /etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.138 ServerApp] Paths used for configuration of default_setting_overrides: /usr/local/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.138 ServerApp] Paths used for configuration of default_setting_overrides: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.138 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.138 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/.jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.141 LabApp] 200 GET /lab/api/settings/@jupyterlab/mainmenu-extension:plugin?1665156148125 (127.0.0.1) 4.86ms [D 2022-10-07 17:22:28.142 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:28.142 ServerApp] Paths used for configuration of default_setting_overrides: /etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.143 ServerApp] Paths used for configuration of default_setting_overrides: /usr/local/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.143 ServerApp] Paths used for configuration of default_setting_overrides: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.143 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.143 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/.jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.143 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:28.144 ServerApp] Paths used for configuration of default_setting_overrides: /etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.144 ServerApp] Paths used for configuration of default_setting_overrides: /usr/local/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.144 ServerApp] Paths used for configuration of default_setting_overrides: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.144 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.144 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/.jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.144 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:28.145 ServerApp] Paths used for configuration of default_setting_overrides: /etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.145 ServerApp] Paths used for configuration of default_setting_overrides: /usr/local/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.145 ServerApp] Paths used for configuration of default_setting_overrides: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.145 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.145 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/.jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.145 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:28.146 ServerApp] Paths used for configuration of default_setting_overrides: /etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.146 ServerApp] Paths used for configuration of default_setting_overrides: /usr/local/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.146 ServerApp] Paths used for configuration of default_setting_overrides: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.146 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.146 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/.jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.148 LabApp] 200 GET /lab/api/settings/@jupyterlab/htmlviewer-extension:plugin?1665156148126 (127.0.0.1) 6.30ms [D 2022-10-07 17:22:28.149 LabApp] 200 GET /lab/api/settings/@jupyterlab/cell-toolbar-extension:plugin?1665156148124 (127.0.0.1) 5.84ms [D 2022-10-07 17:22:28.150 LabApp] 200 GET /lab/api/settings/@jupyterlab/shortcuts-extension:shortcuts?1665156148124 (127.0.0.1) 6.10ms [D 2022-10-07 17:22:28.153 LabApp] 200 GET /lab/api/settings/@jupyterlab/docmanager-extension:plugin?1665156148125 (127.0.0.1) 7.22ms [D 2022-10-07 17:22:28.154 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:28.154 ServerApp] Paths used for configuration of default_setting_overrides: /etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.154 ServerApp] Paths used for configuration of default_setting_overrides: /usr/local/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.155 ServerApp] Paths used for configuration of default_setting_overrides: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.155 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.155 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/.jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.157 LabApp] 200 GET /lab/api/settings/@jupyterlab/notebook-extension:panel?1665156148126 (127.0.0.1) 3.22ms [D 2022-10-07 17:22:28.157 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:28.158 ServerApp] Paths used for configuration of default_setting_overrides: /etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.158 ServerApp] Paths used for configuration of default_setting_overrides: /usr/local/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.158 ServerApp] Paths used for configuration of default_setting_overrides: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.158 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.158 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/.jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:28.160 LabApp] Node v16.16.0 [D 2022-10-07 17:22:28.295 LabApp] Yarn configuration loaded. [D 2022-10-07 17:22:29.200 ServerApp] 200 GET /lab/api/translations/?1665156148154 (127.0.0.1) 1042.68ms [D 2022-10-07 17:22:29.201 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [I 2022-10-07 17:22:29.384 LabApp] Build is up to date [D 2022-10-07 17:22:29.389 LabApp] 200 GET /lab/api/build?1665156148103 (127.0.0.1) 1283.76ms [D 2022-10-07 17:22:29.389 ServerApp] Path @jupyterlab/theme-light-extension/index.css served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/themes/@jupyterlab/theme-light-extension/index.css [D 2022-10-07 17:22:29.406 ServerApp] 200 GET /lab/api/themes/@jupyterlab/theme-light-extension/index.css (127.0.0.1) 22.31ms [D 2022-10-07 17:22:29.407 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:29.407 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:29.408 ServerApp] Paths used for configuration of default_setting_overrides: /etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.408 ServerApp] Paths used for configuration of default_setting_overrides: /usr/local/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.408 ServerApp] Paths used for configuration of default_setting_overrides: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.408 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.409 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/.jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.409 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:29.409 ServerApp] Paths used for configuration of default_setting_overrides: /etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.409 ServerApp] Paths used for configuration of default_setting_overrides: /usr/local/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.410 ServerApp] Paths used for configuration of default_setting_overrides: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.410 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.410 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/.jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.414 LabApp] 200 GET /lab/api/settings/@jupyterlab/filebrowser-extension:widget?1665156148190 (127.0.0.1) 7.30ms [D 2022-10-07 17:22:29.417 LabApp] 200 GET /lab/api/settings/@jupyterlab/csvviewer-extension:csv?1665156148190 (127.0.0.1) 8.02ms [D 2022-10-07 17:22:29.417 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:29.418 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:29.418 ServerApp] Paths used for configuration of default_setting_overrides: /etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.419 ServerApp] Paths used for configuration of default_setting_overrides: /usr/local/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.419 ServerApp] Paths used for configuration of default_setting_overrides: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.419 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.419 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/.jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.430 LabApp] 200 GET /lab/api/workspaces/default?1665156148237 (127.0.0.1) 13.14ms [D 2022-10-07 17:22:29.439 LabApp] 200 GET /lab/api/settings/@jupyterlab/application-extension:context-menu?1665156148246 (127.0.0.1) 21.62ms [D 2022-10-07 17:22:29.440 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:29.441 ServerApp] Paths used for configuration of default_setting_overrides: /etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.441 ServerApp] Paths used for configuration of default_setting_overrides: /usr/local/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.441 ServerApp] Paths used for configuration of default_setting_overrides: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.441 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.442 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/.jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.442 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:29.442 ServerApp] Paths used for configuration of default_setting_overrides: /etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.443 ServerApp] Paths used for configuration of default_setting_overrides: /usr/local/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.443 ServerApp] Paths used for configuration of default_setting_overrides: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.443 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.443 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/.jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.448 LabApp] 200 GET /lab/api/settings/@jupyterlab/shortcuts-extension:shortcuts?1665156148268 (127.0.0.1) 8.37ms [D 2022-10-07 17:22:29.451 LabApp] 200 GET /lab/api/settings/@jupyterlab/csvviewer-extension:tsv?1665156148191 (127.0.0.1) 9.58ms [D 2022-10-07 17:22:29.452 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:29.453 ServerApp] Paths used for configuration of default_setting_overrides: /etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.453 ServerApp] Paths used for configuration of default_setting_overrides: /usr/local/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.454 ServerApp] Paths used for configuration of default_setting_overrides: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.454 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.455 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/.jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.463 LabApp] 200 GET /lab/api/settings/@jupyterlab/fileeditor-extension:plugin?1665156148191 (127.0.0.1) 10.64ms [D 2022-10-07 17:22:29.474 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:29.474 ServerApp] Paths used for configuration of default_setting_overrides: /etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.475 ServerApp] Paths used for configuration of default_setting_overrides: /usr/local/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.475 ServerApp] Paths used for configuration of default_setting_overrides: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.475 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.475 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/.jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.678 LabApp] 200 GET /lab/api/settings?1665156149472 (127.0.0.1) 204.78ms [D 2022-10-07 17:22:29.679 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:29.680 ServerApp] 200 GET /api/sessions?1665156149473 (127.0.0.1) 0.86ms [D 2022-10-07 17:22:29.680 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:29.681 ServerApp] Paths used for configuration of default_setting_overrides: /etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.681 ServerApp] Paths used for configuration of default_setting_overrides: /usr/local/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.682 ServerApp] Paths used for configuration of default_setting_overrides: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.682 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.682 ServerApp] Paths used for configuration of default_setting_overrides: /Users/tahesse/.jupyter/labconfig/default_setting_overrides.json [D 2022-10-07 17:22:29.684 LabApp] 200 GET /lab/api/settings/@jupyterlab/shortcuts-extension:shortcuts?1665156149575 (127.0.0.1) 4.63ms [D 2022-10-07 17:22:29.700 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [I 2022-10-07 17:22:29.701 ServerApp] Request start kernel: kernel_id=None, path='' [D 2022-10-07 17:22:29.701 ServerApp] Request new kernel at: http://enterprise-gateway.ns-jupyter:8888/api/kernels [D 2022-10-07 17:22:29.808 ServerApp] Using contents: services/contents [D 2022-10-07 17:22:29.852 ServerApp] Path favicon.ico served from /Users/tahesse/Library/Python/3.9/lib/python/site-packages/jupyter_server/static/favicon.ico [D 2022-10-07 17:22:29.857 ServerApp] Path style/bootstrap.min.css served from /Users/tahesse/Library/Python/3.9/lib/python/site-packages/jupyter_server/static/style/bootstrap.min.css [D 2022-10-07 17:22:29.866 ServerApp] Path style/bootstrap-theme.min.css served from /Users/tahesse/Library/Python/3.9/lib/python/site-packages/jupyter_server/static/style/bootstrap-theme.min.css [D 2022-10-07 17:22:29.875 ServerApp] Path style/index.css served from /Users/tahesse/Library/Python/3.9/lib/python/site-packages/jupyter_server/static/style/index.css [D 2022-10-07 17:22:29.880 ServerApp] Path logo/logo.png served from /Users/tahesse/Library/Python/3.9/lib/python/site-packages/jupyter_server/static/logo/logo.png [W 2022-10-07 17:22:29.882 ServerApp] 404 GET /static/lab/jlab_core.716b79be8019e5bdc801.js.map?v=716b79be8019e5bdc801 (127.0.0.1) 74.79ms referer=None [D 2022-10-07 17:22:29.926 ServerApp] 200 GET /api/nbconvert?1665156148162 (127.0.0.1) 519.28ms [D 2022-10-07 17:22:29.992 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:29.996 LabApp] 204 PUT /lab/api/workspaces/default?1665156149991 (127.0.0.1) 3.99ms [D 2022-10-07 17:22:29.997 ServerApp] 200 GET /api/nbconvert?1665156148161 (127.0.0.1) 796.41ms [D 2022-10-07 17:22:30.008 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:30.009 ServerApp] 200 GET /api/contents?content=1&1665156150007 (127.0.0.1) 1.86ms [D 2022-10-07 17:22:36.935 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:36.937 ServerApp] 200 GET /api/terminals?1665156156927 (127.0.0.1) 2.82ms [D 2022-10-07 17:22:37.007 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:37.008 ServerApp] Request list kernels: http://enterprise-gateway.ns-jupyter:8888/api/kernels [D 2022-10-07 17:22:37.145 ServerApp] 200 GET /api/kernels?1665156157004 (127.0.0.1) 138.73ms [D 2022-10-07 17:22:39.707 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:39.709 ServerApp] 200 GET /api/sessions?1665156159699 (127.0.0.1) 3.15ms [D 2022-10-07 17:22:40.018 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:40.026 ServerApp] 200 GET /api/contents?content=1&1665156160015 (127.0.0.1) 7.99ms [D 2022-10-07 17:22:46.954 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:46.956 ServerApp] 200 GET /api/terminals?1665156166948 (127.0.0.1) 2.80ms [D 2022-10-07 17:22:47.156 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:47.157 ServerApp] Request list kernels: http://enterprise-gateway.ns-jupyter:8888/api/kernels [D 2022-10-07 17:22:47.293 ServerApp] 200 GET /api/kernels?1665156167151 (127.0.0.1) 136.77ms [D 2022-10-07 17:22:49.720 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:49.723 ServerApp] 200 GET /api/sessions?1665156169716 (127.0.0.1) 4.12ms [D 2022-10-07 17:22:50.041 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:50.048 ServerApp] 200 GET /api/contents?content=1&1665156170034 (127.0.0.1) 7.76ms [D 2022-10-07 17:22:56.973 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:56.976 ServerApp] 200 GET /api/terminals?1665156176966 (127.0.0.1) 3.96ms [D 2022-10-07 17:22:57.304 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:57.306 ServerApp] Request list kernels: http://enterprise-gateway.ns-jupyter:8888/api/kernels [D 2022-10-07 17:22:57.447 ServerApp] 200 GET /api/kernels?1665156177299 (127.0.0.1) 142.16ms [D 2022-10-07 17:22:59.735 ServerApp] Accepting token-authenticated connection from 127.0.0.1 [D 2022-10-07 17:22:59.738 ServerApp] 200 GET /api/sessions?1665156179727 (127.0.0.1) 4.94ms </pre> </details>  If I can provide any more information, please let me know! Thank you for your work on enterprise-gateway, and thanks for any help / pointers in the right direction! :)

where I also outline my hunch where the communication is stuck.

I tried to debug into the network communication using kiali but it is not really helpful because most of the traffic is routed through kubernetes.
I am currently stuck at enterprise-gateway starting the kernel pod. The kernel pod sends its connection information to enterprise-gateway here (enterprise_gateway/launch_ipykernel.py at c3650d0b0e46cbc481e47df04892986701894bd1 · jupyter-server/enterprise_gateway · GitHub) via Pod ip 10.100.xxx.xxx and port 8877 using python socket. I replaced the Pod ip with the service fqdn but it is still not working.

I connected a shell to the kernel pod and was able to directly connect to the enterprise-gateway pod via pod and cluster ip:

jovyan@guest-4d933a6c-9fb8-4b6d-ad55-ec71832677e2:~$ nc -v 172.20.111.3 8877
Connection to 172.20.111.3 8877 port [tcp/*] succeeded!
^C
jovyan@guest-4d933a6c-9fb8-4b6d-ad55-ec71832677e2:~$ nc -v 10.100.34.44 8877
Connection to 10.100.34.44 8877 port [tcp/*] succeeded!
^C

Can anyone please point me in the right direction here?
What other information can I provide?

thesse-neoloan · December 1, 2022, 1:43pm

Does anyone have an idea how I could possibly proceed?

Topic		Replies	Views
Istio 1.7.4 with another prometheus in monitoring namespace	4	2117	December 21, 2020
Separate helm releases for Istio and gateways	5	1619	June 25, 2019
Istio injection failed	1	549	October 5, 2020
Istio 1.5 helm template installation to istioctl installation with no downtime?	0	287	November 27, 2020
Istio-cni "istio-iptables.sh: No such file or directory" Networking	1	970	June 23, 2020

Enable istio for Jupyter enterprise-gateway

Related Topics