Jupyter
The Jupyter Lab and Notebook is a web application for creating and sharing documents that contain code, visualizations, and text. It can be used for data science, statistical modeling, machine learning, and more.
Running on a Compute Node via sbatch
Submitting a job via sbatch will launch Jupyter on a compute node while you can access this from your desktop browser.
In order to do this we need a submission script like the following called jupyter.sh
#!/bin/bash
#SBATCH --job-name=jupyter # Job name
#SBATCH --partition=sixhour # Partition Name (Required)
#SBATCH --ntasks=1 # Run on a single CPU
#SBATCH --mem=2gb # Job memory request
#SBATCH --time=0-06:00:00 # Time limit hrs:min:sec
#SBATCH --output=jupyter.log # Standard output and error log
echo "HOSTNAME: $(hostname)"
module load jupyter-notebook/latest
jupyter-notebook --no-browser --ip=0.0.0.0
HPC Examples
Check out the HPC Examples Gitlab repo which includes the jupyter.sh script
This job launches Jupyter on the allocated compute node which you can access through a ssh tunnel
First, from the submit node, submit the job to the queue:
$ sbatch jupyter.sh
Once the job is running, a log file will be created that is called jupyter.log. The log file contains information on how to connect to Jupyter, and the necessary token.
In order to connect to Jupyter that is running on the compute node, set up a SSH tunnel on your desktop.
SSH Tunnel
First, find the hostname and the port after the hostname in the URL in the jupyter.log file. In this case, the hostname is r16r27n01 and the port is 8888. The jupyter.log will look similar to the one below:
HOSTNAME: r16r27n01
[I 2023-12-11 18:46:22.585 ServerApp] jupyter_lsp | extension was successfully linked.
...
[I 2023-12-11 18:46:23.816 ServerApp] Jupyter Server 2.12.1 is running at:
[I 2023-12-11 18:46:23.816 ServerApp] http://r16r27n01:8888/tree?token=35c72a2ce9e92eb04fdac8cf5168c49c5606a62c4976841d
[I 2023-12-11 18:46:23.816 ServerApp] http://127.0.0.1:8888/tree?token=35c72a2ce9e92eb04fdac8cf5168c49c5606a62c4976841d
[I 2023-12-11 18:46:23.816 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2023-12-11 18:46:23.830 ServerApp]
To access the server, open this file in a browser:
file:///panfs/pfs.local/home/r557e636/.local/share/jupyter/runtime/jpserver-3575-open.html
Or copy and paste one of these URLs:
http://r16r27n01:8888/tree?token=35c72a2ce9e92eb04fdac8cf5168c49c5606a62c4976841d
http://127.0.0.1:8888/tree?token=35c72a2ce9e92eb04fdac8cf5168c49c5606a62c4976841d
Using MobaXterm, macOS, or Linux use the following command in a new terminal:
ssh -N -L <port>:<hostname>:<port> <KU Online ID>@hpc.crc.ku.edu
For the jupyter.log file above, that command will be:
$ ssh -N -L 8888:r16r27n01:8888 r557e636@hpc.crc.ku.edu
Once signed in no text will appear on the terminal window. Do not exit this window or your SSH tunnel will exit.
Copy and paste the URL containing 127.0.0.1 into your desktop browser
http://127.0.0.1:8888/tree?token=35c72a2ce9e92eb04fdac8cf5168c49c5606a62c4976841d
Cleaning Up
First, quit your Jupyter instance and close the browser tab.
On the SSH terminal window doing the SSH tunnel, use the keys ctrl+c to exit the SSH session.
To cancel the Jupyter instance running on the compute node, use the command scancel <jobid> cancel the job. Use the command squeue -u $USER to get the Job IDs of the jobs you are running.