nc_tensorboard

This Tensorboard OnDemand app runs through an authenticating reverse proxy.

Tensorboard doesn't provide any kind of authentication for its web interface, so on a shared environment, anybody knowing the hostname and port number of a running Tensorboard instance can connect to it.

To mitigate this, Stanford Sherlock implemented an authentication mechanism in this app that basically sets a browser cookie in the OnDemand interactive app page (through the "Connect to Tensorboard" button), which is then checked by the authenticating reverse proxy to control access to the Tensorboard web interface.

Without that cookie, access to the Tensorboard web interface is refused. And if the cookie is ever lost, users can re-create it by visiting the "My Interactive Sessions" page and clicking the "Connect" button again.

Setup

To use this OOD app, you need to install the following on every node, e.g. to a shared directory such as NFS or Lustre, or if you don't have either just rsync the generated directories consistently between all nodes.

• Tensorboard Installation - Choose one of the following. Using an environment allows dependencies to avoid getting modified by the system, obtain older or newer packages than the system has, and allow multiple versions of tensorboard to be maintained.
  • Python virtualenv
  • Anaconda Environment tensorboard
• Lmod module tensorboard - This lmod module will then be used to set the command tensorboard in the PATH. You can create different .lua files can be labeled with different Python + tensorboard versions so some users can downgrade as needed.

Install twisted to systemwide python

twisted is used to run an initial proxying webserver.

On Fedora/RHEL/Rocky Linux 9:

sudo dnf install python3-twisted

Create conda environment for tensorboard

As the owner of the lmod module directory (such as hpc_test), run the following commands:

$ module load conda
$ conda create -p /data/apps/miniforge/25.11.0/envs/tensorboard

https://docs.pytorch.org/tutorials/beginner/introyt/tensorboardyt_tutorial.html

Now create an an Lmod module for this environment at /etc/modulefiles/tensorboard/py312-25.11.0.lua, named after the python version used.

local modroot = "/data/apps/miniforge/25.11.0/envs/tensorboard"
local pyver   = "3.12.12"

pushenv("CONDA_PREFIX",modroot)
pushenv("CONDA_DEFAULT_ENV",modroot)
prepend_path("LD_LIBRARY_PATH",modroot.."/lib")
prepend_path("PYTHONPATH",modroot.."/lib/python3.12/site-packages")
pushenv("PYTHONNOUSERSITE","1")

-- Using Python and tools from the environment
prepend_path("PATH",modroot.."/bin")

Now install pytorch and torchvision:

module load tensorboard
conda install pytorch torchvision torchaudio pytorch -c pytorch

Or for nvidia CUDA support:

module load tensorboard
conda install pytorch torchvision torchaudio pytorch-cuda -c pytorch -c nvidia

Deployment

Place this git repo into /var/www/ood/apps/usr . Make sure permissions on it are set to chmod 755, or it will not appear.

cd /var/www/ood/apps/usr/
sudo git clone https://github.com/PATH/nc_common.git
sudo git clone https://github.com/PATH/TO-YOUR-REPO.git
sudo chown root:root -R /var/www/ood/apps/usr/

Ensure that nc_common also exists in the same directory, since it imports form and attributes from it.

Make sure that chmod +x /var/www/ood/apps/usr/nc_tensorboard/templates/* is set. It would already be set if you git cloned, but there might be issues.

You can set chmod 700 on a directory to hide the app from OOD.