PrimeHub Extension is an extension of Jupyter Notebook developed for PrimeHub. We plans to roll out more features to enhance user's ML workflow and experience from Notebook.
The latest images provided by InfuseAI are built with PrimeHub Extension and the corresponding environment to execute extension features smoothly. To Use these features, please make sure the running image is the latest one or is built on top of the latest one from InfuseAI. See the list.
How to build own images including PrimeHub Extensions? See the repo.
After opening a Notebook file (*.ipynb), there is PrimeHub extension on the menu bar of the Notebook. This document explains these extension features briefly.
PrimeHub Extension needs an API Token to submit a Job, Please generate a API Token and setup it.
You could open the PrimeHub Extension dropdown list to find the
API Token setting:
Submit Notebook as a Job
The working group's Group Volume is required.
Users could submit their Notebook to the PrimeHub Jobs. See Simple UseCase.
However, there are differences between jobs from Notebook and jobs from Job Submission directly:
- A job submitted by PrimeHub Extension (a.k.a Notebook Job) always uses the directory where the notebook locates as its working directory, whereas a job submitted by Job Submission uses
- A notebook has to locate inside a group volume.
Path, for example:
There is a group volume
phusers and a notebook,
my-notebook.ipynb, in the sub path
When the notebook
my-notebook.ipynb is submitted into a job, the job takes
/home/jovyan/phusers/experiment-1/ as the working directory and run all of cells of the notebook.
Job Artifacts with a Notebook Job
In terms of a Notebook Job, users could store outputs either in the group volume or in the
Job Artifacts but different working directories as we mention above have to be taken into consideration. Since working directories are different,
relative path in code may cause problems.
The code snippet works in a Job but not in a Notebook Job, because the working directory is at
/home/jovyan and it put data relatively into
/home/jovyan/artifacts which is a correct path.
mkdir -p artifacts/ cp my_model.h5 artifacts/ cp -r logs artifacts/
However, a Notebook Job starts at
/home/jovyan/<group_volume>/path/to/ where the notebook locates, the relative path of same codes above becomes
Hence, we encourage using absolute path in both of cases instead to avoid the mistake and the confusion.
mkdir -p /home/jovyan/artifacts/ cp my_model.h5 /home/jovyan/artifacts/ cp -r logs /home/jovyan/artifacts/
Job Artifacts is optional
We also encourage using group volume to store persistent data.
Job Artifacts is optional when we expect
- Download artifacts from PrimeHub Job without opening a Notebook
- Artifacts will be cleaned automatically