On This Page

Adding Custom Packages To cbcore Container

Adding Custom Packages To cbcore Container

You can add custom packages to the Cerebras cbcore container. This section describes how to add two specific packages: the spektral and networkx, to your Cerebras cbcore container.

Adding spektral and networkx packages

Requirements

Read this before you proceed further

  • Make sure that your system administrator has enabled the user control of binds for Singularity. See here for more: Bind Paths and Mounts — Singularity container 3.0 documentation .

  • Know that you are responsible to maintain the directory of packages you are installing in the Cerebras container (the ~/Cerebras-0.8.0-extra-python-packages directory in the example installation instructions below).

  • Ensure that you map the directory containing these packages to all the Slurm nodes in your cluster, for example via NFS.

Installation

  1. Create a directory shared between and accessible by all the hosts. For example, a directory with the name: ~/Cerebras-0.8.0-extra-python-packages.

  2. Run the following singularity exec command with -B option to perform the following two tasks:

    • Access the ~/Cerebras-0.8.0-extra-python-packages directory as $HOME/.local directory from within the Cerebras container, and

    • Install the spektral and networkx packages in $HOME/.local. In reality these packages are installed in ~/Cerebras-0.8.0-extra-python-packages directory but they are accessible within the Cerebras container as $HOME/.local directory.

      • TensorFlow

      singularity exec -B ~/Cerebras-0.8.0-extra-python-packages:$HOME/.local /<path>/cbcore-0.8.0-<identifier-string>.sif pip install --user spektral==1.0.7 networkx==2.6.2

      • PyTorch

      singularity exec -B ~/Cerebras-0.8.0-extra-python-packages:$HOME/.local /<path>/cbcore-0.8.0-<identifier-string>.sif python-pt -m pip install --user spektral==1.0.7 networkx==2.6.2

Accessing the packages

Run the following command to test that the modules in the installed packages are accessible from within the Cerebras container:

  • TensorFlow

    singularity exec -B ~/Cerebras-0.8.0-extra-python-packages:$HOME/.local /<path>/cbcore-0.8.0-<identifier-string>.sif python -c 'import spektral; print(spektral.__file__)'

  • PyTorch

    singularity exec -B ~/Cerebras-0.8.0-extra-python-packages:$HOME/.local /<path>/cbcore-0.8.0-<identifier-string>.sif python-pt -c 'import spektral; print(spektral.__file__)'

The output should resemble the following, if the $HOME value is /my/home/username:

/my/home/username/.local/lib/python3.7/site-packages/spektral/__init__.py.

previous

Incremental Compile