Managing Virtual Environments Using `pip` and `venv` on ARC

This page describes how to build, organize, and use Python virtual environments (VEs) with pip and venv on ARC systems. This method is lightweight and does not rely on Conda or Miniforge. ARC suggests the use of Miniforge as the preferred way to build conda virtual environments (CVEs). We recommend not to install andaconda on your home as suggested on traditional tutorials. ARC’s provided Miniforge and Miniconda will work faster than user-installed anaconda in $HOME.

1. When to Use pip/venv

Use pip and venv if:

You only need a few Python packages.
You prefer the built-in Python tools.
You do not need Conda’s dependency management or GPU toolkit integrations.

This method creates a fully isolated Python environment using only standard library tools.

2. Steps to Create a pip/venv Virtual Environment

Step 1 — Start an Interactive Session

Always create environments on the same node type where they’ll be used.

interact --account=<account> --partition=<partition_name>

Step 2 — Load a Python Module

Identify available Python versions, then load one.

module spider Python/3
module load Python/3.12.3-GCCcore-13.3.0

Step 3 — Create the Virtual Environment

python -m venv ~/venvs/<env_name>

Step 4 — Activate the Environment

source ~/venvs/<env_name>/bin/activate

Step 5 — Install Packages

Install or upgrade packages using pip.

pip install numpy pandas
pip install --upgrade pip

Step 6 — Check Installed Packages

pip list
python --version

Step 7 — Deactivate the Environment

deactivate

3. Organizing and Managing Virtual Environments

Path	Description
`~/venvs/<name>`	Default per-user environment directory
`/projects/<group>/envs/<name>`	Shared environment directory for group use

Example structure:

~/venvs/
 ├── tc_normal-ml/
 ├── tc_a100-nlp/
 └── bioinfo-tools/

List environments:

ls ~/venvs

4. Using pip/venv Environments in Jupyter OnDemand

You can use a pip/venv environment as a Jupyter kernel to run notebooks on Open OnDemand.

Step 1 — Install ipykernel

Activate your environment, then install the ipykernel package.

source ~/venvs/<env_name>/bin/activate
pip install ipykernel

Step 2 — Register the Kernel

python -m ipykernel install --user --name <env_name> --display-name "Python (<env_name>)"

Step 3 — Select the Kernel in Jupyter

Launch Jupyter via Open OnDemand.
From the top menu, select Kernel → Change Kernel → Python (<env_name>).

5. Troubleshooting

Issue	Solution
Module not found	Verify the package is installed in the active environment.
Kernel not showing in Jupyter	Re-run the `ipykernel install` command.
Permission denied	Avoid using shared directories unless permissions are set.
Python version mismatch	Load the same Python module that was used to create the environment.

7. Summary

Use pip and venv for simple, lightweight Python environments.
Always create environments on the correct node type.
Register environments using ipykernel for Jupyter OnDemand.
For complex dependency chains or GPU use, prefer Miniforge/Miniconda instead.

Managing Virtual Environments Using pip and venv on ARC