Virtual Environments: How to Build Using pip and venv

This information is part of a collection of information on constructing and using virtual environments.

ARC suggests the use of MiniForge as one way to construct conda virtual environments (CVEs). (As noted elsewhere, Ananconda can no longer be used because of changes in Anaconda’s terms of use that are affecting universities nation-wide and beyond.)

However, there may be times when you want to go with a different approach to create a virtual environment (VE). One approach is using pip and venv. The approach is provided here. Note that the resulting VE is not a conda VE.

Steps for Building a Virtual Environment (VE)

The steps are given concisely here. Sections below, named as the bullets here, provide additional detail if needed.

Steps for Build

  1. Log onto the machine on which you wish to run code.

  2. Identify the partition (i.e., queue) to which you will submit your job.

  3. Request resources on a compute node of that partition. These resources will be used to build the VE. A sample form of a request, for TC or Owl:

salloc --account=<account>  --partition=<partition_name>    --nodes=<number of nodes>  --ntasks-per-node=<number of tasks per node> --cpus-per-task=<number of cores per task> --time=<duration of resources>`

An example might be salloc --account=personal  --partition=normal_q    --nodes=1 --ntasks-per-node=1 --cpus-per-task=2 --time=2:00:00, where one is requesting two cores for two hours on the normal_q using their personal account. This will return several pieces of information when slurm provides the resources that you requested to you. Two of the most important are the slurm JOB ID corresponding to this request and the compute node whose resources you will use.

  1. ssh to the compute node returned from the salloc resource request. Enter:

ssh XXX

where XXX is the compute node name that is returned from the salloc command and consists of the cluster name (e.g., owl for the Owl cluster) or an abbreviation of the cluster name (e.g., tc for Tinkercliffs cluster) and a three-digit number. Examples for XXX are tc032 and owl007. So the command whoudl be ssh tc032 or similar.

  1. The version of python that you need. A key aspect of this approach is that one needs a module that resides on the particular cluster for the particular version of python that you need in your VE. This is generally not too difficult of a problem. For example, on Tinkercliffs, the modules that contain various versions of python from 3.9 through 3.12 are:

   Python/3.9.5-GCCcore-10.3.0-bare
   Python/3.9.5-GCCcore-10.3.0
   Python/3.10.4-GCCcore-11.3.0-bare
   Python/3.10.4-GCCcore-11.3.0
   Python/3.10.8-GCCcore-12.2.0-bare
   Python/3.10.8-GCCcore-12.2.0
   Python/3.11.3-GCCcore-12.3.0
   Python/3.11.5-GCCcore-13.2.0
   Python/3.12.3-GCCcore-13.3.0                            (D)

And on Owl, the Python modules run from 3.8 through 3.12, like so:

   Python/3.8.20
   Python/3.9.6-GCCcore-11.2.0-bare
   Python/3.9.20
   Python/3.10.4-GCCcore-11.3.0-bare
   Python/3.10.8-GCCcore-12.2.0-bare
   Python/3.10.8-GCCcore-12.2.0
   Python/3.10.15
   Python/3.11.3-GCCcore-12.3.0
   Python/3.11.5-GCCcore-13.2.0
   Python/3.11.10
   Python/3.12.3-GCCcore-13.3.0
   Python/3.12.7                                           (D)

Note that these listings (options) will change over time.

  1. Reset modules. Enter

module reset

  1. Load a module for the version of python that you need.

The form of the command is:

module load <module-name>

*** You must remember and record the module name that you use here.  This module and the VE are a _pair_: (module, VE) ***

If one needs python version 3.9 on owl, then one can load from the list above the following module by entering module load Python/3.9.20. If you want the default version of Python on a machine, just look for (D) beside a particular module; that is the default version. So, on Tinkercliffs, if you want the default module Python/3.12.3-GCCcore-13.3.0, then you can simply type module load Python (or you can use the full specification: module load Python/3.12.3-GCCcore-13.3.0).

  1. Create the virtual environment (VE). Enter

python -m venv  /path/to/virt-env/<VE name>

Now the VE exists, but you have to activate it to increase its capabilities.

  1. Activate the VE. You must specify the activate script within the bin directory of your VE.

Enter

source /path/to/virt-env/<VE name>/bin/activate

  1. Check the python version in the VE. Enter

python --version

and you should get the same version of python as is in the module.

  1. Add packages to your VE. Enter the following command as many times as you need, each time loading a package (<package_name>) that is not yet in the VE:

python -m pip install <package_name>

So if you want to load pandas, then enter python -m pip install pandas. You will have to press y at least once, maybe a couple of times for the install to complete.

  1. If the system prints a message to update pip, you can update it. Update pip by entering

python -m pip install --upgrade pip

But you do not have to update pip.

  1. List the packages in the VE. Enter

pip list

  1. Deactivate the VE. When you are done adding packages to the VE, enter

deactivate

This will deactivate the VE.

  1. Leave the compute node. After you are done building the CVE, exit off the compute node by typing

exit

  1. Relinquish resources. Enter:

scancel XXX

where XXX is the slurm JOB ID (i.e., an integer) corresponding to the resource request.

Note that if you find you want additional packages in your VE at this point, then you merely repeat the following steps:

  • 1 to 4 and 6 to log in and get onto the compute node.

  • 7 to load the module (remember, you have to record this).

  • 9 through 16 to activate the VE (you do not rebuild it; it exists), add modules to it, deactivate the VE, get off the compute node, and relinquish compute node resources.

Details

Remember to write down and save the Python module that you loaded

for the VE that you created.

There ae several options for a Python module to select, as step 7 showed. You must remember this module: the module and the VE are a pair, like so: (module, VE). One way to do this is to put a README file above the /path/to/virt-env and put in there the (module, VE) pairs.

Log onto the machine on which you wish to run code

From a terminal, type ssh <username>@<clustername>.arc.vt.edu where <username> is your user name and <clustername> is the name of the cluster you are trying to log into. Examples of the latter are tinkercliff2 and owl1.

Identify the partition (i.e., queue) to which you will submit your job

To list the partitions on a cluster, type:

sinfo

or

sinfo --long

or

sinfo |   awk -F " "  'NR > 1 { a[$1]++ } END { for (b in a) { print b } }'

Request resources on a compute node of that partition

To build a VE, it is most likely that you will only need one core of one compute node. For the sample form of resource request, for TC or Owl,

salloc --account=<account>  --partition=<partition_name>    --nodes=<number of nodes>  --ntasks-per-node=<number of tasks per node> --cpus-per-task=<number of cores per task> --time=<duration of resources>

one may take <number of nodes> as 1, <number of tasks per node as 1, and <number of cores per task> as 1. A duration <duration of resources> of two hours, i.e., 2:00:00 will usually suffice.

When slurm returns with your resources, note the names of the compute node(s) given to you and the slurm JOB ID. The names of compute nodes are used to determine which nodes to ssh into. The slurm JOB ID is used to relinquish resources when done with them, as a last step in this process.

How to determine an appropriate Python module

To find all occurrences of python (i.e., to find all python modules), first write to file all of the modules available on the cluster by typing:

module avail >& list.of.all.modules

Then open this file list.of.all.modules and search for Python (note the capital P) to find the versions of python. Use this full name in the module load command above.

Create a VE

There are many ways to create a virtual environment (VE), and if you use multiple ways to construct VEs, then you might want to consider putting -pv- (or similar) in the name of the VE to denote it was built using pip and venv. Different methods of generating modules result in different ways to activate them.

Use of VEs

You can only use a VE on the cluster and with the type of compute nodes that was used to build the VE.