HPC

Difference between revisions of "Quickstart/Virtual Environments"

From HPC
Jump to: navigation , search
(created a new ve page)
 
m (→‎What is a Virtual Environment?)
 
(47 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
==What is a Virtual Environment?==
 
==What is a Virtual Environment?==
 +
 +
A virtual environment is a named, isolated, working copy of Python that maintains its own files, directories, and paths so that you can work with specific versions of '''libraries''' or '''Python''' itself without affecting other Python projects. These can be created in standard Python and also Conda Python.
  
 
==Why should you use a Virtual Environment?==
 
==Why should you use a Virtual Environment?==
 +
 +
Python has various modules and packages for different applications. During a research project, we may require a third-party library or even a particular version, which we install. Another project also uses the same directory for retrieval and storage but doesn't require any other third-party packages. We can see we now have a package issue which will create problems very quickly.
 +
 +
So, the virtual environment can come into play and make a separate isolated environment for both projects, and each project can store and retrieve packages from its specific environment.
 +
 +
Also, let us consider another case where we are creating a Deep learning project using Tensorflow. Suppose you are working on two research projects project-01 and project-02.
 +
 +
If project-01 uses Tensorflow-2.0 and project2 uses Tensorflow-2.6, they would be stored in the same directory with the same name, and the error may occur. Then, in such cases, virtual environments can be really helpful for you to maintain the dependencies of both projects.
  
 
==How to use a Virtual Environment==
 
==How to use a Virtual Environment==
  
 +
Creating a virtual environment is simple and will save you a considerable amount of time and remove a lot of pitfalls as you use Python libraries.
 +
 +
We will use the following steps
 +
 +
# [[#Environment|Start the python module]]
 +
# [[#Creation of Virtual Environment Using Anaconda on Viper|Create the environment]]
 +
# [[#Adding packages|Adding python packages]]
 +
# Using the environment in an [[#Using your environment in an interactive session|interactive session]] or as a [[#Using a BATCH script with a virtual environment|submitted batch]] job.
 +
 +
 +
===Environment===
 +
 +
Let's load up a python module and build our Virtual Environment (VE) from there.
 +
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
[user@login01 ~]$ module load python/anaconda/202111/3.9
 +
[user@login01 ~]$ conda list
 +
# packages in the environment
 +
#
 +
# Name                    Version                  Build  Channel
 +
asn1crypto                0.24.0                  py37_0
 +
ca-certificates          2019.1.23                    0
 +
certifi                  2019.3.9                py37_0
 +
cffi                      1.12.2          py37h2e261b9_1
 +
chardet                  3.0.4                    py37_1
 +
conda                    4.6.14                  py37_0
 +
cryptography              2.6.1            py37h1ba5d50_0
 +
idna                      2.8                      py37_0
 +
libedit                  3.1.20181209        hc058e9b_0
 +
libffi                    3.2.1                hd88cf55_4
 +
libgcc-ng                8.2.0                hdf63c60_1
 +
libstdcxx-ng              8.2.0                hdf63c60_1
 +
ncurses                  6.1                  he6710b0_1
 +
openssl                  1.1.1b              h7b6447c_1
 +
pip                      19.0.3                  py37_0
 +
pycosat                  0.6.3            py37h14c3975_0
 +
pycparser                2.19                    py37_0
 +
pyopenssl                19.0.0                  py37_0
 +
pysocks                  1.6.8                    py37_0
 +
python                    3.7.3                h0371630_0
 +
readline                  7.0                  h7b6447c_5
 +
requests                  2.21.0                  py37_0
 +
ruamel_yaml              0.15.46          py37h14c3975_0
 +
setuptools                41.0.0                  py37_0
 +
six                      1.12.0                  py37_0
 +
sqlite                    3.27.2              h7b6447c_0
 +
tk                        8.6.8                hbc83047_0
 +
urllib3                  1.24.1                  py37_0
 +
wheel                    0.33.1                  py37_0
 +
xz                        5.2.4                h14c3975_4
 +
yaml                      0.1.7                had09818_2
 +
zlib                      1.2.11              h7b6447c_3
 +
 +
</pre>
 +
 +
 +
 +
===Creation of Virtual Environment Using Anaconda on Viper===
 +
 +
 +
To create a virtual environment using anaconda 4.6 with python version 3.7 on Viper,  you would use the conda create command as follows:
 +
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
[user@login01 ~]$ module load python/anaconda/202111/3.9
 +
[user@login01 ~]$ conda create –n tensorflow1
 +
</pre>
 +
 +
* This will solve and create the virtual environment and will then ask you '''Proceed ([y]/n)?''' - press '''y''' to proceed.
 +
 +
 +
{| class="mw-collapsible mw-collapsed wikitable"
 +
|+ Seeing CommandNotFoundError: Your shell has not been properly....?
 +
|-
 +
! scope="col" | Solution
 +
|-
 +
|  If you see an error like this then you need to type '''conda init bash''' at the terminal command line.  Then exit the terminal you're in and restart the session to read the new conda settings.
 +
|}
 +
 +
The above command creates a new virtual environment called tensorflow1.
 +
 +
To activate this virtual environment you would issue the following command:
 +
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
[user@login01 ~]$ conda activate tensorflow1
 +
</pre>
 +
On successful activation of this virtual environment you should the name of your environment in front of your login prompt like so:
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
(tensorflow1)  [user@login01 ~]$
 +
</pre>
 +
 +
* To exit the virtual environment use the key combination '''conda deactivate'''.
 +
 +
===Adding packages===
 +
 +
Once you have installed Miniconda and set up your environment to access it, you can then add whatever packages you wish to the installation using the conda install ... command. For example:
 +
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
(tensorflow1) [user@login01 ~]$ conda install numpy
 +
Fetching package metadata ...............
 +
Solving package specifications:
 +
 +
Package plan for installation in environment /home/t01/t01/user/miniconda3:
 +
 +
The following NEW packages will be INSTALLED:
 +
 +
    blas:        1.1-openblas                  conda-forge
 +
    libgfortran: 3.0.0-1                                 
 +
    numpy:      1.14.0-py36_blas_openblas_200 conda-forge [blas_openblas]
 +
    openblas:    0.2.20-7                      conda-forge
 +
 +
The following packages will be UPDATED:
 +
 +
    conda:      4.3.31-py36_0                            --> 4.3.33-py36_0 conda-forge
 +
 +
The following packages will be SUPERSEDED by a higher-priority channel:
 +
 +
    conda-env:  2.6.0-h36134e3_1                          --> 2.6.0-0      conda-forge
 +
 +
Proceed ([y]/n)? y
 +
</pre>
 +
 +
* Please note, for some package installations it may also be necessary to specify a channel such as conda-forge. For example, the following command installs the pygobject module.
 +
 +
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
(tensorflow1) [user@login01 ~]$ conda install -c conda-forge pygobject
 +
</pre>
 +
 +
* To create an environment with a specific version of a package:
 +
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
[user@login01 ~]$ conda create -n myenv scipy=0.15.0
 +
</pre>
 +
 +
* or even defining the python version at 3.4
 +
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
[user@login01 ~]$ conda create -n myenv python=3.4 scipy=0.15.0 astroid babel
 +
</pre>
 +
 +
===Clone an environment===
 +
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
[user@login01 ~]$ conda create -n OriginalENV --clone NewENV
 +
</pre>
 +
 +
===Removing an environment===
 +
 +
To delete a conda environment, enter the following, where yourenvname is the name of the environment you wish to delete.
 +
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
[user@login01 ~]$ conda remove --name EnvironmentNAME --all
 +
</pre>
 +
 +
 +
 +
===Using your environment in an interactive session===
 +
 +
To test and debug your program it is recommended to use an interactive session, these are also useful for programs that have short runtimes too.
 +
 +
* If you want a CPU-based node to use the commands:
 +
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
[user@login01 ~]$ interactive
 +
[user@c001 ~]$ conda activate pytorch01
 +
{pytorch01} [user@c001 ~]$ python mypytorchprogram.py
 +
</pre>
 +
 +
* If you want a GPU-based node instead use the commands:
 +
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
[user@login01 ~]$ interactive -pgpu
 +
[user@gpu02 ~]$ conda activate pytorch01
 +
{pytorch01} [user@gpu02 ~]$ python mypytorchprogram.py
 +
</pre>
 +
 +
{| class="mw-collapsible mw-collapsed wikitable"
 +
|+ Seeing CommandNotFoundError: Your shell has not been properly....?
 +
|-
 +
! scope="col" | Solution
 +
|-
 +
|  If you see an error like this then you need to type '''conda init bash''' at the terminal command line. Then exit the terminal you're in and restart the session to read the new conda settings.
 +
|}
 +
 +
===Using a BATCH script with a virtual environment===
 +
 +
Here are two examples of BATCH scripts which are using a Python virtual environment
 +
 +
* '''substitute''' ''/home/<user>'' for your own path
 +
 +
====Compute Node Example====
 +
 +
<pre style="background-color: #C8C8C8; color: black; font-family: monospace, sans-serif;">
 +
#!/bin/bash
 +
#SBATCH -J BUILDCPU
 +
#SBATCH -N 1
 +
#SBATCH --ntasks-per-node 1
 +
#SBATCH -D /home/<user>/
 +
#SBATCH -o debug.out
 +
#SBATCH -e debug.err
 +
#SBATCH -p compute
 +
#SBATCH -t 00:10:00
 +
#SBATCH --mail-user= your email address here
 +
 +
echo $SLURM_JOB_NODELIST
 +
 +
module purge
 +
module load python/anaconda/20220712/3.9
 +
 +
source activate /home/<user>/.conda/envs/bioinformatics1
 +
export PATH=/home/<user>/.conda/envs/bioinformatics/bin:${PATH}
 +
 +
python /home/user/TATT-CPU.py
 +
</pre>
 +
 +
 +
====GPU Node Example====
 +
 +
 +
<pre style="background-color: #C8C8C8; color: black; font-family: monospace, sans-serif;">
 +
#!/bin/bash
 +
#SBATCH -J BIDGPU
 +
#SBATCH -N 1
 +
#SBATCH --ntasks-per-node 1
 +
#SBATCH -D /home/<user>/
 +
#SBATCH -o debug.out
 +
#SBATCH -e debug.err
 +
#SBATCH --gres=gpu
 +
#SBATCH -p gpu
 +
#SBATCH -t 00:10:00
 +
#SBATCH --mail-user= your email address here
 +
 +
echo $SLURM_JOB_NODELIST
 +
 +
module purge
 +
module load gcc/5.2.0
 +
module load  python/anaconda/20220712/3.9
 +
module load cuda/11.5.0
 +
 +
source activate /home/<user>/.conda/envs/bioinformatics1
 +
export PATH=/home/<user>/.conda/envs/bioinformatics/bin:${PATH}
 +
 +
python /home/user/TATT-GPU.py
 +
</pre>
 +
 +
===Issues with environments===
 +
 +
{| class="mw-collapsible mw-collapsed wikitable"
 +
|+ Virtual Environment Tips
 +
|-
 +
! scope="col" | Worth noting
 +
|-
 +
| * If you see an error like this then you need to type '''conda init bash''' at the terminal command line. Then exit the terminal you're in and restart the session to read the new conda settings.
 +
|-
 +
| * I recommend using a separate virtual environment for each project.
 +
|-
 +
| * Avoid using pip by itself. Using '''python -m pip''' will always guarantee you are using the pip associated with that specific python being called, instead of potentially calling a pip associated with a different python.
 +
|-
 +
| * You should never copy or move around virtual environments. Always create new ones, or use YAML exports or clone them.
 +
|}
 +
 +
=== Importing and Exporting Virtual Environments===
 +
 +
[[Quickstart/Exporting Virtual Environments|Click here for the advanced importing and exporting of a virtual environment.]]
 +
 +
== Next Steps ==
 +
 +
* [[Programming/Python|Python programming]]
 +
* [[Applications/Anaconda|Anaconda python]]
  
  
−
[[Newmainpage #Quickstart| Back]]
+
[[Main Page #Quickstart| Back]]    /  [[Quickstart/Tutorial Videos| Next (Tutorial Videos)]]

Latest revision as of 10:44, 17 April 2023

What is a Virtual Environment?

A virtual environment is a named, isolated, working copy of Python that maintains its own files, directories, and paths so that you can work with specific versions of libraries or Python itself without affecting other Python projects. These can be created in standard Python and also Conda Python.

Why should you use a Virtual Environment?

Python has various modules and packages for different applications. During a research project, we may require a third-party library or even a particular version, which we install. Another project also uses the same directory for retrieval and storage but doesn't require any other third-party packages. We can see we now have a package issue which will create problems very quickly.

So, the virtual environment can come into play and make a separate isolated environment for both projects, and each project can store and retrieve packages from its specific environment.

Also, let us consider another case where we are creating a Deep learning project using Tensorflow. Suppose you are working on two research projects project-01 and project-02.

If project-01 uses Tensorflow-2.0 and project2 uses Tensorflow-2.6, they would be stored in the same directory with the same name, and the error may occur. Then, in such cases, virtual environments can be really helpful for you to maintain the dependencies of both projects.

How to use a Virtual Environment

Creating a virtual environment is simple and will save you a considerable amount of time and remove a lot of pitfalls as you use Python libraries.

We will use the following steps

  1. Start the python module
  2. Create the environment
  3. Adding python packages
  4. Using the environment in an interactive session or as a submitted batch job.


Environment

Let's load up a python module and build our Virtual Environment (VE) from there. 

[user@login01 ~]$ module load python/anaconda/202111/3.9
[user@login01 ~]$ conda list
# packages in the environment 
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py37_0
ca-certificates           2019.1.23                     0
certifi                   2019.3.9                 py37_0
cffi                      1.12.2           py37h2e261b9_1
chardet                   3.0.4                    py37_1
conda                     4.6.14                   py37_0
cryptography              2.6.1            py37h1ba5d50_0
idna                      2.8                      py37_0
libedit                   3.1.20181209         hc058e9b_0
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 8.2.0                hdf63c60_1
libstdcxx-ng              8.2.0                hdf63c60_1
ncurses                   6.1                  he6710b0_1
openssl                   1.1.1b               h7b6447c_1
pip                       19.0.3                   py37_0
pycosat                   0.6.3            py37h14c3975_0
pycparser                 2.19                     py37_0
pyopenssl                 19.0.0                   py37_0
pysocks                   1.6.8                    py37_0
python                    3.7.3                h0371630_0
readline                  7.0                  h7b6447c_5
requests                  2.21.0                   py37_0
ruamel_yaml               0.15.46          py37h14c3975_0
setuptools                41.0.0                   py37_0
six                       1.12.0                   py37_0
sqlite                    3.27.2               h7b6447c_0
tk                        8.6.8                hbc83047_0
urllib3                   1.24.1                   py37_0
wheel                     0.33.1                   py37_0
xz                        5.2.4                h14c3975_4
yaml                      0.1.7                had09818_2
zlib                      1.2.11               h7b6447c_3


Creation of Virtual Environment Using Anaconda on Viper

To create a virtual environment using anaconda 4.6 with python version 3.7 on Viper, you would use the conda create command as follows: 

[user@login01 ~]$ module load python/anaconda/202111/3.9
[user@login01 ~]$ conda create –n tensorflow1
  • This will solve and create the virtual environment and will then ask you Proceed ([y]/n)? - press y to proceed.


Seeing CommandNotFoundError: Your shell has not been properly....?
Solution
If you see an error like this then you need to type conda init bash at the terminal command line. Then exit the terminal you're in and restart the session to read the new conda settings.

The above command creates a new virtual environment called tensorflow1.

To activate this virtual environment you would issue the following command: 

[user@login01 ~]$ conda activate tensorflow1

On successful activation of this virtual environment you should the name of your environment in front of your login prompt like so: 

(tensorflow1)  [user@login01 ~]$
  • To exit the virtual environment use the key combination conda deactivate.

Adding packages

Once you have installed Miniconda and set up your environment to access it, you can then add whatever packages you wish to the installation using the conda install ... command. For example: 

(tensorflow1) [user@login01 ~]$ conda install numpy
Fetching package metadata ...............
Solving package specifications:

Package plan for installation in environment /home/t01/t01/user/miniconda3:

The following NEW packages will be INSTALLED:

    blas:        1.1-openblas                  conda-forge
    libgfortran: 3.0.0-1                                  
    numpy:       1.14.0-py36_blas_openblas_200 conda-forge [blas_openblas]
    openblas:    0.2.20-7                      conda-forge

The following packages will be UPDATED:

    conda:       4.3.31-py36_0                             --> 4.3.33-py36_0 conda-forge

The following packages will be SUPERSEDED by a higher-priority channel:

    conda-env:   2.6.0-h36134e3_1                          --> 2.6.0-0       conda-forge

Proceed ([y]/n)? y
  • Please note, for some package installations it may also be necessary to specify a channel such as conda-forge. For example, the following command installs the pygobject module.


(tensorflow1) [user@login01 ~]$ conda install -c conda-forge pygobject
  • To create an environment with a specific version of a package:
[user@login01 ~]$ conda create -n myenv scipy=0.15.0
  • or even defining the python version at 3.4
[user@login01 ~]$ conda create -n myenv python=3.4 scipy=0.15.0 astroid babel

Clone an environment

[user@login01 ~]$ conda create -n OriginalENV --clone NewENV

Removing an environment

To delete a conda environment, enter the following, where yourenvname is the name of the environment you wish to delete. 

[user@login01 ~]$ conda remove --name EnvironmentNAME --all


Using your environment in an interactive session

To test and debug your program it is recommended to use an interactive session, these are also useful for programs that have short runtimes too.

  • If you want a CPU-based node to use the commands:
[user@login01 ~]$ interactive
[user@c001 ~]$ conda activate pytorch01
{pytorch01} [user@c001 ~]$ python mypytorchprogram.py
  • If you want a GPU-based node instead use the commands:
[user@login01 ~]$ interactive -pgpu
[user@gpu02 ~]$ conda activate pytorch01
{pytorch01} [user@gpu02 ~]$ python mypytorchprogram.py
Seeing CommandNotFoundError: Your shell has not been properly....?
Solution
If you see an error like this then you need to type conda init bash at the terminal command line. Then exit the terminal you're in and restart the session to read the new conda settings.

Using a BATCH script with a virtual environment

Here are two examples of BATCH scripts which are using a Python virtual environment

  • substitute /home/<user> for your own path

Compute Node Example

#!/bin/bash
#SBATCH -J BUILDCPU
#SBATCH -N 1
#SBATCH --ntasks-per-node 1
#SBATCH -D /home/<user>/
#SBATCH -o debug.out
#SBATCH -e debug.err
#SBATCH -p compute
#SBATCH -t 00:10:00
#SBATCH --mail-user= your email address here

echo $SLURM_JOB_NODELIST

module purge
module load python/anaconda/20220712/3.9

source activate /home/<user>/.conda/envs/bioinformatics1
export PATH=/home/<user>/.conda/envs/bioinformatics/bin:${PATH}

python /home/user/TATT-CPU.py


GPU Node Example

#!/bin/bash
#SBATCH -J BIDGPU
#SBATCH -N 1
#SBATCH --ntasks-per-node 1
#SBATCH -D /home/<user>/
#SBATCH -o debug.out
#SBATCH -e debug.err
#SBATCH --gres=gpu
#SBATCH -p gpu
#SBATCH -t 00:10:00
#SBATCH --mail-user= your email address here

echo $SLURM_JOB_NODELIST

module purge
module load gcc/5.2.0
module load  python/anaconda/20220712/3.9
module load cuda/11.5.0

source activate /home/<user>/.conda/envs/bioinformatics1
export PATH=/home/<user>/.conda/envs/bioinformatics/bin:${PATH}

python /home/user/TATT-GPU.py

Issues with environments

Virtual Environment Tips
Worth noting
* If you see an error like this then you need to type conda init bash at the terminal command line. Then exit the terminal you're in and restart the session to read the new conda settings.
* I recommend using a separate virtual environment for each project.
* Avoid using pip by itself. Using python -m pip will always guarantee you are using the pip associated with that specific python being called, instead of potentially calling a pip associated with a different python.
* You should never copy or move around virtual environments. Always create new ones, or use YAML exports or clone them.

Importing and Exporting Virtual Environments

Click here for the advanced importing and exporting of a virtual environment.

Next Steps


Back / Next (Tutorial Videos)

Retrieved from ‘http://hpc.mediawiki.hull.ac.uk/index.php?title=Quickstart/Virtual_Environments&oldid=41480’