Applications/Miniconda
Contents
- 1 Application Details
- 2 Purpose
- 3 Environment
- 3.1 Creation of Virtual Environment Using Anaconda on Viper
- 3.2 Adding packages
- 3.3 Clone an environment
- 3.4 Removing an environment
- 3.5 Using your environment in an interactive session
- 3.6 Using SLURM with a virtual environment
- 3.7 Creation of a Virtual Environment in Anaconda Using a YAML File
- 3.8 Exporting a Virtual Environment in Anaconda to a YAML File
- 4 Virtual Environment Tips
- 5 Next Steps
Application Details
- Description: Miniconda Python is a high-level interpreted programming language for general-purpose programming, supported by a large number of libraries for many tasks which the user can install for customizing their environment
- Versions: Miniconda (lite version of anaconda)
- Module name: python/anaconda/4.6/miniconda/3.7 and python/anaconda/202111/3.9 (used for virtual environments only)
- License: Free to use - Python Software Foundation License
Purpose
Important: The purpose of the miniconda installation is that it is a basic Anaconda python install with a minimal set of modules to allow users virtual environments. This allows for custom virtual environments which can be quite small.
Definition
A virtual environment is a named, isolated, working copy of Python that maintains its own files, directories, and paths so that you can work with specific versions of libraries or Python itself without affecting other Python projects. Virtual environments make it easy to cleanly separate different projects and avoid problems with different dependencies and version requirements across components.
The conda command is the preferred interface for managing installations and virtual environments with the Anaconda Python distribution.
Note: It is very likely within the next HPC, all Python environments will be all virtual environments.
Environment
This provides a minimal Python configuration, as shown below:
- Important : For building an environment for AI models on a GPU, otherwise continue below:
[user@login01 ~]$ module load python/anaconda/202111/3.9 [user@login01 ~]$ conda list # packages in the environment # # Name Version Build Channel asn1crypto 0.24.0 py37_0 ca-certificates 2019.1.23 0 certifi 2019.3.9 py37_0 cffi 1.12.2 py37h2e261b9_1 chardet 3.0.4 py37_1 conda 4.6.14 py37_0 cryptography 2.6.1 py37h1ba5d50_0 idna 2.8 py37_0 libedit 3.1.20181209 hc058e9b_0 libffi 3.2.1 hd88cf55_4 libgcc-ng 8.2.0 hdf63c60_1 libstdcxx-ng 8.2.0 hdf63c60_1 ncurses 6.1 he6710b0_1 openssl 1.1.1b h7b6447c_1 pip 19.0.3 py37_0 pycosat 0.6.3 py37h14c3975_0 pycparser 2.19 py37_0 pyopenssl 19.0.0 py37_0 pysocks 1.6.8 py37_0 python 3.7.3 h0371630_0 readline 7.0 h7b6447c_5 requests 2.21.0 py37_0 ruamel_yaml 0.15.46 py37h14c3975_0 setuptools 41.0.0 py37_0 six 1.12.0 py37_0 sqlite 3.27.2 h7b6447c_0 tk 8.6.8 hbc83047_0 urllib3 1.24.1 py37_0 wheel 0.33.1 py37_0 xz 5.2.4 h14c3975_4 yaml 0.1.7 had09818_2 zlib 1.2.11 h7b6447c_3
Creation of Virtual Environment Using Anaconda on Viper
- IMPORTANT: If you see an error like this:
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently, supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
You will need to type:
$ conda init bash
To create a virtual environment using Anaconda 4.6 with Python version 3.7 on Viper, you would use the conda create command as follows:
[user@c001 ~ ]$ module load python/anaconda/202111/3.9 [user@c001 ~ ]$ conda create –n myenv1
The above command creates a new virtual environment called myenv1.
To activate this virtual environment you would issue the following command:
[user@c001 ~ ]$ conda activate myenv1
On successful activation of this virtual environment, you should the name of your environment in front of your login prompt like so:
(myenv1) [user@c001 ~ ]$
To exit the virtual environment use the key combination ctrl + d or 'conda deactivate'.
Adding packages
Once you have installed Miniconda and set up your environment to access it, you can then add whatever packages you wish to the installation using the conda install ... command. For example:
(myenv1) user@c001:~> conda install numpy
Fetching package metadata ...............
Solving package specifications:
Package plan for installation in environment /home/t01/t01/user/miniconda3:
The following NEW packages will be INSTALLED:
blas: 1.1-openblas conda-forge
libgfortran: 3.0.0-1
numpy: 1.14.0-py36_blas_openblas_200 conda-forge [blas_openblas]
openblas: 0.2.20-7 conda-forge
The following packages will be UPDATED:
conda: 4.3.31-py36_0 --> 4.3.33-py36_0 conda-forge
The following packages will be SUPERSEDED by a higher-priority channel:
conda-env: 2.6.0-h36134e3_1 --> 2.6.0-0 conda-forge
Proceed ([y]/n)? y
- Please note, for some package installations it may also be necessary to specify a channel such as conda-forge. For example, the following command installs the pygobject module.
(myenv1) [user@c001]$ conda install -c conda-forge pygobject
- To create an environment with a specific version of a package:
[user@c001]$ conda create -n myenv2 scipy=0.15.0
- or even defining the Python version at 3.4
[user@c001]$ conda create -n myenv python=3.4 scipy=0.15.0 astroid babel
Clone an environment
[user@c001]$ conda create -n OriginalENV --clone NewENV
Removing an environment
To delete a conda environment, enter the following, where yourenvname is the name of the environment you wish to delete.
[user@c001]$ conda remove --name EnvironmentNAME --all
Using your environment in an interactive session
To test and debug your program it is recommended to use an interactive session, these are also useful for programs that have short runtimes too.
- If you want a CPU-based node to use the commands:
[user@login01]$ interactive
[user@c001]$ conda activate pytorch01
{pytorch01} [user@c001]$ python mypytorchprogram.py
- If you want a GPU-based node instead use the commands:
[user@login01]$ interactive -pgpu
[user@gpu02]$ conda activate pytorch01
{pytorch01} [user@gpu02]$ python mypytorchprogram.py
If you see an error like this:
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently, supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
You will need to type:
$ conda init bash
then exit the terminal you're in and restart the session to read the new conda settings.
Using SLURM with a virtual environment
Here are two examples of SLURM scripts which are using a Python virtual environment
- substitute /home/<user> for your own path
Compute Node Example
#!/bin/bash
#SBATCH -J BUILDCPU
#SBATCH -N 1
#SBATCH --ntasks-per-node 12
#SBATCH -D /home/<user>/
#SBATCH -o debug.out
#SBATCH -e debug.err
#SBATCH -p compute
#SBATCH -t 00:10:00
#SBATCH --mail-user= your email address here
echo $SLURM_JOB_NODELIST
module purge
module load python/anaconda/20220712/3.9
source activate /home/<user>/.conda/envs/bioinformatics1
export PATH=/home/<user>/.conda/envs/bioinformatics/bin:${PATH}
python /home/user/TATT-CPU.py
GPU Node Example
#!/bin/bash
#SBATCH -J BIDGPU
#SBATCH -N 1
#SBATCH --ntasks-per-node 1
#SBATCH -D /home/<user>/
#SBATCH -o debug.out
#SBATCH -e debug.err
#SBATCH --gres=gpu
#SBATCH -p gpu
#SBATCH -t 00:10:00
#SBATCH --mail-user= your email address here
echo $SLURM_JOB_NODELIST
module purge
module load gcc/5.2.0
module load python/anaconda/20220712/3.9
module load cuda/11.5.0
source activate /home/<user>/.conda/envs/bioinformatics1
export PATH=/home/<user>/.conda/envs/bioinformatics/bin:${PATH}
python /home/user/TATT-GPU.py
Creation of a Virtual Environment in Anaconda Using a YAML File
To create a virtual environment from a YAML file you would issue the following command:
[user@c001 ~ ]$ conda env create -f myenv.yml
The above command is creating a virtual environment from the YAML called myenv.yml. Below is a copy of the markup in the file called "myenv.yml".
name: ytenv channels: - defaults dependencies: - ca-certificates=2017.08.26=h1d4fec5_0 - certifi=2018.1.18=py27_0 - intel-openmp=2018.0.0=hc7b2577_8 - libedit=3.1=heed3624_0 - libffi=3.2.1=hd88cf55_4 - libgcc-ng=7.2.0=h7cc24e2_2 - libgfortran-ng=7.2.0=h9f7466a_2 - libstdcxx-ng=7.2.0=h7a57d05_2 - mkl=2018.0.1=h19d6760_4 - ncurses=6.0=h9df7e31_2 - numpy=1.14.0=py27h3dfced4_1 - openssl=1.0.2n=hb7f436b_0 - pip=9.0.1=py27ha730c48_4 - python=2.7.14=h1571d57_29 - readline=7.0=ha6073c6_4 - setuptools=38.4.0=py27_0 - sqlite=3.22.0=h1bed415_0 - tk=8.6.7=hc745277_3 - wheel=0.30.0=py27h2bc6bb2_1 - zlib=1.2.11=ha838bed_2 - pip: - backports.functools-lru-cache==1.5 - backports.shutil-get-terminal-size==1.0.0 - cycler==0.10.0 - decorator==4.2.1 - enum34==1.1.6 - h5py==2.7.1 - ipython==5.5.0 - ipython-genutils==0.2.0 - matplotlib==2.1.2 - mpmath==1.0.0 - pathlib2==2.3.0 - pexpect==4.4.0 - pickleshare==0.7.4 - prompt-toolkit==1.0.15 - ptyprocess==0.5.2 - pygments==2.2.0 - pyparsing==2.2.0 - python-dateutil==2.6.1 - pytz==2018.3 - scandir==1.7 - simplegeneric==0.8.1 - six==1.11.0 - subprocess32==3.2.7 - sympy==1.1.1 - traitlets==4.3.2 - wcwidth==0.1.7 - yt==3.4.1
Exporting a Virtual Environment in Anaconda to a YAML File
Export a virtual environment to a YAML file so that you or another researcher can replicate your environment using Anaconda can be done using the following steps:
- Activate the Virtual environment you wish to export:
[user@c001 ~ ]$ source activate myenv1
- Export your active virtual environment using the following command:
{myenv1} [user@c001 ~ ]$ conda env export > myenv1.yml
Virtual Environment Tips
- Avoid using pip by itself. Using python -m pip will always guarantee you are using the pip associated with that specific python being called, instead of potentially calling a pip associated with a different python.
- I recommend using a separate virtual environment for each project.
- You should never copy or move around virtual environments. Always create new ones, or use YAML exports.
- Ignore the virtual environment directories from repositories (eg GitHub, GitLab). For example, .gitignore them.