HPC

Difference between revisions of "Applications/Miniconda"

From HPC
Jump to: navigation , search
m
m (→‎Environment)
 
(31 intermediate revisions by the same user not shown)
Line 2: Line 2:
  
 
=== Application Details ===
 
=== Application Details ===
−
* Description: Miniconda python is a high-level interpreted programming language for general-purpose programming, supported by a large number of libraries for many tasks for which the user can install for customising their environment
+
* Description: Miniconda Python is a high-level interpreted programming language for general-purpose programming, supported by a large number of libraries for many tasks which the user can install for customizing their environment
 
* Versions: Miniconda (''lite version of anaconda'')
 
* Versions: Miniconda (''lite version of anaconda'')
−
* Module name: python/anaconda/4.6/miniconda/3.7 ('''used for virtual environments only''')
+
* Module name: python/anaconda/4.6/miniconda/3.7 and '''python/anaconda/202111/3.9'''  ('''used for virtual environments only''')
 
* License: Free to use - [https://en.wikipedia.org/wiki/Python_Software_Foundation_License Python Software Foundation License]
 
* License: Free to use - [https://en.wikipedia.org/wiki/Python_Software_Foundation_License Python Software Foundation License]
  
 
==Purpose==
 
==Purpose==
  
−
The purpose of the miniconda installation is that it is a basic Anaconda python install to allow user virtual environments.
+
'''Important''': The purpose of the miniconda installation is that it is a basic Anaconda python install with a minimal set of modules to allow users virtual environments. This allows for custom virtual environments which can be quite small.
  
 
===Definition===
 
===Definition===
  
−
A virtual environment is a named, isolated, working copy of Python that that maintains its own files, directories, and paths so that you can work with specific versions of libraries or Python itself without affecting other Python projects. Virtual environments make it easy to cleanly separate different projects and avoid problems with different dependencies and version requirements across components.  
+
A virtual environment is a named, isolated, working copy of Python that maintains its own files, directories, and paths so that you can work with specific versions of libraries or Python itself without affecting other Python projects. Virtual environments make it easy to cleanly separate different projects and avoid problems with different dependencies and version requirements across components.  
  
 
The conda command is the preferred interface for managing installations and virtual environments with the Anaconda Python distribution.
 
The conda command is the preferred interface for managing installations and virtual environments with the Anaconda Python distribution.
  
−
'''Note''' : It is very likely within the next HPC, all python environments will be all virtual environments.
+
'''Note''': It is very likely within the next HPC, all Python environments will be all virtual environments.
  
 +
==Environment==
  
−
== Usage Examples ==
+
This provides a minimal Python configuration, as shown below:
−
 
 
−
* Python is provided by the [[applications/Anaconda|Anaconda package]] too.
 
−
* Anaconda is the leading open data science platform powered by Python.
 
  
−
==Environment==
+
* '''Important''' : For building an environment for [[TensorflowforGPU| AI models on a GPU]], otherwise continue below:
  
−
This provides a minimal python configuration, as shown below:
 
  
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
−
[user@login01 ~]$ module load python/anaconda/4.6/miniconda/3.7
+
[user@login01 ~]$ module load python/anaconda/202111/3.9
 
[user@login01 ~]$ conda list
 
[user@login01 ~]$ conda list
−
# packages in environment at /trinity/clustervision/CentOS/7/apps/anaconda/4.6.0/3.7:
+
# packages in the environment  
 
#
 
#
 
# Name                    Version                  Build  Channel
 
# Name                    Version                  Build  Channel
Line 72: Line 69:
 
===Creation of Virtual Environment Using Anaconda on Viper===
 
===Creation of Virtual Environment Using Anaconda on Viper===
  
−
To create a virtual environment using anaconda 4.6 with python version 3.7 on Viper, you would use the conda create command as follows:
+
* '''IMPORTANT''': If you see an error like this:
 +
 
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
 +
To initialize your shell, run
 +
 
 +
    $ conda init <SHELL_NAME>
 +
 
 +
Currently, supported shells are:
 +
  - bash
 +
  - fish
 +
  - tcsh
 +
  - xonsh
 +
  - zsh
 +
  - powershell
 +
</pre>
 +
 
 +
You will need to type:
 +
 
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
$ conda init bash
 +
</pre>
  
−
* '''IMPORTANT NOTE''': By default the virtual environment does not use the python system packages. However because of the way viper is configured it will see python system packages because of the PYTHONPATH environment variable. So it is advised if you would like a clean environment (meaning no system packages being included) to set this environment variable to empty as follows: ''export PYTHONPATH=''
+
To create a virtual environment using Anaconda 4.6 with Python version 3.7 on Viper,  you would use the conda create command as follows:
  
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
−
[user@c001 ~ ]$ module load python/anaconda/4.6/miniconda/3.7
+
[user@c001 ~ ]$ module load python/anaconda/202111/3.9
−
[user@c001 ~ ]$ conda create –n tensorflow1
+
[user@c001 ~ ]$ conda create –n myenv1
 
</pre>
 
</pre>
−
The above command creates a new virtual environment called tensorflow1.
+
The above command creates a new virtual environment called myenv1.
  
 
To activate this virtual environment you would issue the following command:
 
To activate this virtual environment you would issue the following command:
  
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
−
[user@c001 ~ ]$ conda activate tensorflow1
+
[user@c001 ~ ]$ conda activate myenv1
 
</pre>
 
</pre>
−
On successful activation of this virtual environment you should the name of your environment in front of your login prompt like so:
+
 
 +
 
 +
On successful activation of this virtual environment, you should the name of your environment in front of your login prompt like so:
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
−
(tensorflow1)  [user@c001 ~ ]$
+
(myenv1)  [user@c001 ~ ]$
 
</pre>
 
</pre>
 +
 
To exit the virtual environment use the key combination '''ctrl + d''' or 'conda deactivate'.
 
To exit the virtual environment use the key combination '''ctrl + d''' or 'conda deactivate'.
  
 
===Adding packages===
 
===Adding packages===
  
−
Once you have installed Miniconda and setup your environment to access it, you can then add whatever packages you wish to the installation using the conda install ... command. For example:
+
Once you have installed Miniconda and set up your environment to access it, you can then add whatever packages you wish to the installation using the conda install ... command. For example:
  
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
−
(tensorflow1) user@c001:~> conda install numpy
+
(myenv1) user@c001:~> conda install numpy
 
Fetching package metadata ...............
 
Fetching package metadata ...............
−
Solving package specifications: .
+
Solving package specifications:
  
 
Package plan for installation in environment /home/t01/t01/user/miniconda3:
 
Package plan for installation in environment /home/t01/t01/user/miniconda3:
Line 126: Line 147:
  
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
−
(tensorflow1) [user@c001]$ conda install -c conda-forge pygobject  
+
(myenv1) [user@c001]$ conda install -c conda-forge pygobject  
 
</pre>
 
</pre>
  
Line 132: Line 153:
  
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
−
[user@c001]$ conda create -n myenv scipy=0.15.0
+
[user@c001]$ conda create -n myenv2 scipy=0.15.0
 
</pre>
 
</pre>
  
−
* or even defining the python version at 3.4
+
* or even defining the Python version at 3.4
  
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
Line 154: Line 175:
 
[user@c001]$ conda remove --name EnvironmentNAME --all
 
[user@c001]$ conda remove --name EnvironmentNAME --all
 
</pre>
 
</pre>
 +
 +
 +
 +
===Using your environment in an interactive session===
 +
 +
To test and debug your program it is recommended to use an interactive session, these are also useful for programs that have short runtimes too.
 +
 +
* If you want a CPU-based node to use the commands:
 +
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
[user@login01]$ interactive
 +
[user@c001]$ conda activate pytorch01
 +
{pytorch01} [user@c001]$ python mypytorchprogram.py
 +
</pre>
 +
 +
* If you want a GPU-based node instead use the commands:
 +
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
[user@login01]$ interactive -pgpu
 +
[user@gpu02]$ conda activate pytorch01
 +
{pytorch01} [user@gpu02]$ python mypytorchprogram.py
 +
</pre>
 +
 +
If you see an error like this:
 +
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
 +
To initialize your shell, run
 +
 +
    $ conda init <SHELL_NAME>
 +
 +
Currently, supported shells are:
 +
  - bash
 +
  - fish
 +
  - tcsh
 +
  - xonsh
 +
  - zsh
 +
  - powershell
 +
</pre>
 +
 +
You will need to type:
 +
 +
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 +
$ conda init bash
 +
</pre>
 +
 +
then exit the terminal you're in and restart the session to read the new conda settings.
  
  
Line 159: Line 227:
 
===Using SLURM with a virtual environment===
 
===Using SLURM with a virtual environment===
  
−
Here is an example SLURM script which is using a Python virtual environment
+
Here are two examples of SLURM scripts which are using a Python virtual environment
 +
 
 +
* '''substitute''' ''/home/<user>'' for your own path
 +
 
 +
====Compute Node Example====
 +
 
  
 
<pre style="background-color: #C8C8C8; color: black; font-family: monospace, sans-serif;">
 
<pre style="background-color: #C8C8C8; color: black; font-family: monospace, sans-serif;">
 
#!/bin/bash
 
#!/bin/bash
−
#SBATCH -J BIDLSTMATT100
+
#SBATCH -J BUILDCPU
 +
#SBATCH -N 1
 +
#SBATCH --ntasks-per-node 12
 +
#SBATCH -D /home/<user>/
 +
#SBATCH -o debug.out
 +
#SBATCH -e debug.err
 +
#SBATCH -p compute
 +
#SBATCH -t 00:10:00
 +
#SBATCH --mail-user= your email address here
 +
 
 +
echo $SLURM_JOB_NODELIST
 +
 
 +
module purge
 +
module load python/anaconda/20220712/3.9
 +
 
 +
source activate /home/<user>/.conda/envs/bioinformatics1
 +
export PATH=/home/<user>/.conda/envs/bioinformatics/bin:${PATH}
 +
 
 +
python /home/user/TATT-CPU.py
 +
</pre>
 +
 
 +
 
 +
====GPU Node Example====
 +
 
 +
 
 +
<pre style="background-color: #C8C8C8; color: black; font-family: monospace, sans-serif;">
 +
#!/bin/bash
 +
#SBATCH -J BIDGPU
 
#SBATCH -N 1
 
#SBATCH -N 1
 
#SBATCH --ntasks-per-node 1
 
#SBATCH --ntasks-per-node 1
−
#SBATCH -D /home/user/Ekman6/
+
#SBATCH -D /home/<user>/
 
#SBATCH -o debug.out
 
#SBATCH -o debug.out
 
#SBATCH -e debug.err
 
#SBATCH -e debug.err
−
#SBATCH --gres=gpu:tesla
+
#SBATCH --gres=gpu
 
#SBATCH -p gpu
 
#SBATCH -p gpu
 
#SBATCH -t 00:10:00
 
#SBATCH -t 00:10:00
 +
#SBATCH --mail-user= your email address here
  
 
echo $SLURM_JOB_NODELIST
 
echo $SLURM_JOB_NODELIST
Line 177: Line 278:
 
module purge
 
module purge
 
module load gcc/5.2.0
 
module load gcc/5.2.0
−
module load python/anaconda/4.6/miniconda/3.7
+
module load python/anaconda/20220712/3.9
−
module load cuda/10.1.168
+
module load cuda/11.5.0
  
−
source activate /home/user/.conda/envs/tensorflow1
+
source activate /home/<user>/.conda/envs/bioinformatics1
 +
export PATH=/home/<user>/.conda/envs/bioinformatics/bin:${PATH}
  
−
python /home/user/Ekman6/TATT.py
+
python /home/user/TATT-GPU.py
 
</pre>
 
</pre>
  
Line 194: Line 296:
 
</pre>
 
</pre>
  
−
This above command is creating a virtual environment from the YAML called myenv.yml. Below is a copy of the mark-up contained within the file called "myenv.yml".
+
The above command is creating a virtual environment from the YAML called myenv.yml. Below is a copy of the markup in the file called "myenv.yml".
  
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
Line 253: Line 355:
 
===Exporting a Virtual Environment in Anaconda to a YAML File===
 
===Exporting a Virtual Environment in Anaconda to a YAML File===
  
−
To export a virtual environment to a YAML file so that you or another researcher can replicate your environment using Anaconda can be done using the following steps:
+
Export a virtual environment to a YAML file so that you or another researcher can replicate your environment using Anaconda can be done using the following steps:
  
 
*  Activate the Virtual environment you wish to export:
 
*  Activate the Virtual environment you wish to export:
  
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
−
[user@c001 ~ ]$ source activate tensorflow1
+
[user@c001 ~ ]$ source activate myenv1
 
</pre>
 
</pre>
  
Line 264: Line 366:
  
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
−
{tensorflow1} [user@c001 ~ ]$ conda env export > tensorflow1.yml
+
{myenv1} [user@c001 ~ ]$ conda env export > myenv1.yml
 
</pre>
 
</pre>
  
−
== Further Information ==
+
==Virtual Environment Tips==
 +
 
 +
 
 +
* Avoid using pip by itself. Using '''python -m pip''' will always guarantee you are using the pip associated with that specific python being called, instead of potentially calling a pip associated with a different python.
 +
* I recommend using a separate virtual environment for each project.
 +
* You should never copy or move around virtual environments. Always create new ones, or use YAML exports.
 +
* Ignore the virtual environment directories from repositories (''eg GitHub, GitLab''). For example, '''.gitignore''' them.
 +
 
 +
== Next Steps ==
  
 
* [[Programming/Python|Python programming]]
 
* [[Programming/Python|Python programming]]
 
* [[Applications/Anaconda|Anaconda python]]
 
* [[Applications/Anaconda|Anaconda python]]
  
−
==Navigation==
+
{{Modulepagenav}}
−
 
 
−
* [[Main_Page|Home]]
 
−
* [[Applications|Application support]] *
 
−
* [[General|General]]
 
−
* [[Programming|Programming support]]
 

Latest revision as of 09:08, 21 April 2023

Application Details

  • Description: Miniconda Python is a high-level interpreted programming language for general-purpose programming, supported by a large number of libraries for many tasks which the user can install for customizing their environment
  • Versions: Miniconda (lite version of anaconda)
  • Module name: python/anaconda/4.6/miniconda/3.7 and python/anaconda/202111/3.9 (used for virtual environments only)
  • License: Free to use - Python Software Foundation License

Purpose

Important: The purpose of the miniconda installation is that it is a basic Anaconda python install with a minimal set of modules to allow users virtual environments. This allows for custom virtual environments which can be quite small.

Definition

A virtual environment is a named, isolated, working copy of Python that maintains its own files, directories, and paths so that you can work with specific versions of libraries or Python itself without affecting other Python projects. Virtual environments make it easy to cleanly separate different projects and avoid problems with different dependencies and version requirements across components.

The conda command is the preferred interface for managing installations and virtual environments with the Anaconda Python distribution.

Note: It is very likely within the next HPC, all Python environments will be all virtual environments.

Environment

This provides a minimal Python configuration, as shown below: 


[user@login01 ~]$ module load python/anaconda/202111/3.9
[user@login01 ~]$ conda list
# packages in the environment 
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py37_0
ca-certificates           2019.1.23                     0
certifi                   2019.3.9                 py37_0
cffi                      1.12.2           py37h2e261b9_1
chardet                   3.0.4                    py37_1
conda                     4.6.14                   py37_0
cryptography              2.6.1            py37h1ba5d50_0
idna                      2.8                      py37_0
libedit                   3.1.20181209         hc058e9b_0
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 8.2.0                hdf63c60_1
libstdcxx-ng              8.2.0                hdf63c60_1
ncurses                   6.1                  he6710b0_1
openssl                   1.1.1b               h7b6447c_1
pip                       19.0.3                   py37_0
pycosat                   0.6.3            py37h14c3975_0
pycparser                 2.19                     py37_0
pyopenssl                 19.0.0                   py37_0
pysocks                   1.6.8                    py37_0
python                    3.7.3                h0371630_0
readline                  7.0                  h7b6447c_5
requests                  2.21.0                   py37_0
ruamel_yaml               0.15.46          py37h14c3975_0
setuptools                41.0.0                   py37_0
six                       1.12.0                   py37_0
sqlite                    3.27.2               h7b6447c_0
tk                        8.6.8                hbc83047_0
urllib3                   1.24.1                   py37_0
wheel                     0.33.1                   py37_0
xz                        5.2.4                h14c3975_4
yaml                      0.1.7                had09818_2
zlib                      1.2.11               h7b6447c_3

Creation of Virtual Environment Using Anaconda on Viper

  • IMPORTANT: If you see an error like this:
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently, supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

You will need to type: 

$ conda init bash

To create a virtual environment using Anaconda 4.6 with Python version 3.7 on Viper, you would use the conda create command as follows: 

[user@c001 ~ ]$ module load python/anaconda/202111/3.9
[user@c001 ~ ]$ conda create –n myenv1

The above command creates a new virtual environment called myenv1.

To activate this virtual environment you would issue the following command: 

[user@c001 ~ ]$ conda activate myenv1


On successful activation of this virtual environment, you should the name of your environment in front of your login prompt like so: 

(myenv1)  [user@c001 ~ ]$

To exit the virtual environment use the key combination ctrl + d or 'conda deactivate'.

Adding packages

Once you have installed Miniconda and set up your environment to access it, you can then add whatever packages you wish to the installation using the conda install ... command. For example: 

(myenv1) user@c001:~> conda install numpy
Fetching package metadata ...............
Solving package specifications:

Package plan for installation in environment /home/t01/t01/user/miniconda3:

The following NEW packages will be INSTALLED:

    blas:        1.1-openblas                  conda-forge
    libgfortran: 3.0.0-1                                  
    numpy:       1.14.0-py36_blas_openblas_200 conda-forge [blas_openblas]
    openblas:    0.2.20-7                      conda-forge

The following packages will be UPDATED:

    conda:       4.3.31-py36_0                             --> 4.3.33-py36_0 conda-forge

The following packages will be SUPERSEDED by a higher-priority channel:

    conda-env:   2.6.0-h36134e3_1                          --> 2.6.0-0       conda-forge

Proceed ([y]/n)? y
  • Please note, for some package installations it may also be necessary to specify a channel such as conda-forge. For example, the following command installs the pygobject module.


(myenv1) [user@c001]$ conda install -c conda-forge pygobject
  • To create an environment with a specific version of a package:
[user@c001]$ conda create -n myenv2 scipy=0.15.0
  • or even defining the Python version at 3.4
[user@c001]$ conda create -n myenv python=3.4 scipy=0.15.0 astroid babel

Clone an environment

[user@c001]$ conda create -n OriginalENV --clone NewENV

Removing an environment

To delete a conda environment, enter the following, where yourenvname is the name of the environment you wish to delete. 

[user@c001]$ conda remove --name EnvironmentNAME --all


Using your environment in an interactive session

To test and debug your program it is recommended to use an interactive session, these are also useful for programs that have short runtimes too.

  • If you want a CPU-based node to use the commands:
[user@login01]$ interactive
[user@c001]$ conda activate pytorch01
{pytorch01} [user@c001]$ python mypytorchprogram.py
  • If you want a GPU-based node instead use the commands:
[user@login01]$ interactive -pgpu
[user@gpu02]$ conda activate pytorch01
{pytorch01} [user@gpu02]$ python mypytorchprogram.py

If you see an error like this: 

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently, supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

You will need to type: 

$ conda init bash

then exit the terminal you're in and restart the session to read the new conda settings.


Using SLURM with a virtual environment

Here are two examples of SLURM scripts which are using a Python virtual environment

  • substitute /home/<user> for your own path

Compute Node Example

#!/bin/bash
#SBATCH -J BUILDCPU
#SBATCH -N 1
#SBATCH --ntasks-per-node 12
#SBATCH -D /home/<user>/
#SBATCH -o debug.out
#SBATCH -e debug.err
#SBATCH -p compute
#SBATCH -t 00:10:00
#SBATCH --mail-user= your email address here

echo $SLURM_JOB_NODELIST

module purge
module load python/anaconda/20220712/3.9

source activate /home/<user>/.conda/envs/bioinformatics1
export PATH=/home/<user>/.conda/envs/bioinformatics/bin:${PATH}

python /home/user/TATT-CPU.py


GPU Node Example

#!/bin/bash
#SBATCH -J BIDGPU
#SBATCH -N 1
#SBATCH --ntasks-per-node 1
#SBATCH -D /home/<user>/
#SBATCH -o debug.out
#SBATCH -e debug.err
#SBATCH --gres=gpu
#SBATCH -p gpu
#SBATCH -t 00:10:00
#SBATCH --mail-user= your email address here

echo $SLURM_JOB_NODELIST

module purge
module load gcc/5.2.0
module load  python/anaconda/20220712/3.9
module load cuda/11.5.0

source activate /home/<user>/.conda/envs/bioinformatics1
export PATH=/home/<user>/.conda/envs/bioinformatics/bin:${PATH}

python /home/user/TATT-GPU.py


Creation of a Virtual Environment in Anaconda Using a YAML File

To create a virtual environment from a YAML file you would issue the following command: 

[user@c001 ~ ]$ conda env create -f myenv.yml

The above command is creating a virtual environment from the YAML called myenv.yml. Below is a copy of the markup in the file called "myenv.yml". 

name: ytenv
channels:
- defaults
dependencies:
- ca-certificates=2017.08.26=h1d4fec5_0
- certifi=2018.1.18=py27_0
- intel-openmp=2018.0.0=hc7b2577_8
- libedit=3.1=heed3624_0
- libffi=3.2.1=hd88cf55_4
- libgcc-ng=7.2.0=h7cc24e2_2
- libgfortran-ng=7.2.0=h9f7466a_2
- libstdcxx-ng=7.2.0=h7a57d05_2
- mkl=2018.0.1=h19d6760_4
- ncurses=6.0=h9df7e31_2
- numpy=1.14.0=py27h3dfced4_1
- openssl=1.0.2n=hb7f436b_0
- pip=9.0.1=py27ha730c48_4
- python=2.7.14=h1571d57_29
- readline=7.0=ha6073c6_4
- setuptools=38.4.0=py27_0
- sqlite=3.22.0=h1bed415_0
- tk=8.6.7=hc745277_3
- wheel=0.30.0=py27h2bc6bb2_1
- zlib=1.2.11=ha838bed_2
- pip:
  - backports.functools-lru-cache==1.5
  - backports.shutil-get-terminal-size==1.0.0
  - cycler==0.10.0
  - decorator==4.2.1
  - enum34==1.1.6
  - h5py==2.7.1
  - ipython==5.5.0
  - ipython-genutils==0.2.0
  - matplotlib==2.1.2
  - mpmath==1.0.0
  - pathlib2==2.3.0
  - pexpect==4.4.0
  - pickleshare==0.7.4
  - prompt-toolkit==1.0.15
  - ptyprocess==0.5.2
  - pygments==2.2.0
  - pyparsing==2.2.0
  - python-dateutil==2.6.1
  - pytz==2018.3
  - scandir==1.7
  - simplegeneric==0.8.1
  - six==1.11.0
  - subprocess32==3.2.7
  - sympy==1.1.1
  - traitlets==4.3.2
  - wcwidth==0.1.7
  - yt==3.4.1

Exporting a Virtual Environment in Anaconda to a YAML File

Export a virtual environment to a YAML file so that you or another researcher can replicate your environment using Anaconda can be done using the following steps:

  • Activate the Virtual environment you wish to export:
[user@c001 ~ ]$ source activate myenv1
  • Export your active virtual environment using the following command:
{myenv1} [user@c001 ~ ]$ conda env export > myenv1.yml

Virtual Environment Tips

  • Avoid using pip by itself. Using python -m pip will always guarantee you are using the pip associated with that specific python being called, instead of potentially calling a pip associated with a different python.
  • I recommend using a separate virtual environment for each project.
  • You should never copy or move around virtual environments. Always create new ones, or use YAML exports.
  • Ignore the virtual environment directories from repositories (eg GitHub, GitLab). For example, .gitignore them.

Next Steps





Modules | Main Page | Further Topics

Retrieved from ‘http://hpc.mediawiki.hull.ac.uk/index.php?title=Applications/Miniconda&oldid=41505’