Revision as of 13:54, 27 June 2019

Application Details

• Description: Miniconda python is a high-level interpreted programming language for general-purpose programming, supported by a large number of libraries for many tasks
• Versions: Miniconda (lite version of anaconda)
• Module name: python/anaconda/4.6/miniconda/3.7 (used for virtual environments)
• License: Free to use - Python Software Foundation License

Purpose

The purpose of the miniconda installation is that it is a basic Anaconda python to allow user virtual environments.

Usage Examples

• Python is provided by the Anaconda package too.
• Anaconda is the leading open data science platform powered by Python.

Environment

This provides a minimal python configuration, as shown below:

[user@login01 ~]$ module load python/anaconda/4.6/miniconda/3.7
[user@login01 ~]$ conda list
# packages in environment at /trinity/clustervision/CentOS/7/apps/anaconda/4.6.0/3.7:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py37_0
ca-certificates           2019.1.23                     0
certifi                   2019.3.9                 py37_0
cffi                      1.12.2           py37h2e261b9_1
chardet                   3.0.4                    py37_1
conda                     4.6.14                   py37_0
cryptography              2.6.1            py37h1ba5d50_0
idna                      2.8                      py37_0
libedit                   3.1.20181209         hc058e9b_0
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 8.2.0                hdf63c60_1
libstdcxx-ng              8.2.0                hdf63c60_1
ncurses                   6.1                  he6710b0_1
openssl                   1.1.1b               h7b6447c_1
pip                       19.0.3                   py37_0
pycosat                   0.6.3            py37h14c3975_0
pycparser                 2.19                     py37_0
pyopenssl                 19.0.0                   py37_0
pysocks                   1.6.8                    py37_0
python                    3.7.3                h0371630_0
readline                  7.0                  h7b6447c_5
requests                  2.21.0                   py37_0
ruamel_yaml               0.15.46          py37h14c3975_0
setuptools                41.0.0                   py37_0
six                       1.12.0                   py37_0
sqlite                    3.27.2               h7b6447c_0
tk                        8.6.8                hbc83047_0
urllib3                   1.24.1                   py37_0
wheel                     0.33.1                   py37_0
xz                        5.2.4                h14c3975_4
yaml                      0.1.7                had09818_2
zlib                      1.2.11               h7b6447c_3

Creation of Virtual Environment Using Anaconda on Viper

To create a virtual environment using anaconda 4.6 with python version 3.7 on Viper, you would use the conda create command as follows:

• IMPORTANT NOTE: By default the virtual environment does not use the python system packages. However because of the way viper is configured it will see python system packages because of the PYTHONPATH environment variable. So it is advised if you would like a clean environment (meaning no system packages being included) to set this environment variable to empty as follows: export PYTHONPATH=

[user@c001 ~ ]$ module load python/anaconda/4.6/miniconda/3.7
[user@c001 ~ ]$ conda create –n newenv

The above command creates a new virtual environment called newenv and installs pip and numpy within this environment.

To activate this virtual environment you would issue the following command:

[user@c001 ~ ]$ conda activate newenv

On successful activation of this virtual environment you should the name of your environment in front of your login prompt like so:

(newenv)  [user@c001 ~ ]$

To exit the virtual environment use the key combination ctrl + d or 'conda deactivate'.

Using SLURM with virtual environment

Here is an example SLURM script which is using a Python virtual environment

Miniconda Python

#!/bin/bash
#SBATCH -J BIDLSTMATT100
#SBATCH -N 1
#SBATCH --ntasks-per-node 1
#SBATCH -D /home/user/Ekman6/
#SBATCH -o debug.out
#SBATCH -e debug.err
#SBATCH --gres=gpu:tesla
#SBATCH -p gpu
#SBATCH -t 00:10:00

echo $SLURM_JOB_NODELIST

module purge
module load gcc/5.2.0
module load python/anaconda/4.6/miniconda/3.7
module load cuda/10.1.168

source activate /home/user/.conda/envs/tensorflow1

python /home/user/Ekman6/TATT.py

Creation of a Virtual Environment in Anaconda Using a YAML File

To create a virtual environment from a YAML file you would issue the following command:

[user@c001 ~ ]$ conda env create -f myenv.yml

This above command is creating a virtual environment from the YAML called myenv.yml. Below is a copy of the mark-up contained within the file called "myenv.yml".

name: ytenv
channels:
- defaults
dependencies:
- ca-certificates=2017.08.26=h1d4fec5_0
- certifi=2018.1.18=py27_0
- intel-openmp=2018.0.0=hc7b2577_8
- libedit=3.1=heed3624_0
- libffi=3.2.1=hd88cf55_4
- libgcc-ng=7.2.0=h7cc24e2_2
- libgfortran-ng=7.2.0=h9f7466a_2
- libstdcxx-ng=7.2.0=h7a57d05_2
- mkl=2018.0.1=h19d6760_4
- ncurses=6.0=h9df7e31_2
- numpy=1.14.0=py27h3dfced4_1
- openssl=1.0.2n=hb7f436b_0
- pip=9.0.1=py27ha730c48_4
- python=2.7.14=h1571d57_29
- readline=7.0=ha6073c6_4
- setuptools=38.4.0=py27_0
- sqlite=3.22.0=h1bed415_0
- tk=8.6.7=hc745277_3
- wheel=0.30.0=py27h2bc6bb2_1
- zlib=1.2.11=ha838bed_2
- pip:
  - backports.functools-lru-cache==1.5
  - backports.shutil-get-terminal-size==1.0.0
  - cycler==0.10.0
  - decorator==4.2.1
  - enum34==1.1.6
  - h5py==2.7.1
  - ipython==5.5.0
  - ipython-genutils==0.2.0
  - matplotlib==2.1.2
  - mpmath==1.0.0
  - pathlib2==2.3.0
  - pexpect==4.4.0
  - pickleshare==0.7.4
  - prompt-toolkit==1.0.15
  - ptyprocess==0.5.2
  - pygments==2.2.0
  - pyparsing==2.2.0
  - python-dateutil==2.6.1
  - pytz==2018.3
  - scandir==1.7
  - simplegeneric==0.8.1
  - six==1.11.0
  - subprocess32==3.2.7
  - sympy==1.1.1
  - traitlets==4.3.2
  - wcwidth==0.1.7
  - yt==3.4.1

Exporting a Virtual Environment in Anaconda to a YAML File

To export a virtual environment to a YAML file so that you or a third party can replicate your environment using Anaconda can be done using the following steps:

• Activate the Virtual environment you wish to export

[user@c001 ~ ]$ source activate newenv

• Export your active virtual environment using the following command

[user@c001 ~ ]$ conda env export > env.yml