Difference between revisions of "Applications/Miniconda"
From HPC
m |
m |
||
| Line 115: | Line 115: | ||
* Please note, for some package installations it may also be necessary to specify a channel such as conda-forge. For example, the following command installs the pygobject module. | * Please note, for some package installations it may also be necessary to specify a channel such as conda-forge. For example, the following command installs the pygobject module. | ||
| − | <pre style="background-color: # | + | |
| + | <pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;"> | ||
user@c001:~> conda install -c conda-forge pygobject | user@c001:~> conda install -c conda-forge pygobject | ||
</pre> | </pre> | ||
| + | |||
===Using SLURM with virtual environment=== | ===Using SLURM with virtual environment=== | ||
Revision as of 14:25, 27 June 2019
Contents
Application Details
- Description: Miniconda python is a high-level interpreted programming language for general-purpose programming, supported by a large number of libraries for many tasks
- Versions: Miniconda (lite version of anaconda)
- Module name: python/anaconda/4.6/miniconda/3.7 (used for virtual environments)
- License: Free to use - Python Software Foundation License
Purpose
The purpose of the miniconda installation is that it is a basic Anaconda python to allow user virtual environments.
Usage Examples
- Python is provided by the Anaconda package too.
- Anaconda is the leading open data science platform powered by Python.
Environment
This provides a minimal python configuration, as shown below:
[user@login01 ~]$ module load python/anaconda/4.6/miniconda/3.7 [user@login01 ~]$ conda list # packages in environment at /trinity/clustervision/CentOS/7/apps/anaconda/4.6.0/3.7: # # Name Version Build Channel asn1crypto 0.24.0 py37_0 ca-certificates 2019.1.23 0 certifi 2019.3.9 py37_0 cffi 1.12.2 py37h2e261b9_1 chardet 3.0.4 py37_1 conda 4.6.14 py37_0 cryptography 2.6.1 py37h1ba5d50_0 idna 2.8 py37_0 libedit 3.1.20181209 hc058e9b_0 libffi 3.2.1 hd88cf55_4 libgcc-ng 8.2.0 hdf63c60_1 libstdcxx-ng 8.2.0 hdf63c60_1 ncurses 6.1 he6710b0_1 openssl 1.1.1b h7b6447c_1 pip 19.0.3 py37_0 pycosat 0.6.3 py37h14c3975_0 pycparser 2.19 py37_0 pyopenssl 19.0.0 py37_0 pysocks 1.6.8 py37_0 python 3.7.3 h0371630_0 readline 7.0 h7b6447c_5 requests 2.21.0 py37_0 ruamel_yaml 0.15.46 py37h14c3975_0 setuptools 41.0.0 py37_0 six 1.12.0 py37_0 sqlite 3.27.2 h7b6447c_0 tk 8.6.8 hbc83047_0 urllib3 1.24.1 py37_0 wheel 0.33.1 py37_0 xz 5.2.4 h14c3975_4 yaml 0.1.7 had09818_2 zlib 1.2.11 h7b6447c_3
Creation of Virtual Environment Using Anaconda on Viper
To create a virtual environment using anaconda 4.6 with python version 3.7 on Viper, you would use the conda create command as follows:
- IMPORTANT NOTE: By default the virtual environment does not use the python system packages. However because of the way viper is configured it will see python system packages because of the PYTHONPATH environment variable. So it is advised if you would like a clean environment (meaning no system packages being included) to set this environment variable to empty as follows: export PYTHONPATH=
[user@c001 ~ ]$ module load python/anaconda/4.6/miniconda/3.7 [user@c001 ~ ]$ conda create –n tensorflow1
The above command creates a new virtual environment called tensorflow1.
To activate this virtual environment you would issue the following command:
[user@c001 ~ ]$ conda activate tensorflow1
On successful activation of this virtual environment you should the name of your environment in front of your login prompt like so:
(newenv) [user@c001 ~ ]$
To exit the virtual environment use the key combination ctrl + d or 'conda deactivate'.
Adding packages
Once you have installed Miniconda and setup your environment to access it, you can then add whatever packages you wish to the installation using the conda install ... command. For example:
user@c001:~> conda install numpy
Fetching package metadata ...............
Solving package specifications: .
Package plan for installation in environment /home/t01/t01/user/miniconda3:
The following NEW packages will be INSTALLED:
blas: 1.1-openblas conda-forge
libgfortran: 3.0.0-1
numpy: 1.14.0-py36_blas_openblas_200 conda-forge [blas_openblas]
openblas: 0.2.20-7 conda-forge
The following packages will be UPDATED:
conda: 4.3.31-py36_0 --> 4.3.33-py36_0 conda-forge
The following packages will be SUPERSEDED by a higher-priority channel:
conda-env: 2.6.0-h36134e3_1 --> 2.6.0-0 conda-forge
Proceed ([y]/n)? y
- Please note, for some package installations it may also be necessary to specify a channel such as conda-forge. For example, the following command installs the pygobject module.
user@c001:~> conda install -c conda-forge pygobject
Using SLURM with virtual environment
Here is an example SLURM script which is using a Python virtual environment
Miniconda Python
#!/bin/bash #SBATCH -J BIDLSTMATT100 #SBATCH -N 1 #SBATCH --ntasks-per-node 1 #SBATCH -D /home/user/Ekman6/ #SBATCH -o debug.out #SBATCH -e debug.err #SBATCH --gres=gpu:tesla #SBATCH -p gpu #SBATCH -t 00:10:00 echo $SLURM_JOB_NODELIST module purge module load gcc/5.2.0 module load python/anaconda/4.6/miniconda/3.7 module load cuda/10.1.168 source activate /home/user/.conda/envs/tensorflow1 python /home/user/Ekman6/TATT.py
Creation of a Virtual Environment in Anaconda Using a YAML File
To create a virtual environment from a YAML file you would issue the following command:
[user@c001 ~ ]$ conda env create -f myenv.yml
This above command is creating a virtual environment from the YAML called myenv.yml. Below is a copy of the mark-up contained within the file called "myenv.yml".
name: ytenv channels: - defaults dependencies: - ca-certificates=2017.08.26=h1d4fec5_0 - certifi=2018.1.18=py27_0 - intel-openmp=2018.0.0=hc7b2577_8 - libedit=3.1=heed3624_0 - libffi=3.2.1=hd88cf55_4 - libgcc-ng=7.2.0=h7cc24e2_2 - libgfortran-ng=7.2.0=h9f7466a_2 - libstdcxx-ng=7.2.0=h7a57d05_2 - mkl=2018.0.1=h19d6760_4 - ncurses=6.0=h9df7e31_2 - numpy=1.14.0=py27h3dfced4_1 - openssl=1.0.2n=hb7f436b_0 - pip=9.0.1=py27ha730c48_4 - python=2.7.14=h1571d57_29 - readline=7.0=ha6073c6_4 - setuptools=38.4.0=py27_0 - sqlite=3.22.0=h1bed415_0 - tk=8.6.7=hc745277_3 - wheel=0.30.0=py27h2bc6bb2_1 - zlib=1.2.11=ha838bed_2 - pip: - backports.functools-lru-cache==1.5 - backports.shutil-get-terminal-size==1.0.0 - cycler==0.10.0 - decorator==4.2.1 - enum34==1.1.6 - h5py==2.7.1 - ipython==5.5.0 - ipython-genutils==0.2.0 - matplotlib==2.1.2 - mpmath==1.0.0 - pathlib2==2.3.0 - pexpect==4.4.0 - pickleshare==0.7.4 - prompt-toolkit==1.0.15 - ptyprocess==0.5.2 - pygments==2.2.0 - pyparsing==2.2.0 - python-dateutil==2.6.1 - pytz==2018.3 - scandir==1.7 - simplegeneric==0.8.1 - six==1.11.0 - subprocess32==3.2.7 - sympy==1.1.1 - traitlets==4.3.2 - wcwidth==0.1.7 - yt==3.4.1
Exporting a Virtual Environment in Anaconda to a YAML File
To export a virtual environment to a YAML file so that you or a third party can replicate your environment using Anaconda can be done using the following steps:
- Activate the Virtual environment you wish to export
[user@c001 ~ ]$ source activate newenv
- Export your active virtual environment using the following command
[user@c001 ~ ]$ conda env export > env.yml