HPC

Applications/Python

From HPC
Revision as of 10:53, 10 May 2019 by Pysdlb (talk | contribs) (Pysdlb moved page Applications/Xython to Applications/Python over a redirect without leaving a redirect)

Jump to: navigation , search

Application Details

  • Description: Python is a high-level interpreted programming language for general-purpose programming, supported by a large number of libraries for many tasks
  • Versions: Python 2.7.11 and 3.5.1
  • Module names: python/2.7.11 python/3.5.1
  • License: Free to use - Python Software Foundation License


Usage Examples

  • Python is provided by the Anaconda package too.
  • Anaconda is the leading open data science platform powered by Python.

Interactive

Interactive with command line: 

[username@c170 ~]$ module add python/anaconda/4.0/2.7
[username@c170 ~]$ python
Python 2.7.5 (default, Nov 20 2015, 02:00:19)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

Interactive with command line with IPython: 

[username@c170 ~]$ ipython
Python 2.7.13 |Anaconda custom (64-bit)| (default, Dec 20 2016, 23:09:15)
Type "copyright", "credits" or "license" for more information.

IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]:

Batch Submission

#!/bin/bash
#SBATCH -J PythonTest                # Job name, you can change it to whatever you want
#SBATCH -N 1                                  # Number of nodes 
#SBATCH --ntasks-per-node 1    # Number of cores per node
#SBATCH -o %N.%j.out                # Standard output will be written here
#SBATCH -e %N.%j.err                 # Standard error will be written here
#SBATCH -p compute                  # Slurm partition, where you want the job to be queued 

module purge
module add python/3.5.1
 
python PythonTest.py

This is then submitted as follows: 

[username@login01 ~]$ sbatch Pythontest.job
Submitted batch job 289522

Virtual Environments

A Python virtual environment allows users to create a custom environment(s) in which they can have the packages, and versions of those packages that are required without having elevated user privileges.

We recommend the Virtualenv installation when you use a specialised package that would not be used by the wider HPC community. Virtualenv is a virtual Python environment isolated from other Python development, incapable of interfering with or being affected by other Python programs on the same HPC. During the Virtualenv installation process, you will install can install not only the additional package but all the dependencies that go with it. (This is actually pretty easy.) All in all, Virtualenv provides a safe and reliable mechanism for installing and running additional packages.

There are many benefits to using a virtual environment on a system. Here is a short non-exhaustive list of a few of the key benefits:

  • Control over packages and package versions.
  • An arbitrary number of virtual environments can be created for different tasks.
  • Reproducible - A user can replicate a python environment on any system, so as to be able to run or re-run a particular task or job under the same conditions.

Creation of Virtual Environment Using Python 2.7.11 or 3.5.1 on Viper

To create a virtual environment you would issue the following command:

  • IMPORTANT NOTE: By default the virtual environment does not use the python system packages. However because of the way viper is configured it will see python system packages because of the PYTHONPATH environment variable. So it is advised if you would like a clean environment (meaning no system packages being included) to set this environment variable to empty as follows: "export PYTHONPATH="


[user@c001 ~ ]$ module add python/3.5.1
[user@c001 ~ ]$ virtualenv foo

This creates a directory containing your virtual environment named foo . To activate this virtual environment you would issue the following command: 

[user@c001 ~ ]$ source foo/bin/activate

If the environment starts correctly you should see the name of your environment in round brackets in front of the command prompt. For this particular example it should appear as follows: 

(foo)  [user@c001 ~ ]$

To exit the virtual environment use the key combination ctrl + d

Creation of Virtual Environment Using Anaconda on Viper

To create a virtual environment using anaconda 4.1.1 with python version 2.7 on Viper, you would use the conda create command as follows:

  • IMPORTANT NOTE: By default the virtual environment does not use the python system packages. However because of the way viper is configured it will see python system packages because of the PYTHONPATH environment variable. So it is advised if you would like a clean environment (meaning no system packages being included) to set this environment variable to empty as follows: export PYTHONPATH=
[user@c001 ~ ]$ module load python/anaconda/4.0/3.5
[user@c001 ~ ]$ conda create –n newenv pip numpy

The above command creates a new virtual environment called newenv and installs pip and numpy within this environment.

To activate this virtual environment you would issue the following command: 

[user@c001 ~ ]$ source activate newenv

On successful activation of this virtual environment you should the name of your environment in front of your login prompt like so: 

(newenv)  [user@c001 ~ ]$

To exit the virtual environment use the key combination ctrl + d or deactivate.

Using SLURM with virtual environment

Here is an example SLURM script which is using a Python virtual environment

Standard Python

#!/bin/bash
#SBATCH -J PythonTest                # Job name, you can change it to whatever you want
#SBATCH -N 1                                  # Number of nodes 
#SBATCH --ntasks-per-node 1    # Number of cores per node
#SBATCH -o %N.%j.out                # Standard output will be written here
#SBATCH -e %N.%j.err                 # Standard error will be written here
#SBATCH -p compute                  # Slurm partition, where you want the job to be queued 

module purge
module add python/3.5.1
 
source newenv/bin/activate
python PythonTest.py

Anaconda Python

#!/bin/bash
#SBATCH -J PythonTest                # Job name, you can change it to whatever you want
#SBATCH -N 1                                  # Number of nodes 
#SBATCH --ntasks-per-node 1    # Number of cores per node
#SBATCH -o %N.%j.out                # Standard output will be written here
#SBATCH -e %N.%j.err                 # Standard error will be written here
#SBATCH -p compute                  # Slurm partition, where you want the job to be queued 

module purge
module add python/anaconda/4.0/3.5
 
conda activate myenviro
python PythonTest.py


Creation of a Virtual Environment in Anaconda Using a YAML File

To create a virtual environment from a YAML file you would issue the following command: 

[user@c001 ~ ]$ conda env create -f myenv.yml

This above command is creating a virtual environment from the YAML called myenv.yml. Below is a copy of the mark-up contained within the file called "myenv.yml". 

name: ytenv
channels:
- defaults
dependencies:
- ca-certificates=2017.08.26=h1d4fec5_0
- certifi=2018.1.18=py27_0
- intel-openmp=2018.0.0=hc7b2577_8
- libedit=3.1=heed3624_0
- libffi=3.2.1=hd88cf55_4
- libgcc-ng=7.2.0=h7cc24e2_2
- libgfortran-ng=7.2.0=h9f7466a_2
- libstdcxx-ng=7.2.0=h7a57d05_2
- mkl=2018.0.1=h19d6760_4
- ncurses=6.0=h9df7e31_2
- numpy=1.14.0=py27h3dfced4_1
- openssl=1.0.2n=hb7f436b_0
- pip=9.0.1=py27ha730c48_4
- python=2.7.14=h1571d57_29
- readline=7.0=ha6073c6_4
- setuptools=38.4.0=py27_0
- sqlite=3.22.0=h1bed415_0
- tk=8.6.7=hc745277_3
- wheel=0.30.0=py27h2bc6bb2_1
- zlib=1.2.11=ha838bed_2
- pip:
  - backports.functools-lru-cache==1.5
  - backports.shutil-get-terminal-size==1.0.0
  - cycler==0.10.0
  - decorator==4.2.1
  - enum34==1.1.6
  - h5py==2.7.1
  - ipython==5.5.0
  - ipython-genutils==0.2.0
  - matplotlib==2.1.2
  - mpmath==1.0.0
  - pathlib2==2.3.0
  - pexpect==4.4.0
  - pickleshare==0.7.4
  - prompt-toolkit==1.0.15
  - ptyprocess==0.5.2
  - pygments==2.2.0
  - pyparsing==2.2.0
  - python-dateutil==2.6.1
  - pytz==2018.3
  - scandir==1.7
  - simplegeneric==0.8.1
  - six==1.11.0
  - subprocess32==3.2.7
  - sympy==1.1.1
  - traitlets==4.3.2
  - wcwidth==0.1.7
  - yt==3.4.1

Exporting a Virtual Environment in Anaconda to a YAML File

To export a virtual environment to a YAML file so that you or a third party can replicate your environment using Anaconda can be done using the following steps:

  • Activate the Virtual environment you wish to export
[user@c001 ~ ]$ source activate newenv
  • Export your active virtual environment using the following command
[user@c001 ~ ]$ conda env export > env.yml

Further Information

Icon home.png
Retrieved from ‘http://hpc.mediawiki.hull.ac.uk/index.php?title=Applications/Python&oldid=7918’