Applications/Python
Application Details
- Description: Python is a high-level interpreted programming language for general-purpose programming, supported by a large number of libraries for many tasks
- Versions: Python 2.7.11 and 3.5.1
- Module names: python/2.7.11 python/3.5.1
- License: Free to use - Python Software Foundation License
Usage Examples
Python is provided by the Anaconda package too
Interactive
Interactive with command line:
[username@c170 ~]$ module add python/anaconda/4.0/2.7 [username@c170 ~]$ python Python 2.7.5 (default, Nov 20 2015, 02:00:19) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>>
Interactive with command line with IPython:
[username@c170 ~]$ ipython Python 2.7.13 |Anaconda custom (64-bit)| (default, Dec 20 2016, 23:09:15) Type "copyright", "credits" or "license" for more information. IPython 5.1.0 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details. In [1]:
Batch Submission
#!/bin/bash #SBATCH -J PythonTest # Job name, you can change it to whatever you want #SBATCH -N 1 # Number of nodes #SBATCH --ntasks-per-node 1 # Number of cores per node #SBATCH -o %N.%j.out # Standard output will be written here #SBATCH -e %N.%j.err # Standard error will be written here #SBATCH -p compute # Slurm partition, where you want the job to be queued module purge module add python/3.5.1 python PythonTest.py
This is then submitted as follows:
[username@login01 ~]$ sbatch Pythontest.job Submitted batch job 289522
Virtual Environments
A Python Virtual Environment allows users to create a custom environment/s in which they can have the packages, and versions of said packages that they require without having to have elevated user privileges on the system.
There are many benefits to using a virtual environment on a system. Here is a short non-exhaustive list of a few of the key benefits:
- Control over packages and package versions
- An arbitrary number of virtual environments can be created for different tasks
- Reproducibility - A user can replicate a python environment on any system, so as to be able to run or re-run a particular task or job under the same conditions
Creation of Virtual Environment Using Python 2.7.11 or 3.5.1 on Viper
To create a virtual environment you would issue the following command:
IMPORTANT NOTE: By default the virtual environment does not use the python system packages. However because of the way viper is configured it will see python system packages because of the PYTHONPATH environment variable. So it is advised if you would like a clean environment (meaning no system packages being included) to set this environment variable to empty as follows: export PYTHONPATH=
[user@c001 ~ ]$ virtualenv foo
This creates a directory containing your virtual environment named foo . To activate this virtual environment you would issue the following command:
[user@c001 ~ ]$ source foo/bin/activate
If the environment starts correctly you should see the name of your environment in round brackets in front of the command prompt. For this particular example it should appear as follows:
(foo) [user@c001 ~ ]$
To exit the virtual environment use the key combination ctrl + d
Creation of Virtual Environment Using Anaconda on Viper
To create a virtual environment using anaconda 4.1.1 with python version 2.7 on Viper, you would use the conda create command as follows:
IMPORTANT NOTE: By default the virtual environment does not use the python system packages. However because of the way viper is configured it will see python system packages because of the PYTHONPATH environment variable. So it is advised if you would like a clean environment (meaning no system packages being included) to set this environment variable to empty as follows: export PYTHONPATH=
[user@c001 ~ ]$ conda create –n newenv pip numpy
The above command creates a new virtual environment called newenv and installs pip and numpy within this environment.
To activate this virtual environment you would issue the following command:
[user@c001 ~ ]$ source activate newenv
On successful activation of this virtual environment you should the name of your environment in front of your login prompt like so:
(newenv) [user@c001 ~ ]$
To exit the virtual environment use the key combination ctrl + d
Creation of a Virtual Environment in Anaconda Using a YAML File
To create a virtual environment from a YAML file you would issue the following command:
[user@c001 ~ ]$ conda env create -f myenv.yml
This above command is creating a virtual environment from the YAML called myenv.yml. Below is a copy of the mark-up contained within the file called "myenv.yml".
name: ytenv channels: - defaults dependencies: - ca-certificates=2017.08.26=h1d4fec5_0 - certifi=2018.1.18=py27_0 - intel-openmp=2018.0.0=hc7b2577_8 - libedit=3.1=heed3624_0 - libffi=3.2.1=hd88cf55_4 - libgcc-ng=7.2.0=h7cc24e2_2 - libgfortran-ng=7.2.0=h9f7466a_2 - libstdcxx-ng=7.2.0=h7a57d05_2 - mkl=2018.0.1=h19d6760_4 - ncurses=6.0=h9df7e31_2 - numpy=1.14.0=py27h3dfced4_1 - openssl=1.0.2n=hb7f436b_0 - pip=9.0.1=py27ha730c48_4 - python=2.7.14=h1571d57_29 - readline=7.0=ha6073c6_4 - setuptools=38.4.0=py27_0 - sqlite=3.22.0=h1bed415_0 - tk=8.6.7=hc745277_3 - wheel=0.30.0=py27h2bc6bb2_1 - zlib=1.2.11=ha838bed_2 - pip: - backports.functools-lru-cache==1.5 - backports.shutil-get-terminal-size==1.0.0 - cycler==0.10.0 - decorator==4.2.1 - enum34==1.1.6 - h5py==2.7.1 - ipython==5.5.0 - ipython-genutils==0.2.0 - matplotlib==2.1.2 - mpmath==1.0.0 - pathlib2==2.3.0 - pexpect==4.4.0 - pickleshare==0.7.4 - prompt-toolkit==1.0.15 - ptyprocess==0.5.2 - pygments==2.2.0 - pyparsing==2.2.0 - python-dateutil==2.6.1 - pytz==2018.3 - scandir==1.7 - simplegeneric==0.8.1 - six==1.11.0 - subprocess32==3.2.7 - sympy==1.1.1 - traitlets==4.3.2 - wcwidth==0.1.7 - yt==3.4.1
Exporting a Virtual Environment in Anaconda to a YAML File
To export a virtual environment to a YAML file so that you or a third party can replicate your environment using Anaconda can be done using the following steps:
1.) Activate the Virtual environment you wish to export
[user@c001 ~ ]$ source activate newenv
2.)Export your active virtual environment using the following command
[user@c001 ~ ]$ conda env export > env.yml
Further Information
|