Difference between revisions of "Applications/Python"
From HPC
(→Creation of a Virtual Environment in Anaconda Using a Yaml File) |
m (→Python Virtual Environment) |
||
| (66 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
__TOC__ | __TOC__ | ||
| + | |||
=== Application Details === | === Application Details === | ||
* Description: Python is a high-level interpreted programming language for general-purpose programming, supported by a large number of libraries for many tasks | * Description: Python is a high-level interpreted programming language for general-purpose programming, supported by a large number of libraries for many tasks | ||
* Versions: Python 2.7.11 and 3.5.1 | * Versions: Python 2.7.11 and 3.5.1 | ||
| − | * Module names: python/2.7. | + | * Module names: python/anaconda/4.0/2.7, python/anaconda/4.0/3.5, python/anaconda/4.1.1/2.7 and python/anaconda/4.3.31/3.6-VE |
| + | * Additional module: python/anaconda/4.6/miniconda/3.7 and python/anaconda/202111/3.9 (used for virtual environments) | ||
* License: Free to use - [https://en.wikipedia.org/wiki/Python_Software_Foundation_License Python Software Foundation License] | * License: Free to use - [https://en.wikipedia.org/wiki/Python_Software_Foundation_License Python Software Foundation License] | ||
| + | == Introduction == | ||
| + | |||
| + | * Python is provided by the [[applications/Anaconda|Anaconda package]] too. | ||
| + | * Anaconda is the leading open data science platform powered by Python. | ||
| + | |||
| + | == Virtual Environments == | ||
| + | |||
| + | A Python virtual environment allows users to create a custom environment(s) in which they can have the packages, and versions of those packages that are required without having elevated user privileges. | ||
| + | |||
| + | We recommend the Virtualenv installation when you use a specialised package that would not be used by the wider HPC community. Virtualenv is a virtual Python environment isolated from other Python development, incapable of interfering with or being affected by other Python programs on the same HPC. During the Virtualenv installation process, you will install can install not only the additional package but all the dependencies that go with it. (This is actually pretty easy.) All in all, Virtualenv provides a safe and reliable mechanism for installing and running additional packages. | ||
| − | + | There are many benefits to using a virtual environment on a system. Here is a short non-exhaustive list of a few of the key benefits: | |
| − | + | * Control over packages and package versions. | |
| + | * An arbitrary number of virtual environments can be created for different tasks. | ||
| + | * Reproducible - A user can replicate a python environment on any system, so as to be able to run or re-run a particular task or job under the same conditions. | ||
| − | === Interactive === | + | |
| + | ===Creation of Virtual Environment Using Anaconda on Viper=== | ||
| + | |||
| + | * See [[Applications/Miniconda]] page for creating your own virtual environment. This is the preferred method for specialised python modules required such as biopython or TensorFlow for example. | ||
| + | |||
| + | |||
| + | ==Usage Examples== | ||
| + | |||
| + | === Interactive Session === | ||
Interactive with command line: | Interactive with command line: | ||
| Line 25: | Line 47: | ||
</pre> | </pre> | ||
| − | + | ipython with the command line | |
| + | |||
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;"> | <pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;"> | ||
[username@c170 ~]$ ipython | [username@c170 ~]$ ipython | ||
| Line 41: | Line 64: | ||
=== Batch Submission === | === Batch Submission === | ||
| + | |||
<pre style="background-color: #C8C8C8; color: black; font-family: monospace, sans-serif;"> | <pre style="background-color: #C8C8C8; color: black; font-family: monospace, sans-serif;"> | ||
#!/bin/bash | #!/bin/bash | ||
| Line 50: | Line 74: | ||
#SBATCH -p compute # Slurm partition, where you want the job to be queued | #SBATCH -p compute # Slurm partition, where you want the job to be queued | ||
| − | |||
module purge | module purge | ||
| − | module add python/3.5 | + | module add python/anaconda/4.0/3.5 |
python PythonTest.py | python PythonTest.py | ||
| Line 60: | Line 83: | ||
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;"> | <pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;"> | ||
[username@login01 ~]$ sbatch Pythontest.job | [username@login01 ~]$ sbatch Pythontest.job | ||
| − | Submitted batch job | + | Submitted batch job 2895122 |
</pre> | </pre> | ||
| − | == | + | ===Popular Python Packages=== |
| − | |||
| − | |||
| − | |||
| − | * | + | * TensorFlow |
| − | * | + | * Scikit-Learn |
| − | * | + | * Numpy |
| + | * PyTorch | ||
| + | * SciPy | ||
| + | * Matplotlib | ||
| + | * Pandas | ||
| − | === | + | ===Python Virtual Environment=== |
| − | + | This part refers to non-conda virtual environments. | |
| − | + | The creation of a virtual environment is done by executing the command venv: | |
| − | <pre | + | <pre> |
| − | + | python -m venv machinelearn | |
</pre> | </pre> | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | Running this command creates the target directory (creating any parent directories that don’t exist already) and places a pyvenv.cfg file in it with a home key pointing to the Python installation from which the command was run (a common name for the target directory is .venv). | |
| − | To | + | To activate |
| − | <pre | + | <pre> |
| − | + | source machinelearn/bin/activate | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
</pre> | </pre> | ||
| − | |||
| − | + | ==Next Steps== | |
| − | + | * [[Applications/Miniconda|Miniconda Python virtual environments]] | |
| − | + | * [[Applications/Anaconda|Anaconda Python]] | |
| − | * [[ | + | * [[Programming/Python|Python Programming]] |
| − | * [[ | ||
| − | * [[ | ||
| − | { | + | {{Modulepagenav}} |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
Latest revision as of 12:30, 17 February 2023
Contents
Application Details
- Description: Python is a high-level interpreted programming language for general-purpose programming, supported by a large number of libraries for many tasks
- Versions: Python 2.7.11 and 3.5.1
- Module names: python/anaconda/4.0/2.7, python/anaconda/4.0/3.5, python/anaconda/4.1.1/2.7 and python/anaconda/4.3.31/3.6-VE
- Additional module: python/anaconda/4.6/miniconda/3.7 and python/anaconda/202111/3.9 (used for virtual environments)
- License: Free to use - Python Software Foundation License
Introduction
- Python is provided by the Anaconda package too.
- Anaconda is the leading open data science platform powered by Python.
Virtual Environments
A Python virtual environment allows users to create a custom environment(s) in which they can have the packages, and versions of those packages that are required without having elevated user privileges.
We recommend the Virtualenv installation when you use a specialised package that would not be used by the wider HPC community. Virtualenv is a virtual Python environment isolated from other Python development, incapable of interfering with or being affected by other Python programs on the same HPC. During the Virtualenv installation process, you will install can install not only the additional package but all the dependencies that go with it. (This is actually pretty easy.) All in all, Virtualenv provides a safe and reliable mechanism for installing and running additional packages.
There are many benefits to using a virtual environment on a system. Here is a short non-exhaustive list of a few of the key benefits:
- Control over packages and package versions.
- An arbitrary number of virtual environments can be created for different tasks.
- Reproducible - A user can replicate a python environment on any system, so as to be able to run or re-run a particular task or job under the same conditions.
Creation of Virtual Environment Using Anaconda on Viper
- See Applications/Miniconda page for creating your own virtual environment. This is the preferred method for specialised python modules required such as biopython or TensorFlow for example.
Usage Examples
Interactive Session
Interactive with command line:
[username@c170 ~]$ module add python/anaconda/4.0/2.7 [username@c170 ~]$ python Python 2.7.5 (default, Nov 20 2015, 02:00:19) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>>
ipython with the command line
[username@c170 ~]$ ipython Python 2.7.13 |Anaconda custom (64-bit)| (default, Dec 20 2016, 23:09:15) Type "copyright", "credits" or "license" for more information. IPython 5.1.0 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details. In [1]:
Batch Submission
#!/bin/bash #SBATCH -J PythonTest # Job name, you can change it to whatever you want #SBATCH -N 1 # Number of nodes #SBATCH --ntasks-per-node 1 # Number of cores per node #SBATCH -o %N.%j.out # Standard output will be written here #SBATCH -e %N.%j.err # Standard error will be written here #SBATCH -p compute # Slurm partition, where you want the job to be queued module purge module add python/anaconda/4.0/3.5 python PythonTest.py
This is then submitted as follows:
[username@login01 ~]$ sbatch Pythontest.job Submitted batch job 2895122
Popular Python Packages
- TensorFlow
- Scikit-Learn
- Numpy
- PyTorch
- SciPy
- Matplotlib
- Pandas
Python Virtual Environment
This part refers to non-conda virtual environments.
The creation of a virtual environment is done by executing the command venv:
python -m venv machinelearn
Running this command creates the target directory (creating any parent directories that don’t exist already) and places a pyvenv.cfg file in it with a home key pointing to the Python installation from which the command was run (a common name for the target directory is .venv).
To activate
source machinelearn/bin/activate