Difference between revisions of "Applications/Anaconda"

From HPC
Jump to: navigation , search
m
m (Python Packages)
 
(23 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
__TOC__
 
__TOC__
 +
 +
===Deprecation Notice===
 +
 +
'''IMPORTANT NOTICE''': If you're looking to set up a python environment for a particular program (eg. xcube, bwa etc) please use the [[Applications/Miniconda|miniconda virtual environments which are set up here]]. It is very likely in the future all python environments will be ''virtual environments'' using [[Applications/Miniconda|miniconda]].
 +
 
=== Application Details ===
 
=== Application Details ===
 +
 +
There are two types of python provided on Viper, if you looking for the [[Applications/Python|Original Python click here]].
 +
 
* Description: Python is a high-level interpreted programming language for general-purpose programming, supported by a large number of libraries for many tasks
 
* Description: Python is a high-level interpreted programming language for general-purpose programming, supported by a large number of libraries for many tasks
 
* Versions:  
 
* Versions:  
Line 6: Line 14:
 
** Anaconda v4.0 Python 2.7 and 3.5
 
** Anaconda v4.0 Python 2.7 and 3.5
 
** Anaconda v4.1.1 Python 2.7  
 
** Anaconda v4.1.1 Python 2.7  
 +
** '''Anaconda v4.6 miniconda python 3.7''' ([[Applications/Miniconda|Virtual Enviroments only]])
 
* Module names:  
 
* Module names:  
 
** python/2.7.11 python/3.5.1  
 
** python/2.7.11 python/3.5.1  
** python/anaconda/4.0/2.7 python/anaconda/4.0/3.5 python/anaconda/4.1.1/2.7
+
** python/anaconda/4.0/2.7 python/anaconda/4.0/3.5 python/anaconda/4.1.1/2.7 python/anaconda/4.6/miniconda/3.7 (VE)
 
* License: Free to use  
 
* License: Free to use  
 
** Python [https://en.wikipedia.org/wiki/Python_Software_Foundation_License Python Software Foundation License]
 
** Python [https://en.wikipedia.org/wiki/Python_Software_Foundation_License Python Software Foundation License]
Line 14: Line 23:
  
 
== Usage Examples ==
 
== Usage Examples ==
 +
 
=== Interactive ===
 
=== Interactive ===
 +
 
Interactive with command line:
 
Interactive with command line:
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
 
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;">
Line 45: Line 56:
 
<pre style="background-color: #C8C8C8; color: black; font-family: monospace, sans-serif;">
 
<pre style="background-color: #C8C8C8; color: black; font-family: monospace, sans-serif;">
 
#!/bin/bash
 
#!/bin/bash
#SBATCH -J PythonTest       # Job name, you can change it to whatever you want
+
#SBATCH -J PythonTest               # Job name, you can change it to whatever you want
#SBATCH -N 1               # Number of nodes  
+
#SBATCH -N 1                                 # Number of nodes  
#SBATCH -o %N.%j.out       # Standard output will be written here
+
#SBATCH --ntasks-per-node 1    # Number of cores per node
#SBATCH -e %N.%j.err       # Standard error will be written here
+
#SBATCH -o %N.%j.out               # Standard output will be written here
#SBATCH -p compute         # Slurm partition, where you want the job to be queued  
+
#SBATCH -e %N.%j.err                 # Standard error will be written here
 +
#SBATCH -p compute                 # Slurm partition, where you want the job to be queued  
 +
 
 
   
 
   
 
module purge
 
module purge
module add python/anaconda/4.0/2.7
+
module add python/anaconda/4.0/3.5
 
   
 
   
 
python PythonTest.py
 
python PythonTest.py
Line 80: Line 93:
 
| Enhanced interactive shell for Python, including tools for parallel computing
 
| Enhanced interactive shell for Python, including tools for parallel computing
 
| python/anaconda/4.0/2.7 python/anaconda/4.0/3.5 python/anaconda/4.1.1/2.7
 
| python/anaconda/4.0/2.7 python/anaconda/4.0/3.5 python/anaconda/4.1.1/2.7
 +
|-
 +
| Keras
 +
| style="white-space: nowrap;" | import keras
 +
| Wrapper for Neural net building library suitable for ML and DL research, see Tensorflow or Theano
 +
| python/anaconda/4.0/3.5
 
|-
 
|-
 
| Matplotlib
 
| Matplotlib
 
| style="white-space: nowrap;" | import matplotlib  
 
| style="white-space: nowrap;" | import matplotlib  
 
| Plotting library for Python and its numerical mathematics extension NumPy
 
| Plotting library for Python and its numerical mathematics extension NumPy
 +
To disable output to the screen use (''useful for nodes that cannot display these'')
 +
<pre>
 +
import matplotlib
 +
matplotlib.use('Agg')
 +
</pre>
 
| python/anaconda/4.0/2.7 python/anaconda/4.0/3.5 python/anaconda/4.1.1/2.7
 
| python/anaconda/4.0/2.7 python/anaconda/4.0/3.5 python/anaconda/4.1.1/2.7
 
|-
 
|-
Line 93: Line 116:
 
| OBITools
 
| OBITools
 
| style="white-space: nowrap;" | import obitools  
 
| style="white-space: nowrap;" | import obitools  
| Set of programs developed to simplify the manipulation of sequence files, mainly designed to help analyzing Next Generation Sequencer outputs in the context of DNA Metabarcoding
+
| Set of programs developed to simplify the manipulation of sequence files, mainly designed to help analyse Next Generation Sequencer outputs in the context of DNA Metabarcoding
 
| python/anaconda/4.0/2.7 python/anaconda/4.0/3.5
 
| python/anaconda/4.0/2.7 python/anaconda/4.0/3.5
 
|-
 
|-
| Theano
+
| Tensorflow
| style="white-space: nowrap;" | import theano
+
| style="white-space: nowrap;" | import tensorflow
| Numerical computation library, primarily designed for machine learning
+
| Neural net building library suitable for ML and DL research
| python/anaconda/4.0/2.7 python/anaconda/4.1.1/2.7
+
| python/anaconda/4.0/3.5
 
|-
 
|-
 +
 
| YT
 
| YT
 
| import yt
 
| import yt
| Package for analyzing and visualizing volumetric data
+
| Package for analysing and visualising volumetric data
 
| python/anaconda/4.0/2.7 python/anaconda/4.0/3.5 python/anaconda/4.1.1/2.7
 
| python/anaconda/4.0/2.7 python/anaconda/4.0/3.5 python/anaconda/4.1.1/2.7
 
|}
 
|}
  
=== Personal Python Library ===
+
== Further Information ==
  
== Further Information ==
+
* [[Programming/Python|Python Programming]]
* [https://www.hpc.hull.ac.uk/forum/viewforum.php?f=15 UoH Python forum]
+
* [[Applications/Miniconda|Miniconda python virtual environments]]
* [[Programming/Python|Wiki: Python Programming]]
 
* [[Main Page#General_Support|Wiki: General Support]]
 
* [[Main Page|Wiki: Home]]
 
  
{|
+
{{Modulepagenav}}
|style="width:5%; border-width: 0" | [[File:icon_home.png]]
 
|style="width:95%; border-width: 0" |
 
* [[Main_Page|Home]]
 
* [[Applications|Application support]]
 
* [[General|General]]
 
* [[Training|Training]]
 
* [[Programming|Programming support]]
 
|-
 
|}
 

Latest revision as of 09:55, 20 June 2024

Deprecation Notice

IMPORTANT NOTICE: If you're looking to set up a python environment for a particular program (eg. xcube, bwa etc) please use the miniconda virtual environments which are set up here. It is very likely in the future all python environments will be virtual environments using miniconda.

Application Details

There are two types of python provided on Viper, if you looking for the Original Python click here.

  • Description: Python is a high-level interpreted programming language for general-purpose programming, supported by a large number of libraries for many tasks
  • Versions:
    • Python 2.7.11 and 3.5.1
    • Anaconda v4.0 Python 2.7 and 3.5
    • Anaconda v4.1.1 Python 2.7
    • Anaconda v4.6 miniconda python 3.7 (Virtual Enviroments only)
  • Module names:
    • python/2.7.11 python/3.5.1
    • python/anaconda/4.0/2.7 python/anaconda/4.0/3.5 python/anaconda/4.1.1/2.7 python/anaconda/4.6/miniconda/3.7 (VE)
  • License: Free to use

Usage Examples

Interactive

Interactive with command line:

[username@c170 ~]$ module add python/anaconda/4.0/2.7
[username@c170 ~]$ python
Python 2.7.13 |Anaconda custom (64-bit)| (default, Dec 20 2016, 23:09:15)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>>

Interactive with command line with IPython:

[username@c170 ~]$ ipython
Python 2.7.13 |Anaconda custom (64-bit)| (default, Dec 20 2016, 23:09:15)
Type "copyright", "credits" or "license" for more information.

IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]:

Batch Submission

#!/bin/bash
#SBATCH -J PythonTest                # Job name, you can change it to whatever you want
#SBATCH -N 1                                  # Number of nodes 
#SBATCH --ntasks-per-node 1    # Number of cores per node
#SBATCH -o %N.%j.out                # Standard output will be written here
#SBATCH -e %N.%j.err                 # Standard error will be written here
#SBATCH -p compute                  # Slurm partition, where you want the job to be queued 

 
module purge
module add python/anaconda/4.0/3.5
 
python PythonTest.py

This is then submitted as follows:

[username@login01 ~]$ sbatch Pythontest.job
Submitted batch job 289522

Python Packages

Common packages such as SciPy and NumPy are available across all Python modules. Other commonly used packages are listed below:

Python Package Import Command Description Available in
Biopython import Bio Collection of Python tools for computational biology and bioinformatics, containing classes to represent biological sequences and sequence annotations, and it is able to read and write to a variety of file formats. Also allows for a programmatic means of accessing online databases of biological information, such as those at NCBI. Separate modules extend Biopython's capabilities to sequence alignment, protein structure, population genetics, phylogenetics, sequence motifs, and machine learning. python/anaconda/4.0/2.7 python/anaconda/4.0/3.5
IPython import IPython Enhanced interactive shell for Python, including tools for parallel computing python/anaconda/4.0/2.7 python/anaconda/4.0/3.5 python/anaconda/4.1.1/2.7
Keras import keras Wrapper for Neural net building library suitable for ML and DL research, see Tensorflow or Theano python/anaconda/4.0/3.5
Matplotlib import matplotlib Plotting library for Python and its numerical mathematics extension NumPy

To disable output to the screen use (useful for nodes that cannot display these)

import matplotlib
matplotlib.use('Agg')
python/anaconda/4.0/2.7 python/anaconda/4.0/3.5 python/anaconda/4.1.1/2.7
MPI4Py import mpi4py Provides bindings of the Message Passing Interface (MPI) standard for the Python programming language, allowing any Python program to exploit multiple processors. python/anaconda/4.0/3.5
OBITools import obitools Set of programs developed to simplify the manipulation of sequence files, mainly designed to help analyse Next Generation Sequencer outputs in the context of DNA Metabarcoding python/anaconda/4.0/2.7 python/anaconda/4.0/3.5
Tensorflow import tensorflow Neural net building library suitable for ML and DL research python/anaconda/4.0/3.5
YT import yt Package for analysing and visualising volumetric data python/anaconda/4.0/2.7 python/anaconda/4.0/3.5 python/anaconda/4.1.1/2.7

Further Information





Modules | Main Page | Further Topics