Difference between revisions of "General/Modules"
From HPC
m |
(→module avail) |
||
| (15 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
== Introduction == | == Introduction == | ||
| + | |||
Viper uses a module environment to provide access to applications, tools and libraries. Modules allow us to provide multiple versions of applications and libraries without causing conflicts. By loading a module, your environment is configured to allow you to use your chosen application by setting required paths and environment variables. Modules can be loaded and unloaded dynamically, giving you full control over your working environment. | Viper uses a module environment to provide access to applications, tools and libraries. Modules allow us to provide multiple versions of applications and libraries without causing conflicts. By loading a module, your environment is configured to allow you to use your chosen application by setting required paths and environment variables. Modules can be loaded and unloaded dynamically, giving you full control over your working environment. | ||
=== Module command === | === Module command === | ||
| + | |||
The following commands allow you to find out what modules are available on Viper, find out more information about them and control which modules you load. | The following commands allow you to find out what modules are available on Viper, find out more information about them and control which modules you load. | ||
==== module avail ==== | ==== module avail ==== | ||
| + | |||
To see a full list of applications and libraries that are available to use on Viper, you should run '''module avail''' as shown below (showing only a few of the hundreds of modules that are available): | To see a full list of applications and libraries that are available to use on Viper, you should run '''module avail''' as shown below (showing only a few of the hundreds of modules that are available): | ||
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;"> | <pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;"> | ||
[username@login01 ~]$ module avail | [username@login01 ~]$ module avail | ||
| − | ------------------------------ /usr/share/Modules/modulefiles ------------------------------ | + | --------------------------------------------- /usr/share/Modules/modulefiles --------------------------------------------------------------------------------------- |
dot module-git module-info modules null use.own | dot module-git module-info modules null use.own | ||
| − | ------------------------------ /trinity/clustervision/CentOS/7/modulefiles ------------------------------ | + | --------------------------------------------- /trinity/clustervision/CentOS/7/modulefiles/apps ------------------------------------------------------------------------------ |
| − | abyss/1.5.2/gcc- | + | abyss/1.5.2/gcc-5.2.0 centrifuge/1.0.4 hyphy/2.2.7/gcc-6.3.0/openmpi-2.0.2 orca/4.0.0 STAR/2.6.1 |
| − | + | albacore/1.2.6 cp2k/3.0 hypre/intel/2.10.0b orca/4.0.1 starccm+/11.02.010 | |
| − | ansys/v170 | + | ansys/v170 cp2k/6.1.0/gcc-7.3.0/intelmpi-2018 idl/8.4 orca/4.1.0/openmpi212 starccm+/12.04.011 |
| − | ansys/v172 | + | ansys/v172 cp2k/6.1.0/gcc-7.3.0/openmpi-3.0.0 image-magic/7.0.7/mpich-3.2/gcc-5.2.0 orca/4.1.0/openmpi313 starccm+/12.06.011 |
| + | |||
| + | |||
</pre> | </pre> | ||
| + | |||
If you want to use a specific application, you can find which versions are available by running '''module avail modulename''', e.g.: | If you want to use a specific application, you can find which versions are available by running '''module avail modulename''', e.g.: | ||
| Line 24: | Line 30: | ||
[username@login01 ~]$ module avail gcc | [username@login01 ~]$ module avail gcc | ||
| − | ------------------------------ /trinity/clustervision/CentOS/7/modulefiles ------------------------------ | + | -------------------------------------------- /trinity/clustervision/CentOS/7/modulefiles/development --------------------------------------------------------------------------- |
| − | gcc/4.9.3(default) gcc/5.2.0 gcc/6.3.0 gcc/ | + | gcc/4.9.3(default) gcc/5.2.0 gcc/6.3.0 gcc/7.3.0 gcc/8.2.0 |
| + | |||
</pre> | </pre> | ||
| − | In this case the (default) indicates that '''gcc/4.9.3''' is the default GCC module. | + | In this case, the (default) indicates that '''gcc/4.9.3''' is the default GCC module. |
==== module show ==== | ==== module show ==== | ||
| + | |||
To find out what a module file does, run '''module show modulename'''. This will show you brief information about what the module will do (this information is being updated for existing modules to show more relevant information), along with how the module will update your environment: | To find out what a module file does, run '''module show modulename'''. This will show you brief information about what the module will do (this information is being updated for existing modules to show more relevant information), along with how the module will update your environment: | ||
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;"> | <pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;"> | ||
| − | [username@login01 ~]$ module show abyss/1.5.2/gcc- | + | [username@login01 ~]$ module show abyss/1.5.2/gcc-5.2.0 |
------------------------------------------------------------------- | ------------------------------------------------------------------- | ||
| − | /trinity/clustervision/CentOS/7/modulefiles/abyss/1.5.2/gcc- | + | /trinity/clustervision/CentOS/7/modulefiles/abyss/1.5.2/gcc-5.2.0 |
module-whatis ABySS is a de novo sequence assembler intended for short paired-end reads and large genomes | module-whatis ABySS is a de novo sequence assembler intended for short paired-end reads and large genomes | ||
| − | prepend-path PATH /trinity/clustervision/CentOS/7/apps/abyss/1.5.2/gcc- | + | prepend-path PATH /trinity/clustervision/CentOS/7/apps/abyss/1.5.2/gcc-5.2.0/bin |
| − | prepend-path LD_LIBRARY_PATH /trinity/clustervision/CentOS/7/apps/abyss/1.5.2/gcc- | + | prepend-path LD_LIBRARY_PATH /trinity/clustervision/CentOS/7/apps/abyss/1.5.2/gcc-5.2.0/lib |
| − | prepend-path C_INCLUDE_PATH /trinity/clustervision/CentOS/7/apps/abyss/1.5.2/gcc- | + | prepend-path C_INCLUDE_PATH /trinity/clustervision/CentOS/7/apps/abyss/1.5.2/gcc-5.2.0/include |
| − | prepend-path PKG_CONFIG_PATH /trinity/clustervision/CentOS/7/apps/abyss/1.5.2/gcc- | + | prepend-path PKG_CONFIG_PATH /trinity/clustervision/CentOS/7/apps/abyss/1.5.2/gcc-5.2.0/lib/pkgconfig |
------------------------------------------------------------------- | ------------------------------------------------------------------- | ||
</pre> | </pre> | ||
==== module add ==== | ==== module add ==== | ||
| + | |||
To add a module you use '''module add modulename'''. In most cases, running this command will not display anything and will just return you back to the Linux command prompt. | To add a module you use '''module add modulename'''. In most cases, running this command will not display anything and will just return you back to the Linux command prompt. | ||
| Line 56: | Line 65: | ||
<pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;"> | <pre style="background-color: #000000; color: white; border: 2px solid black; font-family: monospace, sans-serif;"> | ||
| − | [username@login01 ~]$ module add gcc/ | + | [username@login01 ~]$ module add gcc/8.2.0 |
</pre> | </pre> | ||
| − | Without putting a version in, you will either load the highest alphanumeric version | + | Without putting a version in, you will either load the highest alphanumeric version or the default if one is set. |
Note: '''module load''' and '''module add''' are the same. | Note: '''module load''' and '''module add''' are the same. | ||
| Line 68: | Line 77: | ||
[username@login01 ~]$ module list | [username@login01 ~]$ module list | ||
Currently Loaded Modulefiles: | Currently Loaded Modulefiles: | ||
| − | 1) gcc/ | + | 1) gcc/8.2.0 |
</pre> | </pre> | ||
==== module delete ==== | ==== module delete ==== | ||
| − | To remove a loaded module and all environment configurations that module has set, run '''module delete modulename'''. Running this command will not display anything and will just return you back to the Linux command prompt | + | |
| + | To remove a loaded module and all environment configurations that the module has set, run '''module delete modulename'''. Running this command will not display anything and will just return you back to the Linux command prompt | ||
Note: module delete, module unload, module del and module rm are the same | Note: module delete, module unload, module del and module rm are the same | ||
==== module purge ==== | ==== module purge ==== | ||
| + | |||
To remove all loaded modules and return your environment to are vanilla state, run '''module purge'''. Running this command will not display anything and will just return you back to the Linux command prompt | To remove all loaded modules and return your environment to are vanilla state, run '''module purge'''. Running this command will not display anything and will just return you back to the Linux command prompt | ||
==== module whatis ==== | ==== module whatis ==== | ||
| + | |||
To find out what modules are available with information about what each application does, run '''module whatis''', for example (selected output): | To find out what modules are available with information about what each application does, run '''module whatis''', for example (selected output): | ||
| Line 85: | Line 97: | ||
[username@login01 ~]$ module whatis | [username@login01 ~]$ module whatis | ||
STAR/2016-10-27 : Spliced Transcripts Alignment to a Reference | STAR/2016-10-27 : Spliced Transcripts Alignment to a Reference | ||
| − | abyss/1.5.2/gcc- | + | abyss/1.5.2/gcc-5.2.0: ABySS is a de novo sequence assembler intended for short paired-end reads and large genomes |
crop/1.33/gcc-6.3.0 : Crop clustering approach to find common taxonomic units in large datasets | crop/1.33/gcc-6.3.0 : Crop clustering approach to find common taxonomic units in large datasets | ||
discovardenovo/52488 : DISCOVAR de novo – large genome assembler | discovardenovo/52488 : DISCOVAR de novo – large genome assembler | ||
| Line 97: | Line 109: | ||
=== Modulefiles === | === Modulefiles === | ||
==== Modulefile actions ==== | ==== Modulefile actions ==== | ||
| + | |||
Typically modulefiles instruct the module command to alter or set shell environment variables such as paths and flags, to provide access to executables or libraries. | Typically modulefiles instruct the module command to alter or set shell environment variables such as paths and flags, to provide access to executables or libraries. | ||
| Line 122: | Line 135: | ||
</pre> | </pre> | ||
| − | Other common options set in a | + | Other common options set in a module file include the following: |
* Paths such as LIBRARY_PATH, CPATH, LD_RUN_PATH | * Paths such as LIBRARY_PATH, CPATH, LD_RUN_PATH | ||
* Compiler flags such as CFLAGS, CPPFLAGS, FCFLAGS and LDFLAGS | * Compiler flags such as CFLAGS, CPPFLAGS, FCFLAGS and LDFLAGS | ||
| − | * Application specific paths or options such as MPI_HOME or MPI_RUN | + | * Application-specific paths or options such as MPI_HOME or MPI_RUN |
* Manual page location via the MANPATH | * Manual page location via the MANPATH | ||
* Other modules to load to meet dependencies | * Other modules to load to meet dependencies | ||
| Line 133: | Line 146: | ||
==== Modulefile naming convention ==== | ==== Modulefile naming convention ==== | ||
| − | |||
| − | However, due to the specific combinations of some of the tools required to build certain applications which may be relevant to those using the application, a new naming convention is being introduced. These modulefiles will take the form application/version/ followed by information about the compiler module used to build the application (and possibly the version of MPI), for example: | + | Most modulefiles take the form applicationname/version, for example '''gcc/5.2.0''' for the GNU Compiler Collection version 5.2.0. There are also other versions of GCC, for example '''gcc/5.2.0''', '''gcc/6.3.0'' and '''gcc/7.3.0'''. A similar example is '''matlab/2016a''' |
| + | |||
| + | However, due to the specific combinations of some of the tools required to build certain applications which may be relevant to those using the application, a new naming convention is being introduced. These modulefiles will take the form of application/version/ followed by information about the compiler module used to build the application (and possibly the version of MPI), for example: | ||
* '''crop/1.33/gcc-6.3.0''' | * '''crop/1.33/gcc-6.3.0''' | ||
| Line 142: | Line 156: | ||
* '''openmpi/1.10.5/intel-2017''' | * '''openmpi/1.10.5/intel-2017''' | ||
| − | If a piece of software doesn't have a specific version number, for example when retrieved from | + | If a piece of software doesn't have a specific version number, for example when retrieved from ''GitHub'', then the version will often be replaced by the date the application was built, for example '''sparsehash/2016-12-21/gcc-4.9.3''' |
| + | |||
| + | ====Test modulefiles==== | ||
| − | + | While applications are being installed and tested, they may be put in a test area and not directly available to users. This is to stop people from using modules that may not be correctly configured or are subject to change while testing the installations. To see the modules that are in testing you need to run '''module add test-modules''' before running '''module avail''' | |
| − | While applications are being installed and tested, they may be put in a test area and not directly available to users. This is to stop people using modules | ||
=== Module Tips and Tricks === | === Module Tips and Tricks === | ||
* Loading a module will be active for that session only | * Loading a module will be active for that session only | ||
| − | * Multiple modules can be loaded in one command, for example '''module add gcc/ | + | * Multiple modules can be loaded in one command, for example '''module add gcc/5.2.0 sparsehash/2016-12-21/gcc-5.2.0''' |
| − | * | + | * '''module add''' commands should be included in job submission scripts |
| − | * The module add command support tab completion | + | * The '''module add''' command support tab completion |
* You can create private modules in your own home directory | * You can create private modules in your own home directory | ||
| − | * | + | * The '''module add''' commands are recorded to track application usage |
| + | |||
| + | |||
| + | ==Navigation== | ||
* [[Main_Page|Home]] | * [[Main_Page|Home]] | ||
| − | * [[Applications|Application support]] | + | * [[Applications|Application support]] * |
| − | * | ||
| − | |||
* [[General|General]] | * [[General|General]] | ||
| − | * [[ | + | * [[Programming|Programming support]] |
Latest revision as of 13:33, 10 November 2022
Contents
Introduction
Viper uses a module environment to provide access to applications, tools and libraries. Modules allow us to provide multiple versions of applications and libraries without causing conflicts. By loading a module, your environment is configured to allow you to use your chosen application by setting required paths and environment variables. Modules can be loaded and unloaded dynamically, giving you full control over your working environment.
Module command
The following commands allow you to find out what modules are available on Viper, find out more information about them and control which modules you load.
module avail
To see a full list of applications and libraries that are available to use on Viper, you should run module avail as shown below (showing only a few of the hundreds of modules that are available):
[username@login01 ~]$ module avail --------------------------------------------- /usr/share/Modules/modulefiles --------------------------------------------------------------------------------------- dot module-git module-info modules null use.own --------------------------------------------- /trinity/clustervision/CentOS/7/modulefiles/apps ------------------------------------------------------------------------------ abyss/1.5.2/gcc-5.2.0 centrifuge/1.0.4 hyphy/2.2.7/gcc-6.3.0/openmpi-2.0.2 orca/4.0.0 STAR/2.6.1 albacore/1.2.6 cp2k/3.0 hypre/intel/2.10.0b orca/4.0.1 starccm+/11.02.010 ansys/v170 cp2k/6.1.0/gcc-7.3.0/intelmpi-2018 idl/8.4 orca/4.1.0/openmpi212 starccm+/12.04.011 ansys/v172 cp2k/6.1.0/gcc-7.3.0/openmpi-3.0.0 image-magic/7.0.7/mpich-3.2/gcc-5.2.0 orca/4.1.0/openmpi313 starccm+/12.06.011
If you want to use a specific application, you can find which versions are available by running module avail modulename, e.g.:
[username@login01 ~]$ module avail gcc -------------------------------------------- /trinity/clustervision/CentOS/7/modulefiles/development --------------------------------------------------------------------------- gcc/4.9.3(default) gcc/5.2.0 gcc/6.3.0 gcc/7.3.0 gcc/8.2.0
In this case, the (default) indicates that gcc/4.9.3 is the default GCC module.
module show
To find out what a module file does, run module show modulename. This will show you brief information about what the module will do (this information is being updated for existing modules to show more relevant information), along with how the module will update your environment:
[username@login01 ~]$ module show abyss/1.5.2/gcc-5.2.0 ------------------------------------------------------------------- /trinity/clustervision/CentOS/7/modulefiles/abyss/1.5.2/gcc-5.2.0 module-whatis ABySS is a de novo sequence assembler intended for short paired-end reads and large genomes prepend-path PATH /trinity/clustervision/CentOS/7/apps/abyss/1.5.2/gcc-5.2.0/bin prepend-path LD_LIBRARY_PATH /trinity/clustervision/CentOS/7/apps/abyss/1.5.2/gcc-5.2.0/lib prepend-path C_INCLUDE_PATH /trinity/clustervision/CentOS/7/apps/abyss/1.5.2/gcc-5.2.0/include prepend-path PKG_CONFIG_PATH /trinity/clustervision/CentOS/7/apps/abyss/1.5.2/gcc-5.2.0/lib/pkgconfig -------------------------------------------------------------------
module add
To add a module you use module add modulename. In most cases, running this command will not display anything and will just return you back to the Linux command prompt.
[username@login01 ~]$ module add gcc
If you want to add a specific version of a module then you should include the version number, for example:
[username@login01 ~]$ module add gcc/8.2.0
Without putting a version in, you will either load the highest alphanumeric version or the default if one is set.
Note: module load and module add are the same.
module list
To see what modules you have loaded, run module list
[username@login01 ~]$ module list Currently Loaded Modulefiles: 1) gcc/8.2.0
module delete
To remove a loaded module and all environment configurations that the module has set, run module delete modulename. Running this command will not display anything and will just return you back to the Linux command prompt
Note: module delete, module unload, module del and module rm are the same
module purge
To remove all loaded modules and return your environment to are vanilla state, run module purge. Running this command will not display anything and will just return you back to the Linux command prompt
module whatis
To find out what modules are available with information about what each application does, run module whatis, for example (selected output):
[username@login01 ~]$ module whatis STAR/2016-10-27 : Spliced Transcripts Alignment to a Reference abyss/1.5.2/gcc-5.2.0: ABySS is a de novo sequence assembler intended for short paired-end reads and large genomes crop/1.33/gcc-6.3.0 : Crop clustering approach to find common taxonomic units in large datasets discovardenovo/52488 : DISCOVAR de novo – large genome assembler expat/gcc/2.2.0 : Expat XML parser ssw/2016-08-08 : SSW (NASA SolarSoft) test-modules : Adds modules under development to your module avail list
Note: This information is currently being updated for existing modules.
Modulefiles
Modulefile actions
Typically modulefiles instruct the module command to alter or set shell environment variables such as paths and flags, to provide access to executables or libraries.
[username@login01 ~]$ module show openmpi/1.10.5/gcc-6.3.0 ------------------------------------------------------------------- /trinity/clustervision/CentOS/7/modulefiles/openmpi/1.10.5/gcc-6.3.0: module-whatis loads the openmpi/1.10.5/gcc-6.3.0 environment module load gcc/6.3.0 conflict mpich conflict openmpi conflict mvapich2 conflict intel prepend-path PATH /trinity/clustervision/CentOS/7/apps/openmpi/1.10.5/gcc-6.3.0/bin prepend-path LD_LIBRARY_PATH /trinity/clustervision/CentOS/7/apps/openmpi/1.10.5/gcc-6.3.0/lib prepend-path LIBRARY_PATH /trinity/clustervision/CentOS/7/apps/openmpi/1.10.5/gcc-6.3.0/lib prepend-path INCLUDE /trinity/clustervision/CentOS/7/apps/openmpi/1.10.5/gcc-6.3.0/include prepend-path C_INCLUDE_PATH /trinity/clustervision/CentOS/7/apps/openmpi/1.10.5/gcc-6.3.0/include prepend-path MANPATH /trinity/clustervision/CentOS/7/apps/openmpi/1.10.5/gcc-6.3.0/share/man setenv MPI_HOME /trinity/clustervision/CentOS/7/apps/openmpi/1.10.5/gcc-6.3.0 setenv MPI_RUN /trinity/clustervision/CentOS/7/apps/openmpi/1.10.5/gcc-6.3.0/bin/mpirun setenv OMPI_HOME /trinity/clustervision/CentOS/7/apps/openmpi/1.10.5/gcc-6.3.0 -------------------------------------------------------------------
Other common options set in a module file include the following:
- Paths such as LIBRARY_PATH, CPATH, LD_RUN_PATH
- Compiler flags such as CFLAGS, CPPFLAGS, FCFLAGS and LDFLAGS
- Application-specific paths or options such as MPI_HOME or MPI_RUN
- Manual page location via the MANPATH
- Other modules to load to meet dependencies
- Any conflicting modules that can't be loaded together
- License file locations, for example for Intel Compiler
Modulefile naming convention
Most modulefiles take the form applicationname/version, for example gcc/5.2.0' for the GNU Compiler Collection version 5.2.0. There are also other versions of GCC, for example gcc/5.2.0, gcc/6.3.0 and gcc/7.3.0. A similar example is matlab/2016a
However, due to the specific combinations of some of the tools required to build certain applications which may be relevant to those using the application, a new naming convention is being introduced. These modulefiles will take the form of application/version/ followed by information about the compiler module used to build the application (and possibly the version of MPI), for example:
- crop/1.33/gcc-6.3.0
- openmpi/1.10.5/gcc-5.2.0
- openmpi/1.10.5/gcc-6.3.0
- openmpi/1.10.5/intel-2017
If a piece of software doesn't have a specific version number, for example when retrieved from GitHub, then the version will often be replaced by the date the application was built, for example sparsehash/2016-12-21/gcc-4.9.3
Test modulefiles
While applications are being installed and tested, they may be put in a test area and not directly available to users. This is to stop people from using modules that may not be correctly configured or are subject to change while testing the installations. To see the modules that are in testing you need to run module add test-modules before running module avail
Module Tips and Tricks
- Loading a module will be active for that session only
- Multiple modules can be loaded in one command, for example module add gcc/5.2.0 sparsehash/2016-12-21/gcc-5.2.0
- module add commands should be included in job submission scripts
- The module add command support tab completion
- You can create private modules in your own home directory
- The module add commands are recorded to track application usage