Difference between revisions of "OpenHPC"
(→Resource Requirements) |
|||
| Line 17: | Line 17: | ||
|128GB (i.e. 128GB for exclusive use) | |128GB (i.e. 128GB for exclusive use) | ||
|} | |} | ||
| − | |||
| − | |||
If a task is terminated due to exceeding the requested amount of memory, you should see a message in your Slurm error log file, such as: | If a task is terminated due to exceeding the requested amount of memory, you should see a message in your Slurm error log file, such as: | ||
| Line 25: | Line 23: | ||
slurmstepd: error: Detected 1 oom-kill event(s) in StepId=319.batch. Some of your processes may have been killed by the cgroup out-of-memory handler. | slurmstepd: error: Detected 1 oom-kill event(s) in StepId=319.batch. Some of your processes may have been killed by the cgroup out-of-memory handler. | ||
</pre> | </pre> | ||
| − | |||
== Job Emails == | == Job Emails == | ||
Revision as of 17:08, 6 January 2022
Contents
Introduction
Slurm upgrade
Resource Requirements
In order to reduce issues with tasks using more resource (CPU or RAM) than they should, Slurm will now uses Linux cgroups to more closely monitor resource use of tasks and will limit (CPU) or terminate (RAM) tasks exceeding what they should use. In particular this means it is important to request the correct amount of RAM needed for a task.
| Default (no specific memory request) | approx 4GB (i.e. 128 GB RAM / 28 cores) |
| #SBATCH --mem=40G | 40GB (i.e. specific amount) |
| #SBATCH --exclusive | 128GB (i.e. 128GB for exclusive use) |
If a task is terminated due to exceeding the requested amount of memory, you should see a message in your Slurm error log file, such as:
slurmstepd: error: Detected 1 oom-kill event(s) in StepId=319.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.
Job Emails
It is now possible to get emails alerts when certain event types occur using Slurms built in --mail-type SBATCH directive support.
The most commonly used valid type values are as follows (multiple type values may be specified in a comma separated list):
- NONE (the default if you don't set --mait-type
- BEGIN
- END
- FAIL
- REQUEUE
- ALL (equivalent to BEGIN, END, FAIL, INVALID_DEPEND, REQUEUE, and STAGE_OUT)
- INVALID_DEPEND (dependency never satisfied)
- TIME_LIMIT, TIME_LIMIT_90 (reached 90 percent of time limit), TIME_LIMIT_80 (reached 80 percent of time limit), TIME_LIMIT_50 (reached 50 percent of time limit) * ARRAY_TASKS (sends emails for each array task otherwise job BEGIN, END and FAIL apply to a job array as a whole rather than generating individual email messages for each task in the job array).
The user to be notified is indicated with --mail-user, however only @hull.ac.uk email addresses are valid.
If you want to be alerted when your task completes, if it advised to use #SBATCH --mail-type=END,FAIL to catch if a job finishes cleanly or if it finishes due to an error.
An example of a completion email is shown below:
Slurm Information
It is now possible to check the details of a job submission script used to submit a job. This is done using sacct -B -j <jobnumber> e.g.:
$ sacct -B -j 317 Batch Script for 317 -------------------------------------------------------------------------------- #!/bin/bash #SBATCH -J jobsubmissionfile #SBATCH -n 1 #SBATCH -o slurm-%j.out #SBATCH -e slurm-%j.out #SBATCH -p compute #SBATCH --exclusive #SBATCH --time=1-00:00:00 #SBATCH --mail-type=END,FAIL #SBATCH --mail-user=<your Hull email address> echo "This is my submission script" sleep 10