Training/Matlab-Viper-plugin
Contents
Matlab viper plugin
please note this page is still under development.
Introduction
There is a feature in Matlab that enables you to submit jobs to clusters directly from your Matlab client (normal desktop Matlab). we were able to develop Matlab-Viper plugin that utilise this feature and integrate Matlab with viper to enable all our Matlab users to submit some Matlab jobs to viper easily without the need to know much about HPC or Linux. Everything could be done from the normal desktop Matlab version that many users are already using. This guide should explain more on how to use viper through the Matlab viper plugin. In-general the toolbox that could utilise such integration is the Matlab parallel computing toolbox. we will try to explain the main concepts here and point you to some Mathworks tutorial that you could use and try on viper.
- If you don't have Matlab-Viper plugin installed and tested on your Matlab contact help@hull.ac.uk or viper@hull.ac.uk and we will help you to install it and configure it.
- This is only available for Matlab 2017a , we will try to support it for all the upcoming versions starting from 2017a. if you have a special use case that requires the usage of older version you could check the possibility of creating this plugin for it by emailing us : viper@hull.ac.uk
- This link https://uk.mathworks.com/products/ineligible_programs.html shows a list of Ineligible programs that couldn't run on Matlab distributed computing server through this plugin.
Matlab to Viper
Main concepts
Before we start we would like to introduce to you some basic concepts in order to be able to use this plugin efficiently.
This plugin works by connecting 2 components together:
- Client:
- The client could run on your laptop or desktop computer. This is where you will run Matlab commands and submit your jobs.
- Worker:
- The worker is and instance of Matlab distributed computing enviornement that runs on Viper in order to perform a calculation for one of the clients. you could submit batch jobs from your clients that runs some simulations in parallel using functions like parfor, you could also run normal Matlab code on it.
limitations
submitting a job to Viper has a a lot of overhead. if you are running a small computing task it might take even longer to run on the cluster as there are a lot of time will be lost in communication and transferring your data to Viper then in sometimes your job could be queued and start running in a later time. however in many other cases it will show you a speedup and as a minimum you will be able to free up your client from doing this computation.
Run Matlab Job
using parallel tool box, you can run your Matlab code on viper using 2 modes:
Batch job
- This is the main way to utilize the Matlab-viper-plugin. This submit your job to Viper queuing system (SLURM) and send back the results to your client once everything is done.
- Very useful for time consuming job that need hours or even days to finish. you will submit your job and you don't need to keep your Matlab running or even the whole computer could be turned off as all your simulation will be running on Viper.
- if you want to know more about the job life cycles and different stages https://uk.mathworks.com/help/distcomp/how-parallel-computing-products-run-a-job.html#f3-11033
- Create Matlab script
- this can be either a normal serial script or a parallel one that uses functions like parfor. if you have an exisiting script make sure that the following is done before running it:
- remove any parpool commands if any
- modify the file paths if any is used in your script. consider that all your files will move to the cluster and run *:there so if your script contains a filepath that exisits only on your client it will not be able to find it there.
- example:
- use
load ('INPUT_FILE_NAME')
- instead of
load ('/INPUT_DIRECTORY_NAME/INPUT_FILE_NAME')
- Batch command: This command begins an automated process that connects to the cluster, submits a job to the scheduler, and initializes MDCS. it will ask you to enter your user name and password. you must also make sure that you are using on the campus network or while using VPN
- For example if you submit a script to Viper with one input file:
viper = parcluster()
job = viper.batch('<your script>', 'pool', 9, 'AttachedFiles', '<name of input file>')
- Specify the name of your script without the “m” prefix. For instance use test, and not test.m
- The argument pool specified the number of workers you need for execution of the script. Note that the MATLAB adds one worker to the number of workers you specified in pool. So for instance if you specify 9 workers in the pool, then MATLAB will use 10 workers.
- The 4th Argument 'AttachedFiles' highlight that we are going to attache input file/files to this batch job. you can send multiple files by including them in braces. For Example
job = viper.batch('test', 'pool', 9, 'AttachedFiles', {input1, input2})
- When you finish working with this job and get all your outputs back. permanently delete its data and remove its reference from the workspace:
delete(job) clear job
Useful commands
below is some useful commands and its detailed offical documentation
| Training Area | Description |
| [getDebugLog ] | shows a details log that is very useful in debugging the job submission process |
| [Job Monitor] | enable you to monitor all your jobs and its status |
| [Parallel Computing Toolbox] | This contains all the details of the Parallel Computing toolbox. |
Parallel pool
- Another mode of operation that is easier if you are testing piece of code or running a small job that you expect to finish in few minutes.
- This requires a running connection to be in place between your computer and viper. at any point you shutdown your Matlab/Wifi or even your laptop everything will be lost and you will need to rerun your code/simulation.
Further Information