Difference between revisions of "Training/Matlab-Viper-plugin"

From HPC
Jump to: navigation , search
m
Line 1: Line 1:
=Matlab viper plugin=
+
__TOC__
please note this page is still under development.  
+
 
 +
=Matlab Viper plugin=
 +
 
 +
Please note this page is in pre-release status and may be changed at any time.
 +
 
 
==Introduction==
 
==Introduction==
There is a feature in Matlab that enables you to submit jobs to clusters directly from your Matlab client (normal desktop Matlab). we were able to develop Matlab-Viper plug-in that utilise this feature and integrate Matlab with viper to enable all our Matlab users to submit some Matlab jobs to viper easily without the need to know much about HPC or Linux. Everything could be done from the normal desktop Matlab version that many users are already using.  
+
There is a feature in Matlab that enables you to submit jobs to clusters directly from your Matlab client (normal desktop Matlab). we were able to develop a Matlab-Viper plug-in that utilizes this feature and integrates Matlab with viper to enable all our Matlab users to submit some Matlab jobs to viper easily without the need to know much about HPC or Linux. Everything could be done from the normal desktop Matlab version that many users are already using.  
This guide should explain more on how to use viper through the Matlab viper plug-in. In-general the toolbox that could utilise such integration is the Matlab parallel computing toolbox. we will try to explain the main concepts here and point you to some MathWorks tutorial that you could use and try on viper.
+
This guide should explain more on how to use viper through the Matlab viper plug-in. In general, the toolbox that could utilize such integration is the Matlab parallel computing toolbox. we will try to explain the main concepts here and point you to some MathWorks tutorials that you could use and try on viper.
  
 
*If you don't have Matlab-Viper plug-in installed and tested on your Matlab contact [https://support.hull.ac.uk/ Topdesk] we will help you to install it and configure it.
 
*If you don't have Matlab-Viper plug-in installed and tested on your Matlab contact [https://support.hull.ac.uk/ Topdesk] we will help you to install it and configure it.
*This is only available for Matlab 2017a , we will try to support it for all the upcoming versions starting from 2017a. if you have a special use case that requires the usage of older version you could check the possibility of creating this plug-in for it contact [https://support.hull.ac.uk/ Topdesk] .
+
*This is only available for Matlab 2017a , we will try to support it for all the upcoming versions starting from 2017a. if you have a special use case that requires the usage of an older version you could check the possibility of creating this plug-in for it contact [https://support.hull.ac.uk/ Topdesk] .
 
* This link https://uk.mathworks.com/products/ineligible_programs.html  shows a list of Ineligible programs that couldn't run on Matlab distributed computing server through this plug-in.
 
* This link https://uk.mathworks.com/products/ineligible_programs.html  shows a list of Ineligible programs that couldn't run on Matlab distributed computing server through this plug-in.
  
Line 15: Line 19:
 
==== This plugin works by connecting 2 components together:====
 
==== This plugin works by connecting 2 components together:====
 
#'''Client:'''
 
#'''Client:'''
#:The client could run on your laptop or desktop computer. This is where you will run Matlab commands and submit your jobs.
+
#: The client could run on your laptop or desktop computer. This is where you will run Matlab commands and submit your jobs.
 
#'''Worker:'''
 
#'''Worker:'''
#:The worker is and instance of Matlab distributed computing environment that runs on Viper in order to perform a calculation for one of the clients. you could submit batch jobs from your clients that runs some simulations in parallel using functions like ''parfor'', you could also run normal Matlab code on it.
+
#: The worker is an instance of Matlab distributed computing environment that runs on Viper in order to perform a calculation for one of the clients. you could submit batch jobs from your clients that run some simulations in parallel using functions like ''parfor'', you could also run normal Matlab code on it.
  
 
==== limitations ====
 
==== limitations ====
submitting a job to Viper has a a lot of overhead. if you are running a small computing task it might take even longer to run on the cluster as there are a lot of time will be lost in communication and transferring your data to Viper then in sometimes your job could be queued and start running in a later time. however in many other cases it will show you a speed-up and as a minimum you will be able to free up your client from doing this computation.
+
 
 +
Submitting a job to Viper has a lot of overhead. If you are running a small computing task it might take even longer to run on the cluster. However, in many other cases, it will show you a speed-up and as a minimum, you will be able to free up your client from doing this computation.
 +
 
 
===Run Matlab Job===
 
===Run Matlab Job===
using parallel tool box, you can run your Matlab code on viper using 2 modes:
+
 
 +
Using a parallel toolbox, you can run your Matlab code on viper using 2 modes:
  
 
====Batch job====
 
====Batch job====
  
*This is the main way to utilize the Matlab-viper-plugin. This submit your job to Viper queuing system (SLURM) and send back the results to your client once everything is done.   
+
* This is the main way to utilize the Matlab-viper-plugin. This submits your job to Viper queuing system (SLURM) and sends back the results to your client once everything is done.   
*Very useful for time consuming job that need hours or even days to finish. you will submit your job and you don't need to keep your Matlab running or even the whole computer could be turned off as all your simulation will be running on Viper.
+
* Very useful for a time-consuming job that needs hours or even days to finish. you will submit your job and you don't need to keep your Matlab running or even the whole computer could be turned off as all your simulation will be running on Viper.
* if you want to know more about the job life cycles and different stages https://uk.mathworks.com/help/distcomp/how-parallel-computing-products-run-a-job.html#f3-11033  
+
* If you want to know more about the job life cycles and different stages https://uk.mathworks.com/help/distcomp/how-parallel-computing-products-run-a-job.html#f3-11033  
 
*'''Create Matlab script'''
 
*'''Create Matlab script'''
*:this can be either a normal serial script or a parallel one that uses functions like parfor. if you have an existing script make sure that the following is done before running it:
+
*:This can be either a normal serial script or a parallel one that uses functions like parfor. if you have an existing script make sure that the following is done before running it:
** remove any parpool commands if any
+
** Remove any parpool commands if any
**modify the file paths if any is used in your script. consider that all your files will move to the cluster and run *:there so if your script contains a filepath that exisits only on your client it will not be able to find it there.  
+
** Modify the file paths if any is used in your script. consider that all your files will move to the cluster and run *:there so if your script contains a filepath that exisits only on your client it will not be able to find it there.  
 
*:example:  
 
*:example:  
 
*:use  
 
*:use  
Line 62: Line 69:
 
|-
 
|-
 
| [[https://uk.mathworks.com/help/distcomp/job-monitor.html/XQuartz| Job Monitor]]  
 
| [[https://uk.mathworks.com/help/distcomp/job-monitor.html/XQuartz| Job Monitor]]  
| enable you to monitor all your jobs and its status
+
| enable you to monitor all your jobs and their status
 
|-
 
|-
 
| [[https://uk.mathworks.com/help/distcomp/index.html| Parallel Computing Toolbox]]
 
| [[https://uk.mathworks.com/help/distcomp/index.html| Parallel Computing Toolbox]]
Line 70: Line 77:
  
 
====Parallel pool====
 
====Parallel pool====
* Another mode of operation that is easier if you are testing piece of code or running a small job that you expect to finish in few minutes.  
+
* Another mode of operation that is easier if you are testing a piece of code or running a small job that you expect to finish in few minutes.  
* This requires a running connection to be in place between your computer and viper. at any point you shutdown your Matlab/Wifi or even your laptop everything will be lost and you will need to rerun your code/simulation.
+
* This requires a running connection to be in place between your computer and viper. at any point you shut down your Matlab/Wifi or even your laptop everything will be lost and you will need to rerun your code/simulation.
  
 
== Further Information ==
 
== Further Information ==

Revision as of 13:06, 24 May 2021

Matlab Viper plugin

Please note this page is in pre-release status and may be changed at any time.

Introduction

There is a feature in Matlab that enables you to submit jobs to clusters directly from your Matlab client (normal desktop Matlab). we were able to develop a Matlab-Viper plug-in that utilizes this feature and integrates Matlab with viper to enable all our Matlab users to submit some Matlab jobs to viper easily without the need to know much about HPC or Linux. Everything could be done from the normal desktop Matlab version that many users are already using. This guide should explain more on how to use viper through the Matlab viper plug-in. In general, the toolbox that could utilize such integration is the Matlab parallel computing toolbox. we will try to explain the main concepts here and point you to some MathWorks tutorials that you could use and try on viper.

  • If you don't have Matlab-Viper plug-in installed and tested on your Matlab contact Topdesk we will help you to install it and configure it.
  • This is only available for Matlab 2017a , we will try to support it for all the upcoming versions starting from 2017a. if you have a special use case that requires the usage of an older version you could check the possibility of creating this plug-in for it contact Topdesk .
  • This link https://uk.mathworks.com/products/ineligible_programs.html shows a list of Ineligible programs that couldn't run on Matlab distributed computing server through this plug-in.

Matlab to Viper

Main concepts

Before we start we would like to introduce to you some basic concepts in order to be able to use this plug-in efficiently.

This plugin works by connecting 2 components together:

  1. Client:
    The client could run on your laptop or desktop computer. This is where you will run Matlab commands and submit your jobs.
  2. Worker:
    The worker is an instance of Matlab distributed computing environment that runs on Viper in order to perform a calculation for one of the clients. you could submit batch jobs from your clients that run some simulations in parallel using functions like parfor, you could also run normal Matlab code on it.

limitations

Submitting a job to Viper has a lot of overhead. If you are running a small computing task it might take even longer to run on the cluster. However, in many other cases, it will show you a speed-up and as a minimum, you will be able to free up your client from doing this computation.

Run Matlab Job

Using a parallel toolbox, you can run your Matlab code on viper using 2 modes:

Batch job

  • This is the main way to utilize the Matlab-viper-plugin. This submits your job to Viper queuing system (SLURM) and sends back the results to your client once everything is done.
  • Very useful for a time-consuming job that needs hours or even days to finish. you will submit your job and you don't need to keep your Matlab running or even the whole computer could be turned off as all your simulation will be running on Viper.
  • If you want to know more about the job life cycles and different stages https://uk.mathworks.com/help/distcomp/how-parallel-computing-products-run-a-job.html#f3-11033
  • Create Matlab script
    This can be either a normal serial script or a parallel one that uses functions like parfor. if you have an existing script make sure that the following is done before running it:
    • Remove any parpool commands if any
    • Modify the file paths if any is used in your script. consider that all your files will move to the cluster and run *:there so if your script contains a filepath that exisits only on your client it will not be able to find it there.
    example:
    use
 load  ('INPUT_FILE_NAME') 
instead of
 load ('/INPUT_DIRECTORY_NAME/INPUT_FILE_NAME')
  • Batch command: This command begins an automated process that connects to the cluster, submits a job to the scheduler, and initializes MDCS. it will ask you to enter your user name and password. you must also make sure that you are using on the campus network or while using VPN
  • For example if you submit a script to Viper with one input file:
 viper = parcluster()
 job = viper.batch('<your script>', 'pool', 9, 'AttachedFiles', '<name of input file>')
  • Specify the name of your script without the “m” prefix. For instance use test, and not test.m
  • The argument pool specified the number of workers you need for execution of the script. Note that the MATLAB adds one worker to the number of workers you specified in pool. So for instance if you specify 9 workers in the pool, then MATLAB will use 10 workers.
  • The 4th Argument 'AttachedFiles' highlight that we are going to attach input file/files to this batch job. you can send multiple files by including them in braces. For Example
     job = viper.batch('test', 'pool', 9, 'AttachedFiles', {input1, input2})
  • When you finish working with this job and get all your outputs back. permanently delete its data and remove its reference from the workspace:
 
delete(job)
clear job

Useful commands

below is some useful commands and its detailed offical documentation

Training Area Description
[getDebugLog ] shows a details log that is very useful in debugging the job submission process
[Job Monitor] enable you to monitor all your jobs and their status
[Parallel Computing Toolbox] This contains all the details of the Parallel Computing toolbox.

Parallel pool

  • Another mode of operation that is easier if you are testing a piece of code or running a small job that you expect to finish in few minutes.
  • This requires a running connection to be in place between your computer and viper. at any point you shut down your Matlab/Wifi or even your laptop everything will be lost and you will need to rerun your code/simulation.

Further Information


Navigation