Difference between revisions of "Training/Matlab-Viper-plugin"
From HPC
m |
|||
| (7 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
| + | <!-- | ||
__TOC__ | __TOC__ | ||
| Line 6: | Line 7: | ||
==Introduction== | ==Introduction== | ||
| − | There is a feature in Matlab that enables you to submit jobs to clusters directly from your Matlab client (normal desktop Matlab). we were able to develop a Matlab-Viper plug-in that utilizes this feature and integrates Matlab with | + | There is a feature in Matlab that enables you to submit jobs to clusters directly from your Matlab client (normal desktop Matlab). we were able to develop a Matlab-Viper plug-in that utilizes this feature and integrates Matlab with Viper to enable all our Matlab users to submit some Matlab jobs to Viper easily without the need to know much about HPC or Linux. Everything could be done from the normal desktop Matlab version that many users are already using. |
| − | This guide should explain more | + | This guide should explain more about how to use Viper through the Matlab Viper plug-in. In general, the toolbox that could utilize such integration is the Matlab parallel computing toolbox. we will try to explain the main concepts here and point you to some MathWorks tutorials that you could use and try on our HPC. |
| − | *If you don't have Matlab-Viper plug-in installed and tested on your Matlab contact [https://support.hull.ac.uk/ | + | *If you don't have the Matlab-Viper plug-in installed and tested on your Matlab contact [https://support.hull.ac.uk/ support portal] we will help you to install it and configure it. |
| − | *This is only available for Matlab 2017a , we will try to support it for all the upcoming versions starting from 2017a. if you have a special use case that requires the usage of an older version you could check the possibility of creating this plug-in for it contact [https://support.hull.ac.uk/ | + | *This is only available for Matlab 2017a, we will try to support it for all the upcoming versions starting from 2017a. if you have a special use case that requires the usage of an older version you could check the possibility of creating this plug-in for it contact our [https://support.hull.ac.uk/ support portal]. |
| − | * This link https://uk.mathworks.com/products/ineligible_programs.html shows a list of Ineligible programs that couldn't run on Matlab distributed computing server through this plug-in. | + | * This link https://uk.mathworks.com/products/ineligible_programs.html shows a list of Ineligible programs that couldn't run on the Matlab distributed computing server through this plug-in. |
==Matlab to Viper== | ==Matlab to Viper== | ||
| Line 21: | Line 22: | ||
#: The client could run on your laptop or desktop computer. This is where you will run Matlab commands and submit your jobs. | #: The client could run on your laptop or desktop computer. This is where you will run Matlab commands and submit your jobs. | ||
#'''Worker:''' | #'''Worker:''' | ||
| − | #: The worker is an instance of Matlab distributed computing environment that runs on Viper in order to perform a calculation for one of the clients. you could submit batch jobs from your clients that run some simulations in parallel using functions like ''parfor'', you could also run normal Matlab code on it. | + | #: The worker is an instance of a Matlab distributed computing environment that runs on Viper in order to perform a calculation for one of the clients. you could submit batch jobs from your clients that run some simulations in parallel using functions like ''parfor'', you could also run normal Matlab code on it. |
==== limitations ==== | ==== limitations ==== | ||
| Line 29: | Line 30: | ||
===Run Matlab Job=== | ===Run Matlab Job=== | ||
| − | Using a parallel toolbox, you can run your Matlab code on | + | Using a parallel toolbox, you can run your Matlab code on Viper using 2 modes: |
====Batch job==== | ====Batch job==== | ||
* This is the main way to utilize the Matlab-viper-plugin. This submits your job to Viper queuing system (SLURM) and sends back the results to your client once everything is done. | * This is the main way to utilize the Matlab-viper-plugin. This submits your job to Viper queuing system (SLURM) and sends back the results to your client once everything is done. | ||
| − | * Very useful for a time-consuming job that needs hours or even days to finish. you will submit your job and you don't need to keep your Matlab running or even the whole computer could be turned off as all your | + | * Very useful for a time-consuming job that needs hours or even days to finish. you will submit your job and you don't need to keep your Matlab running or even the whole computer could be turned off as all your simulations will be running on Viper. |
| − | * If you want to know more about the job life cycles and different stages https://uk.mathworks.com/help/distcomp/how-parallel-computing-products-run-a-job.html | + | * If you want to know more about the job life cycles and different stages https://uk.mathworks.com/help/distcomp/how-parallel-computing-products-run-a-job.html |
| + | |||
*'''Create Matlab script''' | *'''Create Matlab script''' | ||
*:This can be either a normal serial script or a parallel one that uses functions like parfor. if you have an existing script make sure that the following is done before running it: | *:This can be either a normal serial script or a parallel one that uses functions like parfor. if you have an existing script make sure that the following is done before running it: | ||
** Remove any parpool commands if any | ** Remove any parpool commands if any | ||
| − | ** Modify the file paths if any is used in your script. consider that all your files will move to the cluster and run *:there so if your script contains a | + | ** Modify the file paths if any is used in your script. consider that all your files will move to the cluster and run *: there so if your script contains a file path that exists only on your client it will not be able to find it there. |
*:example: | *:example: | ||
*:use | *:use | ||
| Line 46: | Line 48: | ||
<pre style="background-color: #f5f5dc; color: black; font-family: monospace, sans-serif;"> load ('/INPUT_DIRECTORY_NAME/INPUT_FILE_NAME') | <pre style="background-color: #f5f5dc; color: black; font-family: monospace, sans-serif;"> load ('/INPUT_DIRECTORY_NAME/INPUT_FILE_NAME') | ||
</pre> | </pre> | ||
| − | *'''Batch command''': This command begins an automated process that connects to the cluster, submits a job to the scheduler, and initializes MDCS. it will ask you to enter your | + | *'''Batch command''': This command begins an automated process that connects to the cluster, submits a job to the scheduler, and initializes MDCS. it will ask you to enter your username and password. you must also make sure that you are using the campus network or while using VPN |
*For example if you submit a script to Viper with one input file: | *For example if you submit a script to Viper with one input file: | ||
<pre style="background-color: #f5f5dc; color: black; font-family: monospace, sans-serif;"> viper = parcluster() | <pre style="background-color: #f5f5dc; color: black; font-family: monospace, sans-serif;"> viper = parcluster() | ||
| Line 52: | Line 54: | ||
</pre> | </pre> | ||
:*Specify the name of your script without the “m” prefix. For instance use test, and not test.m | :*Specify the name of your script without the “m” prefix. For instance use test, and not test.m | ||
| − | :*The argument pool specified the number of workers you need for execution of the script. Note that | + | :*The argument pool specified the number of workers you need for the execution of the script. Note that MATLAB adds one worker to the number of workers you specified in the pool. So for instance, if you specify 9 workers in the pool, then MATLAB will use 10 workers. |
| − | :*The 4th Argument 'AttachedFiles' | + | :*The 4th Argument 'AttachedFiles' highlights that we are going to attach input file(s) to this batch job. you can send multiple files by including them in braces. For Example<pre style="background-color: #f5f5dc; color: black; font-family: monospace, sans-serif;"> job = viper.batch('test', 'pool', 9, 'AttachedFiles', {input1, input2})</pre> |
| − | *When you finish working | + | *When you finish working on this job and get all your outputs back. permanently delete its data and remove its reference from the workspace: |
<pre style="background-color: #f5f5dc; color: black; font-family: monospace, sans-serif;"> | <pre style="background-color: #f5f5dc; color: black; font-family: monospace, sans-serif;"> | ||
delete(job) | delete(job) | ||
| Line 60: | Line 62: | ||
</pre> | </pre> | ||
==Useful commands== | ==Useful commands== | ||
| − | below | + | below are some useful commands and their detailed official documentation |
{| class="wikitable" | {| class="wikitable" | ||
| style="width:25%" | <Strong>Training Area</Strong> | | style="width:25%" | <Strong>Training Area</Strong> | ||
| Line 77: | Line 79: | ||
====Parallel pool==== | ====Parallel pool==== | ||
| − | * Another mode of operation that is easier if you are testing a piece of code or running a small job that you expect to finish in few minutes. | + | * Another mode of operation that is easier is if you are testing a piece of code or running a small job that you expect to finish in a few minutes. |
| − | * This requires a running connection to be in place between your computer and | + | * This requires a running connection to be in place between your computer and Viper. at any point you shut down your Matlab/Wifi or even your laptop everything will be lost and you will need to rerun your code/simulation. |
== Further Information == | == Further Information == | ||
| Line 86: | Line 88: | ||
| − | |||
| − | + | ||
| − | + | [[Main Page]] / [[FurtherTopics/FurtherTopics#Matlab| Further Topics]] | |
| − | + | --> | |
| − | |||