Difference between revisions of "Training/Matlab-Viper-plugin"
From HPC
(Created page with "=Matlab viper plugin= ==introduction== There is a feature in Matlab that enables you to submit jobs to clusters directly from your matlab client (normal desktop matlab). we we...") |
m |
||
| (39 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
| − | =Matlab | + | <!-- |
| − | == | + | __TOC__ |
| − | There is a feature in Matlab that enables you to submit jobs to clusters directly from your | + | |
| − | This guide should explain more | + | =Matlab Viper plugin= |
| − | If you don't have Matlab-Viper | + | |
| + | Please note this page is in pre-release status and may be changed at any time. | ||
| + | |||
| + | ==Introduction== | ||
| + | There is a feature in Matlab that enables you to submit jobs to clusters directly from your Matlab client (normal desktop Matlab). we were able to develop a Matlab-Viper plug-in that utilizes this feature and integrates Matlab with Viper to enable all our Matlab users to submit some Matlab jobs to Viper easily without the need to know much about HPC or Linux. Everything could be done from the normal desktop Matlab version that many users are already using. | ||
| + | This guide should explain more about how to use Viper through the Matlab Viper plug-in. In general, the toolbox that could utilize such integration is the Matlab parallel computing toolbox. we will try to explain the main concepts here and point you to some MathWorks tutorials that you could use and try on our HPC. | ||
| + | |||
| + | *If you don't have the Matlab-Viper plug-in installed and tested on your Matlab contact [https://support.hull.ac.uk/ support portal] we will help you to install it and configure it. | ||
| + | *This is only available for Matlab 2017a, we will try to support it for all the upcoming versions starting from 2017a. if you have a special use case that requires the usage of an older version you could check the possibility of creating this plug-in for it contact our [https://support.hull.ac.uk/ support portal]. | ||
| + | * This link https://uk.mathworks.com/products/ineligible_programs.html shows a list of Ineligible programs that couldn't run on the Matlab distributed computing server through this plug-in. | ||
| + | |||
| + | ==Matlab to Viper== | ||
| + | ===Main concepts=== | ||
| + | Before we start we would like to introduce to you some basic concepts in order to be able to use this plug-in efficiently. | ||
| + | |||
| + | ==== This plugin works by connecting 2 components together:==== | ||
| + | #'''Client:''' | ||
| + | #: The client could run on your laptop or desktop computer. This is where you will run Matlab commands and submit your jobs. | ||
| + | #'''Worker:''' | ||
| + | #: The worker is an instance of a Matlab distributed computing environment that runs on Viper in order to perform a calculation for one of the clients. you could submit batch jobs from your clients that run some simulations in parallel using functions like ''parfor'', you could also run normal Matlab code on it. | ||
| + | |||
| + | ==== limitations ==== | ||
| + | |||
| + | Submitting a job to Viper has a lot of overhead. If you are running a small computing task it might take even longer to run on the cluster. However, in many other cases, it will show you a speed-up and as a minimum, you will be able to free up your client from doing this computation. | ||
| + | |||
| + | ===Run Matlab Job=== | ||
| + | |||
| + | Using a parallel toolbox, you can run your Matlab code on Viper using 2 modes: | ||
| + | |||
| + | ====Batch job==== | ||
| + | |||
| + | * This is the main way to utilize the Matlab-viper-plugin. This submits your job to Viper queuing system (SLURM) and sends back the results to your client once everything is done. | ||
| + | * Very useful for a time-consuming job that needs hours or even days to finish. you will submit your job and you don't need to keep your Matlab running or even the whole computer could be turned off as all your simulations will be running on Viper. | ||
| + | * If you want to know more about the job life cycles and different stages https://uk.mathworks.com/help/distcomp/how-parallel-computing-products-run-a-job.html | ||
| + | |||
| + | *'''Create Matlab script''' | ||
| + | *:This can be either a normal serial script or a parallel one that uses functions like parfor. if you have an existing script make sure that the following is done before running it: | ||
| + | ** Remove any parpool commands if any | ||
| + | ** Modify the file paths if any is used in your script. consider that all your files will move to the cluster and run *: there so if your script contains a file path that exists only on your client it will not be able to find it there. | ||
| + | *:example: | ||
| + | *:use | ||
| + | <pre style="background-color: #f5f5dc; color: black; font-family: monospace, sans-serif;"> load ('INPUT_FILE_NAME') </pre> | ||
| + | ::instead of | ||
| + | <pre style="background-color: #f5f5dc; color: black; font-family: monospace, sans-serif;"> load ('/INPUT_DIRECTORY_NAME/INPUT_FILE_NAME') | ||
| + | </pre> | ||
| + | *'''Batch command''': This command begins an automated process that connects to the cluster, submits a job to the scheduler, and initializes MDCS. it will ask you to enter your username and password. you must also make sure that you are using the campus network or while using VPN | ||
| + | *For example if you submit a script to Viper with one input file: | ||
| + | <pre style="background-color: #f5f5dc; color: black; font-family: monospace, sans-serif;"> viper = parcluster() | ||
| + | job = viper.batch('<your script>', 'pool', 9, 'AttachedFiles', '<name of input file>') | ||
| + | </pre> | ||
| + | :*Specify the name of your script without the “m” prefix. For instance use test, and not test.m | ||
| + | :*The argument pool specified the number of workers you need for the execution of the script. Note that MATLAB adds one worker to the number of workers you specified in the pool. So for instance, if you specify 9 workers in the pool, then MATLAB will use 10 workers. | ||
| + | :*The 4th Argument 'AttachedFiles' highlights that we are going to attach input file(s) to this batch job. you can send multiple files by including them in braces. For Example<pre style="background-color: #f5f5dc; color: black; font-family: monospace, sans-serif;"> job = viper.batch('test', 'pool', 9, 'AttachedFiles', {input1, input2})</pre> | ||
| + | *When you finish working on this job and get all your outputs back. permanently delete its data and remove its reference from the workspace: | ||
| + | <pre style="background-color: #f5f5dc; color: black; font-family: monospace, sans-serif;"> | ||
| + | delete(job) | ||
| + | clear job | ||
| + | </pre> | ||
| + | ==Useful commands== | ||
| + | below are some useful commands and their detailed official documentation | ||
| + | {| class="wikitable" | ||
| + | | style="width:25%" | <Strong>Training Area</Strong> | ||
| + | | style="width:75%" | <Strong>Description</Strong> | ||
| + | |- | ||
| + | | [[https://uk.mathworks.com/help/distcomp/getdebuglog.html| getDebugLog ]] | ||
| + | | shows a details log that is very useful in debugging the job submission process | ||
| + | |- | ||
| + | | [[https://uk.mathworks.com/help/distcomp/job-monitor.html/XQuartz| Job Monitor]] | ||
| + | | enable you to monitor all your jobs and their status | ||
| + | |- | ||
| + | | [[https://uk.mathworks.com/help/distcomp/index.html| Parallel Computing Toolbox]] | ||
| + | | This contains all the details of the Parallel Computing toolbox. | ||
| + | |- | ||
| + | |} | ||
| + | |||
| + | ====Parallel pool==== | ||
| + | * Another mode of operation that is easier is if you are testing a piece of code or running a small job that you expect to finish in a few minutes. | ||
| + | * This requires a running connection to be in place between your computer and Viper. at any point you shut down your Matlab/Wifi or even your laptop everything will be lost and you will need to rerun your code/simulation. | ||
| + | |||
| + | == Further Information == | ||
| + | |||
| + | * [https://matlabacademy.mathworks.com/ Matlab online courses] | ||
| + | * [https://uk.mathworks.com/help/distcomp/batch.html/ Batch command documentation] | ||
| + | |||
| + | |||
| + | |||
| + | |||
| + | [[Main Page]] / [[FurtherTopics/FurtherTopics#Matlab| Further Topics]] | ||
| + | --> | ||