Getting Started with MATLAB Job Scheduler, Part 4: Run Jobs on Your MATLAB Parallel Server Cluster - MATLAB
Video Player is loading.
Current Time 0:00
Duration 5:20
Loaded: 0.00%
Stream Type LIVE
Remaining Time 5:20
 
1x
  • Chapters
  • descriptions off, selected
  • en (Main), selected
    Video length is 5:20

    Getting Started with MATLAB Job Scheduler, Part 4: Run Jobs on Your MATLAB Parallel Server Cluster

    From the series: Getting Started with MATLAB Job Scheduler

    This is the fourth video of a five-part series. It will provide an overview of how to run jobs on your MATLAB Parallel Server™ cluster.

    MATLAB Parallel Server lets you scale MATLAB® programs and Simulink® simulations to clusters and clouds. You can prototype your programs and simulations on the desktop and then run them on clusters and clouds without recoding. MATLAB Parallel Server supports batch jobs, interactive parallel computations, and distributed computations with large matrices.

    Take advantage of high-end hardware in your organization’s cluster without leaving the MATLAB desktop environment. Use the same MATLAB analytics on small or large volumes of data. From your Windows®, Mac®, or Linux® desktop, you can process big data on Hadoop® clusters enabled by Spark™ or on traditional clusters with standard file systems. Execute calculations that won’t fit in the memory of a single machine without needing to recode your algorithm or use a shared-memory architecture.

    Published: 17 Aug 2023

    Hello. My name is Rebecca Silva Alves. I am a part of the installation and licensing team here at MathWorks. This is the fourth video of a five-part series where I am going to provide an overview of MATLAB Parallel Server and how to set up and use MATLAB Job Scheduler and a MathWorks Network License Manager. The previous video covered how to configure MATLAB to submit jobs to your MATLAB job scheduler. This video will cover how to use MATLAB to run and monitor jobs on your MATLAB Job Scheduler cluster. Let's begin.

    In the previous video, you confirmed that your MATLAB version matched the MATLAB Parallel Server version on the cluster, and you had Parallel Computing Toolbox installed and licensed. You created a cluster profile that points MATLAB to your MATLAB Job Scheduler and optionally validated it to make sure that everything is working correctly. You can run code on the cluster by starting a parallel pool of workers where MATLAB Parallel Server is installed using the parpool command. I am also specifying my cluster name and the number of workers I wish to start. Parallel constructs like parfor are also able to automatically open a parallel pool for you.

    You are now ready to run jobs on your cluster. Here is a parameter sweep example that I am going to run. You can find this example on the MathWorks website. The script includes some serial code, some parallel code, and some helper functions at the bottom of the script.

    I know that a majority of the time is spent in the parfor loop, and I want to utilize my cluster to run the code on more workers in parallel and speed up the computation. When your code runs, any MATLAB code in a parallel construct will automatically run on the resources that parpool is connected to. With interactive workflows, your MATLAB client is a part of the parallel pool and MATLAB must remain open and connected to the cluster while the computation runs.

    You can only have one interactive pool running out of time. Let's run our example. When MATLAB hits the parfor command, you can see, in the bottom left of the MATLAB window, that the parallel workers are doing something. As you can see, the MATLAB client is a part of the computation, and we can see how the computation is progressing.

    In this example, while my code is running, I am unable to use MATLAB for anything else. You can include asynchronous parallel constructs to run things on the cluster and be able to continue using MATLAB. If I shut down MATLAB or my computer, or disconnect from the cluster, the parallel pool will terminate and my code will stop running.

    You can also use the batch function to offload work from your MATLAB session to run on the cluster where MATLAB Parallel Server is installed. Before doing this, make sure the parallel pool you previously started is closed. With batch, your function or script runs on the cluster, and your MATLAB client is not tied to the computation. This

    Batch command will offload the entire parameter sweep problem to the cluster. My MATLAB session will not be a part of the computation, and everything will run solely on the cluster. You can also keep track of your jobs from MATLAB programmatically or using the job monitor.

    After submitting a batch job, you will be able to continue working in MATLAB. You may also shut down MATLAB or your computer and come back later to check the job status or retrieve the results. You can submit multiple batch jobs which will enter a queue and run as resources become available. At this point, the first job has finished, so we can grab the results from that job.

    Depending on your code and workflow, a batch job or an interactive pool may be a better fit for your needs. If, for example, you are running a job that will take a long time to complete, like days or weeks, then batch will be the preferred option, as your MATLAB client doesn't have to remain open and connected to the cluster the entire time. On the other hand, if you want to prototype or debug your code, or if your code utilizes interactive workflows, then an interactive job may be a better fit.

    In this video, we submitted jobs to the MATLAB Job Scheduler, discussed the difference between interactive and batch jobs, and retrieved results from your job. In the next video, we will cover how your cluster admin can monitor MATLAB Parallel Server usage and upgrade versions.