Main Content

Program Communicating Jobs for a Supported Scheduler

Schedulers and Conditions

You can run a communicating job using any type of scheduler. Communicating jobs also include parpool, spmd, and parfor. These instructions show how to program communicating jobs to run on the local Processes cluster, a MATLAB® Job Scheduler cluster, or any third-party scheduler cluster interfaced with MATLAB Parallel Server™. The basic job programming sequence is the same for all the different scheduler types.

Code the Task Function

This example shows the basic principles of programming a communicating job. In this example, the worker whose spmdIndex value is 1 creates a magic square comprised of a number of rows and columns that is equal to the number of workers running the job (spmdSize). In this case, four workers run a communicating job with a 4-by-4 magic square. The first worker broadcasts the matrix using the spmdBroadcast function to all the other workers, each of which calculates the sum of one column of the matrix. All of these column sums are combined with the spmdPlus function to calculate the total sum of the elements of the original magic square.

The function for this example is shown below. 

function total_sum = colsum
if spmdIndex == 1
    % Send magic square to other workers
    A = spmdBroadcast(1,magic(spmdSize)); 
else
    % Receive broadcast on other workers
    A = spmdBroadcast(1);
end

% Calculate sum of column identified by spdmIndex for this worker
column_sum = sum(A(:,spmdIndex));

% Calculate total sum by combining column sum from all workers
total_sum = spmdPlus(column_sum);

Save the function as the file named colsum.m on the path of the MATLAB client. The software sends the file to each worker using the job's AttachedFiles property.

While this example has one worker create the magic square and broadcast it to the other workers, there are alternative methods of getting data to the workers. Each worker could create the matrix for itself. Alternatively, each worker could read its part of the data from a file on disk, the data could be passed in as an argument to the task function, or the data could be sent in a file contained in the job's AttachedFiles property. The solution to choose depends on your network configuration and the nature of the data.

Code in the Client

As with independent jobs, you choose a profile and create a cluster object in your MATLAB client by using the parcluster function. There are slight differences in the profiles, depending on the scheduler you use, but using profiles to define as many properties as possible minimizes coding differences between the scheduler types.

You can create and configure the cluster object with this code:

c = parcluster('MyProfile')

where 'MyProfile' is the name of a cluster profile for the type of scheduler you are using. Any required differences for various cluster options are controlled in the profile. You can have one or more separate profiles for each type of scheduler. For complete details, see Discover Clusters and Use Cluster Profiles. Create or modify profiles according to the instructions of your system administrator.

When your cluster object is defined, you create the job object with the createCommunicatingJob function. The job Type property must be set as 'SPMD' when you create the job.

cjob = createCommunicatingJob(c,'Type','SPMD');

The function file colsum.m (created in Code the Task Function) is on the MATLAB client path, but it has to be made available to the workers. One way to do this is with the job's AttachedFiles property, which can be set in the profile you used, or by entering this code in the Command Window:

cjob.AttachedFiles = {'colsum.m'}

You can also set other properties on the job, for example, setting the number of workers to use. Again, profiles might be useful in your particular situation, especially if most of your jobs require many of the same property settings. To run this example on four workers, you can established this in the profile, or by the following client code:

cjob.NumWorkersRange = 4

You create the job's one task with the usual createTask function. In this example, the task returns only one argument from each worker, and there are no input arguments to the colsum function.

t = createTask(cjob, @colsum, 1, {})

Use submit to run the job.

submit(cjob)

Make the MATLAB client wait for the job to finish before collecting the results. The results consist of one value from each worker. The spmdPlus function in the task shares data between the workers, so that each worker has the same result.

wait(cjob)
results = fetchOutputs(cjob)
results = 
    [136]
    [136]
    [136]
    [136]