Main Content

Run Batch Job and Access Files from Workers

Open Live Script

You can offload your computations to run in the background by using batch.

If your code needs access to files, you can use additional options, such as 'AttachedFiles' or 'AdditionalPaths', to make the data accessible. You can continue working in MATLAB while the computations take place. If you submit your computations to a remote cluster, you can close MATLAB and recover the results later.

Prepare Example

Use the supporting function prepareSupportingFiles to copy the required data for this example to your current working folder.

prepareSupportingFiles;

Your current working folder now contains 4 files: A.dat, B1.dat, B2.dat, and B3.dat.

Run Batch Job

Create a cluster object using parcluster. By default, parcluster uses your default cluster profile. Check your default cluster profile on the MATLAB Home tab, in the Environment section, in Parallel > Select a Default Cluster.

c = parcluster();

Place your code inside a function and submit it as a batch job by using batch. For an example of a custom function, see the supporting function divideData. Specify the expected number of output arguments and a cell array with inputs to the function.

Note that if you send a script file using batch, MATLAB transfers all the workspace variables to the cluster, even if your script does not use them. If you have a large workspace, it impacts negatively the data transfer time. As a best practice, convert your script to a function file to avoid this communication overhead. You can do this by simply adding a function line at the beginning of your script. To reduce overhead in this example, divideData is defined in a file outside of this live script.

If your code uses a parallel pool, use the 'Pool' name-value pair argument to create a parallel pool with the number of workers that you specify. batch uses an additional worker to run the function itself.

By default, batch changes the initial working folder of the workers to the current folder of the MATLAB client. It can be useful to control the initial working folder in the workers. For example, you might want to control it if your cluster uses a different filesystem, and therefore the paths are different, such as when you submit from a Windows client machine to a Linux cluster.

  • To keep the initial working folder of the workers and use their default, set 'CurrentFolder' to '.'.

  • To change the initial working folder, set 'CurrentFolder' to a folder of your choice.

This example uses a parallel pool with three workers and chooses a temporary location for the initial working folder. Use batch to offload the computations in divideData.

job = batch(c,@divideData,1,{}, ...
    'Pool',3, ...
    'CurrentFolder',tempdir);

batch runs divideData on a parallel worker, so you can continue working in MATLAB while computations take place.

If you want to block MATLAB until the job completes, use the wait function on the job object.

wait(job);

To retrieve the results, use fetchOutputs on the job object. As divideData depends on a file that the workers cannot find, fetchOutputs throws an error. You can access error information by using getReport on the Error property of Task objects in the job. In this example, the code depends on a file that the workers cannot find.

getReport(job.Tasks(1).Error)
ans = 
    'Error using divideData (line 4)
     Unable to read file 'B2.dat'. No such file or directory.'

Access Files from Workers

By default, batch automatically analyzes your code and transfers required files to the workers. In some cases, you must explicitly transfer those files -- for example, when you determine the name of a file at runtime.

In this example, divideData accesses the supporting file A.dat, which batch automatically detects and transfers. The function also accesses B1.dat, but it resolves the name of the file at runtime, so the automatic dependency analysis does not detect it.

type divideData.m 
function X = divideData()
    A = load("A.dat"); 
    X = zeros(flip(size(A)));
    parfor i = 1:3
        B = load("B" + i + ".dat");
        X = X + A\B;
    end
end

If the data is in a location that the workers can access, you can use the name-value pair argument 'AdditionalPaths' to specify the location. 'AdditionalPaths' adds this path to the MATLAB search path of the workers and makes the data visible to them.

pathToData = pwd;
job(2) = batch(c,@divideData,1,{}, ...
    'Pool',3, ...
    'CurrentFolder',tempdir, ...
    'AdditionalPaths',pathToData);
wait(job(2));

If the data is in a location that the workers cannot access, you can transfer files to the workers by using the 'AttachedFiles' name-value pair argument. You need to transfer files if the client and workers do not share the same file system, or if your cluster uses the generic scheduler interface in nonshared mode. For more information, see Configure Using the Generic Scheduler Interface (MATLAB Parallel Server).

filenames = "B" + string(1:3) + ".dat";
job(3) = batch(c,@divideData,1,{}, ...
    'Pool',3, ...
    'CurrentFolder',tempdir, ...
    'AttachedFiles',filenames);

Find Existing Job

If you submit the job to a remove cluster, you can close MATLAB after job submission and retrieve the results later. Before you close MATLAB, make a note of the job ID.

job3ID = job(3).ID
job3ID = 25

When you open MATLAB again, you can find the job by using the findJob function.

job(3) = findJob(c,'ID',job3ID);
wait(job(3));

Alternatively, you can use the Job Monitor to track your job. You can open it from the MATLAB Home tab, in the Environment section, in Parallel > Monitor Jobs.

Retrieve Results and Clean Up Data

To retrieve the results of a batch job, use the fetchOutputs function. fetchOutputs returns a cell array with the outputs of the function run with batch.

X = fetchOutputs(job(3))
X = 1×1 cell array
    {40×207 double}

When you have retrieved all the required outputs and do not need the job object anymore, delete it to clean up its data and avoid consuming resources unnecessarily.

delete(job)
clear job

See Also

| | |