Speeding up a batching process that takes too long

I have been using the batch() function to run a needy process in the background using multiple workers in a local pool. The problem I'm finding is that the execution of batch() in my case actually takes a very long time, in some cases almost as long as it would take the process itself to finish. I've tried to re-write it so that the batching process (which takes a while) is called in a script, and that script is then itself batched. What I end up with is an error saying that workers cannot submit jobs to the local cluster. Does anyone know if there is a way to support parallel batching done in the background?

Answer 1

[Edric Ellis]

You can use parfeval on top of a parallel pool to do this. (Note that by opening a parallel pool of workers, you're actually reducing the number of workers available to run batch jobs - workers can work on either parallel pool things ( parfor, parfeval, and spmd ), or batch - but not both at the same time).

The programming interface to parfeval is broadly similar to batch, so it should not be too difficult to adapt your code to use that instead. parfeval has the advantage of lower overheads once your parallel pool is set up ( batch has to launch a new MATLAB worker process for each task that it executes).

[Stephen]

That works so much better!