How to pass data between parfeval

I'd like to run two jobs on the background using parfeval while I hope they can communicate between each other by passing the data from one job to another. For example, job A generates a data and the job B receives it. How should this work out?

Answer 1

[Walter Roberson]

Possibly a pollable queue https://www.mathworks.com/help/distcomp/send.html -- but I cannot tell from the documentation whether it only works for parfor or if it works for parfeval as well.

[Cai Li]

Thanks for responding! I found that I can use queue to pass data from the worker to the client while it doesn't work when you want to pass data to the worker from the client or the worker. And in the function description, it also mentions this.

[Walter Roberson]

If you look at DataQueue and PollableDataQueue https://www.mathworks.com/help/distcomp/parallel.pool.dataqueue.html they say,

"You can construct the queue on the workers and send it back to the client to enable communication in the reverse direction. However, you cannot send a queue from one worker to another. Use spmd, labSend, or labReceive instead."

You would need to use the client as the switchboard for this to work.

For worker-to-worker communications you would need labSend / labReceive. You might also be able to use MPI; see https://www.mathworks.com/matlabcentral/linkexchange/links/840-parallel-programming-with-mpi

[Cai Li]

Hi Walter,

Thanks for your comments. Sorry for such late response, took some time to write the code. I tried using client as a switchboard while I think it simply cannot pass data from client to the worker through queue. As for labSend and labReceive, I think it doesn't work in perfeval. For some reason, I have to use perceval to run my script on the background. Instead, I figured out a way to pass the data by simply saving it to a file in one worker and reading the file in another worker, which is surprisingly fast (at least for me) Thank you!