Can I limit number of items in parallel.p​ool.Pollab​leDataQueu​e?

1 Ansicht (letzte 30 Tage)
Andrew McLean
Andrew McLean am 28 Apr. 2020
Bearbeitet: Andrew McLean am 29 Apr. 2020
I have a problem where I need to read and decode data from multiple packetised data streams. I currently have a system object that reads individual data streams (Reader) and another system object (MultiReader) that calls multiple readers and returns synchronised data.
The decoding is quite computationaly expensive so I'm trying to parallelise this. I have a solution that looks like it might work. I'm basically running the individual readers in parallel. A set of workers from a parallel pool are launched with parfeval and use the existing readers to return decoded data packets asynchronously in a parallel.pool.PollableDataQueue. The client receives the data from the workers, synchronises the streams and returns combined/synchronised data.
At times the workers can read the data faster than the client can consume it, there is a risk of exhausting available memory. However, there doesn't seem to be any easy way to limit the amount of data in the queues.
If I was using Python I would just set the maxsize parameter of the Queue object; adding items would then block when the number of items was above maxsize. Matlab's PollableDataQueue does not seem to have a similar capability.
As a workaround I tried the following in the worker:
while queue.QueueLength >= maxsize
pause(1)
end
send(queue, message)
But this doesn't seem to work. I assume the QueueLength attibute only "works" when called from the client. This behaviour appears undocumented. Am I missing something? I say "does not seem to work" because I'm finding it really difficult to debug code running on workers. Apart from looking at the Error and Diary attributes of the tasks in the client are there any tricks?
Returning to the original problem, the only other idea I had was to create an additional queue for control messages from the client to worker. These queues would need to be created in the workers and passed back to the client! However, this approach really does not fit into the structure of the code at the client end, which is pretty complicated as it is. Is there not a better way to do this?
Any ideas gratefully received.

Antworten (0)

Produkte


Version

R2019a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by