minibatchqueue dispatch in background slow

Is there a reason using minibatchqueue with dispatchInBackground set to true would be slower than false?

I'm using a machine with A100 GPU (40Gig) and 50 cores

When I turn the dispatchInBackground option on, it's like the minibatchqueue preprocess function gets in an infinite loop prepping data. The parallel pool turns on, but it only a portion of the four bars on the parallel indicator ever turns green. If I turn the option off, it's slow but runs fine.

[Joss Knight]

Hard to answer without more information really, like some example code. It might be slower if you're accidentally using a remote cluster instead of a local one or if there is otherwise some problem for your pool workers reading the files. Or perhaps your data is in memory rather than on the filesystem, in which case you usually shouldn't use dispatchInBackground. If file access and preprocessing is trivial in terms of performance then you may find that the real bottleneck is communication between the pool and your client MATLAB. Or it could be you have preprocessing code that is very compute intensive and the workers are competing with each other and the client for resources.

[Luke Hubbard]

My datastore preprocessing function is mostly file i/o that pulls in a table from a given file (the files are not local but rather on a shared network--maybe a source of issue?) with a couple of extra steps adding a few new columns (features) to the table, taking the log of features with large numbers, and then using the datasample() fcn to return a smaller portion of the table. Then I split the features into the predictor sequence array and responses categorical array for use in sequence training.

When I set up my minibatchqueue, I have another preprocessing function to pad batch sequences and return as dlarray. Like I said before, the network runs and learns as expected without the dispatchInBackground feature turned on but hangs otherwise.

I'm not at liberty to share actual code. I couldn't find any other community questions that were similar, so I was curious if this is a known issue.