How can I save matrices in par-for-loop?

Hi all,

I'm using MATLAB's par-for-loop to process some experimental data, which is large in size. The result of each iteration is a matrix, but the sizes of the matrices generated by different iterations are different. I'm using a cell to collect the data, and yet it turns out to be inefficient. What I've implemented now is as follows,

load('dataset.mat', 'data');
load('matsize.mat', 'matsize');
data = parallel.pool.Constant(data);
matsize = parallel.pool.Constant(matsize);

N = 24;
results = cell(N,1);
parfor L = 0:N-1
    cursize = matsize.Value(L+1);
    matrix(1:cursize, 1:cursize) = calculateSomething(data.Value, L); 
    % the matrix is very large, can be up to size of 2^N, whose sizes are differnt for each L
    results{L+1} = matrix(1:cursize, 1:cursize);
end

save('results.mat', 'results');

% MATLAB version: 2019a

I use ticBytes and tocBytes to test the data passing process and find that massive data is transferred from workers to client. Is there any way to avoid this and allow data to be directly written in the same block of memory as pre-allocated, and thus improve the efficiency of the code?

Thanks a lot!

Answer 1

[Walter Roberson]

No, the memory lives in different processes. There is no Mathworks support for shared memory.

There is the FEX contribution https://www.mathworks.com/matlabcentral/fileexchange/28572-sharedmatrix that can take advantage of operating system facilities for shared memory.

The closest official support would possibly be https://www.mathworks.com/help/matlab/ref/memmapfile.html memmapfile() -- that is you would memory map the same file on worker and client. Note however that memory mapping a file usually implies writing to disc. Do you just happen to have enough RAM to create a RAMDISK filesystem to share? Or at least an SSD that will not run out of write cycles for a reasonable duration ?

[Wenjun Zhang]

Thanks a lot. I'm trying FEX. This is the first time I deal with hugh matrices and parallel computation.