Building Distributed/Codistributed Array with .mat files
3 Ansichten (letzte 30 Tage)
Ältere Kommentare anzeigen
I'm trying to run a truncated SVD on a very large (wide) dataset. The dataset will be ~1000x100000, and broken up into multiple pieces by row (25x100000) and saved in different locations/workers as .mat files. If I have a list of these .mat files, is there a way to create a distributed or codistributed array from these files?
I've already tried creating a fileDatastore and converting the results into a distributed, but this just gets me a datastore containing multiple cells. Is there a function similar to cellUnderlying() for distributed arrays? I'm using Matlab R2017a.
0 Kommentare
Antworten (1)
Oli Tissot
am 13 Aug. 2020
The following should do what you want:
ds = datastore('A_rowchunk_*.mat', 'Type', 'file', 'ReadFcn', @importdata, 'UniformRead', true);
dA = distributed(ds);
[U, S, V] = svd(dA);
For this to work the files must be located in a (network) location accessible from all the workers, but this seems to already be the case for you.
0 Kommentare
Siehe auch
Kategorien
Mehr zu Distributed Arrays finden Sie in Help Center und File Exchange
Produkte
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!