Reduce function if concatenated intermediate results doesn't fit in memory

1 Ansicht (letzte 30 Tage)
I'm trying to decode a huge dataset of rougly 500k*10k size. What I'm doing is actually load the file into a datastore, replace the tall table variable names with a meaningful name, and decode the content of all columns with a meaningful sentences in my dictionary.
I have some wiered issues that didn't let my program go through using a tall table. So I'm now trying to use mapreduce.
I find an example here
function subsettingReducer(~, intermValList, outKVStore)
% get all intermediate results from the list
outVal = {};
while hasnext(intermValList)
outVal = [outVal; getnext(intermValList)];
% Note that this approach assumes the concatenated intermediate values (the
% subset of the whole data) fit in memory.
add(outKVStore, 'Null', outVal);
My question is, in this example, if the contanenated intermediate results doesn't fit in memory, is there a better workaround for this?
  2 Kommentare
Peng Li
Peng Li am 9 Mai 2020
Or actually I can write the data store to local in the reduce function?
Peng Li
Peng Li am 11 Mai 2020
anybody who can shed me some light please?

Melden Sie sich an, um zu kommentieren.

Antworten (0)





Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by