Reduce function if concatenated intermediate results doesn't fit in memory

1 view (last 30 days)
I'm trying to decode a huge dataset of rougly 500k*10k size. What I'm doing is actually load the file into a datastore, replace the tall table variable names with a meaningful name, and decode the content of all columns with a meaningful sentences in my dictionary.
I have some wiered issues that didn't let my program go through using a tall table. So I'm now trying to use mapreduce.
I find an example here
function subsettingReducer(~, intermValList, outKVStore)
% get all intermediate results from the list
outVal = {};
while hasnext(intermValList)
outVal = [outVal; getnext(intermValList)];
end
% Note that this approach assumes the concatenated intermediate values (the
% subset of the whole data) fit in memory.
add(outKVStore, 'Null', outVal);
end
My question is, in this example, if the contanenated intermediate results doesn't fit in memory, is there a better workaround for this?
Thanks!

Answers (0)

Tags

Products


Release

R2020a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by