Building Distributed/Codistributed Array with .mat files
1 vue (au cours des 30 derniers jours)
Afficher commentaires plus anciens
I'm trying to run a truncated SVD on a very large (wide) dataset. The dataset will be ~1000x100000, and broken up into multiple pieces by row (25x100000) and saved in different locations/workers as .mat files. If I have a list of these .mat files, is there a way to create a distributed or codistributed array from these files?
I've already tried creating a fileDatastore and converting the results into a distributed, but this just gets me a datastore containing multiple cells. Is there a function similar to cellUnderlying() for distributed arrays? I'm using Matlab R2017a.
0 commentaires
Réponses (1)
Oli Tissot
le 13 Août 2020
The following should do what you want:
ds = datastore('A_rowchunk_*.mat', 'Type', 'file', 'ReadFcn', @importdata, 'UniformRead', true);
dA = distributed(ds);
[U, S, V] = svd(dA);
For this to work the files must be located in a (network) location accessible from all the workers, but this seems to already be the case for you.
0 commentaires
Voir également
Catégories
En savoir plus sur Distributed Arrays dans Help Center et File Exchange
Produits
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!