out of memory: array too long
3 vues (au cours des 30 derniers jours)
Afficher commentaires plus anciens
Hello, I'd like to run function pdist of Matlab on an array whose the length is N*(N-1)/2 where N=340000. Matlab is out of memory to preallocate this array. Could anyone give me a solution please? Kind regards, Winn
1 commentaire
Oleg Komarov
le 24 Sep 2014
As per the you reference in http://www.mathworks.co.uk/matlabcentral/answers/156028#comment_239068, you can block process and keep partial sums. In any case you will need to use for loops.
Réponse acceptée
Sean de Wolski
le 24 Sep 2014
That's only going to require 57.8 gigabytes of memory for a single column. And that's the end result. Surely there'll be some large intermediate arrays as well.
I'd recommend downsampling or chunking up the calls.
Do you need every pairwise distance? Are you looking for something specific? What's the end goal?
8 commentaires
Sean de Wolski
le 25 Sep 2014
That's a good idea. Do you even need the files though? Because they'll take up a ton of space. Could you just gather the info you need from the data and write only the results or pairs you care about?
You might want to look into doing this in parallel with a parfor loop, it could help speed it along. Though writing the files will likely be the bottleneck and it will be a hardware limitation not a software one.
Plus de réponses (1)
Adam
le 24 Sep 2014
Depending on how much over memory it is you could try converting your data to single before you pass it to pdist. That should take half the memory.
I don't know off-hand if pdist is overloaded for integer types or not. If it is then you could also use them depending what level of accuracy you requie.
2 commentaires
Adam
le 24 Sep 2014
Well, yes, but that creates an array of doubles. You can try pre-allocating:
dist = zeros(N*(N-1)/2,1, 'single')
or even
dist = zeros(N*(N-1)/2,1, 'uint8')
but the latter option assumes pdist works on uint8 data and that you really don't care much about accuracy!
Voir également
Catégories
En savoir plus sur Startup and Shutdown dans Help Center et File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!