Matlab crashes with large matrix
14 vues (au cours des 30 derniers jours)
Afficher commentaires plus anciens
valentino dardanoni
le 25 Oct 2023
Commenté : Bruno Luong
le 25 Oct 2023
I need to create a large matrix F=[F1, F2, ... , FN] where each Fn has the same number of rows (about 12,000) but different number of columns. What I am doing now is to create F as a cell array using a loop with element F{n} = Fn. The problem is that when the number of columns of F is very large, Matlab crashes: at some point it shuts down (I run this on a server, and I find matlab closed). I suspect that it shuts down when I convert the cell array into a single matrix, F=cell2mat(F);
Notice that everything works ok when F has a number of columns less than say, say 1,000,000,
Any help is much appreciated. Thanks!
0 commentaires
Réponse acceptée
Walter Roberson
le 25 Oct 2023
Modifié(e) : Walter Roberson
le 25 Oct 2023
Well, that is certainly possible... if you make the arrays large enough then there might just not be enough memory available.
Is it possible to predict the number of columns that will be needed for each Fn before doing the actual callculations? If so then you could consider doing that calculation and totaling them, and pre-allocating the output array, then writing directly into the output array instead of building the cell array. If you pre-build the sizes then you can use cumsum() to create an array that will tell you which indices to write at
As a side note: it is more efficient to use
F = horzcat(F{:});
than F = cell2mat(F); -- but this would not solve your memory problems.
2 commentaires
Bruno Luong
le 25 Oct 2023
No it's not normal. But when all memory are consumed, many tasks are slowing down and requesred memory get swap to hard drive. The PC are in a "non standard" functioning mode and things can get slippy from there. Even pop up a warning message or windows become a hercules task.
Plus de réponses (1)
Bruno Luong
le 25 Oct 2023
Modifié(e) : Bruno Luong
le 25 Oct 2023
For 1e6 column the memory required in Gb for one array is
memgGb = 1e6*12e3*8 / 1e9
And you have the cell array + the concatenate array (single contiguous emory blockà, so you need 192 Gb of free memory.
You might consider using single-class array, write directly to the allocated matrix as Walter suggests or using tall array assuming you store the transposed of the original intended array.
2 commentaires
Bruno Luong
le 25 Oct 2023
licols is Matt's FEX, but yeah I quite participate to popular the idea. Thanks
Voir également
Catégories
En savoir plus sur Loops and Conditional Statements dans Help Center et File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!