How to solve out of memory error when constructing sparse matrix

10 vues (au cours des 30 derniers jours)
lala le 24 Sep 2023
Commenté : lala le 27 Sep 2023
I am construcing a stiffness matrix in finite element computation.
I have got i, j, and v vectors as input arguments for sparse.
Both i and j are 222263280-by-1 int32 vectors while v is a 222263280-by-1 double vector. The stiffness matrix is 1202454-by-1202454.
However I encountered out of memory error when I run the command S = sparse(i,j,v,m,n).
Are there any solutions or suggestions to address this problem?
Why does not sparse support tall array? I think this problem can be solved if tall array used.
Thanks.
0 commentairesAfficher -2 commentaires plus anciensMasquer -2 commentaires plus anciens

Connectez-vous pour commenter.

Réponse acceptée

Bruno Luong le 24 Sep 2023
Modifié(e) : Bruno Luong le 24 Sep 2023
You might try the syntax
nz = ... ; < 222263280
S = sparse(i,j,v,m,n,nz);
with nz is the estimation of non-zero elements of your matrix. If you build P1 FEM, it's the number of edges of your setup. You might use Euler formula to estimte it : https://www.mathsisfun.com/geometry/eulers-formula.html
or using
nz = size(unique([i(:) j(:)], 'row'),1)
Note that the byte size required to store a sparse matrix S is
if isa(S,'double')
elbyte = 8;
else
elbyte = 1;
end
Sbytesize = 8*(size(S,2)+1) + ... % Jc array
8*nnzmax(S) + ... % Ir array
elbyte*nnzmax(S) % data value array
So if you deal well, your S will take less than 3.566 Gb (I assumz nz < 222263280),
>> (1202454+1)*8+222263280*8+222263280*8
ans =
3.5658e+09
plus the space of I, J, V. It will require 7 Gb to build it. If your have other suff around 16 Gb become a little tight.
1 commentaireAfficher -1 commentaires plus anciensMasquer -1 commentaires plus anciens
lala le 27 Sep 2023

Connectez-vous pour commenter.

Plus de réponses (2)

Matt J le 24 Sep 2023
It's a 5 GB matrix. Are you sure you have enough RAM?
4 commentairesAfficher 2 commentaires plus anciensMasquer 2 commentaires plus anciens
lala le 24 Sep 2023
No, I need help. 😭
since all i, j, and v can be maintained in RAM, why sparse failed? Does sparse need more memory than i, j, and v?
Torsten le 24 Sep 2023
If I clear all variables except i, j, and v for sparse, it can construct the sparse matrix successfully.
This should show you that it's not possible with the RAM available.

Connectez-vous pour commenter.

John D'Errico le 24 Sep 2023
Modifié(e) : John D'Errico le 24 Sep 2023
Remember that i and j and v also take memory.
(222263280*4*2 + 222263280*8)/1024^3
ans = 3.3120
So those three variables alone require 3.3 GB. And then the sparse array would require 5 GB. Next, you need to factor in that MATLAB will require CONTIGUOUS memory for any arrays. And MATLAB itself, and your OS take up memory. So I can understand that you may just have room to create that array, if you clear everything else. But you still won't be able to do virtually anything with it once built.
This is not really a problem of needing support for tall arrays to build a sparse matrix. That is just a crutch that will encourage you to build larger arrays yet. If you are going to work with these problems on that computer, then you desperately need to get more RAM. RAM is cheap these days. Honestly, I would suggest 16 GB of RAM is just barely a viable amount these days if you are doing any serious work in MATLAB.
John's law: Computer problems always expand to just a little larger than the memory available on the largest computer you can find.
(So, yes, according to the law I just gave you, even if you do get more RAM, you will quickly decide to solve larger problems yet. Since who wants to solve small problems? But it gives you a few years of grace.)
1 commentaireAfficher -1 commentaires plus anciensMasquer -1 commentaires plus anciens
lala le 27 Sep 2023

Connectez-vous pour commenter.

Catégories

En savoir plus sur Loops and Conditional Statements dans Help Center et File Exchange

R2023a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by