MATLAB (Linux) crashes during parallel read in of files

Hello,

I've had problems with MATLAB crashing (it simply crashes and exits) when I try to read in a large number of files within a parfor loop. I have noticed that it seems to happen when my RAM (monitored using htop) gets full and then MATLAB suddenly crashes.

I have upgraded my RAM to 16GB and it seemed to help in some cases, however the problem still occurs.

Is the only solution to upgrade my RAM, or can something else be done? Actually one other thing I tried was to use less CPU cores which seems to help.

I used to use MATLAB on Windows 11 and it didn't crash due to this.

Thanks!

[Edric Ellis]

One thing you could try is to use parpool("threads") - not everything works there, but if it does, it often uses less memory.

Without some more details, it's hard to know exactly what to suggest - it's not clear if the problem is during the data processing inside the parfor loop, or during the transfer to and from the workers. Both are possible. You can use ticBytes and tocBytes to see how much data transfer there is.

Otherwise, you might need to see if you can reduce the amount of memory needed to process each file. Maybe you can read each file in pieces rather than all at once. Maybe something like datastore could help.

[Andrew Sandeman]

Hi, sorry I didn't see this sooner I didn't recieve some kind of notification / email.

When I read in the same number of files using less cores, or even just a single core (without using the parfor loop) it seems to be less likely to crash. In particular I haven't had any crashes when I just use single core. Does this suggest that the data processing inside the parfor loop is ok? (just that it may use significant memory)

Just to rephrase / rethink: it's possible that either the reading in of a single file takes significant memory, or its the communication of data between cores?

I am in fact already using a datastore when using the parfor read in. That is the difference in the code between the parfor loop and normal loop versions.

Thank you for your suggestions, I will try to use them at some point.