blazing fast performance with parfor except it pukes on itself with big files

This is a general question if anyone has ever seen this. Im not aiming to pick apart complex code here.

I have a data handing program i wrote that is meant to be extremely fast. Lots of care was taken to get it to perform to the highest speed.

Over the course of several versions i have made 100 X timing performance gains.

I started using par for which gave me a 4X speed up which was wonderful. However, when i ask par for to chew on a large version of my test files it just pukes and dies. Locks up my computer and that is the end of it until i force reboot.

My smaller test files are identical except less quantity of lines to parse.

When i use a 10,000 line test csv i get great performance.

When i use a 50,000 line csv it just falls down dead

Has anyone seen this before? i have a fairly significant machine to run this on so its not a resource problem i dont think. In looking at the task manager i do see 90% memory ussage.

Any general thoughts?

[Eike Blechschmidt]

Hello Robert, without any code it is hard to tell but my guess would be that you are loading the files into memory. What functions are you using for loading the data? Did you have a look at Datastore for tabular text files - MATLAB (mathworks.com)?

[Walter Roberson]

As an experiment, try asking for a parpool with one fewer member than you would normally be allocated.

[Robert Scott]

OK, will give that a go Walter, thanks

I suspect Erike's comment above to be true. Looking at the task manager, the local ram percentage used gets to 100% when trying to run large files against my code.

Why do a have a sneaking suspicion that the next thing for me is "tall" arrays

However, im not sure par for works with tall arays? Does it?

[Walter Roberson]

https://www.mathworks.com/help/parallel-computing/run-tall-arrays-on-a-parallel-pool.html

Yes, it is specifically documented to use parfor with tall array.

Could you give an approximate file size that you need to deal with? You mentioned 50000 lines, but about how many columns? Is each column 4 hex digits representing 16 bits?

[Robert Scott]

The true answer is that the row index can be in the millions

The colum index will be in several hundred