Can't open anymore files with fopen above some threshold

Hi guys!

I am trying to open all files contained in a folder, with something like

folder = 'some/folder/path'
files = dir(folder);
for i = 1:length(files)
    filenames{i} = files(i).name;
    output(i).name = filenames{i};
    output(i).file = sprintf('%s/%s',folder,filenames{i});
    output(i).fid = fopen(output(i).file);
    if (output(i).fid == -1)
        output(i).status = 'not read';
    else
        output(i).status = 'read';
    end
end

The folder contains few thousands of very light text files.

The problem is that, after having read some number of this files (in my case always 1020) the files in the list are not read anymore. So I get something like

Now, the problem is not the limit of files which can't be opened by MATLAB, because for my system it is apparently 10240.

If, in this situation, I try to open again manuallt that qc.dat, for example, I get -1 and 'Operation not permitted'. BUT if I try to read again manually that q_plates.dat, i.e. the last file which was read, it is read again with new fid 1021.

And, it seems also not a problem related with the files. If i close everything and only after that I try to manually read that 'qc.dat' and so on, I succeed.

What's the problem? The only thing I can think is a limit of the number of files opened in the same moment which sit in the same folder... is it the case?

I use MATLAB R2021b on a MacBook 10.15.7.

[Rik]

I don't have a solution for your problem, so let me try to figure out the deeper motive here: why exactly do you want to open that many files at once? I try to keep my file interactions as short-term as possible, so I hardly ever have more than 1 active FID.

[Steven Lord]

Having two files open simultaneously (one for reading, one for writing) is a common pattern. But I agree with @Rik; why do you need thousands of files open at once?

Open the file, do whatever processing you need to do with it, then close it. Repeat until you get to the end of the list of files. [Depending on the details of what processing you're performing, you may even be able to parallelize it using a parfor with this approach if you have Parallel Computing Toolbox installed.]

Answer 1

[Walter Roberson]

There is a per-process limit on the number of open files,

!ulimit -n

65535

The limit for the operating system running MATLAB Answers is 65535, but the limit on my Mac is 10240 by default.

If you wanted to change the limit for your system, see the "Sierra and newer systems" description at https://wilsonmar.github.io/maximum-limits/

For systems before Sierra, see https://docs.riak.com/riak/kv/latest/using/performance/open-files-limit/index.html

[Antonello Zito]

Yes, I know that. Also for my case the limit is 10240. Indeed I have no problems in open more files.

My problem is that I cannot open more than N files which sit in the same folder. After I run that code, I am not able to open qc.dat and so on as long as the previous 1020 in that folder are open... but I am able to open other files in other folders which will have resulting fids 1021 and so on.

Answer 2

[Image Analyst]

Why do you need to have them ALL open simultaneously? What I'd do is open one, get it's contents and do something with it, then close it and repeat for all the other files. So you'd have only one oepn at a time.