What is the fastest/best way to create several thousand (text) files?

What is the fastest/best way to create several thousand (text) files?

The files should have the same name base and be numbered accordingly.

[Stephen23]

FOR loop and COPYFILE

[AAA]

is for loop and copyfile really the fastest way because I already implemented this but it is quite slow in my application?

[Matt J]

It should be slow. It's a disk operation.

If the copied text can be held in RAM though, wonder if it would be faster to write it to many different files using fprintf(). That way, you would have one-way RAM-to-disk transfer.

[AAA]

But the issue is that it is gradually getting slower with rising number of copies (-> at 10,000 copies the last copy takes 6 times longer than the first one)

[John D'Errico]

Not at all surprising. Odds are this just says your hard disk is fragmented. That is a natural thing for any drive that has been in use for possibly multiple years. So each time, your OS takes a little more time to allocate the necessary disk space. This is not a MATLAB issue by the way. It is your OS that is doing the work, and taking the time.

I might suggest you do it by working on an empty drive on the side. Now you would probably find it is less hungry for time.

[AAA]

Thank you everyone for your answers!

I analyzed a lot of different implementation possibilities and the main reason for the slow down when implementing it with a for-loop and "copyfile" was that the destination directory where the files get copied to, was in my search path. Seems like it is not an issue of the OS, instead the "copyfile" function triggers matlab to scan all the data in the search path which drastically increases the execution time over time.

When removing the destination directory from the search path the execution time does NOT increase and is drastically decreased.

When using "cellfun(@(FID) copyfile(sourcefile, FID), destnames);" instead of the for-loop the execution time can be further optimized and is a good way to go. (https://de.mathworks.com/matlabcentral/answers/14962-making-multiple-copies-of-a-file#answer_20334)

The fastest implementation for my application however is the method suggested by Kunal Kandhari.

Answer 1

[Kunal Kandhari]

Using "fopen" & "fprintf" is one of the fastest way of creating several thousand text files

below is the sample code for it:

baseFileName = 'textfile';  % The base name for your files
numFiles = 100000;           % The number of files you want to create

for i = 1:numFiles
    % Generate the full file name with a numerical suffix
    fileName = sprintf('%s_%04d.txt', baseFileName, i);
    
    % Open the file for writing
    fileID = fopen(fileName, 'w');
    
    % Write some content to the file (you can customize this part)
    fprintf(fileID, 'This is file number %d\n', i);
    
    % Close the file
    fclose(fileID);
end

The speed of file creation depends on various factors, including the file system's performance, your hardware, and how efficiently you write the files.

The MATLAB code above is a straightforward way to accomplish the task, and it's not inherently slow.

You can read more about the "fprintf" and "fopen" from the following documentation:

• fprintf
• fopen

[AAA]

Thank you everyone for your answers!

I analyzed a lot of different implementation possibilities and the main reason for the slow down when implementing it with a for-loop and "copyfile" was that the destination directory where the files get copied to, was in my search path. Seems like it is not an issue of the OS, instead the "copyfile" function triggers matlab to scan all the data in the search path which drastically increases the execution time over time.

When removing the destination directory from the search path the execution time does NOT increase and is drastically decreased.

When using "cellfun(@(FID) copyfile(sourcefile, FID), destnames);" instead of the for-loop the execution time can be further optimized and is a good way to go. (https://de.mathworks.com/matlabcentral/answers/14962-making-multiple-copies-of-a-file#answer_20334)

The fastest implementation for my application however is the method suggested by Kunal Kandhari.