How can I perform a set of calculations on multiple matrices located in a csv. file?

I currently have a code that reads through multiple csv. files in a folder.

I have defined a matrix M, using: M = readmatrix(filename,'Range','A8277:AR8320');

I then perform a set of calculations on this matrix.

%calculating contact pressure
ContactPressure = max(M(:))
M = M ~= 0;
S = sum(M);
TS = sum(S);
%Calculating contact area
ContactArea = TS*1.6129   

My csv. files have 1440 matrices of the exact same size as the one previously defined (M). I would like to modify this code such that it performs these calculations on each matrix and returns the ContactPressure and ContactArea to a table.

I have currently tried to define each frame using: Frame1 = readmatrix(filename, 'Range', 'A45:AR88'); but cannot figure out what to do next.

Any ideas??

[dpb]

Are there multiple files or does one file contain multiple datasets?

What is the structure of the file if the latter as is sorta' implied/stated?

If it's all just one great big file with an arbitrary size defining which piece belongs to a given set, then simply reading the whole file into memory and manipulating the pieces is probably the simplest. One could use a 3D array and each plane represents one dataset.

Be best to attach a file representative of the dataset...always far easier to answer a specific Q? w/ example than to guess...

Answer 1

[Jessica Benitah]

Hello, thanks for your reply. See sample data set attached.

This is one csv file in a folder with others that need to be read my the code.

Answer 2

[dpb]

Those files have structure to deal with -- generically, one could read the header info and parse it to find out the content -- but, I'll not go quite that far here, but do need to deal with the fact there are a zillion frames in each file so don't want to open/close the file every time. All the new readXXX routines do that; revert back to venerable textscan to handle a case such as this. Or, one could use the large file mapping routines...

fid=fopen('S193714L_30_M.csv');         % open the file
% read first section past the header -- I hardcoded the 44x44 size as well
% as the headerlines section number of lines here.  In general, should scan
% for and determine dynamically
d=reshape(cell2mat(textscan(fid,'%f',44*44,'HeaderLines',44,'Delimiter',',','CollectOutput',1)),44,[]).';
% process this dataset here before going on to next one...
while ~feof(fid)                            % loop until hit EOF() condition...
  fgetl(fid);                               % resynch filepointer to beginning of record
  d=reshape(cell2mat(textscan(fid,'%f',44*44,'HeaderLines',2,'Delimiter',',','CollectOutput',1)),44,[]).';
  % process this dataset here before going on to next one...
end
fid=fclose(fid);                             % close file handle...

This doesn't save the data; it presumes is processed as wanted each time read a new array. To save, need to preallocate a 3D array and write each to the succeeding plane. This may be more than can hold in memory, however...

Answer 3

[Seth Furman]

Take a look at the "Large Files and Big Data" section of the doc. This sounds like a good candidate for tall arrays.

https://www.mathworks.com/help/matlab/large-files-and-big-data.html

[dpb]

This has been a recommendation several times in the past from TMW staff, Seth. I have also asked several times if the recommender would illustrate the use of one or more of these features on the specific file, but so far nobobdy has taken up the challenge.

There are no examples in the documentation to handle files of the sort of OP's which are not truly totally regular; my experimentation so far has never been successful in figuring out how to make any of these tools work for specific files.

I think examples such as this would be a priceless resource for the community in seeing how to manipulate the datastore object to handle such, or, they're good examples to use in further developing the toolset to handle real world issues.