How to average more than 50 3D matrices using nanmean

Hi, I am trying to average a lot of 3D matrices using NaNmean. I have tried using cat but my 3D matrices are huge (351x400x400) which is using a lot of memory. Is there a better way to do this ?

[Adam Danz]

The goal isn't clear. What are you concatenating? Over which dimension are you averaging? What release of Matlab are you using?

A simple example would be best.

[raheem mian]

I have 50 volumes that are size (351,400,400).

So the regular solution I have been seeing is:

a1 = rand(351,400,400);
a2 = rand(351,400,400);
a3 = rand(351,400,400);
...
aN = rand(351,400,400); 
res = cat(4,a1,a2,a3,..,aN);
res = nanmean(res,4);

But since I have more than 50 volumes, a lot of memory is being occupied using this method. Looking for a more optimal solution. Matlab R2019b

[Adam Danz]

I see. So you're averaging element-wise across 50 equally sized 3D arrays. Were those arrays ever together as a single data set or within a single file or table etc? Where are those variables stored? on a mat file?

[raheem mian]

I save the matrices as .mat in a folder, they are individual 3D matrices.

[Matt J]

Are the arrays sparse (i.e., do they contain mostly zeros)?

[Adam Danz]

Hmmmm... concatenating 50 arrays that each have more than 56 million elements isn't going to happen.

Off the bat I can think of a couple ideas.

1) Using 2 loops, you can loop through each file and partially load each 351 x 400 slice so you have 50 of those matricies which would make ~7m data points. If that's still too large you could partially load in each 351x1 column. Then you can do element-wise averaging and store the values as you proceed through the loops. That would involve 50 x 400 loops which isn't a big deal.

2) you can reorganize your data as tall arrays which are designed for large amounts of data.

[raheem mian]

This idea is perfect, Thank you!

Answer 1

[Matt J]

Here's what I would do, I suppose. It assumes each of your .mat files stores the volume under the name 'a'.

Summation=0;
NCounter=0;

files=dir(fullfile('yourFolder','*.mat'));

for i=1:numel(files)
    
   S=load(fullfile('yourFolder',files(i).name)); 
   
   map=isnan(S.a);
   
   S.a(map)=0;
   
   Summation = Summation + S.a;
   NCounter  = NCounter + (~map);
    
end


result = Summation./Ncounter;

[raheem mian]

I like this method too ! Thanks. Method seems faster.

[Adam Danz]

Yep, this is simple and fast!