How to use parfor with a range from a very large object and cause memory problems

1 vue (au cours des 30 derniers jours)
Jon Ward
Jon Ward le 18 Nov 2021
Modifié(e) : Jon Ward le 18 Nov 2021
How can I achieve the following without gettting the broadcast variable warning and minimising memory usage by Matlab trying to copy the full variable R?
This is a simplified example of what I am trying to achieve, but the memory requirements explode when I try to do this (I am using 48 processors in parallel on a high spec AWS box with lots of memory).
Thanks!
T = 10000;
S = 100; %S is actually ~1000
P = 10;
M = 60;
R = rand(T,S,P,M); % This is an example, R and C below get populated from another process, but this is to illustrate the size of R.
C = rand(T,M);
resultsM = zeros(T,S,P);
resultsSD = zeros(T,S,P);
parfor t=50:T
r = R(1:t,:,:,:); % Broadcast warning here
% Do some stuff with r
c = C(1:t,:);
for s=1:S
for p=1:P
rr = squeeze(r(:,s,p,:));
tmpResults = sum(c.*rr,2);
resultsM(t,s,p) = mean(tmpResults); % Mean and below sd are examples, I apply other custom functions to tmpResults
resultsSD(t,s,p) = std(tmpResults);
end
end
end
  2 commentaires
Matt J
Matt J le 18 Nov 2021
The solution would depend on the "Do some stuff with r" part of the code.
Jon Ward
Jon Ward le 18 Nov 2021
Thanks, i have just edited the code to hopefully show more of the type of thing I am trying to do.

Connectez-vous pour commenter.

Réponses (1)

Raymond Norris
Raymond Norris le 18 Nov 2021
Have you considered assigning R in the parfor?
parfor t = 1:T
R = rand(T,S,P,M)
...
end
It's less intuitive for MATLAB serial code, but may be a better practice for MATLAB parallel code. R is now a temporary variable, which can't be referenced after the parfor.
Also look at ticBytes and tocBytes to see how much data is truely getting passed back and forth.
  1 commentaire
Jon Ward
Jon Ward le 18 Nov 2021
Unfortunately not, I was using rand to show the size of R only, it actually gets generated from another process. I have edited the code above to show what I am trying to do.

Connectez-vous pour commenter.

Catégories

En savoir plus sur Parallel for-Loops (parfor) dans Help Center et File Exchange

Produits


Version

R2020b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by