How to start parallel pool in multiple nested loops?

my outer loop is like this, parfor SNRindB=0:2:20 and I have more than 20 loops inside this outer loop. When I try to use parallel pool it gives me following error, Error: Unable to classify the variable 'num_err_bits' in the body of the parfor-loop. I am attaching the main file of my program for reference. I could not use the techniques explained in "https://kr.mathworks.com/help/parallel-computing/troubleshoot-variables-in-parfor-loops.html" becoz my code is too complex. Can anyone help me. I will be grateful.

[Sajid Sarwar]

Can anyone help?

Answer 1

[Walter Roberson]

You have

v=1;
parfor SNRindB=0:2:20
%some code
    num_err_bits(v)=0;
%more code
        for num=1:(m*Nt)
            if in_index(num) ~= out_index(num)
                num_err_bits(v) = num_err_bits(v) + 1;
            end
        end
        kk
    end
    prb(v) = num_err_bits(v)/(B);
    v=v+1;
end

Let us run through this, supposing you have two parallel workers.

Suppose for the purposes of discussion that your code was permitted.

You assign 1 to v before the parfor, so the variable with 1 is copied to both workers when the parfor starts.

Suppose one of the workers gets SNRindB = 14 and the other gets SNRindB = 20 -- remember that parfor does not even try to execute the workers in 0:2:20 increasing order. One worker does not get assigned 0 and the next worker 2, then whatever worker finishes first gets assigned 4, and so on: parfor allocates chunks of range to a worker. At least one of the workers will be assigned the maximum value first, which is an optimization designed to make sure that arrays get allocated as large as needed the first time. So, sure, it could be the case that the first pass, one worker is assigned [20 18 16] and the other [14 12 10] and the rest left in reserve to serve to whichever finishes first.

Both workers have v = 1, so num_err_bits(1) is assigned 0 in both workers.

You do some work, and both workers enter the for num loop. Suppose the if is true, so both workers get to

num_err_bits(v) = num_err_bits(v) + 1

and v is 1 on both workers, so this is

num_err_bits(1) = num_err_bits(1) + 1

Now, let us take two cases:

1. Due to chances of timing, one of the two workers is able to execute the complete statement first and then the other executes. If so, then num_err_bits(1) would assigned 0+1 = 1 and then num_err_bits(2) would be assigned 1+1 = 2. So you might happen to avoid a timing conflict, and on both workers num_err_bits(v) would be 2 (for the purposes of discussion!); OR
2. Due to chances of timing, one of the two workers is able to execute the num_err_bits(v) reading part first, pulling out the 0, and then before the assignment in the first worker, the other worker executes the num_err_bits(v) reading part, also pulling out the 0. Both workers increment the count to 1, and both workers assign that 1 to num_err_bits(1) (for the purpose of discussion)

Now suppose that in one of the workers, in_index(num) ~= out_index(num) is false for all the other iterations of num, so it finishes the for num loop faster, and moves on and gets through to the assignment to prb(v) before the second worker is done with its for num loop.

So that faster worker goes to execute

    prb(v) = num_err_bits(v)/(B);

If the slower worker is in the middle of a num_err_bits(v) = num_err_bits(v) + 1 then does the expression use the value before the increment or after the increment?

And the faster worker moves on to

    v=v+1;

so v becomes 2. Meanwhile the slower worker is back at

    num_err_bits(v) = num_err_bits(v) + 1

but now (for the purpose of discussion) v has become 2. So the slower worker has to read out num_err_bits(2) . But num_err_bits(2) has not been assigned to, perhaps. Or if the faster worker makes it to assigning 0 to num_err_bits(v) for the next iteration, then num_err_bits(v) might perhaps be 0. Or maybe somehow num_err_bits(v) was initialized with something and the slower worker increments it a few times and then the value gets overwritten with 0 when the faster worker gets to the place that it assigns 0 to num_err_bits(v)

So (for the purpose of discussion) eventually the slower worker assigns to prb(v) and increment v to 3... at a time when the faster worker might be counting on v staying the same for its work.

After all the cycles then (for the purpose of discussion) all of the prb are assigned to. How many of them? Well, that depends upon the timing of the incrementing v. In the worst case, prb can end up being short by (number of workers minus 1)

Now... supposing that prb came out the expected length. Your parfor finishes, and you do

SNRindB=0:2:20;
semilogy(SNRindB,prb,'-^b');

Gee, that sounds like you expect a particular position in prb to correspond to a particular SNRindB value. But it doesn't, because parfor executes in what should be considered a random order (it is at least non-deterministic order) and (for the purposes of discussion) you stored into prb in the order of finishing iterations, not according to the SNRindB value that was appropriate for the iteration.

How does MATLAB deal with all of this? Simple: it prohibits you from doing this.

When you assign into a variable inside a parfor, in a variable that is intended to be output (not a temporary variable), then the output variable must be indexed by a simple arithmetic expression of the parfor index. If I recall correctly, division is not permitted, so it would not be valid to index prb(SNRindB/2+1)

The code you posted does not appear to use num_err_bits after the parfor, so perhaps you can turn num_err_bits into a local variable.

Suggested code:

SNRindBvals = 0:2:20;
parfor v = 1 : length(SNRindBvals)
   SNRindB = SNRindBvals(v);
   %your code
   prb(v) = as appropriate
   %and NO increment of v
end
plot(SNRindBvals, prb)

[Sajid Sarwar]

Dear @Walter Roberson thanks for ur answer. I have tried above mentioned code but it gives me following error. may u plz check it by running the code. I have attached other functions as well.

error

Assigning to the parfor-loop variable 'v' is not supported in parfor-loops

I will be grateful

[Sajid Sarwar]

Actually I want to reduce simulation time. it takes more than a week to get the results.

[Walter Roberson]

You did not remove the

v = v + 1;

line.

[Sajid Sarwar]

Dear @Walter Roberson, I have checked by by removing above line. also i have defined varibles with zeros outside every loop. but now it gives following error,

Error using FourbyFour_mimo_ofdm_IM_demoApraz (line 30)

Index exceeds the number of array elements (1).

I could not get where the problem is now. I have attached the updated file

I will be grateful for ur cooperation.

[Walter Roberson]

Updated code is below. It removes all the obvious initialization and indexing problems.

I have no idea how long the current version of the code takes. I added a progress bar, but it has not managed to finish any iterations on my system.

The progress bar I added is https://www.mathworks.com/matlabcentral/fileexchange/71083-waitbar-for-parfor -- to make use of that part you will need to install that contribution.

[Sajid Sarwar]

@Walter RobersonWalter Roberson Thank you very much

[Walter Roberson]

There is an assignment to kk that does not have a semi-colon after it... you should add the semi-colon.

[Walter Roberson]

Apparently it took about 2000 seconds to get through the first 18% (4 iterations), and apparently it took 18346 seconds to get through the first 36% (8 iterations). That is a lot slower for the second set.

If it were able to continue the 18346 for 8 iterations pace, then it would take about 31 hours to finish. I am not convinced.

[Walter Roberson]

A few minutes ago it updated to 51417 used for 45%.

I realized I was miscounting before: As there are only 11 total iterations, 18% should have meant 2 iterations done, 36% should have meant 4 iterations done, and now 45% would correspond go 6 iterations done. Whis is a bit puzzling as it always seems to be jumping by 9% but the pool is 4 cores.

So if 51417 is 45% then...

hours(seconds(2000)/.18)

ans = 3.0864

hours(seconds(18345)/.36)

ans = 14.1551

hours(seconds(51417)/.45)

ans = 31.7389

... It's getting slower. It looks to me as if each additional iteration is pretty much twice the time of the previous one.

I cancelled the program...

[Sajid Sarwar]

Dear @Walter Roberson, I am very thankful for ur cooperation. My simulation is still running, it says 81% done as u can see in the attached screenshot. Although, it's still slow but it's better than simple for loop (without parallel pool) because it was taking more than 10 days. Let's see the final result. It's getting slow.

[Walter Roberson]

You might have as little as 11 hours left (relative to the time you posted.)

[Sajid Sarwar]

Dear @Walter Roberson elapsed time is 388529.958732 seconds, almost 4.5 days. There is 50% improvement in simulation time compared to conventional way but still it's slow. I was using ubuntu 16.04. Is there some better way which could further reduce simulation time? Thanks for your cooperation.

[Walter Roberson]

Some of your loops can be replaced by reshape() or permute()

I have not studied what your code actually does.

Have you profiled the code to see where the time is going?

[Sajid Sarwar]

I have not profiled the code to see time. Although, I will try to replace loops by reshape