Picking only 2 nested for loops among several.

I have a very complex code structure with nested for loops.

Some days i loop over 2 given variables, some days two other ones, some days only one.

I want to limit the number of lines i have to edit every day when i want to run a different version of the code.

params = initialization(params_file_path);
for ii_outer_variable_1 = 1:length(params.outer_variable_1_values)
    outer_variable_1 = params.outer_variable_1_values(ii_outer_variable_1);
    ...
                for ii_outer_variable_5 = 1:length(params.outer_variable_5_values)
                    outer_variable_5 = params.outer_variable_5_values(ii_outer_variable_5);
                    array_1 = get_file_1(outer_variable_1, outer_variable_2, outer_variable_5);
                    array_2 = get_file_1(outer_variable_3, outer_variable_4, outer_variable_5);
                    [array_1, array2] = manipulate_and_combine_those_files(array_1, array2, params);
                    plot_some_visuals_1(array_1, array2, params);
                    for ii_middle_variable_1 = 1:length(params.middle_variable_1_values)
                        middle_variable_1 = params.middle_variable_1_values(ii_middle_variable_1);
                        [array_1, array2] = reshape_my_arrays(array_1, array2, middle_variable_1, params);
                        plot_some_visuals_2(array_1, array2, params);
                        for ii_inner_variable_1 = 1:length(params.inner_variable_1_values)
                            inner_variable_1 = params.inner_variable_1_values(ii_inner_variable_1);
                            ...
                                    for ii_inner_variable_4 = 1:length(params.inner_variable_4_values)
                                        inner_variable_4 = params.inner_variable_4_values(ii_inner_variable_4);
                                        result = iterative_script(array_1,array_2,params,inner_variable_1,inner_variable_2,inner_variable_3,inner_variable_4); % This iterative_script has hundreds of iterations of large calculations and thus runs for hours. The main script runs for 10h to a few days.
                                        % I don't fill the results cell array yet otherwise Matlab would run out of memory.
                                        savename = sprintf("%s%d%s%d%s.mat",params.path,variable_of_the_day_1,name_of_variable_of_the_day_1,variable_of_the_day_2,name_of_variable_of_the_day_2);
                                        save(savename,"result");
                                    end
                                end
                            end
                        end
                    end
                end
            end
        end
    end
end
clear all -except params
results = cell(length(params.variable_of_the_day_1_values),length(params.variable_of_the_day_2_values));
for ii_variable_of_the_day_1 = 1:length(params.variable_of_the_day_1_values)
    variable_of_the_day_1 = variable_of_the_day_1_values(ii_variable_of_the_day_1);
    for ii_variable_of_the_day_2 = 1:length(params.variable_of_the_day_2_values)
        variable_of_the_day_2 = variable_of_the_day_2_values(ii_variable_of_the_day_2);
        results{ii_variable_of_the_day_1,ii_variable_of_the_day_2} = load(sprintf("%s%d%s%d%s.mat",params.path,variable_of_the_day_1,name_of_variable_of_the_day_1,variable_of_the_day_2,name_of_variable_of_the_day_2)).result;
    end
end
my_display(results);

So, some days i will for example have both params.outer_variable_1_values and params.middle_variable_1_values be a 1*10 array each, and all other params.xx_values arrays be only 1*1 scalar arrays to avoid having incredibly complex file names (and so many result files that i can't even plot them vs. each other without flooding the screen).

I don't have hopes of reducing the number of nested for loops, but at least i don't want to everyday have to rewrite every input of every sprintf() call that i run for fig titles and filesave names, and rewrite every field of the two for loops at the end that display the results.

Can i somehow use "@" (handles to variables) tricks for that ?

Answer 1

[Matt J]

Get rid of all the nested loops. Use the combinations command to create a table of all combinations of your variables. Then, you can just use a single for-loop across the rows of the table to get each combination.

T=combinations(1:3,1:2,1:4)

T = 24x3 table
    Var1    Var2    Var3
    ____    ____    ____

     1       1       1  
     1       1       2  
     1       1       3  
     1       1       4  
     1       2       1  
     1       2       2  
     1       2       3  
     1       2       4  
     2       1       1  
     2       1       2  
     2       1       3  
     2       1       4  
     2       2       1  
     2       2       2  
     2       2       3  
     2       2       4  

[m,n]=size(T);

s=table2struct(T);
c=table2cell(T);
nameTemplate="name"+repmat('_%d',1,n)

nameTemplate = "name_%d_%d_%d"

for i=1:m
    
   fname=sprintf(nameTemplate,c{i,:});
    
   si=s(i);
   save(fname,'-struct','si');
    
end

dir '*.mat'

name_1_1_1.mat  name_1_1_4.mat  name_1_2_3.mat  name_2_1_2.mat  name_2_2_1.mat  name_2_2_4.mat  name_3_1_3.mat  name_3_2_2.mat  
name_1_1_2.mat  name_1_2_1.mat  name_1_2_4.mat  name_2_1_3.mat  name_2_2_2.mat  name_3_1_1.mat  name_3_1_4.mat  name_3_2_3.mat  
name_1_1_3.mat  name_1_2_2.mat  name_2_1_1.mat  name_2_1_4.mat  name_2_2_3.mat  name_3_1_2.mat  name_3_2_1.mat  name_3_2_4.mat  

[Matt J]

Also, you would do well to learn about programmatically generating comma-separated lists,

https://www.mathworks.com/help/releases/R2024b/matlab/matlab_prog/comma-separated-lists.html

[Arnaud]

This solution would require all my loops to be outer loops (which obviously is possible but makes the code way slower as it implies reloading the same exact files again and again and reshaping them the exact same way again and again), and it would definitely not solve my issue.

Let's say today i have var1 which varies between diffenrent weights (of a ball or else) and var2 varies between different values of an abstract mathematical coefficient used to control the behavior of iterative_script().

Let's say tomorrow i fix the weight of the ball to a value i liked from today and the mathematical coefficient to a value i like from today, and, still tomorrow, I want to study a 2D sweep where var1 is the length of a road and var2 tries different reshape geometries of array1 and array2.

Then i have to rewrite all my code so that i don't input var1 where the weight input goes anymore but i input var1 in functions in the input slot where the road length usually goes. Same for var2.

With your solution my code would place inputs incorrectly in functions.

With your solution my saved files would all be named 1_1, 1_2 etc and would overwrite each other from the previous day. Even if i add the date of the day in the file name, those file names wouldn't let me keep track of which variables were swept on that day.

[Arnaud]

"you would do well to learn about programmatically generating comma-separated lists"

I do not see where i would need them here. Can you be more specific please ? I believe i already know how to use them.

[Matt J]

I don't see why. You would just regenerate the combinations you want to sweep on a given day. The table of combinations can include values you want held fixed,e.g.,

T=combinations(1:3,2,1:4) %middle variable fixed

T = 12x3 table
    Var1    Var2    Var3
    ____    ____    ____

     1       2       1  
     1       2       2  
     1       2       3  
     1       2       4  
     2       2       1  
     2       2       2  
     2       2       3  
     2       2       4  
     3       2       1  
     3       2       2  
     3       2       3  
     3       2       4  

Same thing with the file names. There is no conceptual difference between varying function names combinatorically and varying data combinatorically.

[Stephen23]

"This solution would require all my loops to be outer loops (which obviously is possible but makes the code way slower as it implies reloading the same exact files again and again and reshaping them the exact same way again and again), and it would definitely not solve my issue."

No, you can in fact arrange it to be exactly equivalent to nested loops. COMBINATIONS returns the input values in lexicographic order. As long as you arrange its inputs such that the slowest changing variables correspond to ones which require importing file-data and you detect when there is a change in the corresponding output element (which is trivial for numeric indices) then you can import file-data just as often as you would with lots of nested loops.

"With your solution my code would place inputs incorrectly in functions."

No, that is not what Matt J suggested.

"With your solution my saved files would all be named 1_1, 1_2 etc and would overwrite each other from the previous day. "

No that is definitely not what Matt J suggested. The given answer lets you specify all of the function inputs just as you would with nested loops: if you can generate some particular combinations of values using nested loops then there is absolutely nothing stopping you from generating those combinations of values using a function like COMBINATIONS. Yes it requires changing how you think about your task and yes it requires refactoring your code, but Matt J's solution understands the essence of the problem and provides a neat, extensible, easily generalizable solution which will let you avoid this: "...lines i have to edit every day when i want to run a different version of the code" by letting you use data to control the flow. For code that takes a long time to run it could also be used to stop-and-start calculations (by storing the current combinations row index).

"I do not see where i would need them here. Can you be more specific please ?"

By replacing all of those anti-pattern numbered variables with a few cell arrays.

[Matt J]

"you would do well to learn about programmatically generating comma-separated lists" .I do not see where i would need them here. Can you be more specific please ?

I used one in this line,

fname=sprintf(nameTemplate,c{i,:});

Instead of passing 3 explicit inputs to sprintf, I used c{i,:} to pass all 3 implicitly. This is generally the kind of thing you do to avoid hardcoding for a specific number of variables.

[Arnaud]

I think we still don't understand each other.

First, what you're focusing on (and that probably cannot be fixed in my opinion):

Let's say i use combinations.

First iteration, all my variables are at their first value. Fine.

Second iteration, my last variable (inner_variable_4) goes to its second value. Since the for loop is written at the top of my script, the for loop will re-run the two get_file_1 calls (which each take 30s), the manipulate_and_combine_those_files() call, the plot_some_visuals_1 call, the reshape_my_arrays call, the plot_some_visuals_2 call, the iterative_script and the save. I do not want that. I do not want to re-calculate the same stuff again. I do not want to plot the same figs again. I want this combination to run only the interative_script and the save.

Second, the part that sounds more feasible to me:

In this example i show 10 variables (5 outer, 1 medium, 4 inner). In the most general case i believe i actually have 9 variables to study on the outermost loop (the one that picks input files) then i have 7 layers of intermediate loops that are different ways of preparing the data before the iterative_script, then 4 inner variables that affect the behavior of the iterative_script only. Which means that i cannot save this information in the file name because it would be an unreasonably long string. (Instead, at each combination i save in the results file alongside my results a table of which value each of these variables was set to.) I still want my filenames to reflect the information of what 2 variables were swept that day and what value this result file corresponds to for each of these two variables. So i want an sprintf call that i never have to modify manually, which detects which 2 of the 9+7+4 variables have a length greater than 1 (for that day), and which prints the current value for only these two variables. I feel like the only way would be to use handles. sprintf(nameTemplate,c{i,:}) would easily make a 50+ characters long filename and Windows/OneDrive would never accept that since i already placed all this study deep in the Windows Explorer folder tree.

[Arnaud]

"and you detect when there is a change in the corresponding output element"

Do you mean i should have this combinations outer loop that starts with:

if c{i,1} ~= c{i-1,1} || ... || c{i,9} ~= c{i-1,9}

array_1 = get_file_1(outer_variable_1, outer_variable_2, outer_variable_5);

array_2 = get_file_1(outer_variable_3, outer_variable_4, outer_variable_5);

[array_1, array2] = manipulate_and_combine_those_files(array_1, array2, params);

plot_some_visuals_1(array_1, array2, params);

end

if c{i,1} ~= c{i-1,1} || ... ... || c{i,9+7} ~= c{i-1,9+7}

[array_1, array2] = reshape_my_arrays(array_1, array2, middle_variable_1, params);

plot_some_visuals_2(array_1, array2, params);

end

result = iterative_script(array_1,array_2,params,inner_variable_1,inner_variable_2,inner_variable_3,inner_variable_4);

savename = sprintf("%s%d%s%d%s.mat",params.path,variable_of_the_day_1,name_of_variable_of_the_day_1,variable_of_the_day_2,name_of_variable_of_the_day_2);

save(savename,"result");

like this ?

[Matt J]

Since the for loop is written at the top of my script, the for loop will re-run the two get_file_1 calls (which each take 30s)

I don't know what "written at the top of my script" means. A loop behaves the same no matter how many lines from the top of the file it is.

Regardless, Stephen already explained that you can make the file reading and whatever else conditional, so that it only happens when the loop counter for that variable updates.

So i want an sprintf call that i never have to modify manually, which detects which 2 of the 9+7+4 variables have a length greater than 1 (for that day), and which prints the current value for only these two variables.

So, use conditional logic for that, too,

longest=strlength( c(i,:) )>1;

assert(sum(longest)==2,'There should be exactly 2 variables of length>1')

sprintf('name_%s_%s', c{i,longest});

[Arnaud]

"use conditional logic for that, too"

Yes, i think that would work, thanks i'll try it !

[Arnaud]

"Stephen already explained that you can make the file reading and whatever else conditional"

Yes, i'm sorry i wrote that before reading the "and you detect when there is a change in the corresponding output element" part.