how to do logical indexing for, for and if loops to create a new dataset every time the condition is met.

Hi, i am trying to learn the loops in matlab and stuck in a situtaion. I have a time series dataset and i would like to do logical indexing of the table based on variable called global solar irradiance. Every time the value is between 180 and 400, the loops should create a new table based on whenever the condition is met. This would mean i should end up with around 30 to 40 or even more subsets/ tables . I have no idea where to begin. I know i would require a for and if loops for this .

I tried using simple logical indexing like

% refmean= 728

lower_threshold= 180 %0.01 * refmean

upper_threshold = 400 %0.80* refmean

highligting_indices=find(T3_filtered_final.("Global Solar Irradiance")>=lower_threshold & T3_filtered_final.("Global Solar Irradiance")<=upper_threshold)
New_Table=T3_filtered_final(highligting_indices,:)

This new table will have all the time intervals where the value of global solar irradiance meets the condition, but i would like to make subsets that whenever the conditions are met , the program should output a new table where the first value is always something near to 180 and last value near to 400. The new tables could have 1 second time gap or maybe 10 minutes time gap.

How to proceed with this program

for i =1:height(T3_filtered_final)% loop through whole table
    if T3_filtered_final.("Global Solar Irradiance")(i)>=180 && T3_filtered_final.("Global Solar Irradiance")(i)<=400
    % how to proceed
    end
    
end

sample data is provided within.

[dpb]

"... I know i would require a for and if loops for this ."

Not necessarily, no. MATrixLABoratory is named this for a reason and while looping is sometimes necessary, vectorized solutions are to be sought before resorting to brute force loops (although loops have become much less of a bottleneck with time and better optimizations than in the distant past where they got a bad reputation).

Let's see what's we got...I'm not positive I understand the intent but one other thing is the idea of building a zillion more tables as subsets is also generally not the proper approach; grouping variables and splitapply processing or rowfun or even retime with timetables can often do it all in a single or few lines of code...

load matlab2
whos

  Name                        Size              Bytes  Class    Attributes

  T3_filtered_final3      25000x22            4406135  table              
  ans                         1x36                 72  char               
  cmdout                      1x33                 66  char               

tT=T3_filtered_final3; clear T3* ans cmd*  % shorten table name for less typing
head(tT)

           Datetime            Latitude    Longitude    Altitude    Aircraft North Velocity    Aircraft East Velocity    Aircraft Down Velocity    Aircraft Roll    Aircraft Pitch    Heading    Air Temperature    Relative Humidity    Reflected Shortwave Irradiance    Emitted Longwave Irradiance    Global Solar Irradiance    Downwelling Longwave Irradiance    Wind Component North    Wind Component East    wind down comp    Surface Temperature    Ground_speed    Wind_speed
    _______________________    ________    _________    ________    _______________________    ______________________    ______________________    _____________    ______________    _______    _______________    _________________    ______________________________    ___________________________    _______________________    _______________________________    ____________________    ___________________    ______________    ___________________    ____________    __________

    2021-07-27T10:46:00.000     53.831      6.8643       122.7               66.7                      -35.2                      -0.2                 -0.5              2.8           -33.5          18.4                 85                          33                              411                          600                            354                          3.9                     5.9                  0.1                18.6               75.418         7.0732  
    2021-07-27T10:46:00.010     53.831      6.8643       122.7               66.7                      -35.2                      -0.2                 -0.5              2.8           -33.5          18.4                 86                          33                              411                          600                            354                          3.9                     5.9                    0                18.6               75.418         7.0725  
    2021-07-27T10:46:00.020     53.831      6.8643       122.7               66.7                      -35.2                      -0.1                 -0.4              2.8           -33.5          18.4                 85                          33                              411                          600                            354                          3.9                     5.9                    0                18.6               75.418         7.0725  
    2021-07-27T10:46:00.030     53.831      6.8643       122.7               66.7                      -35.2                      -0.2                 -0.4              2.8           -33.5          18.4                 86                          33                              411                          600                            354                          3.9                     5.9                  0.1                18.6               75.418         7.0732  
    2021-07-27T10:46:00.040     53.831      6.8642       122.7               66.7                      -35.2                      -0.2                 -0.4              2.8           -33.5          18.4                 85                          33                              411                          599                            354                          3.9                     5.9                    0                18.6               75.418         7.0725  
    2021-07-27T10:46:00.050     53.831      6.8642       122.7               66.7                      -35.2                      -0.1                 -0.4              2.8           -33.5          18.4                 85                          33                              411                          599                            354                          3.9                     5.9                 -0.1                18.6               75.418         7.0732  
    2021-07-27T10:46:00.060     53.831      6.8642       122.7               66.7                      -35.2                      -0.1                 -0.4              2.8           -33.5          18.4                 86                          33                              411                          599                            354                          3.9                     5.9                 -0.1                18.6               75.418         7.0732  
    2021-07-27T10:46:00.070     53.831      6.8642       122.7               66.7                      -35.2                      -0.2                 -0.4              2.8           -33.5          18.4                 85                          33                              411                          599                            354                          3.8                     5.8                    0                18.6               75.418          6.934  

tT.Properties.VariableNames=strrep(tT.Properties.VariableNames,' ','_');    % get rid of blanks in variable names
plot(tT.Datetime,tT.Global_Solar_Irradiance), xlim([tT.Datetime(1) tT.Datetime(end)]) % see what time series looks like
xlabel('Date'), ylabel('Global Solar Irradiance')

Well, there's no data in the attached data set that meets your criteria, the minimum value is >450.

So, now what would you propose that you want to do???

[arjun luther]

Thanks for initiating help. The minimum value of 450 could have been caused by me where i selected part of the .mat file to be used as sample here as the upload limit was 5mb.

We can change the limits so that i can understand how the process is done. How about if you can help to find all the intervals where the lowest was 500 and highest was 650. Extract the time steps in new tables every time the value was observed in that range.

The reason for extracting them and going for zillion tables is that those specific time intervals are to be studied individually and I can't think of better approach rather than making different tables for those specific times but suggestions for better approach is more than welcome.

[Stephen23]

"The reason for extracting them and going for zillion tables is that those specific time intervals are to be studied individually and I can't think of better approach rather than making different tables for those specific times but suggestions for better approach is more than welcome."

Splitting up data generally makes data harder to work with.

Use the tools MATLAB provides for working with groups of data (exactly as dpb already recommended and showed):

https://www.mathworks.com/help/matlab/matlab_prog/split-data-into-groups-and-calculate-statistics.html

https://www.mathworks.com/help/matlab/matlab_prog/perform-calculations-by-group-in-table.html

[arjun luther]

Thank you for your suggestion, i would look into these and try to find a solution bases on grouping . Although @dpb talked about grouping variables and other suggestions, he did not show any example. I tried using compute by group task in Matlab but still i cannot figure out how i will create a filter where everytime new group is formed when the values are in certain range of my interest variable and soon as they go out of range , the program should reset and look for the exact range again from last point.

[dpb]

Other than the two limits, we still don't have a precise definition of the selection criterion with which to work -- given the new range of 500-650 with the above time trace, where would you choose from in the above noisy trace?

load matlab2
tT=T3_filtered_final3; clear T3* ans cmd*  % shorten table name for less typing
tT.Properties.VariableNames=strrep(tT.Properties.VariableNames,' ','_');    % get rid of blanks in variable names
plot(tT.Datetime,tT.Global_Solar_Irradiance), xlim([tT.Datetime(1) tT.Datetime(end)]) % see what time series looks like
xlabel('Date'), ylabel('Global Solar Irradiance')
LO=500; HI=650;
yline(LO,'r-'), yline(HI,'r-')

Is the initial section from 10:46 until it drops below 500 in or out? What about the next double small peaks before drops below again? Or must there be an excursion out the other range, too, not just "within"?

Then, presuming it may be the excursions from LO to HI Is the interest just from the boundary crossings of the excursion between 10:47 and 10:48 or from the local minimum to the local maximum outside the boundaries? Similarly on the return from the peak...oh, so many options to choose...

[arjun luther]

The orignal data starts from 08 in the morning, but due to the upload limitation, i dissected a part of the orignal data to be used as sample here. When i said 500 to 600, i meant them as arbitrary limits. I have hihghlited the points using same limits in order to showcae what i would like to achieve.

I want the stamps:

1-2

3-4

5-6

8-9

to be seperatly extracted so all the assosciated variables in that time range could be studied for the specific time.

Basically i m interested in Dips that are formed in the plot and i want to extract all the dips under certain value in my main data. The data i provided here has coincidentally more peaks than dips, but if i we can figure out a program for peaks, tweaking it for dips would not be hard.

Does my task makes more sense now or is it still unclear what i am aimimg for?

[dpb]

Experiment with findpeaks remembering a dip in x is a peak in -x; the optional parameters in it allow quite a lot of tweaking of what is identified. Then there's islocalmin and its sibling that could also be helpful within subsections.

I'd suggest load the real dataset, zip it if have to or it's probable the decimation by 2 wouldn't change the problem drastically.

Or, you can get really devious and attach a couple of sections as two files to be joined...

[dpb]

Why would 5-6 be included but not 6-7???

[arjun luther]

5-6 has time gaps before and after, while annotating the graph i missed a point between 6-7, which would have started a little after 6. I opted for a little different approach for this whole scenario as i was not getting closer to the solution. I already had a program that would highlight me the dips. Usually between each dip there is a time gap of arbitrary length but more than 5 seconds. I made a logical indexing to find the indices of those time gaps and using these time gap indices, i created my new subsets ,i.e, using adjacent indices. So far the tables look fine to me with this aproach.

Answer 1

[SAI SRUJAN]

Hi Arjun luther,

I understand that you have a time series dataset in MATLAB and you want to create tables based on a condition involving the "Global Solar Irradiance" variable. Specifically, you want to create new tables whenever the value of "Global Solar Irradiance" is between 180 and 400.

You can follow the given example to proceed further.

% Set the lower and upper thresholds
lower_threshold = 180;
upper_threshold = 400; 

% Initialize variables
subsetTables = {}; % Cell array to store subset tables
startIndex = 1; % Index to track the start of each subset - starting index

for i = 1:height(T3_filtered_final)
    if T3_filtered_final.("Global Solar Irradiance")(i) >= lower_threshold && T3_filtered_final.("Global Solar Irradiance")(i) <= upper_threshold
        % If the condition is met, update the start index
        if startIndex == 0
            startIndex = i;
        end
    else
        % If the condition is not met and a subset has started,   
        % create a new table and insert it.
        if startIndex ~= 0
            endIndex = i - 1;
            subsetTables{end+1} = T3_filtered_final(startIndex:endIndex, :);
            startIndex = 0; % Reset the start index
        end
    end
end

% Check if a subset is still in progress at the end of the loop
if startIndex ~= 0
    endIndex = height(T3_filtered_final);
    subsetTables{end+1} = T3_filtered_final(startIndex:endIndex, :);
end

% You will end up with a cell array of tables 
% which are created whenever conditon is met
% Access each subset table using subsetTables{index}

You can refer to the following documentation to understand more about "table" in MATLAB,

https://in.mathworks.com/help/matlab/ref/table.html