Create new variable from an existing variable but deleting some outliers

Hello!

I'm working with a NetCDF file and I have values of T,S,density,depth, chlorophyll and oxygen measured in time.

I plotted a vertical profile of oxygen with depth and I saw there were some outliers that need to be removed. So, I would like to create a new variable (oxycor) from the original oxygen variable but without those values. I would like to remove those values right in the corner with concentration of oxygen equal to zero.

I tried this and then plotting with 'glider.oxycor' but it says vectors must be the same length.

oxycorrection2=find(glider.oxy>150);
glider.oxycor=glider.oxy(oxycorrection2);

What am I doing wrong?

Thank you!!

[Walter Roberson]

Is glider a struct() or is it a table()

If it is a table, then you are trying to have the glider.oxy column be full length but the glider.oxycor column be shorter.

[Alejandra Uguet de Resayre]

Yes it is a table.

I thought about making those values NaN, but I don't want to modify the raw variable, just create a new one without them.

What could I do?

Answer 1

[Walter Roberson]

mask = glider.oxy>150;
glider_subset = glider(mask,:);

[Alejandra Uguet de Resayre]

Thank you so much!

How would it be in case it were a struct()?

Best :)

[Walter Roberson]

Struct case

mask = [glider.oxy] > 150;
glider_subset = glider(mask);

Answer 2

[Steven Lord]

Remove all rows in the table that have an outlier value for the oxy variable. Here's a similar example using the sample patients data.

load patients
P = table(LastName, Age, Height, Weight);
P2 = P; % Make a backup copy before outlier removal
fprintf("P has %d rows before outlier removal\n", height(P))

P has 100 rows before outlier removal

head(P)

ans = 8×4 table
      LastName      Age    Height    Weight
    ____________    ___    ______    ______

    {'Smith'   }    38       71       176  
    {'Johnson' }    43       69       163  
    {'Williams'}    38       64       131  
    {'Jones'   }    40       67       133  
    {'Brown'   }    49       64       119  
    {'Davis'   }    46       68       142  
    {'Miller'  }    33       64       142  
    {'Wilson'  }    40       68       180  

Let's say outliers are patients that are younger than 40.

tooYoung = P.Age < 40;
P(tooYoung, :) = [];
fprintf("P has %d rows after outlier removal\n", height(P))

P has 44 rows after outlier removal

head(P)

ans = 8×4 table
      LastName      Age    Height    Weight
    ____________    ___    ______    ______

    {'Johnson' }    43       69       163  
    {'Jones'   }    40       67       133  
    {'Brown'   }    49       64       119  
    {'Davis'   }    46       68       142  
    {'Wilson'  }    40       68       180  
    {'Anderson'}    45       68       128  
    {'Thomas'  }    42       66       137  
    {'Martin'  }    48       71       181  

Note that several patients younger than 40 appeared in the table before outliers were removed, but afterwards the first patient in the resulting table was Johnson at 43 years old.

Now if you wanted to keep the data but not plot it, you could do that either by just using logical indexing or, if you want to compute with it more often, adding that as a new variable to the table.

P2.tooYoung = tooYoung;
head(P2)

ans = 8×5 table
      LastName      Age    Height    Weight    tooYoung
    ____________    ___    ______    ______    ________

    {'Smith'   }    38       71       176       true   
    {'Johnson' }    43       69       163       false  
    {'Williams'}    38       64       131       true   
    {'Jones'   }    40       67       133       false  
    {'Brown'   }    49       64       119       false  
    {'Davis'   }    46       68       142       false  
    {'Miller'  }    33       64       142       true   
    {'Wilson'  }    40       68       180       false  

plot(P2{tooYoung, "Height"}, P2{tooYoung, "Weight"}, 'o'); % or

figure
scatter(P2{P2.tooYoung, "Height"}, P2{P2.tooYoung, "Weight"})