How does the "sequentialfs" function decide which feature to choose (or drop) in each iteration?

Dear sir or madam,

I am a graduate student working on biomedical data analysis. Thanks to your wonderful product (Statistics and Machine Learning Toolbox), I was able to apply the latest machine learning techniques to my work.

My question regarding the "sequentialfs" function:

I understand that it keeps adding (in "sequential forward selection" mode) features until the criterion does not decrease any more. How does the algorithm decide which feature to add in each iteration? Does it take a greedy approach? E.g., in each iteration, does it try adding each of the remaining features and choose the one that decreases the criterion the most?

Thank you.

Best regards, Xun

Answer 1

[Walter Roberson]

"Starting from an empty feature set, sequentialfs creates candidate feature subsets by sequentially adding each of the features not yet selected. [...]

After computing the mean criterion values for each candidate feature subset, sequentialfs chooses the candidate feature subset that minimizes the mean criterion value. This process continues until adding more features does not decrease the criterion."

[Xun Wu]

According to the online documentation, the answer to my question seems a "yes", doesn't it? But I am still not very sure. Here is the pseudocode I wrote for sequentialfs. Could you take a look at it and let me know whether it matches with what is happening inside the function?

selectedFeature = [] #represented with positive integers, e.g. 1,2,3 ...
unSelectedFeature = fullFeatureSet
decrease = true
criterion = +Inf
while decrease==True:
  nextFeatureToAdd = -1
  for feature in unSelectedFeatures:
    selectedFeatureCopy = selectedFeature.createCopy()
    selectedFeatureCopy.add(feature)
    newCriterion = calculateCriterionWithFeatureSet(selectedFeatureCopy)
    if(newCriterion<criterion):
      criterion = newCriterion
      nextFeatureToAdd = feature
  if nextFeatureToAdd==-1:
    decrease = False
  else:
    selectedFeature.add(nextFeatureToAdd)
                unSelectedFeature.remove(nextFeatureToAdd)
return selectedFeature

[Walter Roberson]

You can view the code yourself

edit sequentialfs

The portion of interest starts about line 325.