Applying a K-Fold cross validated model to predict the response variable for new data

Hi,

I have trained a k-fold cross-validated model using the fitctree classification model. Here is my code:

% Extract predictors and response
predictors = featTable(:, 1:end-1);
response = featTable.ActivityID;

% For reproducible results
rng default

% Partition the data using kFold method
cvp = cvpartition(response,'KFold',5); % setting KFold to 5 gives the lowest loss

% Calculate the model loss
KFoldMdl = fitctree(featTable,'ActivityID', "CVPartition",cvp);
kfLoss = kfoldLoss(KFoldMdl)    %model loss: 0.0128

I now want to apply my newly trained k-fold corss-validated model to classify new data that the model hasn't seen before. I used the following code:

Predictions = predict(KFoldMdl, predictorsNew)

When I run the code on my new unclassified predictors, I get the following error message:

Error using predict

No valid dataset found for the "predict" command. Specify a dataset using an iddata object, a timetable object, idfrd

object, or numeric matrices.

I'm not sure why this is not working as the code above works for the model if it is trained using the holdout option in cvpartition. I've attached my new predictors file. So could you please help with advising how I can apply a k-fold trained cross-validated model to new data? I tried using kfoldPredict but that doesn't work either.

Any help would be most appreciated.

[the cyclist]

It would be helpful if you uploaded all the data needed to run the above code, which looks like it would be

• featTable, or both predictors and response
• predictorsNew

Uploading a subset of the data the gives the same behavior would be fine.

I don't think this is likely a data-specific problem, but it's still easier if we can just replicate what you are doing, rather than create an example of our own for debugging.

Answer 1

[Aakash]

You can use the predict function in MATLAB to predict responses using the cross-validated model KFoldMd and the new data predictorsNew.

The code would look like this:

y_predict = predict(KFoldMd.Trained{1}, predictorsNew);

Note that KFoldMd.Trained{1} is the trained model for the first fold in the cross-validation, you can use any fold that you think has the best performance.

[Impala]

Hi Aakash,

Many thanks! This works perfectly! Much appreciated.

Can I ask, how do you check the performance of a fold?

Thank you once again!

[the cyclist]

Although the method in this solution gives a prediction, I would recommend against it, on theoretical grounds.

The point of doing multiple folds is not to find the fold that gives the best performance. That fold is likely to be over-fit to the training data, and will not be expected to generalize well to new data. It basically defeats the purpose of doing multiple folds in the first place.

Instead, it would be better to use the predictions from all the folds, combined in some way that makes sense. For example, if you are using 5-fold classification, you might chose the prediction that is chosen by the majority of the folds. (You also can get a sense of the error, if the different folds make different predictions.)

Another alternative would be to use fitcensemble to choose a single model. (You can still use k-fold crossvalidation in that framework, but the output is not the k different models for the k folds.)

[Impala]

I see your point about the predict function not being the best to give an accurate prediction.

So what is the syntax if I want to combine the predictions from all folds? Can I still use kfoldPredicts?

[Aakash]

No using kfoldPredict you cannot predict for any new data, you can just see the output labels for entire data that is given to the model through fitctree function.

For checking performance of folds, you can write a function on your own where your predict using the above method I suggested and compare it with truth values.

Answer 2

[the cyclist]

Immediately after writing my comment above (which is still true), I think I see the answer without the additional info.

For k-fold models, you need to use kfoldPredict (for classification tree models).

[Impala]

Hi,

Thanks for your feedback.

Apologies for not including the featTable data (which I've now attached) - I thought it wasnt required as that was the data I had used to train the model, the code for which worked fine.

I was having trouble applying the model to new data (NewPredictors file). I had tried to use the kfoldPredict function but this onlye requires the model as an input - I couldn't see how to include the new predictors in this function. Could you kindly help with this or shall I just stick to using the predict function?

Thanks in advance!