Novice - Help in Understanding the Cross Validation Folds in MATLAB ML Learner Apps

Hi! I wanted to have an understanding of how the MATLAB ML CV fold works, specifically for the Regression Learner App. I have a dataset of 21 rows and 6 columns. When I give fold values of 5, 10 and 15, usually, for the 5 and 10 cases, the models train fine. But, when I use a 15-fold CV case, there are times where the model has an RMSE of 4000, when normally for the 5 and 10 fold cases, it averages near 400.

My understanding is that for 15 folds, there will be 6 folds containing two data points, and 9 folds containing only one data point. Does the fact that there are more singular data points have anything to do with this?

One more thing I noticed while training the optimizable models, the minimum MSE plot, which usually shows the MSE vs Iteration Plot for the 5 fold and 10 fold cases, does not show any variation, while on the right hand side, the Regularization Strength keeps on updating per iteration, and at the end of it, all the points the model has predicted is just at the bottom, most of the time, in the Response Plot.

A final query - I have two datasets with me, one containing 21 rows, the other containing approximately 160. When I give a value for the folds, for the 21 row dataset, it mentions to put values between 2 and 19, while for the 160 row dataset, it says 2 and 50. Is there a particular reason why that is? I can understand that for 21 data, anything above 19 absolutely does not make sense, but for the 160 data, why is it 50?

Any sort of help in understanding this much appreciated, as I have just started my ML journey.

Answer 1

[Tridib]

Dear @Vivek Gopal,

1. In k‑fold cross‑validation with only 21 rows, using a large k (like 15) creates folds with just one or two samples. When a fold has only one test sample, its error is driven by a single prediction, so even a small miss can make that fold’s RMSE spike, which explains jumps from ~400 to ~4000.
2. For optimizable models, MATLAB updates hyperparameters each iteration and evaluates them via CV. With 15 folds on 21 rows, the validation loss is highly noisy and inconsistent across folds, so the optimizer cannot detect a clear improvement trend and the “Minimum MSE vs Iteration” line appears flat despite parameter changes.
3. Predictions collapsing to the bottom occur because the training sets are tiny. Under some hyperparameters the model becomes overly biased and can default to near‑constant predictions. The noisy CV losses make the objective unreliable.
4. MATLAB limits the maximum number of folds to 50 in the case of 160 row dataset to keep training practical and results more stable, preventing very slow runs and overly volatile error estimates.

Hope this helps!