Doubts about cross-validation

I'm trying to test performance of a KNN and a svm (cecoc) classifier for a multiclass problem.

For the knn model I do it by building mdlknn, which is a cross validated model (leave one out) and then using kfoldLoss function

 mdlknn = fitcknn(X,labels, 'NumNeighbors', ...
                   k, 'Distance',@distKNN, 'Leaveout','on',...
                    'HyperparameterOptimizationOptions','UseParallel');
 perf  = 1-kfoldLoss(mdlknn);

Is this correct? My doubts regards the cross-bvalidation:

Using option ('Leaveout','on') in fitcknn is the same as not using it and then calling crossval on the trained model?

 mdlknnNoCrossVal = fitcknn(X,labels, 'NumNeighbors', ...
                   k, 'Distance',@distKNN, 'HyperparameterOptimizationOptions','UseParallel');
 mdlknn = crossval(mdlknnNoCrossVal, 'leaveout','on')
 perf  = 1-kfoldLoss(mdlknn);

Next, since in knn classifier I'm using my distance, I would like to do the same with the cecoc. How can I do that?

At this stage, the code I use for building the cecoc doesn't use it. how can I set it?

  t = templateSVM('Standardize',true,'KernelFunction','rbf');
  mdlsvm = fitcecoc(X,labels, 'Leaveout','on',...
                    'HyperparameterOptimizationOptions','UseParallel',...
                    'coding', 'ternarycomplete','Learners',t);
  perf = 1-kfoldLoss(mdlsvm);
           

And again: Using option ('Leaveout','on') in fitcecoc is the same as not using it and then calling crossval on the trained model, right?

THANKS A LOT

Answer 1

[Jyothis Gireesh]

I am assuming that you want to have some clarification regarding the cross-validation syntax in KNN and SVM

• KNN Classifier

mdlknn = fitcknn(X,labels, 'NumNeighbors', ... 
                   k, 'Distance',@distKNN, 'Leaveout','on',... 
                    'HyperparameterOptimizationOptions','UseParallel'); 
 
mdlknnNoCrossVal = fitcknn(X,labels, 'NumNeighbors', ... 
                   k, 'Distance',@distKNN, 'HyperparameterOptimizationOptions','UseParallel'); 
mdlknn = crossval(mdlknnNoCrossVal, 'leaveout','on') 

These two syntaxes may be equivalent for the creation of a cross-validated KNN model. “mdlknn" is a "ClassificationPartitionModel" classifier. "mdlknnNoCrossVal” creates a "ClassificationKNN" classifier which is cross validated using the “crossval” function.

• ECOC classifier

In the case of ECOC classifier you may use the “templateKNN” as the learner. It provides the feature to customize the distance metric using the ‘Distance’ argument. The distance metric can be passed as a function handle like that of the KNN classifier syntax.

And the two syntaxes are valid for the process of cross validation in the case of ECOC also.

[Elena Casiraghi]

Thanks a lot!

this clarified what I was asking!