Classification Learner App vs. Training and testing a model programmatically, Is there any hidden magical step in the classification learner app?

I am trying to find a good model to explain my dataset. The problem is that I want to do leave-one-person-out cross validation which is not available in the App. So I trained different models (e.g. Tree, SVM, KNN, LDA) using functions like fitctee, fitcsvm, fitcknn, and fitcdiscr. Following the leave-one-person-out procedure I have found average classification accuracy of about 70% for the best model. However, when I use the App to model the data using 10-Fold cross validation, it has much better accuracy and TPR and TNR about 98%. This is really confusing that why this is happening! I was wondering if there are some steps I am missing when I do the modeling programmatically. Or is there any way to do what the App does by writing scripts and probably customizing the cross validation scheme to leave-one-person-out?

Answer 1

[Stephan]

Hi,

A possible way to do this is working with the app and then, when you got a good result, export the code to matlab. This allows you to see the magic steps that are made and modify the code you got, if needed.

There is some more information here.

I could imagine that this procedure will solve your problem.

Best regards

Stephan

[RZM]

Hi Stephan, Thanks for your suggestion.

I actually did that too. I exported the best model. I used the same model properties to define a new model to be trained and tested in the leave-one-person-out procedure. The model performance did not improve at all. Another possibility was if I could have access to one of the models trained by one folding, and to check its response for the unseen fold programmatically. But the only model that can be exported is the best model among all folds which is trained by all the dataset.

BR

[Stephan]

Im not in the near of my computer, but if i remember correctly, then you can choose the number of folds in the app. If you export the code of the model you should also be able to change the number of folds.

[RZM]

Yes, but there is no way to specify/customize the samples for each fold as it is required in leave-one-person-out. Note that it is different with leave-one-out since there are more than one sample which belongs to each subject/person.

[Stephan]

Ok. Then this way is not helpful. My knowledge about this topic ends at this point. I would say, that ether you do the way you want it and accept a lower model quality or you take a good perfoming model and leave out leave one out.

Best regards

Stephan

[RZM]

Thank you very much Stephan, I would rather to choose the one with higher performance but I am afraid if this K-Fold CV does not take inter-subject variability into account. In other words, I am not sure which one of these two CV approaches has higher power in terms of generalization. Regards

[Stephan]

I wish you success.

Best regards

Stephan