Filter löschen
Filter löschen

Why is the accuracy reported in the Classification Learner app much higher than the accuracy of the exported model on the held out validation set?

5 Ansichten (letzte 30 Tage)
Problem description
I used the default 5-fold cross-validation (CV) scheme in the Classification Learner app and trained all the available models. The best model (quadratic SVM) has 74.2% accuracy. I used
export model => generate code
and then ran the generated code, again examining the 5-fold CV accuracy. Surprisingly, the validation accuracy of this generated model was 66.8%, which is much lower than that reported by the app.
This has happened repeatedly - everytime I follow this scheme (with different datasets and models) I observe reduced accuracy for the exported model.
Potential explanations
I understand that not resetting the random number generation seed may lead to some variability between runs, but this effect seems too large and consistent to be explained by chance. I also understand that choosing the empirically-best model on the validation set, without having an additional test set, may lead to overestimation of its performance. If the exported model has a different CV partition, that could explain some of the reduction in accuracy, but I wonder if there might be some other explanation that I am missing.

Antworten (1)

Jalaj Gambhir
Jalaj Gambhir am 6 Mär. 2020
You can have a look at the answer here. Hope it helps!
  1 Kommentar
asaf benjamin
asaf benjamin am 10 Mär. 2020
Thanks for the response, but that answer deals with a very different issue, where one mistakingly compares cross-validation accuracy to accuracy on the training data. Under no circumstances would I try to test my model's performance on the data on which it trained. Rather, I am comparing
  1. The 5-fold cross-validation accuracy reported by the app, to
  2. The 5-fold cross-validation accuracy reported by the code generated by the app.
These should be equal (or at least very close), as I understand it, but they are in fact very different...

Melden Sie sich an, um zu kommentieren.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by