Importance of cross-validation

This example illustrates that omitting cross-validation can result in misleadingly high goodness-of-fit due to overfitting

randomCrossValidation.m illustrates that omitting cross-validation can result in misleadingly high goodness-of-fit due to overfitting

DESCRIPTION

Random Poisson distributed matrix x and vector y are fitted with Poisson Generalised Linear Model [1] and goodness-of-fit is estimated for two cases:
1 Without cross-validation which results in high pseudo-R2 (pR2) value
2 With cross-validation which gives correct low pR2 value

Misleadingly high pR2 value and good fit without cross-validation is due to overfitting. This means that the model fits too much to the data without taking into account essential properties of the data, i.e. this model cannot explain any not used in the training set values.

Note, that pR2 measure is a common goodness-of-fit measure for Poisson distributed data, see [2,3] for pR2 definition and [4] for its MATLAB implementation.

INPUT

nPoints - number of predicted points ( 1 x 1 )
nCovariates - number of predicting covariates ( 1 x 1 )
x - matrix of covariates ( nCovariates x nPoints )
y - response variable ( 1 x nPoints )

OUTPUT

pR2 - pR2 value of fit when not using cross-validation ( 1 x 1 )
pR2crossValidated - pR2 value of fit when using cross-validation ( 1 x 1 )
yEstimated - estimated values of y from x when not using cross-validation ( 1 x nPoints )
yEstimatedCrossValidated - estimated values of y from x when using cross-validation ( 1 x nPoints )

EXAMPLE OF USE

1 Upload scripts using Download button
2 Run randomCrossValidation.m (it will take a few minutes, uncomment line 107 and run plotting section if you do not want to wait)
3 Have a look at the plotted figures for different number of observations (points in response variable)

REFERENCES

[1] J.A. Nelder and R.J. Baker. Generalized linear models. Wiley Online Library, 1972.
[2] Heinzl, H. and Mittlboeck, M., 2003. Pseudo R-squared measures for Poisson regression models with over-or underdispersion. Computational statistics & data analysis, 44(1-2), pp.253-271.
[3] Mittlböck, M. (2002). Calculating adjusted R2 measures for Poisson regression models. Computer Methods and Programs in Biomedicine, 68(3), 205-214.
[4] Unakafova V.A. Pseudo-R squared measure for poisson regression models. https://de.mathworks.com/matlabcentral/fileexchange/67041-pseudo-r-squared-measurefor-poisson-regression-models, 2018.