How can i evaluate prediction interval with interval coverage probability (PICP) for probabilistic forecasting?

I am trying to use Gaussian Process Regression Models https://au.mathworks.com/help/stats/gaussian-process-regression-models.html

And I would like to use prediction interval assessment: the prediction interval coverage probability (PICP) and the mean prediction interval width (MPIW). PICP measures the reliability of the predictions and shows the percentage of the real values that will be covered by the upper and lower bounds using the attached formula?

Thanks in advance

Answer 1

[Sai Pavan]

Hi Israt,

I understand that you are trying to calculate the prediction interval coverage probability (PICP) and the mean prediction interval width (MPIW) for Gaussian Process Regression models.

Please follow the below workflow to calculate PICP and MPIW:

1. Train your Gaussian Process Regression model using the “fitrgp” function
2. Obtain predictions and prediction intervals using the “predict” function
3. To calculate PICP:

• Define the lower and upper bounds of the prediction interval, such as yLower and yUpper, based on the prediction intervals obtained in the previous step.
• Calculate the number of observations within the prediction interval: numObservations = sum(yLower <= YTest & YTest <= yUpper).
• Calculate the PICP as the ratio of the number of observations within the prediction interval to the total number of observations: PICP = numObservations / numel(YTest)

4. To calculate MPIW:

• Calculate the width of each prediction interval: intervalWidths = yUpper - yLower
• Calculate the mean prediction interval width: MPIW = mean(intervalWidths)

Please refer to the below code snippet to calculate the PICP and MPIW:

gprMdl = fitrgp(XTrain, YTrain); % Train Gaussian Process Regression model
[yPred, ysd, yint] = predict(gprMdl, XTest); % Obtain predictions and prediction intervals 
yLower = yint(:, 1);
yUpper = yint(:, 2); 
numObservations = sum(yLower <= YTest & YTest <= yUpper); 
PICP = numObservations / numel(YTest); % Calculate PICP 
intervalWidths = yUpper - yLower; 
MPIW = mean(intervalWidths); % Calculate MPIW 

Please refer to the below documentations to learn more about “fitrgp” and “predict” functions:

• https://www.mathworks.com/help/stats/fitrgp.html
• https://www.mathworks.com/help/stats/compactregressiongp.predict.html

Hope it helps.

Regards,

Sai Pavan