loss

Loss at each horizon step

Since R2023b

Syntax

L = loss(Mdl,TestTbl)

L = loss(Mdl,TestX,TestY)

L = loss(Mdl,TestY)

L = loss(___,LossFun=lossfun)

Description

L = loss(Mdl,TestTbl) returns the test set loss (mean squared error) for the direct forecasting model Mdl at each step of the horizon (Mdl.Horizon). The function uses the test data TestTbl to prepare lagged and leading predictors, and uses the response variable in TestTbl to compute loss values.

L is a 1-by-h numeric vector, where h is the number of elements in Mdl.Horizon.

L = loss(Mdl,TestX,TestY) returns losses for the test set exogenous predictor data TestX and the test set response data TestY. This syntax assumes that Mdl uses exogenous predictors; that is, Mdl.PredictorNames is nonempty.

L = loss(Mdl,TestY) returns test set losses when the model Mdl does not use exogenous predictors. That is, Mdl.PredictorNames must be empty.

L = loss(___,LossFun=lossfun) specifies the loss function in addition to any of the input argument combinations in previous syntaxes.

Examples

Calculate Test Set Mean Squared Error of Direct Forecasting Model

Calculate the test set mean squared error (MSE) of a direct forecasting model.

Load the sample file TemperatureData.csv, which contains average daily temperatures from January 2015 through July 2016. Read the file into a table. Observe the first eight observations in the table.

temperatures = readtable("TemperatureData.csv");
head(temperatures)

    Year       Month       Day    TemperatureF
    ____    ___________    ___    ____________

    2015    {'January'}     1          23     
    2015    {'January'}     2          31     
    2015    {'January'}     3          25     
    2015    {'January'}     4          39     
    2015    {'January'}     5          29     
    2015    {'January'}     6          12     
    2015    {'January'}     7          10     
    2015    {'January'}     8           4     

For this example, use a subset of the temperature data that omits the first 100 observations.

Tbl = temperatures(101:end,:);

Create a datetime variable t that contains the year, month, and day information for each observation in Tbl. Then, use t to convert Tbl into a timetable.

numericMonth = month(datetime(Tbl.Month, ...
    InputFormat="MMMM",Locale="en_US"));
t = datetime(Tbl.Year,numericMonth,Tbl.Day);
Tbl.Time = t;
Tbl = table2timetable(Tbl);

Plot the temperature values in Tbl over time.

plot(Tbl.Time,Tbl.TemperatureF)
xlabel("Date")
ylabel("Temperature in Fahrenheit")

Partition the temperature data into training and test sets by using tspartition. Reserve 20% of the observations for testing.

partition = tspartition(size(Tbl,1),"Holdout",0.20);
trainingTbl = Tbl(training(partition),:);
testTbl = Tbl(test(partition),:);

Create a full direct forecasting model by using the data in trainingTbl. Train the model using a decision tree learner. All three of the predictors (Year, Month, and Day) are leading predictors because their future values are known. To create new predictors by shifting the leading predictor and response variables backward in time, specify the leading predictor lags and the response variable lags.

Mdl = directforecaster(trainingTbl,"TemperatureF", ...
    Learner="tree", ...
    LeadingPredictors="all",LeadingPredictorLags={0:1,0:1,0:7}, ...
    ResponseLags=1:7)

Mdl = 
  DirectForecaster

                  Horizon: 1
             ResponseLags: [1 2 3 4 5 6 7]
        LeadingPredictors: [1 2 3]
     LeadingPredictorLags: {[0 1]  [0 1]  [0 1 2 3 4 5 6 7]}
             ResponseName: 'TemperatureF'
           PredictorNames: {'Year'  'Month'  'Day'}
    CategoricalPredictors: 2
                 Learners: {[1x1 classreg.learning.regr.CompactRegressionTree]}
                   MaxLag: 7
          NumObservations: 372

Mdl is a DirectForecaster model object. By default, the horizon is one step ahead. That is, Mdl predicts a value that is one step into the future.

Calculate the test set MSE. Smaller MSE values indicate better performance.

testMSE = loss(Mdl,testTbl)

testMSE = 
61.0849

Input Arguments

Mdl — Direct forecasting model

DirectForecaster model object | CompactDirectForecaster model object

Direct forecasting model, specified as a DirectForecaster or CompactDirectForecaster model object.

TestTbl — Test set data

table | timetable

Test set data, specified as a table or timetable. Each row of TestTbl corresponds to one observation, and each column corresponds to one variable. TestTbl must have the same data type as the predictor data argument used to train Mdl, and must include all exogenous predictors and the response variable.

TestX — Test set exogenous predictor data

numeric matrix | table | timetable

Test set exogenous predictor data, specified as a numeric matrix, table, or timetable. Each row of TestX corresponds to one observation, and each column corresponds to one predictor. TestX must have the same data type as the predictor data argument used to train Mdl, and must consist of the same exogenous predictors.

TestY — Test set response data

numeric vector | one-column table | one-column timetable

Test set response data, specified as a numeric vector, one-column table, or one-column timetable. Each row of TestY corresponds to one observation.

• If TestX is a numeric matrix, then TestY must be a numeric vector.

• If TestX is a table, then TestY must be a numeric vector or one-column table.

• If TestX is a timetable or it is not specified, then TestY must be a numeric vector, one-column table, or one-column timetable.

If you specify both TestX and TestY, then they must have the same number of observations.

lossfun — Loss function

"mse" (default) | function handle

Loss function, specified as "mse" or a function handle.

• If you specify the built-in function "mse", then the loss function is the mean squared error.

• If you specify your own function using function handle notation, then the function must have the signature lossvalue = lossfun(Y,predictedY), where:

  • The output argument lossvalue is a scalar.

  • You specify the function name (lossfun).

  • Y is an n-by-1 vector of observed numeric responses at a specific horizon step, where n is the number of test set observations.

  • predictedY is an n-by-1 vector of predicted numeric responses at a specific horizon step.

  Specify your function using LossFun=@lossfun.

Data Types: single | double | function_handle

Limitations

• When you use the loss object function, the test set data must contain at least Mdl.MaxLag + max(Mdl.Horizon) observations. The software requires these observations for creating lagged and leading predictors.

Version History

Introduced in R2023b