Cross validation in recurrent neural network

There doesn't seem to be any facility to use cross-validation in recurrent neural networks in the NN Toolbox.

The following script is taken directly from the documentation on the 'layrecnet' function;

[X,T] = simpleseries_dataset; net = layrecnet(1:2,10); [Xs,Xi,Ai,Ts] = preparets(net,X,T); net = train(net,Xs,Ts,Xi,Ai); view(net) Y = net(Xs,Xi,Ai); perf = perform(net,Y,Ts)

Running this script starts the window, but no validation checks are made during training, and the process is stopped either by reaching the maximum epochs or mu value.

I've tried dividerand and divideind, but I cannot get validation checks to occur in any simulation.

Any suggestions?

Also, the example given in the documentation on Design Layer-Recurrent Neural Networks has the same problem. Is it a trait of recurrent neural networks that I was unaware of that validation checks cannot occur, am I doing something wrong, or is this a bug in the program?

Cheers,

Alex

Answer 1

[Greg Heath]

Contrary to some of the data division information stored in the LAYRECNET timeseries

 neto = layrecnet;
 neto = neto       ( NO SEMICOLON!)

 LAYRECNET DOES NOT HAVE DATA DIVISION!

Details are in the NEWSGROUP post

 http://www.mathworks.com/matlabcentral/newsreader/view_thread/341668#936256

Hope this helps.

Thank you for formally accepting my answer

Greg

[M.B]

This should be included in the documentation of layrecnet. Can someone from MathWorks please confirm this "bug"?

Answer 2

[Greg Heath]

Validation checks are a default and should work as long as you are not trying to use TRAINBR. (TRAINBR validation is version dependent).

HOWEVER, the default data division function is DIVIDERAND. Therefore, the validation data is randomly spread along the length of the data set within the training data.

For most time-series prediction purposes it makes more sense to use DIVIDEBLOCK.

DIVIDEBLOCK restricts the val data to be between the training data and the test data.

Hope this helps.

Thank you for formally accepting my data

Greg

[Greg Heath]

You should be able to use DIVIDEIND to mimic DIVIDEBLOCK.

... but why bother?

ALSO: Validation stopping is NOT CROSS-VALIDATION. As far as I'm concerned, the latter is just as invalid for timeseries as designing multiple nets with DIVIDERAND.

[Alex C]

Hey Greg, Thanks for the reply. I will have to maybe look at updating my version of Matlab, (2014b), or the toolbox, because validation checks definitely do not occur when I run the code provided in the documentation, and I am using 'trainlm'. The lack of validation checks is shown by the performance plotting in the training window, which only shows the calibration performance. I agree that I have erroneously called this cross-validation, as opposed to validation checks. Incidentally why do you feel this is invalid for timeseries data?

[Alex C]

Thanks for this. I would like to accept that last comment as the correct answer, but didn't see a facility for that. Until the bug is fixed, I guess a trial and error approach to the amount of training epochs should be used, and a certain amount of data should be withheld by the user for validation.

Thanks for your help Greg

[Greg Heath]

I moved the comment to an Answer box.

Another approach is to

1. Use ALL of the data to minimize the number of parameters

 Np = num(lags) + num(hidden) + num(weights)

subject to a maximum bound constraint on the DEGREE-OF-FREEDOM-ADJUSTED mean-square-error

 MSEa = SSE/(N-Np)

For example

 MSEa <= 0.001*MSE00a % or even 0.01*MSE00a

where

 MSE00a = mean(var(target,0))

Summary statistics (e.g., NMSE cumulative probability or, just the summary tabulation of min/median/mean/stddev/max) of expected performance on unseen operational data can be estimated via multiple random initial weight designs.

2. To obtain less biased performance estimate statistics

a. Use the above topology with a Ntrn/0/Ntst data division
b. Design Ntrials nets with random initial weights.
c. Estimate the summary stats.

Hope this helps.

Greg