Can anyone help to interpret these neural network graphs?

I've been trying a MLP with 2 input variables obtained from another model (It's a hybrid model with ANN) and one output. I've got the following graphs but I don't know how to interpret them and decide if my model is correct. MSE doesn't decrease on testing data. Does that mean my model is not good? Does my regression plot look okay?

Also, should the testing line always be above the validation line in the performance plot or can it be below?

Someone help please! Thank you.

[BN]

Hello kareema

First I'm gonna tell you that in the ANN's it's very important that what percentage of data are used for train, validation, and test. usually in my case of study 70% of the data set using for the train, 15% for validation, and 15% for tests.

the criteria for deciding which ANN's are better is R of the test. bigger R means that your model can predict new condition.

it is good to mention that train is like when you study your lessons, validation is like when you solve the quiz and watch the answers (before the teacher get exam in the class), and test part is like when you have a real test in the class and haven't any idea about that fact that what it could be the test questions.

let's talk about your problem. you have really near 1 R in train data, at a first look, I was thinking that maybe your model is over-trained because R is really near 1. I recommend you to decrease the percentage of train data (like to 65%) and increase data in the validation (like 20%). I think change these percentages can lead us to better test R.

It is very depend on your case of study and your field of study to diside that if this R is good for choose the ANN, but I think R=076 could be good.

MSE doesn't decrease on testing data...

it very depends on your data! let's tell you you can use RMSE which could be accurate than MSE:

net.performFcn = 'mse';  % Mean Squared Error
% For a list of all plot functions type: help nnplot
net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...
    'plotregression', 'plotfit'};
[net,tr] = train(net,x,t); 
% Test the Network
y = net(x);
e = gsubtract(t,y);
performance = perform(net,t,y);
% Recalculate Training, Validation and Test Performance
trainTargets = t .* tr.trainMask{1};
valTargets = t .* tr.valMask{1};
testTargets = t .* tr.testMask{1};
trainPerformance = perform(net,trainTargets,y);
valPerformance = perform(net,valTargets,y);
testPerformance = perform(net,testTargets,y);
RMSE_test=sqrt(testPerformance); % RMSE is sqrt of MSE

see what is the number of RMSE, this number shows you the predicted data in test how many different from real data. you can decide now that if your model gives you good results or not.

sometimes it is a good idea to calculate normalized RMSE which maybe useful depends on your data.

N_RMSE= RMSE_test/(max(test_outputs)-min(test_outputs

Does that mean my model is not good? Does my regression plot look okay?

No, it can't be a reason for the model not good. read above.

your regression looks good. because the regression line exactly creates 45 degrees and that means you are accurate.

for summarize:

I think your model is good. but in order to increase the R in test data, I recommend you to try some changes in the neural network like change percentage of used data, trainFcn, etc...

don't forget always you can change net.divideMode.

in one sentence your model is good but it very depends on your size of data sets and field of study.

Best Regards,

Behzad

[kareema jumoorty]

Hello Behzad.

Thank you for answering my question. I will try these.

I am forecasting 1-min intraday volatility for exchange rate with a hybrid ANN-GARCH-type model and comparing with the GARCH model. Adjusted predicted values from the garch-type model is used as input in the ANN (I'm using Roh(2007) : Forecasting the volatility of stock price index; as reference)

I've one more question. Should I make predictions on data which was used for testing in the ANN or should I use another dataset apart? Because in my ANN, i used forecasted data from the GARCH model for testing and predicted values from the garch model for training and validation.

Concerning the RMSE test, does it mean, the lower the better? what would you say is a good range for accepting the model?

I also use 'sample' as net.divideMode.

Again thanks.

Regards,

Kareema

[BN]

So in general, you have some data as inputs. the model will use these data for train, validation, and test. after you achieve desire network and you certain about network accuracy, you can use this net to produce forecasts using new inputs.

it should be something like this for arbitrary X

Y = net(X); %use trained ANN to produce forecast

please look at: https://www.mathworks.com/matlabcentral/answers/377267-how-to-predict-from-a-trained-neural-network

about RMSE: in general yes lower is better. but please read this reference:

"There is no absolute good or bad threshold, however, you can define it based on your DV. For a datum which ranges from 0 to 1000, an RMSE of 0.7 is small, but if the range goes from 0 to 1, it is not that small anymore. However, although the smaller the RMSE, the better, you can make theoretical claims on levels of the RMSE by knowing what is expected from your DV in your field of research.

You can't fix a particular threshold value for RMSE. We have to look at the comparison of RMSE of both test and train datasets. If your model is good then your RMSE of test data is quite similar to the train dataset. Otherwise below conditions met.

RMSE of test > RMSE of train => OVER FITTING of the data.

RMSE of test < RMSE of train => UNDER FITTING of the data.

"

you can see the reference here: https://stats.stackexchange.com/questions/56302/what-are-good-rmse-values

Best Regards,

Behzad

[kareema jumoorty]

Thanks so much Behzad!!!