How to plot test and validation accuracy every epoch using Computer vision system toolbox? And what about overfitting?

I am finding many blogs of CNN and its related classification strategy in Matlab but I couldn't find how I could actually plot the validation set accuracy for every epoch along with the training set accuracy.

https://www.mathworks.com/help/nnet/ref/trainingoptions.html#namevaluepairarguments

above documentation shows that I could plot training accuracy every epoch but not the validation set accuracy.

If that is not possible how to make sure that my network is not overfitting?

Answer 1

[Sebastian K]

Hi Jay,

There is no option to display the validation set accuracy. However this should not be necessary. Functions in the Neural Network Toolbox employ a technique called Early Stopping, which takes the validation set error into consideration to prevent overfitting. The article I am linking to provides some nice discussions on overfitting, I would recommend you to read the other parts as well.

Cheers,

Sebastian

[Jay]

Hi Sebastian,

I have already referred the link given. However, it seems that it is for conventional neural networks.

In CNN usually, a good network would not overfit when you increase the number of Epoch.

For example, let's say I train the network for 60 epoch in total but my training and validation error is converging around 40 epochs. Now it is always a good practice to run for little more number of epoch, to check whether any parameter is causing the overfitting if the network gets overtrained. It that is the case I would tune the parameters that might cause the overfitting. Last 20 Epoch will perform this operation.

Refer the image below. For above description.

ONE SOLUTION: I have thought about the solution of plotting these types of graph is, let the training complete and for total number of epoch. for every epoch save the check points. Once training gets done, load every checkpoint and measure the accuracy on the validation set for every particular checkpoint.

I am not sure if that is possible though! I haven't implemented it yet.

[Chandrama Sarker]

Hi Jay, Thanks for raising this query as I am facing the similar problem and Matlab does not have any solution for that. I am just curious how would you be able to measure the accuracy on the validation set for every particular checkpoint. My apology I am pretty new in this field. I am able to save the checkpoints but could not understand how to measure the accuracy on the validation data set. Any help in this regard will be highly helpful for me. Regards

[Greg Heath]

I have not used CNN so I am curious why MATLAB cannot continuously plot trn/val/tst error rates the same way it does for skinny nets.

If there is no answer here, I suggest someone who uses CNN post a question on the NEWSGROUP.

Greg

[Keke Zhang]

Hi Jay: I want to know how did you draw the picture you gave? Thank you very much!

[Anu Sebastian]

Hlo jay How to code this

Answer 2

[Evan Koester]

This problem has been addressed in the newly released MATLAB 2018a. If you have a groundtruth of your data, you can load it as a pixelLabelImageDatastore. This can be used in your ValidationData trainingOptions.

An example:

val4data = imageDatastore(location of image data); val4label = load('location of label groundtruth');

val4label = pixelLabelDatastore(val4label.gTruth); val4gt = pixelLabelImageDatastore(val4data,val4label);

opts = trainingOptions('sgdm', ... 'MaxEpochs', 5000, ... 'InitialLearnRate', .05, ... 'VerboseFrequency',validationFrequency,... 'ValidationData',val4gt,... 'ValidationFrequency',5,... 'Plots','training-progress',... 'CheckpointPath', tempdir,... 'MiniBatchSize', 48);

This will plot the validation data loss on the same plot as training loss when training your CNN. In my application I used pixel-wise labeling. Prior to MATLAB version 2018a, there was not a way to perform this without making a checkpoint, test validation data, continue training type of algorithm as mentioned above.

Answer 3

[Saira]

Hi,

I have 5600 training images. I have extracted features using Principal Component Analysis (PCA). Then I am applying CNN on extracted features. My training accuracy is 30%. How to increase training accuracy?

Feature column vector size: 640*1

My training code:

% Convolutional neural network architecture

layers = [

imageInputLayer([1 640 1]);

reluLayer

fullyConnectedLayer(7);

softmaxLayer();

classificationLayer()];

options = trainingOptions('sgdm', 'Momentum',0.95, 'InitialLearnRate',0.0001, 'L2Regularization', 1e-4, 'MaxEpochs',5000, 'MiniBatchSize',8192, 'Verbose', true);

[Sid Khan]

increase dataset size