Problem with bias-variance tradeoff in neural network

Hi everyone, I've a problem with the error values returned by train function. In the first place, using:

[ net tr y e ] = train(net,x,t);

The matrix "e" returned has dimensions [number of classes x number of samples] but most of values has the negative sign. Why? What are those values? In which set these values relate? Training set, validation set or test set? In the second place, using:

biaserr = mean(e(:))
 varerr  = var(e(:))

despite the different size of the network (by inserting a different number of hidden units or by changing the number of hidden layer), I get always a bias value of zero. What are the causes of this phenomenon?

The dataset that use is unbalanced, in addition, whatever the number of hidden units or hidden layer get a maximum of 10% of the classification error.

Answer 1

[Greg Heath]

I don't see how a net trained on a discontinuous target could be very useful for studying the error bias/variance tradeoff.

Therefore, it would be helpful for you to explicitly state what you expect to be demonstrated.

You should use a regression net (e.g., FITNET) with a predominately continuous input/output relation.

Furthermore, for a fixed network structure, differences in outputs will depend on initial weights. Therefore, for a fixed number of hidden nodes and layers at least 10 to 20 different nets from different random initial weights should be considered.

I would begin with the following goal:

Minimize the number of hidden nodes subject to the constraint that NMSE, the normalized mean-square-error, is less than 0.01 (i.e., Rsquare >= 0.99). Not all nets can achieve that goal. However, multiple trials will allow you to find the lowest achievable NMSE

Finally you should begin with the smaller regression data sets obtained using the commands

 help nndatasets

 and

 doc nndatasets

as described in

http://www.mathworks.com/matlabcentral/newsreader/view_thread/337914#928054

Hope this helps.

P.S. I have designed zillions of regression nets in both the NEWSGROUP and ANSWERS.

You can find many of them by just adding adding

 greg fitnet

to your search words.

Hope this helps.

Thank you for formally accepting my answer

Greg

Answer 2

[Greg Heath]

Insufficient explanation:

I assume classification. For which the mean and variance of the error e = t-y is typically, irrelevant.

How many classes? Size of each class?

Dimensions of input and target matrices?

Are the targets [0,1] unit vectors?

Why aren't you using the MATLAB function PATTERNNET ?

Perplexed,

Greg

[Alessandro Lo Presti]

Hi Greg, I use neural network for a supervised classification problem. I've 5 classes normalized with the form: 1 0 0 0 0 for the first class, 1 1 0 0 0 for the second class, etc. Input matrix has 11 x 5473 dimension and Target has 5 x 5473. I use patternnet to create the network but my problem is to study the bias-variance tradeoff in relation with the number of neurons (for example 1 neuron, 10 neurons and 50 neurons) and the number of hidden layer (1 or 2 hidden layers, not more). How do you study the bias-variance-tradeoff in matlab? What functions should I use once I've trained the network with the neural network toolbox?