How are error weights implemented with cross-entropy performance function when training a neural network?

7 views (last 30 days)
James on 7 Jun 2016
Commented: Brendan Hamm on 7 Jun 2016
I am training a neural network for classification to work with several classes that have some class imbalance. I am using the error weight functionality of train to play with compensating for the class imbalance through "cost-sensitive" learning.
There is documentation for how the error weights are applied with the mse cost function. I would like to know how the error weights are applied when using the crossentropy cost function. Is it implemented as roughly as
Sum_s^N Sum_i^K w(s,i) * ( -t(s,i) * log(y(s,i)) )
where w is the weight for sample s and class i, N is the sample size, and K is the number of classes? (In my case the weight for all samples are the same, only the weights of the classes change in proportion to their frequency in the training set. Also, I'm just using a simple feedforward network with one hidden layer, and using trainscg.)
Do these error weights simply modify the networks outputs, or does it actually modify the training (backpropagation) algorithm? For reference as to what I mean see M. Kukar, I. Kononenko, Cost-sensitive learning with neural networks, 1998. My question is specifically trying to asking if Matlab uses something like the "Minimization of the misclassification costs" method mentioned in that paper, or something like two of the other methods such as "Adapting the output of the network", or "Cost-sensitive classification"
Finally, and partially related, since we can set weights for individual samples, could an ensamble method be created using an AdaBoost like method using these error weights that are given to the train function?
  1 Comment
Brendan Hamm
Brendan Hamm on 7 Jun 2016
For alot of the Neural Network toolbox you need to use help instead of doc:
>> help crossentropy
This will describe the formula for ce, which appears to be as you describe (more or less). I do not believe that Neural Networks support ensemble methods. Furthermore I am not aware of being able to update the Weights during different Epochs in the NN training.

Sign in to comment.

Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by