Mapminmax process function causes that NN incorrectly simulates outputs

Hello,

I have a problem with some outputs from a trained custom neural network. I am using MATLAB 2014 with NN ToolBox ver 8.2.

I have created a simple feedforward NN for classification. I have used some Inputs and Targets, trained the NN, and tried to simulate Outputs given Inputs from the same range. The NN I created gives me incorrect outputs. First it returns outputs only with 0,1 range while the targets are in rage of -6 ... 3 and when I add a process function 'mapminimax' to input and output, the results are wrong: 1. when targets are 0 ... 1, there is an offset of 0.5 such that the outputs are 0.5 and 1 2. when targets are e.g. -6 ... 3, the outputs are around -3 ... 3

I am trying to understand what I am doing wrong.

PS. I have already asked this question in SO and also I provided some more details code: http://stackoverflow.com/questions/36449224/mapminmax-process-function-causes-that-nn-incorrectly-simulates-outputs

The code to test NN

    clear all; close all; clc;

    net = NNPatRec;
    %net.inputs{1}.processFcns = {'mapminmax'};
    %net.outputs{2}.processFcns = {'mapminmax'};

    Inputs = -10:10;
    %Targets = [-6*ones(1,11) 3*ones(1,10)];
    Targets = [zeros(1,11) ones(1,10)];
    [net,tr] = train(net,Inputs,Targets);
    net(-10:10)

The code to create NN:

    function net = NNPatternRecognition
        net = nntest;
    end

    function net = nntest
        net = network;
        net.numInputs = 1;
        net.numLayers = 2;
        net.biasConnect = [1 1]';
        net.inputConnect = [1; 0];
        net.layerConnect = [0 0; 1 0];
        net.outputConnect = [0 1];

        % Inputs
        %net.inputs{1}.processFcns = {'mapminmax'};
        net.inputWeights{1}.learnFcn = 'learngdm';

        % layers 1 (at input)
        net.layers{1}.initFcn = 'initnw';
        net.layers{1}.netInputFcn = 'netsum';
        net.layers{1}.transferFcn = 'tansig';
        net.layers{1}.size = 3;
        % layers 2 (hidden)
        net.layers{2}.initFcn = 'initnw';
        net.layers{2}.netInputFcn = 'netsum';
        net.layers{2}.transferFcn = 'purelin';
        net.layers{2}.size = 1;

        % Network functions
        net.adaptFcn = 'adaptwb';
        net.derivFcn = 'defaultderiv';
        net.divideFcn = 'dividerand'; %'divideblock';
        net.initFcn = 'initlay';
        net.performFcn = 'crossentropy';
        net.trainFcn = 'trainscg';

        % Outputs
        %net.outputs{2}.processFcns = {'mapminmax'};
        %net.outputs{2}.exampleOutput = [0 1];

        net.trainParam.showWindow = false;
        net.trainParam.showCommandLine = true;    
    end

Answer 1

[Greg Heath]

 close all, clear all, clc, plt=0, tic
 x       = -10:10;  N = length(x)
 trueind = 1 + [zeros(1,11) ones(1,10)];
 t       = full(ind2vec(trueind))

 plt = plt+1, figure(plt), hold on
 plot( x( 1:11), trueind(  1:11) ,'o' )
 plot( x(12:21), trueind(12:21),'ro' )
 axis([ -11 11 0 3 ])
 title('CLASS INDICES')

 rng('default')
 net            = patternnet;
 [ net tr y e ] = train( net, x, t );
 outind         = vec2ind(y)
 plot( x( 1:11), outind(  1:11) ,'x' ,'LineWidth',2)
 plot( x(12:21), outind(12:21),'rx' ,'LineWidth',2)

 err    = outind~=trueind;
 Nerr   = sum(err)           % 1
 PctErr = 100*Nerr/N       % 4.7619

 Hope this helps.

 For details, remove the semicolon to get

 net = net

ALSO, for a trn/val/tst breakdown use

 tr = tr

Hope this helps.

Thank you for formally accepting my answer

Greg

[Céldor]

Hi Greg,

Thanks for your answer. I followed your code.

In the final matrix produced by Brendan and yourself, there are three categories not two as I used in my example. Is it a standard routine in classification to always add one extra category?

Thanks

[Greg Heath]

I don't know where you got that idea. Did you run my program?

[Brendan Hamm]

There should be exactly 2 rows in the output. The probability of Class_1 and the probability of Class_2. You just refer to these as Class_0 and Class_1 respectively.

Answer 2

[Brendan Hamm]

You would likely have better luck if you just started with the patternnet which is meant for NN classification.

What I see that is wrong with your current implementation is you have a linear transferFcn for your second layer. For classification purposes this should really be a softmax function. That is change

net.layers{2}.transferFcn = 'purelin';

to

net.layers{2}.transferFcn = 'softmax';

There are also 2 functions used for the processFcns for the the input and output:

net.outputs{2}.processFcns == {'removeconstantrows', mapminmax};
net.inputs{1}.processFcns == {'removeconstantrows', mapminmax};

[Céldor]

Hi Brendan and thanks for your answer.

First of all, I apologise for providing wrong code. I was checking everything from classification to regression and this is why I changed the final tr. function from softmax to purelin. I meant to use softmax for classification. But this problem seems to exists even though I use softmax.

Perhaps I would do better with patternnet which is the function to create a simple NN for classification with the softmax transfer function in an output layer. And normally I would do apply it. But (1) I want to understand how NN object works and avoid keeping it as a black box and (2) I have a more complex NN in my mind. The example I have provided here is only simplified to explain the problem.

While you are right by saying there are two functions for processFcn, my problem actually appears after I set these functions! I am trying to figure out why.

[Brendan Hamm]

mapminmax will map the output values to a minimum and maximum value. By default this is set to the range [-1 1]. So, I suppose I missed this part above, we would need to change the range to be [0,1]. This is controlled in the processParams:

p.outputs{2}.processParams{1}.max_range = 0; % params for remove constant rows
p.outputs{2}.processParams{2}.ymin = 0; % Params for mapminmax
p.outputs{2}.processParams{2}.ymax = 1; % Params for mapminmax

Furthermore with a classification problem the output should really be a probability of belonging to each class. For this reason for a binary classification you would expect 2 outputs for every sample, the probability of class1 and the probability of class2.

So, You really want your targets to be a matrix of ones and zeros where 1 indicates belonging to that class (i.e. you know it belongs to class1 so the probability is 1). You can get this result by calling:

dummyvar(Targets+1)';

I use the +1 as dummyvar expects all inputs to be positive integers (or alternatively categorical or strings) So all in all this could have been accomplished with the following:

Inputs = -10:10;
Targets = [ones(1,11) 2*ones(1,10)]; % I'll Increment by 1 here (Class name is irrelevant)
p = patternnet(10); % For classification problems
Targets = dummyvar(Targets)'; % Make this into an indicator matrix
p = train(p,Inputs,Targets); % train the NN
Tpred = p(Inputs) % Verify the results with our training set.
Tpred =
Columns 1 through 3
  1.0000    1.0000    1.0000
  0.0000    0.0000    0.0000
Columns 4 through 6
  1.0000    1.0000    1.0000
  0.0000    0.0000    0.0000
Columns 7 through 9
  1.0000    1.0000    1.0000
  0.0000    0.0000    0.0000
Columns 10 through 12
  1.0000    1.0000    0.0000
  0.0000    0.0000    1.0000
Columns 13 through 15
  0.0000    0.0000    0.0000
  1.0000    1.0000    1.0000
Columns 16 through 18
  0.0000    0.0000    0.0000
  1.0000    1.0000    1.0000
Columns 19 through 21
  0.0000    0.0000    0.0000
  1.0000    1.0000    1.0000

[Brendan Hamm]

As a shameless plug, if you ever want your Machine Learning Algorithms to feel a little bit less "black-box", consider attending the Machine Learning with MATLAB course.

[Céldor]

Many thanks for this Brendan. Everything's working :)

Regarding mapminmax, I also found what I needed to set yesterday. I had to go through the process of how the command patternnet creates a a netowrk and realised it uses feedforwardnet. But somewhere in the code, there's a line where the ymin is set to 0. I have set ymin by myself and everything started to work :)

And thanks for the comment on preparation of Targets for classification. This is another thing I had to go through by myself and realised that NN works much better if there are multiple outputs, each representing individual class.

Thanks :)

PS. I have just learned a new command: dummyvar