a problem with neural network training

Hi everybody I have read in some papers that in order to avoid your neural network getting stuck in local minima during resampling methods, a network is trained on the entire data set to obtain a model ) with weights W0, then, these weights are used as the starting point for vtraining the other samples. I have applied this method on a dataset but, while trying to train the second sample the training is not being completed. I have changed some of the neural network parameters like min_grad periodically however, it did not solve this problem. Is it due to the fact that our initial guess is so close to the minimum? Is the problem related to my code? or there is other problem in this field. The code is as follows: [x,t]=house_dataset; Inputs=cell(1,2); Targets=cell(1,2); for i=1:2 Ind=randi(size(x,2),size(x,2),1); Inputs{i} = x(:, Ind); Targets{i}=t(Ind); end NN=cell(1,2); net=feedforwardnet; rng(0); NN{1}=train(net,Inputs{1},Targets{1}); IW = NN{1}.IW; LW = NN{1}.LW; B = net.b; NN{2}.IW = IW; NN{2}.LW = LW; NN{2}.b = B; net.initFcn=''; NN{2}=train(net,Inputs{2},Targets{2});

Your help is greatly appreciated.

Answer 1

[Greg Heath]

There are MANY thing wrong with your code.

1. Your premise of using a full data design to initialize divided data designs doesn't make sense.

2. Your data selection using randi randomly selects data with replacement. Therefore ,e.g., you may only have 325 unique examples out of 506. The rest are duplicates and 181 are not used at all!

3. The default data division applied to the non-unique data selection not only results in training, validation and test subsets with duplicated data, some data points belong to more than one of the 3 subsets.

4. You train and use a net, NN{1} without checking to see if the design is any good.

5. I see no reason to put networks inside cells.

Thank you for formally accepting my answer

Greg

P.S. Search using

 greg fitnet Ntrials % regression/curvefitting

 greg patternnet Ntrials % classification/pattern-recognition

for some of my sample code.

[Mohamad]

Dear Greg thank you very much.

Considering your answers some of the reasons for using this code are as follows:

1. My premise is based on the following references:

Moody, J. 1994. Prediction Risk and Architecture Selection for Neural Networks,….

Utans, J. and Moody, J. (1991), Selecting neural network architectures via the prediction risk: Application to corporate bond rating prediction, in ‘Proceedings of the First International Conference on Artificial Intelligence Applications on Wall Street’, IEEE Computer Society Press, Los Alamitos, CA.

Moody, J. E. and Utans, J. (1992), Principled architecture selection for neural networks: Application to corporate bond rating prediction, in J. E. Moody, S. J. Hanson and R. P. Lippmann, eds, ‘Advances in Neural Information Processing Systems 4’, Morgan Kaufmann Publishers, San Mateo, CA, pp. 683–690.

2. My data selection is the same procedure which is used by bootstrap method, sampling with replacement, or at least I assume that they are the same.

3. Yes, I agree with you.

4. When I store the NNs in different cell, I can refer to each one of them and consider their performance after training for all of them is complete.

best

[Greg Heath]

1. Thanks for the references. However, I don't have access to them. Moreover, since the dates are ancient, I'll stand by my skepticism.

2. Sampling with replacement does not mean that you should not try to use all of the data in each replication. The last subset should contain all of the data not chosen by the other subset(s). However, you can get away with it if you have a huge data set AND a huge number of replications. Otherwise, you should try to use all of the data in each replication.

4. I don't object to storing nets in a cell. However if you use the command whos, you will see that your second net is not recognized as a net.