"Invalid training data. Predictors and responses must have the same number of observations"

I'd be appreciated it if someone help me to debug my code. Thanks in advance.

clc;
clear;
close all;
check_node_degs_vec=[1,2,3,4,5];
var_node_degs_vec=[1,2,3,4,5];
x=0.5;
decoding_threshold=0.3;
sample_size=1000;
Xtemp=decoding_threshold*rand([1,sample_size]);
for i=1:sample_size
    Xtrain{i}=Xtemp(i);
end
Xtrain=reshape(Xtrain, [1,sample_size]); 
Ytrain=zeros(size(Xtrain));
lgraph = layerGraph;
layer_1=[sequenceInputLayer(1,"Name","seq_input_1")
    functionLayer(@(x) nodeDegreeLayer(x, check_node_degs_vec),"Name","func_ro")
    fullyConnectedLayer(1,"Name","ro")
    reluLayer('Name','relu_1');
    functionLayer(@(x) nodeDegreeLayer(x, var_node_degs_vec),"Name","func_lam")
    fullyConnectedLayer(1,"Name","lam")
    reluLayer('Name','relu_2');
    additionLayer(2,'Name','add_1');
    reluLayer('Name','relu_3');
    regressionLayer
    ];
lgraph = addLayers(lgraph,layer_1);
res_input=functionLayer(@(X) -X,"Name","res_func");
lgraph = addLayers(lgraph,res_input);
lgraph = connectLayers(lgraph,'seq_input_1','res_func');
lgraph = connectLayers(lgraph,'res_func','add_1/in2');
%dlnet = dlnetwork(lgraph);
% figure
% plot(lgraph)
options = trainingOptions("adam", ...
    MaxEpochs=60, ...
    miniBatchSize=1, ...
    Plots="training-progress", ...
    Verbose=0);
net = trainNetwork(Xtrain,Ytrain,lgraph,options);

Error using trainNetwork
Invalid training data. Predictors and responses must have the same number of observations.

function y = nodeDegreeLayer(x, node_degs_vec)
 
y = (1-x).^(node_degs_vec-1);
end

Error using trainNetwork

Invalid training data. Predictors and responses must have the same number of observations.

Error in DAGNET (line 49)

net = trainNetwork(Xtrain,Ytrain,lgraph,options);

[massoud pourmandi]

"GPU"

[massoud pourmandi]

I think that the way that I generate Xtrain and Ytrain is wrong, but I dont know how to gernerate it.

Answer 1

[Cris LaPierre]

Note that, in the image you shared, they also all collapse back to a single node. My suspicion, then, is that you would use the lambdas to weight the result of each value. Again, you would know more about the particulars here than me.

I defined a vector of lambdas and used pagemtimes to compute the output of nodeDegreeLayer

clc;
clear;
close all;
check_node_degs_vec=[1,2,3,4,5];
var_node_degs_vec=[1,2,3,4,5];
lambda = [0.1 0.3 0.2 0.15 0.25]';
x=0.5;
decoding_threshold=0.3;
sample_size=1000;
Xtemp=decoding_threshold*rand([1,sample_size]);
% for i=1:sample_size
%     Xtrain{i}=Xtemp(i);
% end
Xtrain=Xtemp;
Xtrain=reshape(Xtrain, [1,sample_size]); 
Ytrain=zeros(size(Xtrain));
lgraph = layerGraph;
layer_1=[sequenceInputLayer(1,"Name","seq_input_1")
    functionLayer(@(x) nodeDegreeLayer(x, check_node_degs_vec,lambda),"Name","func_ro")
    fullyConnectedLayer(1,"Name","ro")
    reluLayer('Name','relu_1');
    functionLayer(@(x) nodeDegreeLayer(x, var_node_degs_vec,lambda),"Name","func_lam")
    fullyConnectedLayer(1,"Name","lam")
    reluLayer('Name','relu_2');
    additionLayer(2,'Name','add_1');
    reluLayer('Name','relu_3');
    regressionLayer
    ];
lgraph = addLayers(lgraph,layer_1);
res_input=functionLayer(@(X) -X,"Name","res_func");
lgraph = addLayers(lgraph,res_input);
lgraph = connectLayers(lgraph,'seq_input_1','res_func');
lgraph = connectLayers(lgraph,'res_func','add_1/in2');
%dlnet = dlnetwork(lgraph);
% figure
% plot(lgraph)
options = trainingOptions("adam", ...
    MaxEpochs=60, ...
    miniBatchSize=1, ...
    Plots="training-progress", ...
    Verbose=0);
net = trainNetwork(Xtrain,Ytrain,lgraph,options);

function y = nodeDegreeLayer(x, node_degs_vec,lambda)
 
y = pagemtimes((1-x).^(node_degs_vec-1),lambda);
end

[l]

Hello,I have the same problem, but it hasn't been solved yet , so I hope you can give me some advice.My code is as follows:

gwsdata = reshape(gwsdata,205,3627);

tempdata = reshape(tempdata,205,3627);

ndvidata = reshape(ndvidata,205,3627);

precdata = reshape(precdata,205,3627);

petdata = reshape(petdata,205,3627);

lstdata = reshape(lstdata,205,3627);

smdata = reshape(smdata,205,3627);

%划分训练集，验证集

[trainInd, testInd] = dividerand(205, 0.8, 0.2); % 80%的数据作为训练集，20%的数据作为测试集

temp_train = reshape(tempdata(trainInd, :), 39, 93, 1, 142);

ndvi_train = reshape(ndvidata(trainInd, :), 39, 93, 1, 142);

prec_train = reshape(precdata(trainInd, :), 39, 93, 1, 142);

pet_train = reshape(petdata(trainInd, :), 39, 93, 1, 142);

lst_train = reshape(lstdata(trainInd, :), 39, 93, 1, 142);

sm_train = reshape(smdata(trainInd, :), 39, 93, 1, 142);

gws_train = reshape(gwsdata(trainInd, :), 39, 93, 1, 142);

temp_test = reshape(tempdata(testInd, :), 39, 93, 1, 36);

ndvi_test = reshape(ndvidata(testInd, :), 39, 93, 1, 36);

prec_test = reshape(precdata(testInd, :), 39, 93, 1, 36);

pet_test = reshape(petdata(testInd, :), 39, 93, 1, 36);

lst_test = reshape(lstdata(testInd, :), 39, 93, 1, 36);

sm_test = reshape(smdata(testInd, :), 39, 93, 1, 36);

gws_test = reshape(gwsdata(testInd, :), 39, 93, 1, 36);

%数据归一化

temp_train_norm = normalize(temp_train);

temp_test_norm = normalize(temp_test);

%ndvi_train_norm = normalize(ndvi_train);

%ndvi_test_norm = normalize(ndvi_test);

prec_train_norm = normalize(prec_train);

prec_test_norm = normalize(prec_test);

pet_train_norm = normalize(pet_train);

pet_test_norm = normalize(pet_test);

lst_train_norm = normalize(lst_train);

lst_test_norm = normalize(lst_test);

sm_train_norm = normalize(sm_train);

sm_test_norm = normalize(sm_test);

%gws_train_norm = normalize(gws_train);

%gws_test_norm = normalize(gws_test);

% 构造网络结构

layers = [

imageInputLayer([39 93 6],"Name","imageinput")

convolution2dLayer([3 3],50,"Name","conv2d_1","Padding",[1 1 1 1])

reluLayer("Name","Relu_1")

maxPooling2dLayer([2 2],"Name","pool_1","Padding","same","Stride",[2 2])

dropoutLayer(0.2,"Name","dropout_1")

convolution2dLayer([3 3],100,"Name","conv2d_2","Padding",[1 1 1 1])

reluLayer("Name","Relu_2")

maxPooling2dLayer([2 2],"Name","pool_2","Padding","same","Stride",[2 2])

dropoutLayer(0.2,"Name","dropout_2")

convolution2dLayer([3 3],200,"Name","conv2d_3","Padding",[1 1 1 1])

reluLayer("Name","Relu_3")

maxPooling2dLayer([2 2],"Name","pool_3","Padding","same","Stride",[2 2])

dropoutLayer(0.3,"Name","dropout_3")

convolution2dLayer([3 3],300,"Name","conv2d_4","Padding",[1 1 1 1])

reluLayer("Name","Relu_4")

fullyConnectedLayer(200,"Name","fc_1")

reluLayer("Name","relu_5")

dropoutLayer(0.4,"Name","dropout_4")

fullyConnectedLayer(1164*2783,"Name","fc_2")

regressionLayer("Name","Output")];

lgraph = layerGraph(layers);

options = trainingOptions('adam', ...

'MaxEpochs', 32, ...

'MiniBatchSize', 15, ...

'Shuffle', 'every-epoch', ...

'InitialLearnRate', 1e-2, ...

'LearnRateDropFactor', 0.01, ...

'LearnRateDropPeriod', 10, ...

'LearnRateSchedule','piecewise',...

'Plots', 'training-progress',...

'ExecutionEnvironment', 'cpu', ...

"L2Regularization", 0.001,...

'Verbose', true);

net = trainNetwork({temp_train, ndvi_train, prec_train,pet_train,sm_train,lst_train}, gws_train, lgraph, options);

Error for trainnetwork:Invalid training data. The predictor and response must have the same number of observations.

But when I checked the data, I found that the data seemed to be correct. Could you tell me why? Thanks a lot!

[Cris LaPierre]

Without your data, it is pretty hard to provide a specific answer. However, if you have the same error, then your predictor and response inputs do not have the same number of observations.

Answer 2

[Image Analyst]

Instead of

Xtemp=decoding_threshold*rand([1,sample_size]);
for i=1:sample_size
    Xtrain{i}=Xtemp(i);
end
Xtrain=reshape(Xtrain, [1,sample_size]); 
Ytrain=zeros(size(Xtrain));

maybe

Xtemp=decoding_threshold*rand([sample_size, 1]); % Column vector.  But why random numbers?!?
Xtrain=Xtemp; 
Ytrain=zeros(size(Xtrain)); % Huh?!?

but it looks like you're training on random inputs, and a desired ground truth of all zeros. What's up with that? That makes no sense, at least not to me.

[massoud pourmandi]

This is not a conventional nueral network. Here, I'm trying to use NN as a powerfull tool to solve my optimization problem, though all the constraint are not considered yet. Since I got stuck in the initial phase, I just posted it here. Anyway thank you for your time and afford. I have decided to develop the model from scratch without using deel learnnig toolbox.

[l]

Hello,I have the same problem, but it hasn't been solved yet , so I hope you can give me some advice.My code is as follows:

gwsdata = reshape(gwsdata,205,3627);

tempdata = reshape(tempdata,205,3627);

ndvidata = reshape(ndvidata,205,3627);

precdata = reshape(precdata,205,3627);

petdata = reshape(petdata,205,3627);

lstdata = reshape(lstdata,205,3627);

smdata = reshape(smdata,205,3627);

%划分训练集，验证集

[trainInd, testInd] = dividerand(205, 0.8, 0.2); % 80%的数据作为训练集，20%的数据作为测试集

temp_train = reshape(tempdata(trainInd, :), 39, 93, 1, 142);

ndvi_train = reshape(ndvidata(trainInd, :), 39, 93, 1, 142);

prec_train = reshape(precdata(trainInd, :), 39, 93, 1, 142);

pet_train = reshape(petdata(trainInd, :), 39, 93, 1, 142);

lst_train = reshape(lstdata(trainInd, :), 39, 93, 1, 142);

sm_train = reshape(smdata(trainInd, :), 39, 93, 1, 142);

gws_train = reshape(gwsdata(trainInd, :), 39, 93, 1, 142);

temp_test = reshape(tempdata(testInd, :), 39, 93, 1, 36);

ndvi_test = reshape(ndvidata(testInd, :), 39, 93, 1, 36);

prec_test = reshape(precdata(testInd, :), 39, 93, 1, 36);

pet_test = reshape(petdata(testInd, :), 39, 93, 1, 36);

lst_test = reshape(lstdata(testInd, :), 39, 93, 1, 36);

sm_test = reshape(smdata(testInd, :), 39, 93, 1, 36);

gws_test = reshape(gwsdata(testInd, :), 39, 93, 1, 36);

%数据归一化

temp_train_norm = normalize(temp_train);

temp_test_norm = normalize(temp_test);

%ndvi_train_norm = normalize(ndvi_train);

%ndvi_test_norm = normalize(ndvi_test);

prec_train_norm = normalize(prec_train);

prec_test_norm = normalize(prec_test);

pet_train_norm = normalize(pet_train);

pet_test_norm = normalize(pet_test);

lst_train_norm = normalize(lst_train);

lst_test_norm = normalize(lst_test);

sm_train_norm = normalize(sm_train);

sm_test_norm = normalize(sm_test);

%gws_train_norm = normalize(gws_train);

%gws_test_norm = normalize(gws_test);

% 构造网络结构

layers = [

imageInputLayer([39 93 6],"Name","imageinput")

convolution2dLayer([3 3],50,"Name","conv2d_1","Padding",[1 1 1 1])

reluLayer("Name","Relu_1")

maxPooling2dLayer([2 2],"Name","pool_1","Padding","same","Stride",[2 2])

dropoutLayer(0.2,"Name","dropout_1")

convolution2dLayer([3 3],100,"Name","conv2d_2","Padding",[1 1 1 1])

reluLayer("Name","Relu_2")

maxPooling2dLayer([2 2],"Name","pool_2","Padding","same","Stride",[2 2])

dropoutLayer(0.2,"Name","dropout_2")

convolution2dLayer([3 3],200,"Name","conv2d_3","Padding",[1 1 1 1])

reluLayer("Name","Relu_3")

maxPooling2dLayer([2 2],"Name","pool_3","Padding","same","Stride",[2 2])

dropoutLayer(0.3,"Name","dropout_3")

convolution2dLayer([3 3],300,"Name","conv2d_4","Padding",[1 1 1 1])

reluLayer("Name","Relu_4")

fullyConnectedLayer(200,"Name","fc_1")

reluLayer("Name","relu_5")

dropoutLayer(0.4,"Name","dropout_4")

fullyConnectedLayer(1164*2783,"Name","fc_2")

regressionLayer("Name","Output")];

lgraph = layerGraph(layers);

options = trainingOptions('adam', ...

'MaxEpochs', 32, ...

'MiniBatchSize', 15, ...

'Shuffle', 'every-epoch', ...

'InitialLearnRate', 1e-2, ...

'LearnRateDropFactor', 0.01, ...

'LearnRateDropPeriod', 10, ...

'LearnRateSchedule','piecewise',...

'Plots', 'training-progress',...

'ExecutionEnvironment', 'cpu', ...

"L2Regularization", 0.001,...

'Verbose', true);

net = trainNetwork({temp_train, ndvi_train, prec_train,pet_train,sm_train,lst_train}, gws_train, lgraph, options);

Error for trainnetwork:Invalid training data. The predictor and response must have the same number of observations.

But when I checked the data, I found that the data seemed to be correct. Could you tell me why? Thanks a lot!

[Image Analyst]

Start your own question and put all these variables (temp_train, ndvi_train, prec_train,pet_train,sm_train,lst_train}, gws_train, lgraph, options) into a .mat file and attach it with the paperclip icon. Also give your code there in your new question.

Answer 3

[Cris LaPierre]

I believe the problem is due to the fact that Xtrain is a cell array. I would replace your for loop with the following code instead:

Xtrain=Xtemp;

However, that will lead to a second error:

Error using trainNetwork

Targets and network outputs must be the same size. The networks outputs have size 1×5×1000 and the targets have size 1×1×1000.

This is because the network output (result of nodeDegree) is a 1000 x 5. You need to apply your knowledge is this space to have the network output and target variable be the same size. Here is a configuration that got the training to run for me.

clc;
clear;
close all;
check_node_degs_vec=[1,2,3,4,5];
var_node_degs_vec=[1,2,3,4,5];
x=0.5;
decoding_threshold=0.3;
sample_size=1000;
Xtemp=decoding_threshold*rand([1,sample_size]);
% for i=1:sample_size
%     Xtrain{i}=Xtemp(i);
% end
Xtrain=Xtemp;
Xtrain=reshape(Xtrain, [1,sample_size]); 
Ytrain=zeros(size(Xtrain));
lgraph = layerGraph;
layer_1=[sequenceInputLayer(1,"Name","seq_input_1")
    functionLayer(@(x) nodeDegreeLayer(x, check_node_degs_vec),"Name","func_ro")
    fullyConnectedLayer(1,"Name","ro")
    reluLayer('Name','relu_1');
    functionLayer(@(x) nodeDegreeLayer(x, var_node_degs_vec),"Name","func_lam")
    fullyConnectedLayer(1,"Name","lam")
    reluLayer('Name','relu_2');
    additionLayer(2,'Name','add_1');
    reluLayer('Name','relu_3');
    regressionLayer
    ];
lgraph = addLayers(lgraph,layer_1);
res_input=functionLayer(@(X) -X,"Name","res_func");
lgraph = addLayers(lgraph,res_input);
lgraph = connectLayers(lgraph,'seq_input_1','res_func');
lgraph = connectLayers(lgraph,'res_func','add_1/in2');
%dlnet = dlnetwork(lgraph);
% figure
% plot(lgraph)
options = trainingOptions("adam", ...
    MaxEpochs=60, ...
    miniBatchSize=1, ...
    Plots="training-progress", ...
    Verbose=0);
net = trainNetwork(Xtrain,Ytrain,lgraph,options);


function y = nodeDegreeLayer(x, node_degs_vec)
 
y = sum((1-x).^(node_degs_vec-1),2);
end

Answer 4

[massoud pourmandi]

[massoud pourmandi]

As you can see, functionlayer is responsible for expanding x value to a vector in that each element is related of x. Based on your modification, we have a scaler to scaler map not a scaler to vector map. I noticed that when we increrase the batch size to more than 1, It reflects itself in X. do you have any suguesstion to handle that?. thanks in advance.