"Invalid training data. Predictors and responses must have the same number of observations"

39 vues (au cours des 30 derniers jours)
I'd be appreciated it if someone help me to debug my code. Thanks in advance.
clc;
clear;
close all;
check_node_degs_vec=[1,2,3,4,5];
var_node_degs_vec=[1,2,3,4,5];
x=0.5;
decoding_threshold=0.3;
sample_size=1000;
Xtemp=decoding_threshold*rand([1,sample_size]);
for i=1:sample_size
Xtrain{i}=Xtemp(i);
end
Xtrain=reshape(Xtrain, [1,sample_size]);
Ytrain=zeros(size(Xtrain));
lgraph = layerGraph;
layer_1=[sequenceInputLayer(1,"Name","seq_input_1")
functionLayer(@(x) nodeDegreeLayer(x, check_node_degs_vec),"Name","func_ro")
fullyConnectedLayer(1,"Name","ro")
reluLayer('Name','relu_1');
functionLayer(@(x) nodeDegreeLayer(x, var_node_degs_vec),"Name","func_lam")
fullyConnectedLayer(1,"Name","lam")
reluLayer('Name','relu_2');
additionLayer(2,'Name','add_1');
reluLayer('Name','relu_3');
regressionLayer
];
lgraph = addLayers(lgraph,layer_1);
res_input=functionLayer(@(X) -X,"Name","res_func");
lgraph = addLayers(lgraph,res_input);
lgraph = connectLayers(lgraph,'seq_input_1','res_func');
lgraph = connectLayers(lgraph,'res_func','add_1/in2');
%dlnet = dlnetwork(lgraph);
% figure
% plot(lgraph)
options = trainingOptions("adam", ...
MaxEpochs=60, ...
miniBatchSize=1, ...
Plots="training-progress", ...
Verbose=0);
net = trainNetwork(Xtrain,Ytrain,lgraph,options);
Error using trainNetwork
Invalid training data. Predictors and responses must have the same number of observations.
function y = nodeDegreeLayer(x, node_degs_vec)
y = (1-x).^(node_degs_vec-1);
end
Error using trainNetwork
Invalid training data. Predictors and responses must have the same number of observations.
Error in DAGNET (line 49)
net = trainNetwork(Xtrain,Ytrain,lgraph,options);
  2 commentaires
massoud pourmandi
massoud pourmandi le 30 Juin 2022
I think that the way that I generate Xtrain and Ytrain is wrong, but I dont know how to gernerate it.

Connectez-vous pour commenter.

Réponse acceptée

Cris LaPierre
Cris LaPierre le 1 Juil 2022
Note that, in the image you shared, they also all collapse back to a single node. My suspicion, then, is that you would use the lambdas to weight the result of each value. Again, you would know more about the particulars here than me.
I defined a vector of lambdas and used pagemtimes to compute the output of nodeDegreeLayer
clc;
clear;
close all;
check_node_degs_vec=[1,2,3,4,5];
var_node_degs_vec=[1,2,3,4,5];
lambda = [0.1 0.3 0.2 0.15 0.25]';
x=0.5;
decoding_threshold=0.3;
sample_size=1000;
Xtemp=decoding_threshold*rand([1,sample_size]);
% for i=1:sample_size
% Xtrain{i}=Xtemp(i);
% end
Xtrain=Xtemp;
Xtrain=reshape(Xtrain, [1,sample_size]);
Ytrain=zeros(size(Xtrain));
lgraph = layerGraph;
layer_1=[sequenceInputLayer(1,"Name","seq_input_1")
functionLayer(@(x) nodeDegreeLayer(x, check_node_degs_vec,lambda),"Name","func_ro")
fullyConnectedLayer(1,"Name","ro")
reluLayer('Name','relu_1');
functionLayer(@(x) nodeDegreeLayer(x, var_node_degs_vec,lambda),"Name","func_lam")
fullyConnectedLayer(1,"Name","lam")
reluLayer('Name','relu_2');
additionLayer(2,'Name','add_1');
reluLayer('Name','relu_3');
regressionLayer
];
lgraph = addLayers(lgraph,layer_1);
res_input=functionLayer(@(X) -X,"Name","res_func");
lgraph = addLayers(lgraph,res_input);
lgraph = connectLayers(lgraph,'seq_input_1','res_func');
lgraph = connectLayers(lgraph,'res_func','add_1/in2');
%dlnet = dlnetwork(lgraph);
% figure
% plot(lgraph)
options = trainingOptions("adam", ...
MaxEpochs=60, ...
miniBatchSize=1, ...
Plots="training-progress", ...
Verbose=0);
net = trainNetwork(Xtrain,Ytrain,lgraph,options);
function y = nodeDegreeLayer(x, node_degs_vec,lambda)
y = pagemtimes((1-x).^(node_degs_vec-1),lambda);
end
  2 commentaires
l
l le 6 Oct 2023
Hello,I have the same problem, but it hasn't been solved yet , so I hope you can give me some advice.My code is as follows:
gwsdata = reshape(gwsdata,205,3627);
tempdata = reshape(tempdata,205,3627);
ndvidata = reshape(ndvidata,205,3627);
precdata = reshape(precdata,205,3627);
petdata = reshape(petdata,205,3627);
lstdata = reshape(lstdata,205,3627);
smdata = reshape(smdata,205,3627);
%划分训练集,验证集
[trainInd, testInd] = dividerand(205, 0.8, 0.2); % 80%的数据作为训练集,20%的数据作为测试集
temp_train = reshape(tempdata(trainInd, :), 39, 93, 1, 142);
ndvi_train = reshape(ndvidata(trainInd, :), 39, 93, 1, 142);
prec_train = reshape(precdata(trainInd, :), 39, 93, 1, 142);
pet_train = reshape(petdata(trainInd, :), 39, 93, 1, 142);
lst_train = reshape(lstdata(trainInd, :), 39, 93, 1, 142);
sm_train = reshape(smdata(trainInd, :), 39, 93, 1, 142);
gws_train = reshape(gwsdata(trainInd, :), 39, 93, 1, 142);
temp_test = reshape(tempdata(testInd, :), 39, 93, 1, 36);
ndvi_test = reshape(ndvidata(testInd, :), 39, 93, 1, 36);
prec_test = reshape(precdata(testInd, :), 39, 93, 1, 36);
pet_test = reshape(petdata(testInd, :), 39, 93, 1, 36);
lst_test = reshape(lstdata(testInd, :), 39, 93, 1, 36);
sm_test = reshape(smdata(testInd, :), 39, 93, 1, 36);
gws_test = reshape(gwsdata(testInd, :), 39, 93, 1, 36);
%数据归一化
temp_train_norm = normalize(temp_train);
temp_test_norm = normalize(temp_test);
%ndvi_train_norm = normalize(ndvi_train);
%ndvi_test_norm = normalize(ndvi_test);
prec_train_norm = normalize(prec_train);
prec_test_norm = normalize(prec_test);
pet_train_norm = normalize(pet_train);
pet_test_norm = normalize(pet_test);
lst_train_norm = normalize(lst_train);
lst_test_norm = normalize(lst_test);
sm_train_norm = normalize(sm_train);
sm_test_norm = normalize(sm_test);
%gws_train_norm = normalize(gws_train);
%gws_test_norm = normalize(gws_test);
% 构造网络结构
layers = [
imageInputLayer([39 93 6],"Name","imageinput")
convolution2dLayer([3 3],50,"Name","conv2d_1","Padding",[1 1 1 1])
reluLayer("Name","Relu_1")
maxPooling2dLayer([2 2],"Name","pool_1","Padding","same","Stride",[2 2])
dropoutLayer(0.2,"Name","dropout_1")
convolution2dLayer([3 3],100,"Name","conv2d_2","Padding",[1 1 1 1])
reluLayer("Name","Relu_2")
maxPooling2dLayer([2 2],"Name","pool_2","Padding","same","Stride",[2 2])
dropoutLayer(0.2,"Name","dropout_2")
convolution2dLayer([3 3],200,"Name","conv2d_3","Padding",[1 1 1 1])
reluLayer("Name","Relu_3")
maxPooling2dLayer([2 2],"Name","pool_3","Padding","same","Stride",[2 2])
dropoutLayer(0.3,"Name","dropout_3")
convolution2dLayer([3 3],300,"Name","conv2d_4","Padding",[1 1 1 1])
reluLayer("Name","Relu_4")
fullyConnectedLayer(200,"Name","fc_1")
reluLayer("Name","relu_5")
dropoutLayer(0.4,"Name","dropout_4")
fullyConnectedLayer(1164*2783,"Name","fc_2")
regressionLayer("Name","Output")];
lgraph = layerGraph(layers);
options = trainingOptions('adam', ...
'MaxEpochs', 32, ...
'MiniBatchSize', 15, ...
'Shuffle', 'every-epoch', ...
'InitialLearnRate', 1e-2, ...
'LearnRateDropFactor', 0.01, ...
'LearnRateDropPeriod', 10, ...
'LearnRateSchedule','piecewise',...
'Plots', 'training-progress',...
'ExecutionEnvironment', 'cpu', ...
"L2Regularization", 0.001,...
'Verbose', true);
net = trainNetwork({temp_train, ndvi_train, prec_train,pet_train,sm_train,lst_train}, gws_train, lgraph, options);
Error for trainnetwork:Invalid training data. The predictor and response must have the same number of observations.
But when I checked the data, I found that the data seemed to be correct. Could you tell me why? Thanks a lot!
Cris LaPierre
Cris LaPierre le 6 Oct 2023
Without your data, it is pretty hard to provide a specific answer. However, if you have the same error, then your predictor and response inputs do not have the same number of observations.

Connectez-vous pour commenter.

Plus de réponses (3)

Image Analyst
Image Analyst le 1 Juil 2022
Instead of
Xtemp=decoding_threshold*rand([1,sample_size]);
for i=1:sample_size
Xtrain{i}=Xtemp(i);
end
Xtrain=reshape(Xtrain, [1,sample_size]);
Ytrain=zeros(size(Xtrain));
maybe
Xtemp=decoding_threshold*rand([sample_size, 1]); % Column vector. But why random numbers?!?
Xtrain=Xtemp;
Ytrain=zeros(size(Xtrain)); % Huh?!?
but it looks like you're training on random inputs, and a desired ground truth of all zeros. What's up with that? That makes no sense, at least not to me.
  3 commentaires
l
l le 6 Oct 2023
Hello,I have the same problem, but it hasn't been solved yet , so I hope you can give me some advice.My code is as follows:
gwsdata = reshape(gwsdata,205,3627);
tempdata = reshape(tempdata,205,3627);
ndvidata = reshape(ndvidata,205,3627);
precdata = reshape(precdata,205,3627);
petdata = reshape(petdata,205,3627);
lstdata = reshape(lstdata,205,3627);
smdata = reshape(smdata,205,3627);
%划分训练集,验证集
[trainInd, testInd] = dividerand(205, 0.8, 0.2); % 80%的数据作为训练集,20%的数据作为测试集
temp_train = reshape(tempdata(trainInd, :), 39, 93, 1, 142);
ndvi_train = reshape(ndvidata(trainInd, :), 39, 93, 1, 142);
prec_train = reshape(precdata(trainInd, :), 39, 93, 1, 142);
pet_train = reshape(petdata(trainInd, :), 39, 93, 1, 142);
lst_train = reshape(lstdata(trainInd, :), 39, 93, 1, 142);
sm_train = reshape(smdata(trainInd, :), 39, 93, 1, 142);
gws_train = reshape(gwsdata(trainInd, :), 39, 93, 1, 142);
temp_test = reshape(tempdata(testInd, :), 39, 93, 1, 36);
ndvi_test = reshape(ndvidata(testInd, :), 39, 93, 1, 36);
prec_test = reshape(precdata(testInd, :), 39, 93, 1, 36);
pet_test = reshape(petdata(testInd, :), 39, 93, 1, 36);
lst_test = reshape(lstdata(testInd, :), 39, 93, 1, 36);
sm_test = reshape(smdata(testInd, :), 39, 93, 1, 36);
gws_test = reshape(gwsdata(testInd, :), 39, 93, 1, 36);
%数据归一化
temp_train_norm = normalize(temp_train);
temp_test_norm = normalize(temp_test);
%ndvi_train_norm = normalize(ndvi_train);
%ndvi_test_norm = normalize(ndvi_test);
prec_train_norm = normalize(prec_train);
prec_test_norm = normalize(prec_test);
pet_train_norm = normalize(pet_train);
pet_test_norm = normalize(pet_test);
lst_train_norm = normalize(lst_train);
lst_test_norm = normalize(lst_test);
sm_train_norm = normalize(sm_train);
sm_test_norm = normalize(sm_test);
%gws_train_norm = normalize(gws_train);
%gws_test_norm = normalize(gws_test);
% 构造网络结构
layers = [
imageInputLayer([39 93 6],"Name","imageinput")
convolution2dLayer([3 3],50,"Name","conv2d_1","Padding",[1 1 1 1])
reluLayer("Name","Relu_1")
maxPooling2dLayer([2 2],"Name","pool_1","Padding","same","Stride",[2 2])
dropoutLayer(0.2,"Name","dropout_1")
convolution2dLayer([3 3],100,"Name","conv2d_2","Padding",[1 1 1 1])
reluLayer("Name","Relu_2")
maxPooling2dLayer([2 2],"Name","pool_2","Padding","same","Stride",[2 2])
dropoutLayer(0.2,"Name","dropout_2")
convolution2dLayer([3 3],200,"Name","conv2d_3","Padding",[1 1 1 1])
reluLayer("Name","Relu_3")
maxPooling2dLayer([2 2],"Name","pool_3","Padding","same","Stride",[2 2])
dropoutLayer(0.3,"Name","dropout_3")
convolution2dLayer([3 3],300,"Name","conv2d_4","Padding",[1 1 1 1])
reluLayer("Name","Relu_4")
fullyConnectedLayer(200,"Name","fc_1")
reluLayer("Name","relu_5")
dropoutLayer(0.4,"Name","dropout_4")
fullyConnectedLayer(1164*2783,"Name","fc_2")
regressionLayer("Name","Output")];
lgraph = layerGraph(layers);
options = trainingOptions('adam', ...
'MaxEpochs', 32, ...
'MiniBatchSize', 15, ...
'Shuffle', 'every-epoch', ...
'InitialLearnRate', 1e-2, ...
'LearnRateDropFactor', 0.01, ...
'LearnRateDropPeriod', 10, ...
'LearnRateSchedule','piecewise',...
'Plots', 'training-progress',...
'ExecutionEnvironment', 'cpu', ...
"L2Regularization", 0.001,...
'Verbose', true);
net = trainNetwork({temp_train, ndvi_train, prec_train,pet_train,sm_train,lst_train}, gws_train, lgraph, options);
Error for trainnetwork:Invalid training data. The predictor and response must have the same number of observations.
But when I checked the data, I found that the data seemed to be correct. Could you tell me why? Thanks a lot!
Image Analyst
Image Analyst le 6 Oct 2023
Start your own question and put all these variables (temp_train, ndvi_train, prec_train,pet_train,sm_train,lst_train}, gws_train, lgraph, options) into a .mat file and attach it with the paperclip icon. Also give your code there in your new question.

Connectez-vous pour commenter.


Cris LaPierre
Cris LaPierre le 30 Juin 2022
I believe the problem is due to the fact that Xtrain is a cell array. I would replace your for loop with the following code instead:
Xtrain=Xtemp;
However, that will lead to a second error:
Error using trainNetwork
Targets and network outputs must be the same size. The networks outputs have size 1×5×1000 and the targets have size 1×1×1000.
This is because the network output (result of nodeDegree) is a 1000 x 5. You need to apply your knowledge is this space to have the network output and target variable be the same size. Here is a configuration that got the training to run for me.
clc;
clear;
close all;
check_node_degs_vec=[1,2,3,4,5];
var_node_degs_vec=[1,2,3,4,5];
x=0.5;
decoding_threshold=0.3;
sample_size=1000;
Xtemp=decoding_threshold*rand([1,sample_size]);
% for i=1:sample_size
% Xtrain{i}=Xtemp(i);
% end
Xtrain=Xtemp;
Xtrain=reshape(Xtrain, [1,sample_size]);
Ytrain=zeros(size(Xtrain));
lgraph = layerGraph;
layer_1=[sequenceInputLayer(1,"Name","seq_input_1")
functionLayer(@(x) nodeDegreeLayer(x, check_node_degs_vec),"Name","func_ro")
fullyConnectedLayer(1,"Name","ro")
reluLayer('Name','relu_1');
functionLayer(@(x) nodeDegreeLayer(x, var_node_degs_vec),"Name","func_lam")
fullyConnectedLayer(1,"Name","lam")
reluLayer('Name','relu_2');
additionLayer(2,'Name','add_1');
reluLayer('Name','relu_3');
regressionLayer
];
lgraph = addLayers(lgraph,layer_1);
res_input=functionLayer(@(X) -X,"Name","res_func");
lgraph = addLayers(lgraph,res_input);
lgraph = connectLayers(lgraph,'seq_input_1','res_func');
lgraph = connectLayers(lgraph,'res_func','add_1/in2');
%dlnet = dlnetwork(lgraph);
% figure
% plot(lgraph)
options = trainingOptions("adam", ...
MaxEpochs=60, ...
miniBatchSize=1, ...
Plots="training-progress", ...
Verbose=0);
net = trainNetwork(Xtrain,Ytrain,lgraph,options);
function y = nodeDegreeLayer(x, node_degs_vec)
y = sum((1-x).^(node_degs_vec-1),2);
end

massoud pourmandi
massoud pourmandi le 30 Juin 2022
  1 commentaire
massoud pourmandi
massoud pourmandi le 30 Juin 2022
As you can see, functionlayer is responsible for expanding x value to a vector in that each element is related of x. Based on your modification, we have a scaler to scaler map not a scaler to vector map. I noticed that when we increrase the batch size to more than 1, It reflects itself in X. do you have any suguesstion to handle that?. thanks in advance.

Connectez-vous pour commenter.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by