how to Train Network on Image and Feature Data with more then one feature input?

In this example: openExample('nnet/TrainNetworkOnImageAndFeatureDataExample')

I want to change numFeatures fro 1 to 3. I have added a 3 element vector to X2Train

>> preview(dsTrain)

ans =

1×3 cell array

{28×28 double} {[-42 0.9891 0.5122]} {[3]}

layers = [

imageInputLayer(imageInputSize,'Normalization','none','Name','images')

convolution2dLayer(filterSize,numFilters,'Name','conv')

reluLayer('Name','relu')

fullyConnectedLayer(50,'Name','fc1')

concatenationLayer(1,3,'Name','concat')

fullyConnectedLayer(numClasses,'Name','fc2')

softmaxLayer('Name','softmax')];

lgraph = layerGraph(layers);

featInput = featureInputLayer(numFeatures,Name="features");

lgraph = addLayers(lgraph,featInput);

lgraph = connectLayers(lgraph,"features","cat/in2");

lgraph = connectLayers(lgraph,"features","cat/in3");

figure

plot(lgraph)

when I run it I keep getting this error:

Error using trainNetwork

Input datastore returned more than one observation per row for network input 2.

Any help would be appreciated!

Answer 1

[Ben]

The subtle issue here is that the feature data needs to read out of the datastore as a NumFeatures x 1 vector as documented here: https://www.mathworks.com/help/deeplearning/ug/datastores-for-deep-learning.html

So you'll need to transpose your feature data either before it goes into the datastore, or as a transform of your existing datastore (e.g. transformedDsTrain = transform(dsTrain,@(x) [x(1),{x{2}.'},x(3)]);).

However you'll next run into another subtle issue at the concatenationLayer since the output of layer 'fc2' will have size 1(S) x 1(S) x 50(C) x BatchSize(B). This needs squeezing so it can be concatenated with the feature data in shape 3(C) x BatchSize(B). Probably the easiest way to do that is with a functionLayer. Here's some code to get your network running:

imageInputSize = [28,28,1];

filterSize = 3;

numFilters = 8;

numClasses = 10;

numFeatures = 3;

layers = [

imageInputLayer(imageInputSize,'Normalization','none','Name','images')

convolution2dLayer(filterSize,numFilters,'Name','conv')

reluLayer('Name','relu')

fullyConnectedLayer(50,'Name','fc1')

squeezeLayer()

concatenationLayer(1,3,'Name','cat')

fullyConnectedLayer(numClasses,'Name','fc2')

softmaxLayer('Name','softmax')

classificationLayer];

lgraph = layerGraph(layers);

featInput = featureInputLayer(numFeatures,Name="features");

lgraph = addLayers(lgraph,featInput);

lgraph = connectLayers(lgraph,"features","cat/in2");

lgraph = connectLayers(lgraph,"features","cat/in3");

numObservations = 100;

fakeImages = randn([imageInputSize,numObservations]);

imagesDS = arrayDatastore(fakeImages,IterationDimension=4);

fakeFeatures = randn([numObservations,numFeatures]);

featureDS = arrayDatastore(fakeFeatures.',IterationDimension=2);

fakeTargets = categorical(mod(1:numObservations,numClasses));

targetDS = arrayDatastore(fakeTargets,IterationDimension=2);

ds = combine(imagesDS,featureDS,targetDS);

opts = trainingOptions("adam","MaxEpochs",1);

trainNetwork(ds,lgraph,opts);

function layer = squeezeLayer(args)

arguments

args.Name='';

end

layer = functionLayer(@squeezeLayerFcn,"Name",args.Name,"Formattable",true);

end

function x = squeezeLayerFcn(x)

x = squeeze(x);

% Since squeeze will squeeze out some dimensions, we need to relabel x.

% Assumption: x does not have a 'T' dimension.

n = ndims(x);

newdims = [repelem('S',n-2),'CB'];

x = dlarray(x,newdims);

end

As a final note - I notice you're concatenating the feature input layer to itself, alongside the outputs of layer 'fc2'. Maybe that's intentional, it seemed slightly curious to me.

[Andrew Brindle]

Thanks! This is really helpful! I figured it was a formatting problem, I'm kind of new to datastores, but trying to learn.

I'm definitely not try to concatenate the feature layer into itself, I’m trying to concatenate it with the image data. I’d like to take the outputs of the fullyConnectedLayer(fc1) and concatenate them with the 3 features to my mind that should be 50+3 = 53 inputs to next fullyConnectedLayer(fc2). I guess I didn’t do that correctly.

[Kenneth]

Hi ben,

I am trying to concatenate image layer (224 x 224 x 3) with feature input layer (4 features) for a regression CNN but am getting the below error:

Error using trainNetwork

Invalid training data. The output size (1) of the last layer does not match the response size (1569).

I have 1569 observations and since I am doing a regression network I have put the last fully connetced layer to be 1 output but it is returning the above error.

Data store prep:

SimTrainDS = arrayDatastore(SimTrain,IterationDimension=4);

aw_SimTrainDS = arrayDatastore(awSimTrain.',IterationDimension=2);

betaTrainDS = arrayDatastore(betaTrain,IterationDimension=2);

All_TrainDs = combine(SimTrainDS,aw_SimTrainDS,betaTrainDS);

layers = [

imageInputLayer([224 224 3],Normalization="none") % coloured image size, height, width, and number of channels respectively

convolution2dLayer(3,8,'Padding','same')

batchNormalizationLayer

reluLayer

averagePooling2dLayer(2,'Stride',2)

convolution2dLayer(3,16,'Padding','same')

batchNormalizationLayer

reluLayer

averagePooling2dLayer(2,'Stride',2)

convolution2dLayer(3,32,'Padding','same')

batchNormalizationLayer

reluLayer

convolution2dLayer(3,32,'Padding','same')

batchNormalizationLayer

reluLayer

dropoutLayer(0.2)

reluLayer

fullyConnectedLayer(50)

flattenLayer

concatenationLayer(1,4,Name="concat")

fullyConnectedLayer(1)

regressionLayer];

%Convert the layers to a layer graph.

lgraph = layerGraph(layers);

% Add a feature input layer to the layer graph and connect it

% to the concatenation layer.

numFeatures=4;

featInput = featureInputLayer(numFeatures,Name="Param_features");

lgraph = addLayers(lgraph,featInput);

lgraph = connectLayers(lgraph,"Param_features","concat/in2");

lgraph = connectLayers(lgraph,"Param_features","concat/in3");

lgraph = connectLayers(lgraph,"Param_features","concat/in4");

%Visualize the network.

figure

set(gcf, 'Color', 'w');

plot(lgraph)

miniBatchSize = 128;

options = trainingOptions('sgdm', ...

'MiniBatchSize',miniBatchSize, ...

'MaxEpochs',30, ...

'InitialLearnRate',1e-3, ...

'LearnRateSchedule','piecewise', ...

'LearnRateDropFactor',0.1, ...

'LearnRateDropPeriod',20, ...

'Shuffle','every-epoch', ...

'Plots','training-progress', ...

'Verbose',false);

Reg_CNNnetSim = trainNetwork(All_TrainDs,lgraph,options);

[Ben]

The error is suggestive that the issue is with the datastore setup - trainNetwork thinks that your responses/targets have size 1569, but that's actually the batch/observation dimension.

You can find documentation on datastore inputs for trainNetwork here: https://www.mathworks.com/help/deeplearning/ug/datastores-for-deep-learning.html

If you could call:

data = read(All_TrainDs)

and post information about data, we might be able to debug - in particular we want to check the size of each of the inputs/predictors and the output/response in data.

[Kenneth]

Ok.

Thanks so much Ben for the time and prompt response.