Multi-Input CNN for image classification

I have two different folders of images for 5 objects. I want to create a multi-input CNN as attached here. How can I send two different folders in trainNetwork while training the network and for testing as well?

[Srivardhan Gadila]

From R2020b onwards we can directly use arrayDatastore function instead of saving the label data to disk and loading it, as mentioned in the answer. For versions less than R2020b the answer would be the workaround.

Answer 1

[Srivardhan Gadila]

In order to train Multiple input networks you have to use a datastore & After referring to Input Arguments & ds of trainNetwork, Multiple-Input and Multiple-Output Networks, Input Datastore for Training, Validation, and Inference, Read One Array at a Time From Large MAT-File the following is one possible way:

Save all the categorical labels in a mat file & While loading the labels in command window it should look like below:(N(number of training samples) = 4000 in the below case)

>> labels = load('labelData.mat')

labels = 

  struct with fields:

    labels: [4000×1 categorical]

Note that the order of the labels should be i.e., sorted in the same order of the files in the imageDatastore used for input1 and input2.

trainpath1 = fullfile("PATH TO INPUT1 IMAGES");
trainpath2 = fullfile("PATH TO INPUT2 IMAGES");
imds1 = imageDatastore(trainpath1, 'IncludeSubfolders',true, 'FileExtensions','.PNG','LabelSource','foldernames');
imds2 = imageDatastore(trainpath2, 'IncludeSubfolders',true, 'FileExtensions','.PNG','LabelSource','foldernames');
labelds = fileDatastore('labelData.mat','ReadFcn',@myReadFcn,'ReadMode','partialfile');
cds = combine(imds1,imds2,labelds);

% Change the image sizes accordingly
imsize1 = [32 80 3];
imsize2 = [48 64 3];
numClasses = 5;
%% Define muliple input network
layers1 = [
    imageInputLayer(imsize1,'Name','input1')  
    convolution2dLayer(3,16,'Padding','same','Name','conv_1')
    reluLayer('Name','relu_1')
    fullyConnectedLayer(10,'Name','fc11')
    additionLayer(2,'Name','add')
    fullyConnectedLayer(numClasses,'Name','fc12')
    softmaxLayer('Name','softmax')
    classificationLayer('Name','classOutput')];

lgraph = layerGraph(layers1);

layers2 = [imageInputLayer(imsize2,'Name','input2')
    convolution2dLayer(3,16,'Padding','same','Name','conv_2')
    reluLayer('Name','relu_2')
    fullyConnectedLayer(10,'Name','fc21')];

lgraph = addLayers(lgraph,layers2);
lgraph = connectLayers(lgraph,'fc21','add/in2');
plot(lgraph)

%% Define trainingOptions and also set 'Shuffle' to 'never' for this workaround to work
options = trainingOptions('adam', ...
    'InitialLearnRate',0.005, ...
    'LearnRateSchedule','piecewise',...
    'MaxEpochs',3, ...
    'MiniBatchSize',1, ...
    'Verbose',1, ...
    'Plots','training-progress');

net = trainNetwork(cds,lgraph,options);

%% custom read function to be used for the labels fileDatastore 
%to ensure the read(fileDatastore) returns 1 label at a time
function [data,userdata,done] = myReadFcn(filename,userdata)
% If variable list is empty, 
% create list of variables from the file
if isempty(userdata) 
    labels = load(filename);
    userdata = labels.labels;
end
% Load a variable from the list of variables
data = userdata(1);
% Remove the newly-read variable from the list
userdata(1) = []; 
% Move on to the next file if this file is done reading.
done = isempty(userdata); 
end

For testing the network:

testimds1 = imageDatastore(testpath1);
testimds2 = imageDatastore(testpath2);
testcds = combine(testimds1,testimds2);
YPred = predict(net,testcds,'MiniBatchSize',4)

[hongyang zhao]

Dear Gadila , can you give me an example of creating labeldata. mar, or related code. Thank you

[Ala Harb]

I have a question does "labelData.mat" file should contain training labels of both "imds1" & "imds2" in order .. or it is enough to store the labels of one of the imds, So, when we use combine it will consider the labelsds for imds1 & imds2 ? .. Just I need to have clarification of how does data stored in "labelData.mat"

cds = combine(imds1,imds2,labelds);

[Ali Hajnayeb]

Thanks for the answer. I implemented it. Unfortunately, Matlab does not support validation for multi-input networks.

Do you have any suggestion for using validation data?

[Chunyan]

Dear Hajnayed, do you have any solutions to the problem that Matlab does not support validation for multi-input network? looking forward to your reply，thank you！

[MAHMOUD EID]

can you share the code for formatting the labels as a struct with fields?