Why do I get error "Invalid training data for multiple-input network" while training deep neural networks with multiple image inputs?

I have designed a neural network "layerGraph" for binary classification that takes two 3D image inputs. I created a combined datastore from two image datastores and then used it as input argument for "tranNetwork" function.

>> imds_1 = imageDatastore(imds_path_1, 'IncludeSubfolders',true,'LabelSource','foldernames');
>> imds_2 = imageDatastore(imds_path_2, 'IncludeSubfolders',true,'LabelSource','foldernames');
>> train_ds = combine(imds_1, imds_2);
>> net = trainNetwork(combined_ds ,layers,options);

I got the following error during training:

Error using trainNetwork (line ###)
Invalid training data for multiple-input network. For a
network with 2 inputs and 1 output, the datastore read
function must return an M-by-3 cell array, but it
returns an M-by-2 cell array.

I am using following documentation page as reference:

https://www.mathworks.com/help/deeplearning/ug/multiple-input-and-multiple-output-networks.html

How to resolve this issue?

Answer 1

[MathWorks Support Team]

The following documentation page mentions that the datastore should output outputs as a cell array with (numInputs + 1) columns.

https://www.mathworks.com/help/deeplearning/ug/multiple-input-and-multiple-output-networks.html

In this case with two inputs, the final datastore being used for "trainNetwork" should output a Mx3 cell array as following: 

>> read(train_ds) 
ans = 1×3 cell array
{32×32×3 uint8} {32×32×3 uint8} {[Anomalies]}

The first two columns should correspond to inputs and third column should corresponds to categorical response variable. Currently, the datastore outputs a Mx2 cell array with image inputs only and results in an error during network training. 

To resolve this issue, the datastore should output data in the required format (i-e Mx3 cell array). 

This can be achieved by saving metadata (i-e filepaths and class labels) for all the images in individual MAT-files and then creating a "fileDatastore" from MAT-files using a custom "ReadFcn". Make sure that the output of the data from "ReadFcn" is a cell array with 3 columns.

For example:

% save metadata for all samples in indivdual MAT-files
save('sample1.mat','filepath1','filepath2','label')

% create a fileDatastore from MAT-files using a custom "ReadFcn"
trainds = fileDatastore(matFileFolder,'FileExtensions','.mat','ReadFcn',@matRead);

function output = matRead(fn)
s = load(fn);
img1 = imread(s.filepath1);
img2 = imread(s.filepath2);
label = s.label;
output = {img1,img2,label};
end