how to combine/append 2 dagnetworks/LayerGraphs: Add layer graph or network to layer graph or network

Hi,

I am trying to combine/append 2 complex dagnetworks. One is a downloaded pretrained CNN (googlenet). The other is a complex LSTM ([netLSTM,info] = trainNetwork(sequencesTrain,labelsTrain,lg,options)) which I have entered and trained.

I want to use the output of the CNN as the input to the LSTM, with approprite editing and connecting of the input and output layers.

>>netCNN=googlenet;

>>[netLSTM,info] = trainNetwork(sequencesTrain,labelsTrain,lg,options);

>>cnnLayers = layerGraph(netCNN);

>>lstmLayers = layerGraph(netLSTM);

When I use addLayers I get:

">> netest=addLayers(cnnLayers,lstmLayers)

Error using nnet.internal.cnn.util.validateLayersForLayerGraph (line 22)

Expected input to be one of these types:

nnet.cnn.layer.Layer

Instead its type was nnet.cnn.LayerGraph.

Error in nnet.cnn.LayerGraph>iValidateLayers (line 550)

larray = nnet.internal.cnn.util.validateLayersForLayerGraph(larray, existingLayers);

Error in nnet.cnn.LayerGraph/addLayers (line 184)

larray = iValidateLayers(larray, existingLayers);"

I have tried to use netest=addLayers(cnnLayers,lstmLayers.Layers) but that removes the branching that I have used in the lstmLayers dagnetwork/layerGraph. In addition the training weights and biases of the LSTM do not transfer.

In net2=addLayers(net1, larray) net1 may be a dagnetwork/layerGraph, but larray can only be layers connected sequentially, no branching allowed.

Thank you

[Peter]

Example in R2022b (error message has changed)

layers = [

imageInputLayer([28 28 1])

convolution2dLayer(5,16,'Padding','same')

batchNormalizationLayer

reluLayer('Name','relu_1')

convolution2dLayer(3,32,'Padding','same','Stride',2)

batchNormalizationLayer

reluLayer

convolution2dLayer(3,32,'Padding','same')

batchNormalizationLayer

reluLayer

additionLayer(2,'Name','add')

averagePooling2dLayer(2,'Stride',2)

fullyConnectedLayer(10)

softmaxLayer

classificationLayer];

lgraph = layerGraph(layers);

skipConv = convolution2dLayer(1,32,'Stride',2,'Name','skipConv');

lgraph = addLayers(lgraph,skipConv);

lgraph = connectLayers(lgraph,'relu_1','skipConv');

lgraph = connectLayers(lgraph,'skipConv','add/in2');

figure

plot(lgraph);

cnn=googlenet;

cnnLayers=layerGraph(cnn);

lg2=addLayers(cnnLayers,lgraph)

Error using nnet.cnn.LayerGraph/addLayers

lg2=addLayers(cnnLayers,lgraph)

↑

Invalid argument at position 2. Input must be an array of layers.

Answer 1

[Katja Mogalle]

The easiest way to combine complicated networks is to use the app "Deep Network Designer". You can find it in the apps gallery in the MATLAB desktop toolstrip or you just type the following command in the command window: deepNetworkDesigner. In the app, you can import networks from the workspace.

You can also read more about this app here: https://uk.mathworks.com/help/deeplearning/ref/deepnetworkdesigner-app.html

Note that currently, your pretrained CNN expects image data as input. The output of the network is plain features (no more spatial dimensions and no sequence/time dimensions). This might be problematic when you want to use an LSTM network on this output. However, Deep Network Designer let's you make changes to the network structure (e.g. swapping out input layers, deleting layers, and inserting layers) very easily.

I don't have a good understanding of what you're trying to achieve at the end so I can't provide more specific help at this point. But if you're intending to build a network for video processing, you could start by looking at this documentation example: https://uk.mathworks.com/help/deeplearning/ug/classify-videos-using-deep-learning.html

[Peter]

Thank you.

Peter