Visualize Features of a Convolutional Neural Network
This example shows how to visualize the features learned by convolutional neural networks.
Convolutional neural networks use features to classify images. The network learns these features itself during the training process. What the network learns during training is sometimes unclear. However, you can use the
deepDreamImage function to visualize the features learned.
The convolutional layers output a 3D activation volume, where slices along the third dimension correspond to a single filter applied to the layer input. The channels output by fully connected layers at the end of the network correspond to high-level combinations of the features learned by earlier layers.
You can visualize what the learned features look like by using
deepDreamImage to generate images that strongly activate a particular channel of the network layers.
The example requires Deep Learning Toolbox™ and Deep Learning Toolbox Model for GoogLeNet Network support package.
Load Pretrained Network
Load a pretrained GoogLeNet network.
net = googlenet;
Visualize Early Convolutional Layers
There are multiple convolutional layers in the GoogLeNet network. The convolutional layers towards the beginning of the network have a small receptive field size and learn small, low-level features. The layers towards the end of the network have larger receptive field sizes and learn larger features.
analyzeNetwork, view the network architecture and locate the convolutional layers.
Features on Convolutional Layer 1
layer to be the first convolutional layer. This layer is the second layer in the network and is named
layer = 2; name = net.Layers(layer).Name
name = 'conv1-7x7_s2'
Visualize the first 36 features learned by this layer using
deepDreamImage by setting
channels to be the vector of indices
'PyramidLevels' to 1 so that the images are not scaled. To display the images together, you can use
deepDreamImage uses a compatible GPU, by default, if available. Otherwise it uses the CPU. Using a GPU requires Parallel Computing Toolbox™ and a supported GPU device. For information on supported devices, see GPU Support by Release (Parallel Computing Toolbox).
channels = 1:36; I = deepDreamImage(net,name,channels, ... 'PyramidLevels',1);
|==============================================| | Iteration | Activation | Pyramid Level | | | Strength | | |==============================================| | 1 | 0.26 | 1 | | 2 | 6.99 | 1 | | 3 | 14.24 | 1 | | 4 | 21.49 | 1 | | 5 | 28.74 | 1 | | 6 | 35.99 | 1 | | 7 | 43.24 | 1 | | 8 | 50.50 | 1 | | 9 | 57.75 | 1 | | 10 | 65.00 | 1 | |==============================================|
figure I = imtile(I,'ThumbnailSize',[64 64]); imshow(I) title(['Layer ',name,' Features'],'Interpreter','none')
These images mostly contain edges and colors, which indicates that the filters at layer
'conv1-7x7_s2' are edge detectors and color filters.
Features on Convolutional Layer 2
The second convolutional layer is named
'conv2-3x3_reduce', which corresponds to layer 6. Visualize the first 36 features learned by this layer by setting
channels to be the vector of indices
To suppress detailed output on the optimization process, set
'false' in the call to
layer = 6; name = net.Layers(layer).Name
name = 'conv2-3x3_reduce'
channels = 1:36; I = deepDreamImage(net,name,channels, ... 'Verbose',false, ... 'PyramidLevels',1); figure I = imtile(I,'ThumbnailSize',[64 64]); imshow(I) name = net.Layers(layer).Name; title(['Layer ',name,' Features'],'Interpreter','none')
Filters for this layer detect more complex patterns than the first convolutional layer.
Visualize Deeper Convolutional Layers
The deeper layers learn high-level combinations of features learned by the earlier layers.
Increasing the number of pyramid levels and iterations per pyramid level can produce more detailed images at the expense of additional computation. You can increase the number of iterations using the
'NumIterations' option and increase the number of pyramid levels using the '
layer = 97; name = net.Layers(layer).Name
name = 'inception_4e-1x1'
channels = 1:6; I = deepDreamImage(net,name,channels, ... 'Verbose',false, ... "NumIterations",20, ... 'PyramidLevels',2); figure I = imtile(I,'ThumbnailSize',[250 250]); imshow(I) name = net.Layers(layer).Name; title(['Layer ',name,' Features'],'Interpreter','none')
Notice that the layers which are deeper into the network yield more detailed filters which have learned complex patterns and textures.
Visualize Fully Connected Layer
To produce images that resemble each class the most closely, select the fully connected layer, and set
channels to be the indices of the classes.
Select the fully connected layer (layer 142).
layer = 142; name = net.Layers(layer).Name
name = 'loss3-classifier'
Select the classes you want to visualize by setting
channels to be the indices of those class names.
channels = [114 293 341 484 563 950];
The classes are stored in the
Classes property of the output layer (the last layer). You can view the names of the selected classes by selecting the entries in
ans = 6×1 categorical snail tiger zebra castle fountain strawberry
Generate detailed images that strongly activate these classes. Set
'NumIterations' to 100 in the call to
deepDreamImage to produce more detailed images. The images generated from the fully connected layer correspond to the image classes.
I = deepDreamImage(net,name,channels, ... 'Verbose',false, ... 'NumIterations',100, ... 'PyramidLevels',2); figure I = imtile(I,'ThumbnailSize',[250 250]); imshow(I) name = net.Layers(layer).Name; title(['Layer ',name,' Features'])
The images generated strongly activate the selected classes. The image generated for the ‘zebra’ class contain distinct zebra stripes, whilst the image generated for the ‘castle’ class contains turrets and windows.