Experiment with Weight Initializers for Transfer Learning

This example shows how to configure an experiment that initializes the weights of convolution and fully connected layers using different weight initializers for training. To compare the performance of different weight initializers for your task, create an experiment using this example as a guide.

When training a deep learning network, the initialization of layer weights and biases can have a big impact on how well the network trains. The choice of initializer has a bigger impact on networks without batch normalization layers. For more information on weight initializers, see Compare Layer Weight Initializers.

Open Experiment

First, open the example. Experiment Manager loads a project with a preconfigured experiment that you can inspect and run. To open the experiment, in the Experiment Browser, double-click the name of the experiment (WeightInitializerExperiment).

An experiment definition consists of a description, a hyperparameter table, a setup function, and (optionally) a collection of metric functions to evaluate the results of the experiment. For more information, see Configure Deep Learning Experiment.

The Description box contains a textual description of the experiment. For this example, the description is:

Perform transfer learning by initializing the weights of convolution and
fully connected layers in a pretrained network.

The Hyperparameter Table contains the names and values of the hyperparameters used in the experiment. When you run the experiment, Experiment Manager sweeps through the hyperparameter values and trains the network multiple times. Each trial uses a different combination of the hyperparameter values specified in the table. This example uses the hyperparameters WeightsInitializer and BiasInitializer to specify the weight initializers for the convolution and fully connected layers in a pretrained network.

The Setup Function configures the training data, network architecture, and training options for the experiment. To inspect the setup function, under Setup Function, click Edit. The setup function opens in MATLAB Editor. In this example, the setup function:

  • Downloads and extracts the Flowers data set, which is about 218 MB. For more information on this data set, see Image Data Sets.

  • Loads a pretrained GoogLeNet network and initializes the input weight in the convolution and fully connected layers by using the initializers specified in the hyperparameter table. The auxiliary function findLayersToReplace determines which layers in the network architecture can be modified for transfer learning.

  • Defines a trainingOptions object for the experiment. The example trains the network for 10 epochs, using a mini-batch size of 128 and validating the network every 5 epochs.

The Metrics section specifies optional functions that evaluate the results of the experiment. This example does not include any metric functions.

Run Experiment

In the Experiment Manager tab, click Run. Experiment Manager trains the network defined by the setup function multiple times. Each trial uses a different combination of hyperparameters. A table of results displays the accuracy and loss for each trial.

While the experiment is running, click Training Plot to display the training plot and track the progress of each trial.

Click Confusion Matrix to display the confusion matrix for the validation data in each completed trial.

When the experiment finishes, you can sort the results table by column or filter trials by using the Filters pane. For more information, see Sort and Filter Experiment Results.

To test the performance of an individual trial, export the trained network or the training information for the trial. On the Experiment Manager tab, select Export > Trained Network or Export > Training Information, respectively. For more information, see Output Arguments.

Close Experiment

In the Experiment Browser, right-click the name of the project and select Close Project. Experiment Manager saves your results and closes all of the experiments contained in the project.

See Also

| |

Related Topics