Custom Neural Network with learnable weights in activation function

Hi,

I want to create a Custom Neural Network with learnable parameters in the activation function. The learnable parameters are the weights of layers, but the weights are also used inside the activation function in a certain nonlinear relation. Normally, the activation function takes the inputs multiplied with the weights ( e,g, y1=f1(X*W1 + b1)) but here I want to use weights directly inside in a certain nonlinear relation i.e. y1=g1(X*W1 + b1,W1).

Custom layers can be created as metioned here:

https://www.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-layer.html

But the way I want to create a custom layer with a specific activation function is what i need to know. Any help will be highly appreciated.

AR

Answer 1

[Kausthub]

Hi Ahmed Rehan,

I understand that you would like to have a custom activation function that has weights as an additional input i.e., instead of y1= f1(X*W1 + b1) you would like to have y1=g1(X*W1 + b1, W1).

You could leverage the “Define Custom Deep Learning Layer with Formatted Inputs” article which explains how to provide inputs to a custom layer.

• https://www.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-layer-with-formatted-inputs.html

I have attached a simple working snippet regarding the same for your reference. Major changes from the article you mentioned (https://www.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-layer.html) include:

• Adding weights as a parameter in the layer definition

function layer = customActivationLayer(weights, args) 

• Passing weights when using the layer

customActivationLayer(Weights); 

Please have a look at the attached code sample for better understanding.

Hope this helps and clarifies your queries regarding how to provide weights to activation functions using custom layers!