How to customize Neural Networks' activation function

Hi, I would like to implement, using Matlab, a neural network with 3 hidden layers, each using ReLU activation function. How can i do this?

Currently, I know i can set the activation function using:

net.layers{i}.transferFcn = reluLayer();

But this only allows to set a specific type of function that is predefined (like logsig), but ReLU is not one of those functions.

Is there a way to change the layer to the ReLU layer? Thanks

Answer 1

[Darío Pérez]

As far as I am concern, you can use the predefined function 'poslin' (which is a ReLU):

net.layers{i}.transferFcn = 'poslin';

but "other differentiable transfer functions can be created and used if desired": Multilayer Neural Network Architecture.

Not sure how discontinuity at x=0 would affect training stage. In addition, recent articles state that ReLU should be used for regression problems but it achieves worst results than 'tansig' or 'logsig' in one of my examples. Has anyone any thoughts/conclusions in this regard?

Regards!

Answer 2

[daniel]

its useful for deeper nets so depends on your # layers, it's mostly for minimizing vanishing/exploding gradients

[Jan]

[MOVED from flags] Flagged by Amine Bendali on 18 Jan 2019 at 22:31.

i want to check it later

[Please use flags only to inform admins and editors about messages, which should be reviewed. Thanks]

Answer 3

[peter chevo]

1. Copy folder and file of C:\Program Files\MATLAB\MATLAB Production Server\R2015a\toolbox\nnet\nnet\nntransfer\ such as +tansig and tansig.m to current path 2. edit file name such as tansig.m is my_transfer.m 3. edit folders name such as +tansig is +my_transfer 4. edit last line in apply.m to your formula equation

[Abdelwahab Afifi]

This method doesn't work. Because each Activation function has its own files with its own sturcture/values/equation.

Answer 4

[Maria Duarte Rosa]

For Deep Learning networks one can create a custom activation layer using:

https://www.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-layers.html

Answer 5

[David Willingham]

Hi,

@Maria Duarte Rosa gave a good answer on how to create a custom activation layer by visiting this page: Define Custom Deep Learning Layers

This extended answer is aimed at addressing how to define a "Relu" function in MATLAB.

With MATLAB's current Deep Learning framework, ReLu function is a standard layer you can define.

Here is an example:

Create a ReLU layer with the name 'relu1'.

layer = reluLayer('Name','relu1')

layer =

ReLULayer with properties:

Name: 'relu1'

Include a ReLU layer in a Layer array.

layers = [ ...
    imageInputLayer([28 28 1])
    convolution2dLayer(5,20)
    reluLayer
    maxPooling2dLayer(2,'Stride',2)
    fullyConnectedLayer(10)
    softmaxLayer
    classificationLayer]

layers =

7x1 Layer array with layers:

1 '' Image Input 28x28x1 images with 'zerocenter' normalization

2 '' Convolution 20 5x5 convolutions with stride [1 1] and padding [0 0 0 0]

3 '' ReLU ReLU

4 '' Max Pooling 2x2 max pooling with stride [2 2] and padding [0 0 0 0]

5 '' Fully Connected 10 fully connected layer

6 '' Softmax softmax

7 '' Classification Output crossentropyex

For more information on ReLu layer see this page: reluLayer

[Alexander Krauss]

Hi David,

how can this ReLU "layer" be understood?

I thought ReLU ist just an activation function for a neuron.

Does the ReLU layer refer to some sort of layer of activation functions?

Because to me this would mean you would have to specify a layer architecture like fullyConnected in advance in order to specify number of neurons and biases.

And generally how would one specify the number of neurons in one of MATLAB's predefined layers?

I would be very pleased to receive a reply.

Best regards

Alex

[Antoni Woss]

The reluLayer will apply the rectified linear unit operation to the outputs of the preceding layer. As you mentioned the reluLayer is exactly a layer of activation functions. For example, if the reluLayer follows a 2D convolutional layer, where the output of the convolution layer is say 10x10x5 (5 filters each of 10 pixels by 10 pixels), then the reluLayer will apply the rectified linear operation to each of the 10x10x5 values.

As the reluLayer operates elementwise, you do not need to specify architecture, such as specifying the outputSize in the fullyConnectedLayer. The reluLayer infers the correct architecture from the previous layer.

To specify the number of output activations (number of neurons as you are referring to) for MATLABs built in layers, you can take a look at the documentation for the particular layer. For example:

• For the lstmLayer specify the number of hidden units, numHiddenUnits, to specify the number of output activations.
• For the fullyConnectedLayer you can specify the outputSize.

Here are some documentation links to the layers discussed in this post for further reference:

• https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.relulayer.html
• https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.lstmlayer.html
• https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.fullyconnectedlayer.html

[Alexander Krauss]

Thanks a lot Antoni, great and insightful answer. :)

I would just like to know how dropout layers are to be understood.

I mean I know what dropout and dropout rates respectively are and how they work.

But again my question refers to the implementation of dropout layers in MATLAB.

If I want to achieve a dropout rate of say 15 % (for each hidden layer), do I have to add a 15 % dropout layer after every activation layer or will it be enough to just add one dropout layer which then applies to the entire network of hidden layers.

Would be great if you could answer this one as well.

Best regards

Alex

Answer 6

[km y]

可以看一下我写的博客，这里有如何调用和修改激活函数内容：如何在matlab工具箱中自定义激活函数及其使用_web，gui编写爱好者的博客-CSDN博客