Can I read Adaptively Customizing Activation Functions for Various Layers on EtoBox?

Adaptively Customizing Activation Functions for Various Layers by Hu, Haigen; Liu, Aizhu; Guan, Qiu; Li, Xiaoxin; Chen, Shengyong; Zhou, Qianwei is a scholarly article available to read on EtoBox.

What is Adaptively Customizing Activation Functions for Various Layers about?

To enhance the nonlinearity of neural networks and increase their mapping abilities between the inputs and response variables, activation functions play a crucial role to model more complex relationships and patterns in the data. In this work, a novel methodology is proposed to adaptively customize activation functions only by adding very few parameters to the traditional activation functions such as Sigmoid, Tanh, and ReLU. To verify the effectiveness of the proposed methodology, some theoretical and experimental analysis on accelerating the convergence and improving the performance is presented, and a series of experiments are conducted based on various network models (such as AlexNet, VGGNet, GoogLeNet, ResNet and DenseNet), and various datasets (such as CIFAR10, CIFAR100, miniImageNet, PASCAL VOC and COCO) . To further verify the validity and suitability in various optimization strategies and usage scenarios, some comparison experiments are also implemented among different optimization strategies (such as SGD, Momentum, AdaGrad, AdaDelta and ADAM) and different recognition tasks like classification and detection. The results show that the proposed methodology is very simple but

Author

Hu, Haigen; Liu, Aizhu; Guan, Qiu; Li, Xiaoxin; Chen, Shengyong; Zhou, Qianwei

Published

2021

Language

EN