Tangma, defined as x times tanh(x plus a learned shift) plus a learned linear term, reaches 99.09% validation accuracy on MNIST and 78.15% on CIFAR-10 after 10 epochs, slightly beating ReLU, Swish, and GELU in these single runs.
Learn-able parameter guided Activation Functions
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
In this paper, we explore the concept of adding learn-able slope and mean shift parameters to an activation function to improve the total response region. The characteristics of an activation function depend highly on the value of parameters. Making the parameters learn-able, makes the activation function more dynamic and capable to adapt as per the requirements of its neighboring layers. The introduced slope parameter is independent of other parameters in the activation function. The concept was applied to ReLU to develop Dual Line and DualParametric ReLU activation function. Evaluation on MNIST and CIFAR10 show that the proposed activation function Dual Line achieves top-5 position for mean accuracy among 43 activation functions tested with LENET4, LENET5, and WideResNet architectures. This is the first time more than 40 activation functions were analyzed on MNIST andCIFAR10 dataset at the same time. The study on the distribution of positive slope parameter beta indicates that the activation function adapts as per the requirements of the neighboring layers. The study shows that model performance increases with the proposed activation functions
citation-role summary
background 1
citation-polarity summary
fields
cs.NE 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
background 1representative citing papers
citing papers explorer
-
Tangma: A Tanh-Guided Activation Function with Learnable Parameters
Tangma, defined as x times tanh(x plus a learned shift) plus a learned linear term, reaches 99.09% validation accuracy on MNIST and 78.15% on CIFAR-10 after 10 epochs, slightly beating ReLU, Swish, and GELU in these single runs.