Pith. sign in

REVIEW 4 major objections 4 minor 16 references

Tangma: A Tanh-Guided Activation Function with Learnable Parameters

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Tangma, a tanh-based activation with learnable shift and linear-skip parameters, achieves higher final validation accuracy and more stable convergence than ReLU, Swish, and GELU on MNIST and CIFAR-10.

desk verdict A small, clearly explained activation variant whose claimed gains rest on single runs; the math is fine, the evidence isn't yet. read the letter →

arxiv 2507.10560 v1 pith:VWBRKWQG submitted 2025-07-02 cs.NE cs.CVcs.LG

Shreel Golwala This is my paper
classification cs.NEcs.CVcs.LG
keywords activationfunctionlearnableparameterstanhhyperbolictangentgradientflowMNISTCIFAR-10deeplearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Tangma, an activation function defined as $\mathrm{Tangma}(x) = x\cdot\tanh(x+\alpha) + \gamma x$ with learnable parameters $\alpha$ and $\gamma$, and argues that it combines the smooth saturation of tanh with a guaranteed linear gradient path. The author claims that on small convolutional networks trained on MNIST and CIFAR-10, Tangma reaches slightly higher final validation accuracy than ReLU, Swish, and GELU — 99.09% versus at most 98.96% on MNIST and 78.15% versus at most 77.99% on CIFAR-10 — while showing smoother convergence and, on CIFAR-10, shorter average epoch runtimes than Swish and GELU. The reason a reader would care is that activation-function design is a cheap way to improve training dynamics, and a learnable activation that preserves weak gradients could generalize to deeper models without architectural changes.

What carries the argument

The central object is the function $\mathrm{Tangma}(x) = x\cdot\tanh(x+\alpha) + \gamma x$, where $\alpha$ is a learnable horizontal shift of the tanh inflection point and $\gamma$ is a learnable linear-skip coefficient. The argument turns on two of its properties: the derivative $\frac{d}{dx}\mathrm{Tangma}(x) = \tanh(x+\alpha) + x\,\mathrm{sech}^2(x+\alpha) + \gamma$ is nonzero everywhere as long as $\gamma \neq 0$, so gradients always flow; and for large $|x|$ the function becomes linear with slope $\gamma - 1$ on the negative side and $\gamma + 1$ on the positive side, keeping gradient magnitudes controlled. The paper treats $\alpha$ and $\gamma$ as trainable tensors initialized to zero, so the network can adapt the activation's saturation region and linearity during training.

What would settle it

Run the same four-activation comparison multiple times per activation with different random seeds and report the mean and standard deviation of final validation accuracy; if the error bands overlap substantially, the claimed ordering is not established. A second check freezes $\alpha$ and $\gamma$ to zero so the activation is $x\cdot\tanh(x)$; if accuracy is unchanged, the learnable parameters are not the active ingredient.

Watch

Extended reading notes

Core claim

The central claim is that the proposed activation function, $\mathrm{Tangma}(x) = x\cdot\tanh(x+\alpha) + \gamma x$, with $\alpha$ and $\gamma$ learned during training, yields better final validation accuracy and more stable convergence than three standard activations on two image-classification tasks. The author asserts that the tanh term's learnable horizontal shift lets each neuron adjust where it saturates, while the $\gamma x$ term supplies a nonzero derivative everywhere, preventing dead neurons and vanishing gradients; this combination is said to retain low-magnitude features that ReLU discards and to stabilize strong responses that tanh alone would compress. On MNIST, Tangma reports the highest final validation accuracy (99.09%) and lowest validation loss (0.0363) among the four; on CIFAR-10, it reports the highest final accuracy (78.15%) with the lowest average epoch time (8.97 s) among the four, though its final validation loss (0.7631) sits between ReLU's 0.7032 and GELU's 0.7740.

Load-bearing premise

The single training run per activation on a fixed validation split means the reported accuracy gaps (0.13 to 0.73 percentage points) are assumed to reflect systematic differences rather than run-to-run noise.

Editorial extensions

If this is right

  • Networks using Tangma would see a small but consistent accuracy gain — 0.13 to 0.18 percentage points on MNIST and 0.16 to 0.73 on CIFAR-10 — over ReLU, Swish, and GELU on these architectures.
  • Tangma's guaranteed nonzero gradient eliminates the dying-neuron failure mode of ReLU, so training remains stable even when inputs are persistently negative.
  • Because $\gamma x$ acts as a linear skip connection, the activation could be dropped into deeper networks where gradient propagation is a bottleneck.
  • The learnable $\alpha$ and $\gamma$ give the optimizer a way to tune each layer's nonlinearity to the input distribution, which may reduce sensitivity to initialization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the single-run results hold across seeds, the main practical consequence is that a cheap drop-in activation can buy the same accuracy as minor architecture tuning, which is useful in settings where retraining cost dominates.
  • The parameter trajectories, with $\alpha$ and $\gamma$ increasing steadily on CIFAR-10, suggest the optimizer uses the extra degrees of freedom to shift the activation toward a more linear regime as training progresses; checking whether this trajectory is consistent across seeds and architectures would clarify what the network is actually learning.
  • A natural next experiment is to evaluate Tangma in a transformer or residual network where Swish and GELU dominate; the linear skip term may interact favorably with normalization layers, but that is beyond this paper's evidence.
  • The reported faster epoch time on CIFAR-10 (8.97 s versus 11.2–11.3 s for Swish and GELU) may partly reflect implementation-level differences in computing the activation and its derivative; a fair runtime comparison would need identical fused kernels.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Tangma, an activation function defined as Tangma(x) = x·tanh(x+α) + γx, with learnable parameters α (a horizontal shift) and γ (a linear skip coefficient). The authors derive the derivative, analyze asymptotic behavior, and evaluate the function on MNIST and CIFAR-10 using small custom CNNs, comparing against ReLU, Swish, and GELU. They report final validation accuracy, training/validation loss, per-epoch runtime, and the evolution of α and γ over training. The central claim is that Tangma achieves higher final validation accuracy and more stable convergence than the baselines, with competitive or better runtime.

Significance. The activation function itself is cleanly defined and the derivative derivation in Section 3.1 is correct. The paper includes full per-epoch tables (Appendix Tables 3–6) and provides a code link, which aids reproducibility. The proposed mechanism—a learnable shift and a linear skip term—is plausible and could be of interest to the activation-function community. However, the empirical evidence supporting the superiority claim is currently weak: all comparisons rest on a single training run per activation, and the reported accuracy differences are within plausible run-to-run noise. The runtime claim in the abstract is contradicted by the paper's own MNIST results. If repeated-seed experiments confirm a systematic advantage, the contribution would be modest but real; as it stands, the evidence is not yet convincing.

major comments (4)
  1. [Section 5.1 and 5.2, Tables 1–2] The central empirical claim—that Tangma achieves the highest final validation accuracy on both datasets—rests on a single training run per activation with no reported seeds, variance, or statistical comparison. On CIFAR-10, the reported margin over GELU is 0.16 percentage points and over ReLU is 0.73 percentage points; with a 6,000-image validation set, the binomial standard error near 78% accuracy is approximately 0.53 percentage points, so the largest gap is only about 1.4 standard errors. With four activations compared and no multiple-comparison control, a single run cannot distinguish a systematic advantage from seed luck. The paper should provide results over multiple seeds (e.g., 5–10) with mean±std, and ideally a paired significance test or at least an explicit statement of the number of seeds and the observed variance.
  2. [Abstract and Section 5.0.1, Table 1] The abstract claims that Tangma showed 'improved training efficiency with lower average epoch runtimes compared to Swish and GELU.' On MNIST (Table 1), however, Tangma's average epoch time is 3.45 s, which is slightly higher than Swish's 3.44 s and markedly higher than GELU's 2.89 s. The runtime advantage is only observed on CIFAR-10 (Table 2). The claim must be restricted to the dataset where it holds, or corrected.
  3. [Section 3.1, derivative discussion] The statement 'This derivative is non-zero as long as γ ≠ 0, guaranteeing that the neuron remains active and gradients continue to flow' is not correct. Even with γ ≠ 0, the derivative tanh(x+α) + x·sech²(x+α) + γ can vanish for certain values of x and α (e.g., α = 0 and γ = 1.18 yields a zero derivative near x = -1). Moreover, at initialization α = γ = 0, the derivative at x = 0 is exactly zero. The role of γ as a gradient floor is plausible, but the absolute guarantee is not established; the claim should be weakened to say that the linear term adds a gradient contribution of γ, or that for |γ| sufficiently large the derivative is bounded away from zero.
  4. [Section 5.0.3 and Table 2] The superiority claim is based solely on final validation accuracy. On CIFAR-10, Tangma's final validation loss (0.7631) is worse than ReLU's (0.7032), and its final training loss (0.2270) is worse than GELU's (0.1840). The 'faster and more stable convergence' narrative is also drawn from single trajectories without a quantitative measure of stability. If the claim is that Tangma is better, the paper should either justify why accuracy is the appropriate single metric here, or present a multi-metric comparison with repeated runs that accounts for the inconsistency across metrics.
minor comments (4)
  1. [Throughout] The text contains typographical and spacing errors, such as 'CIF AR-10' in place of 'CIFAR-10', 'a equivalent' in Section 2, and 'keep training stability' in Section 3.2. A careful proofreading pass is needed.
  2. [Section 2.1] The derivative of ReLU is stated as '1 if x > 0, 0 otherwise', which ignores the subgradient convention at x = 0. This is a minor technical imprecision.
  3. [Section 3.1, Figure 1] The caption states the plots are for α = 0 and γ = 0; at these settings the derivative at x = 0 equals 0, which is visible in the figure but not commented on. The caption should note the zero-gradient point or use a slightly non-zero γ for illustration.
  4. [Section 7, References] Reference [3] (ELU) is cited in the bibliography but is not discussed in the text. Either add a sentence about ELU in the related work or remove the reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the Tangma definition, derivative analysis, and held-out evaluations are self-contained; the single-run statistical fragility is an evidence-quality concern, not a circular reduction.

full rationale

The paper's claimed derivation chain is self-contained at every link. Tangma(x) = x·tanh(x+α) + γx is introduced as a new parametric definition (Section 2.4); the derivative (Section 3.1), the small-x Taylor expansion (Section 3.2), and the large-|x| asymptotes (Section 3.2) are computed from that definition by elementary calculus and do not assume any empirical outcome. The learnable parameters α and γ are trained on the training split, and the reported 99.09% (MNIST) and 78.15% (CIFAR-10) validation accuracies are genuine held-out measurements from Tables 1-2 and Appendix Tables 3-4; nothing is fitted to the validation set and then renamed a prediction. The paper contains no self-citations at all: references [1]-[15] are external prior work, so there is no load-bearing self-citation chain and no imported uniqueness theorem. The post-hoc narratives in Sections 5.0.2 and 5.0.4 explain the results using Section 3 properties of the formula, but those properties are derived independently of the empirical outcomes, making the explanation speculative rather than circular. Two analytic claims are mathematically suspect — the derivative is asserted non-zero for any γ≠0, yet tanh(x+α)+x·sech²(x+α) dips below -1.1 near x≈-1.2, so the derivative can vanish for small positive γ; and the inflection point of x·tanh(x+α) solves x·tanh(x+α)=1, not x=-α — but these are correctness errors, not circular reductions. The main limitation is statistical: one run per activation with margins of 0.13–0.73 percentage points (roughly 1.4 standard errors on the 6,000-image CIFAR-10 validation set) cannot establish systematic superiority, and the paper honestly notes that Tangma's final validation loss on CIFAR-10 is worse than ReLU's. This is an evidence-quality issue that does not make the derivation circular.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper depends mainly on standard calculus and on a single-run evaluation design. Alpha and gamma are learned from data, not imposed to force the result, so their presence is not circular; the main statistical burden is the absence of repeated seeds.

free parameters (2)
  • alpha (learned shift) = MNIST about 0.283 at epoch 10; CIFAR-10 about 0.399 at epoch 10
    Learned per activation layer and dataset; the central comparison depends on Adam tuning these values.
  • gamma (learned linear-skip coefficient) = MNIST about 0.135 at epoch 10; CIFAR-10 about 0.377 at epoch 10
    Learned per activation layer; nonzero gamma is stated as the mechanism preventing vanishing gradients.
assumptions (4)
  • standard math Backpropagation and the chain rule govern gradient flow.
    Section 2 uses the chain rule to argue that vanishing gradients matter; this is standard background.
  • standard math tanh saturation asymptotics describe practical neuron behavior.
    Section 3.2 uses limits as x goes to plus or minus infinity to claim nonzero gradients and controlled growth.
  • domain assumption One run each on fixed 80/20 and 90/10 splits is a sufficient statistical comparison.
    Section 5 reports single-run final accuracies with no seeds; this assumption is load-bearing for the superiority claim.
  • domain assumption MNIST and CIFAR-10 with the chosen small CNNs, without augmentation or per-activation tuning, are adequate for benchmarking activation functions.
    Section 4 fixes one architecture and hyperparameters; no tuning or augmentation is described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tangma: A Tanh-Guided Activation Function with Learnable Parameters." pith.science (2026). https://pith.science/paper/VWBRKWQG

@misc{pith2026250710560,
  author       = {Pith},
  title        = {Pith review of: Tangma: A Tanh-Guided Activation Function with Learnable Parameters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VWBRKWQG}},
  note         = {Machine review of arXiv:2507.10560}
}
abstract

Activation functions are key to effective backpropagation and expressiveness in deep neural networks. This work introduces Tangma, a new activation function that combines the smooth shape of the hyperbolic tangent with two learnable parameters: $\alpha$, which shifts the curve's inflection point to adjust neuron activation, and $\gamma$, which adds linearity to preserve weak gradients and improve training stability. Tangma was evaluated on MNIST and CIFAR-10 using custom networks composed of convolutional and linear layers, and compared against ReLU, Swish, and GELU. On MNIST, Tangma achieved the highest validation accuracy of 99.09% and the lowest validation loss, demonstrating faster and more stable convergence than the baselines. On CIFAR-10, Tangma reached a top validation accuracy of 78.15%, outperforming all other activation functions while maintaining a competitive training loss. Tangma also showed improved training efficiency, with lower average epoch runtimes compared to Swish and GELU. These results suggest that Tangma performs well on standard vision tasks and enables reliable, efficient training. Its learnable design gives more control over activation behavior, which may benefit larger models in tasks such as image recognition or language modeling.

Figures

Figures reproduced from arXiv: 2507.10560 by the authors.

Figure 1
Figure 1. Plot of the Tangma activation function and its derivative for [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Effect of γ on Tangma activation function with α = 0. 3.3 Role of α The parameter α horizontally shifts the nonlinear response of the function. For standard tanh(x), the inflection point is at x = 0; for Tangma, the inflection occurs at x = −α. This shift determines where the neuron transitions between linear behavior and saturation. • When α > 0, the curve shifts left, activating or saturating earlier. • When α < 0… view at source ↗
Figure 3
Figure 3. Effect of α on Tangma activation function with γ = 0. 4 Methodology A convolutional neural network was implemented for both MNIST and CIFAR-10, differing only in model depth and input dimensionality. MNIST, being simpler, was trained with a smaller batch size, while the more complex CIFAR-10 used larger batches for convergence. Each model was designed to support four pluggable activation functions — Swish, GELU, ReL… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: MNIST forward-pass structure. 4.3 CIFAR-10 architecture For CIFAR-10, the dataset was loaded using torchvision.datasets.CIFAR10 with standard PyTorch DataLoader pipelines. X ∈ R 128×3×32×32 , 8 [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: CIFAR-10 forward-pass structure. 10 [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Performance of the four activation functions on MNIST. The plots show training loss (a), [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Performance of the four activation functions on CIFAR-10. The plots show training loss (a), [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Behavior of parameters α and γ on MNIST across 10 epochs. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Behavior of parameters α and γ on CIFAR-10 across 10 epochs. Throughout the training, the evolution of the learnable parameters reflect how model shifts its focus from fine edge preservation to saturation control, depending on the dataset’s visual complexity. On MNIST,…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 9 canonical work pages

  1. [6]

    Learn-able parameter guided Activation Functions

    S. Balaji, T. Kavya, and Natasha Sebastian. Learn-able Parameter Guided Activation Functions. arXiv preprint arXiv:1912.10752, 2019

    work page Pith review arXiv 1912

  2. [7]

    Goyal, R

    M. Goyal, R. Goyal, B. Lall. Learning Activation Functions: A new paradigm of understanding Neural Networks. arXiv preprint arXiv:1906.09529, 2020

    arXiv 1906

  3. [1]

    Deep Learning using Rectified Linear Units (ReLU)

    Abien Fred Agarap. Deep Learning using Rectified Linear Units (ReLU). arXiv preprint arXiv:1803.08375, 2018

    arXiv 2018

  4. [2]

    Vinod Nair and Geoffrey E. Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. In Proceedings of the 27th International Conference on Machine Learning (ICML), 2010. 15

    work page 2010

  5. [3]

    Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

    Djork-Arn´ e Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). arXiv preprint arXiv:1511.07289, 2016

    arXiv 2016

  6. [4]

    Gaussian Error Linear Units (GELUs)

    Dan Hendrycks and Kevin Gimpel. Gaussian Error Linear Units (GELUs). arXiv preprint arXiv:1606.08415, 2016

    arXiv 2016

  7. [5]

    Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for Activation Functions. arXiv preprint arXiv:1710.05941, 2017

    arXiv 2017

  8. [8]

    Mish: A Self Regularized Non-Monotonic Neural Activation Function

    Diganta Misra. Mish: A Self Regularized Non-Monotonic Neural Activation Function. arXiv preprint arXiv:1908.08681, 2019

    arXiv 1908

Show all 16 references
  1. [9]

    Gradient-Based Learning Applied to Document Recognition

    Yann LeCun, L´ eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998

    1998

  2. [10]

    Learning Multiple Layers of Features from Tiny Images

    Alex Krizhevsky and Geoffrey Hinton. Learning Multiple Layers of Features from Tiny Images. Technical Report, University of Toronto, 2009

    2009

  3. [11]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet Classification with Deep Con- volutional Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS), 2012

    2012

  4. [12]

    Very Deep Convolutional Networks for Large-Scale Image Recognition

    Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  5. [13]

    Deep Residual Learning for Image Recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

    2016

  6. [14]

    Weinberger

    Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely Connected Convolutional Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017

    2017

  7. [15]

    Understanding the log-sum-exp trick

    Gregory Gundersen. Understanding the log-sum-exp trick. Gregory Gundersen ’s Blog, February

  8. [2020]

    TL: Train Loss, VL: Val Loss, V A: Val Accuracy, T: Time

    https://gregorygundersen.com/blog/2020/02/09/log-sum-exp/ 8 Appendix Table 3: Training and validation metrics across epochs for different activation functions on MNIST. TL: Train Loss, VL: Val Loss, V A: Val Accuracy, T: Time. Epoch Tangma ReLU Swish GELUTL VL V A (%) T (s) TL...

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.