Pith. sign in

REVIEW 5 major objections 6 minor 47 references

Hybrid activation functions for deep neural networks: S3 and S4 -- a novel approach to gradient flow optimization

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read S4, a smooth sigmoid-softsign hybrid activation, is claimed to outperform nine baselines across classification and regression while keeping gradients stable.

desk verdict A new named blend and code, but a wrong derivative and selection-biased results make the central claims unsupported. read the letter →

arxiv 2507.22090 v1 pith:5CIXSZOB submitted 2025-07-29 cs.LG cs.AIcs.NEcs.NI

Sergii Kavun This is my paper
classification cs.LGcs.AIcs.NEcs.NI MSC 68T0768T0565D1068Q32
keywords activationfunctionshybridS3functionS4gradientflowdeadneuronsdeepneuralnetworkssteepnessparameter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that a single family of hybrid activation functions, S3 and S4, can keep gradients flowing in deep networks without either killing neurons or saturating. S3 grafts sigmoid for negative inputs to softsign for positive inputs, and S4 smooths the seam with a steepness-controlled weighting function. On the paper's experiments, S4 beats all nine compared activations, reaching 97.4% accuracy on MNIST, 96.0% on Iris, and 18.7 mean squared error on Boston Housing, while reaching lower training epochs than ReLU and keeping gradients in a claimed [0.24, 0.59] band. If the claim holds, practitioners could replace task-specific activation search with one tunable function whose parameter k is set per task.

What carries the argument

The load-bearing object is the S4 mixing formula $S4(x)=\alpha_k(x)\cdot\mathrm{softsign}(x)+(1-\alpha_k(x))\cdot\sigma(x)$, where $\alpha_k(x)=1/(1+e^{-kx})$ is a sigmoidal weight that shifts output from sigmoid-like behavior for negative inputs to softsign-like behavior for positive inputs. The paper's stated derivative of S4 is continuous everywhere, which removes the jump from 0.25 to 1.0 that S3 has at the transition point; this continuity, together with the tunable steepness k, is what the gradient-flow and convergence analysis rests on.

What would settle it

Recompute $S4'(0)$ from the displayed formula: it gives $5/8 - k/8$, which is 0 at the recommended $k=5$, whereas the text states $S4'(0)=k/4$; if the recomputed gradient range no longer stays in [0.24, 0.59] or dead-neuron counts match ReLU's at depth 3, the central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that S4, defined as $S4(x)=\alpha_k(x)\cdot\mathrm{softsign}(x)+(1-\alpha_k(x))\cdot\sigma(x)$ with $\alpha_k(x)=1/(1+e^{-kx})$, is a smooth, parameterized hybrid of the sigmoid and softsign families, and that this smoothness is what gives it an edge. The paper argues that the derivative discontinuity at $x=0$ in S3 is the reason naive hybrids fail, and that removing it while keeping a tunable transition sharpness yields stable gradient flow, faster convergence, and better final accuracy. In the reported comparisons, S4 ranks first on every task, with the tunable parameter k taking task-specific optimal values: k=5 for regression, k=10 for Iris, k=15 for binary classification. The authors present this as evidence that principled smooth hybridization, rather than a specific formula, is the promising direction for activation design.

Load-bearing premise

The central claim depends on the paper's derivative formula for S4 being correct: if the value of the derivative at the transition point is wrong, the claimed gradient range, dead-neuron avoidance, and convergence advantages are not established.

Editorial extensions

If this is right

  • S4 ranks first on every task in the reported comparisons, so it is claimed as a drop-in replacement for fixed activations across binary classification, multi-class classification, regression, and image classification.
  • S4 converges in 8 to 14 epochs on the tested architectures while ReLU needs 12 to 19, which translates into shorter training time for equivalent network depth.
  • Gradients through S4 stay in the [0.24, 0.59] band across network depths, contrasting with an 18% dead-neuron rate for ReLU at depth 3 and severe vanishing gradients for sigmoid beyond depth 2.
  • The k parameter has task-specific optimal ranges (k=5-10 for regression, k=5-15 for multi-class, k=10-20 for binary classification), giving practitioners a tuning rule instead of a search over different activation functions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the smoothing construction is generic, so the same $\alpha_k$ weighting could be applied to other activation pairs, such as ReLU with tanh or ELU with softsign, creating a family of task-adaptive hybrids beyond S4.
  • Editorial inference: because S4's performance degrades for k above 30 and approaches S3's hard switch, learning k during training would be a natural extension and would remove the separate grid-search pass over k.
  • Editorial inference: if the derivative at $x=0$ is recomputed from the displayed formula rather than the paper's quoted value, the claimed [0.24, 0.59] gradient range may change; the empirical rankings could still survive, but the mechanism would need to be restated.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript introduces two hybrid activation functions, S3 and S4. S3 is defined piecewise as sigmoid for x≤0 and softsign for x>0, and S4 smooths the transition with a sigmoidal weight α_k(x). The authors report experiments on a synthetic binary-classification task, Iris, Boston Housing, and MNIST, claiming that S4 outperforms nine baseline activation functions (97.4% MNIST accuracy, 96.0% Iris accuracy, 18.7 regression MSE), converges faster than ReLU, and keeps gradients in [0.24, 0.59] across depths. They also report a task-dependent optimal range for the steepness parameter k and a 1.68x computational speedup after optimization.

Significance. If the claims were correct, a smooth, tunable hybrid activation with stable gradients and no dead neurons would be a practically useful contribution, especially given the reproducible experimental setup: code is promised via Zenodo and fixed random seeds are stated. However, the theoretical foundation is not delivered, the derivative evaluation at the origin is wrong, and the model-selection and statistical reporting are insufficient. As a result, the core claims about gradient health and consistent superiority are not established, and the manuscript's central narrative is internally contradicted by its own tables.

major comments (5)
  1. [Section 3.2 (derivative formula)] The claim S4'(0) = k/4 is incorrect. Applying the product rule to S4(x) = α_k(x) softsign(x) + (1 − α_k(x)) σ(x) and evaluating at x = 0 gives S4'(0) = α_k'(0)(softsign(0) − σ(0)) + α_k(0) softsign'(0) + (1 − α_k(0)) σ'(0) = (k/4)(0 − 1/2) + 1/2·1 + 1/2·1/4 = 5/8 − k/8. At the recommended k = 5 the slope is exactly 0, and just left of the origin it is negative. This directly contradicts the gradient-flow analysis in Section 4, where S4 is reported to maintain gradients in [0.24, 0.59] and to avoid dead neurons; the claimed advantage over ReLU therefore rests on an erroneous derivative value.
  2. [Section 3.1] The text states that the authors 'provide comprehensive theoretical justification ... including formal proofs of convergence properties, stability guarantees, and expressivity bounds,' but no such proofs appear in the manuscript. Section 3.2 gives only a derivative expression and a list of asserted properties. Because the paper's contribution is framed as a theoretically grounded gradient-flow optimization method, the absence of these proofs leaves the central theoretical claim unsupported.
  3. [Section 4, Table 4] The convergence analysis is internally inconsistent. The prose claims that 'S4 with k = 5 achieves the fastest convergence in every configuration,' but the table shows S4(k = 15) with 7 vs 8 epochs on 10-1, 10 vs 11 on 50-2, and 13 vs 14 on 100-3. Thus the data contradict the written claim that k = 5 is fastest, and the subsequent parameter-sensitivity conclusions are not supported by the table they cite.
  4. [Section 4, Parameter Optimization and Table 3] The reported S4 results are obtained after a grid search over k on the same datasets used to report the best performance, with no nested validation or correction for selection. The 'task-specific optimal ranges' (k = 10–20 for binary, k = 5–15 for multi-class, k = 5–10 for regression) are therefore fitted values rather than validated guidelines, and the headline comparisons against fixed baselines are not on equal footing.
  5. [Section 4, experimental design] The manuscript states that all results are means over three runs and that paired t-tests (p < 0.05) and 95% confidence intervals were used, but no test statistics, p-values, variances, or intervals are reported. Given differences of 0.3–0.7 percentage points in several comparisons (e.g., 96.8 vs 96.5 on binary classification), three seeds are insufficient to establish superiority without reporting the actual statistical evidence.
minor comments (6)
  1. [Section 3.1 (S3 derivative limits)] The right-limit sentence says lim(x → 0⁺) S3(x) = 0.5, but the expression immediately above gives the right derivative as 1/(1+|x|)^2 at 0⁺, i.e., 1.0, and the following sentence says the jump is from 0.25 to 1.0; the 0.5 value is inconsistent with both.
  2. [Section 3.1 (S3 differentiability)] The bullet 'At point x = 0: The derivative equals 1' conflicts with the earlier statement that the derivative is discontinuous at x = 0; the manuscript should specify that this is a one-sided (right) derivative.
  3. [Section 4, Computational Performance] A reduction from 2.28 s to 1.36 s is a 1.68x speedup (~40% time reduction), not a '16%' improvement as stated; the '16%' figure appears to be a miscalculation.
  4. [Abstract and Table 4] The abstract's 'faster convergence (-19 for ReLU)' is unclear; Table 4 shows epoch differences of 2–5, not −19, and the notation should be explained or corrected.
  5. [Section 3.2 (S4 range)] The stated range 'approximately (0; 0.909)' is asserted without derivation; since the function is bounded and monotone in some regions, an explicit bound or computation should be provided.
  6. [Tables 1 and 3] There are two different 'Table 1' entries in the manuscript (the comparison table in Section 3.1 and the related-work table in the literature review), and several references to 'Table 1' in the text are ambiguous; renumber the tables and cross-check all in-text callouts.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; central empirical claims are self-contained, though the derivative formula contains a mathematical error that is a correctness risk, not circularity.

full rationale

The paper's central claims—S4's benchmark results, convergence improvements, and gradient-stability range—are presented as empirical measurements obtained by training networks and comparing against nine baselines, not as consequences derived from the activation function's definition in a way that reduces to its inputs. The S4 function is explicitly defined as a weighted mixture of softsign and sigmoid, and the reported performance numbers (e.g., 97.4% MNIST, 18.7 MSE Boston) are experimental outcomes. The k parameter is selected by grid search on the same datasets, and the reported 'optimal ranges' (k=10-20 for binary, k=5-15 for multi-class, k=5-10 for regression) are summaries of those search results rather than independent predictions; this raises a standard model-selection bias concern but does not constitute circularity under the specific patterns enumerated. Self-citations ([24], [25], [27], [38], [41]) are either unrelated background or point to the code/dataset repository for reproducibility; they are not load-bearing for the central argument. No uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. The paper does contain a serious internal inconsistency in the derivative calculation: Section 3.2's equation gives S4'(0)=5/8−k/8, but the bullet claims S4'(0)=k/4; at the recommended k=5 the correct value is 0, contradicting the claimed gradient range [0.24, 0.59]. Also Table 4 shows S4(k=15) converging faster than S4(k=5), contradicting the text. These are correctness errors that undermine the gradient-stability narrative, but they are not instances of circular reasoning. Therefore the circularity score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claims rest on one free parameter, k, which is fitted per task by grid search, and on unproven assumptions that smoothness drives the observed gains and that small dense networks are representative. No new physical or mathematical entities are introduced.

free parameters (1)
  • Steepness parameter k for S4 = 5, 10, 15 task-specific (regression k=5, multi-class k=10, binary k=15)
    Grid-searched on the same tasks over the set {5, 10, 15, 20, 30, 40, 50}. The reported optimal ranges and all S4 performance numbers depend on this fitted choice.
assumptions (3)
  • domain assumption Smoothness of the activation function is sufficient to produce the improved gradient flow and convergence claimed for S4.
    Section 3.2 and the Discussion assert that eliminating the derivative discontinuity yields stable gradients and faster convergence, but no formal proof or controlled experiment isolates smoothness from other differences.
  • domain assumption A sigmoid-weighted convex combination of sigmoid and softsign retains the desirable properties of both components.
    This is the design premise in Section 3.2. It is plausible but not proven; the optimal balance is instead determined empirically through k.
  • domain assumption Small fully connected networks on MNIST, Iris, Boston Housing, and a synthetic set are representative benchmarks for activation function superiority.
    Section 4 uses one small MLP architecture for all main comparisons. The abstract and discussion generalize far beyond this, so the representativeness assumption is load-bearing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hybrid activation functions for deep neural networks: S3 and S4 -- a novel approach to gradient flow optimization." pith.science (2026). https://pith.science/paper/5CIXSZOB

@misc{pith2026250722090,
  author       = {Pith},
  title        = {Pith review of: Hybrid activation functions for deep neural networks: S3 and S4 -- a novel approach to gradient flow optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5CIXSZOB}},
  note         = {Machine review of arXiv:2507.22090}
}
read the original abstract

Activation functions are critical components in deep neural networks, directly influencing gradient flow, training stability, and model performance. Traditional functions like ReLU suffer from dead neuron problems, while sigmoid and tanh exhibit vanishing gradient issues. We introduce two novel hybrid activation functions: S3 (Sigmoid-Softsign) and its improved version S4 (smoothed S3). S3 combines sigmoid for negative inputs with softsign for positive inputs, while S4 employs a smooth transition mechanism controlled by a steepness parameter k. We conducted comprehensive experiments across binary classification, multi-class classification, and regression tasks using three different neural network architectures. S4 demonstrated superior performance compared to nine baseline activation functions, achieving 97.4% accuracy on MNIST, 96.0% on Iris classification, and 18.7 MSE on Boston Housing regression. The function exhibited faster convergence (-19 for ReLU) and maintained stable gradient flow across network depths. Comparative analysis revealed S4's gradient range of [0.24, 0.59] compared to ReLU's 18% dead neurons in deep networks. The S4 activation function addresses key limitations of existing functions through its hybrid design and smooth transition mechanism. The tunable parameter k allows adaptation to different tasks and network depths, making S4 a versatile choice for deep learning applications. These findings suggest that hybrid activation functions represent a promising direction for improving neural network training dynamics.

Figures

Figures reproduced from arXiv: 2507.22090 by the authors.

Figure 1
Figure 1. Activation function S3 and its components [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 32 canonical work pages

  1. [1]

    & Baldi, P

    Agostinelli, F., Hoffman, M., Sadowski, P. & Baldi, P. Learning activation functions to improve deep neural networks. arXiv preprint arXiv:1412.6830 (2014)

    arXiv 2014

  2. [2]

    & Prevete, R

    Apicella, A., Donnarumma, F., Isgrò, F. & Prevete, R. A survey on modern trainable activation functions. Neural Networks 138, 14–32 (2021)

    work page 2021

  3. [3]

    Ashish Rajanand and Pradeep Singh. 2024. ErfReLU: adaptive activation function for deep neural network. Pattern Anal. Appl. 27, 2 (Jun 2024). https://doi.org/10.1007/s10044 -024-01277-w

    doi:10.1007/s10044 2024

  4. [4]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIP S'17). Curran Associates Inc., Red Hook, NY, USA, 6000–6010

    work page 2017

  5. [5]

    Activation functions and their characteristics in deep neural networks,

    B. Ding, H. Qian and J. Zhou, "Activation functions and their characteristics in deep neural networks," 2018 Chinese Control and Decision Conference (CCDC) , Shenyang, China, 2018, pp. 1836 - 1841, doi: 10.1109/CCDC.2018.8407425

    arXiv 2018

  6. [6]

    & Bengio, Y

    Bergstra, J. & Bengio, Y. Random search for hyper -parameter optimization. Journal of Machine Learning Research 13, 281–305 (2012)

    work page 2012

  7. [7]

    & Pandey, A

    Biswas, K., Kumar, S., Banerjee, S. & Pandey, A. K. Smooth maximum unit: smooth activation function for deep networks using smoothing maximum technique. in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 794–803 (2022)

    work page 2022

  8. [8]

    https://www.kaggle.com/c/boston-housing

    Boston Housing dataset : UCI Machine Learning Repository [officially removed from core ML libraries (including scikit -learn since version 1.2.0) due to ethical concerns related to racial bias in data ] (archiv): https://archive.ics.uci.edu/ml/machine-learning-databases/housing/. https://www.kaggle.com/c/boston-housing

    work page

Show all 47 references
  1. [9]

    -A., Unterthiner, T

    Clevert, D. -A., Unterthiner, T. and Hochreiter, S. (2015) Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). 4 th International Conference on Learning Representations, ICLR 2016, San Juan, 2 -4 May 2016, 1 -14. http://arxiv.org/abs/1511.07289

    2015 arXiv

  2. [10]

    Approximation by superpositions of a sigmoidal function

    Cybenko, G. Approximation by superpositions of a sigmoidal function. Math. Control Signal Systems 2, 303–314 (1989). https://doi.org/10.1007/BF02551274

    1989 doi

  3. [11]

    The impact of activation functions on training and performance of a deep neural network,

    D. C. Marcu and C. Grava, "The impact of activation functions on training and performance of a deep neural network," 2021 16th International Conference on Engineering of Modern Electric Systems (EMES), Oradea, Romania, 2021, pp. 1-4, doi: 10.1109/EMES52337.2021.9484108

    2021

  4. [12]

    & Zhou, J

    Ding, B., Qian, H. & Zhou, J. Activation functions and their characteristics in deep neural networks. In 2018 Chinese Control and Decision Conference 1836 –1841 (IEEE, 2018)

    2018

  5. [13]

    R., Singh, S

    Dubey, S. R., Singh, S. K. & Chaudhuri, B. B. Activation functions in deep learning: A comprehensive survey and benchmark. Neurocomputing 503, 92 –108 (2022) , doi: 10.1016/j.neucom.2022.06.111

    2022 doi

  6. [14]

    & Doya, K

    Elfwing, S., Uchibe, E. & Doya, K. Sigmoid -weighted linear units for neural network function approximation in reinforcement learning. Neural Networks 107, 3 –11 (2018) , doi: 10.1016/j.neunet.2017.12.012

    2018 doi

  7. [15]

    O., Ante, J

    Essang, S. O., Ante, J. E., Otobi, A. O., Okeke, S. I., Akpan, U. D., Francis, R. E., Auta, J. T., Essien, D. E., Fadugba, S. E., Kolawole, O. M., Asanga, E. E., Ita, B. I. (2025), Optimizing Neural Networks with Convex Hybrid Activations for Improved Grad ient Flow. Advanced ...

    2025 doi

  8. [16]

    Farhadi, F., Nia, V. P. and Lodi, A. (2020). Activation Adaptation in Neural Networks . In Proceedings of the 9th International Conference on Pattern Recognition Applications and Methods - ICPRAM; ISBN 978 -989-758-397-1; ISSN 2184 -4313, SciTePress, pages 249 -257. DOI: 10.52...

    2020 doi

  9. [17]

    and Bengio, Y

    Glorot, X. and Bengio, Y. (2010) Understanding the Difficulty of Training Deep Feedforward Neural Networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, 9, 249 -256. http://proceedin...

    2010

  10. [18]

    and Bengio, Y

    Glorot, X., Bordes, A. and Bengio, Y. (2011) Deep Sparse Rectifier Neural Networks. Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, 15, 315-323

    2011

  11. [19]

    & Courville, A

    Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning. (MIT Press, 2016)

    2016

  12. [20]

    Hochreiter, S. et al. Gradient flow in recurrent nets: the difficulty of learning long -term dependencies. in A Field Guide to Dynamical Recurrent Neural Networks (IEEE Press, 2001)

    2001

  13. [21]

    https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_iris.html

    Iris dataset: UCI Machine Learning Repository: https://archive.ics.uci.edu/ml/datasets/iris. https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_iris.html

  14. [22]

    Jagtap, A. D. & Karniadakis, G. E. How important are activation functions in regression and classification? A survey, performance comparison, and future directions. arXiv preprint arXiv:2209.02681 (2022)

    2022 arXiv

  15. [23]

    Delving Deep into Rectifiers: Surpassing Human -Level Performance on ImageNet Classification,

    K. He, X. Zhang, S. Ren and J. Sun, "Delving Deep into Rectifiers: Surpassing Human -Level Performance on ImageNet Classification," 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 2015, pp. 1026-1034, doi: 10.1109/ICCV.2015.123

    2015 doi

  16. [24]

    Kavun S., Zamula, A., Mizurin, V. (2019). Intelligent Evaluation Method for Complex Systems in The Big Data Environment, 2019 IEEE 2nd Ukraine Conference on Electrical and Computer Engineering (UKRCON), July 2-6, Lviv, Ukraine, PP. 951-957. DOI: 10.1109/UKRCON.2019.8880024. Av...

    2019

  17. [25]

    Preprint article

    Kavun, Sergii, Multiple Data -Driven Missing Imputation. Preprint article . Available at SSRN: https://arxiv.org/abs/2507.03061

    arXiv

  18. [26]

    Kunc, V., & Kl'ema, J. (2024). Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks. ArXiv, abs/2402.09092

    2024 arXiv

  19. [27]

    Computational aspects of critical infrastructures security

    Kuznetsov, A., Kalashnikov, V., Brumnik, R., & Kavun, S. (2020). Editorial “ Computational aspects of critical infrastructures security”, “security and post-quantum cryptography”. International Journal of Computing, 19(2), 233 -236. Retrieved from https://computingonline.net/c...

    2020

  20. [28]

    LeCun, Y. et al. Efficient BackProp. In Neural Networks: Tricks of the Trade 9 –48 (Springer, 1998)

    1998

  21. [29]

    & Hinton, G

    LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436 –444 (2015) , doi: 10.1038/nature14539

    2015 doi

  22. [30]

    LeCun, Y.A., Bottou, L., Orr, G.B., Müller, KR. (2012). Efficient BackProp . In: Montavon, G., Orr, G.B., Müller, KR. (eds) Neural Networks: Tricks of the Trade. Lecture Notes in Computer Science, vol

    2012

  23. [31]

    & Goldstein, T

    Li, H., Xu, Z., Taylor, G., Studer, C. & Goldstein, T. Visualizing the loss landscape of neural nets. in Advances in Neural Information Processing Systems 6389–6399 (2018)

    2018

  24. [32]

    and Ng, A.Y

    Maas, A.L., Hannun, A.Y. and Ng, A.Y. (2013) Rectifier Nonlinearities Improve Neural Network Acoustic Models. Proceedings of the 30th International Conference on Machine Learning, Vol. 28, 3. https://ai.stanford.edu/~amaas/papers/relu_hybrid_icml2013_final.pdf

    2013

  25. [33]

    & Vrahatis, A

    Maniakas, A. & Vrahatis, A. G. Piecewise linear activations for deep reinforcement learning. in 2018 IEEE 30th International Conference on Tools with Artificial Intelligence 852 –859 (IEEE, 2018)

    2018

  26. [34]

    Mercioni, M. A. & Holban, S. The Most Used Activation Functions: Classic Versus Current. in 2020 International Conference on Development and Application Systems (DAS) 141 –145 (2020) , doi: 10.1109/DAS49615.2020.9108942

    2020

  27. [35]

    Mish: A self-regularized non-monotonic activation function

    Misra, D. Mish: A self-regularized non-monotonic activation function. arXiv preprint arXiv:1908.08681 (2019)

    2019 arXiv

  28. [36]

    https://www.kaggle.com/datasets/hojjatk/mnist-dataset

    MNIST dataset: official site (Yann LeCun): http://yann.lecun.com/exdb/mnist/. https://www.kaggle.com/datasets/hojjatk/mnist-dataset. https://www.openml.org/d/554

  29. [37]

    & Bengio, Y

    Pascanu, R., Mikolov, T. & Bengio, Y. On the difficulty of training recurrent neural networks. In International conference on machine learning 1310–1318 (PMLR, 2013)

    2013

  30. [38]

    Pavlyshyn, I., Petrenko, A., Opryshko, B., Oliinyk, B., Kavun, S., & Vasylchuk S. (2024). Social Media Impact on the ‘Cosmos’ Blockchain Ecosystem: State and Prospect. Data Science Journal, 23(1), p

    2024

  31. [39]

    Ramachandran, P., Zoph, B. & Le, Q. V. Searching for activation functions. arXiv preprint arXiv:1710.05941 (2017)

    2017 arXiv

  32. [40]

    https://doi.org/10.5334/dsj-2024-017

    2024 doi

  33. [41]

    Sergii Kavun. (2025). s -kav/s3_s4_activation_function: v.1.0 (v.1.0). Zenodo. https://doi.org/10.5281/zenodo.16459162

    2025 doi

  34. [42]

    Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37 (ICML'15). JMLR.org, 448–456

    2015

  35. [43]

    Snopov, Pavel & Musin, Oleg. (2025). Topology -Aware Activation Functions in Neural Networks. 29-34. 10.14428/esann/2025.ES2025-211

    2025 doi

  36. [44]

    Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Mądry. 2018. How does batch normalization help optimization? In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS'18). Curran Associates Inc., Red Hook, NY, USA, 2488–2498

    2018

  37. [45]

    Yuen, B., Hoang, M.T., Dong, X. et al. Universal activation function for machine learning. Sci Rep 11, 18757 (2021). https://doi.org/10.1038/s41598-021-96723-8

    2021 doi

  38. [46]

    Xu, B., Wang, N., Chen, T. & Li, M. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853 (2015)

    2015 arXiv

  39. [7700]

    https://doi.org/10.1007/978-3-642-35289-8_3

    Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-35289-8_3

    doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.