REVIEW 5 major objections 4 minor 28 references
A parametric activation function based on Wendland RBF
T0 review · 5 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper's enhanced Wendland RBF activation reaches 99.33% on MNIST and 92.47% on Fashion-MNIST, ahead of every standard activation in its comparison.
desk verdict A genuinely new parametric activation formula, but the paper never ties the precise mathematical definition to the experiments, so the headline accuracy numbers cannot be trusted as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the enhanced Wendland activation $$\$\varphi$(r)=(1-\$\alpha$ r)^k_+(k\$\alpha$ r+1)+\$\lambda$ r+\epsilon $e^{{-\beta r}}$,\qquad x\mapsto x\odot\$\varphi$(r),$$ where $r$ is the channel-wise $\ell^2$ norm of the input vector, $(z)_+=\max(0,z)$, $k=4$ by default, $\alpha$ is trainable, and $\lambda,\epsilon,\beta$ are small positive constants. This object carries the argument: it is a smooth, compactly supported radial scaling that stays non-vanishing at large radius and non-singular near zero, so the network can learn a per-channel locality while preserving gradient flow.
What would settle it
Re-run the VGG and LeNet runs on MNIST and Fashion-MNIST with the activation applied pointwise (each scalar input using $r=|x|$) and compare against the same runs using the full channel-wise vector norm. If the margin over ReLU disappears in the pointwise version, the radial norm rather than the Wendland smoothing is carrying the reported improvement.
Extended reading notes
Core claim
On the paper's own terms, the central claim is that the enhanced Wendland activation $x\odot[(1-\alpha r)^k_+(k\alpha r+1)+\lambda r+\epsilon e^{-\beta r}]$, with $r$ the channel-wise L2 norm of $x$, matches or surpasses standard activations while improving training stability. The Wendland term supplies compact support and smoothness, the linear term keeps gradients alive at large $r$, and the exponential term stabilizes behavior near $r=0$, with the learnable $\alpha$ controlling the radius. The numerical experiments list $99.33\%/99.27\%$ accuracy on MNIST and $92.47\%/92.07\%$ on Fashion-MNIST with VGG and LeNet, above all other activation functions in Table 1. The paper further claims that the compact support reduces overfitting and improves generalization.
Load-bearing premise
The formula is defined on a whole input vector through a channel-wise Euclidean norm, and the paper never states how that vector-level distance was computed inside the layers used in the reported experiments; if the experiments actually used a pointwise activation, the proposed formula is not what was tested.
Editorial extensions
If this is right
- If the reported results hold, the enhanced Wendland activation outperforms ReLU-family activations on standard vision benchmarks, with the clearest margin on Fashion-MNIST.
- Compact support means inputs beyond a learned radius are clipped, which the paper argues mitigates overfitting and improves generalization.
- The linear and exponential terms keep gradients flowing at large and small radial distances, reducing the vanishing-gradient failure mode of sigmoid and tanh.
- The learnable $\alpha$ parameter lets the network adjust the locality radius per channel or layer, making the activation adaptive rather than fixed-shape.
Reading between the lines
- Editorial extension: if the vector-level norm is the real driver, the Wendland activation is closer to a lightweight normalization or attention mechanism than to a pointwise activation, so its benefits may transfer to architectures that exploit channel statistics.
- Editorial extension: a direct test on smooth regression tasks, such as physics-informed or function-approximation networks, would clarify whether the claimed smoothness is what stabilizes training, since that is where compact-support RBFs have a theoretical pedigree.
- Editorial extension: the compact support could act as an implicit regularizer, so measuring the train/test accuracy gap under identical augmentation would test the overfitting claim more directly than final accuracy.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a parametric activation function built from a Wendland radial basis function, augmented with a linear term and an exponential tail. The activation is defined in Section 4.1 as an elementwise scaling of the input by a function of a radial distance r, with trainable parameters α, λ, β, ε and polynomial degree k. The authors claim that this activation has useful theoretical properties—smoothness, compact support, adaptability—and report empirical comparisons on MNIST and Fashion-MNIST in Table 1, as well as qualitative experiments on sine-wave regression (Example 5.1) and moons/circles classification (Example 5.2). The abstract and conclusions claim superior accuracy in certain scenarios, particularly regression, and better generalization through localized, smooth transformations.
Significance. If the central empirical claim were substantiated—namely that the proposed Wendland activation outperforms ReLU, ELU, Swish, PReLU and others on standard benchmarks—the paper would report a useful addition to the activation-function literature. The idea of using compactly supported radial functions as trainable activations is interesting and connects approximation theory with deep learning. However, as written, the significance cannot be assessed: the activation is not defined unambiguously, the experimental section provides almost no protocol or numerical results for the regression task, and the claimed theoretical analysis is absent. The paper does not ship code, machine-checked proofs, or parameter-free derivations, and the empirical tables are not reproducible from the information given.
major comments (5)
- [§4.1, activation definition] The proposed activation is written as x ⊙ [(1−αr)_+^k (kαr+1) + λr + εe^{−βr}], where r is called the 'channel-wise L2 norm' of the input. This is ambiguous in a way that directly affects the experiments. In a standard fully connected or convolutional layer, the pre-activation of each neuron is a scalar, for which a 'channel-wise L2 norm' has no meaning; if r is instead the norm of a feature vector across channels, the activation becomes a radial scaling that multiplies all channels by one common factor, which is a different mathematical map from a pointwise scalar activation. The manuscript never states which interpretation was implemented in the experiments behind Table 1, and no code is provided. The reported MNIST and Fashion-MNIST numbers therefore cannot be reliably attributed to the function as written.
- [§5, Table 1 and experimental protocol] Table 1 reports accuracy numbers for MNIST and Fashion-MNIST with VGG and LeNet variants, but no experimental protocol is given: there is no statement of the exact architectures, optimizer, learning rate, batch size, number of epochs, weight initialization, data augmentation, or number of independent runs. Without error bars or multiple seeds, the claimed improvements—for example Fashion-MNIST VGG accuracy of 92.47 for Wend versus 89.69 for ReLU—cannot be distinguished from run-to-run variation or from differences in training setup. This is a load-bearing gap because the paper's central claim is the empirical superiority of the proposed activation.
- [§5, Examples 5.1 and 5.3] The abstract emphasizes superior accuracy in regression tasks, but the sine-wave approximation experiment in Example 5.1 reports no numerical results whatsoever—no final loss, no accuracy metric, no comparison table—and Example 5.3 is empty, stating only that the activation is 'evaluated on a slightly more complex dataset' with no dataset description, results, or analysis. Consequently, the claim of superiority in regression is unsupported by any data in the manuscript.
- [Abstract, §1, §6] The paper repeatedly claims a 'theoretical analysis' demonstrating smoothness, stability, adaptive scaling, and improved gradient propagation. However, no theorem, proposition, derivation, or formal argument appears anywhere in the text; the only mathematical content is a restatement of standard Wendland properties and the definition of the proposed function. The theoretical contribution claimed in the abstract and conclusions is therefore not present in the manuscript.
- [§4.1, parameterization] The parameters α, λ, β, ε, and k are introduced with suggested defaults, but the manuscript does not state whether all of them are learned, how they are initialized, whether they are shared across layers or per-layer, or how the compact support radius (governed by α) interacts with weight initialization and normalization layers. This matters because the paper's stated advantage includes trainable adaptability, and without this information the reported experiments cannot be reproduced or even interpreted.
minor comments (4)
- [Throughout] The manuscript contains numerous typographical and grammatical errors that impede reading, including 'active functionted' and 'gractive functionted' in Section 3, 'active functionfects' in Example 5.1, and inconsistent use of 'active function' versus 'activation function'. A thorough proofreading pass is needed.
- [§4.1, notation] The text writes a general polynomial P(αr) and then immediately specializes to (kαr+1), but it is not clear whether P is meant to be a fixed polynomial of degree k or a learnable polynomial. The notation should be made precise, and the role of k as a hyperparameter versus a learned parameter should be clarified.
- [Figures 1 and 2] Figure 1 and Figure 2 are referenced in Examples 5.1 and 5.2, but the captions do not fully describe the axes, the curves, or the experimental conditions, and the figures are not analyzed in the text in a way that supports the claimed conclusions. Please describe what is plotted and what qualitative conclusion the reader should draw.
- [References] Reference [2] (Baes 2024) appears in the reference list but is not cited in the body of the paper. Please check that all listed references are cited and that all citations in the text are included in the list.
Circularity Check
No significant circularity: the Wendland activation's trainable parameters are fitted to training data in the standard supervised way, and the empirical claims are evaluated against independent benchmarks.
full rationale
The paper proposes a parametric activation function whose parameters (alpha, lambda, beta, epsilon) are learned from training data. That is ordinary supervised fitting, not a case where a prediction is equivalent to an input by construction. The activation formula does not define the benchmark accuracies in terms of itself, and Table 1 compares Wend against standard activations on external datasets (MNIST and Fashion-MNIST) rather than against a quantity derived from the same fitted parameters. No load-bearing argument is justified solely by a self-citation, and no uniqueness theorem from the authors is invoked to force a choice. The main substantive concern in the paper is an implementation ambiguity in Section 4.1: the radial distance r is described as a channel-wise L2 norm, which makes the activation vector-wise rather than a standard pointwise scalar activation, and the paper does not state which interpretation was used in the experiments. That ambiguity undermines reproducibility and the attribution of the reported numbers to the proposed formula, but it is not a circularity: the reported results are not equivalent to the paper's inputs by definition, nor is any fitted parameter renamed as a prediction. Example 5.3 being empty is an omitted experimental detail, not a circular step. Overall, the derivation chain contains no self-referential reduction, so the circularity score is 0.
Assumptions & free parameters
free parameters (5)
- alpha (α) =
learnable
- lambda (λ) =
0.1 default
- epsilon (ε) =
0.01 default
- beta (β) =
not specified
- k (polynomial degree) =
4 default
assumptions (4)
- standard math Wendland functions are positive definite and compactly supported.
- domain assumption Smooth activations improve gradient flow and convergence in deep networks.
- ad hoc to paper The proposed radial scaling f(x)=x·g(||x||) preserves spatial structure and is suitable as an activation for standard networks.
- domain assumption A standard fully connected or convolutional network with the proposed activation can approximate the sine function and classify benchmark images.
Cite this review
Pith. "Pith review of A parametric activation function based on Wendland RBF." pith.science (2026). https://pith.science/paper/RWJ4SLA7
@misc{pith2026250711493,
author = {Pith},
title = {Pith review of: A parametric activation function based on Wendland RBF},
year = {2026},
howpublished = {\url{https://pith.science/paper/RWJ4SLA7}},
note = {Machine review of arXiv:2507.11493}
}
read the original abstract
This paper introduces a novel parametric activation function based on Wendland radial basis functions (RBFs) for deep neural networks. Wendland RBFs, known for their compact support, smoothness, and positive definiteness in approximation theory, are adapted to address limitations of traditional activation functions like ReLU, sigmoid, and tanh. The proposed enhanced Wendland activation combines a standard Wendland component with linear and exponential terms, offering tunable locality, improved gradient propagation, and enhanced stability during training. Theoretical analysis highlights its mathematical properties, including smoothness and adaptability, while empirical experiments on synthetic tasks (e.g., sine wave approximation) and benchmark datasets (MNIST, Fashion-MNIST) demonstrate competitive performance. Results show that the Wendland-based activation achieves superior accuracy in certain scenarios, particularly in regression tasks, while maintaining computational efficiency. The study bridges classical RBF theory with modern deep learning, suggesting that Wendland activations can mitigate overfitting and improve generalization through localized, smooth transformations. Future directions include hybrid architectures and domain-specific adaptations.
Figures
Reference graph
Works this paper leans on
-
[1]
Apicella, A., Donnarumma, F., Isgro, F., & Prevete, R. (2021). A survey on modern trainable activation functions. Neural Networks, 138, 14-32. 10
work page 2021
-
[2]
Baes, M. (2024). Self-consistent dynamical models with a finite extent–IV. Wendland models based on compactly supported radial basis functions. Monthly Notices of the Royal Astronomical Society, 531(4), 5097-5108
work page 2024
-
[3]
Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2), 157-166
work page 1994
-
[4]
Chen, J., & Pan, Z. (2023). Saturated Non-Monotonic Activation Functions. arXiv preprint arXiv:2305.07537
arXiv 2023
-
[5]
A., Unterthiner, T., & Hochreiter, S
Clevert, D. A., Unterthiner, T., & Hochreiter, S. (2015). Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289
arXiv 2015
-
[6]
R., Singh, S
Dubey, S. R., Singh, S. K., & Chaudhuri, B. B. (2022). Activation functions in deep learning: A comprehensive survey and benchmark. Neurocomputing, 503, 92-108
2022
-
[7]
Gulcehre, C., Moczulski, M., Denil, M., & Bengio, Y. (2016, June). Noisy activation functions. In International conference on machine learning (pp. 3059-3068). PMLR
work page 2016
-
[8]
Glorot, X., Bordes, A., & Bengio, Y. (2011, June). Deep sparse rectifier neural networks. In Proceed- ings of the fourteenth international conference on artificial intelligence and statistics (pp. 315-323). JMLR Workshop and Conference Proceedings
work page 2011
Show all 28 references
-
[9]
Goyal, M., Goyal, R., & Lall, B. (2019). Learning activation functions: A new paradigm for understanding neural networks. arXiv preprint arXiv:1906.09529
2019 arXiv
-
[10]
(2019, May)
Hayou, S., Doucet, A., & Rousseau, J. (2019, May). On the impact of the activation function on deep neural networks training. In International conference on machine learning (pp. 2672-2680). PMLR
2019
-
[11]
He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision (pp. 1026-1034)
2015
-
[12]
Hendrycks, D., & Gimpel, K. (2016). Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415
2016 arXiv
-
[13]
(2019, December)
Li, H., Li, J., Guan, X., Liang, B., Lai, Y., & Luo, X. (2019, December). Research on overfitting of deep learning. In 2019 15th international conference on computational intelligence and security (CIS) (pp. 78-81). IEEE
2019
-
[14]
A., & Zulqarnain, M
Javid, I., Ghazali, R., Syed, I., Husaini, N. A., & Zulqarnain, M. (2022, October). Developing Novel T-Swish Activation Function in Deep Learning. In 2022 International Conference on IT and Industrial Technologies (ICIT) (pp. 1-7). IEEE
2022
-
[15]
Molina, A., Schramowski, P., & Kersting, K. (2019). Pad´ e activation units: End-to-end learning of flexible activation functions in deep networks. arXiv preprint arXiv:1907.06732
2019 arXiv
-
[16]
Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10) (pp. 807-814)
2010
-
[17]
V., & Edla, D
Naresh Babu, K. V., & Edla, D. R. (2017). New algebraic activation function for multi-layered feed forward neural networks. IETE Journal of Research, 63(1), 71-79
2017
-
[18]
Nwankpa, C., Ijomah, W., Gachagan, A., & Marshall, S. (2018). Activation functions: Comparison of trends in practice and research for deep learning. arXiv preprint arXiv:1811.03378. 11
2018 arXiv
-
[19]
S., Bhateja, P
Job, M. S., Bhateja, P. H., Gupta, M., Bingi, K., & Prusty, B. R. (2022). Fractional rectified linear unit activation function and its variants. Mathematical Problems in Engineering, 2022(1), 1860779
2022
-
[20]
Paul, A., Bandyopadhyay, R., Yoon, J.H., Geem, Z.W., & Sarkar, R. (2022). SinLU: Sinu- Sigmoidal Linear Unit. Mathematics 10(3), p. 337
2022
-
[21]
Ramachandran, P., Zoph, B., & Le, Q. V. (2017). Searching for activation functions. arXiv preprint arXiv:1710.05941
2017 arXiv
-
[22]
Sun, H., Wu, Z., Xia, B., Chang, P., Dong, Z., Yuan, Y., ...& Wang, X. (2024). A Method on Searching Better Activation Functions. arXiv preprint arXiv:2405.12954
2024 arXiv
-
[23]
Strack, L., Sactive functionari, M., & Hutter, F. (2024). Efficient Search for Customized Activation Functions with Gradient Descent. arXiv preprint arXiv:2408.06820
2024 arXiv
-
[24]
Xu, B., Huang, R., & Li, M. (2016). Revise saturated activation functions. arXiv preprint arXiv:1602.05980
2016 arXiv
-
[25]
Xu, B., Wang, N., Chen, T., & Li, M. (2015). Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853
2015 arXiv
-
[26]
Wang, X., Ren, H., & Wang, A. (2022). Smish: A novel activation function for deep learning methods. Electronics, 11(4), 540
2022
-
[27]
Wendland, H. (2004). Scattered data approximation (Vol. 17). Cambridge university press
2004
-
[28]
Zhang, J., & Ding, C. (2025). Simple yet effective adaptive activation functions for physics- informed neural networks. Computer Physics Communications, 307, 109428
2025
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.