REVIEW 3 major objections 7 minor 31 references
Semi-Periodic Activation for Time Series Classification
T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper claims that LeakySineLU, a semi-periodic activation whose derivative is periodic, is the best-ranked activation for time series classification on 112 equal-length UCR datasets in both an MLP and an FCN.
desk verdict A modest Snake variant whose FCN results are plausible but whose MLP pairwise counts sum to 127, not 112—so the central claim is currently not supported as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the LeakySineLU activation, $\sigma(x)=\sin^2(x)+x$ for $x>0$ and $\sigma(x)=(\sin^2(x)+x)/2$ otherwise. Its derivative is $\sigma'(x)=\sin(2x)+1$ on the positive side and $\sigma'(x)=(\sin(2x)+1)/2$ on the negative side, so the derivative is periodic with period $\pi$ on each side while the function itself is unbounded and monotonic. The discontinuity at $x=0$ is handled with a sub-derivative, exactly as ReLU and PReLU handle theirs. This combination—unbounded, monotonic, non-linear, and with a periodic derivative—is what the paper argues lets a network keep negative-valued observations (no dying ReLU) while still expressing periodic patterns.
What would settle it
Run the same MLP and FCN comparison with per-activation hyperparameter search and include the variable-length UCR datasets, reporting accuracy over multiple random seeds; if LeakySineLU no longer holds the best average rank in either architecture, or if its rank advantage over ReLU and PReLU falls within seed noise, the paper's central claim fails.
Extended reading notes
Core claim
On 112 equal-length datasets from the UCR archive, the paper reports that LeakySineLU achieves the best average rank among ten activation functions in both architectures tested: average rank 4.2402 in the MLP and 3.0491 in the FCN. In pairwise one-versus-one comparisons it wins a majority of datasets against ReLU, PReLU, and Snake, and in the FCN it is statistically different from ELU and Snake under the Wilcoxon test with Holm correction. Notably, in the MLP its mean accuracy (0.7081) is slightly below ReLU (0.7098) and PReLU (0.7097), so the rank advantage is not the same as a mean-accuracy advantage. The paper interprets the results as evidence that a semi-periodic activation captures periodic structure in time series better than monotonic or purely periodic alternatives, while avoiding the vanishing-gradient and dying-ReLU failure modes of bounded and zero-slope activations.
Load-bearing premise
The entire comparison uses one fixed training schedule (optimizer, learning rate, and epoch count) applied identically to every activation, and it covers only equal-length UCR datasets, so if that schedule happens to suit LeakySineLU or variable-length series are where periodic derivatives matter, the best-rank result will not generalize.
Editorial extensions
If this is right
- LeakySineLU is a drop-in activation with no extra learnable parameters, applicable to both dense and convolutional time series classifiers.
- On equal-length UCR classification, it ranks first among ten activations in both the MLP and FCN settings by average rank.
- It preserves negative-valued observations instead of zeroing them, addressing the dying-ReLU information loss while remaining unbounded and non-linear.
- Its derivative is periodic in both the positive and negative domains, giving networks a mechanism to represent periodic structure, consistent with the paper's Fourier-series motivation.
- In pairwise comparisons, it wins a majority of datasets against ReLU, PReLU, and Snake in both architectures.
Reading between the lines
- Because the evaluation used one fixed training schedule for every activation, a testable extension is to repeat the comparison with per-activation hyperparameter tuning and with variable-length UCR datasets included; the rank ordering could change under either condition.
- The gap between LeakySineLU's best average rank and its slightly lower mean accuracy in the MLP suggests the rank advantage may come from avoiding worst-case failures on particular datasets, so the activation could be most valuable where other activations collapse.
- The factor of 1/2 on the negative branch is an arbitrary design choice; testing other scales would reveal whether the exact constant matters for the reported advantage.
- The same periodic-derivative principle could be tested on other time series tasks such as forecasting and extrinsic regression, where seasonal structure is even more central.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes LeakySineLU, a piecewise activation defined as sigma(x) = sin^2(x) + x for x > 0 and (sin^2(x) + x)/2 otherwise, designed to combine unboundedness, non-linearity, monotonicity, and a semi-periodic derivative. It evaluates this activation against ReLU, PReLU, ELU, GeLU, SiLU, Snake, TanH, Sigmoid, and Sine in an MLP and an FCN on 112 equal-length UCR time-series classification datasets. The central claim is that LeakySineLU obtains the best average ranking in all comparative scenarios, supported by critical-difference diagrams, pairwise win/draw/loss counts, and multi-comparison matrices.
Significance. If the empirical claim holds, the paper offers a simple, drop-in activation that improves rank-based aggregate performance for time-series classification in simple MLP and FCN models, and its property checklist is a useful organizing device. The study is broad in benchmark coverage, uses standard non-parametric statistical post-hoc analysis, and makes the code available. There is no evident circularity: LeakySineLU is hand-designed from a property checklist and evaluated on external benchmarks, with no constants fitted to the UCR results. The significance is limited by the small mean-accuracy differences and the lack of repeated-seed variance; the contribution is incremental rather than a major advance, and the claims need to be calibrated to the actual evidence.
major comments (3)
- [§5, Figs. 4 and 6] The MLP pairwise win/draw/loss counts are inconsistent with the stated 112-dataset benchmark. In Fig. 4, LeakySineLU vs ReLU is 67/3/57 (sum 127), vs Snake is 75/3/49 (sum 127), and vs PReLU is 68/3/56 (sum 127); every comparison in the MLP MCM of Fig. 6 also sums to 127, whereas the FCN counts in Figs. 7 and 9 sum to 112. Since the captions and text state that the experiments use 112 equal-length datasets, the MLP ranking in Fig. 5 and the MCM in Fig. 6 may have been computed on a different dataset set. This is a load-bearing inconsistency for the abstract claim of best average ranking in all comparative scenarios. The authors must either rerun the MLP analysis on the declared 112 equal-length subset or re-scope the claim to the actual dataset set used; the FCN results alone cannot support the MLP claim.
- [§5, Figs. 6 and 9] The headline ranking result is accompanied by lower mean accuracy and mostly non-significant pairwise differences in both architectures. In the MLP MCM, LeakySineLU has mean accuracy 0.7081 against ReLU 0.7098 and PReLU 0.7097, with Wilcoxon p-values 0.8267 and 0.9042; in the FCN MCM it has 0.8006 against ReLU 0.8025, with p = 0.8583. No repeated-seed variance is reported, so the rank ordering may be unstable. The paper should either temper the claim from best to conditionally competitive, or add confidence intervals and repeated seeds and show that the average-rank advantage is not an artifact of a single run. A majority of wins in pairwise counts is not sufficient evidence of superiority when the mean accuracy is lower and the test is not significant.
- [§4.1, footnote 1] The benchmark is restricted to the 112 equal-length UCR datasets and to one fixed training schedule per architecture (Adadelta with lr = 1.0 and 1000 epochs for MLP; Adam with lr = 0.001 and 2000 epochs for FCN), while the abstract and conclusion speak of time series classification without this qualification. The paper should either narrow the claims to equal-length classification under this fixed protocol, or add evidence that the ranking is stable under variable-length series and different hyperparameters. Otherwise the generalization claim is not supported by the reported experiments.
minor comments (7)
- [§3.2, Eq. (5)] The otherwise branch of the definition is ambiguous; as printed, sin2(x)+x over 2 could be read as sin^2(x) + x/2 rather than (sin^2(x)+x)/2. Please add parentheses.
- [§3.3, Eqs. (7)-(8)] Because sin^2(+∞) is undefined, the limit argument should be written using inequalities, for example sin^2(x) ≥ 0, so x + sin^2(x) → +∞ and (x + sin^2(x))/2 → -∞.
- [§2, Definition 3] Definition 3 requires σ′(x+T) = σ′(x) for all x, but the derivative of LeakySineLU is undefined at x = 0 and the one-sided limits differ; please state that the periodicity holds on each branch of the derivative.
- [§3.5] The sub-derivative definition in Section 2 is stated for convex functions, but LeakySineLU is not convex; the treatment of the discontinuity at x = 0 should instead use one-sided derivatives or a clearly defined generalized subgradient.
- [§3.6] The Fourier-series motivation leading to Eq. (17) is not rigorous: cosine and sine basis terms with different arguments cannot be collapsed into a single sin(X) applied to an input matrix without specifying phase and argument-matching conditions.
- [§3.1] The sentence introducing the ReLU family is garbled: it says ReLU, ELU as an exponential linear unit (PReLU) for its learnable parameters, which conflates ELU and PReLU and introduces LeakyReLU elsewhere without defining it.
- [§5] The Friedman test is invoked following [5], but no Friedman statistic or p-value is reported; please include it or clarify that only the Wilcoxon signed-rank test with Holm correction is used.
Circularity Check
No circularity: LeakySineLU is hand-designed and its headline claim is an empirical ranking on external UCR benchmarks, not a quantity re-derived from fitted inputs.
full rationale
The paper's central derivation is self-contained. LeakySineLU is explicitly defined in Eq. 5 as σ(x)=sin²(x)+x for x>0 and (sin²(x)+x)/2 otherwise, with derivative in Eq. 6. No parameter of the activation is fitted to UCR accuracies; the evaluation is a post-hoc comparison on an external benchmark suite. The theoretical motivation in Sec. 3.6 uses Fourier-series reasoning to justify periodic or semi-periodic activation, but it does not derive the benchmark result from the activation formula. The only arguable overlap is that the positive branch equals Snake with a=1 (Snake formula in Table 1 is x+sin²(ax)/a), which is a design similarity, not a circular dependency: the paper does not claim to derive Snake or to predict its ranking from its own formula. There are no load-bearing self-citations: references [31], [5], and [10] are external prior work. The reported win/loss/draw counts that sum to 127 rather than 112 for MLP (Figs. 4 and 6) are an internal consistency concern about the empirical claim, not a circularity of the derivation chain.
Assumptions & free parameters
free parameters (3)
- Negative branch scale factor =
1/2
- Snake period parameter a =
1
- Branch threshold =
0
assumptions (3)
- domain assumption Universal Extrapolation Theorem from [31] is valid and implies usefulness of periodic components
- domain assumption The 112 equal-length UCR datasets are a representative benchmark for time series classification
- standard math Sub-gradient optimization at the x=0 kink behaves like ReLU and PReLU
Cite this review
Pith. "Pith review of Semi-Periodic Activation for Time Series Classification." pith.science (2026). https://pith.science/paper/K3BFEHNC
@misc{pith2026241209889,
author = {Pith},
title = {Pith review of: Semi-Periodic Activation for Time Series Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/K3BFEHNC}},
note = {Machine review of arXiv:2412.09889}
}
read the original abstract
This paper investigates the lack of research on activation functions for neural network models in time series tasks. It highlights the need to identify essential properties of these activations to improve their effectiveness in specific domains. To this end, the study comprehensively analyzes properties, such as bounded, monotonic, nonlinearity, and periodicity, for activation in time series neural networks. We propose a new activation that maximizes the coverage of these properties, called LeakySineLU. We empirically evaluate the LeakySineLU against commonly used activations in the literature using 112 benchmark datasets for time series classification, obtaining the best average ranking in all comparative scenarios.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
IEEE Transactions on Neural Networks5(2), 157–166 (1994)
Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks5(2), 157–166 (1994). https://doi.org/10.1109/72.279181
-
[2]
Clevert, D., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (elus). In: Bengio, Y., LeCun, Y. (eds.) 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2- 4, 2016, Conference Track Proceedings (2016),http://arxiv.org/abs/1511.07289
arXiv 2016
-
[3]
Dau, H.A., Keogh, E., Kamgar, K., Yeh, C.C.M., Zhu, Y., Gharghabi, S., Ratanama- hatana, C.A., Yanping, Hu, B., Begum, N., Bagnall, A., Mueen, A., Batista, G., Hexagon-ML: The ucr time series classification archive (October 2018), https://www.cs.ucr.edu/~eamonn/time_series_data_2018/
work page 2018
-
[4]
CoRRabs/1702.03118 (2017), http://arxiv.org/abs/1702.03118
Elfwing, S., Uchibe, E., Doya, K.: Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. CoRRabs/1702.03118 (2017), http://arxiv.org/abs/1702.03118
arXiv 2017
-
[5]
Fawaz, H.I., Forestier, G., Weber, J., Idoumghar, L., Muller, P.A.: Deep learning for time series classification: a review. Data Mining and Knowledge Discovery33(4), 917–963 (mar 2019).https://doi.org/10.1007/s10618-019-00619-1, https:// doi.org/10.1007%2Fs10618-019-00619-1
-
[6]
Data Mining and Knowledge Discovery34(6), 1936–1962 (sep 2020)
Fawaz, H.I., Lucas, B., Forestier, G., Pelletier, C., Schmidt, D.F., Weber, J., Webb, G.I., Idoumghar, L., Muller, P.A., Petitjean, F.: InceptionTime: Finding AlexNet for time series classification. Data Mining and Knowledge Discovery34(6), 1936–1962 (sep 2020). https://doi.org/10.1007/s10618-020-00710-y, https: //doi.org/10.1007%2Fs10618-020-00710-y
-
[7]
Foumani, N.M., Miller, L., Tan, C.W., Webb, G.I., Forestier, G., Salehi, M.: Deep learning for time series classification and extrinsic regression: A current survey (2023)
work page 2023
-
[8]
In: 2015 IEEE International Conference on Computer Vision (ICCV)
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In: 2015 IEEE International Conference on Computer Vision (ICCV). pp. 1026–1034 (2015).https://doi.org/10.1109/ ICCV.2015.123
work page 2015
Show all 31 references
-
[9]
CoRRabs/1606.08415 (2016), http://arxiv.org/ abs/1606.08415
Hendrycks, D., Gimpel, K.: Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRRabs/1606.08415 (2016), http://arxiv.org/ abs/1606.08415
2016 arXiv
-
[10]
arXiv preprint arXiv:2305.11921 (2023)
Ismail-Fawaz, A., Dempster, A., Tan, C.W., Herrmann, M., Miller, L., Schmidt, D.F., Berretti, S., Weber, J., Devanne, M., Forestier, G., et al.: An approach to multiple comparison benchmark evaluations that is stable under manipulation of the comparate set. arXiv preprint arXi...
2023 arXiv
-
[11]
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization (2017)
2017
-
[12]
Neurocomputing 216, 718–734 (2016)
Liew, S.S., Khalil-Hani, M., Bakhteri, R.: Bounded activation functions for en- hanced training stability of deep neural networks on visual pattern recogni- tion problems. Neurocomputing 216, 718–734 (2016). https://doi.org/https: //doi.org/10.1016/j.neucom.2016.08.037, https:...
2016 doi
-
[13]
Big Data Research34, 100407 (2023).https://doi.org/https://doi.org/10
Lima, F.T., Souza, V.M.: A large comparison of normalization methods on time se- ries. Big Data Research34, 100407 (2023).https://doi.org/https://doi.org/10. 1016/j.bdr.2023.100407, https://www.sciencedirect.com/science/article/ pii/S2214579623000400
2023
-
[14]
Misra, D.: Mish: A self regularized non-monotonic activation function (2020) Semi-Periodic Activation for Time Series Classification 15
2020
-
[15]
In: Proceedings of the 27th international conference on machine learning (ICML-10)
Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th international conference on machine learning (ICML-10). pp. 807–814 (2010)
2010
-
[16]
Parascandolo, G., Huttunen, H., Virtanen, T.: Taming the waves: sine as activation function in deep neural networks (2017), https://openreview.net/forum?id= Sks3zF9eg
2017
-
[17]
Pedamonti, D.: Comparison of non-linear activation functions for deep neural networks on mnist classification task (2018)
2018
-
[18]
Acta Numerica 8, 143–195 (1999).https://doi.org/10.1017/S0962492900002919
Pinkus, A.: Approximation theory of the mlp model in neural networks. Acta Numerica 8, 143–195 (1999).https://doi.org/10.1017/S0962492900002919
1999 doi
-
[19]
In: 2020 IEEE 18th World Symposium on Applied Machine Intelligence and Informatics (SAMI)
Rasamoelina, A.D., Adjailia, F., Sinčák, P.: A review of activation function for artificial neural network. In: 2020 IEEE 18th World Symposium on Applied Machine Intelligence and Informatics (SAMI). pp. 281–286 (2020).https://doi.org/10. 1109/SAMI48414.2020.9108717
2020
-
[20]
Data Mining and Knowledge Discovery35, 401–449 (2021)
Ruiz, A.P., Flynn, M., Large, J., Middlehurst, M., Bagnall, A.: The great multi- variate time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Mining and Knowledge Discovery35, 401–449 (2021). https://doi.org/https://doi...
2021 doi
-
[21]
In: IJCNN’99
Silvescu, A.: Fourier neural networks. In: IJCNN’99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339). vol. 1, pp. 488–491 vol.1 (1999). https://doi.org/10.1109/IJCNN.1999.831544
1999
-
[22]
Applied and Computational Harmonic Analysis43(2), 233– 268 (2017)
Sonoda, S., Murata, N.: Neural network with unbounded activation functions is universal approximator. Applied and Computational Harmonic Analysis43(2), 233– 268 (2017). https://doi.org/https://doi.org/10.1016/j.acha.2015.12.005, https://www.sciencedirect.com/science/article/pi...
2017 doi
-
[23]
In: 1999 Ninth International Conference on Artificial Neural Networks ICANN 99
Sopena, J., Romero, E., Alquezar, R.: Neural networks with periodic and monotonic activation functions: a comparative study in classification problems. In: 1999 Ninth International Conference on Artificial Neural Networks ICANN 99. (Conf. Publ. No. 470). vol. 1, pp. 323–328 vo...
1999 doi
-
[24]
Wang, Z., Yan, W., Oates, T.: Time series classification from scratch with deep neural networks: A strong baseline (2016)
2016
-
[25]
In: Proceedings of the Thirtieth Interna- tional Joint Conference on Artificial Intelligence
Wen, Q., Sun, L., Yang, F., Song, X., Gao, J., Wang, X., Xu, H.: Time series data augmentation for deep learning: A survey. In: Proceedings of the Thirtieth Interna- tional Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence ...
2021 doi
-
[26]
Wen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J., Sun, L.: Transformers in time series: A survey (2023)
2023
-
[27]
In: 2019 IEEE International Conference on Smart Manufacturing, Industrial & Logistics Engineering (SMILE)
Wu, J., Zhang, Z., Ji, Y., Li, S., Lin, L.: A resnet with ga-based structure optimiza- tion for robust time series classification. In: 2019 IEEE International Conference on Smart Manufacturing, Industrial & Logistics Engineering (SMILE). pp. 69–74 (2019). https://doi.org/10.11...
2019
-
[28]
Knowledge-Based Systems 245, 108606 (2022)
Yang, X., Zhang, Z., Cui, R.: Timeclr: A self-supervised contrastive learn- ing framework for univariate time series representation. Knowledge-Based Systems 245, 108606 (2022). https://doi.org/https://doi.org/10.1016/j. knosys.2022.108606, https://www.sciencedirect.com/science...
2022
-
[29]
Zeiler, M.D.: Adadelta: An adaptive learning rate method (2012)
2012
-
[30]
Zhumekenov, A., Uteuliyeva, M., Kabdolov, O., Takhanov, R., Assylbekov, Z., Castro, A.J.: Fourier neural networks: A comparative study (2019) 16 J. G. B. de Medeiros Júnior et al
2019
-
[31]
In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H
Ziyin, L., Hartwig, T., Ueda, M.: Neural networks fail to learn periodic functions and how to fix it. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 1583–1594. Curran Associates, Inc. (20...
2020
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.