pith. sign in

arxiv: 2606.11104 · v1 · pith:IH7WUNTUnew · submitted 2026-06-09 · 💻 cs.LG · math.CA· stat.ML

Limitations of Learning Tanh Neural Networks with Finite Precision

Philipp Grohs , Mat\v{e}j Tr\"odler This is my paper

Pith reviewed 2026-06-27 13:58 UTC · model grok-4.3

classification 💻 cs.LG math.CAstat.ML
keywords tanh neural networksfinite precisionlearning limitationsMonte Carlo ratebump functionsL^p convergenceadaptive algorithmspoint evaluations
0
0 comments X

The pith

Finite precision prevents any adaptive algorithm from learning tanh networks faster than Monte Carlo rates in L^p unless samples grow exponentially with network size.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that finite-precision computations impose strict limits on learning tanh neural networks from point samples with L^p accuracy guarantees. It constructs sharply localized bump functions using iterated tanh activations to show that adaptive randomized algorithms cannot exceed the Monte Carlo convergence rate of O(m^{-1/p}) unless the number of samples grows exponentially with the network's parameter count and depth. A sympathetic reader would care because this reveals why efficient learning of functions with sharp local features fails under realistic arithmetic constraints, extending similar barriers previously shown for ReLU networks.

Core claim

Using a novel construction of sharply localized bump functions via iterated tanh activations, the paper shows that in a finite-precision setting, no adaptive randomized algorithm based on m samples can achieve a convergence rate higher than the Monte Carlo rate O(m^{-1/p}) in the L^p norm, unless the sampling budget grows exponentially with the size of the network parameters and architecture. The results reveal fundamental limitations imposed by finite precision on the learnability of classes containing localized bump functions.

What carries the argument

Sharply localized bump functions constructed by iterating tanh activations, which create functions that remain hard to distinguish under limited precision and samples.

If this is right

  • Learning classes of functions that contain localized bump functions cannot exceed Monte Carlo rates under finite precision.
  • Adaptive randomized sampling strategies fail to improve upon O(m^{-1/p}) convergence unless the sample budget scales exponentially with network parameters and architecture.
  • The same finite-precision barrier previously identified for ReLU networks extends directly to tanh networks.
  • Pointwise evaluation alone cannot efficiently capture sharp local variations when arithmetic precision is fixed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Hardware that increases floating-point precision may be as critical as additional samples for overcoming these learning barriers.
  • The limitation likely applies to any activation that can generate similar localized features through iteration.
  • Theoretical analyses of neural network training should incorporate discretization effects when claiming super-Monte-Carlo rates.

Load-bearing premise

The construction of sharply localized bump functions via iterated tanh activations remains valid and distinguishable under the finite-precision arithmetic model.

What would settle it

An explicit finite-precision algorithm that recovers a tanh network containing a localized bump to L^p accuracy better than O(m^{-1/p}) using only polynomially many samples in the network size would falsify the claim.

Figures

Figures reproduced from arXiv: 2606.11104 by Mat\v{e}j Tr\"odler, Philipp Grohs.

Figure 1
Figure 1. Figure 1: Screenshot of the application for numerical verification of the main result in Theorem [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Layer-by-layer construction of the bump network [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Iterates of σ n a for a = 2, converging to the step-like limit sign(x) x⋆. 10 [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of the sharpening mechanism for [PITH_FULL_IMAGE:figures/full_fig_p022_4.png] view at source ↗
read the original abstract

We investigate limitations of learning $\tanh$ neural networks from point evaluations under finite-precision computations and $L^p$ accuracy guarantees, building on Berner, Grohs, and Voigtl\"ander (2023). Our approach is based on a novel construction of sharply localized bump functions via iterated $\tanh$ activations. Using this mechanism, we show that, in a finite-precision setting, no adaptive randomized algorithm based on $m$ samples can achieve a convergence rate higher than the Monte Carlo rate $O(m^{-1/p})$ in the $L^p$ norm, unless the sampling budget grows exponentially with the size of the network parameters and architecture. The results reveal fundamental limitations imposed by finite precision on the learnability of classes containing localized bump functions, extending previous results for ReLU networks to the $\tanh$ setting.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper extends results from Berner-Grohs-Voigtländer (2023) on ReLU networks to tanh networks. It constructs a family of sharply localized bump functions realized by iterated tanh activations and uses this family to prove that, in a finite-precision arithmetic model, no adaptive randomized algorithm based on m point evaluations can achieve an L^p convergence rate better than the Monte Carlo rate O(m^{-1/p}) unless the sample budget m grows exponentially with the network depth, width, and parameter count.

Significance. If the central construction is valid under the stated finite-precision model, the result supplies a concrete, architecture-dependent sample-complexity lower bound that applies to any adaptive randomized learner. This strengthens the understanding of information-theoretic barriers imposed by rounding and distinguishes the finite-precision regime from the exact-arithmetic setting. The explicit bump construction is a technical contribution that could be reused for other activation families.

major comments (2)
  1. [Construction of bump functions (likely §3–4)] The load-bearing step is the claim that the iterated-tanh bump functions remain nonzero at some query point and mutually distinguishable from the zero function (and from each other) after rounding in the finite-precision model of Berner et al. (2023). No error-propagation analysis, numerical verification for depth-dependent iteration counts, or explicit bound on the rounding threshold appears in the manuscript; without this, the exponential-sample lower bound does not follow.
  2. [Main theorem statement] The statement that the lower bound holds “unless the sampling budget grows exponentially with the size of the network parameters and architecture” is not accompanied by an explicit dependence of the required m on depth, width, or bit precision. A quantitative statement relating the iteration depth k needed for localization to the precision parameter would make the exponential blow-up precise.
minor comments (2)
  1. [Preliminaries] Notation for the finite-precision rounding operator and the precise model of adaptive querying should be introduced once and used consistently.
  2. [Abstract and §1] The abstract and introduction both state the Monte Carlo rate O(m^{-1/p}); a short remark clarifying whether the constant hidden in O(·) depends on the network class would help readers.

Simulated Author's Rebuttal

2 responses · 0 unresolved

Thank you for the careful review and for recognizing the potential reuse of the bump construction. We address the two major comments below. Where the manuscript lacks explicit supporting analysis, we will revise to include it.

read point-by-point responses

  1. Referee: [Construction of bump functions (likely §3–4)] The load-bearing step is the claim that the iterated-tanh bump functions remain nonzero at some query point and mutually distinguishable from the zero function (and from each other) after rounding in the finite-precision model of Berner et al. (2023). No error-propagation analysis, numerical verification for depth-dependent iteration counts, or explicit bound on the rounding threshold appears in the manuscript; without this, the exponential-sample lower bound does not follow.

    Authors: We agree that a self-contained error-propagation argument is needed to make the distinguishability claim fully rigorous under the finite-precision model. The construction inherits the rounding model of Berner et al. (2023), but the iterated tanh requires tracking how the localization width and height degrade with iteration count k. In the revision we will insert a new lemma that bounds the accumulated rounding error after k iterations by O(k · 2^{-b}), where b is the bit precision, and shows that the bump height remains above the rounding threshold provided k = O(b). This supplies the missing explicit threshold and confirms that the family remains pairwise distinguishable, thereby restoring the exponential lower bound on m. revision: yes

  2. Referee: [Main theorem statement] The statement that the lower bound holds “unless the sampling budget grows exponentially with the size of the network parameters and architecture” is not accompanied by an explicit dependence of the required m on depth, width, or bit precision. A quantitative statement relating the iteration depth k needed for localization to the precision parameter would make the exponential blow-up precise.

    Authors: We accept that the current qualitative phrasing leaves the dependence implicit. The proof already tracks that the number of distinguishable bumps scales as roughly 2^{\Theta(k)} for iteration depth k (which is linear in network depth), so any algorithm achieving a rate better than Monte Carlo must use m \ge 2^{c k / p} samples. In the revision we will restate the main theorem with this explicit form, inserting the constants c that arise from the error bound above and relating k to width and total parameter count via the network architecture. This makes the exponential blow-up with depth, width, and bit precision fully quantitative. revision: yes

Circularity Check

0 steps flagged

No significant circularity; lower bound rests on explicit novel construction of iterated-tanh bumps

full rationale

The paper's central result is a lower bound showing that no adaptive randomized m-sample algorithm can exceed Monte Carlo rate O(m^{-1/p}) in L^p unless m grows exponentially with network size. This is derived from a novel construction of sharply localized bump functions realized by iterated tanh activations, which the abstract explicitly presents as original to this work. The finite-precision arithmetic model is inherited from the cited Berner-Grohs-Voigtländer 2023 paper (overlapping author), but the load-bearing step is the new bump family and its distinguishability properties under that model, not a reduction to a fitted parameter, self-definition, or unverified self-citation chain. No equation or claim reduces by construction to its own inputs; the derivation is self-contained via the explicit construction.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The claim rests on the finite-precision model and L^p approximation framework from the 2023 reference plus the validity of the new bump construction; no free parameters or invented entities are introduced in the abstract.

axioms (2)
  • domain assumption Finite-precision arithmetic model and L^p norm guarantees as defined in Berner, Grohs, Voigtländer (2023)
    The paper explicitly builds on this prior framework for the computational model.
  • standard math Standard properties of tanh and iterated compositions in real analysis
    Invoked to justify the bump-function construction.

pith-pipeline@v0.9.1-grok · 5671 in / 1297 out tokens · 20548 ms · 2026-06-27T13:58:41.326378+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    Tight exponential bounds for hyperbolic tangent,

    Y . J. Bagul, R. M. Dhaigude, C. Chesneau, and M. Kosti´c, “Tight exponential bounds for hyperbolic tangent,” Jordan Journal of Mathematics and Statistics, vol. 15, no. 4A, pp. 807–821, 2022

    2022

  2. [2]

    Universal approximation bounds for superpositions of a sigmoidal function,

    A. R. Barron, “Universal approximation bounds for superpositions of a sigmoidal function,”IEEE Transactions on Information theory, vol. 39, no. 3, pp. 930–945, 1993

    1993

  3. [3]

    Berner, P

    J. Berner, P. Grohs, and A. Jentzen, “Analysis of the generalization error: Empirical risk minimization over deep artificial neural networks overcomes the curse of dimensionality in the numerical approximation of black–scholes partial differential equations,”SIAM Journal on Mathematics of Data Science, vol. 2, no. 3, pp. 631–657, 2020

    2020

  4. [4]

    Learning relu networks to high uniform accuracy is intractable,

    J. Berner, P. Grohs, and F. V oigtlaender, “Learning relu networks to high uniform accuracy is intractable,” in Proceedings of the Eleventh International Conference on Learning Representations (ICLR), 2023. [Online]. Available:https://openreview.net/forum?id=nchvKfvNeX0

    2023

  5. [5]

    Optimal approximation with sparsely connected deep neural networks,

    H. Bolcskei, P. Grohs, G. Kutyniok, and P. Petersen, “Optimal approximation with sparsely connected deep neural networks,”SIAM Journal on Mathematics of Data Science, vol. 1, no. 1, pp. 8–45, 2019

    2019

  6. [6]

    Introduction to statistical learning theory,

    O. Bousquet, S. Boucheron, and G. Lugosi, “Introduction to statistical learning theory,” inSummer school on machine learning, Springer, 2003, pp. 169–207

    2003

  7. [7]

    On the properties of neural machine translation: Encoder–decoder approaches,

    K. Cho, B. Van Merriënboer, D. Bahdanau, and Y . Bengio, “On the properties of neural machine translation: Encoder–decoder approaches,” inProceedings of SSST-8, eighth workshop on syntax, semantics and structure in statistical translation, 2014, pp. 103–111

    2014

  8. [8]

    On the mathematical foundations of learning,

    F. Cucker and S. Smale, “On the mathematical foundations of learning,”Bulletin of the American mathematical society, vol. 39, no. 1, pp. 1–49, 2002

    2002

  9. [9]

    Non-euclidean contraction analysis of continuous-time neural networks,

    A. Davydov, A. V . Proskurnikov, and F. Bullo, “Non-euclidean contraction analysis of continuous-time neural networks,”IEEE Transactions on Automatic Control, vol. 70, no. 1, pp. 235–250, 2024

    2024

  10. [10]

    On the approximation of functions by tanh neural networks,

    T. De Ryck, S. Lanthaler, and S. Mishra, “On the approximation of functions by tanh neural networks,”Neural Networks, vol. 143, pp. 732–750, 2021

    2021

  11. [11]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M. -W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” inProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), 2019, pp. 4171–4186

    2019

  12. [12]

    Deep neural network approximation theory,

    D. Elbrächter, D. Perekrestenko, P. Grohs, and H. Bölcskei, “Deep neural network approximation theory,”IEEE Transactions on Information Theory, vol. 67, no. 5, pp. 2581–2623, 2021

    2021

  13. [13]

    Ern and J.-L

    A. Ern and J.-L. Guermond,Theory and practice of finite elements. Springer, 2004, vol. 159

    2004

  14. [14]

    Approximation error of sobolev regular functions with tanh neural networks: Theoretical impact on pinns,

    B. Girault, R. Emonet, A. Habrard, J. Patracone, and M. Sebban, “Approximation error of sobolev regular functions with tanh neural networks: Theoretical impact on pinns,” inJoint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer, 2024, pp. 266–282

    2024

  15. [15]

    Quantization,

    R. M. Gray and D. L. Neuhoff, “Quantization,”IEEE transactions on information theory, vol. 44, no. 6, pp. 2325– 2383, 2002

    2002

  16. [16]

    Grohs, F

    P. Grohs, F. Hornung, A. Jentzen, and P. V on Wurstemberger,A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black–Scholes partial differential equations. American Mathematical Society, 2023, vol. 284

    2023

  17. [17]

    Proof of the theory-to-practice gap in deep learning via sampling complexity bounds for neural network approximation spaces,

    P. Grohs and F. V oigtlaender, “Proof of the theory-to-practice gap in deep learning via sampling complexity bounds for neural network approximation spaces,”Foundations of Computational Mathematics, vol. 24, no. 4, pp. 1085–1143, 2024

    2024

  18. [18]

    Sobolev-type embeddings for neural network approximation spaces,

    P. Grohs and F. V oigtlaender, “Sobolev-type embeddings for neural network approximation spaces,”Constructive Approximation, vol. 57, no. 2, pp. 579–599, 2023

    2023

  19. [19]

    Gaussian Error Linear Units (GELUs)

    D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),”arXiv preprint arXiv:1606.08415, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  20. [20]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,”Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997. 29 APREPRINT- JUNE10, 2026

    1997

  21. [21]

    Robust implicit networks via non-euclidean contrac- tions,

    S. Jafarpour, A. Davydov, A. Proskurnikov, and F. Bullo, “Robust implicit networks via non-euclidean contrac- tions,”Advances in Neural Information Processing Systems, vol. 34, pp. 9857–9868, 2021

    2021

  22. [22]

    Risk minimization by median-of-means tournaments,

    G. Lugosi and S. Mendelson, “Risk minimization by median-of-means tournaments,”Journal of the European Mathematical Society, vol. 22, no. 3, pp. 925–965, 2019

    2019

  23. [23]

    Approximation theory of the mlp model in neural networks,

    A. Pinkus, “Approximation theory of the mlp model in neural networks,”Acta numerica, vol. 8, pp. 143–195, 1999

    1999

  24. [24]

    Impossibility of fast stable approximation of analytic functions from equispaced samples,

    R. B. Platte, L. N. Trefethen, and A. B. Kuijlaars, “Impossibility of fast stable approximation of analytic functions from equispaced samples,”SIAM review, vol. 53, no. 2, pp. 308–318, 2011

    2011

  25. [25]

    Improving language understanding by generative pre-training,

    A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al., “Improving language understanding by generative pre-training,” 2018

    2018

  26. [26]

    Physics-informed neural networks: A deep learning frame- work for solving forward and inverse problems involving nonlinear partial differential equations,

    M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics-informed neural networks: A deep learning frame- work for solving forward and inverse problems involving nonlinear partial differential equations,”Journal of Computational physics, vol. 378, pp. 686–707, 2019

    2019

  27. [27]

    Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network,

    A. Sherstinsky, “Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network,” Physica d: Nonlinear phenomena, vol. 404, p. 132 306, 2020. 30

    2020