Pith. sign in

REVIEW 4 major objections 4 minor 62 references

Standard Transformers Achieve the Minimax Rate in Nonparametric Regression with $C^{s,\lambda}$ Targets

T0 review · 4 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Standard Transformers reach the minimax-optimal rate for smooth nonparametric regression.

desk verdict Serious transformer approximation/regression paper with a real but fixable bug in the memorization step; the stress-test's domain objection does not hold, but the positional-encoding mismatch does. read the letter →

arxiv 2602.20555 v2 pith:BAU3CNHB submitted 2026-02-24 stat.ML cs.ITcs.LGmath.IT

Yanming Lai , Defeng Sun This is my paper · ORCID
classification stat.MLcs.ITcs.LGmath.IT MSC 41A2562G0868T07
keywords TransformerapproximationHölderspacesnonparametricregressionminimaxoptimalratesoftmaxattentionmemorizationcapacitycoveringnumbersLipschitzconstant
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to prove that a Transformer in its standard form—softmax self-attention, ReLU feedforward blocks, no task-specific redesign—can do two things. First, it can approximate any Hölder function C^{s,λ} on [0,1]^{d×n} entrywise in L^t to arbitrary precision, with a parameter count that improves as the smoothness γ=s+λ grows. Second, plugging that approximation into least-squares regression yields an estimator whose excess risk is O(m^{-2γ/(2γ+dn)}(log m)^2), matching the classical minimax lower bound up to log factors. A sympathetic reader would care because it gives a quantitative theoretical reason for why plain Transformers are expressive enough to compete with specialised nonparametric methods.

What carries the argument

The load-bearing mechanism is a cell-wise Taylor approximation: a discretization feedforward block maps each grid cell to its grid point; a memorization Transformer built on contextual mapping outputs the Taylor coefficients at those points; and ReLU feedforward networks evaluate monomials and accumulate coefficient×monomial products. The size tuple and dimension vector are the bookkeeping devices that state exactly which widths, depths, heads, and dimensions are needed. For regression, the minimax rate follows by bounding the covering number of the Transformer class through a Lipschitz estimate for the whole architecture.

What would settle it

For d=n=1, s=1 (a quadratic target), build the Transformer from Proposition 1 and evaluate it at a point X on the boundary of a grid cell, where X−X^{(j)} is negative and has magnitude (1−δ)/K; if the entrywise error is not O(ε), or if the monomial network from the lemma is tested on [−c,c] and its error grows with c, then the ε-approximation theorem and the m^{-2γ/(2γ+dn)} regression rate collapse.

Watch

Extended reading notes

Core claim

Central claim: standard Transformers approximate any Hölder C^{s,λ} function on [0,1]^{d×n} to precision ε using O(ε^{-dn/γ}) parameters and O(log 1/ε) depth, in L^t and L∞ (Theorems 1–2); and a least-squares estimator over such Transformers has excess risk O(m^{-2γ/(2γ+dn)}(log m)^2), the minimax rate (Theorem 3). The construction partitions the cube into grid cells, approximates the target by Taylor polynomials on each cell, and uses softmax attention as a memorization mechanism to retrieve the needed Taylor coefficients.

Load-bearing premise

The approximation theorem assumes that a monomial-approximation lemma proved for inputs in [0,1]^{dn} still gives O(ε) accuracy on the shifted, possibly negative cell coordinates (X−X^{(j)}) used in the Taylor expansion; the paper does not spell out the rescaling, and if that bound fails the cell-wise error budget does not close.

Editorial extensions

If this is right

  • Approximation error scales as ε^{-dn/γ}: smoother targets (larger γ=s+λ) are learned with fewer parameters, and the exponent reveals the effective dimension dn of the flattened token space.
  • In regression, Transformers attain the same rate as classical kernel, spline, and neural estimators for Hölder classes, up to log factors, under i.i.d. Gaussian noise and bounded responses.
  • Pointwise (L∞) approximation is possible but costs an exponential-in-dn factor in width and depth compared with L^t approximation.
  • The Lipschitz and covering-number bounds provide quantitative control over the whole Transformer class, not just individual instances, so the same tools can be reused for generalization and optimization questions.
  • The memorization result improves on earlier memorization constructions by removing bias terms from the attention layers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: The proof suggests softmax attention is being used as an addressing/memorization primitive; if that is the true role, the approximation scheme may extend to anisotropic or manifold-structured targets by replacing the grid with a more efficient covering.
  • Editorial: Because the effective dimension is dn (the flattened token count), the rate does not exploit sequential structure of tokens; a target with smoothness defined along the sequence axis might yield faster rates than dn suggests.
  • Editorial: A concrete testable extension is to instantiate the constructed Transformer for small d, n, s and check whether the empirical L² approximation error tracks ε^{-dn/γ}; plateauing at cell boundaries would pinpoint the monomial-domain issue.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper claims the first quantitative approximation rates for standard (softmax) Transformers approximating Hölder functions in C^{s,λ}([0,1]^{d×n}) under L^t norms, and uses these rates to show that a least-squares estimator over a Transformer class achieves the minimax optimal rate m^{-2γ/(2γ+dn)} in nonparametric regression. The proof constructs a Transformer cellwise from Taylor polynomials of the target, with grid-point memorization (Lemma 7) and ReLU FNN monomial approximation (Lemma 9) as building blocks, then applies empirical-process covering-number bounds. Intermediate results include memorization capacity and a Lipschitz/covering estimate for standard Transformers. The overall strategy is coherent, but as written several load-bearing steps are not justified.

Significance. If repaired, this would be a substantial contribution: it moves Transformer approximation theory from C^{0,λ} to general C^{s,λ} targets and provides the first minimax-optimal regression-rate proof for standard softmax Transformers without attention bias. The modular proof — memorization, parallelization, Lipschitz bounds, and covering-number estimates — contains ingredients of independent interest. The paper also gives unusually detailed size/dimension/weight specifications. However, the listed gaps are in the central derivation, so significance is conditional on those being fixed.

major comments (4)
  1. [§3.1, Step 3 (Eq. (9))] For X∈Ω_j, the proof sets \bar X = X-X^{(j)} and applies Lemma 9 directly. But the entries of \bar X lie in [0,(1-δ)/K]^{dn}, not [0,1]^{dn}; Lemma 9 is stated only on [0,1]^d. No rescaling or shift is described, and the induced changes in the monomial and FNN weights are not tracked in Steps 4–7. Inequality (9) is therefore unsupported. A repair such as z=K\bar X/(1-δ) with f_i(y)=K^{-|α_i|}g_i(Ky/(1-δ)) is plausible, but it is absent and the stated error and weight bounds do not follow as written.
  2. [§3.1, Step 2; Lemma 7] Lemma 7 defines the positional encoding as E=3r√d [1_{d×1},2_{d×1},…,n_{d×1}]. With r=√d this gives entries 3kd, whereas Step 2 invokes Lemma 7 on X^{(j)} + 1_{d×1}eBEB with eBEB=(3,6,…,3n), i.e. entries 3k. The proof of Lemma 7 itself in (14) uses ||E_{:,k}||≈3kr, which is consistent with the Step 2 encoding but not with the stated E (whose norm is 3krd). Thus the memorization claim used in Proposition 1 is not exactly covered by the lemma as stated; either the lemma’s E or the Step 2 encoding must be corrected.
  3. [§3.1, multi-index set and Step 6] The proof defines the Taylor index set as {α:∑α_{uv}≤s} but states its cardinality as C_{s+dn-1}^{dn-1}. In dn variables, the number of multi-indices with |α|≤s is C_{dn+s}^{dn}, while C_{dn+s-1}^{dn-1} counts only |α|=s. If the intended polynomial is of order s−1, the count should be C_{dn+s-1}^{dn}. As written, the sums in Step 6 run over a set that omits all lower-degree monomials, so the identification with the Taylor polynomial in (8) and the Step 6 triangle bound are invalid. This affects Proposition 1 and Theorems 1–2, although the asymptotic ε-exponents would likely survive a corrected count.
  4. [§3.1, Step 3; Lemma 9] The monomial approximation step does not cover the constant term or the low-smoothness cases. Lemma 9 requires \bar α=|α|≥1 to define its depth/width, but the Taylor polynomial includes α=0. Moreover the proof uses 3^{⌈log2(s−1)⌉−1}, which is undefined for s=0 or 1, while Theorems 1–2 claim the result for s∈N_{\ge0}. Thus C^{0,λ} and C^{1,λ} targets are not handled by the construction as written. A separate treatment of α=0 (identity/constant FNN) and of \bar α=1 is required.
minor comments (4)
  1. [§6, Proof of Proposition 2] The proof mixes the sample size notation: several displays use n where m is the sample size, e.g. the definition of Z_g and the sup over 1/n ∑_{i=1}^n. Please correct to m throughout.
  2. [§3.1 / §3.3] The phrase “s−1 order Taylor polynomial” is inconsistent with the stated index set |α|≤s and with the cardinality formula. Please clarify the intended order and align the notation.
  3. [§7, Lemma 29 proof] In the sensitivity bound for feedforward blocks, the sentence “From the derivation of Lemma 24” should refer to Lemma 28, not Lemma 24.
  4. [General] There are numerous typos (“recursviely”, “defiend”, “resepctively”, “ahieve”, “contruct”, “appromants”) that should be corrected in a revision.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the Transformer approximation and minimax regression results are derived from in-paper constructions and external FNN lemmas; self-citations are only contextual.

full rationale

Walking the derivation chain: Theorem 1 (and Theorem 2) is built in Proposition 1 from Taylor polynomials with standard remainder estimates (eq. 8), a Transformer memorization lemma (Lemma 7, proved in Section 4 from Lemmas 14–20), a Transformer-to-FNN conversion (Lemma 8, proved in Section 3.2), and a monomial approximation by ReLU FNNs (Lemma 9, proved in Section 3.3 from Lemma 13 and [57, Prop 3]). None of these inputs contain the conclusion: the width/depth/weight bounds are solved from the error budget ϵ, and the Taylor coefficients are coefficients of f, not fitted to the final regression estimator. Theorem 3 then is a standard empirical-risk decomposition (Proposition 2) over the Transformer class whose approximation error is supplied by Theorem 2 and whose metric entropy is supplied by Lemma 4 (proved in Section 7); the rate m^{-2γ/(2γ+dn)} is obtained by balancing ϵ against m, and the lower bound is external (Stone). There is no step where a parameter fitted to the target data is later called a prediction, no uniqueness theorem imported from the authors' prior work, and no ansatz adopted solely by self-citation. The self-citations [18,19] appear only in the introduction as context and are not used in any proof. The Proposition 1 Step 3 issue raised by the reader (Lemma 9 stated on [0,1]^{dn} while \bar X lives on a small cell) is a potential correctness gap in a proof, not a circularity: it does not identify the theorem's conclusion with an input by construction. Hence score 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. The 'size tuple' and 'dimension vector' are structural descriptors of the architecture, not postulated objects requiring independent evidence. All free parameters in the construction (K, δ, ε1, ε2) are analytical choices made to balance an error budget, not fitted to data. The core assumptions are standard mathematical tools plus the tokenwise separation condition on the grid inputs.

assumptions (5)
  • standard math Taylor's theorem with remainder for C^{s,λ} functions: on a cell of side 1/K, the s−1 order Taylor polynomial approximates f to O(K^{-(s+λ)})
    Invoked in Proposition 1 Step 6, eq. (8), to bound the polynomial approximation error on grid cells.
  • standard math Yarotsky's ReLU FNN approximation results for products and monomials (Lemmas 9, 10, 13)
    Used to build monomial approximations and multiplication within the Transformer's feedforward blocks; cited from [57] and adapted.
  • standard math Empirical process theory: Bernstein's inequality, Borell-Sudakov-Tsirelson concentration, and Dudley entropy integral (Section 6)
    Underpins Proposition 2, which converts covering-number bounds into excess-risk bounds.
  • domain assumption Tokenwise (r,φ)-separatedness of the grid inputs in the memorization lemma holds with r=√d and φ=1/K
    Grid points in {0,...,K−1}^{d×n}/K are distinct and any two distinct columns are either equal or differ by at least 1/K in Euclidean distance, satisfying Definition 2. This is necessary for Lemma 7.
  • domain assumption The 'standard Transformer' architecture as defined in Section 1.2, with no bias terms in the self-attention score computation
    The results are specific to this architecture; prior memorization results [25] allowed attention biases, and removing them is part of the claimed contribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Standard Transformers Achieve the Minimax Rate in Nonparametric Regression with $C^{s,\lambda}$ Targets." pith.science (2026). https://pith.science/paper/BAU3CNHB

@misc{pith2026260220555,
  author       = {Pith},
  title        = {Pith review of: Standard Transformers Achieve the Minimax Rate in Nonparametric Regression with $C^s,\lambda$ Targets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BAU3CNHB}},
  note         = {Machine review of arXiv:2602.20555}
}
abstract

The tremendous success of Transformer models in fields such as large language models and computer vision necessitates a rigorous theoretical investigation. To the best of our knowledge, this paper is the first work proving that standard Transformers can approximate H\"older functions $ C^{s,\lambda}\left([0,1]^{d\times n}\right) $$ (s\in\mathbb{N}_{\geq0},0<\lambda\leq1) $ under the $L^t$ distance ($t \in [1, \infty]$) with arbitrary precision. Building upon this approximation result, we demonstrate that standard Transformers achieve the minimax optimal rate in nonparametric regression for H\"older target functions. It is worth mentioning that, by introducing two metrics: the size tuple and the dimension vector, we provide a fine-grained characterization of Transformer structures, which facilitates future research on the generalization and optimization errors of Transformers with different structures. As intermediate results, we also derive the upper bounds for the Lipschitz constant of standard Transformers and their memorization capacity, which may be of independent interest. These findings provide theoretical justification for the powerful capabilities of Transformer models.

Figures

Figures reproduced from arXiv: 2602.20555 by the authors.

Figure 1
Figure 1. Illustration of the proof process In the proof of Proposition 1, we also require the following two trivial lemmas, which state that both the feedforward block and the self-attention layer can realize the identity mapping. We may sometimes use these two lemmas without explicit mention, particularly when we employ Lemma 3 to parallel two feedforward blocks of different depths. Lemma 11. There exists a ReLU FNN functio… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 5 linked inside Pith

  1. [1]

    Pearson, 2 edition, 1974

    Tom Apostol.Mathematical Analysis. Pearson, 2 edition, 1974

    1974

  2. [2]

    Low-rank bottleneck in multi-head attention models

    Srinadh Bhojanapalli, Chulhee Yun, Ankit Singh Rawat, Sashank Reddi, and San- jiv Kumar. Low-rank bottleneck in multi-head attention models. InInternational conference on machine learning, pages 864–873. PMLR, 2020

    2020

  3. [3]

    Minshuo Chen, Haoming Jiang, Wenjing Liao, and Tuo Zhao. Nonparametric re- gression on low-dimensional manifolds using deep relu networks: Function approxi- mation and statistical recovery.Information and Inference: A Journal of the IMA, 11(4):1203–1253, 2022

    2022

  4. [4]

    A unified framework for establishing the universal approximation of transformer-type architectures

    Jingpu Cheng, Ting Lin, Zuowei Shen, and Qianxiao Li. A unified framework for establishing the universal approximation of transformer-type architectures. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  5. [5]

    Efficient and minimax-optimal in-context nonparametric regression with transformers.arXiv preprint arXiv:2601.15014, 2026

    Michelle Ching, Ioana Popescu, Nico Smith, Tianyi Ma, William G Underwood, and Richard J Samworth. Efficient and minimax-optimal in-context nonparametric regression with transformers.arXiv preprint arXiv:2601.15014, 2026

    arXiv 2026

  6. [6]

    Bert: Pre- training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. InProceed- ings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019

    2019

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representations, 2021

    2021

  8. [8]

    Inductive biases and variable creation in self-attention mechanisms

    Benjamin L Edelman, Surbhi Goel, Sham Kakade, and Cyril Zhang. Inductive biases and variable creation in self-attention mechanisms. InInternational Conference on Machine Learning, pages 5793–5831. PMLR, 2022. 54

    2022

Show all 62 references
  1. [9]

    How do noise tails impact on deep relu networks?The Annals of Statistics, 52(4):1845–1871, 2024

    Jianqing Fan, Yihong Gu, and Wen-Xin Zhou. How do noise tails impact on deep relu networks?The Annals of Statistics, 52(4):1845–1871, 2024

    2024

  2. [10]

    Deep neural networks for esti- mation and inference.Econometrica, 89(1):181–213, 2021

    Max H Farrell, Tengyuan Liang, and Sanjog Misra. Deep neural networks for esti- mation and inference.Econometrica, 89(1):181–213, 2021

    2021

  3. [11]

    Cambridge university press, 2021

    Evarist Gin´ e and Richard Nickl.Mathematical foundations of infinite-dimensional statistical models. Cambridge university press, 2021

    2021

  4. [12]

    Approximation rates for neural networks with encodable weights in smoothness spaces.Neural Networks, 134:107–130, 2021

    Ingo G¨ uhring and Mones Raslan. Approximation rates for neural networks with encodable weights in smoothness spaces.Neural Networks, 134:107–130, 2021

    2021

  5. [13]

    On the rate of convergence of a classifier based on a transformer encoder.IEEE Transactions on Information Theory, 68(12):8139–8155, 2022

    Iryna Gurevych, Michael Kohler, and G¨ ozde G¨ ul S ¸ahin. On the rate of convergence of a classifier based on a transformer encoder.IEEE Transactions on Information Theory, 68(12):8139–8155, 2022

    2022

  6. [14]

    Understanding scaling laws with statisti- cal and approximation theory for transformer neural networks on intrinsically low- dimensional data

    Alexander Havrilla and Wenjing Liao. Understanding scaling laws with statisti- cal and approximation theory for transformer neural networks on intrinsically low- dimensional data. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  7. [15]

    Minimal width for universal property of deep rnn.Journal of Machine Learning Research, 24(121):1–41, 2023

    Chang hoon Song, Geonho Hwang, Jun ho Lee, and Myungjoo Kang. Minimal width for universal property of deep rnn.Journal of Machine Learning Research, 24(121):1–41, 2023

    2023

  8. [16]

    Universal approximation with softmax attention.arXiv preprint arXiv:2504.15956, 2025

    Jerry Yao-Chieh Hu, Hude Liu, Hong-Yu Chen, Weimin Wu, and Han Liu. Universal approximation with softmax attention.arXiv preprint arXiv:2504.15956, 2025

    2025

  9. [17]

    Approximation rate of the transformer architecture for sequence modeling

    Haotian Jiang and Qianxiao Li. Approximation rate of the transformer architecture for sequence modeling. InThe Thirty-eighth Annual Conference on Neural Informa- tion Processing Systems, 2024

    2024

  10. [18]

    Approxima- tion bounds for transformer networks with application to regression.arXiv preprint arXiv:2504.12175, 2025

    Yuling Jiao, Yanming Lai, Defeng Sun, Yang Wang, and Bokai Yan. Approxima- tion bounds for transformer networks with application to regression.arXiv preprint arXiv:2504.12175, 2025

    2025 arXiv

  11. [19]

    Transformers can overcome the curse of dimensionality: A theoretical study from an approximation perspective

    Yuling Jiao, Yanming Lai, Yang Wang, and Bokai Yan. Transformers can overcome the curse of dimensionality: A theoretical study from an approximation perspective. arXiv preprint arXiv:2504.13558, 2025

    2025 arXiv

  12. [20]

    Deep nonparametric regression on approximate manifolds: Nonasymptotic error bounds with polynomial prefactors.The Annals of Statistics, 51(2):691–716, 2023

    Yuling Jiao, Guohao Shen, Yuanyuan Lin, and Jian Huang. Deep nonparametric regression on approximate manifolds: Nonasymptotic error bounds with polynomial prefactors.The Annals of Statistics, 51(2):691–716, 2023

    2023

  13. [21]

    Approximation bounds for recurrent neural networks with application to regression.arXiv preprint arXiv:2409.05577, 2024

    Yuling Jiao, Yang Wang, and Bokai Yan. Approximation bounds for recurrent neural networks with application to regression.arXiv preprint arXiv:2409.05577, 2024

    2024

  14. [22]

    Are transformers with one layer self-attention using low-rank weight matrices universal approximators? InThe Twelfth International Conference on Learning Representations, 2024

    Tokio Kajitsuka and Issei Sato. Are transformers with one layer self-attention using low-rank weight matrices universal approximators? InThe Twelfth International Conference on Learning Representations, 2024. 55

    2024

  15. [23]

    On the optimal memorization capacity of trans- formers

    Tokio Kajitsuka and Issei Sato. On the optimal memorization capacity of trans- formers. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  16. [24]

    The lipschitz constant of self-attention

    Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention. InInternational Conference on Machine Learning, pages 5562–5571. PMLR, 2021

    2021

  17. [25]

    Provable memorization capac- ity of transformers

    Junghwan Kim, Michelle Kim, and Barzan Mozafari. Provable memorization capac- ity of transformers. InThe Eleventh International Conference on Learning Repre- sentations, 2023

    2023

  18. [26]

    Transformers are minimax optimal non- parametric in-context learners

    Juno Kim, Tai Nakamaki, and Taiji Suzuki. Transformers are minimax optimal non- parametric in-context learners. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  19. [27]

    On the rate of convergence of fully connected deep neural network regression estimates.The Annals of Statistics, 49(4):2231–2249, 2021

    Michael Kohler and Sophie Langer. On the rate of convergence of fully connected deep neural network regression estimates.The Annals of Statistics, 49(4):2231–2249, 2021

    2021

  20. [28]

    Univer- sal approximation under constraints is possible with transformers

    Anastasis Kratsios, Behnoosh Zamanlooy, Tianlin Liu, and Ivan Dokmani´ c. Univer- sal approximation under constraints is possible with transformers. InInternational Conference on Learning Representations, 2022

    2022

  21. [29]

    Approximation and optimization theory for linear continuous-time recurrent neural networks.Journal of Machine Learning Research, 23(42):1–85, 2022

    Zhong Li, Jiequn Han, Qianxiao Li, et al. Approximation and optimization theory for linear continuous-time recurrent neural networks.Journal of Machine Learning Research, 23(42):1–85, 2022

    2022

  22. [30]

    Generalization analysis of transformers in distribu- tion regression.Neural Computation, 37(2):260–293, 2025

    Peilin Liu and Ding-Xuan Zhou. Generalization analysis of transformers in distribu- tion regression.Neural Computation, 37(2):260–293, 2025

    2025

  23. [31]

    Deep network approxi- mation for smooth functions.SIAM Journal on Mathematical Analysis, 53(5):5465– 5506, 2021

    Jianfeng Lu, Zuowei Shen, Haizhao Yang, and Shijun Zhang. Deep network approxi- mation for smooth functions.SIAM Journal on Mathematical Analysis, 53(5):5465– 5506, 2021

    2021

  24. [32]

    Upper and lower memory capacity bounds of transformers for next- token prediction.arXiv preprint arXiv:2405.13718, 2024

    Liam Madden. Upper and lower memory capacity bounds of transformers for next- token prediction.arXiv preprint arXiv:2405.13718, 2024

    2024

  25. [33]

    Memorization capacity of multi-head attention in transformers

    Sadegh Mahdavi, Renjie Liao, and Christos Thrampoulidis. Memorization capacity of multi-head attention in transformers. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  26. [34]

    Rates of approximation by relu shallow neural networks.Journal of Complexity, 79:101784, 2023

    Tong Mao and Ding-Xuan Zhou. Rates of approximation by relu shallow neural networks.Journal of Complexity, 79:101784, 2023

    2023

  27. [35]

    Adaptive approximation and generalization of deep neural network with intrinsic dimensionality.Journal of Machine Learning Research, 21(174):1–38, 2020

    Ryumei Nakada and Masaaki Imaizumi. Adaptive approximation and generalization of deep neural network with intrinsic dimensionality.Journal of Machine Learning Research, 21(174):1–38, 2020

    2020

  28. [36]

    Provable memorization via deep neural networks using sub-linear parameters

    Sejun Park, Jaeho Lee, Chulhee Yun, and Jinwoo Shin. Provable memorization via deep neural networks using sub-linear parameters. InConference on learning theory, pages 3627–3661. PMLR, 2021. 56

    2021

  29. [37]

    Equivalence of approximation by convolu- tional neural networks and fully-connected networks.Proceedings of the American Mathematical Society, 148(4):1567–1581, 2020

    Philipp Petersen and Felix Voigtlaender. Equivalence of approximation by convolu- tional neural networks and fully-connected networks.Proceedings of the American Mathematical Society, 148(4):1567–1581, 2020

    2020

  30. [38]

    Representational strengths and limitations of transformers

    Clayton Sanford, Daniel Hsu, and Matus Telgarsky. Representational strengths and limitations of transformers. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  31. [39]

    Nonparametric regression using deep neural net- works with relu activation function.Annals of statistics, 48(4):1875–1897, 2020

    Anselm Johannes Schmidt-Hieber. Nonparametric regression using deep neural net- works with relu activation function.Annals of statistics, 48(4):1875–1897, 2020

    2020

  32. [40]

    The kolmogorov–arnold representation theorem revisited

    Johannes Schmidt-Hieber. The kolmogorov–arnold representation theorem revisited. Neural networks, 137:119–126, 2021

    2021

  33. [41]

    Understanding in- context learning on structured manifolds: Bridging attention to kernel methods

    Zhaiming Shen, Alexander Hsu, Rongjie Lai, and Wenjing Liao. Understanding in- context learning on structured manifolds: Bridging attention to kernel methods. In The Fourteenth International Conference on Learning Representations, 2026

    2026

  34. [42]

    Deep network approximation char- acterized by number of neurons.Communications in Computational Physics, 28(5), 2020

    Zuowei Shen, Haizhao Yang, and Shijun Zhang. Deep network approximation char- acterized by number of neurons.Communications in Computational Physics, 28(5), 2020

    2020

  35. [43]

    Optimal approximation rate of relu networks in terms of width and depth.Journal de Math´ ematiques Pures et Appliqu´ ees, 157:101–135, 2022

    Zuowei Shen, Haizhao Yang, and Shijun Zhang. Optimal approximation rate of relu networks in terms of width and depth.Journal de Math´ ematiques Pures et Appliqu´ ees, 157:101–135, 2022

    2022

  36. [44]

    Optimal approximation rates for deep relu neural networks on sobolev and besov spaces.Journal of Machine Learning Research, 24(357):1–52, 2023

    Jonathan W Siegel. Optimal approximation rates for deep relu neural networks on sobolev and besov spaces.Journal of Machine Learning Research, 24(357):1–52, 2023

    2023

  37. [45]

    Optimal rates of convergence for nonparametric estimators.The annals of Statistics, pages 1348–1360, 1980

    Charles J Stone. Optimal rates of convergence for nonparametric estimators.The annals of Statistics, pages 1348–1360, 1980

    1980

  38. [46]

    Adaptivity of deep reLU network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality

    Taiji Suzuki. Adaptivity of deep reLU network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality. InInternational Conference on Learning Representations, 2019

    2019

  39. [47]

    Approximation and estimation ability of trans- formers for sequence-to-sequence functions with infinite dimensional input

    Shokichi Takakura and Taiji Suzuki. Approximation and estimation ability of trans- formers for sequence-to-sequence functions with infinite dimensional input. InInter- national Conference on Machine Learning, pages 33416–33447. PMLR, 2023

    2023

  40. [48]

    Approximation of permutation invariant polynomials by transformers: Efficient construction in column-size.arXiv preprint arXiv:2502.11467, 2025

    Naoki Takeshita and Masaaki Imaizumi. Approximation of permutation invariant polynomials by transformers: Efficient construction in column-size.arXiv preprint arXiv:2502.11467, 2025

    2025 arXiv

  41. [49]

    Weak convergence

    Aad W Van Der Vaart and Jon A Wellner. Weak convergence. InWeak convergence and empirical processes: with applications to statistics, pages 16–28. Springer, 1996

    1996

  42. [50]

    On the optimal memorization power of reLU neural networks

    Gal Vardi, Gilad Yehudai, and Ohad Shamir. On the optimal memorization power of reLU neural networks. InInternational Conference on Learning Representations, 2022. 57

    2022

  43. [51]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  44. [52]

    Prompt tuning transformers for data memorization

    Haiyu Wang and Yuanyuan Lin. Prompt tuning transformers for data memorization. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  45. [53]

    Zixuan Wang, Stanley Wei, Daniel Hsu, and Jason D. Lee. Transformers provably learn sparse token selection while fully-connected nets cannot. InForty-first Inter- national Conference on Machine Learning, 2024

    2024

  46. [54]

    Statistically meaningful approximation: a case study on approximating turing machines with transformers

    Colin Wei, Yining Chen, and Tengyu Ma. Statistically meaningful approximation: a case study on approximating turing machines with transformers. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors,Advances in Neural Information Processing Systems, 2022

    2022

  47. [55]

    On the optimal approximation of sobolev and besov functions using deep relu neural networks.arXiv preprint arXiv:2409.00901, 2024

    Yunfei Yang. On the optimal approximation of sobolev and besov functions using deep relu neural networks.arXiv preprint arXiv:2409.00901, 2024

    2024 arXiv

  48. [56]

    Nonparametric regression using over- parameterized shallow relu neural networks.Journal of Machine Learning Research, 25(165):1–35, 2024

    Yunfei Yang and Ding-Xuan Zhou. Nonparametric regression using over- parameterized shallow relu neural networks.Journal of Machine Learning Research, 25(165):1–35, 2024

    2024

  49. [57]

    Error bounds for approximations with deep relu networks.Neural networks, 94:103–114, 2017

    Dmitry Yarotsky. Error bounds for approximations with deep relu networks.Neural networks, 94:103–114, 2017

    2017

  50. [58]

    Optimal approximation of continuous functions by very deep relu networks

    Dmitry Yarotsky. Optimal approximation of continuous functions by very deep relu networks. InConference on learning theory, pages 639–649. PMLR, 2018

    2018

  51. [59]

    Are transformers universal approximators of sequence-to-sequence func- tions? InInternational Conference on Learning Representations, 2020

    Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence func- tions? InInternational Conference on Learning Representations, 2020

    2020

  52. [60]

    O (n) connections are expressive enough: Universal approximability of sparse transformers.Advances in Neural Information Processing Systems, 33:13783–13794, 2020

    Chulhee Yun, Yin-Wen Chang, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. O (n) connections are expressive enough: Universal approximability of sparse transformers.Advances in Neural Information Processing Systems, 33:13783–13794, 2020

    2020

  53. [61]

    Theory of deep convolutional neural networks: Downsampling

    Ding-Xuan Zhou. Theory of deep convolutional neural networks: Downsampling. Neural Networks, 124:319–327, 2020

    2020

  54. [62]

    Universality of deep convolutional neural networks.Applied and computational harmonic analysis, 48(2):787–794, 2020

    Ding-Xuan Zhou. Universality of deep convolutional neural networks.Applied and computational harmonic analysis, 48(2):787–794, 2020. 58

    2020

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.