Pith. sign in

REVIEW 5 major objections 6 minor 40 references

Transformers Are Universally Consistent

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Softmax-attention transformers are universally consistent regressors over the Poincaré ball: the empirical risk minimizer converges almost surely to the Bayes-optimal predictor, at rate $\mathcal{O}(t^{-1/2d})$.

desk verdict The paper claims a significant result, but the proof chain breaks at multiple load-bearing points, and the architecture as defined cannot even compose blocks. read the letter →

arxiv 2505.24531 v1 pith:253XK2MU submitted 2025-05-30 cs.LG

Sagar Ghosh , Kushal Bose , Swagatam Das This is my paper
classification cs.LG MSC 68T0762G0868Q32
keywords hyperbolicspacePoincaréballtransformeruniversalconsistencystatisticalapproximationsamplecomplexityempiricalriskminimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a transformer with ordinary softmax attention is a statistically reliable learner: given enough input tokens, the model selected by minimizing training error provably converges to the best possible predictor, the regression function that minimizes expected squared error, for any data distribution with finite second moment. The argument is carried out in hyperbolic space, with tokens living on the Poincaré ball, and ordinary Euclidean transformers fall out as the curvature tends to zero. The authors also supply a quantitative rate: the excess error decays as $\mathcal{O}(t^{-1/2d})$, where $t$ is the number of tokens and $d$ the embedding dimension. If the claim holds, attention-based sequence models are not merely expressive but asymptotically optimal estimators, meeting a classical standard of statistical learning theory.

What carries the argument

The object carrying the argument is the HyT block, a transformer layer defined on the Poincaré ball $\mathbb{D}_c^d$: hyperbolic attention (HypAttn) maps tokens to the tangent space at the origin via $\log_0^c$, runs softmax attention there, and returns results to the ball through the exponential map, with Möbius addition supplying the residual connection; hyperbolic feed-forward (HypFF) does the same for the two-layer ReLU network. Two quantitative instruments do the heavy lifting. First, capacity control: Lemma 5 bounds the pseudo-dimension of the truncated class $\pi_M(T_{P,H}^{h,s,r})$ by the parameter count times the log of the neuron count, and Lemma 6 converts that into a covering-number bound through a hyperbolic analogue of the standard pseudo-dimension packing bound (Theorem V.2). Second, a concentration inequality (Lemma 7) bounds the probability that empirical and generalization error differ uniformly over the class in terms of those covering numbers. Finally, Lemma 8 transfers the Euclidean universal-approximation theorem to the hyperbolic setting by conjugating with the exponential map, and Theorem V.1 splits the excess risk into eight terms that each vanish under the stated growth conditions.

What would settle it

A decisive check would take a regression problem inside the Poincaré disk whose Bayes-optimal function is known in closed form, train the $(2,1,4)$ HyT with steadily increasing token counts, and test whether the excess risk actually vanishes almost surely at the promised $\mathcal{O}(t^{-1/2d})$ rate; the theorem falls if any single distribution makes the error plateau, or if the asserted re-expression of a Euclidean transformer as HyT blocks cannot be realized inside the ball.

Watch

Extended reading notes

Core claim

The paper's central claim is Theorem V.1: the truncated empirical risk minimizer $\pi_M f_D^{2,1,4}$ over the HyT function class $T_{P,H}^{2,1,4}$, transformers with two single-unit attention heads and a four-unit feed-forward layer operating on tokens in the Poincaré ball, is strongly universally consistent. Concretely, for any Borel distribution on inputs and outputs for which $\log_0^c(Y)$ is square-integrable, the expected squared error of the trained model converges almost surely to the minimum achievable value, attained by the hyperbolic regression function $f_\rho(x) = \int \log_0^c(y)\,d\rho(y \mid x)$, as the token count $t$ grows. The convergence happens at rate $\mathcal{O}(t^{-1/2d})$, with $d$ the embedding dimension, and the ordinary Euclidean transformer is recovered as the curvature parameter $c$ tends to zero. The proof controls the gap between empirical and generalization error through capacity estimates, a pseudo-dimension bound for the truncated class and a resulting metric-entropy bound, and closes the remaining gap with a universal-approximation step.

Load-bearing premise

The proof depends on a single unproved transfer step: that a Euclidean transformer computing on the tangent-space coordinates of the data can be rewritten exactly as a chain of hyperbolic transformer blocks, so that the known Euclidean approximation guarantee carries over to the hyperbolic architecture; if that composition cannot be realized, the approximation step of the main proof collapses.

Editorial extensions

If this is right

  • Asymptotically, no other sequence-to-sequence regressor can beat a trained HyT on expected squared error, because the empirical risk minimizer converges almost surely to the Bayes-optimal function as tokens grow.
  • A minimal architecture suffices for the guarantee: consistency holds already at $(h,s,r) = (2,1,4)$, namely two single-unit attention heads and a four-unit feed-forward layer.
  • The Euclidean transformer is a limiting case: as the Poincaré curvature $c \to 0$, the hyperbolic theorem reduces to a consistency guarantee for ordinary transformers on data manifolds of intrinsic dimension $d$.
  • The rate $\mathcal{O}(t^{-1/2d})$ implies the token budget needed for a target error level grows exponentially in the embedding dimension, a concrete scaling rule for data-hungry training of sequence models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The claimed rate $\mathcal{O}(t^{-1/2d})$ coincides with the classical minimax rate for $d$-dimensional nonparametric regression, so the paper's result implicitly says transformers are statistically optimal estimators; spelling out that comparison is left undone.
  • Because the whole construction runs through log/exp maps at the origin, the same consistency argument would plausibly extend to other manifolds with well-defined tangent-space maps, such as the sphere, whenever a Euclidean approximation theorem can be transported along the exponential map.
  • The reported empirical check compares test RMSE against a $t^{-1/257}$ reference curve on question-answering data with an unknown target function, so it cannot directly confirm the claimed rate; a synthetic benchmark with a known Bayes-optimal function and controlled intrinsic dimension would test the bound directly.
  • If the rate is tight, transformer scaling laws would follow from statistics rather than optimization alone: error should fall as a power law in the token budget, with the exponent set by the intrinsic dimension of the target function, offering a principled way to size models and datasets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript introduces a Hyperbolic Transformer (HyT) on the Poincaré ball, with hyperbolic attention (HypAttn) and hyperbolic feed-forward (HypFF) layers, and claims that for h=2, s=1, r=4 the empirical risk minimizer over the class T_{P,H}^{2,1,4} is strongly universally consistent for regression of log_0^c(Y) on X, with a convergence rate O(t^{-1/2d}). The proof proceeds through pseudo-dimension and covering-number estimates (Lemmas 4-6), a concentration inequality (Lemma 7), a universal approximation transfer (Lemma 8), and an eight-term decomposition in Theorem V.1. Experiments on question-answering datasets compare different curvatures.

Significance. If the main theorem were correct, this would be a notable extension of distribution-free consistency results to transformers with softmax attention in hyperbolic geometry. The paper does not, however, provide a sound proof: the class definition does not admit the compositions used in the main theorem, Lemma 7 is circular, Lemma 8's transfer from Euclidean to hyperbolic transformers is unproved, and the algebraic decomposition in Theorem V.1 does not yield the stated inequality. Because the central claim rests on these load-bearing points, I cannot regard the theoretical result as established. The experimental section is auxiliary and reports single un-replicated runs, so it does not compensate for the theoretical gaps.

major comments (5)
  1. [Section IV, Eqs. (IV.1)-(IV.3)] The class T_H^{h,s,r} is defined as finite compositions of HyT blocks t_H^{h,s,r}: D_c^{d×t} -> R^{d×t}. But HypFF in Eq. (IV.2) applies log_0^c to the final Möbius sum, so its output lies in R^{d×t}, not in D_c^{d×t}. Consequently, a composition of two or more blocks is undefined: the second block would require an input in D_c^{d×t} but receive a Euclidean matrix. Under the formal definitions, T_{P,H}^{h,s,r} contains only single-block functions, which contradicts the intended analogy with the multi-layer Euclidean transformer class and invalidates the use of Lemma 8. If one repairs the codomain to D_c^{d×t} to allow depth, then the parameter count n_param in Section V.A and the pseudo-dimension bound in Lemma 5 must depend on depth; neither is supplied. This issue alone breaks the approximation leg (term A7) of Theorem V.1.
  2. [Lemma 7] The proof of Lemma 7 is circular. It states 'By putting α=β=1 and ϵ=t^{-θ} in Lemma 7 we get ...', referring to the very lemma under proof, and concludes with 'Combining this with the Strong Law of Large Numbers ... completing the proof of Lemma 7.' The concentration inequality invoked is taken from Lemma 17 of [33] without derivation or verification that its assumptions apply to the HyT class; [33] is the authors' own prior preprint. Since Lemma 7 is the source of the almost-sure convergence of terms A2 and A4 in Theorem V.1, this is a load-bearing gap.
  3. [Proof of Theorem V.1] The eight-term decomposition is algebraically inconsistent with the left-hand side. Summing the displayed A_1,...,A_8 gives, with the notation used in the proof, -E(π_M f_D) + (1+ε)E_{π_M}(π_M f_D) - E(fρ), not E(π_M f_D) - E(fρ). Additionally, A_8 = ((1+ε)^2-1)E(fρ) is a fixed positive term for any ε>0 and does not vanish as t→∞; the sentence 'which definitely goes to 0 as ε>0 is arbitrary' attempts to let ε→0 after the limit has been taken, which is not valid. The claim that each A_i→0 therefore does not establish the theorem.
  4. [Lemma 8] The statement that f:=t∘log_0^c belongs to T_{P,H}^{2,1,4} for an arbitrary Euclidean transformer t∈T_P^{2,1,4} is asserted without proof. A Euclidean transformer is built from linear maps, softmax attention, ReLU feed-forward layers, and residual connections, whereas HyT blocks use Möbius addition, exponential/logarithmic maps, and attention on log-space representations; no simulation argument shows that the former can be realized by the latter. Moreover, the change-of-variables estimate in Eq. (V.11) bounds the integral by sup_{x∈R}||log_0^c(x)|| times the Euclidean integral; the correct factor is the Jacobian determinant of exp_0^c, not the sup norm of log_0^c. Hence the universal approximation property needed for term A7 is not established.
  5. [Lemmas 5 and 6] Lemma 5 applies Theorem 7 of [36] and Theorem 14.1 of [37], which are pseudo-dimension bounds for piecewise-polynomial or piecewise-linear networks, to the HyT class containing softmax, tanh, exp/log, and Möbius operations. No argument is given that the required activation structure and parameterization hold. If the composition issue is repaired by allowing depth L, n_param must be multiplied by L and the covering-number estimates in Lemma 6 change accordingly; the proof as written only counts a single block. Thus the capacity estimates behind terms A2 and A6 are also unsupported.
minor comments (6)
  1. [Throughout] The notation for truncation is inconsistent: M denotes both the truncation threshold in π_M and, via M' = c^{-1/2} tanh^{-1}(M√c), the effective radius in Lemma 7; this makes the bounds hard to follow.
  2. [Section VI] The text says the experiments cover 'the five datasets previously introduced' but only three datasets (SQuAD, BoolQ, TweetQA) are described, and curvature values 100 and 10.0 are both used for the same setting without explanation.
  3. [Section VI.E] The empirical sample-complexity plot is compared with y=t^{-1/257}, whereas Remark V.5 with d=128 gives t^{-1/256}; the discrepancy is not discussed.
  4. [Lemma 1] Lemma 1 already contains a complete conditional-expectation proof that g(x)=E[log_0^c(Y)|X=x] minimizes the L2 error; the concluding citation to Lemma 8 of [33] is unnecessary and imports an unverified external result.
  5. [Remark V.1] The assertion that Euclidean transformers follow by 'lim c→0' is not a derivation: the Poincaré ball maps, Möbius operations, and the conditions of Theorem V.1 involve 1/√c and tanh^{-1}, so a limiting argument is needed and none is supplied.
  6. [Typos] There are multiple typographical errors, including 'Reimannian', 'emperical', 'diffemorphism', 'considerbale', and inconsistent 'Möbius' typesetting; these should be corrected.

Circularity Check

3 steps flagged · score 8.0 of 10

Self-referential Lemma 7 makes the core concentration bound circular, and Lemma 8 asserts the required HyT approximation by fiat, so the main consistency proof rests on assumed rather than derived inputs.

  1. other [Section V.B, proof of Lemma 7 (after Eqs. (V.9)–(V.10))]
    "By putting α=β=1 and ϵ=t^{-θ} in Lemma 7 we get, (E_{π_M}(π_M f_D^{h,s,r}) − E_{π_M}(f_ρ)) − (E_{π_M,D}(π_M f_D^{h,s,r}) − E_{π_M,D}(π_M f_ρ)) ≤ 8d(1/√c tanh^{-1}(M√c))^2 t^{-θ}"

    The proof of Lemma 7, which is supposed to establish concentration inequality (V.10), invokes “Lemma 7” itself with specific parameter choices. The desired bound is therefore assumed rather than derived. This is load-bearing because Lemma 7 supplies term A2 in the proof of Theorem V.1; without it the almost-sure convergence of the empirical error to the generalization error is not established.

  2. self citation load bearing [Section V.B, before Lemma 7]
    "As in [33], we have to use a version of concentration inequality, which is a slightly more generalized version of Lemma 17 [33], which is in turn a version of Lemma 11.4 [34]."

    The concentration inequality at the heart of Lemma 7 is imported from the authors' own prior work [33] rather than proved here. Since [33] is a preprint by the same authors and is not independently verified in this paper, and since the paper's proof of Lemma 7 then invokes Lemma 7 itself, the core estimate is not independent of the paper's own prior results. This self-citation chain is load-bearing because the convergence of term A2 in Theorem V.1 depends on it.

1 more flagged steps
  1. other [Section V.C, proof of Lemma 8]
    "Define f := t ∘ log_0^c. Then f ∈ T_{P,H}^{2,1,4}."

    Lemma 8 must prove that there exists a HyT network in T_{P,H}^{2,1,4} approximating g. The proof imports a Euclidean transformer t from Yun et al. [38] and simply declares that t ∘ log_0^c belongs to the hyperbolic class. That membership is exactly the property to be established; it is not derived from the HyT block definitions (IV.1)–(IV.3), which use Möbius addition, exp/log maps, and Möbius residuals. Thus the universal approximation premise A7 is assumed rather than proved.

full rationale

The paper's main universal-consistency theorem rests on two load-bearing estimates: the concentration bound (Lemma 7, used for term A2) and the universal approximation transfer (Lemma 8, used for term A7). The proof of Lemma 7 is directly self-referential: after stating the inequality, it “proves” the bound by substituting α=β=1 and ϵ=t^{-θ} “in Lemma 7”, i.e., it assumes the lemma it is supposed to establish. The same lemma is introduced by citing the authors' own prior preprint [33] for the concentration inequality, so the core estimate is not independently derived in this paper. Lemma 1 also redundantly cites Lemma 8 of [33] after already deriving the identity, further evidencing reliance on the same authors' prior work. Separately, Lemma 8 constructs an approximator f := t ∘ log_0^c and asserts f ∈ T_{P,H}^{2,1,4} without deriving membership from the HyT block definitions in Section IV; that membership is the very conclusion needed to make the Euclidean universal approximation theorem applicable in hyperbolic space, so the approximation step is question-begging rather than derived. Additionally, the definition in (IV.3) sets t_H: D_c^{d×t} → R^{d×t}, making “finite composition” of such blocks ill-defined for depth greater than one; if the class effectively contains only single-block functions, the universal approximation claim cannot hold, and if depth is allowed, the pseudo-dimension bound in Lemma 5 would need to depend on depth. These are correctness defects, noted here because they make the asserted or assumed steps load-bearing rather than harmless. On the circularity scale, the self-referential Lemma 7 plus the imported [33] concentration bound push the core convergence estimate into the “forced by self-citation chain” range. The theorem's conclusion may be true, but the derivation chain as presented does not establish it independently.

Assumptions & free parameters 3 free parameters · 5 assumptions · 2 invented entities

The proof imports its central concentration inequality from the authors' own prior work [33], assumes without proof that Euclidean transformers composed with log maps lie in the HyT class, and uses a class definition whose composition is ill-defined. The final inequality in the proof of Theorem V.1 also fails algebraically. These are not standard background facts; they are load-bearing and unverified.

free parameters (3)
  • Truncation parameter M_t = M_t → 1/√c with t^{-θ} M_t^2(1 + tanh^{-1}(M_t√c)/(M_t√c))^2 → 0
    Introduced ad hoc in Theorem V.1 conditions (1)-(2); the consistency conclusion depends on the existence of such a sequence, but none is constructed.
  • Rate exponent θ = θ ∈ (0, 1/2d), arbitrary
    The claimed O(t^{-1/2d}) rate is not actually proven; the theorem yields O(t^{-θ}) for θ < 1/2d, and the endpoint is excluded.
  • Curvature c (experiments) = 0, 0.0001, 1.0, 10.0
    Hand-chosen in Section VI for the empirical runs; not derived from data.
assumptions (5)
  • ad hoc to paper Lemma 17 of [33] concentration inequality is valid and applies to the HyT class
    Basis of Lemma 7's probability bound; imported from the authors' own previous paper, not proved here.
  • ad hoc to paper t∘log_0^c ∈ T_{P,H}^{2,1,4} for any Euclidean transformer t ∈ T_P^{2,1,4}
    Needed in Lemma 8 to transfer universal approximation to the hyperbolic setting; asserted without proof.
  • ad hoc to paper Finite compositions of HyT blocks are well-defined on D_c^{d×t}
    Section IV.3 defines T_H via compositions but blocks map D_c^{d×t} → R^{d×t}; composition requires an input domain match that is not stated.
  • domain assumption Pseudo-dimension bounds for piecewise-linear networks (Bartlett et al. [36], Anthony-Bartlett [37]) apply to the HyT parameterization
    Used in Lemma 5 without verifying the parameter-to-function map of HyT has the required piecewise structure.
  • domain assumption The Bayes predictor for the hyperbolic regression problem is the conditional expectation of log_0^c(Y)
    Lemma 1 derives this by L2 expansion; it assumes squared error in the tangent space at 0 is the right loss for hyperbolic outputs.
invented entities (2)
  • Hyperbolic Transformer (HyT) with HypAttn and HypFF layers
    purpose: Serves as the architecture for which universal consistency is claimed; replaces Euclidean attention/FF with Möbius-based operations.
    New architecture introduced in Section IV; no external falsifiable prediction is provided, and the empirical validation does not isolate the theorem's predictions.
  • Hyperbolic positional encoding E ∈ D_c^{d×t}
    purpose: Added via Möbius addition to inject token-position information, mirroring Euclidean positional encodings.
    Introduced in IV.4; no evidence outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transformers Are Universally Consistent." pith.science (2026). https://pith.science/paper/253XK2MU

@misc{pith2026250524531,
  author       = {Pith},
  title        = {Pith review of: Transformers Are Universally Consistent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/253XK2MU}},
  note         = {Machine review of arXiv:2505.24531}
}
abstract

Despite their central role in the success of foundational models and large-scale language modeling, the theoretical foundations governing the operation of Transformers remain only partially understood. Contemporary research has largely focused on their representational capacity for language comprehension and their prowess in in-context learning, frequently under idealized assumptions such as linearized attention mechanisms. Initially conceived to model sequence-to-sequence transformations, a fundamental and unresolved question is whether Transformers can robustly perform functional regression over sequences of input tokens. This question assumes heightened importance given the inherently non-Euclidean geometry underlying real-world data distributions. In this work, we establish that Transformers equipped with softmax-based nonlinear attention are uniformly consistent when tasked with executing Ordinary Least Squares (OLS) regression, provided both the inputs and outputs are embedded in hyperbolic space. We derive deterministic upper bounds on the empirical error which, in the asymptotic regime, decay at a provable rate of $\mathcal{O}(t^{-1/2d})$, where $t$ denotes the number of input tokens and $d$ the embedding dimensionality. Notably, our analysis subsumes the Euclidean setting as a special case, recovering analogous convergence guarantees parameterized by the intrinsic dimensionality of the data manifold. These theoretical insights are corroborated through empirical evaluations on real-world datasets involving both continuous and categorical response variables.

Figures

Figures reproduced from arXiv: 2505.24531 by the authors.

Figure 1
Figure 1. Test RMSE Loss over the number of training iterations of the three Question-Answering datasets shown [PITH_FULL_IMAGE:figures/full_fig_p014_1.png] view at source ↗
Figure 2
Figure 2. Test RMSE Loss over the number of training tokens (in [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 22 canonical work pages

  1. [33]

    On the universal statistical consistency of expansive hyperbolic deep convolutional neural networks,

    S. Ghosh, K. Bose, and S. Das, “On the universal statistical consistency of expansive hyperbolic deep convolutional neural networks,” arXiv preprint arXiv:2411.10128, 2024

    arXiv 2024

  2. [36]

    Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks,

    P. L. Bartlett, N. Harvey, C. Liaw, and A. Mehrabian, “Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks,”Journal of Machine Learning Research, vol. 20, no. 63, pp. 1–17, 2019

    work page 2019

  3. [37]

    Anthony and P

    M. Anthony and P. L. Bartlett,Neural network learning: Theoretical foundations. cambridge university press, 2009. 16

    work page 2009

  4. [1]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,”Advances in Neural Information Processing Systems, 2017

    2017

  5. [2]

    Learning deep transformer models for machine translation,

    Q. Wang, B. Li, T. Xiao, J. Zhu, C. Li, D. F. Wong, and L. S. Chao, “Learning deep transformer models for machine translation,”arXiv preprint arXiv:1906.01787, 2019

    arXiv 1906

  6. [3]

    Augmenting human innovation teams with artificial intelligence: Exploring transformer-based language models,

    S. G. Bouschery, V . Blazevic, and F. T. Piller, “Augmenting human innovation teams with artificial intelligence: Exploring transformer-based language models,”Journal of Product Innovation Management, vol. 40, no. 2, pp. 139–153, 2023

    work page 2023

  7. [4]

    A survey of controllable text generation using transformer-based pre-trained language models,

    H. Zhang, H. Song, S. Li, M. Zhou, and D. Song, “A survey of controllable text generation using transformer-based pre-trained language models,”ACM Computing Surveys, vol. 56, no. 3, pp. 1–37, 2023

    work page 2023

  8. [5]

    Transformer-based neural network for answer selection in question answering,

    T. Shao, Y . Guo, H. Chen, and Z. Hao, “Transformer-based neural network for answer selection in question answering,”IEEE Access, vol. 7, pp. 26 146–26 156, 2019

    work page 2019

Show all 40 references
  1. [6]

    Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition,

    L. Dong, S. Xu, and B. Xu, “Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition,” in2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, pp. 5884–5888

    2018

  2. [7]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wuet al., “Conformer: Convolution-augmented transformer for speech recognition,”arXiv preprint arXiv:2005.08100, 2020

    2005 arXiv

  3. [8]

    Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences,

    A. Rives, J. Meier, T. Sercu, S. Goyal, Z. Lin, J. Liu, D. Guo, M. Ott, C. L. Zitnick, J. Maet al., “Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences,”Proceedings of the National Academy of Sciences, vol. 118, no. 15,...

    2021

  4. [9]

    Molecular transformer: a model for uncertainty- calibrated chemical reaction prediction,

    P. Schwaller, T. Laino, T. Gaudin, P. Bolgar, C. A. Hunter, C. Bekas, and A. A. Lee, “Molecular transformer: a model for uncertainty- calibrated chemical reaction prediction,”ACS central science, vol. 5, no. 9, pp. 1572–1583, 2019

    2019

  5. [10]

    Improving multi-head attention with capsule networks,

    S. Gu and Y . Feng, “Improving multi-head attention with capsule networks,” inNatural Language Processing and Chinese Computing: 8th CCF International Conference, NLPCC 2019, Dunhuang, China, October 9–14, 2019, Proceedings, Part I 8. Springer, 2019, pp. 314–326

    2019

  6. [11]

    A survey of transformers,

    T. Lin, Y . Wang, X. Liu, and X. Qiu, “A survey of transformers,”AI open, vol. 3, pp. 111–132, 2022

    2022

  7. [13]

    Universal consistency of deep convolutional neural networks,

    S.-B. Lin, K. Wang, Y . Wang, and D.-X. Zhou, “Universal consistency of deep convolutional neural networks,”IEEE Transactions on Information Theory, vol. 68, no. 7, pp. 4610–4617, 2022

    2022

  8. [14]

    Generating long sequences with sparse transformers,

    R. Child, S. Gray, A. Radford, and I. Sutskever, “Generating long sequences with sparse transformers,”arXiv preprint arXiv:1904.10509, 2019

    1904 arXiv

  9. [15]

    Star-transformer,

    G. Qipeng, Q. Xipeng, L. Pengfei, S. Yunfan, X. Xiangyang, and Z. Zheng, “Star-transformer,” inProceedings of HLT-NAACL, 2019, pp. 1315–1325

    2019

  10. [16]

    Longformer: The long-document transformer,

    I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,”arXiv preprint arXiv:2004.05150, 2020

    2004 arXiv

  11. [17]

    Big bird: Transformers for longer sequences,

    M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yanget al., “Big bird: Transformers for longer sequences,”Advances in neural information processing systems, vol. 33, pp. 17 283–17 297, 2020

    2020

  12. [18]

    Axial attention in multidimensional transformers,

    J. Ho, N. Kalchbrenner, D. Weissenborn, and T. Salimans, “Axial attention in multidimensional transformers,”arXiv preprint arXiv:1912.12180, 2019

    1912 arXiv

  13. [19]

    Sparse sinkhorn attention,

    Y . Tay, D. Bahri, L. Yang, D. Metzler, and D.-C. Juan, “Sparse sinkhorn attention,” inInternational Conference on Machine Learning. PMLR, 2020, pp. 9438–9447

    2020

  14. [20]

    Approximation by superpositions of a sigmoidal function,

    G. Cybenko, “Approximation by superpositions of a sigmoidal function,”Mathematics of control, signals and systems, vol. 2, no. 4, pp. 303–314, 1989

    1989

  15. [21]

    Approximation capabilities of multilayer feedforward networks,

    K. Hornik, “Approximation capabilities of multilayer feedforward networks,”Neural networks, vol. 4, no. 2, pp. 251–257, 1991

    1991

  16. [22]

    The expressive power of neural networks: A view from the width,

    Z. Lu, H. Pu, F. Wang, Z. Hu, and L. Wang, “The expressive power of neural networks: A view from the width,”Advances in neural information processing systems, vol. 30, 2017

    2017

  17. [23]

    Resnet with one-neuron hidden layers is a universal approximator,

    H. Lin and S. Jegelka, “Resnet with one-neuron hidden layers is a universal approximator,”Advances in neural information processing systems, vol. 31, 2018

    2018

  18. [24]

    What does bert look at? an analysis of bert’s attention,

    K. Clark, “What does bert look at? an analysis of bert’s attention,”arXiv preprint arXiv:1906.04341, 2019

    1906 arXiv

  19. [25]

    Universality of deep convolutional neural networks,

    D.-X. Zhou, “Universality of deep convolutional neural networks,”Applied and computational harmonic analysis, vol. 48, no. 2, pp. 787–794, 2020

    2020

  20. [26]

    Universal approximations of permutation invariant/equivariant functions by deep neural networks,

    A. Sannai, Y . Takai, and M. Cordonnier, “Universal approximations of permutation invariant/equivariant functions by deep neural networks,” arXiv preprint arXiv:1903.01939, 2019

    1903 arXiv

  21. [27]

    L. W. Tu,Differential geometry: connections, curvature, and characteristic classes. Springer, 2017, vol. 275

    2017

  22. [28]

    do Carmo,Riemannian Geometry, ser

    M. do Carmo,Riemannian Geometry, ser. Mathematics (Boston, Mass.). Birkh ¨auser, 1992

    1992

  23. [29]

    Lang,Differential and Riemannian manifolds

    S. Lang,Differential and Riemannian manifolds. Springer Science & Business Media, 1995

    1995

  24. [30]

    J. M. Lee,Riemannian manifolds: an introduction to curvature. Springer Science & Business Media, 2006, vol. 176

    2006

  25. [31]

    Ungar,A gyrovector space approach to hyperbolic geometry

    A. Ungar,A gyrovector space approach to hyperbolic geometry. Springer Nature, 2022

    2022

  26. [32]

    A geometric interpretation of ungar’s addition and of gyration in the hyperbolic plane,

    J. Vermeer, “A geometric interpretation of ungar’s addition and of gyration in the hyperbolic plane,”Topology and its Applications, vol. 152, no. 3, pp. 226–242, 2005

    2005

  27. [34]

    Gy ¨orfi, M

    L. Gy ¨orfi, M. Kohler, A. Krzyzak, and H. Walk,A distribution-free theory of nonparametric regression. Springer Science & Business Media, 2006

    2006

  28. [35]

    Decision theoretic generalizations of the pac model for neural net and other learning applications,

    D. Haussler, “Decision theoretic generalizations of the pac model for neural net and other learning applications,”Information and computation, vol. 100, no. 1, pp. 78–150, 1992

    1992

  29. [38]

    Are transformers universal approximators of sequence-to-sequence functions?

    C. Yun, S. Bhojanapalli, A. S. Rawat, S. J. Reddi, and S. Kumar, “Are transformers universal approximators of sequence-to-sequence functions?”arXiv preprint arXiv:1912.10077, 2019

    1912 arXiv

  30. [39]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gellyet al., “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  31. [40]

    Decision transformer: Reinforcement learning via sequence modeling,

    L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch, “Decision transformer: Reinforcement learning via sequence modeling,”Advances in neural information processing systems, vol. 34, pp. 15 084–15 097, 2021

    2021

  32. [41]

    Rewon child, scott gray, alec radford, jeffrey wu, and dario amodei. scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, and B. Chess, “Rewon child, scott gray, alec radford, jeffrey wu, and dario amodei. scaling laws for neural language models,”arXiv preprint arXiv:2001.08361, vol. 1, no. 2, p. 4, 2020. 17

    2001 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.