Pith. sign in

REVIEW 3 major objections 5 minor 35 references

Equipping a transformer with a hierarchical grading reduces the data needed to reach a fixed target error exactly when the target's energy profile and the data's variance profile diverge, at a rate fixed by their Bhattacharyya affinity.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:41 UTC pith:P7PXNEZU

load-bearing objection Clean theory, unmeasured premise: the sample-complexity math is sound, but the LLM benefit rests on an untested Bhattacharyya affinity assumption. the 3 major comments →

arxiv 2607.22757 v1 pith:P7PXNEZU submitted 2026-07-23 cs.LG cs.AI

Hierarchical Grading in Large Language Models

T. Shaska This is my paper
classification cs.LG cs.AI MSC 68T0768T5068Q3262C2016W50
keywords graded vector spacesgraded transformerssample complexityminimax lower boundshierarchical attentioninductive biasBhattacharyya affinitylarge language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that the entire benefit of grading a transformer's representation space is a single number: the Bhattacharyya affinity between the target's energy profile and the data's variance profile. When those profiles diverge, the graded model reaches a fixed generalization error with fewer samples, in the closed-form ratio BC(α,τ)²; when they agree, grading neither helps nor hurts. The paper proves this ratio for Rademacher-style bounds, characterizes the improving grades as an open convex cone with the standard transformer on its boundary, and proves a matching minimax separation in the level-stratified regime. It also shows that a trained graded transformer compiles to a standard transformer with zero inference overhead, so the only cost is a conditioning penalty that grows linearly in the grade clip while the sample saving grows quadratically.

Core claim

The central claim is that grading a linear readout over a level-decomposed embedding changes the required sample size by the ratio Λ(g) = (Σ_j α_j g_j^{-2})(Σ_j τ_j g_j²), where α is the target's energy profile and τ is the data's variance profile. The optimal grades are g*_j ∝ α_j^{1/4} τ_j^{-1/4}, giving Λ* = BC(α,τ)² ≤ 1, with equality only when the profiles coincide. The ordinary transformer, with g = 1, sits on the boundary of the admissible cone, not at a local optimum. In the level-stratified geometric regime, the ratio is a genuine minimax risk ratio over all estimators throughout an explicit window of sample sizes, so the separation between the graded prior and its absence is establ

What carries the argument

The graded vector space R^d_q with scalar action λ⋆x = (λ^{q_j} x_j), propagated through embeddings and self-attention as a diagonal transformation G. The load-bearing identity is the sample-complexity ratio Λ(g) = (Σ_j α_j g_j^{-2})(Σ_j τ_j g_j²), minimized by the fourth-root law g*_j ∝ α_j^{1/4} τ_j^{-1/4} to the squared Bhattacharyya affinity BC(α,τ)². The admissible grades form an open convex cone entered exactly when ⟨q, α−τ⟩ > 0; the minimax separation is obtained by Assouad packing on the ellipsoidal constraint geometry that the grades define.

Load-bearing premise

Assumption 1 (Profile Divergence): for hierarchical language targets, the features that decide the target carry little corpus variance — the target's energy profile α and the data's variance profile τ are far apart in Bhattacharyya affinity; if they agree, the graded benefit Λ* is 1 and the entire sample-complexity advantage disappears.

What would settle it

Estimate α and τ offline on an annotated corpus: train a linear probe on the target to get α and compute the normalized per-coordinate variance over the data to get τ. If BC(α,τ) is close to 1 on hierarchical tasks, Assumption 1 is falsified and every conditional prediction of the paper collapses, at the cost of two estimates and no model training. Alternatively, run the six-arm selection ladder: if token efficiency does not order by the admissibility inner product ⟨q, α̂−τ̂⟩, the first-order criterion of Prop. 8 fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the target and data profiles diverge as Assumption 1 asserts, a graded model reaches a fixed target error with a fraction Λ* of the tokens a uniform transformer needs, and that fraction is computable offline before any training.
  • The standard transformer is not a distinguished optimum: whenever α ≠ τ, a strictly descending direction from isotropy exists, so the uniform architecture is forgoing a measurable gain.
  • Every trained GLLM compiles to a standard transformer of identical architecture and inference cost, so the inductive bias is paid for once during training and is free at deployment.
  • Optimal grades are the solution of a convex program, certified by a single inner product computed from estimated profiles; estimation error degrades the attained gain only quadratically.
  • In the level-stratified geometric regime, the ratio Λ(g) is attained as a minimax risk ratio over all estimators throughout a sample-size window, so the sample-complexity advantage is a property of the prior itself, not an artifact of the bounds.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's analysis is proved for a single linear read-out; if the same ratio Λ(g) survives a layerwise extension through the full graded stack, the framework would apply unchanged to deep transformers, and the convex grade-selection procedure would carry over.
  • The framework implies a general recipe for any domain with measurable profiles: fix a basis, estimate the target-energy and data-variance profiles, test their affinity, then solve a convex program. Domains such as vision or multimodal learning with structured, hierarchical targets might exhibit the same divergence and could be graded without architectural changes.
  • Because the clip C prices the trade as a linear conditioning penalty against a quadratic sample saving, sweeping C should trace an explicit cost–benefit curve; the paper plans this sweep, but the curve itself is a direct quantitative prediction of the framework.
  • The geometric-invariant-theory dictionary suggests a consistency test in canonically graded domains: if a domain carries known weights (as in weighted projective spaces), the estimated optimal grades should recover those weights, providing a non-linguistic check of the entire selection procedure.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Graded Large Language Models (GLLMs), which equip the representation space of a transformer with a grading and propagate the induced weighted scalar action through embeddings, attention, and the loss. The central object is the gain function Λ(g) = (Σ_j α_j g_j^{-2})(Σ_j τ_j g_j^2), where α and τ are target and data energy profiles; its minimum is the squared Bhattacharyya affinity BC(α,τ)^2 ≤ 1 (Prop. 7). The paper characterizes the admissible grades by an inner-product criterion (Prop. 8), proves an exponential-in-depth decay under geometric level profiles (Cor. 3), and establishes a two-sided minimax separation in a Gaussian level-stratified model (Thm. 4.1). It also analyzes the consistency of the graded training objective (Prop. 9), the effect of grade clipping (Prop. 10), and an offline grade-selection pipeline for language (Secs. 6–7). No experiments are reported; the LLM-specific benefit is explicitly conditional on Assumption 1, that the two profiles are far apart for hierarchical language targets.

Significance. If Assumption 1 holds and the level-stratified Gaussian model is a faithful approximation, the paper gives a parameter-free, closed-form quantification of the sample-complexity benefit of an inductive bias, with zero inference overhead after training. The proofs are self-contained and the authors are unusually careful: Prop. 7 is explicitly a comparison of upper bounds, the general minimax claim is isolated as Conjecture 2, and Rem. 11 lists the limitations. Theorem 4.1 is a genuine two-sided minimax result in its regime, and Lemma 2's quadratic stability to profile-estimation error is a valuable addition. The offline divergence test in Sec. 7.3 is a genuinely falsifiable design. However, the central empirical premise for language is never measured, and the general two-sided guarantee is not proved; thus the practical claim for LLMs remains conditional.

major comments (3)
  1. [Secs. 7.3, 8; Assumption 1] The paper's own load-bearing empirical premise is not tested. Prop. 7(iv) makes the entire gain equal to BC(α,τ)^2, and Prop. 10(iii) caps the clipped gain by C^{-2}; if BC measured near 1, the benefit disappears. Section 7.3 defines exactly the needed divergence test, but Sec. 7 states 'The estimates themselves are not reported here' and Sec. 8 states 'No experiments have been conducted at the time of writing.' Table 3 is arithmetic for hypothetical (ν,μ), not data. For a manuscript claiming benefits for LLMs, this is a load-bearing gap. I recommend either reporting the offline profile estimates for the named tasks/corpora, or explicitly limiting the contribution to the conditional theory.
  2. [Sec. 4.3, Thm. 4.1; Rem. 11; Conj. 2] The two-sided minimax separation over all estimators is proved only in a Gaussian, level-homogeneous, squared-loss model with geometric profiles, known profiles, and sample sizes in the window W of Lem. 4. In the general setting of Prop. 7 the comparison remains a ratio of two Rademacher upper bounds. This is acknowledged in Rem. 11 and Conj. 2, but the abstract's phrase 'a real reduction in required data' and the introduction's framing can be read as a general theorem. Please consistently qualify the 'real reduction' claim as the stratified-Gaussian theorem and keep the general case labelled as a conjecture.
  3. [Sec. 7.2, Definition 2] The target profile α is estimated as the normalized squared weight of a ridge linear probe. This identifies α only up to the probe's inductive bias, and ridge shrinkage biases the estimate toward uniformity. The paper notes this, but the consequence is that Assumption 1, stated about 'structurally decisive features,' may not be what the offline test actually measures. Since all downstream predictions inherit the measured α, the empirical work must include probe ablations or a clear statement that the theory applies to the linear-probe profile, not to an unobserved 'true' feature profile.
minor comments (5)
  1. [Sec. 2.3 vs Sec. 8.1.1] The symbol C is used both for compute budget and for the grade clip. The paper notes the collision, but the two uses appear close together in Secs. 2.3 and 8.1.1; a different symbol for one of them would improve readability.
  2. [Def. 2, Prop. 7(ii)] The Rademacher computation uses σ_j^2 = E[x_j^2] without stating whether the input distribution is assumed centered. If x has nonzero mean, the linear class without an intercept may need a bias term, and E∥Gx∥^2 is not the variance. Please state the zero-mean assumption explicitly or adjust the definitions.
  3. [Sec. 7.5, Prop. 11] The reduced program Λ_L is stated for level-constant profiles. It would help to state explicitly that the input allocation d_l cancels in both the profiles and the optimal grades, as shown in Cor. 3, since the reader may otherwise wonder whether the level dimensions affect the gain.
  4. [Sec. 8.2.2] The planned sample-efficiency measurement uses 'the ratio of token counts at which each first attains a fixed target error.' Please specify how the target error is chosen (e.g., from validation curves) and whether the ratio is adjusted for the fact that the graded model may train with a different effective objective weight per token.
  5. [Sec. 6.7, Prop. 12] The GIT dictionary is interesting and clearly marked as non-load-bearing, but it is long. A short table of correspondences would make the subsection more digestible without cutting content.

Circularity Check

1 steps flagged

One self-acknowledged circularity in the basis construction; the core sample-complexity derivation is self-contained and independent of the self-citations.

specific steps
  1. self definitional [Section 7.1, Remark 23 (with Sec. 7.2 and Sec. 7.5)]
    "the construction above requires a representation of width d before the model that will carry it exists, and the circularity is real rather than apparent: at initialisation W_e is isotropic by construction, so the profiles read in its coordinates are uniform and carry no information. ... For step (6), the educated grading q⋆_C, the reference is the embedding matrix of the uniform arm itself."

    The profiles (α, τ) that determine the optimal grades are supported on the graded basis {P_l} of Eq. (13). For the educated grading, that basis is obtained from the uniform arm's own embedding matrix, so the inputs to the grade-selection program are read in the coordinate system of the very baseline the graded model is supposed to improve upon. At a freshly initialized W_e the profiles would be uniform by construction, implying BC=1 and no benefit; the paper must import a learned reference to avoid that. The sample-complexity ratio Prop. 7 is still a genuine computation once profiles are given, and the paper mitigates the dependence by using a public encoder for the sign-only certification (Prop. 8(iv)) and by acknowledging the transport as an assumption. The circularity is therefore real

full rationale

The paper's central derivation chain is self-contained: Prop. 7 computes the grading gain Λ(g) from the target and data profiles and proves the closed-form minimum BC(α,τ)^2 by elementary Cauchy-Schwarz; Prop. 8 derives the admissible cone, convexity, and first-order criterion directly from that computation; Cor. 3 evaluates the closed form under geometric stratification; and Thm. 4.1, together with Lemmas 3–4, provides an in-manuscript minimax lower-bound argument (Assouad/Fano packing) rather than importing the result from prior work. Assumption 1 is explicitly an empirical premise, not a theorem, and the paper designs a falsifiable offline divergence test for it; the absence of run results means the framework's applicability to LLMs is conditional, but conditionality is not circularity. The heavy self-citation to [26–29] supplies the graded-vector-space construction and the exponential-grading ansatz, but the load-bearing inequalities do not reduce to those citations; the exponential form is adopted as a construction choice, not presented as a forced consequence of prior theorems. The one genuinely circular passage is Remark 23, where the paper admits that building the graded basis from a model that has not yet been trained would yield uninformative uniform profiles, and resolves it by reading the profiles from a fixed public encoder or from the uniform control arm. This is an acknowledged provenance issue in the empirical pipeline, not a hidden equivalence between prediction and input, and the sign-only certification criterion limits its impact. Overall the mathematical predictions have independent content, so the appropriate score is low.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 1 invented entities

The core sample-complexity theorem is self-contained once profiles and a Gaussian stratified design are granted. The exponential gain rests on unverified profile-shape assumptions, and the practical benefit for language rests on Assumption 1, which is empirically testable but untested. The grade clip and geometric exponents are the main free parameters; the mathematical derivation itself does not hide additional fitted constants.

free parameters (4)
  • Geometric profile exponents (ν, μ) = e.g., (0.5, 0.2) in Table 3
    Cor. 3 and Thm. 4.1 assume a_l ∝ ν^{L-1-l}, s_l ∝ μ^l; the exponential gain Θ(ν^L) depends on these empirical constants, which are to be estimated from data.
  • Grade clip C = 2 at planned configs; swept 1.5–4
    Prop. 10 caps attainable gain at C^{-2}; the cost–benefit tradeoff is priced by this hand-chosen hyperparameter.
  • Grading base λ = e^{1/d_k}
    Sec. 3.1 fixes λ small so that clipped grades span a controlled range; affects the smoothness penalty in Lem. 5.
  • Regularization weights ν_mag, ν_div, ν_orth = not specified beyond ν_mag > ν_div H
    Eq. (20) hyperparameters governing grade magnitude, diversity, and projection orthogonality; not central to the main theorems.
axioms (6)
  • domain assumption Assumption 1: for hierarchical language targets, α and τ are far apart; decisive features carry little corpus variance.
    Sec. 2.4; the entire benefit of grading rests on this. Sec. 7.3 proposes an offline divergence test to falsify it.
  • domain assumption Geometric stratification: level profiles satisfy a_l ∝ ν^{L-1-l}, s_l ∝ μ^l with 0 < μ < ν < 1.
    Cor. 3 and Sec. 4.3; needed for the exponential Θ(ν^L) decay and for Thm. 4.1's window. It is an empirical refinement of Assumption 1, not a theorem.
  • domain assumption Gaussian level-homogeneous design with squared loss and known profiles for Thm. 4.1.
    Sec. 4.3; the minimax separation is proven only in this regime. The general case is Conjecture 2.
  • domain assumption Full support of profiles; zero coordinates are deleted.
    Def. 2; needed for coercivity in Prop. 12(ii) and for positivity of the optimal grades.
  • standard math Standard Rademacher/Bartlett–Mendelson bounds, Fano/Assouad/Le Cam inequalities, convexity of log-sum-exp, Cauchy–Schwarz.
    Used in Props. 7–8, Lem. 3, and Thm. 4.1 as unproved background results.
  • domain assumption The linguistic hierarchy subword→word→phrase→sentence→discourse is the correct stratification.
    Secs. 4.1 and 7.1; the taxonomy fixes the basis and the level structure, so the measured profiles depend on this choice.
invented entities (1)
  • Level-specific subspaces V_l of the embedding space no independent evidence
    purpose: Give linguistic strata explicit coordinates in the representation geometry and define the α/τ profiles.
    Introduced by this paper; no independent empirical evidence yet. The proposed structural-attention-mass and grade-stratified accuracy measurements (Sec. 8.2.6) could provide falsifiable handles if run, but no results are reported.

pith-pipeline@v1.3.0-alltime-deepseek · 47402 in / 19183 out tokens · 194416 ms · 2026-08-01T06:41:23.461464+00:00 · methodology

0 comments
read the original abstract

We introduce Graded Large Language Models (GLLMs), an algebraic framework that equips the representation space of a transformer with a grading and propagates the induced weighted scalar action through embeddings, self-attention, and the training objective. The construction extends the theory of graded neural networks and graded transformers to autoregressive language models while preserving expressive power, asymptotic computational complexity, and inference cost. The governing geometric picture is that of geometric invariant theory. The benefit of a grading is expressed by a Kempf--Ness functional on the grading torus; the grades that improve upon the uniform architecture form an open convex cone whose membership is decided by a Hilbert--Mumford-type criterion pairing a grade direction against two measurable profiles of the target and the data; the optimal grades are the coincidence point of two moment maps, given in closed form; and the ordinary transformer appears as a semistable isotropic point on the boundary of the cone: one member of a larger graded family rather than a distinguished optimum. Separately, for level-stratified targets we prove a minimax separation between the graded prior and its absence: over all estimators the risks of the graded and uniform target classes separate throughout an explicit window of sample sizes, by a factor that decays exponentially in the number of levels under geometric stratification. Both profiles are estimable offline, so the optimal grades solve a convex program certified before training begins. Because the grading is absorbed into the learned parameters after training, every GLLM compiles to a standard transformer of identical architecture and inference complexity.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 12 linked inside Pith

  1. [1]

    J. L. Ba, J. R. Kiros, and G. E. Hinton,Layer normalization, arXiv preprint (2016), available atarXiv:1607.06450

    Pith/arXiv arXiv 2016

  2. [2]

    Bartlett and Shahar Mendelson,Rademacher and gaussian complexities: Risk bounds and structural results, Journal of Machine Learning Research3(2002), 463–482

    Peter L. Bartlett and Shahar Mendelson,Rademacher and gaussian complexities: Risk bounds and structural results, Journal of Machine Learning Research3(2002), 463–482

    2002

  3. [3]

    Bengio, R

    Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin,A neural probabilistic language model, Journal of Machine Learning Research3(2003), 1137–1155

    2003

  4. [4]

    Elira Curri,On the stability of binary forms and their weighted heights, Albanian J. Math. 16(2022), no. 1, 3–23. MR4448533

    2022

  5. [5]

    Dao and A

    T. Dao and A. Gu,Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, Icml, 2024. Also available as arXiv:2405.21060

    Pith/arXiv arXiv 2024

  6. [6]

    L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy,The pile: An 800gb dataset of diverse text for language modeling, arXiv preprint (2020), available atarXiv:2101.00027

    Pith/arXiv arXiv 2020

  7. [7]

    Albert Gu and Tri Dao,Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint (2023), available atarXiv:2312.00752

    Pith/arXiv arXiv 2023

  8. [8]

    Hoffmann and et

    J. Hoffmann and et. al.,Training compute-optimal large language models, arXiv preprint (2022), available atarXiv:2203.15556

    Pith/arXiv arXiv 2022

  9. [9]

    Huang, Y

    Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, and Z. Chen,Gpipe: Efficient training of giant neural networks using pipeline parallelism, Neurips, 2019

    2019

  10. [10]

    Kaplan, S

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei,Scaling laws for neural language models, arXiv preprint (2020), available atarXiv:2001.08361

    Pith/arXiv arXiv 2020

  11. [11]

    Kempf and L

    G. Kempf and L. Ness,The length of vectors in representation spaces, Algebraic geometry (copenhagen, 1978), 1979, pp. 233–243

    1978

  12. [12]

    Kudo and J

    T. Kudo and J. Richardson,Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing, Emnlp: System demonstrations, 2018. HIERARCHICAL GRADING IN LARGE LANGUAGE MODELS 49

    2018

  13. [13]

    Loshchilov and F

    I. Loshchilov and F. Hutter,Sgdr: Stochastic gradient descent with warm restarts, Iclr, 2017. Also available as arXiv:1608.03983

    Pith/arXiv arXiv 2017

  14. [14]

    ,Decoupled weight decay regularization, Iclr, 2019

    2019

  15. [15]

    Mumford, J

    D. Mumford, J. Fogarty, and F. Kirwan,Geometric invariant theory, Third, Ergebnisse der Mathematik und ihrer Grenzgebiete (2), vol. 34, Springer-Verlag, Berlin, 1994

    1994

  16. [16]

    Bo Peng, Eric Alcaide, Quentin Anthony, et al.,Rwkv: Reinventing rnns for the transformer era, Findings of the association for computational linguistics: Emnlp 2023, 2023

    2023

  17. [17]

    Radford, K

    A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever,Improving language understanding by generative pre-training, OpenAI, 2018

    2018

  18. [18]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever,Language models are unsupervised multitask learners, OpenAI Blog1(2019), no. 8, 9

    2019

  19. [19]

    Rafailov, A

    R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn,Direct preference optimization: Your language model is secretly a reward model, Neurips, 2023

    2023

  20. [20]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu,Exploring the limits of transfer learning with a unified text-to-text transformer, Journal of Machine Learning Research21(2020), no. 140, 1–67

    2020

  21. [21]

    Rajbhandari, J

    S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He,Zero: Memory optimizations toward train- ing trillion parameter models, Sc20, 2020, pp. 1–16

    2020

  22. [22]

    Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo,Are emergent abilities of large language models a mirage?, Advances in neural information processing systems 36, 2023

    2023

  23. [23]

    Sennrich, B

    R. Sennrich, B. Haddow, and A. Birch,Neural machine translation of rare words with subword units, Acl, 2016

    2016

  24. [24]

    Shaham, E

    U. Shaham, E. Segal, M. Ivgi, A. Efrat, O. Yoran, A. Haviv, A. Gupta, W. Xiong, M. Geva, J. Berant, and O. Levy,Scrolls: Standardized comparison over long language sequences, Emnlp,

  25. [25]

    C. E. Shannon,Prediction and entropy of printed english, The Bell System Technical Journal 30(1951), no. 1, 50–64

    1951

  26. [26]

    Shaska,Internalizing Tools as Morphisms in Graded Transformers, arXiv preprint (2025), available atarXiv:2511.17840

    T. Shaska,Internalizing Tools as Morphisms in Graded Transformers, arXiv preprint (2025), available atarXiv:2511.17840

    arXiv 2025

  27. [27]

    Also available as arXiv:2507.20108

    ,Graded Transformers, AI in Mathematics and Theoretical Physics(to appear) (2026). Also available as arXiv:2507.20108

    Pith/arXiv arXiv 2026

  28. [28]

    Tony Shaska,Graded Neural Networks, Int. J. Data Sci. Math. Sci.3(2025), no. 2, 87–116. MR5061522

    2025

  29. [29]

    ,Artificial Neural Networks on Graded Vector Spaces, Artificial Intelligence and Math- ematics Research, [2026]©2026, pp. 1–71. MR5023824

    2026

  30. [30]

    Shazeer,Glu variants improve transformer, arXiv preprint (2020), available atarXiv: 2002.05202

    N. Shazeer,Glu variants improve transformer, arXiv preprint (2020), available atarXiv: 2002.05202

    Pith/arXiv arXiv 2020

  31. [31]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin,Attention is all you need, Advances in neural information processing systems 30, 2017, pp. 5998–6008

    2017

  32. [32]

    Jason Wei, Yi Tay, Rishi Bommasani, et al.,Emergent abilities of large language models, Transactions on Machine Learning Research (2022), available atarXiv:2206.07682

    Pith/arXiv arXiv 2022

  33. [33]

    Xiong, Y

    R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T.-Y. Liu,On layer normalization in the transformer architecture, Icml, 2020, pp. 10524– 10533

    2020

  34. [34]

    Zhang and R

    B. Zhang and R. Sennrich,Root mean square layer normalization, Advances in neural infor- mation processing systems 32, 2019, pp. 12360–12371. Also available as arXiv:1910.07467. Department of Computer Science and Engineering,, Oakland University, Rochester, MI 48309 Email address:shaska@oakland.edu

    Pith/arXiv arXiv 2019

  35. [2022]

    Also available as arXiv:2201.03533

    Pith/arXiv arXiv