Pith. sign in

REVIEW 4 major objections 5 minor 14 cited by

A lightweight 'Canon layer' that mixes neighboring tokens can increase language-model reasoning depth 2–4× and revive architectures with no position embeddings.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 15:18 UTC pith:7ISGGDM6

load-bearing objection The 'canon layer' story — residual conv1d helps learning speed and Mamba2's internal conv1d matters — is plausible and systematically argued, but the real-world transfer claim rests on noise-level evidence. the 4 major comments →

arxiv 2512.17351 v2 pith:7ISGGDM6 submitted 2025-12-19 cs.CL

Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers

Zeyuan Allen-Zhu This is my paper
classification cs.CL
keywords Canon layershorizontal information flowsynthetic pretrainingreasoning depthlinear attentionstate-space modelsposition embeddingsmulti-hop reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Architecture comparisons at academic scale are swamped by seed noise and by data that mixes many skills, so this paper builds five synthetic pretraining tasks that each isolate one capability: mental reasoning depth, reasoning breadth, knowledge capacity, knowledge manipulation, and hierarchical structure. Using those tasks, it introduces Canon layers — fixed-window 1-d causal convolutions inserted before, inside, or around the attention and MLP sublayers — that act as horizontal residual links across neighboring tokens. The paper argues these layers substantially improve reasoning depth, breadth, knowledge capacity, and knowledge manipulation across Transformers, linear attention, and state-space models; revive NoPE Transformers to RoPE level; and lift plain GLA to rival Mamba2/GDN. It also argues that linear models' shallow reasoning comes from compression and retrieval errors rather than from limited memory, and that real-world academic-scale pretraining shows the same coarse trends.

Core claim

The central claim is that horizontal information flow within a layer — not attention, recurrence, or positional embeddings by themselves — is a load-bearing architectural ingredient. Canon layers compute h'_t = h_t + conv1d([h_t, h_{t-1}, h_{t-2}, h_{t-3}]), a kernel-size-4 causal convolution with a residual connection, and can be inserted at four points: before attention, inside attention, before the MLP, and inside the MLP. In the synthetic playground, adding these layers with less than 0.5% more trainable parameters lets a RoPE Transformer solve substantially deeper multi-hop reasoning than the same model without them, brings NoPE models up to or beyond RoPE performance, and makes GLA com

What carries the argument

The Canon layer: a residual causal convolution over a four-token window, h'_t = h_t + w1⊙h_{t-1} + w2⊙h_{t-2} + w3⊙h_{t-3}, which provides cheap local context at chosen sublayer positions (A: before attention, B: inside attention, C: before MLP, D: inside MLP). This mechanism carries the argument by giving every token a lightweight horizontal 'residual link' to its neighbors, so global attention and recurrent states can be used for targeted retrieval rather than for local copying; it is supported by the five-task synthetic pretraining setup that isolates each capability separately.

Load-bearing premise

The whole argument rests on the assumption that the five synthetic tasks isolate skills whose learning-speed rankings transfer to real-world pretraining at larger scale; if those rankings flip outside the synthetic playground, the architecture conclusions lose their real-world force.

What would settle it

Pretrain GLA+Canon and Mamba2/GDN at a scale where multi-hop reasoning actually emerges (e.g., 8B parameters on 1–2T tokens) and compare on real-world multi-hop benchmarks: if GLA+Canon falls clearly behind GDN/Mamba2, or NoPE+Canon falls clearly behind RoPE+Canon, the synthetic ranking would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Adding Canon layers to a standard Transformer at negligible parameter cost increases reasoning depth by 2–4× in the synthetic tasks, letting smaller models master multi-hop problems they otherwise cannot solve.
  • NoPE plus Canon layers matches or beats RoPE plus Canon layers, so heavy rotary positional encoding can be reduced or removed without a performance penalty, improving length generalization.
  • A simple GLA plus Canon-AbCD setup rivals Mamba2 and GDN, suggesting that much of the benefit of recent linear-model refinements may come from internal convolution-style mixing rather than the state-space mechanism itself.
  • Removing Mamba2's built-in conv1d drops its performance to GLA level, while replacing it with full Canon layers improves it further, identifying the conv1d as a key load-bearing component.
  • Academic-scale real-world pretraining (1.3B parameters, 100B tokens) is too noisy to resolve fine architectural differences, yet the coarse patterns — Canon helping NoPE and GLA, linear models lagging on retrieval — mirror the synthetic results.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the synthetic-to-real transfer holds, architecture search could be performed at tiny scale on controlled data, making systematic ablations feasible before expensive industrial-scale training.
  • A testable consequence of the paper's view is that many familiar components — positional encodings, gating, delta rules — may partly work by providing local mixing; replacing each with an explicit Canon layer should recover most of their benefit.
  • The paper's 'prediction of future architectures' claim implicitly extends beyond architecture to data curation and RL post-training, but that extension is not tested here and would need its own synthetic benchmarks.
  • Varying Canon kernel width, layer placement, and dynamic input-dependent weights is a natural next experiment; the current paper uses only a fixed kernel size 4, so the claimed benefits are a lower bound on what this design family might achieve.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces 'Canon layers' — lightweight causal conv1d residual blocks (Eq. 4.1) inserted at various sub-layer positions — and argues, via a suite of five synthetic pretraining tasks (Depo, Brevo, Capo, Mano, Lano), that these layers substantially improve reasoning depth, breadth, knowledge capacity, and knowledge manipulation across Transformers, linear attention (GLA), and state-space models (Mamba2, GDN). It further claims that the synthetic rankings 'mirror' academic-scale real-world pretraining at 1.3B/100B tokens, and that the synthetic playground can predict future architecture behavior as training pipelines improve. Twelve key results are presented, leading to recommendations such as Canon-ABCD for Transformers and Canon-AbCD(res) for linear models.

Significance. If the claims hold, the paper would offer a practical architectural primitive — a few lines of code — that improves multiple capabilities across model families, and a low-cost synthetic evaluation methodology for architecture science. The empirical scope is unusually broad: 3×4 mini scaling laws per task, ablations over Canon positions, residualness, activations, and comparisons against recent baselines. The paper is also commendably explicit about the noise and limited resolution of academic-scale pretraining, and it ships or promises code and datasets. However, the headline quantitative claims and the synthetic-to-real transfer argument rest on evaluation protocols that need tightening: best-of-LR selection without a validation split, reduced difficulty for weaker models, and single-seed real-world validation. These issues are local enough to be addressed in revision, but they directly affect the stated multipliers and the 'mirror' claim.

major comments (4)
  1. [§3.1 and Appendix A.1/A.3] All synthetic results are reported as the best accuracy over 4 learning rates (Depo/Brevo: Appendix A.1/A.2; Mano: 4 LRs × 2 seeds, A.4; Capo: best of 2 LRs, A.3), with no held-out validation split for LR selection and no confidence intervals. Since the paper itself notes grokking-driven randomness in Mano and 2–4% seed swings in real-world data, selecting the maximum over LRs on the test distribution inflates apparent gains and makes the precise multipliers (2–4×, 30%, 10–15%) in Results 2–5 fragile. Please report a validation-based selection procedure or show the full LR × seed distribution for the headline comparisons.
  2. [§2.1, Figures 4, 7, 14] Weaker models are evaluated on Depo with K=4, while the reasoning-depth improvements in Result 2 are claimed from Depo1(K=8) and Depo2(K=16) where the original models score 0% or near-0%. Figure 14 further states that for linear models 'meaningful results only for Depo(K=4)'. Consequently, the '2–4× deeper reasoning' comparison is not made at matched difficulty across all architectures; the 4× figure is a ratio against a 0% baseline. Please provide matched-K comparisons for all models in the claim, or explicitly restrict the depth-ratio conclusion to the difficulty range where all compared models were evaluated.
  3. [§8, Result 12, Figure 16] The real-world transfer evidence consists of single-seed runs (seed 20) for almost every architecture, with best-of-LR selection, while only Llama(RoPE)-original has seeds 20–27. The paper documents 2–4% seed noise and states that 'academic-scale pretraining cannot reliably distinguish finer architectural differences', yet the abstract claims Canon layers are 'validated through ... real-world academic-scale pretraining' and Result 12 asserts that trends 'mirror' synthetic results. The claimed equalities (NoPE+Canon≈RoPE, GLA+Canon≈GDN) fall within the paper's own noise band. The future-work sentence about 1–8B/1–2T pretrains provides no data or protocol. Please provide multi-seed statistics with confidence intervals for the key real-world comparisons, or downgrade the transfer claim to tentative.
  4. [§3.1 vs. Abstract/Result 2] Section 3.1 explicitly reframes all synthetic results as 'differences in the speed of skill acquisition, not absolute capability,' yet the abstract and Result 2 state absolute-sounding multipliers ('reasoning depth increases by 2–4×', 'knowledge capacity by 10–15%', 'knowledge manipulation length by 30%') measured after a fixed budget with best-of-LR reporting. These are speed-of-learning statements on specific synthetic tasks. Please carry the speed/absolute distinction through every quantitative claim, e.g., 'within our fixed training budget and under best-of-LR reporting, Canon reaches the same depth with fewer steps' rather than 'increases depth by 4×'.
minor comments (5)
  1. [Eq. (4.1)] The formal definition uses h'_t = h_t + conv1d([h_t, ..., h_{t-3}]), but the earlier general form writes h'_t = w_0⊙h_t + w_1⊙h_{t-1} + ... . Clarify that Eq. (4.1) is the residual parameterization with learned kernel weights and no bias.
  2. [§2.1] Task name typo: 'Breo' is introduced in the text but the task is called 'Brevo' throughout the rest of the paper and in Figure 3. Harmonize.
  3. [Figure 16] The table is extremely dense and difficult to read, with 30+ models × 32 metrics. Consider splitting into panels or moving the full table to the appendix and showing only the key comparisons (NoPE±Canon, GLA±Canon, Mamba2±conv1d) in the main text.
  4. [§5.2] The notation RoPE(♩), representing reduced RoPE (1/4 of dimensions), is used before it is formally introduced. Define it when first used and keep a consistent symbol list.
  5. [§4, Related Work] The aside about an ICML reviewer comment ('too surprising to be true') is out of place in a formal paper; remove it or move it to acknowledgments.

Circularity Check

0 steps flagged

No significant circularity: the paper is an empirical architecture study whose claims are measured against independently defined synthetic and real-world tasks; limitations and self-citations do not reduce the results to their inputs.

full rationale

The paper does not derive its predictions from its inputs by construction. Canon layers are defined independently in Eq. (4.1) as h'_t = h_t + conv1d([h_t, h_{t-1}, h_{t-2}, h_{t-3}]), and the five synthetic tasks (Depo, Brevo, Capo, Mano, Lano) are specified by graph, CFG, and arithmetic generators that do not mention Canon layers, so the reported accuracy differences are measured rather than definitionally forced. The recommended Canon configurations (Canon-ABCD, Canon-AbCD(res)) are selected via ablations on the synthetic playground and then carried into the real-world section; this is a model-selection and potential overfitting risk, but Section 8 evaluates on held-out natural-language benchmarks, so the real-world comparison is not statistically forced by the synthetic fits. Section 3.1 explicitly reframes all synthetic results as 'differences in the speed of skill acquisition, not absolute capability,' and Result 12 concedes that 'Academic-scale pretraining cannot reliably distinguish finer architectural differences' — these are honest limitations, not evidence that a prediction equals its input. The paper's self-citations (e.g., [3,4] for hierarchical-learning theory, [8] for Capo, [75] for the Brevo preprocessing observation) supply background and are acknowledged as prior work; they are not used to forbid alternatives or to define the central Canon claim. The paper also transparently credits causal conv1d to H3/Mamba/GLA and Primer, saying those 'roughly correspond to Canon-B(no-res),' so the Canon idea is presented as a generalization rather than a hidden renaming. The unsupported sentence that 'Preliminary results from larger pretrains (1-8B models on 1-2T tokens) closely align' is a missing-evidence issue for the transfer claim, not a circular derivation. No equation reduces to another equation, and no fitted parameter is relabeled as a prediction. Verdict: no significant circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 1 invented entities

The paper's contribution is mostly empirical: the module (Eq. 4.1) introduces one hand-chosen kernel size and placement scheme, and the evaluation introduces several hand-chosen task parameters (sampling distributions, K values, best-of-LR reporting). The strongest load-bearing assumptions are the transferability of synthetic tasks and the interpretability of 'learning speed' as architectural quality. The only invented 'entity' is the Canon layer itself, which has independent empirical support but is not a fundamentally new mechanism.

free parameters (6)
  • Canon kernel size = 4
    Hand-chosen; no sweep reported, though kernel size controls the receptive field for horizontal mixing.
  • Canon placement and residual design = Canon-ABCD / Canon-AbCD(res)
    Recommended variants were selected via ablations on the same synthetic tasks; different models get different variants (e.g., GLA/Mamba2/GDN use AbCD), which weakens the 'universal/never hurts' claim.
  • Best-of-LR reporting protocol = best of {3e-4, 5e-4, 1e-3, 2e-3}, sometimes 8 runs
    Evaluations report best accuracy across multiple learning rates on the test setup; inflates measured performance and removes variance estimates.
  • n-sampling distribution for Depo/Brevo = p(n) ∝ 1/sqrt(N+n)
    Curriculum choice asserted to be 'tested thoroughly' by the author (Remark A.2) but not independently justified; shapes task difficulty progression.
  • Reduced difficulty for weak models = K=4 instead of K=8/16
    Weaker architectures are evaluated on easier setups, making cross-architecture 'depth' comparisons not fully apples-to-apples.
  • Real-world training budget = 1.3B params, 100B tokens, context 4096
    Fixed academic-scale budget; the paper itself documents ≥2-4% seed variance and near-random 2-hop performance at this scale.
axioms (5)
  • domain assumption Synthetic tasks isolate atomic capabilities and learning-speed differences on them rank real-world architectures.
    Central methodological premise (Sections 1-2); real-world Section 8 only qualitatively mirrors it at a scale the paper calls noise-dominated.
  • domain assumption Faster skill acquisition is an interpretable architectural advantage.
    Section 3.1: 'results should be seen as differences in the speed of skill acquisition, not absolute capability' — a normative choice.
  • domain assumption Mamba2/GDN public implementations are faithful baselines and removing conv1d cleanly isolates the SSM behavior.
    Sections 6.2-6.3; relies on correctness of external codebases and that the ablation does not damage training stability in other ways.
  • domain assumption Academic-scale pretraining on FineWeb-Edu/SlimPajama is a valid (if noisy) real-world test of the synthetic predictions.
    Section 8 and Result 12; the paper explicitly acknowledges limited resolution and high variance.
  • ad hoc to paper Random fixed-weight Canon layers are already effective.
    Claimed in Section 4 and Figure 24 (appendix), but not shown in full detail in the main text; used to argue the mechanism is not from learned weights.
invented entities (1)
  • Canon layer independent evidence
    purpose: Horizontal information flow between adjacent tokens across multiple sublayers (A/B/C/D) of sequence models.
    Ablations across five synthetic tasks and qualitative real-world confirmation support its empirical utility; however, it is essentially a re-labeling of residual causal conv1d already present in prior work.

pith-pipeline@v1.3.0-alltime-deepseek · 80790 in / 12196 out tokens · 128378 ms · 2026-08-03T15:18:03.849604+00:00 · methodology

0 comments
read the original abstract

Understanding architectural differences in language models is challenging, especially at academic-scale pretraining (e.g., 1.3B parameters, 100B tokens), where results are often dominated by noise and randomness. To overcome this, we introduce controlled synthetic pretraining tasks that isolate and evaluate core model capabilities. Within this framework, we discover CANON LAYERS: lightweight architectural components -- named after the musical term "canon" -- that promote horizontal information flow across neighboring tokens. Canon layers compute weighted sums of nearby token representations and integrate seamlessly into Transformers, linear attention, state-space models, or any sequence architecture. We present 12 key results. This includes how Canon layers enhance reasoning depth (e.g., by $2\times$), reasoning breadth, knowledge manipulation, etc. They lift weak architectures like NoPE to match RoPE, and linear attention to rival SOTA linear models like Mamba2/GDN -- validated both through synthetic tasks and real-world academic-scale pretraining. This synthetic playground offers an economical, principled path to isolate core model capabilities often obscured at academic scales. Equipped with infinite high-quality data, it may even PREDICT how future architectures will behave as training pipelines improve -- e.g., through better data curation or RL-based post-training -- unlocking deeper reasoning and hierarchical inference.

Figures

Figures reproduced from arXiv: 2512.17351 by Zeyuan Allen-Zhu.

Figure 1
Figure 1. Figure 1: Architecture search in noisy real-life pretraining (good luck!) vs. our synthetic playground (scientific rigor). [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Our design criteria for synthetic pretrain tasks. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of our five synthetic tasks, each isolating an atomic skill for rigorous architectural comparison. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Initial comparison of base models on five synthetic tasks. GLA performs weakest; Mamba2(mlp) excels in knowledge (Capo, Mano); GDN strengthens reasoning and surpasses Llama(RoPE) on Brevo (reasoning breadth), while RoPE remains best on Depo+Lano (depth and structural reasoning). These results confirm our synthetic playground as effective for architectural comparison, but adding Canon layers (see later) wil… view at source ↗
Figure 6
Figure 6. Figure 6: Illustration of Canon layers. Canon layers: Implementation variants. Canon layers can be implemented in many ways. Even a simple version with fixed, random weights—aggregating past three tokens as horizontal residual links—already notably enhances performance ( [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Column 1→2: Canon layers dramatically enhance RoPE, improving reasoning depth by 2–4×. Column 4→5: Canon transforms NoPE into a strong performer on par with RoPE-based models. Column 2+5→3: With Canon, RoPE usage can be reduced — RoPE + ˇ“Canon (RoPE enabled for 1/4 dimensions) outperforms both RoPE/NoPE + Canon, great news for length generalization! Remark. This figure uses Depo1(K=8) and Depo2(K=16). Ear… view at source ↗
Figure 8
Figure 8. Figure 8: Training curves for RoPE models w/+w/o Canon, on Depo2(K = 16), evaluated at k = 1, 2, 4, 8, 16 and maximum size n = N, shown in two best LRs. More model sizes/data are in [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Detailed accuracies for Task Brevo1, shown overall and stratified by dependency graph depths 1, 2, 3, 4, 5. by depth of the dependency depth. Recall each query in Brevo requires the model to identify all vertices it recursively depends on, forming a sub-DAG of varying (minimum) depth. In [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Ablation study on 12-layer, 768-dim Transformers—NoPE (top) and RoPE (bottom)—with Canon variants (A–D), residual links, activation functions, ALiBi, and H-Alibi. Blank entries indi￾cate untested configs due to resource limits. Additional ablation studies (with more model sizes) are in [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Evaluation of knowledge capacity (Capo) across architectures, measured as bits per parameter. The first row represents baseline models, while the second row shows improvements with Canon layers added. Conclusion: Canon layers enhance knowledge storage for architectures that are slower to train, such as gated MLP and MoE, mitigating the capacity gap between gated and standard MLP as identified in [8]. addi… view at source ↗
Figure 12
Figure 12. Figure 12: Columns 1, 2, 3, 5: Canon drastically improves GLA, making it better than Mamba2 (Result 6.1). Columns 1, 4, 5: Removing conv1d reduces Mamba2’s performance back to match GLA (Result 7.1). Remark. Synthetic results here predict similar trends in real-life experiments (Result 12 and [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Mamba(mlp) and GDN architectures with no conv1d, with conv1d (original), and with full Canon. recall and length generalization. However, our experiments ( [PITH_FULL_IMAGE:figures/full_fig_p022_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Final comparison of base architectures equipped with full-score Canon layers: RoPE(ˇ“), NoPE, Mamba2, GLA and GDN. Most notably, with Canon layers added, Mamba2/GLA/GDN still underperform Transformers by 2× in reasoning depth, with meaningful results only for Depo(K=4). innovations may largely replicate Canon-like horizontal mixing rather than introduce fundamentally new computation. While such mechanisms… view at source ↗
Figure 15
Figure 15. Figure 15: Training curves for 12L768D architectures on Depo2(K=4), evaluated at k = 1, 2, 4 and n = N, with results shown across two best LRs for each k. Results for other data are in [PITH_FULL_IMAGE:figures/full_fig_p026_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Performance of 1.3B models pretrained on 100B tokens across discriminative (left), generative (middle), and 1/2-hop reasoning (right) tasks. Best of 2 learning rates for Llama; 3 for GLA, Mamba, and GDN. GPT2 variants (e.g., squared ReLU) shown in [PITH_FULL_IMAGE:figures/full_fig_p027_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Task Lano: our constructed dataset cfg3k,cfg3j against the cfg3f dataset from [6]. <bos> <len_3> + * a b - c d <ans> ans This corresponds to the expression ((a × b) + (c − d)) mod 23, where operands a, b, c, and d are integers sampled uniformly from [0, 22]. The task involves three operations (+, -, *), each represented as distinct tokens, with all computations performed modulo 23. The factual base consis… view at source ↗
Figure 18
Figure 18. Figure 18: A trivial experiment for copying 500 tokens, evaluated only on correctly copying the first [PITH_FULL_IMAGE:figures/full_fig_p037_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: This is an extension of [PITH_FULL_IMAGE:figures/full_fig_p042_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: This is an extension of [PITH_FULL_IMAGE:figures/full_fig_p043_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Extended results to [PITH_FULL_IMAGE:figures/full_fig_p044_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: This is identical to Figure 16 but [PITH_FULL_IMAGE:figures/full_fig_p045_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Results on the Babilong + S-NIAH dataset evaluating multi-hop reasoning across varied junk context lengths. Most architectural comparisons are statistically insignificant. Key findings include: 1. Linear models consistently underperform Transformers, even in short contexts without junk. 2. Models with reduced RoPE (NoPE, RoPEˇ“) achieve notable improvements in long-context accuracy. 3. S-NIAH is too easy:… view at source ↗
Figure 24
Figure 24. Figure 24: Columns 1,2,3: Constant Canon implementation (random, non-trained average of the past 3 tokens, denoted cst-Canon) already achieves strong performance, clearly outperforming vanilla Llama. Columns 2,4,5: Canon layers also perform strongly on GPT2 models (with standard MLP). Our playground reveals standard MLP is slightly weaker than gated MLP, especially in knowledge manipulation (cf. Result 5). 8L512D 12… view at source ↗
Figure 25
Figure 25. Figure 25: Effect of ReLU2 activation on standard vs. gated MLP. Columns 1→2, 5→6: gated MLP outperforms standard MLP with silu. Columns 2→4, 6→8: adding ReLU2 to standard MLP yields slight gains. Columns 1→3, 5→7: adding ReLU2 to gated MLP hurts performance. 46 [PITH_FULL_IMAGE:figures/full_fig_p047_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Transformer+Canon with varying RoPE configurations. From left to right: (1) RoPE; (2) RoPEˇ“ : half of heads each with half RoPE dimensions; (3) RoPEˇ“ˇ“ : a quarter of heads with full RoPE dimensions; (4) RoPEˇ“ˇ“ˇ“ : all heads each with quarter RoPE dimensions; (5) NoPE. Conclusion: Canon layers eliminate the need for extensive RoPE usage, and reducing RoPE usage to 1/4 is even preferable, outperforming… view at source ↗
Figure 27
Figure 27. Figure 27: Llama(RoPE) family: (from left to right) original, Canon-B, -AC, -BD, -ACD, -ABC, -ABCD. This figure complements [PITH_FULL_IMAGE:figures/full_fig_p049_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Llama(RoPE) family: (left to right) original, Canon-B(no-res), Canon-ABCD(res), ˇ“Canon-B(no-res), ˇ“Canon-ABCD(res). This figure complements [PITH_FULL_IMAGE:figures/full_fig_p049_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: Llama(NoPE) family: (from left to right) original, Canon-B, -AC, -BD, -ACD, -ABC, -ABCD. This figure complements [PITH_FULL_IMAGE:figures/full_fig_p050_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: Ablation study of Mamba2 models of 12L768D size with Canon layers, Canon residuals, original non-linear conv1d, mimetic initialization. Full ablation studies (with additional model sizes, such as the effectiveness of Canon-ACD) are in [PITH_FULL_IMAGE:figures/full_fig_p051_30.png] view at source ↗
Figure 31
Figure 31. Figure 31: Mamba2 variants (left to right): original (conv1d), mimetic (w/ conv1d), no conv1d, Canon-AB(no￾res), Canon-AB(res), Canon-Ab(no-res), Canon-Ab(res). 8L512D 12L512D 8L768D 12L768D N=225 N=300 N=375 12/65% 29/67% 24/74% 43/84% 4/35% 12/46% 13/62% 13/61% 1/22% 13/56% 7/33% 10/42% Task Depo1(K=4, k=4/2) Mamba2(mlp) - original (conv1d) 8L512D 12L512D 8L768D 12L768D N=225 N=300 N=375 5/34% 16/55% 6/50% 16/80% … view at source ↗
Figure 32
Figure 32. Figure 32: Mamba2(mlp) variants (left to right): original (conv1d), mimetic (w/ conv1d), no conv1d, Canon￾ABCD(no-res), Canon-ABCD(res), Canon-AbCD(no-res), Canon-AbCD(res), Canon-ACD(res). 51 [PITH_FULL_IMAGE:figures/full_fig_p052_32.png] view at source ↗
Figure 33
Figure 33. Figure 33: Ablation study on 12L768D GLA with Canon/conv1d layers, residual vs. non-residual, identity feature map vs non-linear (ϕ(x) = elu(x) + 1) feature map. 8L512D 12L512D 8L768D 12L768D N=225 N=300 N=375 7/46% 2/14% 14/55% 19/62% 1/11% 1/9% 1/22% 4/31% 0/4% 0/0% 0/6% 1/16% Task Depo1(K=4, k=4/2) GLA - original 8L512D 12L512D 8L768D 12L768D N=225 N=300 N=375 14/70% 37/91% 72/96% 72/97% 2/41% 22/77% 15/80% 27/86… view at source ↗
Figure 34
Figure 34. Figure 34: GLA variants (left to right): original, original + conv1d, original + Canon-ABCD(no-res), Canon￾ABCD(res), Canon-AbCD(no-res), Canon-AbCD(res), Canon-ACD(res). 52 [PITH_FULL_IMAGE:figures/full_fig_p053_34.png] view at source ↗
Figure 35
Figure 35. Figure 35: Ablation study on 12L768D GDN with Canon/conv1d layers, residual vs. non-residual. 53 [PITH_FULL_IMAGE:figures/full_fig_p054_35.png] view at source ↗
Figure 36
Figure 36. Figure 36: GDN variants (left to right): no conv1d, original (w/ conv1d), Canon-ABCD(no-res), Canon￾ABCD(res), Canon-AbCD(no-res), Canon-AbCD(res), Canon-ACD(res). References [1] Marah Abdin, Jyoti Aneja, Harkirat Behl, S´ebastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024. [2] Zeyu… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Smooth Scaling Laws Hide Stepwise Token Learning

    cs.CL 2026-06 conditional novelty 7.0

    Power-law LLM scaling laws are largely the aggregate of stepwise token learning events whose heavy-tailed learning-time spectrum reconstructs loss derivatives along step, data, and model axes.

  2. Smooth Scaling Laws Hide Stepwise Token Learning

    cs.CL 2026-06 unverdicted novelty 7.0

    Token loss trajectories follow localized sigmoids whose learning-time spectrum quantitatively reconstructs scaling-law derivatives on T, D, and M axes and enables faster training via distribution reshaping.

  3. The Key to Going Linear: Analysis-Driven Transformer Linearization

    cs.LG 2026-07 conditional novelty 6.0

    Delta-rule linear attention faithfully approximates softmax attention through key-dependent rank-1 projections, enabling efficient post-hoc linearization of LLMs up to 32B parameters.

  4. Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts

    cs.LG 2026-07 conditional novelty 6.0

    In looped transformers, halting-gate failures come mainly from how gate training reshapes the trajectory; fixed-prior depth supervision plus simple confidence readouts yields better accuracy per unit of compute.

  5. Dynamic Short Convolutions Improve Transformers

    cs.LG 2026-06 unverdicted novelty 6.0

    Dynamic short convolutions applied to key/query/value projections and linear layers in Transformers yield consistent performance gains and 1.33-1.60x compute advantages over standard models on language modeling from 1...

  6. Beyond Similarity: Temporal Operator Attention for Time Series Analysis

    cs.LG 2026-05 unverdicted novelty 6.0

    TOA augments attention with learnable sequence-space operators and stochastic regularization to enable signed temporal mixing, yielding gains on forecasting and related benchmarks when added to PatchTST and iTransformer.

  7. Convergent Evolution: How Different Language Models Learn Similar Number Representations

    cs.CL 2026-04 unverdicted novelty 6.0

    Diverse language models converge on similar periodic number features with a two-tier hierarchy of Fourier sparsity and geometric separability, acquired via language co-occurrences or multi-token arithmetic.

  8. The Topological Trouble With Transformers

    cs.LG 2026-04 conditional novelty 6.0

    Feedforward transformers push each state update into deeper layers, exhausting the model's depth, so the authors argue step-wise recurrence is required and propose a taxonomy of recurrent transformer designs.

  9. ATMA: Length-Invariant Language Modeling via Polar Attention and Gated-Delta Compression Memory

    cs.LG 2026-06 unverdicted novelty 5.0

    ATMA combines three-channel polar attention with gated-delta recurrent memory to achieve monotonic perplexity reduction and over 90% needle-in-haystack accuracy at 32x training context length.

  10. ATMA: Length-Invariant Language Modeling via Polar Attention and Gated-Delta Compression Memory

    cs.LG 2026-06 unverdicted novelty 5.0

    ATMA combines polar attention (direction + bounded-magnitude channels) with gated-delta recurrent compression to achieve length-invariant perplexity and >90% needle retrieval at 64K tokens after 2K training.

  11. Beyond Similarity: Temporal Operator Attention for Time Series Analysis

    cs.LG 2026-05 unverdicted novelty 5.0

    Temporal Operator Attention augments softmax attention with learnable sequence-space operators for signed temporal mixing and uses stochastic regularization to enable practical training, yielding consistent gains on t...

  12. The Topological Trouble With Transformers

    cs.LG 2026-04 unverdicted novelty 5.0

    Transformers face a topological limitation in dynamic state tracking because their feedforward architecture pushes evolving state representations deeper into layers until depth is exhausted, requiring a shift to recur...

  13. Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention

    cs.LG 2026-06 unverdicted novelty 3.0

    Argues that parametric attention forms are necessary for lifelong in-context learning in transformers to maintain constant memory footprint over arbitrary sequence lengths.

  14. The Topological Trouble With Transformers

    cs.LG 2026-04 reject novelty 3.0

    An intermediate-fusion model concatenating ResNet-18 image embeddings with MLP-derived clinical embeddings reports improved mitosis-class AUC, but the image and tabular datasets are unpaired, undermining the fusion claim.

Reference graph

Works this paper leans on

85 extracted references · 50 linked inside Pith · cited by 9 Pith papers

  1. [1]

    Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    Marah Abdin, Jyoti Aneja, Harkirat Behl, S´ ebastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality, 2025

    Zeyuan Allen-Zhu. Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality, 2025. URLhttps://physics.allen-zhu. com/part-4-architecture-design/part-4-2. Code released athttps://github.com/ facebookresearch/PhysicsLM4

    2025

  3. [3]

    Can SGD Learn Recurrent Neural Networks with Provable Gener- alization? InNeurIPS, 2019

    Zeyuan Allen-Zhu and Yuanzhi Li. Can SGD Learn Recurrent Neural Networks with Provable Gener- alization? InNeurIPS, 2019. Full version available athttp://arxiv.org/abs/1902.01028

    Pith/arXiv arXiv 2019

  4. [4]

    Backward Feature Correction: How Deep Learning Performs Deep (Hierarchical) Learning

    Zeyuan Allen-Zhu and Yuanzhi Li. Backward Feature Correction: How Deep Learning Performs Deep (Hierarchical) Learning. InConference on Learning Theory, COLT ’23, 2023. Full version available at http://arxiv.org/abs/2001.04413

    Pith/arXiv arXiv 2023

  5. [5]

    Physics of Language Models: Part 3.1, Knowledge Storage and Extraction

    Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.1, Knowledge Storage and Extraction. InProceedings of the 41st International Conference on Machine Learning, ICML 2024,

    2024

  6. [6]

    Physics of Language Models: Part 1, Learning Hierarchical Language Structures.Transactions on Machine Learning Research, 2025

    Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 1, Learning Hierarchical Language Structures.Transactions on Machine Learning Research, 2025. Full version available athttp://arxiv. org/abs/2305.13673

    Pith/arXiv arXiv 2025

  7. [7]

    Physics of Language Models: Part 3.2, Knowledge Manipulation

    Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.2, Knowledge Manipulation. In Proceedings of the 13th International Conference on Learning Representations, ICLR 2025, 2025. Full version available athttp://arxiv.org/abs/2309.14402

    Pith/arXiv arXiv 2025

  8. [8]

    Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws

    Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws. InProceedings of the 13th International Conference on Learning Representations, ICLR 2025,

    2025

  9. [9]

    Just read twice: closing the recall gap for recurrent language models.arXiv preprint arXiv:2407.05483, 2024

    Simran Arora, Aman Timalsina, Aaryan Singhal, Benjamin Spector, Sabri Eyuboglu, Xinyi Zhao, Ashish Rao, Atri Rudra, and Christopher R´ e. Just read twice: closing the recall gap for recurrent language models.arXiv preprint arXiv:2407.05483, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Titans: Learning to memorize at test time.arXiv preprint arXiv:2501.00663, 2024

    Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time.arXiv preprint arXiv:2501.00663, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Curriculum learning

    Yoshua Bengio, J´ erˆ ome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009

    2009

  12. [12]

    PIQA: Reasoning about physical common- sense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about physical common- sense in natural language. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020

    2020

  13. [13]

    GPT-NeoX-20B: An open-source autoregressive language model

    Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregressive language model. InProceedings of the ACL Workshop on Cha...

    Pith/arXiv arXiv 2022

  14. [14]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  15. [15]

    Rethinking attention with performers.arXiv preprint arXiv:2009.14794, 2020

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers.arXiv preprint arXiv:2009.14794, 2020

    Pith/arXiv arXiv 2009

  16. [16]

    Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1–113, 2023

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1–113, 2023

    2023

  17. [17]

    BoolQ: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2924–2936, 2019. doi: 10.186...

    doi:10.18653/v1/n19-1300 2019

  18. [18]

    Think you have solved question answering? try ARC, the AI2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    Pith/arXiv arXiv 2018

  19. [19]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060, 2024

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060, 2024. URLhttps://arxiv.org/abs/ 2405.21060

    Pith/arXiv arXiv 2024

  20. [20]

    Griffin: Mixing gated linear recurrences with local attention for efficient language models.arXiv preprint arXiv:2402.19427, 2024

    Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models.arXiv preprint arXiv:2402.19427, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)...

    doi:10.18653/v1/n19-1246 2019

  22. [22]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.The Journal of Machine Learning Research, 23(1):5232–5270, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.The Journal of Machine Learning Research, 23(1):5232–5270, 2022

    2022

  23. [24]

    A framework for few-shot language model evaluation, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework...

    arXiv 2024

  24. [25]

    Multi-token attention.arXiv preprint arXiv:2504.00927, 2025

    Olga Golovneva, Tianlu Wang, Jason Weston, and Sainbayar Sukhbaatar. Multi-token attention.arXiv preprint arXiv:2504.00927, 2025

    Pith/arXiv arXiv 2025

  25. [26]

    Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023. URLhttps://arxiv.org/abs/2312.00752

    Pith/arXiv arXiv 2023

  26. [27]

    Conformer: Convolution-augmented transformer for speech recognition.arXiv preprint arXiv:2005.08100, 2020

    Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmented transformer for speech recognition.arXiv preprint arXiv:2005.08100, 2020

    Pith/arXiv arXiv 2005

  27. [28]

    Deberta: Decoding-enhanced bert with disentangled attention.arXiv preprint arXiv:2006.03654, 2020

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention.arXiv preprint arXiv:2006.03654, 2020

    Pith/arXiv arXiv 2006

  28. [29]

    Ruler: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

    Pith/arXiv arXiv 2024

  29. [30]

    Tutel: Adaptive mixture-of-experts at scale.CoRR, abs/2206.03382, June 2022

    Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, Joe Chau, Peng Cheng, Fan Yang, Mao Yang, and Yongqiang Xiong. Tutel: Adaptive mixture-of-experts at scale.CoRR, abs/2206.03382, June 2022. URLhttps://arxiv.org/ pdf/2206.03382.pdf

    Pith/arXiv arXiv 2022

  30. [31]

    Repeat after me: Transformers are better than state space models at copying.arXiv preprint arXiv:2402.01032, 2024

    Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying.arXiv preprint arXiv:2402.01032, 2024

    Pith/arXiv arXiv 2024

  31. [32]

    Survey of hallucination in natural language generation.ACM Computing Surveys, 55(12):1–38, 2023

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation.ACM Computing Surveys, 55(12):1–38, 2023. doi: 10.1145/3571730. URLhttps://doi.org/10.1145/3571730

    doi:10.1145/3571730 2023

  32. [33]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    Pith/arXiv arXiv 2023

  33. [34]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, 2017

    2017

  34. [35]

    Transformers are RNNs: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran¸ cois Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. InInternational conference on machine learning, pages 5156–5165. PMLR, 2020

    2020

  35. [36]

    Bert: Pre-training of deep bidi- rectional transformers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. InProceedings of NAACL-HLT, pages 4171–4186, 2019

    2019

  36. [37]

    Babilong: Testing the limits of llms with long context reasoning-in-a-haystack.Ad- vances in Neural Information Processing Systems, 37:106519–106554, 2024

    Yury Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack.Ad- vances in Neural Information Processing Systems, 37:106519–106554, 2024

    2024

  37. [38]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural ques- tions: A benchmark for question answering research.Transa...

    doi:10.1162/tacl 2019

  38. [39]

    Self- improving transformers overcome easy-to-hard and length generalization challenges.arXiv preprint arXiv:2502.01612, 2025

    Nayoung Lee, Ziyang Cai, Avi Schwarzschild, Kangwook Lee, and Dimitris Papailiopoulos. Self- improving transformers overcome easy-to-hard and length generalization challenges.arXiv preprint arXiv:2502.01612, 2025. URLhttps://arxiv.org/abs/2502.01612

    Pith/arXiv arXiv 2025

  39. [40]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 56

    2023

  40. [41]

    The LAMBADA dataset: Word prediction requiring a broad discourse context

    Denis Paperno, Germ´ an Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern´ andez. The LAMBADA dataset: Word prediction requiring a broad discourse context. InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1...

    doi:10.18653/v1/p16-1144 2016

  41. [42]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydl ´ ıˇ cek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URLhttps://arxiv.org/abs/2406.17557

    Pith/arXiv arXiv 2024

  42. [43]

    Rwkv: Reinventing rnns for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023

    Pith/arXiv arXiv 2023

  43. [44]

    Grokking: General- ization beyond overfitting on small algorithmic datasets.arXiv preprint arXiv:2201.02177, 2022

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: General- ization beyond overfitting on small algorithmic datasets.arXiv preprint arXiv:2201.02177, 2022. URL https://arxiv.org/abs/2201.02177

    Pith/arXiv arXiv 2022

  44. [45]

    Train short, test long: Attention with linear biases enables input length extrapolation.arXiv preprint arXiv:2108.12409, 2021

    Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation.arXiv preprint arXiv:2108.12409, 2021

    Pith/arXiv arXiv 2021

  45. [46]

    Hierarchically gated recurrent neural network for sequence modeling.Advances in Neural Information Processing Systems, 36:33202–33221, 2023

    Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling.Advances in Neural Information Processing Systems, 36:33202–33221, 2023

    2023

  46. [47]

    Qwen3-Next: Towards Ultimate Training & Inference Efficiency, Sep 2025

    QwenTeam. Qwen3-Next: Towards Ultimate Training & Inference Efficiency, Sep 2025. URL https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd&from=research. latest-advancements-list

    2025

  47. [48]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  48. [49]

    Know what you don’t know: Unanswerable questions for SQuAD

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-2124. URLhttps://aclantholog...

    doi:10.18653/v1/p18-2124 2018

  49. [50]

    Samba: Simple hybrid state space models for efficient unlimited context language modeling.arXiv preprint arXiv:2406.07522, 2024

    Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling.arXiv preprint arXiv:2406.07522, 2024

    Pith/arXiv arXiv 2024

  50. [51]

    WinoGrande: An adversarial winograd schema challenge at scale.arXiv preprint arXiv:1907.10641, 2019

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale.arXiv preprint arXiv:1907.10641, 2019

    Pith/arXiv arXiv 1907

  51. [52]

    Socialiqa: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. InProceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463–4473, 2019. doi: 10.18653/v1/D19-1454

    doi:10.18653/v1/d19-1454 2019

  52. [53]

    Teaching transformers modular arithmetic at scale.arXiv preprint arXiv:2410.03569, 2024

    Eshika Saxena, Alberto Alfarano, Emily Wenger, and Kristin Lauter. Teaching transformers modular arithmetic at scale.arXiv preprint arXiv:2410.03569, 2024. URLhttps://arxiv.org/abs/2410. 03569

    Pith/arXiv arXiv 2024

  53. [54]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  54. [55]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  55. [56]

    Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Noam Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Pith/arXiv arXiv 2002

  56. [57]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. InInternational Conference on Learning Representations, 2016

    2016

  57. [58]

    Simplified state space layers for 57 sequence modeling.arXiv preprint arXiv:2208.04933, 2022

    Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for 57 sequence modeling.arXiv preprint arXiv:2208.04933, 2022

    Pith/arXiv arXiv 2022

  58. [59]

    Primer: Searching for efficient transformers for language modeling

    DR So, W Manke, H Liu, Z Dai, N Shazeer, and QV Le. Primer: Searching for efficient transformers for language modeling. arxiv 2021.arXiv preprint arXiv:2109.08668, 2021

    Pith/arXiv arXiv 2021

  59. [60]

    SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.https://www.cerebras

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.https://www.cerebras. net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, June

  60. [61]

    Roformer: Enhanced transformer with rotary position embedding, 2021

    Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021

    2021

  61. [62]

    Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    Pith/arXiv arXiv 2023

  62. [63]

    Falcon-h1: A family of hybrid-head language models redefining efficiency and performance, May 2025

    Falcon-LLM Team. Falcon-h1: A family of hybrid-head language models redefining efficiency and performance, May 2025. URLhttps://falcon-lm.github.io/blog/falcon-h1

    2025

  63. [64]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  64. [65]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  65. [66]

    Mimetic initialization helps state space models learn to recall.arXiv preprint arXiv:2410.11135, 2024

    Asher Trockman, Hrayr Harutyunyan, J Zico Kolter, Sanjiv Kumar, and Srinadh Bhojanapalli. Mimetic initialization helps state space models learn to recall.arXiv preprint arXiv:2410.11135, 2024

    Pith/arXiv arXiv 2024

  66. [67]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  67. [68]

    Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768, 2020

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768, 2020

    Pith/arXiv arXiv 2006

  68. [69]

    Towards ai-complete question answering: A set of prerequisite toy tasks

    Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merri¨ enboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015

    Pith/arXiv arXiv 2015

  69. [70]

    Cvt: Intro- ducing convolutions to vision transformers

    Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Intro- ducing convolutions to vision transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 22–31, 2021

    2021

  70. [71]

    Fla: A triton-based library for hardware-efficient implementations of linear attention mechanism, January 2024

    Songlin Yang and Yu Zhang. Fla: A triton-based library for hardware-efficient implementations of linear attention mechanism, January 2024. URLhttps://github.com/fla-org/flash-linear-attention

    2024

  71. [72]

    Gated linear attention transformers with hardware-efficient training.arXiv preprint arXiv:2312.06635, 2023

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training.arXiv preprint arXiv:2312.06635, 2023

    Pith/arXiv arXiv 2023

  72. [73]

    Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464, 2024

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464, 2024

    Pith/arXiv arXiv 2024

  73. [74]

    Parallelizing linear transformers with the delta rule over sequence length.arXiv preprint arXiv:2406.06484, 2024

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length.arXiv preprint arXiv:2406.06484, 2024

    Pith/arXiv arXiv 2024

  74. [75]

    Physics of Language Models: Part 2.1, Grade- School Math and the Hidden Reasoning Process

    Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of Language Models: Part 2.1, Grade- School Math and the Hidden Reasoning Process. InProceedings of the 13th International Conference on Learning Representations, ICLR 2025, 2025. Full version available athttps://arxiv.org/abs/2407. 20311

    2025

  75. [76]

    Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems

    Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems. InProceedings of the 13th International Conference on Learning Representations, ICLR 2025, 2025. Full version available athttp://arxiv. org/abs/2408.16293. 58

    Pith/arXiv arXiv 2025

  76. [77]

    Distilling system 2 into system 1.arXiv preprint arXiv:2407.06023, 2024

    Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system 1.arXiv preprint arXiv:2407.06023, 2024

    Pith/arXiv arXiv 2024

  77. [78]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention.arXiv preprint arXiv:2502.11089, 2025

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention.arXiv preprint arXiv:2502.11089, 2025

    Pith/arXiv arXiv 2025

  78. [79]

    HellaSwag: Can a ma- chine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a ma- chine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019. doi: 10.18653/v1/P19-1472

    doi:10.18653/v1/p19-1472 2019

  79. [80]

    Gated slot attention for efficient linear-time sequence modeling.Advances in Neural Information Processing Systems, 37:116870–116898, 2024

    Yu Zhang, Songlin Yang, Rui-Jie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, et al. Gated slot attention for efficient linear-time sequence modeling.Advances in Neural Information Processing Systems, 37:116870–116898, 2024

    2024

  80. [81]

    Relu 2 wins: Discovering efficient activation functions for sparse llms.arXiv preprint arXiv:2402.03804, 2024

    Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song, Zhiyuan Liu, Zeyu Mi, and Maosong Sun. Relu 2 wins: Discovering efficient activation functions for sparse llms.arXiv preprint arXiv:2402.03804, 2024

    Pith/arXiv arXiv 2024

Showing first 80 references.