Pith. sign in

REVIEW 3 major objections 4 minor 87 references

This paper claims that a Transformer with a gated recurrent pathway running through its middle layers outperforms matched Transformers on reasoning tasks, adding only a few percent of inference overhead.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 23:54 UTC pith:KWHR2GVD

load-bearing objection A genuinely new recurrent pathway for Transformers — fusing a cached middle-layer state into an earlier layer of the next token — with honest controlled experiments that are not yet fully training-compute-matched. the 3 major comments →

arxiv 2607.15178 v2 pith:KWHR2GVD submitted 2026-07-16 cs.CL cs.AI

T²MLR: Transformer with Temporal Middle-Layer Recurrence

Ziyang Cai , Xingyu Zhu , Yihe Dong , Yinghui He , Sanjeev Arora This is my paper
classification cs.CL cs.AI
keywords T2MLRmiddle-layer recurrencelatent reasoningTransformersautoregressive decodingstate trackingJacobi approximationtemporal recurrence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the token-level information bottleneck of autoregressive Transformers can be relaxed by carrying a cached representation from the middle layers of the previous token into an earlier middle layer of the current token. The proposed architecture, T2MLR, fuses this cached state through a lightweight gated module, allowing intermediate reasoning states to persist and evolve across decoding steps without looping over the whole network. In data- and parameter-matched comparisons, T2MLR outperforms plain Transformers on a synthetic state-tracking-plus-retrieval task, on pretraining perplexity and downstream NLP benchmarks, and on finetuned multi-hop and math reasoning tasks. The authors also show that applying recurrence to only a localized middle block (as little as 20% of the layers) often beats full-layer recurrence, and that the pathway can be grafted onto an existing pretrained 1.7B model, raising grade-school math accuracy from 35.8% to 39.9% and a harder math benchmark from 12.8% to 18.0% after brief finetuning. The practical stake is that effective latent reasoning may not require expensive full-depth loops; targeted middle-layer recurrence can deliver similar or better gains at near-standard decoding cost.

Core claim

The central discovery is that temporal recurrence placed specifically in the middle layers of a Transformer yields reasoning gains that full-layer recurrence or no recurrence do not. Concretely, the paper defines T2MLR by choosing two layer indices, ℓ_start and ℓ_end, maintaining a recurrent cache R_t that stores a normalized sum of the current token's layer-ℓ_end representation and the previous cache, and fusing R_{t-1} into the current token's representation just before layer ℓ_start via a gated sum. This establishes a deep-to-shallow shortcut: information computed late for token t−1 becomes an explicit input early for token t, without changing the autoregressive interface. On the S5-Retri

What carries the argument

The load-bearing mechanism is the gated fusion module Φ combined with the recurrent cache update. Φ takes the current token's pre-layer-ℓ_start representation h_t and the previous recurrent cache R_{t-1}, and computes a residual-style update with two input-dependent sigmoid gates and two scalar tanh gates (γ_cur, γ_rec), so the recurrent stream is added only when it contributes beyond the current-token stream. The cache is updated after layer ℓ_end as R_t = RMSNorm(h_t^{ℓ_end} + R_{t-1}), making the recurrence a temporal residual. A second, equally essential mechanism is the Jacobi fixed-point training scheme: for scalable teacher-forced training, the cache for all token positions is approxi

Load-bearing premise

The load-bearing premise is that 16 Jacobi fixed-point iterations during training produce a recurrent cache close enough to the exact sequential recurrence for every task, scale, and difficulty level tested; if that approximation degrades with task difficulty or interacts with model scale, the reported gains could be an artifact of the approximation rather than of middle-layer recurrence.

What would settle it

Train a small T2MLR with exact sequential recurrent states (no Jacobi approximation) on S5-Retrieval and on one math task, and compare to the Jacobi-trained model and to the baseline Transformer. If exact-recurrence training does not reproduce or improve upon the reported gains over the baseline, the benefit depends on the approximation rather than on the temporal recurrent pathway. Alternatively, zero out the trained fusion gates (set γ_rec=0) at inference and measure accuracy: if accuracy does not fall toward the baseline, the recurrent pathway is not the cause of the improvement.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If middle-layer recurrence is the right inductive bias, latent-reasoning architectures do not need to loop over all layers; routing recurrence through a localized block can give better or equal gains with far less compute.
  • Because the pathway is a simple gated residual added to an existing Transformer, it can be retrofitted into already-pretrained models with brief finetuning, so the benefit may be achievable without a costly full pretraining run.
  • At inference, T2MLR adds only a constant gated-fusion cost per token (measured at most ~8% overhead, decreasing with model size and generation length), making it attractive for deployment where looped or pause-token methods multiply per-token compute.
  • The gains persist and grow when scaling from 135M to 1B parameters and from 10B to 50B tokens, and a fixed-width ablation shows that middle-layer placement, not just recurrence depth, is what drives the improvement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the recurrent cache is a fixed-size vector, it functions as a learned, task-dependent memory of the recent reasoning state; one could test whether combining T2MLR with long-context segment-level memory compounds the benefit.
  • The Jacobi approximation's success suggests that the exact recurrence may not be needed during training; a cheaper alternative like a learned predictor of the cache or a lower refinement depth could be explored, especially for RL settings where exact states are naturally produced during rollout.
  • The finding that middle-layer recurrence beats full-layer recurrence hints that placing recurrence at the depth where abstract computation is most active is a general design principle, which might transfer to other architectures such as mixture-of-experts or state-space models.
  • One could test the robustness of the claim by varying the task family: if middle-layer recurrence mainly helps tasks that require maintaining a continuous latent state (state tracking, multi-hop composition), it may not help purely lexical associative tasks; the paper's task set samples this space but not exhaustively.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces T2MLR, a Transformer variant that maintains a fixed-size recurrent cache computed from the representation after a middle layer ℓ_end at token t and fuses it into the representation before an earlier middle layer ℓ_start at token t+1 via a gated fusion module. Training uses a temporal-parallel Jacobi approximation (Algorithm 1, d_forward=16, d_backward=4) to avoid sequential recurrence, with inference remaining a standard single forward pass plus a small fusion overhead. The authors evaluate T2MLR on a purpose-built S5-Retrieval task, on 135M–1B parameter pretraining with up to 50B FineWeb-Edu tokens, on finetuned reasoning tasks (Variable Assignment, ProsQA-Hard, HotpotQA-Simple, GSM-Aug), and on retrofitting a SmolLM2-1.7B-Instruct model with continued finetuning. Headline claims are that T2MLR consistently outperforms data- and parameter-matched Transformers, that middle-layer recurrence often outperforms full-layer recurrence, that the Jacobi approximation closely matches exact recurrent training, and that the architecture adds at most ~8% per-token inference overhead.

Significance. If the central claim holds, the paper makes a meaningful contribution: it identifies middle-layer recurrence as a high-leverage design choice, shows that approximate temporal-parallel training can scale latent-reasoning architectures under dense teacher forcing, and demonstrates a practical retrofit path for existing pretrained models. Strengths of the manuscript include the parameter/data-matched comparisons in Tables 1, 2, 9, and 10; the direct comparison of Jacobi-trained versus exact-recurrent-trained models at three scales in Appendix B.4; the gradient-similarity analysis in B.3; the explicit inference-overhead measurements in B.5; and the unusually transparent statement of the training-compute tradeoff in Section 6. The main weakness is that the reasoning gains are not yet separated from the substantially larger training budget of T2MLR, and all headline results are single runs without variance estimates. These issues are fixable and do not invalidate the architectural idea, but they must be addressed before the central reasoning claim can be considered established.

major comments (3)
  1. [Section 4.3 / Figure 6; Section C.2 / Table 12] The central claim that T2MLR improves reasoning is not yet isolated from training-compute differences. All reasoning-task comparisons (Variable Assignment, ProsQA-Hard, HotpotQA-Simple, GSM-Aug, and the 1.7B retrofit in Table 3) use a baseline trained for one epoch, while T2MLR's Jacobi refinement makes training roughly 2.24× more expensive for the (13,18) configuration. The only training-compute-matched control, Table 12, is limited to zero-shot NLP and shows that a Transformer trained for 2.24 epochs surpasses T2MLR (45.30 vs 44.14 average). No equivalent control is provided for the reasoning suite or for Figure 6. Similarly, the comparison across recurrence boundaries is not compute-matched: full recurrence D=30 costs more per step than D=6, so the finding that middle-layer recurrence outperforms full-layer recurrence could reflect an optimization-budget effect rather than recurrence
  2. [Section 6; Tables 1, 2, 9, 10; Figure 6; Table 3] All headline results are single runs, with no error bars, confidence intervals, or seed variance. Many zero-shot differences are small in absolute terms (e.g., 42.83 vs 44.14 average in Table 1, individual task gaps of 1–3 points), and the reasoning-task gaps in Figure 6 are plotted without quantifying run-to-run variability. Section 6 defers multi-seed variance to future work, but given the small effect sizes, this is load-bearing for the 'consistently outperforms' claim. Reporting at least 3–5 seeds for the principal pretraining and reasoning comparisons, with mean and standard deviation, would materially strengthen the paper.
  3. [Section 2.4; Appendix B.4; Appendix B.3] The entire training pipeline rests on the assumption that d_forward=16 Jacobi iterations produce a cache close enough to exact sequential recurrence for all tested tasks and recurrence boundaries. Appendix B.4 validates this on validation loss/perplexity at three scales for ℓ_start=8 after 1B tokens, and B.3 reports gradient similarity for one checkpoint. However, no convergence guarantee is given, and the validation does not cover the downstream reasoning tasks or the full range of recurrence placements (D=2 through D=30) used in the main experiments. If Jacobi error interacts with task difficulty or recurrence location, the reported gains could depend on the approximation rather than on middle-layer recurrence per se. Please include an ablation of d_forward on at least one reasoning finetuning task and an exact-recurrent rollout evaluation for the main model variants, or otherwise boun
minor comments (4)
  1. [Abstract / Section 6] The abstract's 'consistently outperforms data- and parameter-matched Transformer baselines' is technically accurate but should be qualified as 'under fixed parameter/data/inference-compute budgets,' given the training-compute-matched result in Table 12. The paper already does this in Section 6, but the qualification should appear where the headline claim is made.
  2. [Section 2.3] Equation (2.3) states that the scalar gates are initialized to zero and 'the gates randomly'; if γ_cur and γ_rec are initialized to zero, the tanh factors are zero and the gated terms are identically zero, so the phrase 'gates randomly' is misleading. Please clarify the exact initialization and what is random.
  3. [Section 2.4] The sentence 'T2MLR cannot directly adopt the standard token-parallel training procedure used by vanilla Transformers' is followed by a description that relies on 'ShiftRight' and cache indexing (R^⟨1⟩, R^⟨2⟩). The index alignment in Algorithm 1 between the initial cache seed and the first recurrent refinement is easy to misread; a short worked example or a note that R^⟨1⟩ corresponds to the cache for the first token position would help.
  4. [Section 4.3 / Appendix F.2] The HotpotQA result is restricted to the easy subset and uses GPT-4o-mini-generated CoT data; the paper is transparent about this, but the main-text phrasing 'HotpotQA' could oversell the finding. Please state 'easy subset' in the main text whenever the result is discussed, and consider reporting a filtered accuracy on medium/hard for completeness.

Circularity Check

0 steps flagged

No circularity found: T2MLR is an empirically evaluated architecture whose claims rest on external benchmarks and controlled comparisons, not on fitted parameters renamed as predictions.

full rationale

The paper's central claims — that middle-layer temporal recurrence improves reasoning over matched Transformers and that it can be retrofitted into a pretrained model — are supported by direct experiments on external benchmarks (FineWeb-Edu pretraining, GSM8K, MATH500, HotPotQA, ARC, etc.), not by a derivation that assumes the conclusion. The architecture's recurrence location is swept across multiple boundaries, and the empirical finding that localized middle-layer recurrence outperforms full recurrence is a comparison result, not a definitional consequence. The Jacobi approximation used in training is not treated as a prediction: it is an algorithmic approximation, and the paper explicitly compares it against exact sequential recurrent training (Table 6, Section B.3-B.4), showing near-equal validation loss. The S5-Retrieval task is purpose-built to highlight the intended inductive bias, but the evidence there is empirical accuracy, and the baselines are external architectures; this is experimental design, not circular reasoning. No load-bearing self-citation chain is present: the cited mechanistic and theoretical results (Saunshi et al., Merrill et al., Li et al., Wu et al.) are external works, and no uniqueness or ansatz is imported from the authors' own prior publications. The training-compute-matched comparison in Table 12 and the limitation statement in Section 6 concern whether the reported advantage is attributable to training budget rather than to the architecture — a validity/fairness concern, not a circularity concern. Accordingly, the paper does not reduce any of its outputs to its inputs by construction, and the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim rests on a handful of architectural choices: recurrence boundaries and the Jacobi approximation depths, which are selected/tuned rather than derived, plus the domain assumption that middle layers are where abstraction lives. No new physical entities are introduced; the recurrent cache and fusion modules are implemented model components rather than hypothesized entities with external falsifiable signatures.

free parameters (3)
  • Recurrence boundaries (ℓ_start, ℓ_end) = (13,18), (9,22), (5,26), (1,30) at 135M; (5,28) for 1.7B
    Architecture hyperparameters scanned over layer positions; the headline 'middle-layer recurrence is best' depends on selecting these locations, so they are free parameters of the empirical claim.
  • d_forward (Jacobi forward depth) = 16
    Chosen as a balance between approximation quality and training cost; validated empirically in B.3, not derived.
  • d_backward (TBPTT depth) = 4
    Truncated backpropagation depth selected for efficiency; B.3 shows gradients can deviate substantially when d_backward ≤ 4.
axioms (4)
  • domain assumption Middle layers are the primary locus of abstract reasoning, so injecting recurrence there is beneficial.
    Invoked in Section 2.1 and the recurrence-location ablation in C.1; motivates the architecture but is not proven by this paper.
  • ad hoc to paper The constant-depth Jacobi fixed-point iteration approximates exact sequence recurrence well enough for training.
    Section 2.4 and B.3/B.4 rely on this; no convergence theorem is given, only empirical gradient and loss comparisons.
  • ad hoc to paper The cache update R_t = RMSNorm(h^{ℓ_end}_t + R_{t-1}) is a sufficient encoding of prior reasoning state.
    Equation 2.4 is a design choice; the paper does not compare alternatives such as gated recurrent cache updates.
  • standard math Cited circuit-complexity lower bounds for S5 state tracking are valid for interpreting baseline limits.
    Section 3.1 cites Merrill et al. (2024) and Li et al. (2025); these are external theoretical results used to interpret the synthetic experiment.

pith-pipeline@v1.3.0-alltime-deepseek · 27173 in / 15238 out tokens · 125761 ms · 2026-08-01T23:54:45.447415+00:00 · methodology

0 comments
read the original abstract

Transformer reasoning is limited by autoregressive decoding, which repeat edly compresses rich hidden computation through token space and makes it difficult for intermediate reasoning states to persist across time. We in troduce Transformers with Temporal Middle-Layer Recurrence (T2MLR), a transformers-based latent reasoning architecture that fuses a cached middle layer representation from the previous token directly into an earlier layer of the current token position, enabling abstract intermediate computation to persist across decoding steps with little inference overhead. Across natural-language pretraining and multi-hop reasoning finetuning, T2MLR consistently outperforms data- and parameter-matched Transformer base lines. Moreover, applying recurrence to only a localized middle-layer block (as little as 20% of the network) often outperforms full-layer recurrence. Im portantly, T2MLR does not require pretraining from scratch: retrofitting the recurrent pathway into an existing pretrained 1.7B Transformer and briefly finetuning substantially improves math reasoning, lowering the barrier to practical adoption. These results suggest that effective latent reasoning in Transformers does not require looping over all layers as in previous works, but can instead emerge more strongly from targeted middle-layer recurrence.

Figures

Figures reproduced from arXiv: 2607.15178 by Sanjeev Arora, Xingyu Zhu, Yihe Dong, Yinghui He, Ziyang Cai.

Figure 1
Figure 1. Figure 1: T2MLR fuses representation from a deep layer from the previous token position into a shallow layer of the current token position (left). It gets better pretraining perplexity (see Section 4.1) and reasoning downstream performance (see Section 4.3 ) reasoning is known to occur more prominently (Tenney et al., 2019; Geva et al., 2021; Meng et al., 2022; Saunshi et al., 2024; Atanas & Liu, 2025). Concurrently… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of the T2MLR architecture (left) and the representation fusion module Φ (right). When doing forward pass for the t-th token, we use the representation after layer ℓend to update the recurrent cache Rt . During the forward pass for the t + 1-th token, Rt is then merged with the representation before layer ℓstart via the representation fusion module. 4 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Approximated training scheme of T2MLR. The ground truth residual cache R ∗ is approximated by a Jacobi iteration passing through the middle layers for dforward times. design particularly helpful for early training stability as we could initialize γ’s to zero and the gates randomly. The fused representation Φ(h (ℓstart−1) t , Rt−1) the follows the standard transformers forward pass from ℓstart to layer ℓend… view at source ↗
Figure 4
Figure 4. Figure 4: Performance comparison between LSTM, Transformer, and T [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Validation cross￾entropy loss vs ℓstart for language modeling pre￾training runs at 16k steps. We further find that these improvements are not specific to the 135M from-scratch setting: they persist and grow when scaling to 361M and 1B parameters and when extending pretraining to 50B tokens, and a fixed-width ablation confirms that middle-layer placement remains optimal at fixed recurrence depth. We defer t… view at source ↗
Figure 6
Figure 6. Figure 6: Fine-tuning performance comparison between variants of T [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Cosine similarity and relative ℓ2 distance between gradients attained under different setup of (dforward, dbackward) against setting dforward = 32, dbackward = 32. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Plot of gating parameters γcur and γrec with respect to training steps on the FineWeb dataset. This illustrates the learned behavior of input and recurrent gating over the course of training. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

87 extracted references · 12 linked inside Pith

  1. [1]

    2025 , month = jul, url =

    SmolLM3: smol, multilingual, long-context reasoner , author =. 2025 , month = jul, url =

    2025

  2. [2]

    2025 , eprint=

    Hybrid Latent Reasoning via Reinforcement Learning , author=. 2025 , eprint=

    2025

  3. [3]

    ACL , year=

    BERT Rediscovers the Classical NLP Pipeline , author=. ACL , year=

  4. [4]

    ACL , year=

    Analyzing the Structure of Attention in a Transformer Language Model , author=. ACL , year=

  5. [5]

    EMNLP , year=

    Transformer Feed-Forward Layers Are Key-Value Memories , author=. EMNLP , year=

  6. [6]

    NeurIPS , year=

    Locating and Editing Factual Associations in GPT , author=. NeurIPS , year=

  7. [7]

    arXiv preprint arXiv:2402.01739 , year=

    OpenMoE: An Energy-Efficient Multi-Expert Language Model , author=. arXiv preprint arXiv:2402.01739 , year=

    Pith/arXiv arXiv

  8. [8]

    Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS) , year=

    Gated Sparse Attention: Redefining Routing for Billion-Parameter Models , author=. Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS) , year=

  9. [9]

    JMLR , volume=

    Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity , author=. JMLR , volume=

  10. [10]

    ICLR , year=

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer , author=. ICLR , year=

  11. [11]

    EMNLP , year=

    Learning Phrase Representations using RNN Encoder--Decoder for Statistical Machine Translation , author=. EMNLP , year=

  12. [12]

    Neural Computation , volume=

    Long Short-Term Memory , author=. Neural Computation , volume=

  13. [13]

    2024 , eprint =

    Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries , author =. 2024 , eprint =

    2024

  14. [14]

    2024 , eprint =

    Do Large Language Models Latently Perform Multi-Hop Reasoning? , author =. 2024 , eprint =

    2024

  15. [15]

    2025 , eprint =

    Efficient Reasoning with Hidden Thinking , author =. 2025 , eprint =

    2025

  16. [16]

    2024 , eprint =

    Let's Think Dot by Dot: Hidden Computation in Transformer Language Models , author =. 2024 , eprint =

    2024

  17. [17]

    2023 , eprint =

    Implicit Chain of Thought Reasoning via Knowledge Distillation , author =. 2023 , eprint =

    2023

  18. [18]

    2024 , eprint =

    From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step , author =. 2024 , eprint =

    2024

  19. [19]

    2024 , eprint =

    Think before you speak: Training Language Models With Pause Tokens , author =. 2024 , eprint =

    2024

  20. [20]

    Forty-second International Conference on Machine Learning (ICML) , year =

    Layer by Layer: Uncovering Hidden Representations in Language Models , author =. Forty-second International Conference on Machine Learning (ICML) , year =

  21. [21]

    2025 , eprint =

    AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset , author =. 2025 , eprint =

    2025

  22. [22]

    2024 , eprint =

    Training Large Language Models to Reason in a Continuous Latent Space , author =. 2024 , eprint =

    2024

  23. [23]

    2025 , eprint =

    LLM Pretraining with Continuous Concepts , author =. 2025 , eprint =

    2025

  24. [24]

    2025 , eprint =

    CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation , author =. 2025 , eprint =

    2025

  25. [25]

    2025 , eprint =

    Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning , author =. 2025 , eprint =

    2025

  26. [26]

    2025 , eprint =

    Latent Thought Models with Variational Bayes Inference-Time Computation , author =. 2025 , eprint =

    2025

  27. [27]

    2023 , eprint =

    LaRS: Latent Reasoning Skills for Chain-of-Thought Reasoning , author =. 2023 , eprint =

    2023

  28. [28]

    2019 , eprint =

    Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context , author =. 2019 , eprint =

    2019

  29. [29]

    2019 , eprint =

    Compressive Transformers for Long-Range Sequence Modelling , author =. 2019 , eprint =

    2019

  30. [30]

    2022 , eprint =

    Recurrent Memory Transformer , author =. 2022 , eprint =

    2022

  31. [31]

    Advances in Neural Information Processing Systems , year=

    Attention Is All You Need , author=. Advances in Neural Information Processing Systems , year=

  32. [32]

    International Conference on Learning Representations , year=

    Universal Transformers , author=. International Conference on Learning Representations , year=

  33. [33]

    arXiv preprint arXiv:2108.12409 , year=

    Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation , author=. arXiv preprint arXiv:2108.12409 , year=

    Pith/arXiv arXiv

  34. [34]

    Transactions of the Association for Computational Linguistics , year=

    Theoretical Limitations of Self-Attention in Neural Sequence Models , author=. Transactions of the Association for Computational Linguistics , year=

  35. [35]

    arXiv , year =

    Merrill, William and Sabharwal, Ashish , title =. arXiv , year =

  36. [36]

    arXiv preprint arXiv:2303.08774 , year=

    GPT-4 Technical Report , author=. arXiv preprint arXiv:2303.08774 , year=

    Pith/arXiv arXiv

  37. [37]

    International Conference on Learning Representations , year =

    GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding , author =. International Conference on Learning Representations , year =

  38. [38]

    Advances in Neural Information Processing Systems , year =

    Training Very Deep Networks , author =. Advances in Neural Information Processing Systems , year =

  39. [39]

    Proceedings of the AAAI Conference on Artificial Intelligence , year =

    ReZero is All You Need: Fast Convergence at Large Depth , author =. Proceedings of the AAAI Conference on Artificial Intelligence , year =

  40. [40]

    arXiv preprint arXiv:2201.11903 , year=

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author=. arXiv preprint arXiv:2201.11903 , year=

    Pith/arXiv arXiv

  41. [41]

    International Conference on Learning Representations (ICLR) , year=

    Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues , author=. International Conference on Learning Representations (ICLR) , year=

  42. [42]

    Proceedings of the 41st International Conference on Machine Learning (ICML) , year=

    Repeat After Me: Transformers are Better than State Space Models at Copying , author=. Proceedings of the 41st International Conference on Machine Learning (ICML) , year=

  43. [43]

    International Conference on Learning Representations (ICLR) , year=

    Understanding Factual Recall in Transformers via Associative Memories , author=. International Conference on Learning Representations (ICLR) , year=

  44. [44]

    2024 , eprint =

    Titans: Learning to Memorize at Test Time , author =. 2024 , eprint =

    2024

  45. [45]

    2021 , eprint =

    Efficiently Modeling Long Sequences with Structured State Spaces , author =. 2021 , eprint =

    2021

  46. [46]

    2023 , eprint =

    Hyena Hierarchy: Towards Larger Convolutional Language Models , author =. 2023 , eprint =

    2023

  47. [47]

    2023 , eprint =

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces , author =. 2023 , eprint =

    2023

  48. [48]

    2023 , eprint =

    Retentive Network: A Successor to Transformer for Large Language Models , author =. 2023 , eprint =

    2023

  49. [49]

    2024 , eprint =

    Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models , author =. 2024 , eprint =

    2024

  50. [50]

    2024 , eprint =

    Jamba: A Hybrid Transformer-Mamba Language Model , author =. 2024 , eprint =

    2024

  51. [51]

    arXiv preprint arXiv:2110.14168 , year=

    Training Verifiers to Solve Math Word Problems , author=. arXiv preprint arXiv:2110.14168 , year=

    Pith/arXiv arXiv

  52. [52]

    Forty-first International Conference on Machine Learning , year=

    The Illusion of State in State-Space Models , author=. Forty-first International Conference on Machine Learning , year=

  53. [53]

    2024 , eprint=

    The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale , author=. 2024 , eprint=

    2024

  54. [54]

    2025 , eprint=

    SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model , author=. 2025 , eprint=

    2025

  55. [55]

    Soft Thinking: Unlocking the Reasoning Potential of

    Zhen Zhang and Xuehai He and Weixiang Yan and Ao Shen and Chenyang Zhao and Xin Eric Wang , booktitle=. Soft Thinking: Unlocking the Reasoning Potential of. 2025 , url=

    2025

  56. [56]

    2026 , eprint=

    Multiplex Thinking: Reasoning via Token-wise Branch-and-Merge , author=. 2026 , eprint=

    2026

  57. [57]

    2025 , eprint=

    Soft Tokens, Hard Truths , author=. 2025 , eprint=

    2025

  58. [58]

    2025 , eprint=

    Latent Collaboration in Multi-Agent Systems , author=. 2025 , eprint=

    2025

  59. [59]

    arXiv preprint arXiv:2511.08577 , year=

    Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models , author=. arXiv preprint arXiv:2511.08577 , year=

    Pith/arXiv arXiv

  60. [60]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Mixture of Inputs: Text Generation Beyond Discrete Token Sampling , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  61. [61]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  62. [62]

    fairseq: A Fast, Extensible Toolkit for Sequence Modeling

    Ott, Myle and Edunov, Sergey and Baevski, Alexei and Fan, Angela and Gross, Sam and Ng, Nathan and Grangier, David and Auli, Michael. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics (Demonstrations). 2019. doi:10.18653/v1/N19-4009

    doi:10.18653/v1/n19-4009 2019

  63. [63]

    2023 , eprint=

    Looped Transformers as Programmable Computers , author=. 2023 , eprint=

    2023

  64. [64]

    2025 , eprint=

    A Modular Dataset to Demonstrate LLM Abstraction Capability , author=. 2025 , eprint=

    2025

  65. [65]

    2020 , note =

    Lan, Zhenzhong and Chen, Mingda and Goodman, Sebastian and Gimpel, Kevin and Sharma, Piyush and Soricut, Radu , booktitle =. 2020 , note =

    2020

  66. [66]

    Proceedings of the 40th International Conference on Machine Learning (ICML) , year =

    Looped Transformers as Programmable Computers , author =. Proceedings of the 40th International Conference on Machine Learning (ICML) , year =

  67. [67]

    International Conference on Learning Representations (ICLR) , year =

    Reasoning with Latent Thoughts: On the Power of Looped Transformers , author =. International Conference on Learning Representations (ICLR) , year =

  68. [68]

    arXiv preprint , year =

    Scaling Latent Reasoning via Looped Language Models , author =. arXiv preprint , year =

  69. [69]

    arXiv preprint arXiv:1803.05457 , year=

    Think you have solved question answering? try arc, the ai2 reasoning challenge , author=. arXiv preprint arXiv:1803.05457 , year=

    Pith/arXiv arXiv

  70. [70]

    arXiv preprint arXiv:1905.07830 , year=

    Hellaswag: Can a machine really finish your sentence? , author=. arXiv preprint arXiv:1905.07830 , year=

    Pith/arXiv arXiv 1905

  71. [71]

    arXiv preprint arXiv:1809.02789 , year=

    Can a suit of armor conduct electricity? a new dataset for open book question answering , author=. arXiv preprint arXiv:1809.02789 , year=

    Pith/arXiv arXiv

  72. [72]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Piqa: Reasoning about physical commonsense in natural language , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  73. [73]

    arXiv preprint arXiv:1707.06209 , year=

    Crowdsourcing multiple choice science questions , author=. arXiv preprint arXiv:1707.06209 , year=

    Pith/arXiv arXiv

  74. [74]

    Communications of the ACM , volume=

    Winogrande: An adversarial winograd schema challenge at scale , author=. Communications of the ACM , volume=. 2021 , publisher=

    2021

  75. [75]

    Advances in Neural Information Processing Systems , volume=

    On the inductive bias of stacking towards improving reasoning , author=. Advances in Neural Information Processing Systems , volume=

  76. [76]

    The Eleventh International Conference on Learning Representations , year=

    Transformers Learn Shortcuts to Automata , author=. The Eleventh International Conference on Learning Representations , year=

  77. [77]

    Forty-second International Conference on Machine Learning , year=

    (How) Do Language Models Track State? , author=. Forty-second International Conference on Machine Learning , year=

  78. [78]

    arXiv preprint arXiv:2302.13971 , year=

    Llama: Open and efficient foundation language models , author=. arXiv preprint arXiv:2302.13971 , year=

    Pith/arXiv arXiv

  79. [79]

    doi:10.5281/zenodo.12608602 , url =

    Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and Reynolds, Laria and Schoelkopf, Hailey and Skowron, Aviya and Sutawika, Lintang...

    doi:10.5281/zenodo.12608602

  80. [80]

    2024 , eprint=

    GPT-4o System Card , author=. 2024 , eprint=

    2024

Showing first 80 references.