Pith. sign in

REVIEW 3 major objections 6 minor 17 cited by

A Survey on Latent Reasoning

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This survey claims that latent chain-of-thought — reasoning in continuous hidden states rather than words — can overcome the expressive limits of explicit CoT, and that all current methods share a depth-from-optimization principle.

desk verdict A genuinely useful taxonomy of latent reasoning, with a central unification claim that is currently asserted rather than shown. read the letter →

arxiv 2507.06203 v2 pith:755WHYYQ submitted 2025-07-08 cs.CL

Rui-Jie Zhu , Tianhao Peng , Tianhao Cheng , Xingwei Qu , Jinfa Huang , Dawei Zhu , Hao Wang , Kaiwen Xue
show 25 more authors
Xuanliang Zhang Yong Shan Tianle Cai Taylor Kergan Assel Kembay Andrew Smith Chenghua Lin Binh Nguyen Yuqi Pan Yuhong Chou Zefan Cai Zhenhe Wu Yongchi Zhao Tianyu Liu Jian Yang Wangchunshu Zhou Chujie Zheng Chongxuan Li Yuyin Zhou Zhoujun Li Zhaoxiang Zhang Jiaheng Liu Ge Zhang Wenhao Huang Jason Eshraghian
This is my paper · ORCID
classification cs.CL
keywords latentchain-of-thoughtcontinuoushiddenstateactivation-basedrecurrencehidden-statemaskeddiffusionmodelstest-timetraininglooptransformersmechanisticinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Explicit chain-of-thought makes an LLM spell out each reasoning step in words, and that constraint costs expressive power: a token carries roughly 15 bits, while one 2560-dimensional hidden state carries about 40,000 bits. The survey argues that latent chain-of-thought, which keeps the reasoning process in the continuous hidden state via $z_{t+1} = \mathrm{Transform}(z_t, S_t)$, can sidestep that bottleneck and may exceed what word-based reasoning can express. It organizes the field into vertical recurrence (looping the same layers to deepen computation), horizontal recurrence (evolving a compressed hidden state across a sequence), and infinite-depth reasoning via diffusion and optimizer-style updates. The organizing claim is that all three families are one phenomenon: depth emerges from optimization over time, so longer sequences and more loop iterations are the same resource. If this framing holds, non-linguistic reasoning paths and test-time scaling that does not generate more words become available to any model, not just new architectures.

What carries the argument

The central object is the hidden state $S^{l}_{t}$ together with the latent update rule $z_{t+1} = \mathrm{Transform}(z_t, S_t)$ that replaces the explicit-CoT rule $y_{t+1} = \mathrm{Decode}(\mathrm{Transform}(x_t, S_t))$. The load-bearing identity of the survey is the gradient-state recurrence $S_t = S_{t-1} - \eta_t \nabla_S \ell(S_{t-1}; k_t, v_t)$, which makes each incoming token one step of an online optimizer refining a fast-weight layer; DeltaNet's closed-form update is shown to be exactly one gradient step on a least-squares objective. Around this core, the survey places three mechanism families: vertical recurrence (the same layer stack run for multiple iterations), horizontal recurrence (matrix or vector state updated per token), and diffusion-based spatial refinement (masked or embedding-level denoising over the whole sequence). These mechanisms are connected by the claim that depth can be traded for time, making the hidden-state update the common substrate of latent reasoning.

What would settle it

Run a matched comparison on a fixed reasoning task, such as multi-hop question answering or GSM8K-style arithmetic, with two models of identical parameter count: one that loops the same layer stack for T iterations and one that processes T hidden-state updates along the sequence. If doubling T improves only the looped model, or if the two models diverge in accuracy at equal T, the depth-time equivalence at the core of the survey's unification is false for real transformers.

Watch

Extended reading notes

Core claim

The paper's central claim is that reasoning need not be verbalized to be performed: a transformer can carry out multi-step inference entirely in continuous representations by eliminating the decoding step that forces each thought into a token. In the survey's unified formalism, every method is a spatial transformation across layers plus a temporal transformation of a hidden state $S^{l}_{t}$ that can be a key-value (KV) cache, a fixed-size linear-attention matrix, or a recurrent vector. Activation-based methods (Universal Transformer, looped and recursive transformers, Coconut) deepen the computational graph by reusing layers; hidden-state methods (linear attention, RetNet, Mamba-2, TTT, Titans, DeltaNet) evolve a compressed memory; and text diffusion models refine the whole sequence bidirectionally, with confidence-thresholded KV caches that make the process reversible and self-correcting. The survey further claims that these branches are unified by a gradient-based view in which the hidden state is a fast-weight layer updated by an optimizer, so processing a longer sequence is mathematically equivalent to running the same layer for more optimization iterations. The conclusion is that latent reasoning is a coherent design space with a common mathematical core, and that the field's main open problems are training stability, memory efficiency, and the absence of standardized benchmarks.

Load-bearing premise

The load-bearing premise is that depth and time are interchangeable: Section 5.2 asserts that processing a longer sequence is mathematically equivalent to running the same layer for more optimization iterations, but the survey does not prove this equivalence for real transformers, and the unification of vertical and horizontal recurrence rests on it.

Editorial extensions

If this is right

  • If latent chain-of-thought works as claimed, models can reason about problems that have no clean linguistic expression, because thoughts no longer have to pass through a finite vocabulary.
  • Long-context models become deep reasoners: feeding a model more tokens can serve as extra optimization steps, so sequence length and network depth become interchangeable reasoning resources.
  • Diffusion-based reasoning can revise earlier decisions, unlike autoregressive generation which commits to tokens and cannot revisit them, enabling globally consistent and self-correcting chains.
  • Existing pretrained transformers can acquire latent reasoning through training alone, for example with Coconut's continuous thought vectors, CODI's self-distillation, or pause tokens, without architectural changes, so the capability is accessible to current LLMs.
  • The unification predicts that performance on reasoning tasks scales with the number of hidden-state refinement steps, independent of whether those steps come from loops, recurrent states, or denoising iterations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the depth-time equivalence is real, then current linear-attention and state-space models should show measurable latent-reasoning gains on multi-step benchmarks, not just language modeling and recall; the survey notes that such evidence does not yet exist, so this is a testable prediction rather than a reported result.
  • The bandwidth argument implies a sharp interpretability trade-off: latent thoughts are unreadable, so safety and verification may require auxiliary probes or reconstructive decoders rather than inspection of the reasoning trace itself.
  • The same equivalence suggests that deployed reasoning models could replace explicit thinking tokens with silent hidden-state iterations, changing the cost and latency profile of test-time compute scaling.
  • One concrete extension: a controlled comparison of looped transformers and linear-recurrent models with matched parameters and step counts would settle whether vertical and horizontal recurrence are truly the same resource; the survey's taxonomy makes this experiment natural.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This survey organizes the emerging literature on latent reasoning in large language models into a taxonomy with two principal branches: vertical recurrence (activation-based methods that loop over layers) and horizontal recurrence (hidden-state-based methods that propagate compressed states over time), extended by a third branch on diffusion-based infinite-depth reasoning. It proposes a unified mathematical framework in Section 2, an optimization-based equivalence between temporal and depth recurrence in Section 5.2, and a review of mechanistic interpretability evidence in Section 4. The survey is candid about the absence of direct empirical comparisons and includes critical discussion of Infini-attention's reported practical failures.

Significance. If taken as a map of an emerging field, the survey is timely and useful. It covers a wide range of recent work, provides a structured categorization, and includes welcome critical coverage (e.g., the Infini-attention reproduction issues in Section 5.2.1) and explicit limitation statements (Section 6). Its main contribution is the conceptual organization, not new empirical results. The central unification claim—that temporal and depth recurrence are equivalent—is, however, asserted rather than demonstrated, and the horizontal branch is admitted to lack evidence of enhanced reasoning. With that claim appropriately tempered, the survey would be a valuable resource; as written, the significance depends on an unsupported equivalence.

major comments (3)
  1. [Section 5.2, Eq. (13)] The claim that 'processing a longer sequence is mathematically equivalent to running the same layer for more optimization iterations' is not supported by the update rule in Eq. (13), where each new token supplies a fresh (k_t, v_t) and therefore a different objective; vertical recurrence in Eq. (2), by contrast, iterates the same transformation f on a refined representation. The paper provides no derivation or empirical test of this equivalence, and Section 3.2 itself concedes that hidden-state models 'have not yet produced evidence demonstrating enhanced reasoning capabilities.' This makes the unifying claim load-bearing and currently unsupported; I recommend either proving the equivalence under stated conditions, citing evidence for it, or recasting the taxonomy as a classification with open questions about whether horizontal recurrence constitutes latent reasoning.
  2. [Section 3.2.1, Table 2] The optimization-based unification is overgeneralized. Only DeltaNet and Gated DeltaNet (and the gradient-state models in Table 2) have updates that are exactly gradient steps on a regression objective; the linear-state updates for Linear Attention, RetNet, GLA, Mamba-2, and HGRN-2 are associative decay/add rules that do not, in general, minimize a shared objective. The text claims a conceptual unification of temporal and depth recurrence based on DeltaNet, but the table itself shows that the linear-state family is not an optimizer. Please restrict the unification claim to gradient-state and delta-rule models, or show the objective for each update rule.
  3. [Section 3.2 and Section 4.3] The survey labels hidden-state sequence models as a principal branch of latent reasoning despite admitting that these methods have not yet demonstrated enhanced reasoning capabilities (Section 3.2). This conflates memory/linear recurrence with reasoning. Similarly, Section 4.3's title promises Turing completeness of layer-based latent CoT, but the cited results concern Turing completeness of Transformers, RNNs, and prompting, not of latent CoT specifically. Please align the claims with the evidence, for example by explicitly distinguishing demonstrated latent reasoning from architectures that merely have potential for it.
minor comments (6)
  1. [Figure 1] The bandwidth comparison compares bits per token for explicit CoT with bits per hidden state for latent reasoning, but the two quantities are not matched in time granularity or information content; please state the assumptions and compare over a full reasoning step.
  2. [Section 2.1, Eq. (1)] The indexing mixes time steps—x^{l+1}_{t+1} is computed from x^l_{t+1} and S^l_t; please define the time convention consistently so the spatial and temporal updates are unambiguous.
  3. [Section 3.1.1, Eq. (2)] The text says the same transformation f is applied to 'the same input,' but each call passes a different hidden state S^{l+i-1}_t; please rephrase as shared parameters with an evolving state.
  4. [Section 5.2.1, Ref. [73]] The critical assessment of Infini-attention relies on a blog post whose first author is listed as 'neuralink'; please verify the author attribution and, if possible, cite a more archival source or clearly mark the blog as non-peer-reviewed.
  5. [Section 5.1.1 and 5.1.2, Ref. [124]] DoT-SEDD and DoT-Plaid are both cited to Ref. [124] even though the text treats them as distinct frameworks; please clarify the relationship.
  6. [Section 4.3] The subsection title promises Turing completeness of layer-based latent CoT, while the cited results concern Transformers, RNNs, and prompting; consider retitling to avoid overstatement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the survey's taxonomy and equations are organizational definitions; its strongest claim is asserted rather than derived, and the paper openly flags missing empirical evidence.

full rationale

This is a survey, not a derivation, so the classic circularity modes (fitted inputs renamed as predictions, uniqueness theorems imported from the authors, ansatz smuggled by citation) mostly do not apply. The general framework in Section 2.1 (Eqs. 1-3) and the latent update rules in Section 2.2 (Eqs. 4-5) are definitions used to organize the literature, and the taxonomy in Section 3 is a classification of existing architectures. The optimization view in Section 3.2.2 (Eq. 13) is also presented as a unifying update rule taken from the cited TTT/DeltaNet line, not as a newly derived result. The one potentially load-bearing statement in Section 5.2, 'processing a longer sequence is mathematically equivalent to running the same layer for more optimization iterations,' is asserted rather than proved, and the survey itself concedes in Section 3.2 that hidden-state models 'have not yet produced evidence demonstrating enhanced reasoning capabilities.' That is an unsupported extrapolation, which is a correctness and rigor risk, but it is not circular: the claim does not make its conclusion an input to Eq. 13, nor is it a fitted quantity. The self-citations in the masked-diffusion section (e.g., refs. 74-76 and 139) anchor the taxonomy, but they report externally falsifiable benchmark results and are accompanied by many independent references (SEDD, D3PM, Diffusion-LM, DoT, etc.), so no central conclusion reduces to a self-citation chain. The paper also explicitly disclaims direct empirical comparison in Section 6, further showing that it is not claiming to derive a prediction from fitted parameters. Accordingly, no circular step meets the evidentiary bar of exhibiting a reduction of a claimed result to its own inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The survey does not fit parameters or introduce new model entities. Its load-bearing assumptions are interpretive: that layers encode reasoning steps, that the proposed equations cover all methods, that temporal recurrence equals depth, and that diffusion denoising is infinite-depth reasoning. These are organizational postulates, not empirically tested claims.

assumptions (4)
  • domain assumption Transformer depth is a faithful proxy for reasoning depth; layers are the basic computational units of latent reasoning.
    Section 4 builds the 'Layer Specialization' theory on this premise and uses it to argue layer stacks form a latent CoT; Section 4.1 raises the question 'Are layers the basic computational units of reasoning?' but the survey does not answer it.
  • ad hoc to paper The recurrence equations in Section 2.1 (Eqs 1 to 10), including the distinction between activation-based and hidden-state-based methods, capture all relevant latent reasoning approaches.
    This is a survey-internal formalization; no evidence is given that a method exists outside the framework, nor that all methods fit without distortion.
  • domain assumption Temporal recurrence is equivalent to additional depth: a longer sequence equals more iterations of an optimizer on a fast-weight layer.
    Stated in Section 5.2 ('processing a longer sequence is mathematically equivalent to running the same layer for more optimization iterations'); used to unify horizontal and vertical recurrence, but no proof is provided for real transformers.
  • domain assumption Diffusion models performing parallel denoising steps constitute 'infinite-depth reasoning'.
    Section 5 asserts that unbounded denoising steps provide unbounded reasoning depth; this is an interpretive mapping, not a formal equivalence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Survey on Latent Reasoning." pith.science (2026). https://pith.science/paper/755WHYYQ

@misc{pith2026250706203,
  author       = {Pith},
  title        = {Pith review of: A Survey on Latent Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/755WHYYQ}},
  note         = {Machine review of arXiv:2507.06203}
}
read the original abstract

Large Language Models (LLMs) have demonstrated impressive reasoning capabilities, especially when guided by explicit chain-of-thought (CoT) reasoning that verbalizes intermediate steps. While CoT improves both interpretability and accuracy, its dependence on natural language reasoning limits the model's expressive bandwidth. Latent reasoning tackles this bottleneck by performing multi-step inference entirely in the model's continuous hidden state, eliminating token-level supervision. To advance latent reasoning research, this survey provides a comprehensive overview of the emerging field of latent reasoning. We begin by examining the foundational role of neural network layers as the computational substrate for reasoning, highlighting how hierarchical representations support complex transformations. Next, we explore diverse latent reasoning methodologies, including activation-based recurrence, hidden state propagation, and fine-tuning strategies that compress or internalize explicit reasoning traces. Finally, we discuss advanced paradigms such as infinite-depth latent reasoning via masked diffusion models, which enable globally consistent and reversible reasoning processes. By unifying these perspectives, we aim to clarify the conceptual landscape of latent reasoning and chart future directions for research at the frontier of LLM cognition. An associated GitHub repository collecting the latest papers and repos is available at: https://github.com/multimodal-art-projection/LatentCoT-Horizon/.

Figures

Figures reproduced from arXiv: 2507.06203 by the authors.

Figure 1
Figure 1. Explicit reasoning transmits discrete tokens ( [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Taxonomy of Latent Reasoning. Finally, we explore advanced paradigms at the frontier of LLM cognition, focusing on the pursuit of infinite-depth reasoning. This concept refers to a model’s ability to devote unbounded computational steps to refine a solution, moving beyond fixed-depth architectures. Our discussion centers on spatial infinite reasoning as realized by text diffusion models [74, 124]. Unlike traditional… view at source ↗
Figure 3
Figure 3. Comparison of Activation-Based and Hidden-state-Based Latent Reasoning. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Conceptual diagram of a Pre/Loop/Coda architecture with per-iteration input 𝑥𝑡 , hidden state 𝑆𝑡 (KV-cache), depth embedding 𝑑𝑡 , and a dynamic-stop gate. computational resource that can be allocated based on task complexity. Extending activation￾reuse beyond Universal…
Figure 5
Figure 5. Figure 5: An evolutionary graph of the text diffusion models, including three architectural [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Latent On-Policy Self-Distillation

    cs.LG 2026-08 conditional novelty 7.0 of 10

    Latent On-Policy Self-Distillation learns the self-teacher's privileged context end-to-end as continuous tokens retrieved from past experience, improving final performance and sample efficiency over fixed-context OPSD...

  2. Bridging Compute- and Data-Optimal Pretraining

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.

  3. Latent Memory Palace: Reasoning for Control as Autoregressive Variational Inference

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Variable-length autoregressive latent sequences, trained as variational inference with a PPO-style objective, give robot policies adaptive test-time compute and yield a reusable action tokenizer.

  4. Scaling Latent Reasoning via Looped Language Models

    cs.CL 2025-10 unverdicted novelty 7.0 of 10

    Looped language models with latent iterative computation and entropy-regularized depth allocation achieve performance matching up to 12B standard LLMs through superior knowledge manipulation.

  5. Hierarchical Latent Reasoning for LLM-based Recommendation

    cs.IR 2026-07 conditional novelty 6.0 of 10

    HiLaR aligns LLM latent reasoning states with temporally quantized user preference hierarchies and optimizes them with layer-aware process rewards, improving recommendation accuracy on four Amazon datasets.

  6. Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    LOTUS uses a looped padded Transformer with parallel cross-entropy supervision on gold CoT tokens to match explicit CoT performance at 3B parameters while reducing thought-phase latency 2.5x-6.9x.

  7. MUX: Continuous Reasoning via Multiplexed Tokens

    cs.AI 2026-05 conditional novelty 6.0 of 10

    MUX trains language models to reason with continuous latent tokens that encode spans of discrete reasoning as lossless weighted superpositions, improving accuracy and efficiency over latent-reasoning baselines.

  8. LaRe: Latent Refocusing for Multimodal Reasoning

    cs.CV 2025-11 reject novelty 6.0 of 10

    LaRe performs iterative visual refocusing in latent space and reports accuracy gains with fewer tokens, but its main experiments compare against baselines trained with less data.

  9. Navigating the Latent Space Dynamics of Neural Models

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Autoencoders implicitly define a latent vector field whose attractors encode the model's memorized and generalized knowledge, enabling data-free probing and out-of-distribution detection.

  10. MORES: Mobile Reasoning-as-a-Service via Distributed LLM Inference-Time Scaling

    cs.NI 2026-07 conditional novelty 5.5 of 10

    A device–server split of recurrent latent LLM reasoning plus semantic MoE-SAC scheduling yields about 18% higher simulated system throughput than plain SAC under energy, recurrence, and latency budgets.

  11. Think Deep, Speak Once: Relit, A Recursive Latent Implicit Transformer Framework

    cs.AI 2026-08 reject novelty 5.0 of 10

    ReLIT reaches 98.6% on ProofWriter and 97.6% on RuleTaker by adding a recursive latent block to a frozen TinyLlama backbone.

  12. How Far Do Simple Transformations Translate Across Text Embedding Models?

    cs.LG 2026-08 conditional novelty 5.0 of 10

    Simple linear translators between text embedding models work only for architecturally and training-similar pairs, so embedding spaces are not universally related by such maps.

  13. LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

    cs.AI 2026-05 reject novelty 5.0 of 10

    LISA replaces standard attention with linear attention plus indexer-selected sparse attention, claiming O(nM) inference cost and accuracy gains, but the complexity analysis ignores the indexer's full-context scoring.

  14. Bridging Search and Recommendation through Latent Cross Reasoning

    cs.IR 2025-08 conditional novelty 5.0 of 10

    A latent cross reasoning model with contrastive learning and GRPO reinforcement learning improves search-enhanced recommendation on Qilin and KuaiSAR.

  15. Refine Thought: A Test-Time Inference Method for Embedding Model Reasoning

    cs.CL 2025-10 conditional novelty 4.0 of 10

    RT improves embedding-model retrieval on some reasoning-heavy tasks by iteratively refining the query embedding at test time, but reported gains are mixed and the key implementation detail is unspecified.

  16. Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey

    cs.AI 2025-07 conditional novelty 3.0 of 10

    A comprehensive review that categorizes methods for shortening and adaptively triggering chain-of-thought reasoning in large language models.

  17. Critique of Impure Reason: Unveiling the reasoning behaviour of medical Large Language Models

    cs.CL 2024-12 conditional novelty 3.0 of 10

    A narrative review arguing that medical LLM evaluations should examine reasoning behaviour, not only accuracy, and proposing two conceptual transparency frameworks.

Reference graph

Works this paper leans on

140 extracted references · 5 canonical work pages · cited by 17 Pith papers

  1. [1]

    Structured denoising diffusion models in discrete state-spaces

    Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34: 0 17981--17993, 2021

    2021

  2. [2]

    Relaxed recursive transformers: Effective parameter sharing with layer-wise lora

    Sangmin Bae, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Seungyeon Kim, and Tal Schuster. Relaxed recursive transformers: Effective parameter sharing with layer-wise lora. arXiv preprint arXiv:2410.20672, 2024

    arXiv 2024

  3. [3]

    Titans: Learning to memorize at test time

    Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.00663, 2024

    arXiv 2024

  4. [4]

    Atlas: Learning to optimally memorize the context at test time

    Ali Behrouz, Zeman Li, Praneeth Kacham, Majid Daliri, Yuan Deng, Peilin Zhong, Meisam Razaviyayn, and Vahab Mirrokni. Atlas: Learning to optimally memorize the context at test time. arXiv preprint arXiv:2505.23735, 2025 a

    arXiv 2025

  5. [5]

    It's all connected: A journey through test-time memorization, attentional bias, retention, and online optimization

    Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni. It's all connected: A journey through test-time memorization, attentional bias, retention, and online optimization. arXiv preprint arXiv:2504.13173, 2025 b

    arXiv 2025

  6. [6]

    Emergent abilities in large language models: A survey

    Leonardo Berti, Flavio Giorgi, and Gjergji Kasneci. Emergent abilities in large language models: A survey. arXiv preprint arXiv:2503.05788, 2025

    arXiv 2025

  7. [7]

    Transformers to ssms: Distilling quadratic knowledge to subquadratic models

    Aviv Bick, Kevin Li, Eric Xing, J Zico Kolter, and Albert Gu. Transformers to ssms: Distilling quadratic knowledge to subquadratic models. Advances in Neural Information Processing Systems, 37: 0 31788--31812, 2024

    2024

  8. [8]

    Llamba: Scaling distilled recurrent models for efficient language processing

    Aviv Bick, Tobias Katsch, Nimit Sohoni, Arjun Desai, and Albert Gu. Llamba: Scaling distilled recurrent models for efficient language processing. arXiv preprint, cs.LG, 2025. URL https://arxiv.org/abs/2502.14458

    arXiv 2025

Show all 140 references
  1. [9]

    Hopping too late: Exploring the limitations of large language models on multi-hop queries

    Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. Hopping too late: Exploring the limitations of large language models on multi-hop queries. arXiv preprint arXiv:2406.12775, 2024

    2024 arXiv

  2. [10]

    Recurrent memory transformer

    Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. Advances in Neural Information Processing Systems, 35: 0 11079--11091, 2022

    2022

  3. [11]

    Iteration head: A mechanistic study of chain-of-thought

    Vivien Cabannes, Charles Arnal, Wassim Bouaziz, Xingyu Yang, Francois Charton, and Julia Kempe. Iteration head: A mechanistic study of chain-of-thought. Advances in Neural Information Processing Systems, 37: 0 109101--109122, 2024

    2024

  4. [12]

    Large language models to diffusion finetuning

    Edoardo Cetin, Tianyu Zhao, and Yujin Tang. Large language models to diffusion finetuning. arXiv preprint arXiv:2501.15781, 2025

    2025 arXiv

  5. [13]

    Bypassing the exponential dependency: Looped transformers efficiently learn in-context by multi-step gradient descent

    Bo Chen, Xiaoyu Li, Yingyu Liang, Zhenmei Shi, and Zhao Song. Bypassing the exponential dependency: Looped transformers efficiently learn in-context by multi-step gradient descent. arXiv preprint arXiv:2410.11268, 2024 a

    2024 arXiv

  6. [14]

    Unveiling induction heads: Provable training dynamics and feature learning in transformers

    Siyu Chen, Heejune Sheen, Tianhao Wang, and Zhuoran Yang. Unveiling induction heads: Provable training dynamics and feature learning in transformers. arXiv preprint arXiv:2409.10559, 2024 b

    2024 arXiv

  7. [15]

    Analog bits: Generating discrete data using diffusion models with self-conditioning, 2023

    Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning, 2023. URL https://arxiv.org/abs/2208.04202

    2023 arXiv

  8. [16]

    Compressed chain of thought: Efficient reasoning through dense representations

    Jeffrey Cheng and Benjamin Van Durme. Compressed chain of thought: Efficient reasoning through dense representations. arXiv preprint arXiv:2412.13171, 2024

    2024 arXiv

  9. [17]

    Learning phrase representations using RNN encoder-decoder for statistical machine translation

    Kyunghyun Cho, Bart Van Merri \"e nboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Meth...

    2014

  10. [18]

    Investigating recurrent transformers with dynamic halt

    Jishnu Ray Chowdhury and Cornelia Caragea. Investigating recurrent transformers with dynamic halt. arXiv preprint arXiv:2402.00976, 2024

    2024 arXiv

  11. [19]

    Do language models use their depth efficiently? arXiv preprint arXiv:2505.13898, 2025

    R \'o bert Csord \'a s, Christopher D Manning, and Christopher Potts. Do language models use their depth efficiently? arXiv preprint arXiv:2505.13898, 2025

    2025

  12. [20]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024

    2024 arXiv

  13. [21]

    Simulation of graph algorithms with looped transformers

    Artur Back De Luca and Kimon Fountoulakis. Simulation of graph algorithms with looped transformers. arXiv preprint arXiv:2402.01107, 2024

    2024 arXiv

  14. [22]

    Universal transformers

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and ukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819, 2018

    2018 arXiv

  15. [23]

    From explicit cot to implicit cot: Learning to internalize cot step by step

    Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to internalize cot step by step. arXiv preprint arXiv:2405.14838, 2024

    2024 arXiv

  16. [24]

    Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler

    Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H. Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler. Continuous diffusion for categorical data, 2022. UR...

    2022 arXiv

  17. [25]

    Reinforcement pre-training

    Qingxiu Dong, Li Dong, Yao Tang, Tianzhu Ye, Yutao Sun, Zhifang Sui, and Furu Wei. Reinforcement pre-training. arXiv preprint arXiv:2506.08007, 2025

    2025 arXiv

  18. [26]

    Tenenbaum

    Yilun Du, Jiayuan Mao, and Joshua B. Tenenbaum. Learning iterative reasoning through energy diffusion. In International Conference on Machine Learning (ICML), 2024

    2024

  19. [27]

    How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning

    Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, and Tanmoy Chakraborty. How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning. arXiv preprint arXiv:2402.18312, 2024

    2024 arXiv

  20. [28]

    Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14 0 (2): 0 179--211, 1990

    1990

  21. [29]

    Algoformer: An efficient transformer framework with algorithmic structures

    Yihang Gao, Chuanyang Zheng, Enze Xie, Han Shi, Tianyang Hu, Yu Li, Michael K Ng, Zhenguo Li, and Zhaoqiang Liu. Algoformer: An efficient transformer framework with algorithmic structures. arXiv preprint arXiv:2402.13572, 2024

    2024 arXiv

  22. [30]

    Can looped transformers learn to implement multi-step gradient descent for in-context learning? arXiv preprint arXiv:2410.08292, 2024

    Khashayar Gatmiry, Nikunj Saunshi, Sashank J Reddi, Stefanie Jegelka, and Sanjiv Kumar. Can looped transformers learn to implement multi-step gradient descent for in-context learning? arXiv preprint arXiv:2410.08292, 2024

    2024 arXiv

  23. [31]

    Scaling up test-time compute with latent reasoning: A recurrent depth approach

    Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05171, 2025

    2025 arXiv

  24. [32]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Team Gemini. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. 2025 a

    2025

  25. [33]

    Gemini diffusion is our new experimental research model

    Team Gemini. Gemini diffusion is our new experimental research model. 2025 b

    2025

  26. [34]

    Transformer feed-forward layers are key-value memories

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913, 2020

    2012 arXiv

  27. [35]

    Looped transformers as programmable computers

    Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers. In International Conference on Machine Learning, pages 11398--11442. PMLR, 2023

    2023

  28. [36]

    Scaling diffusion language models via adaptation from autoregressive models

    Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling diffusion language models via adaptation from autoregressive models. In The Thirteenth International Conference on Lea...

    2025

  29. [37]

    Think before you speak: Training language models with pause tokens

    Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/for...

    2024

  30. [38]

    Bayesian flow networks, 2025

    Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez. Bayesian flow networks, 2025. URL https://arxiv.org/abs/2308.07037

    2025 arXiv

  31. [39]

    Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887, 2024. URL https://arxiv.org/abs/2403.17887

    2024 arXiv

  32. [40]

    Hashimoto

    Ishaan Gulrajani and Tatsunori B. Hashimoto. Likelihood-based diffusion language models, 2023. URL https://arxiv.org/abs/2305.18619

    2023 arXiv

  33. [41]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025 a

    2025 arXiv

  34. [42]

    How do llms perform two-hop reasoning in context? arXiv preprint arXiv:2502.13913, 2025 b

    Tianyu Guo, Hanlin Zhu, Ruiqi Zhang, Jiantao Jiao, Song Mei, Michael I Jordan, and Stuart Russell. How do llms perform two-hop reasoning in context? arXiv preprint arXiv:2502.13913, 2025 b

    2025 arXiv

  35. [43]

    How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model

    Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems, 36: 0 76033--76060, 2023

    2023

  36. [44]

    Training large language models to reason in a continuous latent space

    Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769, 2024

    2024 arXiv

  37. [45]

    Long short-term memory

    Sepp Hochreiter and J \"u rgen Schmidhuber. Long short-term memory. Neural Computation, 9 0 (8): 0 1735--1780, 1997

    1997

  38. [46]

    Towards a mechanistic interpretation of multi-step reasoning capabilities of language models

    Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, and Mrinmaya Sachan. Towards a mechanistic interpretation of multi-step reasoning capabilities of language models. arXiv preprint arXiv:2310.14491, 2023

    arXiv

  39. [47]

    Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, J. H. Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, and Wei Chu. Opencoder: The open cookbook for top-tie...

    2024 arXiv

  40. [48]

    Reinforcing the diffusion chain of lateral thought with diffusion language models, 2025

    Zemin Huang, Zhiyang Chen, Zijun Wang, Tiancheng Li, and Guo-Jun Qi. Reinforcing the diffusion chain of lateral thought with diffusion language models, 2025. URL https://arxiv.org/abs/2505.10446

    2025

  41. [49]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  42. [50]

    Disentangling memory and reasoning ability in large language models

    Mingyu Jin, Weidi Luo, Sitao Cheng, Xinyi Wang, Wenyue Hua, Ruixiang Tang, William Yang Wang, and Yongfeng Zhang. Disentangling memory and reasoning ability in large language models. arXiv preprint arXiv:2411.13504, 2024

    2024 arXiv

  43. [51]

    Michael I. Jordan. An outsider's view of neural nets. Cognitive Science, 10 0 (1): 0 17--21, 1986

    1986

  44. [52]

    Lattice: Learning to efficiently compress the memory

    Mahdi Karami and Vahab Mirrokni. Lattice: Learning to efficiently compress the memory. arXiv preprint arXiv:2504.05646, 2025

    2025

  45. [53]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran c ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156--5165. PMLR, 2020

    2020

  46. [54]

    micse: Mutual information contrastive learning for low-shot sentence embeddings

    Tassilo Klein and Moin Nabi. micse: Mutual information contrastive learning for low-shot sentence embeddings. arXiv preprint arXiv:2211.04928, 2022

    2022 arXiv

  47. [55]

    Mercury: Ultra-fast language models based on diffusion, 2025

    Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, Stefano Ermon, Aditya Grover, and Volodymyr Kuleshov. Mercury: Ultra-fast language models based on diffusion, 2025

    2025

  48. [56]

    Liger: Linearizing large language models to gated recurrent structures

    Disen Lan, Weigao Sun, Jiaxi Hu, Jusen Du, and Yu Cheng. Liger: Linearizing large language models to gated recurrent structures. arXiv preprint arXiv:2503.01496, 2025

    2025 arXiv

  49. [57]

    The representation and recall of interwoven structured knowledge in llms: A geometric and layered analysis

    Ge Lei and Samuel J Cooper. The representation and recall of interwoven structured knowledge in llms: A geometric and layered analysis. arXiv preprint arXiv:2502.10871, 2025

    2025 arXiv

  50. [58]

    Seek in the dark: Reasoning via test-time instance-level policy gradient in latent space

    Hengli Li, Chenxi Li, Tong Wu, Xuekai Zhu, Yuxuan Wang, Zhaoxin Yu, Eric Hanchen Jiang, Song-Chun Zhu, Zixia Jia, Ying Nian Wu, et al. Seek in the dark: Reasoning via test-time instance-level policy gradient in latent space. arXiv preprint arXiv:2505.13308, 2025

    2025

  51. [59]

    Constant bit-size transformers are turing complete

    Qian Li and Yuyi Wang. Constant bit-size transformers are turing complete. arXiv preprint arXiv:2506.12027, 2025

    2025

  52. [60]

    Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting

    Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems, 32, 2019

    2019

  53. [61]

    Hashimoto

    Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion-lm improves controllable text generation, 2022. URL https://arxiv.org/abs/2205.14217

    2022 arXiv

  54. [62]

    Understanding and patching compositional reasoning in llms

    Zhaoyi Li, Gangwei Jiang, Hong Xie, Linqi Song, Defu Lian, and Ying Wei. Understanding and patching compositional reasoning in llms. arXiv preprint arXiv:2402.14328, 2024 a

    2024 arXiv

  55. [63]

    Chain of thought empowers transformers to solve inherently serial problems

    Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transformers to solve inherently serial problems. arXiv preprint arXiv:2402.12875, 1, 2024 b

    2024 arXiv

  56. [64]

    A comprehensive survey on long context language modeling

    Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, et al. A comprehensive survey on long context language modeling. arXiv preprint arXiv:2503.17407, 2025 a

    2025

  57. [65]

    dllm-cache: Accelerating diffusion large language models with adaptive caching, 2025 b

    Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. dllm-cache: Accelerating diffusion large language models with adaptive caching, 2025 b . URL https://arxiv.org/abs/2506.06295

    2025 arXiv

  58. [66]

    Discrete diffusion modeling by estimating the ratios of the data distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Proceedings of the 41st International Conference on Machine Learning, pages 32819--32848, 2024

    2024

  59. [67]

    dkv-cache: The cache for diffusion language models

    Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models. arXiv preprint arXiv:2505.15781, 2025

    2025 arXiv

  60. [68]

    Peters, and Arman Cohan

    Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew E. Peters, and Arman Cohan. Tess: Text-to-text self-conditioned simplex diffusion, 2024. URL https://arxiv.org/abs/2305.08379

    2024 arXiv

  61. [69]

    Linearizing large language models

    Jean Mercat, Igor Vasiljevic, Sedrick Keh, Kushal Arora, Achal Dave, Adrien Gaidon, and Thomas Kollar. Linearizing large language models. arXiv preprint arXiv:2405.06640, 2024

    2024 arXiv

  62. [70]

    A little depth goes a long way: The expressive power of log-depth transformers

    William Merrill and Ashish Sabharwal. A little depth goes a long way: The expressive power of log-depth transformers. arXiv preprint arXiv:2503.03961, 2025

    2025

  63. [71]

    Cotformer: A chain-of-thought driven architecture with budget-adaptive computation cost at inference

    Amirkeivan Mohtashami, Matteo Pagliardini, and Martin Jaggi. Cotformer: A chain-of-thought driven architecture with budget-adaptive computation cost at inference. arXiv preprint arXiv:2310.10845, 2023

    2023 arXiv

  64. [72]

    Leave no context behind: Efficient infinite context transformers with infini-attention

    Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention. arXiv preprint arXiv:2404.07143, 101, 2024

    2024 arXiv

  65. [73]

    A failed experiment: Infini-Attention , and why we should keep trying?, August 2024

    neuralink, Leandro von Werra, and Thomas Wolf. A failed experiment: Infini-Attention , and why we should keep trying?, August 2024. URL https://huggingface.co/blog/infini-attention. Hugging Face Blog post

    2024

  66. [74]

    Scaling up masked diffusion models on text

    Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410.18514, 2024

    2024 arXiv

  67. [75]

    Large language diffusion models

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, JUN ZHOU, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. In ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, 2025

    2025

  68. [76]

    Your absorbing discrete diffusion secretly models the conditional distributions of clean data

    Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint arXiv:2406.03736, 2024

    2024 arXiv

  69. [77]

    Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence

    Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Teddy Ferdinan, Haowen Hou, Przemys aw Kazienko, et al. Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892, 3, 2024

    2024 arXiv

  70. [78]

    On the turing completeness of modern neural network architectures

    Jorge P \'e rez, Javier Marinkovi \'c , and Pablo Barcel \'o . On the turing completeness of modern neural network architectures. arXiv preprint arXiv:1901.03429, 2019

    1901 arXiv

  71. [79]

    Jacob Pfau, William Merrill, and Samuel R. Bowman. Let s think dot by dot: Hidden computation in transformer language models. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=NikbrdtYvG

    2024

  72. [80]

    Hgrn2: Gated linear rnns with state expansion

    Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgrn2: Gated linear rnns with state expansion. arXiv preprint arXiv:2404.07904, 2024

    2024 arXiv

  73. [81]

    Ask, and it shall be given: On the turing completeness of prompting

    Ruizhong Qiu, Zhe Xu, Wenxuan Bao, and Hanghang Tong. Ask, and it shall be given: On the turing completeness of prompting. arXiv preprint arXiv:2411.01992, 2024

    2024 arXiv

  74. [82]

    D- CPT law: Domain-specific continual pre-training scaling law for large language models

    Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, ZhiqiBai, JiakaiWang, Yuanxing Zhang, Xu Tan, Jie Fu, Jiamang Wang, Lin Qu, Wenbo Su, and Bo Zheng. D- CPT law: Domain-specific continual pre-training scaling law for large language models. ...

    2024

  75. [83]

    Simple and effective masked diffusion language models

    Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37: 0 130136--130184, 2024

    2024

  76. [84]

    Inheritune: Training smaller yet more attentive language models

    Sunny Sanyal, Ravid Shwartz-Ziv, Alex Dimakis, and Sujay Sanghavi. Inheritune: Training smaller yet more attentive language models. arXiv preprint arXiv:2404.08634, 2024

    2024

  77. [85]

    On the inductive bias of stacking towards improving reasoning

    Nikunj Saunshi, Stefani Karp, Shankar Krishnan, Sobhan Miryoosefi, Sashank Jakkam Reddi, and Sanjiv Kumar. On the inductive bias of stacking towards improving reasoning. Advances in Neural Information Processing Systems, 37: 0 71437--71464, 2024

    2024

  78. [86]

    Reasoning with latent thoughts: On the power of looped transformers

    Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J Reddi. Reasoning with latent thoughts: On the power of looped transformers. arXiv preprint arXiv:2502.17416, 2025

    2025 arXiv

  79. [87]

    Implicit language models are RNN s: Balancing parallelization and expressivity

    Mark Sch \"o ne, Babak Rahmani, Heiner Kremer, Fabian Falck, Hitesh Ballani, and Jannes Gladrow. Implicit language models are RNN s: Balancing parallelization and expressivity. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum...

    2025

  80. [88]

    Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks

    Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. Advances in Neural Information Processing Systems, 34: 0 6695--6706, 2021

    2021

  81. [89]

    Distributional reasoning in llms: Parallel reasoning processes in multi-hop reasoning

    Yuval Shalev, Amir Feder, and Ariel Goldstein. Distributional reasoning in llms: Parallel reasoning processes in multi-hop reasoning. arXiv preprint arXiv:2406.13858, 2024

    2024 arXiv

  82. [90]

    Void in language models

    Mani Shemiranifar. Void in language models. arXiv preprint arXiv:2505.14467, 2025

    2025 arXiv

  83. [91]

    Codi: Compressing chain-of-thought into continuous space via self-distillation

    Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing chain-of-thought into continuous space via self-distillation. arXiv preprint arXiv:2502.21074, 2025

    2025 arXiv

  84. [92]

    Understanding layer significance in llm alignment

    Guangyuan Shi, Zexin Lu, Xiaoyu Dong, Wenlong Zhang, Xuanyu Zhang, Yujie Feng, and Xiao-Ming Wu. Understanding layer significance in llm alignment. arXiv preprint arXiv:2410.17875, 2024 a

    2024 arXiv

  85. [93]

    Simplified and generalized masked diffusion for discrete data

    Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37: 0 103131--103167, 2024 b

    2024

  86. [94]

    Siegelmann and Eduardo D

    Hava T. Siegelmann and Eduardo D. Sontag. On the computational power of neural nets. Journal of Computer and System Sciences, 50 0 (1): 0 132--150, 1995

    1995

  87. [95]

    Does representation matter? exploring intermediate layers in large language models

    Oscar Skean, Md Rifat Arefin, Yann LeCun, and Ravid Shwartz-Ziv. Does representation matter? exploring intermediate layers in large language models. arXiv preprint arXiv:2412.09563, 2024

    2024 arXiv

  88. [96]

    Layer by layer: Uncovering hidden representations in language models

    Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. arXiv preprint arXiv:2502.02013, 2025

    2025 arXiv

  89. [97]

    A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis

    Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. arXiv preprint arXiv:2305.15054, 2023

    2023 arXiv

  90. [98]

    Token assorted: Mixing latent and text tokens for improved language model reasoning

    DiJia Su, Hanlin Zhu, Yingchen Xu, Jiantao Jiao, Yuandong Tian, and Qinqing Zheng. Token assorted: Mixing latent and text tokens for improved language model reasoning. arXiv preprint arXiv:2502.03275, 2025

    2025 arXiv

  91. [99]

    The curse of depth in large language models

    Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, and Shiwei Liu. The curse of depth in large language models. arXiv preprint arXiv:2502.05795, 2025

    2025

  92. [100]

    Learning to (learn at test time): Rnns with expressive hidden states

    Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, et al. Learning to (learn at test time): Rnns with expressive hidden states. arXiv preprint arXiv:2407.04620, 2024

    2024 arXiv

  93. [101]

    Retentive network: A successor to transformer for large language models

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023

    2023 arXiv

  94. [102]

    Tess 2: A large-scale generalist diffusion language model

    Jaesung Tae, Hamish Ivison, Sachin Kumar, and Arman Cohan. Tess 2: A large-scale generalist diffusion language model. arXiv preprint arXiv:2502.13917, 2025

    2025 arXiv

  95. [103]

    An explainable transformer circuit for compositional generalization

    Cheng Tang, Brenden Lake, and Mehrdad Jazayeri. An explainable transformer circuit for compositional generalization. arXiv preprint arXiv:2502.15801, 2025

    2025 arXiv

  96. [104]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  97. [105]

    Unpacking robustness in inflectional languages: Adversarial evaluation and mechanistic insights

    Pawe \'L Walkowiak, Marek Klonowski, Marcin Oleksy, and Arkadiusz Janz. Unpacking robustness in inflectional languages: Adversarial evaluation and mechanistic insights. arXiv preprint arXiv:2505.07856, 2025

    2025 arXiv

  98. [106]

    Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization

    Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization. arXiv preprint arXiv:2405.15071, 2024 a

    2024 arXiv

  99. [107]

    Loss landscape geometry reveals stagewise development of transformers

    George Wang, Matthew Farrugia-Roberts, Jesse Hoogland, Liam Carroll, Susan Wei, and Daniel Murfet. Loss landscape geometry reveals stagewise development of transformers. In High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning, 2024 b

    2024

  100. [108]

    Interpretability in the wild: a circuit for indirect object identification in gpt-2 small

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022

    2022 arXiv

  101. [109]

    System-1.5 reasoning: Traversal in language and latent spaces with dynamic shortcuts

    Xiaoqiang Wang, Suyuchen Wang, Yun Zhu, and Bang Liu. System-1.5 reasoning: Traversal in language and latent spaces with dynamic shortcuts. arXiv preprint arXiv:2505.18962, 2025

    2025 arXiv

  102. [110]

    Guiding language model reasoning with planning tokens

    Xinyi Wang, Lucas Caccia, Oleksiy Ostapenko, Xingdi Yuan, William Yang Wang, and Alessandro Sordoni. Guiding language model reasoning with planning tokens. In First Conference on Language Modeling, 2024 c . URL https://openreview.net/forum?id=wi9IffRhVM

    2024

  103. [111]

    Unveiling factual recall behaviors of large language models through knowledge neurons

    Yifei Wang, Yuheng Chen, Wanting Wen, Yu Sheng, Linjing Li, and Daniel Dajun Zeng. Unveiling factual recall behaviors of large language models through knowledge neurons. arXiv preprint arXiv:2408.03247, 2024 d

    2024 arXiv

  104. [112]

    Embedding trajectory for out-of-distribution detection in mathematical reasoning

    Yiming Wang, Pei Zhang, Baosong Yang, Derek Wong, Zhuosheng Zhang, and Rui Wang. Embedding trajectory for out-of-distribution detection in mathematical reasoning. Advances in Neural Information Processing Systems, 37: 0 42965--42999, 2024 e

    2024

  105. [113]

    Latent space chain-of-embedding enables output-free llm self-evaluation

    Yiming Wang, Pei Zhang, Baosong Yang, Derek F Wong, and Rui Wang. Latent space chain-of-embedding enables output-free llm self-evaluation. arXiv preprint arXiv:2410.13640, 2024 f

    2024 arXiv

  106. [114]

    Towards understanding how transformer perform multi-step reasoning with matching operation

    Zhiwei Wang, Yunji Wang, Zhongwang Zhang, Zhangchen Zhou, Hui Jin, Tianyang Hu, Jiacheng Sun, Zhenguo Li, Yaoyu Zhang, and Zhi-Qin John Xu. Towards understanding how transformer perform multi-step reasoning with matching operation. arXiv e-prints, pages arXiv--2405, 2024 g

    2024

  107. [115]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  108. [116]

    Parallel continuous chain-of-thought with jacobi iteration

    Haoyi Wu, Zhihao Teng, and Kewei Tu. Parallel continuous chain-of-thought with jacobi iteration. arXiv preprint arXiv:2506.18582, 2025

    2025

  109. [117]

    Unifying and verifying mechanistic interpretations: A case study with group operations

    Wilson Wu, Louis Jaburi, Jacob Drori, and Jason Gross. Unifying and verifying mechanistic interpretations: A case study with group operations. arXiv preprint arXiv:2410.07476, 2024

    2024 arXiv

  110. [118]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025 a

    2025 arXiv

  111. [119]

    Mmada: Multimodal large diffusion language models

    Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505.15809, 2025 b

    2025 arXiv

  112. [120]

    Do large language models latently perform multi-hop reasoning? arXiv preprint arXiv:2402.16837, 2024 a

    Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? arXiv preprint arXiv:2402.16837, 2024 a

    2024 arXiv

  113. [121]

    Gated linear attention transformers with hardware-efficient training

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635, 2023

    2023 arXiv

  114. [122]

    Gated delta networks: Improving mamba2 with delta rule

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXiv:2412.06464, 2024 b

    2024 arXiv

  115. [123]

    Parallelizing linear transformers with the delta rule over sequence length

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv:2406.06484, 2024 c

    2024 arXiv

  116. [124]

    Diffusion of thoughts: Chain-of-thought reasoning in diffusion language models, 2024

    Jiacheng Ye, Shansan Gong, Liheng Chen, Lin Zheng, Jiahui Gao, Han Shi, Chuan Wu, Xin Jiang, Zhenguo Li, Wei Bi, and Lingpeng Kong. Diffusion of thoughts: Chain-of-thought reasoning in diffusion language models, 2024. URL https://arxiv.org/abs/2402.07754

    2024 arXiv

  117. [125]

    Beyond autoregression: Discrete diffusion for complex reasoning and planning, 2025 a

    Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning, 2025 a . URL https://arxiv.org/abs/2410.14157

    2025 arXiv

  118. [126]

    Dream 7b, 2025 b

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025 b . URL https://hkunlp.github.io/blog/2025/dream

    2025

  119. [127]

    Enhancing auto-regressive chain-of-thought through loop-aligned reasoning

    Qifan Yu, Zhenyu He, Sijie Li, Xun Zhou, Jun Zhang, Jingjing Xu, and Di He. Enhancing auto-regressive chain-of-thought through loop-aligned reasoning. arXiv preprint arXiv:2502.08482, 2025 a

    2025 arXiv

  120. [128]

    Do llms really think step-by-step in implicit reasoning? arXiv preprint arXiv:2411.15862, 2024

    Yijiong Yu. Do llms really think step-by-step in implicit reasoning? arXiv preprint arXiv:2411.15862, 2024

    2024 arXiv

  121. [129]

    Back attention: Understanding and enhancing multi-hop reasoning in large language models

    Zeping Yu, Yonatan Belinkov, and Sophia Ananiadou. Back attention: Understanding and enhancing multi-hop reasoning in large language models. arXiv preprint arXiv:2502.10835, 2025 b

    2025 arXiv

  122. [130]

    Why lift so heavy? slimming large language models by cutting off the layers

    Shuzhou Yuan, Ercong Nie, Bolei Ma, and Michael F \"a rber. Why lift so heavy? slimming large language models by cutting off the layers. arXiv preprint arXiv:2402.11700, 2024

    2024 arXiv

  123. [131]

    Quiet-star: Language models can teach themselves to think before speaking

    Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09629, 2024

    2024 arXiv

  124. [132]

    Pretraining language models to ponder in continuous space

    Boyi Zeng, Shixiang Song, Siyuan Huang, Yixuan Wang, He Li, Ziwei He, Xinbing Wang, Zhiyu Li, and Zhouhan Lin. Pretraining language models to ponder in continuous space. arXiv preprint arXiv:2505.20674, 2025

    2025

  125. [133]

    Map-neo: Highly capable and transparent bilingual large language model series

    Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du, Yiming Liang, Yinghao Ma, Yizhi Li, Ziyang Ma, Bill Lin, Emmanouil Benetos, Huan Yang, Junting Zhou, Kaiji...

    2024 arXiv

  126. [134]

    Lightthinker: Thinking step-by-step compression

    Jintian Zhang, Yuqi Zhu, Mengshu Sun, Yujie Luo, Shuofei Qiao, Lun Du, Da Zheng, Huajun Chen, and Ningyu Zhang. Lightthinker: Thinking step-by-step compression. arXiv preprint arXiv:2502.15589, 2025 a

    2025

  127. [135]

    Lolcats: On low-rank linearizing of large language models

    Michael Zhang, Simran Arora, Rahul Chalamala, Alan Wu, Benjamin Spector, Aaryan Singhal, Krithik Ramesh, and Christopher R \'e . Lolcats: On low-rank linearizing of large language models. arXiv preprint arXiv:2410.10254, 2024 b

    arXiv

  128. [136]

    Test-time training done right

    Tianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang, Fujun Luan, Songlin Yang, Kalyan Sunkavalli, William T Freeman, and Hao Tan. Test-time training done right. arXiv preprint arXiv:2505.23884, 2025 b

    2025 arXiv

  129. [137]

    Investigating layer importance in large language models

    Yang Zhang, Yanfei Dong, and Kenji Kawaguchi. Investigating layer importance in large language models. arXiv preprint arXiv:2409.14381, 2024 c

    2024 arXiv

  130. [138]

    d1: Scaling reasoning in diffusion large language models via reinforcement learning, 2025

    Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. d1: Scaling reasoning in diffusion large language models via reinforcement learning, 2025. URL https://arxiv.org/abs/2504.12216

    2025 arXiv

  131. [139]

    Llada 1.5: Variance-reduced preference optimization for large language diffusion models, 2025 a

    Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Llada 1.5: Variance-reduced preference optimization for large language diffusion models, 2025 a . URL https://arxiv.org/abs/2505.19223

    2025 arXiv

  132. [140]

    Soft reasoning: Navigating solution spaces in large language models through controlled embedding exploration

    Qinglin Zhu, Runcong Zhao, Hanqi Yan, Yulan He, Yudong Chen, and Lin Gui. Soft reasoning: Navigating solution spaces in large language models through controlled embedding exploration. arXiv preprint arXiv:2505.24688, 2025 b

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.