Pith. sign in

REVIEW 4 major objections 5 minor 299 references

The paper claims that a two-step per-layer loop, paired with a memory-aware scaling recipe, lets looped MoE language models outperform vanilla Transformers trained under matched wall-clock pretraining cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 21:29 UTC pith:PABK2LOT

load-bearing objection A concrete, hardware-aware recipe for looped MoE pretraining with a genuine wall-clock win, but the 'same compute budget' claim is not FLOP-matched — the selected model does ~42% more per-token block work. the 4 major comments →

arxiv 2607.16051 v2 pith:PABK2LOT submitted 2026-07-17 cs.CL cs.AI

Loop the Loopies!

Zitian Gao , Yilong Chen , Yihao Xiao , Xinyu Yang , Ran Tao , Joey Zhou , Bryan Dai This is my paper
classification cs.CL cs.AI
keywords looped transformersmixture-of-expertslayer-loop recurrencecompute-matched scalingactivation memorysupervised pre-trainingreasoning language modelsscaling ladder
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to settle whether recurrent (looped) Transformers can compete with ordinary Transformers when pre-training compute is held fixed — not just when parameter counts are matched. It argues they can, provided recurrence is scheduled as a short per-layer loop (each stored layer runs twice before the next layer) and the model is sized by the Loopie Recipe: halve the stored layer count, use the resulting activation-memory savings to double the per-device microbatch, and reinvest the measured efficiency gain into wider and deeper stored layers. The paper reports that this recipe holds across a four-rung scaling ladder (0.15B to 1B baseline size) and that a 20B-A2B Loopie overtakes a 30B-A3B vanilla MoE reproduction after about 600B tokens at matched step time. If true, recurrence becomes a practical third scaling axis for large MoE pretraining, not just a parameter-efficiency curiosity.

Core claim

On its own terms, the paper's central claim is that looped MoE Transformers can outperform compute-matched vanilla MoE baselines when (i) recurrence is organized as layer-loop — each stored Attention/MoE layer is applied twice consecutively before the next layer — rather than whole-model looping, and (ii) stored width, stored depth, and loop count are chosen jointly so that measured per-optimizer-step wall-clock time matches the reference, not FLOPs. The enabling mechanism is an activation-memory transfer: with all recurrent applications inside one checkpointed unit, activation memory scales with stored depth rather than executed depth, so halving stored layers frees memory that is converted

What carries the argument

The central object is the layer-loop recurrence schedule: a stored Transformer/MoE layer is applied R times (here R=2) before control passes to the next stored layer, so effective depth is L·R while stored depth is L. The Loopie Recipe is the scaling rule that makes this competitive: halve stored depth, loop twice to preserve the leading-order block-execution count, use the checkpointing arrangement (all recurrent applications inside one checkpointed unit, so activation memory is proportional to stored depth) to double the per-device microbatch, and reinvest the measured speedup into width and depth until step time matches a chosen vanilla reference. This converts recurrence from a parameter

Load-bearing premise

The load-bearing premise is the checkpointing assumption that activation memory scales with stored depth L, not executed depth R·L: if each recurrent pass needs its own stored layer-boundary activations, the doubled microbatch — and with it the wall-clock compute match — disappears.

What would settle it

Re-train the same Loopie configuration under a checkpointing scheme that stores per-loop-boundary activations (so memory scales with R·L), keeping the microbatch at the reference size; if the measured step time no longer matches the vanilla baseline, or if a strictly FLOP-matched vanilla baseline (roughly 1.42x larger or trained on 1.42x more tokens) beats it, the central claim fails. The cleanest direct check is a strict-FLOP reproduction of the 30B-A3B comparison.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Recurrence becomes a viable third scaling axis: at a fixed wall-clock pretraining budget, a model that repeats a small stored stack twice can beat a larger single-pass model, so compute-matched looping should be considered alongside parameter and token scaling.
  • The layer-loop schedule itself matters: an ablation that removes it while keeping everything else fixed substantially lowers downstream scores, so the ordering of recurrent computation is an architectural choice, not an implementation detail.
  • The checkpointing assumption implies a hardware-software co-design opportunity: if activation checkpointing keeps all recurrent applications of a layer within one unit, activation memory scales with stored depth rather than executed depth, and future training stacks can exploit this to push loop counts or microbatch sizes further.
  • Supervised pre-training at pretraining scale — loss on target tokens only, 2T tokens, batch size 1024, sequence length 131K — improves both general knowledge and reasoning metrics while showing no catastrophic forgetting over roughly ten epochs, a claim separable from the looping story.
  • The scaling-ladder result suggests that looped MoE models trained on far fewer tokens (3.5T versus 25T for comparable post-trained models) can still match or exceed larger vanilla models on knowledge and reasoning benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper defines 'compute-matched' by measured optimizer-step wall-clock time, not FLOPs; its selected Loopie carries about 1.42x the leading-order block work of the 30B reference (per the paper's own Table 1). Under strict FLOP accounting, the comparison baseline would need to be larger or longer-trained, and the headline advantage could shrink or reverse.
  • The layer-loop advantage plausibly interacts with the MoE router: repeating the same layer locally may reinforce expert specialization and routing consistency, which is a testable prediction — the layer-loop benefit should be larger for MoE backbones than for matched dense backbones.
  • The supervised pre-training result suggests that catastrophic forgetting in instruction tuning is an artifact of small-batch optimization scale rather than of supervised masking itself; a direct test is to apply the same SPT-scale schedule to any strong base model and check whether pretraining metrics keep rising.
  • The activation-memory transfer has an inference-side corollary the paper leaves implicit: the same stored-depth-versus-executed-depth decoupling could be used to trade memory for effective depth at inference time, though the paper explicitly defers inference-time compute studies to future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript presents Loopie, a pair of looped Mixture-of-Experts language models (20B-A2B and 6B-A0.6B) built on layer-loop recurrence: each stored layer is applied twice before passing activations onward. The central methodological contribution is the Loopie Recipe, a hardware-aware procedure that halves stored depth, loops each layer twice, uses the resulting activation-memory savings to double the per-device microbatch, and spends the measured training-efficiency gain on wider/deeper stored networks, with final selection by measured per-optimizer-step wall-clock time. The paper claims that, under this operational definition of compute matching, Loopie-20B-A2B overtakes a Qwen3-30B-A3B-style reproduction after roughly 600B tokens (Figure 3) and that the advantage persists across a four-rung scaling ladder (Figure 4). A post-training pipeline (supervised pre-training plus GSPO/DAPO-style RL) is described, and Table 3 reports competitive reasoning results.

Significance. If the wall-clock-matched comparison were accepted as the relevant notion of compute budget, the paper would provide rare large-scale evidence that recurrent computation can be a viable scaling axis for MoE pretraining, rather than merely a parameter-efficiency device. The paper is also notable for shipping two trained models and for a transparent, measurement-driven selection procedure (Section 2.4). Its practical impact is potentially high. However, the central claim is currently expressed in FLOP-adjacent language ('same pre-training compute budget') while the evidence is wall-clock matched, and the selected model performs more arithmetic per token; this gap must be closed before the headline conclusion is established.

major comments (4)
  1. [Section 2.4, Table 1, Figure 3] The paper defines compute matching as equal measured end-to-end optimizer-step time (Section 2.4: 'We match models by realized pre-training cost rather than by exact theoretical FLOPs'). Under this definition, Loopie-20B-A2B is selected with normalized leading-order block compute proxy C-hat = 1.424 relative to the reference (Table 1), and Figure 3 reports realized throughput of 261.53 TFLOPS vs 189.65 TFLOPS (ratio ~1.38). Thus the same wall-clock window corresponds to substantially more nominal arithmetic for Loopie. Because all scaling-ladder rungs (Appendix Table 7) also give Loopie higher leading-order compute (e.g., rung 4: 25*1408^2*2 ~ 99.1e6 vs 46*1280^2 ~ 75.4e6), no comparison in the paper is FLOP-matched. The observed downstream advantage can therefore be explained by the additional arithmetic alone. The abstract and conclusion state 'same compute budget' without the wall-clo
  2. [Section 2.6, Figure 4, Appendix Table 7] The scaling ladder is presented as evidence that the Loopie advantage persists across model scales, but the matched-compute claim is not actually demonstrated. No measured optimizer-step times or throughput numbers are reported for the ladder; the text only asserts that each Loopie model's per-step wall-clock time 'exactly matches' the baseline. Moreover, every rung gives Loopie a higher leading-order FLOP proxy than its vanilla counterpart, so the ladder inherits the same FLOP-mismatch issue as the main comparison. The reported gaps (+1.1, +0.6, +1.7, +2.2 average points) come from single runs with no error bars or multiple seeds, making it impossible to assess whether these differences are significant. I would like to see the actual timing/matching data for each rung, or a clear statement that these are wall-clock-matched but not FLOP-matched comparisons.
  3. [Section 2.4, activation memory equation] The entire Loopie Recipe depends on the claim that activation memory scales with stored depth L rather than executed depth R*L, based on the checkpointing assumption that 'all recurrent applications of a stored layer are enclosed in the same checkpointed unit.' This is plausibly true for layer-loop, but it is a critical implementation detail: if recurrent forward/backward passes require separate stored activations per loop application, the memory headroom that funds the doubled microbatch — and therefore the wall-clock compute match — disappears. The paper says the measured memory footprint was used for candidate feasibility, but no memory profiles or checkpointing diagrams are reported. Please provide measurements or a precise description of the checkpointing implementation to substantiate this assumption.
  4. [Section 4.1, Table 2, Future Work] The supervised pre-training (SPT) section makes strong comparative claims: Table 2 asserts that SFT degrades pre-training metrics, PT leaves reasoning metrics unchanged, and SPT improves both. However, no experiments comparing SPT against SFT or PT baselines are reported in the paper; only SPT curves are shown (Figures 10-11). The Future Work section itself concedes, 'Due to limited computational resources, we were unable to conduct a sufficiently comprehensive ablation study of supervised pre-training.' This is an explicit admission that the SPT contribution is not yet supported by controlled evidence. The claims in Section 4.1 and Table 2 should be softened to reflect what is actually measured, or the missing baseline experiments should be provided.
minor comments (5)
  1. [Section 2.7, Figure 5] The figure caption says 'Loopie-5B' while the text refers to Loopie-6B-A0.6B; also the label '2.14x speedup' is unexplained — speedup relative to what, and measured under which conditions? Please clarify.
  2. [Section 2.8, Figure 6] The text explicitly states that the 2x layer baseline has substantially higher actual training compute than the 2x loop model and that 'the figure should not be read as evidence that 2x looping is dominated by 2x layer scaling.' This caveat undermines the section's conclusion that the marginal benefit of recurrence is largest at R=2. Either present a genuinely compute-matched comparison or remove the figure from the argument.
  3. [Table 3] The 'Pre-training tokens' column has a formatting issue ('Unknown3.5T'), and the comparison with GPT-OSS-20B-A2B High is weakened by the unknown pretraining budget. Please state which token counts include both Stage-1 and Stage-2 pretraining.
  4. [Front matter] The Models and Code entries at the top of the paper do not include URLs. For reproducibility, provide links to the checkpoints and code repositories.
  5. [Appendix C, Table 7] The column 'Width/Depth' is computed as D/(L*N), but the table header does not define N. Please add the definition and note that N is the layer-loop count.

Circularity Check

0 steps flagged

No significant circularity: the Loopie Recipe is an empirical selection procedure and all central comparisons are measured, not derived from fitted inputs.

full rationale

The paper's central derivation chain is the Loopie Recipe (Section 2.4): starting from a non-recurrent Qwen3-like 30B-A3B reference, it halves stored layers, sets R=2, uses the resulting memory headroom to double the per-device microbatch size, and then selects the final (D,L) configuration whose measured optimizer-step wall-clock time most closely matches the reference. No step predicts the downstream benchmark scores from a fitted parameter; the layer-loop-vs-model-loop comparison (Figure 2), the Loopie-vs-vanilla comparison (Figure 3), the scaling ladder (Figure 4), and the layer-loop ablation (Figure 5) are all directly measured. The only analytical expressions used, C ∝ L D^2 R and M_act ∝ s b D L, are explicitly presented as scaling proxies rather than predictors of final timing: the paper states 'These quantities are used to describe the candidates, not to predict their final optimizer-step times.' The compute-matching criterion is the directly measured relation t_step(D1,L1,R1;2b0,g0/2) ≈ t_step(D0,L0,R0;b0,g0), and the selected model's 1.424× leading-order block compute proxy is presented transparently as nominal work, not as a matched-FLOP equivalence. Self-citations (Gao et al. 2025b,c) supply background motivation and data-efficiency arguments but are not load-bearing for the central performance claims, which stand on the paper's own measurements. The wall-clock-versus-FLOPs definition of 'compute-matched' is a potential interpretation issue or confound for a FLOP-based reading of the abstract, but it is not circular: the paper explicitly disclaims analytical FLOP equality and defines its operational criterion. No equation reduces the claimed result to its own inputs, and no fitted value is renamed as a prediction. Therefore the circularity burden is low and no circular step is present.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claim rests primarily on engineering assumptions: the checkpointing-memory model, the wall-clock definition of compute, and the faithfulness of the reproduced Qwen3 baseline. No new physical or mathematical entities are introduced. The free parameters are architectural and schedule choices selected empirically; they are not fitted to the final benchmark scores but they do determine the compute-matched comparison.

free parameters (4)
  • Recurrent loop count R = 2
    Chosen empirically from the loop-count sweep in Section 2.8; the paper states the marginal benefit of recurrence is largest at R=2. Not derived from first principles.
  • Stored width/depth for Loopie-20B-A2B = D=2304, L=27, R=2
    Selected from the grid search in Table 1 by matching measured per-step wall-clock time to the Qwen3-30B-A3B reference. The selection criterion is infrastructure timing rather than a fit to benchmark scores.
  • Microbatch / gradient-accumulation schedule = b1=2b0, g1=g0/2
    Chosen to exploit the memory headroom from halved stored depth; this schedule is load-bearing for the wall-clock compute-matching claim (Section 2.4).
  • Supervised pre-training token budget = 2T tokens (~10 epochs)
    Chosen scale for SPT in Section 4.1; the paper admits in Section 6 that no comprehensive SPT ablation was run.
axioms (5)
  • domain assumption Activation memory under Megatron checkpointing scales with stored depth L, not executed depth R·L.
    Section 2.4 states 'all recurrent applications of a stored layer are enclosed in the same checkpointed unit', giving M_act ∝ s·b·D·L. The memory headroom for the doubled microbatch depends on this.
  • ad hoc to paper Measured end-to-end optimizer-step time defines matched pre-training compute.
    Section 2.4: 'we match models by realized pre-training cost rather than by exact theoretical FLOPs'. This operational definition is load-bearing for the 'same compute budget' claim.
  • standard math Leading-order pre-training compute satisfies C ∝ L·D²·R.
    Used in Section 2.4 to construct the recurrent seed model and candidate configurations; a standard FLOP proxy following Hoffmann et al.
  • domain assumption The Megatron reproduction of Qwen3-30B-A3B is a faithful baseline.
    Section 2.5 compares against 'our reproduction of Qwen3-30B-A3B'; no external validation of the reproduction is provided.
  • domain assumption The eight-benchmark average is a sufficient proxy for general model quality.
    Section 3.1 defines the evaluation used for scaling curves and ablations; a different benchmark set could change the reported rankings.

pith-pipeline@v1.3.0-alltime-deepseek · 32907 in / 14884 out tokens · 135658 ms · 2026-08-01T21:29:50.990457+00:00 · methodology

0 comments
read the original abstract

We present the Loopie series, consisting of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N times increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. With a novel post-training method, Loopie develops strong reasoning abilities and achieves frontier-level reasoning performance.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

299 extracted references · 1 canonical work pages

  1. [1]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    Transformer Layers as Painters , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =. 2025 , doi =

    2025

  2. [2]

    2025 , eprint=

    Group Sequence Policy Optimization , author=. 2025 , eprint=

    2025

  3. [3]

    2026 , eprint=

    Achieving Gold-Medal-Level Olympiad Reasoning via Simple and Unified Scaling , author=. 2026 , eprint=

    2026

  4. [4]

    2019 , doi =

    Tenney, Ian and Das, Dipanjan and Pavlick, Ellie , booktitle =. 2019 , doi =

    2019

  5. [5]

    What Does

    Jawahar, Ganesh and Sagot, Beno. What Does. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages =. 2019 , doi =

    2019

  6. [6]

    Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise

    Bae, Sangmin and Fisch, Adam and Harutyunyan, Hrayr and Ji, Ziwei and Kim, Seungyeon and Schuster, Tal , booktitle =. Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise

  7. [7]

    2026 , eprint =

    What Survives When You Compress a Recursive Reasoner for the Edge? , author =. 2026 , eprint =

    2026

  8. [8]

    2026 , eprint =

    On the Residual Scaling of Looped Transformers: Stability and Transferability , author =. 2026 , eprint =

    2026

  9. [9]

    2026 , eprint =

    Looped Transformers with Layer Normalization Provably Learn the Power Method , author =. 2026 , eprint =

    2026

  10. [10]

    2026 , eprint =

    Chain-of-Thought and Compressed Looped Transformers: A Memory-Budget Separation , author =. 2026 , eprint =

    2026

  11. [11]

    2026 , eprint =

    Interaction Locality in Hierarchical Recursive Reasoning , author =. 2026 , eprint =

    2026

  12. [12]

    2026 , eprint =

    One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformer , author =. 2026 , eprint =

    2026

  13. [13]

    2026 , eprint =

    Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models , author =. 2026 , eprint =

    2026

  14. [14]

    2026 , eprint =

    Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent , author =. 2026 , eprint =

    2026

  15. [15]

    Flat Iteration in Shared-Weight Transformers , author =

    Hierarchical vs. Flat Iteration in Shared-Weight Transformers , author =. 2026 , eprint =

    2026

  16. [16]

    2026 , eprint =

    A Mechanistic Analysis of Looped Reasoning Language Models , author =. 2026 , eprint =

    2026

  17. [17]

    2026 , eprint =

    Relational Preference Encoding in Looped Transformer Internal States , author =. 2026 , eprint =

    2026

  18. [18]

    2026 , eprint =

    Understanding Dynamic Compute Allocation in Recurrent Transformers , author =. 2026 , eprint =

    2026

  19. [19]

    2025 , eprint =

    Two-Scale Latent Dynamics for Recurrent-Depth Transformers , author =. 2025 , eprint =

    2025

  20. [20]

    2026 , eprint =

    DiscoLoop: Looping Discrete Embeddings and Continuous Hidden States for Multi-hop Reasoning , author =. 2026 , eprint =

    2026

  21. [21]

    2026 , eprint =

    Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers , author =. 2026 , eprint =

    2026

  22. [22]

    2026 , eprint =

    CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield , author =. 2026 , eprint =

    2026

  23. [23]

    2026 , eprint =

    Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping , author =. 2026 , eprint =

    2026

  24. [24]

    2026 , eprint =

    LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling , author =. 2026 , eprint =

    2026

  25. [25]

    2026 , eprint =

    Looped World Models , author =. 2026 , eprint =

    2026

  26. [26]

    2026 , eprint =

    Skip a Layer or Loop It? Learning Program-of-Layers in LLMs , author =. 2026 , eprint =

    2026

  27. [27]

    2026 , eprint =

    When to Re-Plan: Subgoal Persistence in Hierarchical Latent Reasoning , author =. 2026 , eprint =

    2026

  28. [28]

    2026 , eprint =

    CART: Context-Anchored Recurrent Transformer -- A Parameter-Efficient Architecture with Learned Stability , author =. 2026 , eprint =

    2026

  29. [29]

    2026 , eprint =

    Fixed-Point Masked Generative Modeling , author =. 2026 , eprint =

    2026

  30. [30]

    2026 , eprint =

    CosmicFish-HRM: Adaptive Reasoning via Hierarchical Recurrent Mechanisms in Compact Language Models , author =. 2026 , eprint =

    2026

  31. [31]

    2026 , eprint =

    Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models , author =. 2026 , eprint =

    2026

  32. [32]

    2026 , eprint =

    Looped Diffusion Language Models , author =. 2026 , eprint =

    2026

  33. [33]

    2026 , eprint =

    HRM-Text: Efficient Pretraining Beyond Scaling , author =. 2026 , eprint =

    2026

  34. [34]

    2026 , eprint =

    Solve the Loop: Attractor Models for Language and Reasoning , author =. 2026 , eprint =

    2026

  35. [35]

    2026 , eprint =

    Simply Stabilizing the Loop via Fully Looped Transformer , author =. 2026 , eprint =

    2026

  36. [36]

    2026 , eprint =

    LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models , author =. 2026 , eprint =

    2026

  37. [37]

    2026 , eprint =

    Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models , author =. 2026 , eprint =

    2026

  38. [38]

    2026 , eprint =

    Hyperloop Transformers , author =. 2026 , eprint =

    2026

  39. [39]

    2026 , eprint =

    LASER: Low-Rank Activation SVD for Efficient Recursion , author =. 2026 , eprint =

    2026

  40. [40]

    2026 , eprint =

    ELT: Elastic Looped Transformers for Visual Generation , author =. 2026 , eprint =

    2026

  41. [41]

    2026 , eprint =

    Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation , author =. 2026 , eprint =

    2026

  42. [42]

    2026 , eprint =

    Recursive Inference Machines for Neural Reasoning , author =. 2026 , eprint =

    2026

  43. [43]

    2026 , eprint =

    AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depth , author =. 2026 , eprint =

    2026

  44. [44]

    2026 , eprint =

    SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion , author =. 2026 , eprint =

    2026

  45. [45]

    2026 , eprint =

    Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models , author =. 2026 , eprint =

    2026

  46. [46]

    2026 , eprint =

    Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Models , author =. 2026 , eprint =

    2026

  47. [47]

    2026 , eprint =

    Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves , author =. 2026 , eprint =

    2026

  48. [48]

    International Conference on Learning Representations , year =

    ChainGPT: Dual-Reasoning Model with Recurrent Depth and Multi-Rank State Updates , author =. International Conference on Learning Representations , year =

  49. [49]

    International Conference on Learning Representations , year =

    MoDr: Mixture-of-Depth-Recurrent Transformers for Test-Time Reasoning , author =. International Conference on Learning Representations , year =

  50. [50]

    2025 , eprint =

    Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs , author =. 2025 , eprint =

    2025

  51. [51]

    2025 , eprint =

    Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA , author =. 2025 , eprint =

    2025

  52. [52]

    2025 , eprint =

    AlgoFormer: An Efficient Transformer Framework with Algorithmic Structures , author =. 2025 , eprint =

    2025

  53. [53]

    2025 , eprint =

    CoTFormer: A Chain-of-Thought Driven Architecture with Budget-Adaptive Computation Cost at Inference , author =. 2025 , eprint =

    2025

  54. [54]

    2026 , eprint =

    G-RRM: Guiding Symbolic Solvers with Recurrent Reasoning Models , author =. 2026 , eprint =

    2026

  55. [55]

    2026 , eprint =

    MPCoT: Reward-Guided Multi-Path Latent Reasoning for Test-Time Scalable Vision-Language-Action , author =. 2026 , eprint =

    2026

  56. [56]

    2026 , eprint =

    Test-Time Compute Scaling for ASR with Depth-Conditioned Looped Transformers , author =. 2026 , eprint =

    2026

  57. [57]

    2026 , eprint =

    BA-T: An Iterative Transformer for Two-View Bundle Adjustment , author =. 2026 , eprint =

    2026

  58. [58]

    2026 , eprint =

    Déjà View: Looping Transformers for Multi-View 3D Reconstruction , author =. 2026 , eprint =

    2026

  59. [59]

    2026 , eprint =

    Recursive Vision Transformer with Dynamic Depth and Width Adjustment for Resource-Efficient Image Semantic Communication , author =. 2026 , eprint =

    2026

  60. [60]

    2026 , eprint =

    PERL: Parameter Efficient Reasoning in CLIP Latent Space , author =. 2026 , eprint =

    2026

  61. [61]

    2026 , eprint =

    Recurrent-Depth VLA: Implicit Test-Time Compute Scaling of Vision-Language-Action Models via Latent Iterative Reasoning , author =. 2026 , eprint =

    2026

  62. [62]

    2025 , eprint =

    Recurrent Transformer-Based Near- and Far-Field THz Wideband Channel Estimation for UM-MIMO , author =. 2025 , eprint =

    2025

  63. [63]

    2026 , eprint =

    Soft Mixture-of-Recursions: Going Deeper with Recursive Vision Transformers , author =. 2026 , eprint =

    2026

  64. [64]

    2026 , eprint =

    Rethinking Depth: A study of the Recursive-Transformer for Speech Recognition , author =. 2026 , eprint =

    2026

  65. [65]

    2026 , eprint =

    RD-ViT: Recurrent-Depth Vision Transformer for Semantic Segmentation with Reduced Data Dependence Extending the Recurrent-Depth Transformer Architecture to Dense Prediction , author =. 2026 , eprint =

    2026

  66. [66]

    2025 , eprint =

    Hyper-SET: Designing Transformers via Hyperspherical Energy Minimization , author =. 2025 , eprint =

    2025

  67. [67]

    2025 , eprint =

    Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning , author =. 2025 , eprint =

    2025

  68. [68]

    European Conference on Computer Vision , year =

    Sliced Recursive Transformer , author =. European Conference on Computer Vision , year =. 2111.05297 , archivePrefix =

    Pith/arXiv arXiv

  69. [69]

    Ouroboros: Dynamic Weight Generation for Recursive Transformers via Input-Conditioned

    Jaber, Jaber and Jaber, Osama , year =. Ouroboros: Dynamic Weight Generation for Recursive Transformers via Input-Conditioned. 2604.02051 , archivePrefix =

    arXiv

  70. [70]

    2026 , eprint =

    Associative-State Universal Transformers: Sparse Retrieval Meets Structured Recurrence , author =. 2026 , eprint =

    2026

  71. [71]

    arXiv preprint arXiv:1707.06347 , year =

    Proximal Policy Optimization Algorithms , author =. arXiv preprint arXiv:1707.06347 , year =

    Pith/arXiv arXiv

  72. [72]

    2025 , eprint=

    Parallel Loop Transformer for Efficient Test-Time Computation Scaling , author=. 2025 , eprint=

    2025

  73. [73]

    arXiv preprint arXiv:2402.03300 , year =

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , author =. arXiv preprint arXiv:2402.03300 , year =

    Pith/arXiv arXiv

  74. [74]

    arXiv preprint arXiv:2503.14476 , year =

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale , author =. arXiv preprint arXiv:2503.14476 , year =

    Pith/arXiv arXiv

  75. [75]

    arXiv preprint arXiv:2506.14965 , year =

    Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective , author =. arXiv preprint arXiv:2506.14965 , year =

    Pith/arXiv arXiv

  76. [76]

    arXiv preprint arXiv:2505.22312 , year =

    Skywork Open Reasoner 1 Technical Report , author =. arXiv preprint arXiv:2505.22312 , year =

    Pith/arXiv arXiv

  77. [77]

    2025 , howpublished =

    DeepScaleR: Effective RL Scaling of Reasoning Models via Iterative Context Lengthening , author =. 2025 , howpublished =

    2025

  78. [78]

    arXiv preprint arXiv:2504.14655 , year =

    LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs , author =. arXiv preprint arXiv:2504.14655 , year =

    Pith/arXiv arXiv

  79. [79]

    arXiv preprint arXiv:2312.14852 , year =

    TACO: Topics in Algorithmic COde Generation Dataset , author =. arXiv preprint arXiv:2312.14852 , year =

    Pith/arXiv arXiv

  80. [80]

    2025 , howpublished =

    TACO-Verified , author =. 2025 , howpublished =

    2025

Showing first 80 references.