Pith. sign in

REVIEW 2 major objections 5 minor 35 references

Looped Transformers' learned halting gates fail mainly because their training objective warps the recurrent trajectory, not because the gate is too weak.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:09 UTC pith:Z7OTU5G7

load-bearing objection The trajectory-readout decomposition is real and the experiments are careful; the causal story about loss-weighting is the one piece that outruns the evidence. the 2 major comments →

arxiv 2607.20519 v1 pith:Z7OTU5G7 submitted 2026-07-08 cs.LG

Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts

Andrei Cristian Popescu , Haitz S\'aez de Oc\'ariz Borde , Pietro Li\`o This is my paper
classification cs.LG
keywords adaptive depthlooped Transformerslearned halting gatesearly exitPonderNettrajectory readoutfixed-prior depth supervisionlatency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that adaptive depth in looped Transformers is best understood as a trajectory–readout problem, not a gate-learning problem. The same learned exit distribution typically serves both as the inference-time stopping rule and as the training-time weighting of per-depth losses, so the gate changes the very trajectory it reads out from. The authors claim that this entanglement, rather than limited gate expressivity, is what makes learned halting gates underperform: jointly trained gates warp the recurrent trajectory, while post-hoc readouts on fixed-prior trajectories match or beat them. Evidence comes from controlled synthetic tasks and from frozen billion-parameter checkpoints, where simple confidence readouts are competitive with pretrained ponder gates.

Core claim

On the paper's own terms, the central discovery is that the trajectory induced by the training objective, not the halting gate's capacity, is the binding constraint on adaptive compute in looped Transformers. Training with an input-independent fixed prior over depths produces difficulty-aware recurrent states—harder inputs stay uncertain longer—and on those trajectories cheap confidence or convergence readouts reach high accuracy at low average depth. When the same gate class is fit post hoc on frozen fixed-prior trajectories, it recovers strong early exits, but fitting it on trajectories produced by joint gate training does not, localizing the failure to the trajectory. The pattern transfer

What carries the argument

The central object is the recurrent trajectory τ(x) = {h_1, …, h_T} produced by a shared Transformer block, together with the exit distribution q_t(x) used in PonderNet-style halting gates (gates that learn a distribution over exit depths). The load-bearing mechanism is the entanglement between q's two roles: at inference q chooses the exit, while during training q weights the per-depth cross-entropy losses, so optimizing q also reshapes the trajectory it reads out from. Separating the roles via fixed-prior depth supervision—loss weights π independent of the input—lets the authors attribute adaptive-compute failure to trajectory formation rather than gate expressivity.

Load-bearing premise

The load-bearing premise is that the joint-gate trajectory's weakness is caused by the loss-weighting entanglement itself, not by confounds such as the added gate head, the KL regularizer, or a shifted loss landscape; the paper's frozen-trajectory control isolates the readout side only.

What would settle it

Train a looped model with fixed-prior depth supervision but attach an auxiliary gate head whose predicted distribution does not enter the backbone loss (or whose weights are detached from the backbone). If the forced-exit accuracy curve looks as weak as the joint-gate curves, the entanglement attribution is falsified; if it looks like the fixed-prior curves, the paper's claim is supported. A scale test would retrain a small looped LM under the decoupled objective and compare its Pareto frontier against joint-gate training.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A learned, input-dependent halting policy is not necessary for useful adaptive depth: fixed-prior depth supervision alone yields trajectories whose intermediate states support strong early exits.
  • Gate expressivity is not the binding constraint: the same linear/MLP gate class recovers strong exits when fit post hoc on a frozen fixed-prior trajectory, but not on a trajectory produced by joint gate training.
  • On Ouro-1.4B and 2.6B, the pretrained ponder gate is competitive but not uniformly Pareto-optimal; simple confidence or convergence readouts improve both accuracy and compute in some benchmark–model pairs.
  • Reducing average exit depth translates into measured latency savings (e.g., 1.24–1.58× on Ouro-1.4B benchmarks and roughly 1.3× on MANO), so better readouts are practically useful, not just a metric artifact.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: an untested corollary of the attribution is that retraining or fine-tuning looped models with fixed-prior-style depth supervision—or decoupling the gate from the loss—should sharpen the Pareto frontier more than further gate engineering; the Ouro experiments only freeze trajectories, so this remains open.
  • Beyond the paper: the difficulty-aware signals that persist out of distribution (harder expressions stay uncertain longer) suggest the trajectory itself could serve as a difficulty estimator, useful for routing, deferral, or curriculum design beyond early exiting.
  • Beyond the paper: the geometric-prior sweep finding an optimum at moderate early-depth weighting points to a cheap design knob for future looped models: depth-loss schedules that concentrate early supervision but retain late coverage may be tuned per compute budget.
  • Beyond the paper: because the winning readout varies across model size and benchmark, a practical extension would be adaptive pooling of readouts (e.g., selecting among entropy, margin, and hidden-state cosine similarity per example or per domain), which the paper does not test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a trajectory–readout decomposition for adaptive depth in looped Transformers. It argues that learned halting gates (PonderNet-style) fail primarily because their training-time role as per-depth loss weights degrades the recurrent trajectory itself, rather than because of gate expressivity or readout quality. The authors train fixed-prior and learned-gate models on MANO and parity tasks, evaluate post-hoc confidence/convergence readouts, fit gates on frozen trajectories, and compare pretrained Ouro-1.4B/2.6B ponder gates against simple readouts. They report that fixed-prior trajectories with simple readouts often match or beat learned gates, that frozen-trajectory experiments localize the failure to trajectory quality, and that pretrained ponder gates are competitive but not uniformly Pareto-optimal.

Significance. If the main claim holds, the paper offers a useful reframing of adaptive-depth diagnostics and a concrete caution about PonderNet-style objectives in looped Transformers. The evaluation protocol is a genuine strength: MANO readout thresholds are selected on seed-43 validation and evaluated on seed-44 test, MANO results are reported over three seeds with standard deviations, forced-exit curves separate trajectory quality from readout quality, and frozen-trajectory experiments isolate the readout side. The latency measurements ground the loop-count metric in practical savings. The central causal attribution, however, is not fully supported by the current experiments, and the large-scale claims lack uncertainty estimates.

major comments (2)
  1. [§5.3–5.4, Eq. (2)–(3)] The paper's central attribution — that joint learned-gate training degrades trajectories mainly through loss-weighting entanglement — is not directly identified. The comparison between the joint-gate objective (Eq. 2) and fixed-prior objective (Eq. 3) changes several things at once: the per-depth weights become input-dependent, a gate head with parameters φ is added, and a KL(β) regularizer is introduced. Forced-exit curves (Fig. 5) show that the resulting trajectories differ, and frozen-trajectory experiments (Fig. 6, App. H) show that a post-hoc gate cannot repair a jointly trained trajectory, but neither isolates the loss-weighting mechanism itself. A gate trained on frozen states cannot alter the trajectory, so it controls only the readout side. The conclusion that fixed-prior supervision works because it decouples trajectory formation from exit selection requires an ablation in whic
  2. [Table 3, §5.5] The large-scale claim that pretrained ponder gates are not uniformly Pareto-optimal rests on single-run comparisons in Table 3. Several decisive contrasts are small: Ouro-1.4B MMLU 67.6 vs 67.3, Ouro-1.4B ARC-Challenge 59.8 vs 59.5, Ouro-2.6B ARC-Challenge 66.6 vs 65.8; in the three 'better on both' cases the accuracy differences are 0.1–0.3 points. No confidence intervals, bootstrap estimates, or multiple held-out subsets are reported. In addition, the post-hoc readout is selected as the best among several candidates on the same validation split used for threshold selection, which introduces selection bias in its favor. Please report uncertainty (e.g., bootstrap over examples or multiple held-out splits) and account for multiple-readout selection, or soften the claim to 'competitive' rather than 'not uniformly Pareto-optimal.'
minor comments (5)
  1. [§5.6 / Table 4a] The text says 'measured end-to-end latency' but Table 4a is captioned with 'estimated latency reductions' and the prose later says 'The Ouro estimates provide the same latency-side sanity check.' Specify which rows are measured and which are estimated, and describe the estimation procedure.
  2. [Appendix B.1] 'Block size 32' is not defined. If this is the sequence length, attention window, or something else, please state it explicitly.
  3. [Appendix F] Training examples for parity are described as 'generated online and balanced across bit lengths,' but the validation/test split for threshold selection is not specified. State how a held-out split is constructed under online generation.
  4. [Appendix C.4] The title 'Difficulty Extrapolation' is somewhat misleading because the authors themselves note this is not a length-generalization benchmark. Consider renaming to something like 'Difficulty Diagnostics Beyond the Training Range.'
  5. [General] The paper does not state whether code, configuration files, and evaluation scripts will be released. For reproducibility of the threshold-grid and Ouro evaluation protocols, please include an availability statement.

Circularity Check

0 steps flagged

No significant circularity: all headline results are held-out empirical evaluations with validation-calibrated thresholds and external Ouro checkpoints.

full rationale

The paper's central claims are empirical evaluations rather than derivations from fitted inputs. Fixed-prior and learned-gate models are trained under distinct objectives (Section 3.2 vs 3.3), and post-hoc readout thresholds are calibrated on validation data (MANO seed 43; Ouro validation or a 20% validation subset) and evaluated on separate held-out test splits (MANO seed 44; Ouro test or 80% held-out subset), as specified in Sections 4.4, B.5, and B.8. This is standard model selection, not circular confirmation. The Ouro experiments use released checkpoints as-is without training or modification, so the pretrained gate is treated as external input. The one self-citation (Català et al., 2026, in Related Work) is background regarding entropy dynamics and is not load-bearing for any result. The causal attribution that joint gate training degrades trajectories through depth-weighted losses is not fully isolated—joint training also adds a gate head and KL regularizer—but that is a causal-identification limitation, not circularity: the paper does not define the conclusion into the setup or fit a parameter and then rename it a prediction. No equation reduces to its own input, and no load-bearing claim rests on an unverified self-citation chain.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claims rest on the five domain assumptions above plus validation-calibrated threshold parameters. There is no free parameter that a derivation depends on in the equation-level sense — the paper makes no deductive derivations, so the ledger's honest cost is the transfer assumption from synthetic to scale, the shared-head validity assumption, and the difficulty-ordering assumption. No invented entities are introduced: the trajectory–readout lens, fixed-prior supervision, and frozen-trajectory gate fitting are methodological constructs rather than postulated entities with independent falsifiable handles.

free parameters (3)
  • Geometric prior rate λ (fixed-prior depth supervision) = 0.3 in headline results; swept over {0.2, 0.3, 0.5, 0.7}
    Controls how strongly early recurrent depths are supervised (Sections 3.3, B.4). The favorable MANO frontiers (D@99 ≈ 1.5 loops) depend on selecting λ=0.3; the paper shows the optimum lies inside the swept range, but the headline is partly a validation-selected hyperparameter result, and larger λ degrades the accuracy ceiling.
  • KL regularization strength β for learned gates = Swept per prior; 'highest-performing β' selected per gate class (Table 2 caption)
    The learned-gate comparison explicitly selects the best β per prior and gate class (B.4), biasing the comparison in the gates' favor; the fact that gates still lose to fixed-prior readouts strengthens the trajectory-formation conclusion rather than weakening it.
  • Readout thresholds (γ/ε/α per readout) = Validation-calibrated; 51-point empirical-quantile grid (B.5)
    Each post-hoc readout and gate threshold is fitted to validation statistics then applied to test. These are legitimate held-out free parameters; calibration is properly separated from test evaluation, and the same validation/test split is used for all readouts.
axioms (5)
  • domain assumption Operation count on MANO is a valid ground-truth difficulty ordering
    The 'difficulty-aware trajectory' claim is measured as correlation between operation count and entropy/margin/KL signals (Tables 5–6, App. C). If operation count were not the operative difficulty factor, the difficulty-ordering interpretation would not carry the weight the paper assigns.
  • domain assumption Per-depth logits with a shared LM head give meaningful intermediate predictions
    Forced-exit evaluation and all confidence/convergence readouts assume the shared prediction head applied at every loop measures trajectory quality (A.1). The paper provides no check of intermediate-head calibration against per-depth-head variants, though early-exit literature supports this assumption.
  • domain assumption Diagnostic conclusions from MANO/parity (T=6, small models) transfer to large looped LLMs
    Ouro checkpoints are frozen; trajectory formation is never manipulated at scale (Section 5.5). The mechanism claim at scale is assumed rather than tested; only the readout-side pattern is observed.
  • domain assumption Ouro's released checkpoints implement the described two-stage PonderNet gate and T_max=4 trajectory
    All Ouro readout comparisons treat the released gate and trajectories as faithful to the architecture description in A.2 (Stage I entropy-regularized training, Stage II marginal-utility gate fine-tuning); the paper does not independently verify gate internals.
  • domain assumption Entropy/margin convergence indicates 'readiness to halt'
    Confidence and convergence readouts assume predictive uncertainty over the vocabulary is a valid proxy for task correctness. The oracle-earliest-correct markers in App. G show a gap between these signals and the true correctness ceiling, so this assumption limits all readout frontiers, both post-hoc and native.

pith-pipeline@v1.3.0-alltime-deepseek · 21073 in / 20769 out tokens · 198808 ms · 2026-08-02T08:09:47.064791+00:00 · methodology

0 comments
read the original abstract

Looped Transformers increase test-time computation by repeatedly applying a shared recurrent block. Learned halting objectives in looped Transformers typically use a single exit distribution both as the inference-time stopping rule and as the training-time weighting of per-depth losses. This entangles exit selection with trajectory formation: the gate not only chooses which recurrent state to use, but also determines how strongly each intermediate state is supervised. Consequently, poor adaptive-compute performance can arise from the readout, the induced trajectory, or their interaction. We study adaptive depth in looped Transformers through this trajectory--readout lens, across controlled synthetic tasks (modular arithmetic and binary parity) and large-scale Ouro-1.4B and 2.6B checkpoints. We find that fixed-prior depth supervision, which shapes the trajectory without an input-dependent halting policy, produces difficulty-aware trajectories whose intermediate states expose useful stopping signals, and that simple post-hoc confidence readouts often match or outperform learned linear and MLP gates. Fitting gates on frozen trajectories localizes the failure: it appears to stem mainly from the trajectory induced by joint gate training rather than from limited gate expressivity. The same pattern is present in Ouro evaluations, where pretrained ponder gates are competitive but not uniformly Pareto-optimal, and measured latency confirms that the resulting reductions in average exit depth translate into practical inference-time savings. Our systematic diagnostic evaluation reframes adaptive depth in looped Transformers as a joint problem of trajectory formation and exit readout, rather than gate learning alone, highlighting a distinction that prior learned-halting work has often left implicit.

Figures

Figures reproduced from arXiv: 2607.20519 by Andrei Cristian Popescu, Haitz S\'aez de Oc\'ariz Borde, Pietro Li\`o.

Figure 1
Figure 1. Figure 1: Adaptive depth is a trajectory–readout problem, not just a gate-learning problem. Left: a looped Transformer’s recurrent trajectory h1, . . . , hT is shaped by the training objective, while a separate readout (a learned gate, or a simple heuristic) decides when to stop. Right: on MANO, post-hoc readouts on a fixed-prior trajectory reach near-ceiling accuracy at lower exit depth than a fixed-depth baseline,… view at source ↗
Figure 2
Figure 2. Figure 2: Post-hoc readout curves for fixed-prior trajectories. We plot test accuracy against average exit depth for [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Trajectory signals for the geometric fixed-prior model with [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Fixed-prior readouts compared with learned linear and MLP gates under uniform and geometric priors. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Forced-exit trajectories and learned exit distributions for the geometric prior with [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Post-hoc gate training on frozen MANO trajectories: we freeze the backbone and LM head, fit new linear [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Large-scale Ouro readout comparisons. Each panel plots benchmark accuracy against average loops. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Full geometric fixed-prior sweep. Each panel shows threshold-swept post-hoc readouts for a different [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Difficulty-stratified trajectory diagnostics for the uniform fixed-prior model. Examples are grouped by the [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Difficulty-stratified trajectory diagnostics for the geometric fixed-prior model with [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Difficulty extrapolation diagnostics for the geometric fixed-prior MANO model with [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Difficulty extrapolation diagnostics for the uniform fixed-prior MANO model. The model is trained on [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Fixed-prior readouts compared with learned linear-gate models across all fixed-prior settings. Each [PITH_FULL_IMAGE:figures/full_fig_p022_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Fixed-prior readouts compared with learned MLP-gate models across all fixed-prior settings. Each panel [PITH_FULL_IMAGE:figures/full_fig_p023_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Forced-exit accuracy by task difficulty across fixed-prior settings. Each panel evaluates the prediction at [PITH_FULL_IMAGE:figures/full_fig_p024_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Learned exit distributions by task difficulty across prior settings. Each panel shows the average learned [PITH_FULL_IMAGE:figures/full_fig_p025_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Parity diagnostic comparing fixed-prior readouts with learned gates. [PITH_FULL_IMAGE:figures/full_fig_p026_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Difficulty-aware trajectory diagnostics on parity. The geometric fixed-prior model is trained on bit strings [PITH_FULL_IMAGE:figures/full_fig_p026_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Full readout comparison for Ouro-1.4B. Each panel plots benchmark accuracy against average loops. [PITH_FULL_IMAGE:figures/full_fig_p027_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Full readout comparison for Ouro-2.6B. Each panel plots benchmark accuracy against average loops. [PITH_FULL_IMAGE:figures/full_fig_p027_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Post-hoc gate training on frozen fixed-prior MANO trajectories. Both linear and MLP post-hoc gates are [PITH_FULL_IMAGE:figures/full_fig_p028_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Post-hoc gate training on frozen MANO trajectories originally trained with a linear learned gate. The [PITH_FULL_IMAGE:figures/full_fig_p029_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Post-hoc gate training on frozen MANO trajectories originally trained with an MLP learned gate. Across [PITH_FULL_IMAGE:figures/full_fig_p029_23.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 29 linked inside Pith

  1. [4]

    Shaojie Bai, J

    URLhttps://arxiv.org/abs/2507.10524. Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models,

    arXiv

  2. [9]

    Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli

    URLhttps://arxiv.org/abs/1807.03819. Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer,

    Pith/arXiv arXiv

  3. [12]

    Angeliki Giannou, Shashank Rajput, Jy yong Sohn, Kangwook Lee, Jason D

    URLhttps://arxiv.org/abs/2502.05171. Angeliki Giannou, Shashank Rajput, Jy yong Sohn, Kangwook Lee, Jason D. Lee, and Dimitris Papailiopou- los. Looped transformers as programmable computers,

    Pith/arXiv arXiv

  4. [13]

    Alex Graves

    URLhttps://arxiv.org/abs/2301.13196. Alex Graves. Adaptive computation time for recurrent neural networks,

    Pith/arXiv arXiv

  5. [15]

    DanHendrycks, CollinBurns, StevenBasart, AndyZou, MantasMazeika, DawnSong, andJacobSteinhardt

    URLhttps://arxiv.org/ abs/2412.06769. DanHendrycks, CollinBurns, StevenBasart, AndyZou, MantasMazeika, DawnSong, andJacobSteinhardt. Measuring massive multitask language understanding,

    Pith/arXiv arXiv

  6. [16]

    Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu

    URLhttps://arxiv.org/abs/2009.03300. Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth,

    Pith/arXiv arXiv 2009

  7. [17]

    Wei Huang, Yizhe Xiong, Xin Ye, Zhijie Deng, Hui Chen, Zijia Lin, and Guiguang Ding

    URLhttps://arxiv.org/abs/2004.04037. Wei Huang, Yizhe Xiong, Xin Ye, Zhijie Deng, Hui Chen, Zijia Lin, and Guiguang Ding. Fast quiet-star: Thinking without thought tokens,

    Pith/arXiv arXiv 2004

  8. [18]

    Ahmadreza Jeddi, Marco Ciccone, and Babak Taati

    URLhttps://arxiv.org/abs/2505.17746. Ahmadreza Jeddi, Marco Ciccone, and Babak Taati. Loopformer: Elastic-depth looped transformers for latent reasoning via shortcut modulation,

    Pith/arXiv arXiv

  9. [19]

    Harsh Kohli, Srinivasan Parthasarathy, Huan Sun, and Yuekun Yao

    URLhttps://arxiv.org/abs/2602.11451. Harsh Kohli, Srinivasan Parthasarathy, Huan Sun, and Yuekun Yao. Loop, think, & generalize: Implicit reasoning in recurrent-depth transformers,

    arXiv

  10. [20]

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut

    URLhttps://arxiv.org/abs/2604.07822. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations,

    Pith/arXiv arXiv

  11. [21]

    org/abs/1909.11942

    URLhttps://arxiv. org/abs/1909.11942. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering,

    Pith/arXiv arXiv 1909

  12. [22]

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam San- toro

    URLhttps://arxiv.org/abs/1809.02789. David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam San- toro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models,

    Pith/arXiv arXiv

  13. [23]

    Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J

    URLhttps://arxiv.org/abs/2404.02258. Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J. Reddi. Reasoning with latent thoughts: On the power of looped transformers,

    Pith/arXiv arXiv

  14. [24]

    Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q

    URLhttps://arxiv.org/abs/2502.17416. Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling,

    Pith/arXiv arXiv

  15. [25]

    Shixiang Song, He Li, Zitong Wang, Boyi Zeng, Feichen Song, Yixuan Wang, Zhiqin John Xu, Ziwei He, and Zhouhan Lin

    URLhttps://arxiv.org/abs/2207.07061. Shixiang Song, He Li, Zitong Wang, Boyi Zeng, Feichen Song, Yixuan Wang, Zhiqin John Xu, Ziwei He, and Zhouhan Lin. Adaponderlm: Gated pondering language models with token-wise adaptive depth,

    Pith/arXiv arXiv

  16. [26]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant

    URLhttps://arxiv.org/abs/2603.01914. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question an- swering challenge targeting commonsense knowledge,

    arXiv

  17. [27]

    Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin

    URLhttps://arxiv.org/abs/1811.00937. Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference,

    Pith/arXiv arXiv

  18. [28]

    Liu Yang, Kangwook Lee, Robert Nowak, and Dimitris Papailiopoulos

    URLhttps://arxiv.org/abs/2004.12993. Liu Yang, Kangwook Lee, Robert Nowak, and Dimitris Papailiopoulos. Looped transformers are better at learning learning algorithms,

    Pith/arXiv arXiv 2004

  19. [29]

    Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D

    URLhttps://arxiv.org/abs/2311.12424. Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. Quiet-star: Language models can teach themselves to think before speaking,

    Pith/arXiv arXiv

  20. [30]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi

    URLhttps://arxiv.org/abs/ 2403.09629. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?,

    Pith/arXiv arXiv

  21. [31]

    Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei

    URLhttps://arxiv.org/abs/1905.07830. Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. Bert loses patience: Fast and robust inference with early exit,

    Pith/arXiv arXiv 1905

  22. [32]

    URLhttps://arxiv.org/abs/2006.04152. 14 Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, He Xing, Lu Li, Jiajun Shi, Kaijing Ma, Shanda Li, Taylor Kergan, Andrew Smith, Xingwei Qu, Mude Hui, Bohong Wu, Qiyang Min, Hongzhi Huang, Xun Zhou, Wei Ye, Jiaheng Liu, Jian Yang, Yunfeng Shi, Chenghua Lin, Enduo Z...

    Pith/arXiv arXiv 2006

  23. [33]

    A Model Background This appendix provides additional background on the architectures studied in this work

    URLhttps://arxiv.org/abs/2510.25741. A Model Background This appendix provides additional background on the architectures studied in this work. We first summarize the looped Transformer formulation used throughout the paper, then review the Ouro architecture and training procedure. A.1 Looped Transformer Architecture A looped Transformer replaces a stack ...

    Pith/arXiv arXiv

  24. [34]

    implements a Looped Language Model by repeatedly applying a shared stack ofL standard decoder-only Transformer layers (multi-head attention with RoPE, SwiGLU feed-forward blocks, sandwich RMSNorm) up toTmax times, rather than stackingLdistinctlayers as in a conventional Trans- former. The released checkpoints are Ouro-1.4B (24 layers) and Ouro-2.6B (48 la...

    2048

  25. [35]

    We balance examples across operation-count bins and use operation count as the difficulty variable in trajectory diagnostics

    Expressions are represented in prefix notation and contain up to 10 operations. We balance examples across operation-count bins and use operation count as the difficulty variable in trajectory diagnostics. Unlike MANO setups that provide the expression length as an input token, our implementation does not include a length token, in order to avoid giving t...

    2000

  26. [2017]

    13 Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian

    URLhttps://arxiv.org/ abs/1603.08983. 13 Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space,

    Pith/arXiv arXiv

  27. [2018]

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser

    URL https://arxiv.org/abs/1803.05457. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal trans- formers,

    Pith/arXiv arXiv

  28. [2019]

    org/abs/1909.01377

    URLhttps://arxiv. org/abs/1909.01377. Andrea Banino, Jan Balaguer, and Charles Blundell. Pondernet: Learning to ponder,

    Pith/arXiv arXiv 1909

  29. [2020]

    URL https://arxiv.org/abs/1910.10073. Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, and Carole-Jean Wu. Layerskip: Enabling early exit inference and self-speculative decoding. InProceedings of the 62nd Annual Meeting of the Asso...

    Pith/arXiv arXiv 1910

  30. [2021]

    Mar Gonzàlez I Català, Haitz Sáez de Ocáriz Borde, George D

    URLhttps: //arxiv.org/abs/2107.05407. Mar Gonzàlez I Català, Haitz Sáez de Ocáriz Borde, George D. Montañez, and Pietro Liò. The stepwise informativeness assumption: Why are entropy dynamics and reasoning correlated in llms?,

    Pith/arXiv arXiv

  31. [2022]

    Sangmin Bae, Jongwoo Ko, Hwanjun Song, and Se-Young Yun

    URLhttps://arxiv.org/abs/2207.04901. Sangmin Bae, Jongwoo Ko, Hwanjun Song, and Se-Young Yun. Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.),Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5910–59...

    Pith/arXiv arXiv 2023

  32. [2023]

    doi: 10.18653/v1/2023.emnlp-main.362

    Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.362. URLhttps://aclanthology.org/2023.emnlp-main.362/. Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Haru- tyunyan, Ziwei Ji, Aaron Courville, and Se-Young Yun. Mixture-of-recursions: Learning dynamic recursive depths for adaptive tok...

    doi:10.18653/v1/2023.emnlp-main.362 2023

  33. [2024]

    URLhttp://dx

    doi: 10.18653/v1/2024.acl-long.681. URLhttp://dx. doi.org/10.18653/v1/2024.acl-long.681. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.ArXiv, abs/2101.03961,

    Pith/arXiv arXiv 2024

  34. [2025]

    Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur

    URLhttps://arxiv.org/abs/2512.17351. Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models,

    Pith/arXiv arXiv

  35. [2026]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord

    URL https://arxiv.org/abs/2604.06192. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge,

    Pith/arXiv arXiv