Pith. sign in

REVIEW 3 major objections 4 minor 23 references

A depth-recurrent transformer's latent state settles into a per-token fixed point, and a training-free halting rule cuts average depth by 38% at unchanged quality, beating a learned router.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 02:06 UTC pith:Z7V6WVBZ

load-bearing objection The per-token convergence measurement is new and the training-free exit beats a learned baseline, but the 38% compute saving is a FLOP proxy from a teacher-forced harness, not a realized speedup. the 3 major comments →

arxiv 2607.14427 v1 pith:Z7V6WVBZ submitted 2026-07-15 cs.AI

Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers

Joe Logan This is my paper
classification cs.AI
keywords depth-recurrent transformerper-token fixed pointadaptive computation timeearly exitconvergence exitdepth allocationrandomized-depth trainingelastic depth
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper shows that a depth-recurrent transformer—one that applies the same weight-tied block a variable number of times—does not need a learned halting policy to allocate compute per token. Measuring the model's own recurrent state, it finds that each token converges to a fixed point at its own rate: the median token settles by loop six, while a ten-percent tail keeps updating through the training-mean depth of eight. A zero-parameter rule that freezes a token as soon as its successive outputs stop changing reaches the quality of uniform depth-8 using only 4.94 average loops, a 38% reduction in depth, whereas a linear router trained on the same model's convergence labels needs nearly full depth and saves nothing. The paper also proposes that a looped model's useful depth is set by the training-time loop distribution, giving practitioners a direct lever on the quality-compute tradeoff.

Core claim

On a 135M-class model trained with randomized recursion counts, the recurrent state converges to a per-token fixed point: mean successive-output KL divergence falls from 3.9e-1 at loop 2 to 8.5e-6 at loop 16, and the fraction of tokens still updating drops from 99.8% to 10% at loop 8, with the median converged by loop 6. Convergence depth is ordered by token type—whitespace shallowest, content words deepest—so uniform depth either wastes compute on the early majority or under-serves the late tail. A training-free convergence exit that freezes each token once its output stabilizes attains uniform depth-8 quality at 4.94 average loops (38% lower average depth) and matches uniform depth across

What carries the argument

The central mechanism is the weight-tied recurrent core with random-depth training, noise state initialization, and per-loop input re-injection, which together push the model toward an input-conditioned fixed point rather than an iteration-count-memorizing trajectory. The convergence exit policy (A1) is the load-bearing object: after each loop it computes the per-token KL divergence between successive output distributions and freezes any token below a threshold, tracing an (average-depth, loss) curve as the threshold is swept. A linear router (A2) trained on convergence labels from the same model serves as the comparison that reads a single state snapshot instead of successive outputs, and l

Load-bearing premise

The 38% depth saving is booked in a teacher-forced masking harness where every token is still looped to the maximum and frozen tokens are simply masked; if per-token fixed points are not stable under autoregressive decoding with per-recursion key-value caching, the saving will not become real throughput.

What would settle it

Run the trained model autoregressively with genuine per-token halting and per-recursion caching, then measure end-to-end wall-clock time: if speedup over uniform depth-8 is not in the 1.48–1.51x bracket (or if halted tokens' logits drift once future tokens attend to updated contexts), the allocation claim fails. Independently, train a mean-16 randomized-depth model at the same scale: if its loss does not saturate near depth 16, the training-mean saturation hypothesis is refuted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Per-token adaptive depth is achievable on a frozen looped backbone without any learned halting unit, ponder loss, or joint-training calibration.
  • Because the allocation policy is post-hoc and parameter-free, it can be tuned to any compute budget at serving time by adjusting one threshold, without retraining or modifying the weights.
  • The same allocation rules apply to any depth-robust looped model, so improvements in the backbone (better training objectives, larger scale) and in the allocation policy compose additively.
  • If the training-mean saturation hypothesis holds, the quality-compute spread of a deployed model is chosen by the practitioner through the training loop distribution, and deepening a model's useful range means raising the mean recursion count during pretraining.
  • The recursive-core design keeps the FLOP proxy and wall-clock savings close because exited tokens free whole forward passes of a shared-weight core, making the efficiency gain more realizable than layer-skip early exit.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The failure of the trained router suggests a general lesson for adaptive-computation design: predicting 'will this token change?' from one state snapshot is strictly harder than observing two successive snapshots, so learned halting should use recurrence history as input rather than a single hidden state.
  • If the training-mean saturation rule generalizes, it predicts a testable scaling law: a mean-16 looped model should saturate at depth 16, and per-token exit curves on reasoning-heavy tasks should spread wider than on web text, where the paper's average-loss results already show a 10% late-converging tail.
  • A natural extension is to couple the convergence exit with depth-wise batching (as the paper notes), where tokens at different loop counts are packed by recursion depth; the 1.5x best-case wall-clock bracket suggests the practical payoff is real but bounded, and serving systems that reclaim the slack would decide the matter.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies a depth-recurrent transformer trained with randomized recursion counts. It measures per-token convergence of the recurrent state using successive-output KL divergence and state-change magnitude, finding that convergence depth varies substantially across tokens. It then proposes a training-free convergence exit (A1) that halts each token once its output distribution stabilizes, and compares this to a learned linear router (A2) and uniform depth (A0). In a teacher-forced masking harness, A1 attains uniform depth-8 quality at 4.94 average loops (a 38% reduction in average depth) while A2 yields no reduction. The paper also reproduces depth elasticity, reports token-type ordering of exit depths, and offers a hypothesis that saturation depth tracks the training-time loop distribution. The study is conducted at 135M-class scale on one seed, with code and checkpoints released.

Significance. The paper provides a clean, reproducible measurement of per-token convergence in a depth-recurrent transformer, and the simple convergence-exit policy is an interesting algorithmic result: it suggests that per-token adaptive depth can be read off a frozen backbone without a learned halting unit, at least in the teacher-forced setting. The paper is transparent about its limitations, including the fact that the 38% saving is an average-depth/FLOP proxy and not a realized speedup. If the allocation result survives statistical scrutiny and transfers to autoregressive serving, it would be a meaningful step toward instance-adaptive compute for looped models. The main weaknesses are the teacher-forced evaluation harness, the unfair comparison between A1 and A2, and the lack of uncertainty quantification.

major comments (3)
  1. [5.3 and 6] The headline reduction (4.94 average loops vs. 8, 38%) is computed in a teacher-forced masking harness where every token is still looped to rmax and frozen tokens are masked; the paper acknowledges that this skips no FLOPs and that realized throughput does not beat uniform depth. The key-value cache story for autoregressive decoding is explicitly not stress-tested (Section 6). Because the contribution is framed as 'directly exploitable,' this is a load-bearing limitation: the measured saving is an algorithmic quantity whose translation to wall-clock depends on unimplemented depth-wise batching and on convergence behavior persisting under model-generated context. I recommend either providing a proof-of-concept in a realistic decode setting (even batch-1) or explicitly stating in the abstract and conclusion that the reduction is a FLOP-proxy property contingent on future systems work.
  2. [3.3 and 5.3] The A1 vs. A2 comparison is not apples-to-apples. A1 halts when KL(p_i || p_{i-1}) is below a threshold, while A2 is trained to predict whether the argmax prediction changes after loop i. These are different convergence criteria. The strong negative result for A2 may be an artifact of the label choice rather than evidence that learned halting policies cannot predict the same signal. To support the claim that 'reading it outperforms learning to predict it,' the authors should train A2 on the same KL-based convergence labels used by A1 (or on the same epsilon objective) and report the result, or restrict the claim to the specific probe architecture and label used.
  3. [Table 4 and 5.3] The differences between A1 and A0 at matched average depth are 0.002--0.007 nats (Table 4). The paper reports no standard errors or confidence intervals, yet claims 'A1 never higher' across depths 4--7 and 'matches uniform depth.' With 128 evaluation windows, these differences may be within noise. The authors should report uncertainty (e.g., bootstrap intervals or per-window standard errors) for the allocation table, particularly for the central claim that A1 reaches the depth-8 quality bar at 4.94 average loops.
minor comments (4)
  1. [Table 4 caption] The column 'avg depth' for A1/A2 is the average number of active loops, not wall-clock depth; the caption should state this explicitly to avoid misinterpretation.
  2. [5.4] The mean exit depths for token classes (space 3.58, numeral 3.81, punctuation 4.16, word 4.25) are given without a table or distribution; a small table or figure would improve clarity.
  3. [5.2] The phrase 'the state reaches a fixed point' could be softened to 'approximate fixed point' since 0.1% of tokens still move at loop 16 (Table 3).
  4. [References] Reference [5] is dated 2026; please verify the citation is correct and not a placeholder.

Circularity Check

1 steps flagged

No load-bearing circularity; central results are empirical; minor definitional overlap between the KL convergence diagnostic and the A1 halt signal.

specific steps
  1. self definitional [Section 3.3 (A1) and Section 5.2 / Table 3]
    "compute KL(pi ∥p i−1)between successive output distributions per token; once it falls below a thresholdϵ, freeze that token ... the fraction of tokens still moving (KL above10−3)"

    The convergence diagnostic and the A1 exit rule threshold the same successive-output KL statistic. Thus 'the median token has converged by loop six' and 'A1 halts the converged majority early' are partly restatements of the operational definition, not fully independent discoveries. The claim is not reduced to the definition, because state-change decay and the matched-depth loss comparison are separate measurements, and the 4.94-loop figure is a measured point on a swept threshold curve rather than a quantity forced by the definition.

full rationale

The paper's derivation chain is an empirical measurement chain, not a formal derivation. Depth robustness is established by a local loss-versus-loops sweep (Table 2) and by a fixed-depth control, not by citation alone; the Huginn/LoopFormer citations provide background and one consistency check but are not load-bearing. The fixed-point claim is supported by three independent diagnostics (successive KL, state-change norm, and flat loss beyond the training mean). Although A1 uses the same successive-KL statistic as the convergence diagnostic, the headline 4.94-average-loop result is a measured curve point under a swept threshold, not a quantity forced by the definition; matched-depth loss is an independent quality check. The FLOP-proxy and teacher-forced-harness caveats are stated transparently and are scoping limitations, not circularity. No self-citation chain, uniqueness import, or ansatz-via-citation is present. Overall, there is no significant circularity; the only definitional overlap is minor and partially mitigated by independent checks.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical or architectural entities; the 'fixed point' is an observed property of the existing model, not a new object. The central empirical claims rely on several hand-chosen training hyperparameters (recursion distribution, noise scale) and on thresholds that define convergence and the A1 exit rule. The most load-bearing domain assumptions are the transferability of teacher-forced results to autoregressive decode and the sufficiency of KL stability as a convergence signal.

free parameters (7)
  • mean recursion count r̄ = 4 (S0), 8 (S1)
    Chosen by hand; sets the training-time loop distribution and, per the paper's open hypothesis, the saturation depth (Section 3.2, Section 6).
  • max recursion count rmax = 16 (S0), 32 (S1)
    Cap on sampled recursion count; chosen by hand as training configuration.
  • LogNormal shape parameter = 0.5
    Controls spread of sampled depths; chosen without a stated criterion.
  • state noise scale = sqrt(2/5)
    Initial recurrent state noise; chosen to push the core toward an input-conditioned fixed point (Section 3.1).
  • convergence threshold for 'moving' tokens = KL > 1e-3
    Hand-set threshold; determines the reported 10% tail of tokens still moving at loop 8 (Table 3).
  • A1 exit threshold epsilon = not stated; swept
    The 4.94-average-loop operating point is selected by sweeping this threshold on the evaluation set (Section 3.3, Section 5.3).
  • truncated-BPTT horizon k = 4 (S0), 8 (S1)
    Training detail with no direct dependence on the central allocation claim, but part of the released configuration.
axioms (5)
  • domain assumption Randomized-depth pretraining produces a depth-robust backbone (established by Geiping et al. and reproduced here).
    Used as the fixed setting for all measurement and allocation results; not derived in this paper (Section 1, Section 5.1).
  • domain assumption The looped core with noise initialization and input re-injection behaves as a contraction toward an input-conditioned fixed point.
    Invoked to explain why extrapolated depths are safe (Section 3.1, Section 5.2); the paper measures convergence but does not prove contraction.
  • domain assumption Teacher-forced evaluation is an adequate proxy for autoregressive allocation behavior.
    Acknowledged as untested in Section 6; the entire allocation result is measured under a teacher-forced masking harness.
  • ad hoc to paper KL stability of successive outputs is the right criterion for 'convergence' and the right signal for early stopping.
    This defines the central object of the paper; independent state-change and flat-loss checks support it, but the specific threshold is arbitrary.
  • domain assumption Validation loss on FineWeb-Edu is the quality metric for allocation; downstream task quality is not evaluated.
    The paper explicitly leaves downstream evaluation at each depth to future work (Section 6).

pith-pipeline@v1.3.0-alltime-deepseek · 3753 in / 3876 out tokens · 148292 ms · 2026-08-02T02:06:06.342002+00:00 · methodology

0 comments
read the original abstract

A depth-recurrent transformer applies a weight-tied core a variable number of times, and prior work has shown that training with a randomized recursion count yields one checkpoint usable across a range of inference depths. We ask what such a model actually computes per token, and measure it directly. On a 135M-class model trained on FineWeb-Edu, the recurrent state converges to a per-token fixed point: mean successive-output KL divergence falls from 3.9e-1 at the second loop to 8.5e-6 by the sixteenth, and per-token state change decays in step. Crucially, this convergence is not uniform across tokens. The median token converges by loop six, while approximately 10 percent of tokens continue to update at the training-mean depth of eight, and mean convergence depth is ordered by token type (whitespace shallowest, content words deepest). This per-token variation is the central object of the paper. We show it is directly readable and that reading it outperforms learning to predict it: a training-free rule that halts each token once its output stabilizes attains uniform depth-8 quality at 4.94 average loops (a 38 percent reduction in average depth) and matches uniform depth across the average-depth range, whereas a linear router trained on convergence labels harvested from the same model requires nearly full depth and yields no reduction. The elasticity that makes this possible reproduces here as background (validation loss decreases monotonically from 3.80 at one loop to 3.20 at eight and remains stable to 32 loops). We report average depth as a FLOP proxy with a three-point wall-clock bracket rather than a realized speedup, make no FLOP-matched parity claim, and note that the allocation results are established at a single scale and seed. The complete study runs on a single RTX 4090 in approximately 100 GPU-hours.

Figures

Figures reproduced from arXiv: 2607.14427 by Joe Logan.

Figure 1
Figure 1. Figure 1: Validation loss as a function of inference loop count. Left: the S1 model (135M-class [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Per-loop convergence at S1. Left: mean successive-output KL decays log-linearly to [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Loss against average loops per token for the three policies (S1 final). The dotted line is [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Mean exit depth by token class at both scales. Structural tokens (whitespace) exit [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 19 linked inside Pith

  1. [1]

    GQA: Training generalized multi-query transformer models from multi-head checkpoints.arXiv preprint arXiv:2305.13245, 2023

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints.arXiv preprint arXiv:2305.13245, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    SmolLM2: When smol goes big – data-centric training of a small language model.arXiv preprint arXiv:2502.02737, 2025

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, et al. SmolLM2: When smol goes big – data-centric training of a small language model.arXiv preprint arXiv:2502.02737, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation.arXiv preprint arXiv:2507.10524, 2025

    Sangmin Bae, Adam Fisch, Hrayr Harutyunyan, et al. Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation.arXiv preprint arXiv:2507.10524, 2025

    arXiv 2025

  4. [4]

    PonderNet: Learning to ponder.arXiv preprint arXiv:2107.05407, 2021

    Andrea Banino, Jan Balaguer, and Charles Blundell. PonderNet: Learning to ponder.arXiv preprint arXiv:2107.05407, 2021

    Pith/arXiv arXiv 2021

  5. [5]

    Universal transformers need memory: Depth-state trade-offs in recurrent language models.arXiv preprint arXiv:2604.21999, 2026

    Róbert Csordás et al. Universal transformers need memory: Depth-state trade-offs in recurrent language models.arXiv preprint arXiv:2604.21999, 2026

    Pith/arXiv arXiv 2026

  6. [6]

    Uni- versal transformers

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Uni- versal transformers. InInternational Conference on Learning Representations (ICLR), 2019. arXiv:1807.03819

    Pith/arXiv arXiv 2019

  7. [7]

    SkipDecode: Autoregressive skip decoding with batching and caching for efficient llm inference.arXiv preprint arXiv:2307.02628, 2023

    Luciano Del Corro, Allie Del Giorno, Sahaj Agarwal, Bin Yu, Ahmed Awadallah, and Subhabrata Mukherjee. SkipDecode: Autoregressive skip decoding with batching and caching for efficient llm inference.arXiv preprint arXiv:2307.02628, 2023. 11

    Pith/arXiv arXiv 2023

  8. [8]

    TinyStories: How small can language models be and still speak coherent english?arXiv preprint arXiv:2305.07759, 2023

    Ronen Eldan and Yuanzhi Li. TinyStories: How small can language models be and still speak coherent english?arXiv preprint arXiv:2305.07759, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    LayerSkip: Enabling early exit inference and self-speculative decoding

    Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, et al. LayerSkip: Enabling early exit inference and self-speculative decoding. InAnnual Meeting of the Association for Computational Linguistics (ACL), 2024. arXiv:2404.16710

    Pith/arXiv arXiv 2024

  10. [10]

    Scaling up test-time compute with latent reasoning: A recurrent depth approach.arXiv preprint arXiv:2502.05171, 2025

    Jonas Geiping, Sean McLeish, Neel Jain, et al. Scaling up test-time compute with latent reasoning: A recurrent depth approach.arXiv preprint arXiv:2502.05171, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Adaptive computation time for recurrent neural networks.arXiv preprint arXiv:1603.08983, 2016

    Alex Graves. Adaptive computation time for recurrent neural networks.arXiv preprint arXiv:1603.08983, 2016

    Pith/arXiv arXiv 2016

  12. [12]

    LoopFormer: Elastic-depth looped transformers with shortcut consistency

    Ahmad Jeddi et al. LoopFormer: Elastic-depth looped transformers with shortcut consistency. arXiv preprint arXiv:2602.11451, 2026

    arXiv 2026

  13. [13]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations (ICLR), 2019. arXiv:1711.05101

    Pith/arXiv arXiv 2019

  14. [14]

    The FineWeb datasets: Decanting the web for the finest text data at scale.arXiv preprint arXiv:2406.17557, 2024

    Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, et al. The FineWeb datasets: Decanting the web for the finest text data at scale.arXiv preprint arXiv:2406.17557, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Rao et al

    D. Rao et al. DREX: Dynamic rebatching for early-exit inference serving.arXiv preprint arXiv:2512.15705, 2025

    arXiv 2025

  16. [16]

    Mixture-of-depths: Dynamically allocating compute in transformer-based language models.arXiv preprint arXiv:2404.02258, 2024

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models.arXiv preprint arXiv:2404.02258, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Confident adaptive language modeling

    Tal Schuster, Adam Fisch, Jai Gupta, et al. Confident adaptive language modeling. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. arXiv:2207.07061

    Pith/arXiv arXiv 2022

  18. [18]

    GLU variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Noam Shazeer. GLU variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Pith/arXiv arXiv 2002

  19. [19]

    RoFormer: Enhanced transformer with rotary position embedding.Neurocomputing, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding.Neurocomputing, 2024. arXiv:2104.09864

    Pith/arXiv arXiv 2024

  20. [20]

    LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  21. [21]

    Whitfield et al

    S. Whitfield et al. TIDE: Token-informed depth execution for efficient inference.arXiv preprint arXiv:2603.21365, 2026

    arXiv 2026

  22. [22]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. InAdvances in Neural Information Processing Systems (NeurIPS), 2019. arXiv:1910.07467

    Pith/arXiv arXiv 2019

  23. [23]

    Zhu et al

    Q. Zhu et al. Scaling latent reasoning via looped language models.arXiv preprint arXiv:2510.25741, 2025. A Configuration Full configuration JSONs ship with the code release. 12 Table 1: Architecture and training configuration. S0 (toy) S1 (main) Data TinyStories FineWeb-Edu Tokens 0.4B 12.9B dmodel 384 576 Heads (KV heads) 6 (2) 9 (3) FFN dim 1024 1536 Pr...

    Pith/arXiv arXiv 2025