Pith. sign in

REVIEW 4 major objections 4 minor 20 references

Reusing Transformer blocks changes the depth-stability rule: the residual exponent must rise from 1/4 to 1/2 when shared blocks are revisited.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 04:59 UTC pith:S232JPYP

load-bearing objection A clean, one-line residual-scaling rule for looped Transformers with a plausible but unmeasured worst-case threshold. the 4 major comments →

arxiv 2607.13491 v1 pith:S232JPYP submitted 2026-07-15 cs.LG cs.AI

DeepLoop: Depth Scaling for Looped Transformers

Shuzhen Li , Yifan Zhang , Jiacheng Guo , Quanquan Gu , Mengdi Wang This is my paper
classification cs.LG cs.AI
keywords looped transformersrecurrent depthresidual scalingweight tyingvisit alignmenttraining stabilityDeepNormPost-LN
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Looped Transformers apply a small stack of physical blocks for multiple rounds, so the same parameters accumulate gradients from many visits and are then read back by those same visits. The paper argues that this tied-depth effect is absent from standard residual-scaling analyses, which assume each unrolled layer owns its parameters. Through a first-order perturbation bound with a visit-alignment coefficient κ_R, it shows that stable training at fixed physical depth and growing loop count requires the residual-scaling exponent p to be at least 1/2 in the worst-case aligned regime. DeepLoop is the resulting one-line rule — α=(2N)^(1/2), β=(8N)^(-1/2) — applied to an otherwise unchanged Post-LN DeepNorm architecture. If the paper is right, recurrent depth becomes a practical compute axis: looped Transformers remain trainable without extra parameters and improve validation loss and downstream accuracy once revisits begin.

Core claim

The paper’s central claim is that depth-wise weight sharing changes the residual-scaling condition from the untied DeepNorm bound M(β/α)^2 = O(1) to the tied bound M κ_R (β/α)^2 = O(1), where κ_R measures how coherently visit-wise gradients and output sensitivities align across R rounds. In the conservative aligned regime, κ_R = Θ(R) at fixed physical depth K, which pushes the exponent threshold from p=1/4 to p=1/2. DeepLoop implements this as α=(2N)^(1/2) and β=(8N)^(-1/2) without gates, learned coefficients, or architecture changes. The paper presents this as the minimal exponent for uniform boundedness under aligned revisits, and reports that on GPT-style looped language models the rule i

What carries the argument

The visit-alignment coefficient κ_R is the ratio between the norm of the summed visit-wise sensitivity/update products and the product of their per-visit bounds, normalized so 0≤κ_R≤R. It quantifies whether the R gradient contributions from revisiting the same physical sublayer add coherently (κ_R=Θ(R)) or cancel/decorrelate (κ_R=O(1)). Plugging κ_R into the first-order perturbation bound converts the untied condition into Mκ_R(β/α)^2=O(1), which is what forces the exponent p≥1/2 in the aligned fixed-depth regime.

Load-bearing premise

The p=1/2 threshold rests on the unmeasured assumption that visit-wise gradients and sensitivities of a reused physical block stay coherently aligned, so κ_R grows like R; if revisits actually decorrelate, the usual p=1/4 exponent would suffice and DeepLoop would be needlessly conservative.

What would settle it

Measure κ_R directly from a trained looped Transformer by recording per-visit effective updates and linearized output-sensitivity operators across rounds. If κ_R stays O(1) as R increases while validation loss continues to improve, the paper’s worst-case bound overstates the required exponent and the p=1/2 rule is not the binding constraint.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • DeepLoop keeps a looped Transformer stable as loop count grows at fixed physical depth, making recurrent depth a usable axis for spending more compute without adding stored parameters.
  • At R=1, when no block is revisited, DeepLoop matches the baseline, so adopting the rule does not hurt ordinary single-pass training.
  • At R≥3 in the paper’s GPT-style ablations, DeepLoop lowers validation loss and improves average accuracy on an eight-task language-model suite.
  • Under a one-step gradient approximation, the bound uses the gradient-visible visit count rather than the full forward count, so hierarchical recurrent reasoners inherit the same p=1/2 threshold for the module that realizes effective depth.
  • If visit-wise contributions decorrelate, the same bound recovers the DeepNorm exponent p=1/4, so DeepLoop is a conservative correction rather than a competing architecture.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If κ_R is ever measured directly and found to grow sublinearly with R, the p=1/2 rule is over-conservative and a smaller exponent would train faster; the paper itself points to this measurement as the natural next step.
  • The per-module decomposition suggests a testable extension: in a hierarchical reasoner, only the module whose recurrent depth comes from aligned revisits needs p=1/2, so assigning different exponents to different modules could be safer and less conservative than one global rule.
  • A running estimate of κ_R during training could support an adaptive scheduler that starts near p=1/4 and tightens to p=1/2 only when alignment actually accumulates, trading worst-case safety for faster learning.
  • Extending the single point p-sweep at R=3 to larger loop counts and physical depths would test whether the p=1/2 boundary persists beyond the configuration reported in the paper.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies residual-scaling stability for looped Transformers, in which a small stack of K physical blocks is reused for R rounds (unrolled depth N=KR). It introduces a visit-alignment coefficient κ_R and argues that, in the conservative aligned regime κ_R=Θ(R), the first-order stability condition Mκ_R(β/α)^2=O(1) forces the residual-scaling exponent p to be 1/2 rather than the DeepNorm value 1/4. The proposed DeepLoop rule sets α=(2N)^{1/2}, β=(8N)^{-1/2}. Experiments on GPT-2-style looped language models and on a hierarchical recurrent reasoner (ARC-AGI) report that DeepLoop matches baseline at R=1 and improves validation loss/downstream accuracy at R≥3.

Significance. If the central claim is correct, the paper gives a one-line, parameter-free residual-scaling rule for looped and hierarchical recurrent Transformers, with a clear falsifiable prediction. The derivation is transparent and the paper is honest about its conservative worst-case framing: no constants are fitted, κ_R is defined rather than estimated, and the experiments are controlled single-seed ablations. The main significance is therefore conditional: the theoretical mechanism is plausible and useful, but the empirical and measurement evidence for the decisive κ_R=Θ(R) regime is currently missing, so the paper establishes a sufficient conservative rule rather than establishing that looped depth requires the exponent increase to 1/2.

major comments (4)
  1. [§3.4, Eq. (3.6) and Proposition 3.2] The p≥1/2 threshold is load-bearing and depends entirely on κ_R=Θ(R). κ_R is defined through operator norms of summed visit-sensitivity/update operators but is never measured anywhere in the paper; the concluding paragraph explicitly says future work should measure κ_R or cross-round gradient alignment directly. Without such a measurement, or at least an indirect empirical signature of visit alignment, the paper cannot distinguish its claim from the alternative that κ_R=O(1), in which DeepNorm's p=1/4 already suffices and DeepLoop is over-conservative. Please provide a direct estimate of κ_R on trained looped models, or an indirect test (e.g., comparing p=1/4 vs p=1/2 under identical looped training).
  2. [Appendix C, Figure 6] The p-sweep is the main evidence for the p=1/2 threshold, but the reported empirical phase boundary is near p≈0.45, not 0.50, and p∈[0.45,0.50) trains with lower mean loss when it does not diverge. The text says the boundary 'lines up' with p=1/2, yet a boundary at 0.45 is quantitatively inconsistent with an exact sharp threshold at 1/2 under the stated worst-case aligned model. Moreover, the same sweep is used both to confirm the prediction and to justify choosing p=1/2 as the default, which is circular. Please reconcile the 0.45 boundary, report seed-level behavior for p=1/4, and state how the sweep distinguishes the aligned κ_R=Θ(R) explanation from a generic small-p instability.
  3. [§5.1, Table 1] The main experiments compare DeepLoop against a baseline with α=β=1, not against the DeepNorm parameterization p=1/4. The paper's theoretical claim is that looped depth raises the required exponent from 1/4 to 1/2; Table 1 does not test whether p=1/4 fails. A controlled comparison among p=1/4, p=1/2 (DeepLoop), and perhaps p=0.45 at R≥3 is needed to establish that the exponent increase is necessary, rather than merely that some non-identity residual scaling is beneficial at large R. This is a necessary condition for the paper's abstract claim that stable recurrent depth 'requires' residual scaling rules that account for parameter visits.
  4. [§4.3 and Table 3] The ARC-AGI result is presented as confirming the p=1/2 prediction for hierarchical recurrent reasoners, but the comparison is only DeepLoop(p=1/2) versus vanilla HRM. The prediction specifically requires the inner-module alignment coefficient κ_g^(L)=Θ(C_L); this is asserted on the grounds that weight tying makes revisits implement the same operation, but no κ_g measurement or decorrelation diagnostic is given. As with the language-model experiments, the result supports the sufficiency of p=1/2, not the claimed necessity. Please add a p=1/4 control and, if possible, a gradient-alignment statistic, or soften the claim accordingly.
minor comments (4)
  1. [Appendix C, last paragraph] The text refers to a 'Limitations paragraph (§6)', but Section 6 is Related Work and no Limitations section exists. Please correct the cross-reference.
  2. [§5.1] All headline results are single-seed (downstream and validation). The paper acknowledges this, but the reported Δ values of 0.01–0.03 nats should be interpreted with that caveat; some Table 2 cells (e.g., DeepLoop R=5 1-shot is 0.23 points below baseline) are negative, so the 'improves every column' statement in §5.3 should not be generalized from one seed.
  3. [Acknowledgements] Typo: 'constructful' should be 'constructive'.
  4. [§3.4, Eq. (3.6)] The definition of κ_R is notationally dense; it would help to state explicitly that the operator norms and constants C_U, C_G are worst-case over inputs and visits, and to give one concrete example of a fully aligned versus fully decorrelated visit pattern in a small K=R case.

Circularity Check

0 steps flagged

No significant circularity: the p=1/2 rule is a conditional consequence of the stated κ_R=Θ(R) worst-case assumption, and no predicted quantity is fitted from data.

full rationale

The derivation chain is self-contained and conditional, not circular. The central bound is Eq. (3.7), which follows from the definition of κ_R in Eq. (3.6): κ_R is defined as the normalized worst-case product of visit-sum sensitivity and update operators, so the inequality ∥ΔF∥ ≤ C'' M κ_R (β/α)^2 is an accounting identity for that coefficient. This makes the bound true by construction, but it is not circular in the harmful sense: the content of Proposition 3.2 is the algebraic threshold p ≥ (1+γ)/4 that follows from M=2N, N=KR, β/α=(cd)^{-p}N^{-2p}, and the assumed scaling κ_R=Θ(R^γ). The p=1/2 choice is the γ=1 specialization of that conditional statement, not an estimate recovered from the experiments. No free constant is fitted: α=(2N)^{1/2}, β=(8N)^{-1/2} are set from the worst-case bound and directly give M R (β/α)^2 = 1/(8K) = O(1). The assumption κ_R=Θ(R) is unmeasured, and the paper explicitly concedes this in the future-work paragraph ('Future work should measure κ_R or cross-round gradient alignment directly'); that is an acknowledged limitation and underdetermination, not a circular fit. The Appendix C p-sweep is used both to support the p=1/2 prediction and to justify selecting p=1/2 as the default, and the empirical boundary is near p≈0.45 rather than exactly 0.5, with no p=1/4 baseline. This entangles validation with design choice and weakens the empirical case, but it does not reduce any predicted equation to the sweep's fitted values: no parameter of the theoretical bound is estimated from the sweep. The only self-citation (Zhang et al. 2026) is used for the downstream evaluation suite and is not load-bearing for the scaling derivation; the DeepNorm result is an external prior work (Wang et al. 2024) with no author overlap. There is no imported uniqueness theorem, no ansatz smuggled through self-citation, and no renaming of a known result as a new unification. Overall, the paper's central claim is an honest conditional derivation with a clearly stated, unverified alignment assumption, so no circular step is established.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The central derivation introduces one bookkeeping coefficient, κ_R, but no new physical or architectural entity. The free parameter ledger is minimal: the method's exponent p is a design constant derived from a conservative worst-case bound, not a number fitted to the main experiments.

free parameters (1)
  • residual-scaling exponent p = 1/2 (default; confirmed by Appendix C sweep)
    The exponent is not estimated from data as a fitted constant; it is the theoretical threshold under the aligned worst-case regime, with p=1/2 also chosen as the smallest reliably stable value in the Appendix C sweep. It is a hand-chosen design constant of DeepLoop.
axioms (4)
  • domain assumption DeepNorm-scale local sensitivity: for every unrolled visit, the norm of the linearized output map from a residual-branch parameter perturbation and the norm of the visit-wise effective update are both O(β/α).
    This is Assumption 3.1, inherited from DeepNet. It is not proven for looped Transformers and controls every subsequent bound.
  • domain assumption Worst-case visit alignment κ_R=Θ(R) for reused physical sublayers at fixed physical depth.
    Section 3.4 and Corollary A.4 invoke this as the conservative regime; it is never measured. It is the load-bearing premise that moves the threshold from p=1/4 to p=1/2.
  • domain assumption First-order double-sum expansion Eq. (3.5) treats visit-wise sensitivity operators U_{r,j} and updates G_{r,j} as independent and discards O(η²).
    The derivation of Proposition A.3 depends on this schematic expansion; the paper states that tensor contractions and optimizer-dependent constants are absorbed.
  • domain assumption The residual branch z is small relative to αx at every sublayer, so RMS(z)/α ≤ c < 1 in Lemma A.1.
    The 1/α exposure factor that motivates the β/α ratio comes from this Taylor expansion of RMSNorm.

pith-pipeline@v1.3.0-alltime-deepseek · 17123 in / 14309 out tokens · 143524 ms · 2026-08-02T04:59:54.799889+00:00 · methodology

0 comments
read the original abstract

Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds, increasing unrolled depth without increasing stored parameters. This reuse changes the residual-scaling problem: in an untied Transformer, each residual branch receives and applies its own parameter update, whereas in a looped Transformer one shared update aggregates gradients from repeated visits and is read back by those same visits in the next linearized forward pass. We formalize this tied-depth effect through a first-order perturbation bound controlled by a visit-alignment coefficient $\kappa_R$. The bound recovers the DeepNorm exponent when visits decorrelate, but in the conservative aligned regime it requires the exponent to increase from $1/4$ to $1/2$ as loop count grows at fixed physical depth. The resulting method, \textbf{DeepLoop}, keeps the Post-LN DeepNorm architecture and sets $\alpha=(2N)^{1/2}$ and $\beta=(8N)^{-1/2}$ for unrolled depth $N$. On GPT-style looped language models at GPT-2 small and GPT-2 medium scale, DeepLoop is neutral when no physical block is revisited and improves validation loss and downstream accuracy once recurrent depth is activated. These results show that stable recurrent depth requires residual scaling rules that account for parameter visits, not only nominal layer count.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 15 linked inside Pith

  1. [4]

    We test this prediction directly with a single-axis sweep over the exponentp at fixedR=3 on the GPT-2 small backbone

    22 C Empiricalp-sweep at fixed loop count Proposition 3.2 predicts a sharpp=1/2 training-stability threshold under the worst-case aligned regime. We test this prediction directly with a single-axis sweep over the exponentp at fixedR=3 on the GPT-2 small backbone. All other ingredients, tied embeddings, input-embedding RMSNorm, FineWeb-Edu 50BT data pipeli...

    2000

  2. [6]

    Scaling up test-time compute with latent reasoning: A recurrent depth approach.arXiv preprint arXiv:2502.05171,

    Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach.arXiv preprint arXiv:2502.05171,

    Pith/arXiv arXiv

  3. [7]

    Think before you speak: Training language models with pause tokens.arXiv preprint arXiv:2310.02226,

    Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens.arXiv preprint arXiv:2310.02226,

    Pith/arXiv arXiv

  4. [8]

    Adaptive computation time for recurrent neural networks.arXiv preprint arXiv:1603.08983,

    Alex Graves. Adaptive computation time for recurrent neural networks.arXiv preprint arXiv:1603.08983,

    Pith/arXiv arXiv

  5. [10]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W Rae, Oriol Vinyals, and Laurent ...

    Pith/arXiv arXiv

  6. [11]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Pith/arXiv arXiv 2001

  7. [12]

    Albert: A lite bert for self-supervised learning of language representations.arXiv preprint arXiv:1909.11942,

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations.arXiv preprint arXiv:1909.11942,

    Pith/arXiv arXiv 1909

  8. [14]

    Subformer: Exploring weight sharing for parameter efficiency in generative transformers

    16 Machel Reid, Edison Marrese-Taylor, and Yutaka Matsuo. Subformer: Exploring weight sharing for parameter efficiency in generative transformers. InFindings of the Association for Computational Linguistics: EMNLP 2021, pages 4081–4090,

    2021

  9. [15]

    Reasoning with latent thoughts: On the power of looped transformers.arXiv preprint arXiv:2502.17416,

    Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J Reddi. Reasoning with latent thoughts: On the power of looped transformers.arXiv preprint arXiv:2502.17416,

    Pith/arXiv arXiv

  10. [17]

    Tensor programs vi: Feature learning in infinite-depth neural networks.arXiv preprint arXiv:2310.02244, 2023a

    Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor programs vi: Feature learning in infinite-depth neural networks.arXiv preprint arXiv:2310.02244, 2023a. Liu Yang, Kangwook Lee, Robert Nowak, and Dimitris Papailiopoulos. Looped transformers are better at learning learning algorithms.arXiv preprint arXiv:2311.12424, 2023b. Hongyi Zhang, Yann N Dau...

    Pith/arXiv arXiv 1901

  11. [18]

    Deep delta learning.arXiv preprint arXiv:2601.00417,

    Yifan Zhang, Yifeng Liu, Mengdi Wang, and Quanquan Gu. Deep delta learning.arXiv preprint arXiv:2601.00417,

    Pith/arXiv arXiv

  12. [1024]

    24); training uses the same FineWeb-Edu 50BT data and schedule but on4×H200 141GB GPUs

    and 12 layers (vs. 24); training uses the same FineWeb-Edu 50BT data and schedule but on4×H200 141GB GPUs. The same eight-task lm-evaluation-harness protocol applies. Table 4 reports per-task accuracy and Avg; Figures 4 and 5 visualize the 1-shot Avg and the per-task 1-shot accuracy, respectively. Table 4Downstream accuracy (%) for the GPT-2 small backbon...

    arXiv 2048

  13. [2016]

    Training large language models to reason in a continuous latent space.arXiv preprint arXiv:2412.06769,

    Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space.arXiv preprint arXiv:2412.06769,

    Pith/arXiv arXiv

  14. [2017]

    Hierarchical reasoning model.arXiv preprint arXiv:2506.21734,

    Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori. Hierarchical reasoning model.arXiv preprint arXiv:2506.21734,

    Pith/arXiv arXiv

  15. [2019]

    Understanding the difficulty of training transformers

    Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5747–5763,

    2020

  16. [2020]

    Universal transformers.arXiv preprint arXiv:1807.03819,

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers.arXiv preprint arXiv:1807.03819,

    Pith/arXiv arXiv

  17. [2021]

    Pondernet: Learning to ponder.arXiv preprint arXiv:2107.05407,

    Andrea Banino, Jan Balaguer, and Charles Blundell. Pondernet: Learning to ponder.arXiv preprint arXiv:2107.05407,

    Pith/arXiv arXiv

  18. [2022]

    Depthwise hyperpa- rameter transfer in residual networks: Dynamics and scaling limit.arXiv preprint arXiv:2309.16620,

    Blake Bordelon, Lorenzo Noci, Mufan Bill Li, Boris Hanin, and Cengiz Pehlevan. Depthwise hyperpa- rameter transfer in residual networks: Dynamics and scaling limit.arXiv preprint arXiv:2309.16620,

    Pith/arXiv arXiv

  19. [2023]

    On the measure of intelligence.arXiv preprint arXiv:1911.01547,

    François Chollet. On the measure of intelligence.arXiv preprint arXiv:1911.01547,

    Pith/arXiv arXiv 1911

  20. [2024]

    15 Khashayar Gatmiry, Nikunj Saunshi, Sashank J Reddi, Stefanie Jegelka, and Sanjiv Kumar

    URLhttps://zenodo.org/records/12608602. 15 Khashayar Gatmiry, Nikunj Saunshi, Sashank J Reddi, Stefanie Jegelka, and Sanjiv Kumar. Can looped transformers learn to implement multi-step gradient descent for in-context learning?arXiv preprint arXiv:2410.08292,

    arXiv