Pith. sign in

REVIEW 4 major objections 7 minor 13 references

A trained weight-tied looped transformer runs a linear computation frontier whose speed is set by the training budget, v ≈ n_train/T_train—not a fixed-point or log-depth scan.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:08 UTC pith:YL3LKNA6

load-bearing objection Strong instrument paper whose headline 'budget law' is better read as a necessary-condition regularity than a selection principle. the 4 major comments →

arxiv 2607.20594 v1 pith:YL3LKNA6 submitted 2026-07-22 cs.LG cs.AIstat.ML

When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers

Tong Zhang , Junhao Hu , Yun Peng , Tao Xie This is my paper
classification cs.LG cs.AIstat.ML
keywords looped transformersweight tyingcomputation frontierbudget lawconvergence-time scalinglength generalizationtest-time computegroup word problems
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks when a looped transformer—one block applied many times—actually implements a step-by-step algorithm. On controlled group word problems, free training installs a linear computation frontier: each loop advances the solved prefix by v positions, with v set by the training contract, v ≈ n_train/T_train (exponent 0.98, R²=0.99). The claim matters because it turns looped transformers into serial machines with predictable test-time compute, yielding a principled halting rule T* = ceil(n/v). It also shows the algorithm lives in the head of the trajectory, while standard tail instruments provably saturate at converged fixed points and miss the entire mechanism.

Core claim

The central discovery is the budget law. Across dozens of weight-tied looped transformers trained on prefix products over Z60, S4, A5, and S5, every model that learns the task implements a computation frontier: the contiguous solved prefix advances by a constant v positions per loop, uniform across input lengths, and v is priced by the training contract. Empirically v ≈ n_tr/T_tr with exponent 0.98 ± 0.04 and R²=0.99 over 23 models spanning an 8× demand range, and exactly v=1.0 under the T=n contract. The paper interprets SGD as selecting the minimum speed the contract demands—meeting it, never exceeding it. A paired discovery is that the mechanism lives in the head of the trajectory: the co

What carries the argument

The paper's central object is the linear computation frontier: the boundary of the contiguous solved prefix in a (position, loop) grid, advancing by v positions per loop with v constant per model. The key instrument is convergence-time scaling τ(n,i), defined as the first loop k at which the decoded output at position i reaches its final value and stays there; the scaling class of τ_max(n)—flat, log n, or linear—identifies the mechanism as shortcut, parallel scan, or serial step-indexed computation. The argument rests on a tail-blindness proposition: once the loop map reaches a fixed point, skipping loops, cross-step Jacobian similarity, and feature-dictionary attributions all become constan

Load-bearing premise

The load-bearing premise is that the budget law reflects SGD actively selecting the minimum demanded speed, which requires that faster-but-equally-trainable solutions exist at the same contract—the paper does not constructively demonstrate this.

What would settle it

Train a large population at a consistent contract such as T=n/4 on S4, searching over seeds, learning rates, and small curriculum variations. If any run that reaches full in-distribution accuracy and OOD generalization shows a measured frontier speed v significantly above ~4 (the demand), the 'never exceeded' part of the law is falsified. Conversely, if a constructive search exhibits a faster-but-trainable solution that SGD never finds, the selection principle is confirmed; if no such solution exists, the law reduces to a tautological lower bound.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Test-time loops buy frontier completion, not convergence: the halting rule T* = ceil(n/v-hat) recovers near-oracle accuracy and avoids the overthinking collapse of overlong budgets, giving looped models a principled compute setting.
  • Standard interpretability instruments—skip tolerance, cross-step similarities, shared-dictionary attribution—are uninformative for looped models whenever trajectories converge with slack; head instruments like τ are required to see the algorithm.
  • Architecture prior, not expressivity, selects the algorithm: weight tying flips SGD from parallel scans (standard-depth) to the serial frontier, even with positional addressing for a log-depth scan, and tied models learn A5 where untied ones fail.
  • Learning walls on group word problems are optimization pathologies, not complexity results: NC1-complete A5 trains fully, while S5's 120×120 operator deadlocks joint learning—until an operator-first curriculum cracks it in every seed.
  • Mechanism portability: warm-starting across loop budgets transfers the learned algorithm in every seed, re-pricing its frontier speed to the new contract, while imposed seriality through streaming input schedules fails where free training succeeds.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The budget law's 'never exceeded' claim is a selection principle only if faster-but-equally-trainable solutions exist at the same contract; a natural extension is to search deliberately for such faster frontiers (e.g., with curricula or regularization) and check whether SGD avoids them—if they are learnable and still avoided, selection is confirmed.
  • The frontier speed v could serve as a test-time compute cost model: training budget and loops can be traded along the v ≈ n_tr/T_tr curve, and the halting rule T* could be integrated into serving of looped language models to allocate compute per input length.
  • On the easy-to-hard benchmark, the learned frontier breaks at 128–256 bits while convolutional Deep Thinking nets reach 512; this suggests global attention's length fragility, not the frontier mechanism, bounds extrapolation—so local or masked attention is a concrete testable extension.
  • Since τ is forward-only and defined on solved inputs, a drop in τ slope as length grows could serve as an early OOD-failure detector during training, a signal that tail metrics cannot provide within a mechanism class.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This paper studies weight-tied looped transformers—one block applied T times—trained on per-position group word problems. Its central claim is a 'budget law': free training installs a linear computation frontier whose speed v (positions solved per loop) tracks the training contract, v ≈ n_tr/T_tr (exponent 0.98±0.04, R²=0.99), and a model never exceeds the demanded minimum. The paper introduces a convergence-time scaling instrument τ(n,i), proves a saturation proposition for tail instruments, reports that weight tying flips SGD from parallel scans to serial frontiers, finds that NC1-completeness is not the binding difficulty (A5 learns while S5 deadlocks, and an operator-first curriculum dissolves the S5 wall), and shows warm-starting transfers mechanisms across loop budgets. Results are replicated on an easy-to-hard prefix-sum benchmark.

Significance. If the central claims hold, the paper is significant: it offers a predictive account of what test-time loops buy in weight-tied transformers, a principled halting rule, an instrument-validity constraint (Proposition 1 is simple but useful), and evidence that optimization/curriculum can dominate circuit-complexity predictions. The paper is commendably transparent about its main weakness—the absence of a constructive search for faster solutions—and it ships substantial empirical support in the form of activation-patching damage cones, tying-degree controls, and an open-benchmark replication. However, the headline budget law currently rests on the saturation of a necessity bound and on several inferred parameters, so the empirical content needs sharpening before the selection claim can be accepted.

major comments (4)
  1. [§6, Fig. 4/Table 1] The 'selection' claim in the budget law is indistinguishable from lower-bound saturation. For any model that solves all training positions by the final training loop T_tr, a contiguous frontier advancing v positions per loop must satisfy v ≥ n_tr/T_tr. The consistent-demand rows of Table 1 (v=0.51, 1.00, 2.00, 4.00 for demands 0.5, 1, 2, 4) are exactly the signature of a quantity pinned to its lower bound. The R²=0.99 fit of v against n_tr/T_tr is therefore not independent evidence, and the paper's own admission that no constructive search for faster-but-equally-trainable solutions was performed (§6) means 'SGD never exceeds the demand' is supported only as a necessity condition, not as a selection principle. Please add one of: (a) a constructive search over seeds/curricula/regularizers at a fixed contract for solutions with v > demand on S4/A5; (b) a negative control with a known cheap
  2. [§6, 'Cross-task form of the law'] The general form v ≈ clip(n_tr/T_tr, v_free, v_max) has at least three quantities—the exponent, v_free, and v_max—inferred from the same data used to report the law. For parity, v_free is defined as the measured floor; for A5, v_max is invoked where demand 4 yields v≈2.5. Without an independent measurement or a pre-specified generative model, the clip form is highly flexible and its agreement with the 8× demand sweep is not strong evidence for a law. Please quantify the number of free parameters per task, estimate them on training data only, and test the clip form on a held-out contract or task to make the law falsifiable.
  3. [Abstract and §7, Table 7] The claim 'NC1-completeness costs nothing (A5 generalizes fully)' overstates the evidence. Table 7 reports only 8/22 from-scratch T=n seeds cracking chained A5, with the remaining seeds never passing the second position; the abstract's 'generalizes fully' applies only to a subset with a favorable seed lottery. The paper does disclose this in §7, but the abstract and contribution (3) should carry the qualification, or report the median/range separately for cracked seeds. As written, the headline finding (3) is not supported by the full population.
  4. [§8, Table 8] The prediction race may be partly circular in the bimodal cells. 'Frontier completion at n=32' is defined only for seeds that have already learned the task at training lengths; failed seeds have no measurable frontier, and assigning them a zero makes the predictor essentially 'did the seed crack by the training length.' Reporting AUROC up to 1.00 for such a predictor overstates the instrument's value. Please restrict the race to seeds with a measurable frontier, or report the predictor's performance separately within the cracked-seed population, where the head instrument is claimed to retain signal.
minor comments (7)
  1. [§6, 'The frontier is causal'] The cone validation is based on only two models ('both models tested' in all 72 grid cells). This is a strong but narrow validation; please state explicitly that the causal cones are illustrative for the population, or extend to a representative sample of frontier models.
  2. [References] Several references are incomplete: 'et al. Blayney', 'et al. Kaissis', 'DifFRACT authors', and 'et al. Yu' lack proper author lists. Please fix.
  3. [Table 1] The A5 row pools cracked and non-cracked seeds. Report the frontier speed v for cracked seeds separately; the pooled median (0.96) with range [0.08, 1.03] hides the bimodality that is central to §7.
  4. [Table 2] The 'overthinking' degradation for tight-budget models is reported on group means. Clarify whether accuracy is computed on solved examples only or on all examples, and report the number of seeds per cell.
  5. [Definition 1] The definition of τ_max(n) as 'median over inputs' should specify the tie-breaking rule and the number of solved inputs required; Appendix C's 'at least 3 lengths' rule appears only later.
  6. [§7] The annealing arms report '4/4 seeds' with no individual values or variance. Given the small N and the importance of the re-pricing result, show the per-seed v before/after transfers.
  7. [§3] The term 'free training' is used to mean 'no imposed seriality,' which may be confused with parameter-free training. Define the term at first use.

Circularity Check

1 steps flagged

Budget law's v≈n_tr/T_tr is partly a definitional lower bound; the 'SGD selects the minimum' claim is explicitly unsearched.

specific steps
  1. self definitional [§6 'The budget law' (Fig. 4, Table 1)]
    "so SGD consistently arrives at a frontier whose speed matches the minimum the contract demands, and v is an experimenter-controllable knob (Fig. 3, left). We state this as a robust empirical regularity rather than a selection theorem: demonstrating that faster-but-equally-trainable solutions exist at the same contract (and are avoided) would require a constructive search over solutions we have not performed; what the data establish is that the demanded minimum is met, never exceeded, across an 8× range of consistent demands."

    v is measured as positions-solved per loop on models trained to exact match all n_tr positions by the final training loop T_tr. Any such successful model has average throughput n_tr/T_tr over the training interval; a linear frontier from the start to the completed prefix must reproduce this chord slope. Hence the observed v≈n_tr/T_tr (and the R²=0.99 fit) is a necessary consequence of training success at least as much as a discovery about SGD's selection. The paper's own caveat—that a selection claim would require a constructive search for faster-but-equally-trainable solutions, which it has not performed—concedes the equality alone cannot distinguish 'the demanded minimum is met' from 'the demanded minimum is all that is possible'. Independent content (cross-length constancy, damage cones

full rationale

The only load-bearing circularity I can exhibit is in the budget law itself. Because v is defined as the speed of the solved-prefix frontier on models trained to solve all n_tr positions by the final training loop T_tr, the average speed over that interval is n_tr/T_tr by construction. Regressing v against n_tr/T_tr therefore partly correlates a measured quantity with its own lower bound, especially in the T=n case where the table reports v=1.00 [1.00, 1.00]. The paper explicitly limits the claim: it is 'a robust empirical regularity rather than a selection theorem,' and admits no constructive search for faster-but-equally-trainable solutions was performed. That admission is the key circularity-relevant fact. I found no load-bearing self-citation: the cited complexity results (Barrington, Merrill & Sabharwal, Liu et al.) are external and used as a hardness ladder, not to establish the budget law. The frontier mechanism itself is supported by independent evidence—cross-length constancy of v, fixed-length rescue, causal damage cones whose slopes reproduce v, and annealing that re-prices speed—so the paper is not wholly circular. The moderate score reflects that the central pricing relation is partly definitional while the mechanism claims retain independent empirical content.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The central claims rest on a measurement instrument (τ and frontier speed) whose interpretation is partly definitional, on a budget law whose lower-bound component is tautological, and on an implicit assumption that solved-training-length measurements reveal the test-time mechanism.

free parameters (3)
  • power-law exponent for budget law = 0.98 ± 0.04
    Fit to measured frontier speed vs training demand over 23 models; the near-unity exponent makes the law's functional form close to the tautological lower bound v≥n_tr/T_tr.
  • parity speed floor v_free = ≈1–1.5
    Estimated from parity models; used in cross-task form v≈clip(n_tr/T_tr, v_free, v_max) to explain why loose-contract parity models do not slow below this.
  • A5 operator cap v_max = ≈2.5
    Estimated from the A5 demand sweep; attributed to the 60×60 composition operator capping frontier speed.
axioms (4)
  • standard math Fixed-point saturation: at a fixed point h* of the loop map, skip tolerance, Jacobian agreement, and feature-dictionary agreement are uninformative about earlier loops.
    Proposition 2 follows from idempotence; no empirical premise needed beyond the model reaching a fixed point.
  • domain assumption The training contract can be summarized by the demand ratio n_tr/T_tr and frontier speed is a per-model constant across lengths.
    The budget law assumes measured v on {16,32,64} training lengths transfers to the mechanism; the paper provides evidence but no proof that v is invariant across curriculum stages and test lengths.
  • domain assumption The decoded output's stabilization time τ(n,i) is a valid upper bound on when the underlying computation finished, and causal damage cones on a subset of models license mechanistic reading for all frontier models.
    τ is defined on solved inputs and final decoded output; stabilization may lag or lead the true computation, and cones were validated on 'both models tested' (two models), not the whole population.
  • domain assumption Per-position supervision on prefix products with a linear readout head is sufficient to identify whether models implement an algorithm as opposed to memorization.
    The task family blocks answer-only shortcuts and gives every intermediate position ground truth, but generalization results still depend on this specific supervision scheme.
invented entities (2)
  • Linear computation frontier independent evidence
    purpose: Describes the mechanism by which a looped transformer extends its solved prefix by v positions per loop.
    It generates falsifiable predictions: per-position rescue with extra loops, halting rule T*=ceil(n/v), and damage-cone slopes that reproduce v; these are checked in the paper.
  • Convergence-time scaling τ(n,i) independent evidence
    purpose: Head-of-trajectory instrument that reports the loop at which the decoded output at each position stabilizes.
    It orders seed generalization where tail metrics do not (Table 8) and is calibrated on streaming control, though that control is definitionally forced.

pith-pipeline@v1.3.0-alltime-deepseek · 13914 in / 14213 out tokens · 115915 ms · 2026-08-01T10:08:43.443274+00:00 · methodology

0 comments
read the original abstract

When does a weight-tied looped transformer -- one block applied T times -- implement an actual algorithm? We answer with four findings from controlled populations on group word problems. (1) The budget law: free training installs a linear computation frontier, a mechanism that solves v positions per loop, whose speed is priced by the training contract: v ~ n_train/T_train (exponent 0.98 +/- 0.04, R^2=0.99), exactly unity under T=n training. SGD selects a frontier matching the minimum the contract demands; granting more test-time loops than ever trained rescues late positions at fixed input length, yielding a principled halting rule T* = ceil(n / v-hat). (2) Architecture prior, not expressivity, picks the algorithm: standard-depth transformers learn parallel scans on this family; weight tying flips the selection to the serial frontier, even when positional addressing for a log-depth scan is supplied. At matched depth and parameters, untied models extrapolate worst and fail to learn A5 at all. (3) The walls are not where circuit complexity says: NC1-completeness costs nothing (A5 generalizes fully), while group order does (S5's 120x120 operator deadlocks joint learning) -- and an operator-first curriculum dissolves the wall in every seed. (4) Mechanisms are portable, not mandatable: warm-starting across budget contracts transfers the algorithm in every seed, re-pricing its speed, while imposing seriality through the input schedule fails where free training succeeds. These results are invisible to standard instruments, which provably saturate at the fixed points trained loops converge to. We introduce a head instrument, the convergence-time scaling tau(n,i), validate it causally via damage cones whose slope reproduces v, and show in-distribution head measurements predict out-of-distribution fate where tail metrics do not. Results replicate on the public easy-to-hard benchmark.

Figures

Figures reproduced from arXiv: 2607.20594 by Junhao Hu, Tao Xie, Tong Zhang, Yun Peng.

Figure 1
Figure 1. Figure 1: Overview. Left: a weight-tied looped transformer applies one shared block T times. Middle: trained models solve per-position tasks via a computation frontier that advances a fixed number of positions per loop; tail instruments only see the converged region, where they provably saturate, while the algorithm lives in the head of the trajectory. Right: the scaling class of the convergence time τ (n, i) (flat … view at source ↗
Figure 2
Figure 2. Figure 2: Per-(loop, position) accuracy heatmaps at [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Two organizing findings. Left: the budget law—the training contract prices the frontier speed; the demanded minimum is met, never exceeded. Right: mechanism portability—the algo￾rithm is found by a basin lottery, transfers across contracts by warm-starting, and cannot be installed through the input schedule [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Left (budget law): measured frontier speed tracks the training demand ntr/Ttr (dashed: v = demand) for consistent demands; the log-T cell (red, mixed demands averaging 4) compromises at v ≈ 2. Right (causal cones): total downstream damage from patching state at (position i, loop t). Damage is strictly downstream (zero upstream in every cell), tracks the moving frontier, and the cone slope reproduces each m… view at source ↗
Figure 5
Figure 5. Figure 5: The learning walls track operator size, not circuit-complexity class: the non-solvable [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Left: the budget law densified—frontier speed vs. training demand over 23 models span￾ning 8× in demand; the fit is v ∝ demand0.98±0.04 , R2=0.99 (dashed: v = demand). Right: the law as a halting rule—stopping at T ⋆=⌈n/vˆ⌉ with vˆ estimated in distribution (blue) nearly matches an oracle T sweep (green) and far exceeds the training schedule’s own fixed T (grey), while avoid￾ing the overthinking collapse o… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 9 linked inside Pith

  1. [4]

    Looped transformers for length generalization.arXiv preprint arXiv:2409.15647,

    Ying Fan, Yilun Du, Kannan Ramchandran, and Kangwook Lee. Looped transformers for length generalization.arXiv preprint arXiv:2409.15647,

    Pith/arXiv arXiv

  2. [6]

    Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun

    arXiv:2402.09963. Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. NeurIPS,

    Pith/arXiv arXiv

  3. [8]

    et al. Kohli. Loop, think, & generalize: Implicit reasoning in recurrent-depth transformers.arXiv preprint arXiv:2604.07822,

    Pith/arXiv arXiv

  4. [10]

    12 Preprint Table 10: Frontier-state transplant (A’s prefix states spliced intoB’s run at the frontier, loopt=16): retention vs

    arXiv:2503.03961. 12 Preprint Table 10: Frontier-state transplant (A’s prefix states spliced intoB’s run at the frontier, loopt=16): retention vs. repair is governed by the storage phenotype of Table

    arXiv

  5. [11]

    Gleb Rodionov and Liudmila Prokhorenkova

    arXiv:2404.08819. Gleb Rodionov and Liudmila Prokhorenkova. Neural algorithmic reasoning without intermediate supervision.NeurIPS,

    Pith/arXiv arXiv

  6. [13]

    arXiv:2502.08482. A LOCALIZATION,TRANSPORT,AND EDITING UNDER WEIGHT TYING The survey lines that dominate interpretability—neuron localization (Dai et al., 2022; Wang et al., 2022; Tang et al., 2024), causal-tracing-guided editing (Meng et al., 2022), and circuit discov- ery (Conmy et al., 2023; Yao et al., 2024)—all index structure byparameter address(lay...

    Pith/arXiv arXiv 2022

  7. [1989]

    et al. Blayney. A mechanistic analysis of looped reasoning language models.arXiv preprint arXiv:2604.11791,

    Pith/arXiv arXiv

  8. [2019]

    Diffract: Diffusion feature reconstruction and attribution for circuit tracing

    DifFRACT authors. Diffract: Diffusion feature reconstruction and attribution for circuit tracing. arXiv preprint arXiv:2606.15796,

    arXiv

  9. [2022]

    Looped transformers are better at learning learning algorithms.arXiv preprint arXiv:2311.12424,

    Liu Yang, Kangwook Lee, Robert Nowak, and Dimitris Papailiopoulos. Looped transformers are better at learning learning algorithms.arXiv preprint arXiv:2311.12424,

    Pith/arXiv arXiv

  10. [2023]

    et al. Kaissis. Step-resolved data attribution for looped transformers.arXiv preprint arXiv:2602.10097,

    arXiv

  11. [2024]

    Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein

    Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach.arXiv preprint arXiv:2502.05171,

    Pith/arXiv arXiv

  12. [2025]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov

    arXiv:2507.02199. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT.NeurIPS,

    arXiv

  13. [2026]

    Ouro: Looped language models.arXiv preprint arXiv:2510.25741,

    ByteDance Seed. Ouro: Looped language models.arXiv preprint arXiv:2510.25741,

    Pith/arXiv arXiv