REVIEW 4 major objections 4 minor 23 references
Reproducing Recurrent Transformers: The CoTFormer
T0 review · 4 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read A reproduction of CoTFormer finds that repeated latent computation helps with iterative retrieval tasks like p-hop induction, but does not by itself produce robust out-of-distribution counting on shifted-start. The paper argues CoTFormer is
desk verdict Honest reproduction with a useful negative result on ADM, but the task-dependent claim rests on single-seed p-hop. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the CoTFormer's recurrent block: a sequence of weight-tied transformer layers applied multiple times, with intermediate states preserved as attendable representations that mimic explicit chain-of-thought traces. This preserves the parameter footprint of a shallow model while allowing repeated composition of operations. The paper uses a p-hop induction task as a behavioral probe for iterative retrieval, where each hop requires finding the nearest previous occurrence of a query symbol and reading the next token; success requires composing this operation 32 times. The contrast with shifted-start counting, which requires maintaining a successor state, isolates when repeated
What would settle it
Run the p-hop induction experiment with at least 5 seeds and matched compute budgets across looped and shallow models. If a shallow Transformer with equivalent FLOPs matches or beats the looped models' accuracy, or if the looped advantage disappears under seed variation, the paper's task-dependent claim for iterative retrieval is not supported.
Extended reading notes
Core claim
The central claim is that recurrent latent computation, as embodied by CoTFormer's weight-tied repeated block with attendable cached states, provides a task-dependent inductive bias. In a controlled p-hop induction task (32 hops, sequences of length 256, alphabet size 4), looped models reach about 0.66 test accuracy versus 0.47 for a depth-1 Transformer, showing that repeated computation supports iterative retrieval. In contrast, on shifted-start counting (train length 25, test length 200), all models cap around 0.26-0.27 OOD accuracy regardless of loop depth, so recurrence does not induce a robust counting procedure. The paper also reproduces the original perplexity results within statistic
Load-bearing premise
The conclusion that recurrence specifically aids iterative retrieval rests on a single-seed p-hop result; if seed variance changes the ordering of CoTFormer/BUT versus shallow Transformers, the claim that recurrence helps iterative retrieval would be weakened.
Editorial extensions
If this is right
- If the task-dependent conclusion holds, looped transformer architectures should be applied selectively to tasks involving iterative retrieval or pointer-chasing, where they may outperform shallow models at matched or modest compute.
- The lack of benefit on shifted-start counting suggests that repeating layers does not by itself instill abstract state-tracking or counting abilities; architectural inductive biases beyond recurrence are needed for such tasks.
- The reproduced LayerNorm benefit (post-repeat normalization) supports the use of feature scaling in recurrent latent streams to prevent representation drift, even if the mechanism is not fully explained.
- The ADM's reported dynamic routing benefits require a clearer definition of efficiency; the paper shows the router uses deeper repeats more over time but does not verify a predictive compute-accuracy benefit at matched inference cost.
- Stronger conclusions about recurrence and algorithmic generalization require matched-compute evaluations and multi-seed results, as the paper's p-hop conclusions rest on a single seed.
Reading between the lines
- The phase-change attention reorganization observed around 27k steps could be a signature of the model transitioning from shallow to compositional use of repeat states, and might be testable as a general phenomenon in looped transformers rather than a CoTFormer-specific quirk.
- Because CoTFormer's advantage over a vanilla looped baseline (BUT) is modest on p-hop, the value of the attendable cache may be primarily in parameter sharing and training dynamics, not in a fundamentally different algorithmic capability; this could be probed by ablating cache attention.
- A practical testable extension: design a task suite that separates retrieval-heavy and successor-heavy algorithmic structure, then measure whether looped models consistently help only on the former, which would strengthen the task-dependent claim.
- The ADM router's sensitivity to extraction method suggests that published router-usage statistics in adaptive-compute architectures may need standardized reporting conventions to be reproducible.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reproduces the CoTFormer architecture and its structural variants (Reserved Layers, LN-CoTFormer, ADM) on language-modeling perplexity and compute-efficiency metrics, then extends evaluation to two controlled algorithmic tasks: shifted-start counting (OOD extrapolation) and p-hop induction (repeated retrieval). The main findings are: (i) base CoTFormer perplexity results reproduce within the paper's single-seed confidence intervals, with a small divergence for LN-CoTFormer at 60k steps attributed to a learning-rate scheduler change; (ii) ADM's claimed compute-routing benefit and deep-repeat usage are not cleanly confirmed, partly due to a replicated bug in the router-weight extraction script and backend/RNG confounds; (iii) recurrent computation helps on p-hop induction, where the target resembles iterative retrieval, but does not induce robust OOD counting on shifted-start. The paper concludes that CoTFormer offers task-dependent algorithmic benefits rather than a general latent chain-of-thought mechanism.
Significance. If the p-hop result is robust, this reproducibility study usefully sharpens the empirical picture of looped/weight-tied transformers: recurrent latent computation appears to provide a task-dependent inductive bias, helping on repeated-retrieval problems while not automatically yielding counter-like OOD extrapolation. The paper is also transparent about confounds (LR-scheduler invalidation, Gloo/NCCL backend changes, single-seed caveats) and explicitly flags where stronger claims would require further experiments. It identifies a concrete bug in the original ADM extraction script, a valuable contribution for the original authors and future users. However, the central p-hop conclusion currently rests on a single-seed comparison, and the abstract's 'out-of-distribution generalisation' framing overstates what the p-hop task actually tests. With additional seeds and matched-compute checks, the task-dependent conclusion would be substantially strengthened.
major comments (4)
- [§4.3, Table 5] The central task-dependent claim rests on a single-seed comparison. The key gap between Base Transformer 6×1 (test acc 0.6133) and CoTFormer 1×6 (0.6619) is 0.049, and there are no error bars for any configuration. Given that §4.2 reports seed sensitivity in shifted-start (e.g., Max OOD Acc varies by ~0.03 across seeds), a gap of this size could plausibly be within seed-level variance. The paper's own concluding caveat ('Stronger claims require matched-compute evaluations, more seeds') acknowledges this, but the abstract and §4.3 already assert that looped models 'clearly outperform' shallow transformers. Please provide multi-seed runs (at least 3 seeds) with SEM for the p-hop configurations, or soften the conclusion to a preliminary single-run observation.
- [Table 1] The claim of 'exact reproducibility' is based on comparing a single-seed per-batch 95% confidence interval to the original 3-seed SEM. These intervals measure minibatch noise for one trained model, not seed-to-seed variability; they are not a valid equivalence test. The ∆ values in Table 1 may be well within seed variance even when they fall outside the per-batch CI. Please either provide seed-level mean ± SEM for at least one configuration to validate the equivalence claim, or rephrase Table 1's caption and §3 text to say 'within our single-seed estimate' rather than 'exact reproducibility.'
- [§4.3, Dataset construction] The p-hop task is not an out-of-distribution test: train_constructive, val_constructive, and test_constructive are generated by the same generator at the same p=32, n=256, |Σ|=4. The abstract's claim that the paper tests 'out-of-distribution generalisation on inductive reasoning tasks' is therefore only supported by the shifted-start experiment, where recurrence does not help. The p-hop results are still informative about whether recurrence aids a repeated-retrieval task, but they should not be described as evidence about OOD generalization. Please align the abstract and §4.3 wording to distinguish in-distribution algorithmic probing from true OOD extrapolation.
- [§3, ADM CoTFormer] The paper identifies a real bug in get_router_weights.py (lines 129-131) and replicates it in fig. 2(c), but the conclusion 'ADM final repeat usage is lower than claimed' is confounded by the backend change from NCCL to Gloo (section 2.iii). The histogram differences between the paper's result and the corrected extraction could reflect the backend/RNG divergence rather than the extraction fix. The paper partly acknowledges this ('confounded by back-end divergences'), but the heading 'Irreproducible claims' and the sentence 'ADM final repeat usage is lower than claimed' go beyond what the evidence supports. Please rephrase to state that the original extraction method is incorrect and that the corrected method, under the current training setup, shows lower usage, without asserting the original distribution is necessarily wrong.
minor comments (4)
- [§4.1] Typo: 'cosistent' should be 'consistent'.
- [§4.1] The phase-change analysis is descriptive and relies on a single run. The paper correctly notes that establishing algorithmic recurrent reasoning requires additional experiments, but the phrase 'significant shift' is used without a statistical test. Consider adding a quantitative threshold or error bars to support 'significant.'
- [§2, item ii] The LR-scheduler change for LN-CoTFormer 40k→60k makes the +0.40 PPL difference in Table 3 non-attributable. This is acknowledged, but the table caption should explicitly note that the difference is not a clean reproduction.
- [Table 4] The note 'Formatted to four decimal places to maintain the precision limits mandated by p-hop experiments' is cryptic. Clarify why p-hop experiments mandate four decimals and whether this format hides meaningful differences.
Circularity Check
No significant circularity: the paper is an experimental reproduction and evaluation, with no prediction derived from fitted parameters or load-bearing self-citation.
full rationale
The paper's claims are empirical rather than derivational. It reproduces CoTFormer perplexity results against published numbers with confidence intervals, evaluates ADM router behavior as a measurement question, and runs two controlled algorithmic tasks (shifted-start counting and p-hop induction). No parameter is fitted to a subset of data and then renamed as a prediction; the p-hop task is constructed from an explicit iterative retrieval procedure, but the model is trained only on final labels and the comparison across architectures is an independent test. The task-dependent conclusion is supported by single-seed p-hop results and the paper itself flags the need for more seeds and matched compute, which is a statistical robustness concern, not circularity. References to prior work are external to the present authors and are not used to forbid alternatives or import uniqueness conclusions. The ADM extraction discrepancy is a measurement/reproducibility issue, not a circular argument. Overall, the derivation chain, where any exists, is self-contained and does not reduce to its own inputs.
Assumptions & free parameters
free parameters (1)
- Model width / head count in shifted-start task =
width=128, heads=4
assumptions (4)
- ad hoc to paper Per-batch 95% confidence intervals from one seed are a valid equivalence test against a 3-seed SEM
- domain assumption The constructive p-hop generator yields valid, representative instances of p-hop induction
- domain assumption Closed-form MAC estimates faithfully reflect compute cost
- domain assumption Model behavior is preserved despite batch-size, gradient-accumulation, and NCCL-to-Gloo changes
Cite this review
Pith. "Pith review of Reproducing Recurrent Transformers: The CoTFormer." pith.science (2026). https://pith.science/paper/UFOKPVJF
@misc{pith2026260719405,
author = {Pith},
title = {Pith review of: Reproducing Recurrent Transformers: The CoTFormer},
year = {2026},
howpublished = {\url{https://pith.science/paper/UFOKPVJF}},
note = {Machine review of arXiv:2607.19405}
}
read the original abstract
The CoTFormer architecture formalizes Chain-of-Thought as a form of recurrent latent computation, preserving intermediate states as attendable representations to mimic explicit reasoning traces. In this work, we evaluate CoTFormer and its structural variants across perplexity and compute efficiency metrics. Furthermore, we extend evaluation to controlled algorithmic settings to determine whether this recurrent framework improves out-of-distribution generalisation on inductive reasoning tasks.
Figures
Reference graph
Works this paper leans on
-
[1]
The Thirteenth International Conference on Representation Learning (ICLR) , year=
CoTFormer: A Chain-of-Thought Driven Architecture with Budget-Adaptive Computation Cost at Inference , author=. The Thirteenth International Conference on Representation Learning (ICLR) , year=
-
[2]
International Conference on Learning Representations (ICLR) , year=
Universal Transformers , author=. International Conference on Learning Representations (ICLR) , year=
-
[3]
arXiv preprint arXiv:2507.02199 , year=
Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer , author=. arXiv preprint arXiv:2507.02199 , year=
-
[4]
arXiv preprint arXiv:2601.10242 , year=
Loop as a Bridge: Can Looped Transformers Truly Link Representation Space and Natural Language Outputs? , author=. arXiv preprint arXiv:2601.10242 , year=
-
[5]
arXiv preprint arXiv:2509.25239 , year=
A Formal Comparison Between Chain of Thought and Latent Thought , author=. arXiv preprint arXiv:2509.25239 , year=
-
[6]
arXiv preprint arXiv:2404.05971 , year=
Does Transformer Interpretability Transfer to RNNs? , author=. arXiv preprint arXiv:2404.05971 , year=
-
[7]
arXiv preprint arXiv:2210.16156 , year=
Reliability of CKA as a Similarity Measure in Deep Learning , author=. arXiv preprint arXiv:2210.16156 , year=
-
[8]
ICLR 2024 Re-Align Workshop , year=
Correcting Biased Centered Kernel Alignment Measures in Biological and Artificial Neural Networks , author=. ICLR 2024 Re-Align Workshop , year=
2024
Show all 23 references
-
[9]
arXiv preprint arXiv:2502.15104 , year=
Estimating Neural Representation Alignment from Sparsely Sampled Inputs and Features , author=. arXiv preprint arXiv:2502.15104 , year=
-
[10]
Frontiers in Earth Science , year=
Re-evaluation of the Power of the Mann-Kendall Test for Detecting Monotonic Trends in Hydrometeorological Time Series , author=. Frontiers in Earth Science , year=
-
[11]
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=
Designing and Interpreting Probes with Control Tasks , author=. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=
2019
-
[12]
transformer-circuits.pub , year=
Toy Models of Superposition , author=. transformer-circuits.pub , year=
-
[13]
arXiv preprint arXiv:2601.21582 , year=
Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves , author=. arXiv preprint arXiv:2601.21582 , year=
-
[14]
arXiv preprint arXiv:2502.05171 , year=
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach , author=. arXiv preprint arXiv:2502.05171 , year=
-
[15]
arXiv preprint arXiv:2104.06022 , year=
Lessons on Parameter Sharing across Layers in Transformers , author=. arXiv preprint arXiv:2104.06022 , year=
-
[16]
arXiv preprint arXiv:2310.07096 , year=
Sparse Universal Transformer , author=. arXiv preprint arXiv:2310.07096 , year=
-
[17]
International Conference on Machine Learning (ICML) , year=
Similarity of Neural Network Representations Revisited , author=. International Conference on Machine Learning (ICML) , year=
-
[18]
Advances in Neural Information Processing Systems (NeurIPS) , year=
Attention Is All You Need , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=
-
[19]
International Conference on Learning Representations (ICLR) , year=
Neural GPUs Learn Algorithms , author=. International Conference on Learning Representations (ICLR) , year=
-
[20]
Advances in Neural Information Processing Systems (NeurIPS) , year=
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=
-
[21]
International Conference on Learning Representations (ICLR) , year=
CoTFormer: A Chain-of-Thought Driven Architecture with Budget-Adaptive Computation Cost at Inference , author=. International Conference on Learning Representations (ICLR) , year=
-
[22]
International Conference on Learning Representations (ICLR) , year=
Reasoning with Latent Thoughts: On the Power of Looped Transformers , author=. International Conference on Learning Representations (ICLR) , year=
-
[23]
International Conference on Learning Representations (ICLR) , year=
Language Models Need Inductive Biases to Count Inductively , author=. International Conference on Learning Representations (ICLR) , year=
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.