REVIEW 3 major objections 5 minor 19 references
Intra-Layer Recurrence in Transformers for Language Modeling
T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The largest perplexity gains come from reusing early transformer layers, not later ones, with no new parameters.
desk verdict The granular reuse-map idea is worth a look, but the early-layer optimality claim rests on single runs and contradicts the paper's own table. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the reuse map $R=[r_1,\ldots,r_L]$, a vector assigning each transformer layer a repetition count within one forward pass. It converts the usual layer recursion $h^{(l)}=f_\theta^{(l)}(h^{(l-1)})$ into a two-stage recurrence: first $h^{(l,1)}=f_\theta^{(l)}(h^{(l-1)})$, then $h^{(l,k)}=f_\theta^{(l)}(h^{(l,k-1)})$ for $k=2,\ldots,r_l$. The reuse map is the entire mechanism: changing one integer per layer changes the model's effective depth without changing its weights, and the paper's experiments vary this map to locate which layers pay off when repeated. Backpropagation through the repetitions produces products of per-layer Jacobians, which the paper cites as the reason high reuse in a single layer risks gradient explosion or vanishing.
What would settle it
Train the 1.2M-parameter model with the same reuse maps while compensating for the added compute, for example by reducing training tokens or steps so every configuration receives the same total FLOPs, and check whether early-layer maps like $[4,2,1,1]$ still beat both the baseline and a later-focused map like $[1,1,1,4]$; repeat with several seeds to confirm the ordering is stable. If early-layer maps no longer win under FLOP-matched or seed-averaged training, the central claim is falsified.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that a transformer's forward pass can be re-routed to loop individual layers, and that the return on those loops is largest at the bottom of the network. The mechanism is a reuse map $R=[r_1,\ldots,r_L]$, where $r_l$ is the number of times layer $l$ fires before the next layer sees the representation; gradients accumulate across all repetitions of a layer during backpropagation. With this mechanism, the paper reports that early-focused maps such as $[4,2,1,1]$ and $[3,2,2,1]$ reduce test perplexity relative to the baseline in a 1.2M-parameter decoder-only transformer, that the improvement holds across NoPE, RoPE, Learned Absolute, and ALiBi positional encodings, and that at 100M parameters a single early-layer reuse map $[1,2,1,\ldots,1]$ gives consistent gains of roughly 0.1-0.2 perplexity. The paper reads these results as evidence that early layers carry the representational load and benefit from recurrent self-refinement, while later layers mostly refine and are not worth repeating.
Load-bearing premise
The conclusion that early-layer reuse is what drives the gains assumes the perplexity differences are not simply effects of the extra forward and backward passes or of run-to-run noise, because every recurrent configuration costs more FLOPs than the baseline and each configuration was run with a single seed.
Editorial extensions
If this is right
- A fixed parameter count can buy lower test perplexity by re-running selected early layers, so the relevant currency for ILR becomes compute per token rather than weights.
- Because ILR improves perplexity under NoPE, RoPE, Learned Absolute, and ALiBi at small scale, the early-layer benefit is not tied to one positional encoding scheme.
- Selective per-layer reuse can beat block-level recurrence, which repeats every layer equally: in the small-scale table the best ILR map is better than block recurrence for all four encodings.
- The best reuse map is position-dependent: $[2,2,2,2]$ wins for Learned PE while $[4,2,1,1]$ wins for RoPE and ALiBi, so 'earlier is better' is a tendency with exceptions.
- Within the paper's results, scaling ILR is only weakly established: the 100M model tests a single reuse map, so the gains exist but the optimal map at that scale is not known.
Reading between the lines
- I infer that if the early-layer effect survives a compute-matched control, then layer location and effective depth are separable design resources, and a learned controller could route extra iterations per input rather than fixing one reuse map for the whole run.
- I infer from Table 2 that the perplexity gain may partly be a training-compute effect: every recurrent configuration costs more FLOPs than baseline, so without holding FLOPs or optimizer steps constant the layer-location story is not isolated.
- I infer that the dependence of the best small-scale map on positional encoding is an under-explored signal: it suggests reuse maps should be tuned jointly with the encoding scheme, and that a single universal 'early layers' rule may not transfer across architectures.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Intra-Layer Recurrence (ILR), a scheme in which individual transformer layers are re-entered within a single forward pass according to a per-layer reuse map R=[r_1,...,r_L]. The authors train 1.2M-parameter and 100M-parameter LLaMA-style decoder-only models on a FineWeb-Edu subset and report test perplexity for various reuse maps under four positional encoding schemes (NoPE, RoPE, Learned Absolute PE, ALiBi). They report consistent perplexity improvements of ILR configurations over the no-recurrence baseline at no increase in parameter count, and the abstract claims that allocating more iterations to earlier layers yields optimal results. The paper also provides gradient equations for backpropagation through reused layers, a comparison with block-level recurrence, and a discussion of compute and training limitations.
Significance. If established, the central finding would be a useful empirical contribution: selective layer reuse can improve perplexity at a fixed parameter count, and the granularity of per-layer reuse maps is a design dimension worth studying. The paper is transparent about limitations, releases code, and evaluates across four positional encoding schemes, which strengthens the empirical base. However, the headline claim that early layers benefit most is currently supported only by unreplicated single runs and by a large-scale result that was selected after inspecting small-scale results. Since the quantitative ranking of reuse maps is the entire basis of the paper's conclusion, the lack of variance information and the internal inconsistency across positional encodings are load-bearing issues that must be resolved before the claim can be accepted.
major comments (3)
- [§4.1, Table 1] The claim that "allocating more iterations to earlier layers yields optimal results" (Abstract) is not consistently supported by the small-scale results in Table 1. Among maps with the same total recurrence (the 'doubled depth' class with sum r_i = 8), the best map is [4,2,1,1] for RoPE and ALiBi, but it is [3,2,2,1] for NoPE (14.62 vs 14.64 for [4,2,1,1]) and [2,2,2,2] for Learned PE (14.17 vs 14.20 for [4,2,1,1]). Moreover, for RoPE the single-layer map [1,2,1,1] (13.93) beats [2,1,1,1] (14.40), so the 'earlier layer is better' pattern is not monotone in layer index. This internal inconsistency is directly relevant to the paper's central claim and needs to be addressed.
- [§4.1, Table 1 and §4.2] Every perplexity number in Table 1 is from a single run, with no seeds, variance, or statistical tests. Several decisive differences are very small: ALiBi 13.63 vs 13.64 for [4,2,1,1] vs [3,2,2,1], and NoPE 14.62 vs 14.64 for [3,2,2,1] vs [4,2,1,1]. For a 1.2M-parameter model, such differences are within typical seed-to-seed noise, and without replication the ranking that motivates the early-layer conclusion could flip. The manuscript should report multiple seeds with error bars, or at minimum a replicated comparison for the maps that are used to support the headline claim.
- [§4, §5] The large-scale confirmation is a single map, [1,2,1,...,1], chosen 'guided by results from the small-scale experiments' (Section 4), and Section 5 concedes that the 100M model was trained on only 3B tokens (500M tokens × 6 epochs), which is likely undertrained. This makes the large-scale experiment incapable of independently confirming early-layer optimality. To use the large-scale result as evidence, the selection rule should be pre-specified and at least a small set of maps should be compared at this scale under matched compute; as it stands, the large-scale result is a single post-hoc datapoint.
minor comments (5)
- [§4, Figure 3] The block-recurrence state mapping is described only briefly; it would help to state explicitly that the implementation uses x = h + e, and to clarify how this differs from the learned adapter in Geiping et al.
- [Table 2] Training FLOPs are reported only for broad categories (baseline, reuse single layer, doubled depth). Since the paper argues about the relative merits of different reuse maps, it would be useful to report FLOPs per row of Table 1, especially to confirm that maps with equal total recurrence indeed have equal FLOPs.
- [§3, Eq. (3)] The gradient formula for parameters accumulates products of Jacobians; the notation is understandable but the dimensions of the final product (vector ∂f/∂θ times δ) could be spelled out more precisely to avoid confusion.
- [§4.1] Evaluation details are incomplete: please specify the number of test tokens, the sequence length used for evaluation, and whether perplexity is computed with a sliding window or full context. This matters for reproducibility.
- [§4.2, References [12,13]] The justification that early layers are most influential cites work on BERT (an encoder) and a blog post; consider citing layerwise analyses of decoder-only language models to better support the transfer of this intuition to the present setting.
Circularity Check
No circularity: the ILR claim is an empirical, held-out perplexity comparison, not a quantity derived from its own definitions or fitted to the target result.
full rationale
The paper's central claim—that allocating more iterations to earlier layers yields optimal results—is an empirical observation from Table 1, where each reuse map is evaluated on a test set and compared against baselines. No reuse map is fitted to the test perplexities, and no parameter is tuned to produce the conclusion. The small-scale results are direct measurements, and the large-scale configuration ([1,2,1,...,1]) was selected from small-scale runs and evaluated on a different model scale (100M vs 1.2M), making it an out-of-sample check rather than a construction. The reuse map is defined in Section 3 as R = [r1,...,rL], and the forward pass is specified independently of any target result; the perplexity values are then measured, not derived from the definition. The gradient analysis is standard backpropagation and is not used to infer early-layer optimality. Citations [12,13] are external empirical studies about layer contributions, not self-citations, and they are used only as supporting context, not as the proof of the paper's results. The limitations section concedes that the large model was undertrained (3B tokens), which weakens the confirmatory value of the large-scale experiment but does not create circularity. A numerical inconsistency exists in Section 4.2, where the text attributes 14.62 to [4,2,1,1] for NoPE while Table 1 lists 14.64 for that map and 14.62 for [3,2,2,1]; this is a correctness or reporting issue, not a circularity issue, because the conclusion is not defined into existence by that statement. Overall, the derivation chain is self-contained: the architecture is specified, the experiments are run, and the conclusions are drawn from measured perplexity differences against held-out test data.
Assumptions & free parameters
free parameters (1)
- reuse map R = [r_1,...,r_L] =
hand-selected maps: [4,2,1,1], [3,2,2,1], [2,2,2,2], [1,2,1,1], [1,1,2,4], etc. (Table 1)
assumptions (4)
- standard math Backpropagation through a reused layer accumulates gradients across all recurrence steps, as in the gradient equations of Section 3.
- domain assumption Test perplexity on a FineWeb-Edu subset is a valid and sufficiently stable metric to rank configurations.
- domain assumption The 1.2M-parameter model is representative enough to choose the large-scale reuse map.
- domain assumption Models trained on the same number of tokens are directly comparable even though recurrent models consume more FLOPs per token.
Cite this review
Pith. "Pith review of Intra-Layer Recurrence in Transformers for Language Modeling." pith.science (2026). https://pith.science/paper/HDMGIAHX
@misc{pith2026250501855,
author = {Pith},
title = {Pith review of: Intra-Layer Recurrence in Transformers for Language Modeling},
year = {2026},
howpublished = {\url{https://pith.science/paper/HDMGIAHX}},
note = {Machine review of arXiv:2505.01855}
}
read the original abstract
Transformer models have established new benchmarks in natural language processing; however, their increasing depth results in substantial growth in parameter counts. While existing recurrent transformer methods address this issue by reprocessing layers multiple times, they often apply recurrence indiscriminately across entire blocks of layers. In this work, we investigate Intra-Layer Recurrence (ILR), a more targeted approach that applies recurrence selectively to individual layers within a single forward pass. Our experiments show that allocating more iterations to earlier layers yields optimal results. These findings suggest that ILR offers a promising direction for optimizing recurrent structures in transformer architectures.
Figures
Reference graph
Works this paper leans on
-
[1]
Gomez, Lukasz Kaiser, and Illia Polosukhin.Attention Is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.Attention Is All You Need. 2017. arXiv: 1706 . 03762 [cs.CL]. url: https://arxiv.org/abs/1706.03762
arXiv 2017
-
[2]
Tom B. Brown et al. Language Models are Few-Shot Learners. 2020. arXiv: 2005 . 14165 [cs.CL]. url: https://arxiv.org/abs/2005.14165
arXiv 2020
-
[3]
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser.Uni- versal Transformers. 2019. arXiv:1807.03819 [cs.CL]. url: https://arxiv.org/abs/1807. 03819
arXiv 2019
-
[4]
Lee, and Dim- itris Papailiopoulos
Angeliki Giannou, Shashank Rajput, Jy yong Sohn, Kangwook Lee, Jason D. Lee, and Dim- itris Papailiopoulos. Looped Transformers as Programmable Computers. 2023. arXiv: 2301. 13196 [cs.LG]. url: https://arxiv.org/abs/2301.13196
arXiv 2023
-
[5]
Liu Yang, Kangwook Lee, Robert Nowak, and Dimitris Papailiopoulos.Looped Transformers are Better at Learning Learning Algorithms. 2024. arXiv:2311.12424 [cs.LG]. url: https: //arxiv.org/abs/2311.12424
arXiv 2024
-
[6]
Ying Fan, Yilun Du, Kannan Ramchandran, and Kangwook Lee.Looped Transformers for Length Generalization. 2024. arXiv: 2409.15647 [cs.LG] . url: https://arxiv.org/abs/ 2409.15647
arXiv 2024
-
[7]
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein.Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. 2025. arXiv: 2502 . 05171 [cs.LG]. url: https://arxiv.org/abs/2502.05171
arXiv 2025
-
[8]
SeppHochreiterandJürgenSchmidhuber.“Longshort-termmemory”.In: Neural computation 9.8 (1997), pp. 1735–1780
work page 1997
Show all 19 references
-
[9]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. 2019. arXiv: 1810.04805 [cs.CL]. url: https://arxiv.org/abs/1810.04805
2019 arXiv
-
[10]
Lan- guage Models are Unsupervised Multitask Learners
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. “Lan- guage Models are Unsupervised Multitask Learners”. In: (2019)
2019
-
[11]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Ro- driguez, Armand Joulin, Edouard Grave, and Guillaume Lample.LLaMA: Open and Efficient Foundation Langu...
2023 arXiv
-
[12]
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky.Revealing the Dark Secrets of BERT. 2019. arXiv: 1908.08593 [cs.CL] . url: https://arxiv.org/abs/1908. 08593
2019 arXiv
-
[13]
Interpreting GPT: The Logit Lens
Nostalgebraist. Interpreting GPT: The Logit Lens. LessWrong. 2020. url: https : / / www . lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens
2020
-
[14]
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf.The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. 2024. arXiv: 2406 . 17557 [cs.CL]. url: https : //arxiv.org/abs/2406.17557
2024 arXiv
-
[15]
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy.The Impact of Positional Encoding on Length Generalization in Transformers
-
[16]
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.RoFormer: Enhanced Transformer with Rotary Position Embedding. 2023. arXiv: 2104.09864 [cs.CL] . url: https://arxiv.org/abs/2104.09864
2023 arXiv
-
[17]
Smith, and Mike Lewis.Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Ofir Press, Noah A. Smith, and Mike Lewis.Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. 2022. arXiv: 2108.12409 [cs.CL] . url: https: //arxiv.org/abs/2108.12409
2022 arXiv
-
[18]
Training Compute-Optimal Large Language Models
Jordan Hoffmann et al. Training Compute-Optimal Large Language Models. 2022. arXiv: 2203.15556 [cs.CL]. url: https://arxiv.org/abs/2203.15556
2022 arXiv
-
[2023]
url: https://arxiv.org/abs/2305.19466
arXiv: 2305.19466 [cs.CL]. url: https://arxiv.org/abs/2305.19466
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.