{"id":"c344331f-c677-43e3-b02a-c8df5da9cd89","arxiv_id":"2505.01855","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Selectively reusing early transformer layers inside a forward pass improves language modeling perplexity without adding parameters, but at extra compute cost and with evidence only from small-scale runs.","lead":"This paper tests a variant of transformer architecture where selected layers are run multiple times within one forward pass, and reports that putting extra passes on early layers improves language modeling perplexity at a fixed parameter count. It is a useful read for anyone tracking compute-for-quality trade-offs in efficient language modeling, though the gains are small and come with extra computation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Early-layer optimality rests on unreplicated single numbers and a post-hoc chosen large-scale map; with equal total recurrences the best reuse map varies by positional encoding.","rationale":"The reader's conditional verdict is appropriate, and my stress test does not move it. The general direction—selective per-layer recurrence can improve test perplexity without increasing parameter count—is reasonably supported: most ILR maps in Table 1 beat the baseline, and the code is released. The load-bearing problem is specifically the abstract's early-layer optimality claim. Even before worrying about FLOPs or seed noise, the paper's own numbers show that the best total-recurrence-8 map is [2,2,2,2] under Learned PE, not an early-heavy map, and that [4,2,1,1] ties or loses to [3,2,2,1] under NoPE. The large-scale run adds little because it reuses layer 2, not layer 1; it was chosen post hoc; and the paper itself states the model was undertrained. A compute-matched, multi-seed rerun of the small-scale comparison would settle whether the early-layer ranking is real. Until then, the headline overclaims and the correct verdict remains conditional acceptance with a request for variance reporting and a pre-specified reuse-map selection rule.","tokens_in":5239,"tokens_out":7521,"duration_ms":72340,"concrete_test":"Rerun the small-scale experiments for the five total-recurrence-8 maps ([4,2,1,1], [3,2,2,1], [2,2,2,2], [1,2,2,3], [1,1,2,4]) plus block recurrence and the no-recurrence baseline, using at least 5 seeds per map per positional encoding, with identical token counts and step counts, and report mean test perplexity ± standard error. Then test whether the most early-heavy map [4,2,1,1] significantly beats [2,2,2,2] and [3,2,2,1] within each positional encoding. If the best map is not consistently early-heavy across positional encodings or is within seed noise, restrict the claim to 'some layer-wise reuse improves perplexity without adding parameters' and remove the early-layer optimality sentence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'allocating more iterations to earlier layers yields optimal results'—is load-bearing and currently insecure for three concrete reasons. (1) The paper's own Table 1 does not show a consistent early-layer winner. Among the total-recurrence-8 maps, which have the same FLOP-per-token class, the best small-scale map is [4,2,1,1] for RoPE and ALiBi, but [3,2,2,1] for NoPE and [2,2,2,2] for Learned PE. Moreover, [4,2,1,1] is not even the best early-heavy map in the NoPE column (14.64 vs 14.62 for [3,2,2,1]), so singling out [4,2,1,1] as the representative early-layer map is not justified. (2) Every reported perplexity is a single run with no seed or variance information; differences of 0.01–0.1 (e.g., ALiBi 13.63 vs 13.64) are within typical seed noise for a 1.2M-parameter model, so the ranking that motivates the abstract could flip under repetition. (3) The large-scale confirmation is one map, [1,2,1,...,1], which reuses layer 2 rather than layer 1, and Section 4 states it was selected 'guided by results from the small-scale experiments' only after those results were known. Section 5 further concedes that the 100M model was undertrained (3B tokens), so it cannot independently confirm early-layer optimality. The reader's compute/noise concern is valid, but the internal inconsistency across positional encodings is itself enough to block the headline as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Intra-Layer Recurrence (ILR), a scheme in which individual transformer layers are re-entered within a single forward pass according to a per-layer reuse map R=[r_1,...,r_L]. The authors train 1.2M-parameter and 100M-parameter LLaMA-style decoder-only models on a FineWeb-Edu subset and report test perplexity for various reuse maps under four positional encoding schemes (NoPE, RoPE, Learned Absolute PE, ALiBi). They report consistent perplexity improvements of ILR configurations over the no-recurrence baseline at no increase in parameter count, and the abstract claims that allocating more iterations to earlier layers yields optimal results. The paper also provides gradient equations for backpropagation through reused layers, a comparison with block-level recurrence, and a discussion of compute and training limitations.","tokens_in":5565,"tokens_out":3758,"duration_ms":35057,"significance":"If established, the central finding would be a useful empirical contribution: selective layer reuse can improve perplexity at a fixed parameter count, and the granularity of per-layer reuse maps is a design dimension worth studying. The paper is transparent about limitations, releases code, and evaluates across four positional encoding schemes, which strengthens the empirical base. However, the headline claim that early layers benefit most is currently supported only by unreplicated single runs and by a large-scale result that was selected after inspecting small-scale results. Since the quantitative ranking of reuse maps is the entire basis of the paper's conclusion, the lack of variance information and the internal inconsistency across positional encodings are load-bearing issues that must be resolved before the claim can be accepted.","major_comments":[{"comment":"The claim that \"allocating more iterations to earlier layers yields optimal results\" (Abstract) is not consistently supported by the small-scale results in Table 1. Among maps with the same total recurrence (the 'doubled depth' class with sum r_i = 8), the best map is [4,2,1,1] for RoPE and ALiBi, but it is [3,2,2,1] for NoPE (14.62 vs 14.64 for [4,2,1,1]) and [2,2,2,2] for Learned PE (14.17 vs 14.20 for [4,2,1,1]). Moreover, for RoPE the single-layer map [1,2,1,1] (13.93) beats [2,1,1,1] (14.40), so the 'earlier layer is better' pattern is not monotone in layer index. This internal inconsistency is directly relevant to the paper's central claim and needs to be addressed.","section":"§4.1, Table 1"},{"comment":"Every perplexity number in Table 1 is from a single run, with no seeds, variance, or statistical tests. Several decisive differences are very small: ALiBi 13.63 vs 13.64 for [4,2,1,1] vs [3,2,2,1], and NoPE 14.62 vs 14.64 for [3,2,2,1] vs [4,2,1,1]. For a 1.2M-parameter model, such differences are within typical seed-to-seed noise, and without replication the ranking that motivates the early-layer conclusion could flip. The manuscript should report multiple seeds with error bars, or at minimum a replicated comparison for the maps that are used to support the headline claim.","section":"§4.1, Table 1 and §4.2"},{"comment":"The large-scale confirmation is a single map, [1,2,1,...,1], chosen 'guided by results from the small-scale experiments' (Section 4), and Section 5 concedes that the 100M model was trained on only 3B tokens (500M tokens × 6 epochs), which is likely undertrained. This makes the large-scale experiment incapable of independently confirming early-layer optimality. To use the large-scale result as evidence, the selection rule should be pre-specified and at least a small set of maps should be compared at this scale under matched compute; as it stands, the large-scale result is a single post-hoc datapoint.","section":"§4, §5"}],"minor_comments":[{"comment":"The block-recurrence state mapping is described only briefly; it would help to state explicitly that the implementation uses x = h + e, and to clarify how this differs from the learned adapter in Geiping et al.","section":"§4, Figure 3"},{"comment":"Training FLOPs are reported only for broad categories (baseline, reuse single layer, doubled depth). Since the paper argues about the relative merits of different reuse maps, it would be useful to report FLOPs per row of Table 1, especially to confirm that maps with equal total recurrence indeed have equal FLOPs.","section":"Table 2"},{"comment":"The gradient formula for parameters accumulates products of Jacobians; the notation is understandable but the dimensions of the final product (vector ∂f/∂θ times δ) could be spelled out more precisely to avoid confusion.","section":"§3, Eq. (3)"},{"comment":"Evaluation details are incomplete: please specify the number of test tokens, the sequence length used for evaluation, and whether perplexity is computed with a sliding window or full context. This matters for reproducibility.","section":"§4.1"},{"comment":"The justification that early layers are most influential cites work on BERT (an encoder) and a blog post; consider citing layerwise analyses of decoder-only language models to better support the transfer of this intuition to the present setting.","section":"§4.2, References [12,13]"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of cs.CL and the central idea is worth pursuing. The main weakness is not the architecture itself but the strength of the evidence for the headline claim. Single-run perplexities and a post-hoc large-scale selection are common in workshop-scale submissions, but here they are load-bearing because the paper's main conclusion is a ranking among reuse maps. I would recommend the authors add seed variance, restrict the claim to what the data support, and either pre-register a large-scale comparison or present the large-scale result as exploratory."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things before reading. First, it's a clean, small study of a simple idea: reuse individual transformer layers a chosen number of times in a forward pass, governed by a \"reuse map,\" and measure test perplexity. Second, the headline claim—that earlier layers benefit most from recurrence—is not actually supported by the paper's own table; the best map flips across positional encodings, and every number comes from a single run.\n\nWhat's new: previous recurrent-transformer work (Universal Transformers, looped transformers, depth-recurrent transformers) reuses whole layers or blocks uniformly. The granular per-layer reuse map is a reasonable extension, and the paper gives it a clean formalization and gradient derivation. The authors also test four positional encodings, which is more thorough than one might expect at this scale, and they ship code. Credit is due for the setup.\n\nThe soft spots are substantial. The abstract says \"allocating more iterations to earlier layers yields optimal results,\" but Table 1 tells a different story. Among the eight-recurrence maps, the best map is [4,2,1,1] for RoPE and ALiBi, [3,2,2,1] for NoPE, and [2,2,2,2] for Learned PE. The text even misquotes its own table, claiming [4,2,1,1] reached 14.62 on NoPE when the table shows 14.64; the 14.62 belongs to [3,2,2,1]. So the \"early layer\" conclusion depends on cherry-picking one map per setting.\n\nSecond, there are no error bars or multiple seeds anywhere. Differences of 0.01–0.1 in perplexity (e.g., ALiBi 13.63 vs 13.64) are likely within run-to-run noise for a 1.2M-parameter model. That alone would make me hesitate to trust the ranking that motivates the claim.\n\nThird, the large-scale confirmation is one map, [1,2,1,...,1], selected after inspecting small-scale results, and it reuses layer 2, not layer 1. Section 5 concedes the 100M model was undertrained on 3B tokens. So the large-scale result can't independently confirm early-layer optimality.\n\nFourth, compute is not matched. Table 2 shows all recurrent configs use more FLOPs, and there's no baseline that spends the same FLOPs on a deeper non-recurrent model. The paper's \"without increasing parameter count\" is true but misleading, because the gains might just be from more compute per token.\n\nWho is this for? Anyone working on recurrent transformers or parameter-efficient architectures will find the granular reuse-map idea worth a quick look. But as written, the empirical support for the central claim is too thin. I'd send it to referees if it came to me—the idea deserves a serious check—but I'd expect major revision: add seeds/variance, a compute-matched baseline, and a systematic selection rule for the reuse map, or soften the claim accordingly.","headline":"The granular reuse-map idea is worth a look, but the early-layer optimality claim rests on single runs and contradicts the paper's own table.","tokens_in":6077,"tokens_out":2652,"would_cite":false,"duration_ms":22731,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The largest perplexity gains come from reusing early transformer layers, not later ones, with no new parameters.","keywords":["intra-layer recurrence","reuse map","transformer language modeling","perplexity","early layers","recurrent transformer","layer reuse","positional encoding"],"falsifier":"Train the 1.2M-parameter model with the same reuse maps while compensating for the added compute, for example by reducing training tokens or steps so every configuration receives the same total FLOPs, and check whether early-layer maps like $[4,2,1,1]$ still beat both the baseline and a later-focused map like $[1,1,1,4]$; repeat with several seeds to confirm the ordering is stable. If early-layer maps no longer win under FLOP-matched or seed-averaged training, the central claim is falsified.","tokens_in":5014,"feed_emoji":"🔁","tokens_out":11894,"duration_ms":100794,"temperature":0.7,"pith_summary":"The paper aims to establish that transformer layers are not interchangeable under recurrence: re-entering an individual layer within a single forward pass helps most when that layer sits early in the stack. It introduces intra-layer recurrence (ILR), governed by a reuse map that says how many times each layer is applied, and tests it on decoder-only language models of 1.2M and 100M parameters under four positional encodings. In the small-scale runs, several reuse maps beat the no-recurrence baseline on test perplexity (for example, $[4,2,1,1]$ improves NoPE from 16.57 to 14.64 and ALiBi from 14.38 to 13.63), with the best maps concentrating iterations on earlier layers. In the 100M-parameter run, reusing an early layer with map $[1,2,1,\\ldots,1]$ improves perplexity over baseline for all four encodings. The reason to care is that this offers a way to make a fixed parameter budget yield a better language model by trading compute per token instead of weights.","feed_headline":"More passes through early layers beats later-layer reuse","feed_subtitle":"Selective layer reuse in one forward pass cuts perplexity in 1.2M and 100M language models, no new weights.","key_machinery":"The central object is the reuse map $R=[r_1,\\ldots,r_L]$, a vector assigning each transformer layer a repetition count within one forward pass. It converts the usual layer recursion $h^{(l)}=f_\\theta^{(l)}(h^{(l-1)})$ into a two-stage recurrence: first $h^{(l,1)}=f_\\theta^{(l)}(h^{(l-1)})$, then $h^{(l,k)}=f_\\theta^{(l)}(h^{(l,k-1)})$ for $k=2,\\ldots,r_l$. The reuse map is the entire mechanism: changing one integer per layer changes the model's effective depth without changing its weights, and the paper's experiments vary this map to locate which layers pay off when repeated. Backpropagation through the repetitions produces products of per-layer Jacobians, which the paper cites as the reason high reuse in a single layer risks gradient explosion or vanishing.","core_discovery":"On its own terms, the paper's discovery is that a transformer's forward pass can be re-routed to loop individual layers, and that the return on those loops is largest at the bottom of the network. The mechanism is a reuse map $R=[r_1,\\ldots,r_L]$, where $r_l$ is the number of times layer $l$ fires before the next layer sees the representation; gradients accumulate across all repetitions of a layer during backpropagation. With this mechanism, the paper reports that early-focused maps such as $[4,2,1,1]$ and $[3,2,2,1]$ reduce test perplexity relative to the baseline in a 1.2M-parameter decoder-only transformer, that the improvement holds across NoPE, RoPE, Learned Absolute, and ALiBi positional encodings, and that at 100M parameters a single early-layer reuse map $[1,2,1,\\ldots,1]$ gives consistent gains of roughly 0.1-0.2 perplexity. The paper reads these results as evidence that early layers carry the representational load and benefit from recurrent self-refinement, while later layers mostly refine and are not worth repeating.","pith_inferences":["I infer that if the early-layer effect survives a compute-matched control, then layer location and effective depth are separable design resources, and a learned controller could route extra iterations per input rather than fixing one reuse map for the whole run.","I infer from Table 2 that the perplexity gain may partly be a training-compute effect: every recurrent configuration costs more FLOPs than baseline, so without holding FLOPs or optimizer steps constant the layer-location story is not isolated.","I infer that the dependence of the best small-scale map on positional encoding is an under-explored signal: it suggests reuse maps should be tuned jointly with the encoding scheme, and that a single universal 'early layers' rule may not transfer across architectures."],"forward_implications":["A fixed parameter count can buy lower test perplexity by re-running selected early layers, so the relevant currency for ILR becomes compute per token rather than weights.","Because ILR improves perplexity under NoPE, RoPE, Learned Absolute, and ALiBi at small scale, the early-layer benefit is not tied to one positional encoding scheme.","Selective per-layer reuse can beat block-level recurrence, which repeats every layer equally: in the small-scale table the best ILR map is better than block recurrence for all four encodings.","The best reuse map is position-dependent: $[2,2,2,2]$ wins for Learned PE while $[4,2,1,1]$ wins for RoPE and ALiBi, so 'earlier is better' is a tendency with exceptions.","Within the paper's results, scaling ILR is only weakly established: the 100M model tests a single reuse map, so the gains exist but the optimal map at that scale is not known."],"supporting_citations":[{"why":"provides the block-recurrence baseline that ILR is designed to outperform and to offer finer control than.","marker":"[7]"},{"why":"defines Learned Absolute Positional Embeddings, one of the four positional encodings under test.","marker":"[9]"},{"why":"supplies the LLaMA-style decoder-only architecture used for both small and large models.","marker":"[11]"},{"why":"supplies evidence that early transformer layers capture core representations, motivating the hypothesis that early layers benefit most.","marker":"[12]"},{"why":"introduces the logit lens observation that token predictions become well-formed early in the stack.","marker":"[13]"},{"why":"provides the deduplicated FineWeb-Edu subset used for pretraining and test perplexity.","marker":"[14]"},{"why":"defines the NoPE positional-encoding variant.","marker":"[15]"},{"why":"defines RoPE, one of the four positional encodings under test.","marker":"[16]"},{"why":"defines ALiBi, one of the four positional encodings under test.","marker":"[17]"}],"fun_headline_variants":["Loop early transformer layers for better language models","Early layers deserve more passes in transformers","Repeat early layers, not late ones, to cut perplexity","Intra-layer recurrence favors early layers in LM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that early-layer reuse is what drives the gains assumes the perplexity differences are not simply effects of the extra forward and backward passes or of run-to-run noise, because every recurrent configuration costs more FLOPs than the baseline and each configuration was run with a single seed.","fun_headline_variants_meta":{"raw":{"variants":["Loop early transformer layers for better language models","Early layers deserve more passes in transformers","Repeat early layers, not late ones, to cut perplexity","Intra-layer recurrence favors early layers in LM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1319,"prompt_tokens":880,"completion_tokens":439,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":380}},"tokens_in":496,"tokens_out":439,"duration_ms":4724,"temperature":1.0,"reasoning_tokens":380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:08:27.475117+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the 1.2M-parameter model with the same reuse maps while compensating for the added compute, for example by reducing training tokens or steps so every configuration receives the same total FLOPs, and check whether early-layer maps like $[4,2,1,1]$ still beat both the baseline and a later-focused map like $[1,1,1,4]$; repeat with several seeds to confirm the ordering is stable. If early-layer maps no longer win under FLOP-matched or seed-averaged training, the central claim is falsified.","supporting_citations":[{"cited_title":"Interpreting GPT: The Logit Lens","cited_arxiv_id":null,"evidence_quote":"introduces the logit lens observation that token predictions become well-formed early in the stack."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines the NoPE positional-encoding variant."}],"review_version":1}