{"id":"1a861d32-fbe5-458a-9355-acfffe42ef5a","arxiv_id":"2502.08482","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Loop-aligned supervision lets a looped Transformer generate CoT chains beyond training length, and those chains improve an auto-regressive CoT model's length generalization.","lead":"The paper trains a looped Transformer to emit Chain-of-Thought reasoning steps one loop iteration at a time, then uses the looped model's generated chains to fine-tune a standard autoregressive model. The result is improved accuracy on arithmetic, edit distance, and longest-increasing-subsequence problems longer than the training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The fine-tuned AR model is evaluated only at the exact lengths used in its fine-tuning data; no test on longer, held-out lengths supports the claimed length generalization.","rationale":"The reader's verdict identifies the hand-crafted round-iteration alignment as the weakest assumption, which is a real generality limitation but does not threaten the demonstrated results on the three synthetic tasks. My concern is different and, I argue, more directly load-bearing for the central claim: the paper never demonstrates that the fine-tuned AR model extrapolates beyond the lengths on which it was fine-tuned. The term 'length generalization' is used throughout, but the experimental protocol conflates generalization with direct training on the target lengths. Stage II constructs generated data for lengths 16-25 (Arithmetic), 31-40 (ED), and 101-120 (LIS), then fine-tunes the AR model on this merged dataset. Figure 4's x-axis covers exactly these lengths. Thus the RELAY-enhanced curve shows that the AR model can fit the longer-length distribution when provided with high-quality supervision at those lengths; it does not show that the model has acquired the looped model's capacity to reason at even greater lengths. The motivation of the paper is precisely to overcome the AR model's dramatic accuracy drop when reasoning length exceeds training length, so the method's value depends on whether the improvement transfers to unseen longer lengths. A simple held-out evaluation would settle this. If the improvement does not transfer, the paper's contribution reduces to 'high-quality synthetic data generation at target lengths,' which is useful but not the claimed length generalization. I therefore recommend keeping the CONDITIONAL verdict (i.e., no change), but adding this extrapolation test as an explicit acceptance condition.","tokens_in":18826,"tokens_out":15583,"duration_ms":156110,"concrete_test":"Using the identical RELAY-enhanced AR checkpoint from Figure 4 (no additional fine-tuning), evaluate final-answer accuracy on held-out problem lengths beyond the maximum length seen during fine-tuning: Arithmetic operators 26-35, ED string lengths 41-50, LIS sequence lengths 121-140. Compare against the untuned AR-CoT baseline and the looped model with explicit CoT alignment at these lengths. If RELAY-enhanced accuracy collapses to near the baseline (e.g., less than a 15-point gap), then the improvement is confined to the fine-tuned length range and the length-generalization claim for the AR model is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of RELAY is that loop-aligned supervision lets a looped Transformer generate accurate CoT chains for problems beyond its training length, and that fine-tuning an auto-regressive CoT model on these chains improves the AR model's length generalization. The first half is supported by the hit-matrix and bit-accuracy analyses. The second half is not: in Stage II the AR model is fine-tuned on generated data covering exactly the test lengths reported in Figure 4 (Arithmetic 16-25, ED 31-40, LIS 101-120), and evaluation is performed only on these same ranges. The AR model is therefore directly trained on the target lengths; its high accuracy on them is an expected consequence of supervised learning on that distribution, not evidence of generalization beyond its original training length. The paper never evaluates the RELAY-enhanced AR model on lengths exceeding the fine-tuning range (e.g., Arithmetic 26-30, ED 41-45, LIS 121-130). Without such a test, the claim that RELAY 'enhances auto-regressive chain-of-thought' by improving length generalization is unsubstantiated: the model may simply have memorized the longer-length distribution present in the generated data. This is load-bearing because the paper's motivation is specifically the AR model's inability to extrapolate to longer reasoning chains; if the improvement does not transfer beyond the fine-tuned lengths, the framework does not solve the stated problem.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RELAY, a two-stage framework that aims to transfer the length-generalization ability of looped Transformers to standard autoregressive Chain-of-Thought (CoT) models. In Stage I, a looped Transformer is trained with an auxiliary loss that supervises each loop iteration to predict the tokens of the corresponding CoT round, aligned via a hand-crafted round-to-iteration mapping and right-aligned padding. In Stage II, this looped model generates CoT reasoning chains for problems longer than the original training length, and these chains are merged with the original training data to fine-tune an autoregressive CoT model. The paper reports experiments on three synthetic tasks (Arithmetic, Edit Distance, Longest Increasing Subsequence) showing that the looped model with CoT alignment maintains high final-answer accuracy beyond the training length and that the fine-tuned autoregressive model improves on the lengths covered by the generated data.","tokens_in":19073,"tokens_out":9864,"duration_ms":95232,"significance":"If fully substantiated, the core idea of aligning loop iterations with CoT rounds to produce interpretable, reliable reasoning chains for out-of-length problems is a novel and potentially useful bridge between looped and autoregressive architectures. The paper includes a useful hit-matrix analysis of reasoning-step quality and compares against self-generated chains filtered by final-answer verification, which is a reasonable baseline. However, the headline claim that RELAY 'enhances auto-regressive chain-of-thought' in the sense of length generalization is not yet established, because the fine-tuned autoregressive model is evaluated only on the same lengths that appear in its fine-tuning data. The evidence base is also narrow (three synthetic algorithmic tasks) for the 'general reasoner' claim. The manuscript is clearly early in development and would benefit from additional experiments and more careful framing.","major_comments":[{"comment":"The RELAY-enhanced autoregressive model is fine-tuned on generated CoT chains for problem lengths [15,25], [30,40], and [100,120] for Arithmetic, ED, and LIS, respectively, and Figure 4 evaluates accuracy only within these exact ranges. No experiment tests the fine-tuned model on longer, held-out lengths (e.g., Arithmetic 26–30, ED 41–45, LIS 121–130). Consequently, the observed improvement over the baseline may reflect direct supervised learning on the target length distribution rather than improved length generalization beyond the fine-tuning distribution. This is load-bearing because the paper's stated motivation is the autoregressive model's inability to extrapolate to longer reasoning chains. I request either (a) new experiments that evaluate the RELAY-enhanced model on lengths beyond the fine-tuning range, or (b) an explicit reframing of the contribution as 'length extension via generated data' rather than 'length generalization.'","section":"Section 4.2, Figure 4"},{"comment":"The total loss is defined as L = L_ans + λL_iter, where λ is a hyperparameter balancing final-answer supervision and intermediate CoT-token supervision. The value of λ is never reported anywhere in the paper, including in the hyperparameter tables in Appendix D.1. This omission makes the Stage I training non-reproducible and prevents readers from understanding the trade-off between answer accuracy and CoT-token accuracy. Please report the exact value(s) of λ used for each task and, ideally, a small sensitivity study over λ.","section":"Section 3.3, Eq. (10)"},{"comment":"The generation procedure for Stage II is underspecified. The looped model predicts all tokens of a reasoning round in parallel through the intermediate head P(z̃_t | e_t), but the paper does not explain how these per-round predictions are assembled into a single sequential CoT chain z, whether decoding uses argmax or sampling, or how the final answer token y is included in the generated demonstration. Without this algorithmic detail, the data-generation step cannot be reproduced, and potential inconsistencies between independently predicted rounds cannot be assessed. Please provide a precise step-by-step description of how (z, y) is obtained from the trained looped model.","section":"Section 3.3 and Section 4.2, Eq. (11)"},{"comment":"The claim that a single looped Transformer serves as a 'general reasoning engine' or 'general-purpose reasoner' is supported by only three synthetic algorithmic tasks (Arithmetic, ED, LIS), all of which require structured multi-step computation and have hand-crafted round-to-iteration alignments. This is too narrow a basis for the term 'general.' In addition, the evaluation of the autoregressive model's improvement lacks an oracle-teacher baseline fine-tuned on ground-truth CoT chains for the long problem lengths. The existing baselines 'AR-CoT + Self Chains & GT Answers' use self-generated intermediate tokens with only the final answer verified; they do not provide ground-truth CoT. Adding a ground-truth-CoT fine-tuning baseline would directly measure how much utility the RELAY-generated chains provide relative to the gold standard. Please either expand the task suite or temper the generality claims.","section":"Section 4.1 and Section 4.2"},{"comment":"All accuracy, bit-accuracy, and hit-matrix results are reported as single curves or matrices without error bars, confidence intervals, or multiple seeds. Several quantitative claims (e.g., that RELAY-enhanced CoT 'approaches and even slightly surpasses' the looped model, and that self-generated chains with GT answer filtering show only 'slight improvement') depend on small or visible differences that could vary with initialisation. Without variance estimates, the statistical significance and stability of the reported improvements cannot be assessed. Please run at least three seeds for each reported condition and report means with standard deviations or confidence intervals.","section":"Section 4.1–4.3, Figures 2, 4, 7"}],"minor_comments":[{"comment":"There are typographical issues: 'comlexity' should be 'complexity' in the paragraph after Eq. (5), and 'denote asx = ...' and 'denote asz = ...' are missing spaces between 'as' and the variable.","section":"Section 3.1"},{"comment":"The notation for the round tokens is inconsistent: the text defines z[kt:kt+1−1] but Eq. (8) uses z̃_t without explicitly connecting the two. Clarify the relationship between the round index t and the token positions k_t.","section":"Section 3.3, Eq. (8)"},{"comment":"The task weights in the multitask loss differ between the looped model with CoT alignment (all weights equal to 1) and the autoregressive CoT and vanilla looped models (weights 10 for ED and 5 for LIS). The authors should justify this discrepancy, as it may affect the comparability of the multitask results.","section":"Appendix D.1, Table 1"},{"comment":"The wording 'extensive experiments' and 'general reasoner' overstates the scope of the evaluation, which uses only three synthetic tasks. Consider adopting more measured language, such as 'experiments on three representative synthetic reasoning tasks' and 'multi-task reasoner for the studied tasks.'","section":"Abstract and Introduction"},{"comment":"The references to 'OpenAI O11' and 'Qwen QwQ 2' appear as footnotes; please include full bibliographic entries for these models in the reference list.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper is explicitly marked 'Working in Progress,' which is consistent with several missing details (λ, the Stage II generation algorithm, seeds, and error bars). The core idea is interesting and the hit-matrix analysis is a useful contribution, but the headline evaluation gap regarding lengths beyond the fine-tuning range is a substantive issue that should be addressed before the paper is considered for publication. The 'general reasoner' claim is overreaching relative to the evidence. I see no indication of circularity or improper fitting to the test set. A major revision with additional experiments and careful reframing would be appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2502.08482. First, the core idea—train a looped transformer to emit CoT-aligned intermediate tokens via right-aligned padding and masked supervision, then use its extrapolated chains to fine-tune an autoregressive model—is genuinely new and worth engaging. Second, the headline claim that this improves autoregressive length generalization is not actually tested: the AR model is fine-tuned on generated data for exactly the lengths it is then evaluated on, so the accuracy gains could be ordinary supervised learning on the target distribution, not extrapolation. That evaluation gap is load-bearing, and the paper's central motivation is exactly the AR model's inability to extrapolate.\n\nWhat's good: the alignment scheme is clever. Packing each CoT round into a fixed-length right-aligned slot with a mask that predicts the last pad token is a concrete, non-obvious way to give the looped model an interpretable, token-level output. The empirical observation that the looped model with this alignment maintains high answer accuracy on lengths beyond its training (Arithmetic ≤15 trained, up to 25 tested; ED ≤30 up to 40; LIS ≤100 up to 120) and simultaneously produces high bit-accuracy chains is a real result, and the hit-matrix comparison with AR self-generated chains (which decay badly in later steps) is a useful diagnostic. The two-stage data-generation pipeline is also sensible as a way to reduce human effort in chain synthesis.\n\nThe soft spots are mostly in the second half. The stress-test note is correct: Stage II generates chains for the same length ranges as the test set. There is no evaluation at lengths beyond those used in fine-tuning (e.g., Arithmetic 26–30, ED 41–45, LIS 121–130), so the claim of improved length generalization for the AR model is unsubstantiated. An oracle-teacher baseline—fine-tuning directly on ground-truth CoT for those long problems—would calibrate how much the RELAY chains contribute beyond simply training on longer problems. Also missing: error bars, the value of λ, and code despite the stated intent. The round-to-iteration alignment is hand-crafted per task; the paper acknowledges this, but it limits the generality claim.\n\nBottom line: the idea deserves a serious referee and a revision. The looped-side results are solid; the AR-side claim needs a held-out length test, an oracle baseline, and released code before the central conclusion can be trusted. I'd send it to review but flag the length-generalization evaluation as the make-or-break issue.","headline":"Novel loop-to-CoT alignment with a solid loop-side result, but the AR length-generalization claim is not actually tested—the fine-tuning and evaluation lengths are identical.","tokens_in":19631,"tokens_out":2856,"would_cite":false,"duration_ms":29098,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RELAY trains a looped Transformer with per-iteration CoT supervision so it can generate correct reasoning chains beyond training length, and uses those chains to fine-tune an autoregressive CoT model, improving its accuracy on longer…","keywords":["Chain-of-Thought reasoning","looped Transformer","length generalization","intermediate supervision","reasoning chain generation","autoregressive language model","synthetic reasoning data"],"falsifier":"Run Stage I on a task whose CoT chain has a natural round longer than the input problem, such as multi-digit multiplication where the partial-product round exceeds the operand token count; if the aligned looped model's generated chains show low token-level accuracy on out-of-training-length problems, the fixed-length round assumption fails.","tokens_in":18613,"feed_emoji":"🔁","tokens_out":8917,"duration_ms":82298,"temperature":0.7,"pith_summary":"Chain-of-Thought reasoning in autoregressive language models breaks down when the reasoning chain grows longer than the training sequences. This paper claims that looped Transformers, which solve problems implicitly by iterating a shared block, can be trained to emit explicit reasoning steps by aligning each loop iteration with one round of CoT and supervising that round directly. With this iteration-wise supervision, the looped model keeps its length generalization and becomes able to generate correct CoT chains for problems beyond the training length. Those generated chains are then used to fine-tune an autoregressive CoT model, which gains accuracy on longer problems without changing its architecture. If the claim holds, RELAY turns the looped architecture's length generalization into reusable training data for standard autoregressive models.","feed_headline":"Looped Transformers can write reasoning chains AR models can't","feed_subtitle":"RELAY lines up each loop with one reasoning step, preserving length generalization and creating step-by-step training data.","key_machinery":"The central mechanism is loop-aligned intermediate supervision, realized through right-aligned padding: for each iteration $t$, the ground-truth reasoning tokens of round $t$ are placed in a fixed-length sequence of the input problem length, padded on the left with $\\langle\\text{pad}\\rangle$, and a binary mask $M_t$ restricts the cross-entropy loss to valid tokens plus the last pad token. A per-iteration prediction head $P(\\tilde{z}_t \\mid e_t; \\theta_{\\text{pred-cot}})$ and a final-answer head define the total loss $L = L_{\\text{ans}} + \\lambda L_{\\text{iter}}$. This alignment is what gives the looped model the new ability to output explicit, reliable CoT tokens while retaining length generalization, and it is what makes the generated chains usable as fine-tuning data in Stage II.","core_discovery":"The central discovery is that a looped Transformer can act as a teacher for autoregressive CoT models if its iterations are forced to correspond to CoT rounds. During Stage I, each iteration $t$ is trained to predict the $t$-th round of ground-truth reasoning tokens, right-aligned to a fixed length with a mask so only valid tokens and the last pad token contribute loss; the final iteration also predicts the answer. The paper reports that this aligned looped model generalizes to longer problems on Arithmetic, Edit Distance, and LIS, and that the CoT chains it generates for those longer problems are reliable enough that fine-tuning an autoregressive model on them yields accuracy close to and sometimes above the looped teacher, while self-generated chains from the autoregressive model do not. The paper's own framing is that the bridge between the two paradigms works because the looped model's implicit iterative computation and CoT's explicit stepwise reasoning share a round structure.","pith_inferences":["The paper does not test RELAY on natural-language reasoning; a reader-visible consequence is that the method's reach is exactly the set of tasks whose CoT chains can be segmented into rounds no longer than the input problem, so failure to segment is a testable boundary.","Because Stage II is just token-level supervised fine-tuning on generated chains, RELAY is orthogonal to positional-encoding fixes for length generalization and could plausibly be combined with them, although the paper does not explore that combination.","The high token accuracy of RELAY-generated chains even beyond training length suggests a bootstrap loop — use the looped teacher to extend one length increment, retrain the autoregressive model, then repeat — that the paper does not run."],"forward_implications":["A single looped Transformer trained with RELAY can handle multiple reasoning tasks at once while keeping near-perfect accuracy on training lengths and substantially higher accuracy than the autoregressive baseline beyond training lengths.","An autoregressive CoT model fine-tuned on RELAY-generated chains can match or slightly surpass the aligned looped teacher on out-of-training-length problems, even though its architecture is unchanged.","Filtering self-generated CoT data by final-answer correctness, whether with a looped verifier or with ground truth, gives only a slight gain, so the reliability of intermediate reasoning steps is what makes generated data useful.","RELAY reduces the human effort of synthetic CoT data creation from problem construction plus solution-strategy design to mainly length-extended problem construction."],"supporting_citations":[{"why":"Introduces Chain-of-Thought prompting, the reasoning format that RELAY aligns to loop iterations and targets for enhancement.","marker":"Wei et al., 2022"},{"why":"Demonstrates looped Transformers' length generalization on RASP-L tasks, the capability RELAY builds on and preserves.","marker":"Fan et al., 2024"},{"why":"Provides the Arithmetic, Edit Distance, and LIS tasks and the theoretical view of CoT length growth that motivates the looped alternative.","marker":"Feng et al., 2024"},{"why":"Establishes looped Transformers as programmable computers, the iterative computation basis for treating loop iterations as reasoning rounds.","marker":"Giannou et al., 2023"},{"why":"Represents the step-by-step verification and synthetic data pipeline whose human effort RELAY aims to reduce.","marker":"Lightman et al., 2024"},{"why":"Supplies evidence that CoT can be unfaithful despite correct answers, motivating RELAY's per-round intermediate supervision.","marker":"Paul et al., 2024"}],"fun_headline_variants":["Looped Transformers write reasoning chains AR can't","Loop-aligned reasoning boosts auto-regressive CoT","RELAY: Loops as stepwise teachers for AR models","Looped Transformers generate CoT data for AR fine-tuning","Aligning loop iterations with CoT steps helps AR models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes every reasoning problem can be split into as many rounds as the looped model has iterations, with each round short enough to fit in the original problem's token length, and that looped Transformers actually extrapolate to longer problems.","fun_headline_variants_meta":{"raw":{"variants":["Looped Transformers write reasoning chains AR can't","Loop-aligned reasoning boosts auto-regressive CoT","RELAY: Loops as stepwise teachers for AR models","Looped Transformers generate CoT data for AR fine-tuning","Aligning loop iterations with CoT steps helps AR models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000932,"raw_usage":{"total_tokens":3991,"prompt_tokens":949,"completion_tokens":3042,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":2961}},"tokens_in":565,"tokens_out":3042,"duration_ms":23356,"temperature":1.0,"reasoning_tokens":2961,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T04:55:35.817931+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Stage I on a task whose CoT chain has a natural round longer than the input problem, such as multi-digit multiplication where the partial-product round exceeds the operand token count; if the aligned looped model's generated chains show low token-level accuracy on out-of-training-length problems, the fixed-length round assumption fails.","supporting_citations":[],"review_version":1}