{"id":"dc45895b-8fd6-4caf-acb7-926d449a8579","arxiv_id":"2506.09251","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Length generalization on a short-trained main task can be inherited from a longer-trained related auxiliary task trained jointly with it.","lead":"Transformers trained on a longer related task can learn to solve a different, shorter-trained task at lengths they never saw. The result suggests multitask training and language pretraining build reusable skills that transfer to out-of-distribution inputs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main figures plot only the best of 5 seeds while the paper documents high seed-to-seed variance, so the headline transfer effect may reflect selection rather than a typical training outcome.","rationale":"The paper's central claim is an existential one, but the way the evidence is presented matters. The reported curves are best-of-5 runs, and the paper itself documents instability. This makes the headline plots unrepresentative. I considered whether the more conceptual concern that transfer is due to input-format sharing rather than algorithmic structure is more important; the copy-first-op and reverse controls already speak to that, reducing its force. The seed selection issue, by contrast, directly affects the magnitude and reproducibility of every reported positive result. A test that reports all seeds or medians would settle it. I also note the paper's strengths: the control pairs, the length-ratio heatmap, the pretraining checkpoint trend, and the circuit-sharing correlations are all real evidence, and the task pairs are carefully constructed. Those strengths do not remove the need to compare distributions rather than order statistics. The reader's conditional verdict remains appropriate; my concern does not move it but sharpens the required revision.","tokens_in":17389,"tokens_out":8193,"duration_ms":94815,"concrete_test":"For the reverse add + {no carry, carry only} task group (Figures 3a and 11), retrain with the same hyperparameters and report the full set of 5 seed curves for co-trained and main-only conditions, or at minimum the mean/median and per-seed accuracies at lengths 17-32 and 33-40. Compute the fraction of seeds with main-task accuracy above 50% at length 32 under each condition, and test whether the median co-trained accuracy exceeds the median main-only accuracy by a margin larger than the seed noise. If it does not, the transfer effect is an artifact of best-seed selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline evidence for length generalization transfer rests on runs selected after the fact. The captions of Figures 3-7 state that each model is trained with 5 seeds and the 'best-performing runs are shown in bold'; Section 6.2 then reports that 'not all random seeds exhibit successful length generalization transfer' and shows large run-to-run variance (Figure 11). Thus the plotted extrapolation curves may be the top order statistic of a noisy process, not a typical or reproducible outcome. If transfer succeeds in only a minority of seeds, the abstract's 'can be transferred' remains existentially true but the practical claim that co-training with a related longer task leads to length generalization is not supported. The single-task baselines are presumably also selected by the same best-seed rule, so the comparison is not unbiased; it compares the best of 5 in each condition, and with high variance the best co-trained run is likely to beat the best single-task run even if the underlying distributions overlap. Since the central quantitative evidence is this comparison, the selection rule is a load-bearing assumption that needs to be tested.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether length generalization can transfer across related tasks in transformers. In their setup, a 'main' task is trained only on short inputs while a related 'auxiliary' task is trained on longer inputs; the authors report that joint training lets the main task extrapolate to lengths it never saw, often matching the auxiliary task's generalization range. The phenomenon is demonstrated on three task families (reverse addition with related arithmetic auxiliaries, string transformations, and maze navigation), with two control pairs showing no transfer when tasks are unrelated. The authors further report that finetuning SmolLM-360M checkpoints at increasing pretraining steps improves OOD accuracy on reverse add and shortest path, and they provide attention-based analyses suggesting that transfer correlates with shared attention-head usage across tasks.","tokens_in":17582,"tokens_out":5819,"duration_ms":63810,"significance":"If the central claim holds, the paper identifies a new route to length generalization: rather than architectural changes or data augmentation, a model can inherit extrapolation behavior from a longer, related training task. The study is well positioned in the length-generalization literature and has practical implications for multitask training and for understanding why pretrained models generalize. Strengths include on-the-fly data generation with exact-match evaluation, three diverse task families, control pairs, public pretrained checkpoints, and an explicit acknowledgment of training instability. The main weaknesses are the best-seed presentation of the headline curves, the absence of an equivalent-compute baseline for the pretraining trend, and the informal nature of the mechanistic correlation analysis. The phenomenon is plausible and interesting, but the current evidence does not yet establish how typical the transfer effect is.","major_comments":[{"comment":"The main quantitative evidence is presented through best-of-five-seed curves, and this selection rule is load-bearing for the central claim. The captions of Figures 3–7 state that 'best-performing runs are shown in bold,' while Section 6.2 states that 'not all random seeds exhibit successful length generalization transfer' and Figure 11 documents substantial run-to-run variability. Because both the co-trained conditions and the single-task baselines are selected by the same rule, the headline comparison can reflect the top order statistic of a noisy process rather than a typical training outcome. Please report the full seed distribution (individual runs, or median with spread) for every condition, quantify the fraction of seeds that achieve transfer in each task group, and state explicitly whether the conclusion holds for median runs as well as best runs.","section":"Section 6.2 and Figures 3–7"},{"comment":"The pretraining evidence is a checkpoint trend without a matched equivalent-compute baseline. Figure 9 shows that later SmolLM checkpoints finetune to higher OOD accuracy on reverse add and shortest path, but there is no from-scratch, randomly initialized, or early-step model finetuned for the same number of steps and with the same data. Without such a control, the monotonic improvement could reflect differences in finetuning dynamics or optimization landscapes rather than reusable scaffolding acquired during pretraining. Please add an equivalent-compute from-scratch baseline and report seed variation for the pretrained-model experiments, or substantially soften the causal wording in Section 5.","section":"Section 5 and Figure 9"},{"comment":"The mechanistic claim is stated more strongly than the evidence supports. The text says the attention similarity metrics correlate with the generalization gap 'in most cases,' but Figure 18 shows that the raw attention-matrix difference does not track the gap for the string task pairs, while the head-ablation-map difference does. Figure 19 also shows mixed behavior for the control pair. The analysis is an informal visual comparison across checkpoints without correlation coefficients or statistical summaries. Please either report quantitative correlations for the measures that actually track the gap, or explicitly present the circuit-sharing results as a qualitative, initial observation rather than a confirmed correlation.","section":"Section 6.4 and Figures 18–19"}],"minor_comments":[{"comment":"Please define the criterion for 'best-performing runs' and clarify that the bold curves correspond to a single selected seed; this information is essential for interpreting the plots.","section":"Figure captions (Figures 3–7)"},{"comment":"The generalization gap is described only in the caption; please give an explicit formula in the main text and label the axes of the heatmap clearly so the reader knows whether rows or columns correspond to main versus auxiliary length.","section":"Section 6.1 and Figure 10"},{"comment":"Please clarify the precise difference from task hinting [Awasthi and Gupta, 2023], which also reports improved length generalization from auxiliary tasks; the current text does not explain why the longer-auxiliary setup here is a distinct phenomenon.","section":"Section 2, Related Work"},{"comment":"There is a typo: 'or arithmetic' should be 'For arithmetic,' and 'with unrelated task' should agree in number with the plural examples ('unrelated tasks').","section":"Section 4.4"},{"comment":"The phrase 'significantly weaker transfer' is not supported by a statistical test; please use 'substantially weaker' or provide a significance test for the RoPE versus NoPE comparison.","section":"Section 6.3"},{"comment":"The notation 'n×3CoT multiply' is confusing because the example uses a three-digit multiplier (502); please define the notation or rename the task to clarify the length parameter.","section":"Appendix B.2, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for the journal and the phenomenon is plausibly real, but the best-seed presentation is a genuine threat to the headline claim. If the authors can show that the transfer holds for median seeds or quantify a high success rate across seeds, I would support acceptance after a revision. I do not see grounds for rejection, provided the seed-distribution issue and the pretraining-control issue are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper shows that co-training a main task on short inputs with a related auxiliary task on longer inputs can push the main task's extrapolation range out to the auxiliary's — but the headline curves are the best of five seeds, and the paper's own Section 6.2 shows that not all seeds transfer. So read it as an existence proof with a robustness caveat, not as a recipe.\n\nWhat's actually new: Awasthi and Gupta's task hinting used simpler auxiliary tasks at similar lengths. Here the auxiliary is trained longer, and the main task inherits the longer range. That asymmetry is the new bit. They show it in three task families (arithmetic, string, maze), with two control pairs (unrelated tasks) that fail to transfer. They also finetune SmolLM checkpoints and see a trend: later pretraining checkpoints generalize better. The limitations section is honest, and the experimental details are complete enough to reimplement. The mechanistic part (attention head ablation maps) is correlational but reasonable. Credit where due: this is a real phenomenon, and the paper doesn't hide its weakness.\n\nThe soft spots: (1) Best-of-seed reporting is load-bearing. The captions say 'best-performing runs are shown in bold'; Section 6.2 documents high run-to-run variance. With only 5 seeds, the top order statistic can beat the single-task top order statistic even if typical co-trained runs are no better. The paper never reports seed-averaged curves or the fraction of seeds that transfer. This is the main fix needed. (2) The pretraining evidence is a checkpoint trend with no compute-matched from-scratch baseline, so 'pretraining helps' is suggestive, not established. (3) The mechanistic metrics don't align for string tasks on raw attention; only the ablation map difference tracks. That's okay for an 'initial evidence' claim, but don't over-read it. (4) The generality of 'relatedness' rests on only two control pairs.\n\nThe central argument holds up as an existence proof. It does not yet show that co-training reliably produces transfer.\n\nWho it's for: people working on length generalization, multitask training, and mechanistic interpretability of arithmetic tasks. It deserves a serious referee — send it out — but the revision should include seed-averaged results and a success-rate analysis.","headline":"Genuine new phenomenon, but the best-of-five-seed reporting makes the headline effect look stronger than the data support.","tokens_in":18094,"tokens_out":3066,"would_cite":true,"duration_ms":31125,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Co-training with a longer related task transfers length generalization to a short main task.","keywords":["length generalization","task transfer","multitask training","transformer interpretability","attention head sharing","out-of-distribution generalization","pretrained language models","algorithmic tasks"],"falsifier":"Pair reverse add (short, length 16) with a longer auxiliary task that is algorithmically related but uses a different surface format—for example, addition in a different numeral base—and check whether the main task still extrapolates; if it does not, the phenomenon depends on format overlap rather than structural relatedness, and if it does, the relatedness claim generalizes beyond the paper's exact formats.","tokens_in":17216,"feed_emoji":"🔁","tokens_out":8604,"duration_ms":78703,"temperature":0.7,"pith_summary":"This paper sets out to show that length generalization—a transformer's ability to handle inputs beyond its training lengths—can be passed from one task to another. The authors train a main task on short examples together with an auxiliary task on longer examples and report that the main task then succeeds at lengths it never saw, inheriting the auxiliary task's generalization range. This transfer appears in arithmetic, string manipulation, and maze navigation. The paper also finds that natural-language pretraining acts like an implicit auxiliary task: later pretraining checkpoints finetune into better length generalization on synthetic tasks. A control result anchors the claim: unrelated task pairs such as reverse add with copy-first-op show no transfer, suggesting relatedness is what carries the effect.","feed_headline":"Co-training transfers length generalization across tasks","feed_subtitle":"Pairing a short task with a longer related one extends the model's range; unrelated pairs do not.","key_machinery":"The carrying object is task association realized through joint multitask training: at each step a task is sampled uniformly from a group containing a short main task and longer auxiliary tasks, so the model is forced to learn the auxiliary task at lengths the main task has never seen. The quantitative instrument is the generalization gap, the average difference between the main and auxiliary accuracy-versus-length curves normalized to [0,1], with smaller values meaning stronger transfer; the paper shows transfer is strongest when the auxiliary-to-main training-length ratio lies between 0.5 and 2. The mechanistic instrument is the attention-head mean-ablation map, a matrix recording the accuracy drop when each head's output is replaced by its mean activation; similarity of these maps between tasks tracks the generalization gap over training, identifying shared head usage as the correlate of transfer. RoPE, the rotary position encoding, is the positional scheme whose presence the paper finds supports transfer, with NoPE giving weaker transfer under the same task settings.","core_discovery":"The paper's discovery is that extrapolation by association is real in transformers: co-training a task at a short length with a related task at a longer length makes the short task extrapolate to the longer task's range, even though the main task never saw examples at those lengths. The authors demonstrate this in three domains—reverse addition inheriting range from no-carry, carry-only, reverse-subtract, or chain-of-thought multiplication; string copy inheriting from multi-query associative recall; capitalize-reverse inheriting from capitalize and reverse; and maze DFS-trace and shortest-path, where transfer works in both directions. They further find that pretraining on natural language provides the same kind of transfer, with extrapolation improving steadily as pretraining progresses, and that rotary position encodings produce the effect more strongly than no positional encoding. Mechanistically, the paper reports that successful transfer coincides with the two tasks sharing attention heads: the per-head mean-ablation importance maps of the tasks converge as the generalization gap closes, while unrelated control pairs do not show this convergence. This is offered as evidence that length generalization transfer corresponds to reuse of the same internal computation across tasks.","pith_inferences":["If shared attention circuits are the mechanism, transfer should occur between tasks that share a subprocedure but differ in surface format; this could be tested by pairing reverse addition with digit-wise addition in another base or with a different delimiter format.","The length-ratio band of 0.5–2 suggests a curriculum strategy the authors do not explicitly propose: to reach very long lengths, one could stage auxiliary tasks, each new task within a factor of two of the previous, rather than jumping directly from short to very long.","The seed-to-seed instability the paper reports implies that in practice transfer is not guaranteed by the training setup alone; deployment would likely require multiple seeds or a checkpoint-selection rule, which the paper does not provide.","The pretraining result suggests a testable scaling prediction: models pretrained on corpora with more long-range structure should show progressively better downstream length generalization on synthetic tasks, even when the tasks share no surface format with the pretraining data."],"forward_implications":["A practical recipe for improving length generalization is to co-train a short target task with a related task at the desired longer length, rather than changing the model architecture.","Task relatedness is a necessary condition for the effect: unrelated auxiliary tasks should not be expected to extend a main task's generalization range.","Pretrained language models carry reusable computational scaffolding from natural language that transfers to novel synthetic tasks, so finetuning a later pretraining checkpoint should yield better extrapolation.","Attention-head mean-ablation similarity can serve as an observable signature that transfer is occurring, allowing early detection or monitoring of successful multitask length generalization.","The auxiliary task's length should be within roughly a factor of two of the main task's training length; beyond that ratio, the model tends to overfit the length difference and transfer weakens."],"supporting_citations":[{"why":"Supplies the reversed-order reverse add task format that serves as the main arithmetic task.","marker":"Lee et al. [2023]"},{"why":"Establishes the training-short-test-long length generalization setting this paper extends.","marker":"Zhou et al. [2023]"},{"why":"Closely related task-hinting result that training with simpler auxiliary tasks improves length generalization.","marker":"Awasthi and Gupta [2023]"},{"why":"Defines the MQAR task used as the auxiliary string task for copy.","marker":"Arora et al. [2023]"},{"why":"Provides SmolLM-360M and its intermediate pretraining checkpoints used to show pretraining acts as an implicit auxiliary task.","marker":"Allal et al. [2024]"},{"why":"Prior finding that NoPE aids length generalization, which the paper contrasts with its observation that RoPE supports transfer.","marker":"Kazemnejad et al. [2024]"},{"why":"Introduces RoPE, the positional encoding whose presence the paper finds promotes length generalization transfer.","marker":"Su et al. [2023]"},{"why":"Provides the activation-patching methodology behind the mean-ablation head importance maps used for mechanistic evidence.","marker":"Wang et al. [2022]"}],"fun_headline_variants":["Short task gains long range from related longer task","Co-training with longer task extends extrapolation","Borrowing length generalization from a related task","Transferring extrapolation: short learns from long","How transformers share length extrapolation skills"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume the observed improvement comes from transferring algorithmic structure between related tasks, not from the auxiliary task sharing superficial input-format features, and this assumption rests on only two unrelated control pairs.","fun_headline_variants_meta":{"raw":{"variants":["Short task gains long range from related longer task","Co-training with longer task extends extrapolation","Borrowing length generalization from a related task","Transferring extrapolation: short learns from long","How transformers share length extrapolation skills"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000147,"raw_usage":{"total_tokens":1195,"prompt_tokens":966,"completion_tokens":229,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":161}},"tokens_in":582,"tokens_out":229,"duration_ms":3179,"temperature":1.0,"reasoning_tokens":161,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:52:54.290965+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pair reverse add (short, length 16) with a longer auxiliary task that is algorithmically related but uses a different surface format—for example, addition in a different numeral base—and check whether the main task still extrapolates; if it does not, the phenomenon depends on format overlap rather than structural relatedness, and if it does, the relatedness claim generalizes beyond the paper's exact formats.","supporting_citations":[],"review_version":1}