{"id":"21f16750-a9be-4a95-8cba-82629aa9c38b","arxiv_id":"2604.00004","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LinearARD restores RoPE-scaled LLMs by exact linear-memory KL distillation of Q/Q, K/K, and V/V self-relations, reaching ~95% short-text recovery with 4.25M tokens.","lead":"LinearARD is a self-distillation method that repairs LLMs after context-window extension by aligning Q/Q, K/K, and V/V attention-relation distributions with a frozen teacher, using a linear-memory kernel. It reports LLaMA2-7B short-text recovery of 94.8% and RULER 63.2 with 4.25M training tokens, versus 256M for continued pre-training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 60x token-efficiency claim lacks same-token-budget baselines; with cross-budget comparison only, 'LongReD/CPT require 256M' is unsupported. Mismatched CPT ablation (Table 3 PI4x vs Table 4 PI8x) further clouds attribution.","rationale":"The reader's weakest_assumption points to the theoretical sufficiency of Q/Q, K/K, V/V row-wise relation supervision. That is a plausible concern, but the empirical design has a more immediately load-bearing gap: the headline efficiency claim is a cross-budget comparison. The paper never shows that LongReD or CPT fails at 4.25M tokens, nor that LinearARD's advantage is robust to equalizing token budgets. This is directly analogous to comparing a method at one compute budget and claiming other methods 'require' a larger budget without measuring them at the smaller budget. The internal mismatch between Table 3 (PI4x) and Table 4 (PI8x) further weakens the ablation logic supporting the two-stage recipe. I agree with the reader that the paper is promising and the kernel's linear-memory/exactness claims are well supported by Propositions 3.2/3.3 and Table 8, so no rejection is warranted. But the central efficiency claim needs a matched-budget control, so the conditional verdict should stand. I set verdict_should_be to UNCHANGED because my concern does not move the verdict away from the reader's CONDITIONAL; it reinforces it.","tokens_in":18848,"tokens_out":4846,"duration_ms":41805,"concrete_test":"Run LongReD and CPT on LLaMA2-7B PI8x with exactly the LinearARD token budget (4.25M tokens, including the same 2M-token CPT stage where applicable), using the same data, optimizer, and context-length schedule. Report the eight-task short-text average and RULER at 8K/16K/32K. If either baseline reaches RULER above 60 or Avg. recovery above 94%, the '60x fewer tokens' claim collapses. In addition, run LinearARD with the same 256M token budget as the baselines; if its advantage does not grow or reverses, the comparison is confounded by compute rather than method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that LinearARD restores short-context accuracy and long-context RULER using 4.25M tokens versus 256M for CPT and LongReD. This is a comparative efficiency claim, but Table 1 only compares LinearARD at 4.25M against baselines at 256M. There is no matched-budget experiment: LongReD and CPT are never run with 4.25M tokens, and LinearARD is never run with 256M. 'Required by LongReD and CPT' is therefore asserted, not demonstrated. If LongReD or CPT reaches comparable accuracy at, say, 8M or even 4.25M tokens, the '60x fewer' headline loses its force. The claim also conflates method efficacy with total compute because the reported LinearARD results include a 2M-token CPT stage; Table 5 shows RULER drops from 63.2 to 36.8 without that CPT stage, so the long-context headline is substantially produced by the CPT component rather than by relation distillation alone. A further internal inconsistency weakens the attribution: the CPT ablation for short-text (Table 3) is run with PI4x, while the long-context ablation (Table 4) and the main LLaMA2-7B result use PI8x. The conclusion that 'CPT is negligible for short-text' is therefore not established for the actual main setting. These gaps are fixable, but until they are addressed the central efficiency comparison is not secure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LinearARD, a self-distillation method that restores short-context performance of RoPE-scaled LLMs by aligning row-wise softmax distributions of Q/Q, K/K, and V/V self-relation matrices with those of a frozen native-RoPE teacher. To make this feasible at long sequence lengths, it introduces an IO-aware tiled kernel that computes the exact KL divergence and gradients with O(n) memory instead of O(n^2). The method is evaluated on LLaMA2-7B, LLaMA3-8B, and Mistral-7B-v0.1 extended to 32K context, with short-text benchmarks (MMLU, LAMBADA, BoolQ, etc.) and RULER. The paper claims that LinearARD achieves comparable or better restoration than CPT and LongReD using 4.25M training tokens versus 256M, roughly a 60x reduction, and provides theoretical proofs of linear memory and gradient exactness.","tokens_in":19275,"tokens_out":5008,"duration_ms":49636,"significance":"If the empirical claims hold, this is a practically valuable contribution: it makes dense structural attention distillation feasible at sequence lengths where materializing n x n relation maps is prohibitive, and it offers a concrete, theoretically grounded alternative to hidden-state distillation for RoPE restoration. The linear-memory KL kernel is elegant, with a clear proof of exactness and empirical numerical verification, and the code is released. The two-stage recipe (relation distillation followed by a light CPT stage) is well motivated and the ablations isolate the contribution of the kernel and loss terms. However, the central token-efficiency claim is currently supported only by cross-budget comparisons, and one ablation uses mismatched RoPE scaling settings, so the significance is conditional on additional experiments.","major_comments":[{"comment":"The headline claim of a 60x token reduction is not supported by a same-token-budget baseline. Table 1 compares LinearARD at 4.25M tokens only against CPT and LongReD at 256M tokens. No experiment shows what CPT or LongReD achieve at 4.25M tokens, nor what LinearARD would achieve at larger budgets. The statement that LongReD and CPT 'require' 256M tokens is asserted, not demonstrated. A matched-budget comparison (or at least a token-budget learning curve) is needed to substantiate the efficiency claim.","section":"§4.1, Table 1, Appendix A.1"},{"comment":"The CPT ablation is internally inconsistent with the main setting. Table 3, which shows that CPT has negligible short-text effect, is run on LLaMA2-7B with PI4x, while Table 4 and the main Table 1 use PI8x. Therefore the conclusion that ARD alone is sufficient for short-text restoration is not established in the exact configuration used for the headline result. Furthermore, Table 4 shows RULER dropping from 63.2 to 36.8 without the CPT stage, so the long-context improvement is substantially produced by the 2M-token CPT component. The attribution of long-context gains to relation distillation versus CPT needs a same-scaling-factor ablation.","section":"§4.3.2, Tables 3 and 4"},{"comment":"All reported results are single runs without error bars or significance tests. Some differences that the text interprets are small: on Mistral-7B-v0.1, LinearARD's RULER is 60.8 vs. LongReD's 62.3 and short-text Avg is 57.91 vs. 59.38. Without variance estimates, the claims of 'competitive' or 'surpassing' performance are difficult to evaluate, especially at the 8K/16K lengths where differences are smaller. Reporting multiple seeds or at least bootstrap confidence intervals would substantially strengthen the comparative claims.","section":"§4.2, Table 1"},{"comment":"The inclusion of V/V relation distillation is a distinctive design choice, but the rationale is not fully developed. V/V relations do not enter the attention logits (which use QK^T), so aligning V/V is an extra assumption about value-feature geometry. The only evidence is the Table 2 ablation, which is on a different architecture (Qwen3-0.6B) and shows a small drop from 43.7 to 43.4 without V-rel. The paper should either provide a more direct analysis of why V/V alignment is necessary for RoPE restoration or acknowledge that this term is an empirical heuristic whose transferability across models is untested.","section":"§3.2, Eq. (4c), Table 2"}],"minor_comments":[{"comment":"The abstract states 'recovers 98.3% of the short-text performance of state-of-the-art baselines.' This figure corresponds only to LLaMA2-7B in Table 1 (51.87/52.75); for LLaMA3 and Mistral the ratios are 97.2% and 97.5%. The claim should be qualified.","section":"Abstract"},{"comment":"The token-budget formula should specify whether L_s is the number of valid tokens per sample or the context length with padding, and how gradient accumulation interacts with the per-GPU batch size. The decomposition of 4.25M into the distillation and CPT stages is not shown, which makes it hard to verify the accounting.","section":"Appendix A.1, Eq. (8)"},{"comment":"The training description says 'only the attention-module Q/K/V projection weights are updated' but also mentions optional LoRA/QLoRA. It is unclear which configuration produced Table 1. Please state explicitly for each experiment whether LoRA was used and what rank.","section":"§4.1"},{"comment":"The main experiments use LLaMA2, LLaMA3, and Mistral, but the ARD component ablation is on Qwen3-0.6B. A sentence explaining why this smaller model was chosen, and whether the conclusions are expected to transfer, would be helpful.","section":"Table 2 caption"},{"comment":"The correlational training-dynamics plot is interpreted as 'strong causal link.' Correlations alone do not establish causation; consider adding an intervention (e.g., early stopping, gradient norm analysis) or softening the causal language.","section":"§A.8.1, Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The kernel contribution and the core idea are solid, and the paper is well within scope for a CL/ML venue. The main risk is the unsupported token-efficiency claim, which is the paper's headline. I would like to see matched-budget baselines and a corrected, same-scaling-factor CPT ablation. If those experiments confirm the current numbers, the paper would be a strong accept. If not, the contribution would still be useful but the claims would need substantial rescaling."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading for the kernel alone. The key identity—the KL gradient for a row-wise softmax is just the difference of the two probabilities—lets you recompute logits tile-by-tile and never materialize the n×n matrix. The tiled algorithm is a straightforward extension of FlashAttention, and their proof of linear memory and numerical exactness is fine. The verification table shows errors at floating-point level. That part is genuinely useful.\n\nThe application to RoPE restoration is new, and the design is sensible: align Q/Q, K/K, and V/V row distributions with a frozen native-RoPE teacher, then run a short CPT. The ablation is informative—without the CPT stage, RULER drops from 63.2 to 36.8 on LLaMA2, so the relation distillation is doing the short-context restoration and CPT is doing the long-context adaptation. This division of labor is the paper's best empirical insight.\n\nHowever, the headline claim that LinearARD needs roughly 60x fewer tokens than CPT and LongReD is not supported by the experiments as run. You compare your method at 4.25M tokens against those baselines at 256M, but you never run the baselines at 4.25M or LinearARD at 256M. 'Required by LongReD and CPT' is therefore asserted, not demonstrated. A cheap CPT might get most of the way there; we don't know.\n\nThere are two concrete internal issues. First, the CPT ablation uses PI4x for short-text (Table 3) and PI8x for long-context (Table 4), so the conclusion 'CPT is negligible for short-text' is only shown for the 4x setting, not the main 8x setting. Second, the abstract says 'surpassing them on long-context benchmarks,' but on Mistral you trail LongReD (60.8 vs 62.3) and on LLaMA3 you trail CPT (68.3 vs 81.3). The text admits the latter, but the abstract overreaches. Also: single runs, no error bars, and the training corpus is not stated in the main setup.\n\nNone of this kills the paper. The kernel is a solid contribution, the method is plausible, and the short/long division of labor is interesting. But the token-efficiency claim needs to be re-framed or backed by matched-budget experiments. Send it to a careful reviewer and ask for that.","headline":"Solid linear-memory KL kernel and plausible distillation recipe, but the 60x token-efficiency claim is not actually demonstrated by the experiments.","tokens_in":19704,"tokens_out":3526,"would_cite":true,"duration_ms":32230,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RoPE scaling breaks short-text accuracy; LinearARD restores it using 4.25M tokens (60x fewer than baselines) by aligning Q/Q, K/K, and V/V attention-relation distributions.","keywords":["RoPE scaling","context window extension","attention distillation","self-relation matrices","linear-memory kernel","KL divergence","continued pre-training","long-context LLMs"],"falsifier":"Take a new backbone/scaling pair not used in the paper (e.g., a 1B model with YaRN 4x) and run LinearARD with the V/V term ablated. If short-text recovery stays at the full-method level, the claimed holism of Q/Q-K/K-V/V alignment is wrong. Conversely, if a student whose Q/Q and K/K match the teacher but whose V/V is deliberately misaligned (e.g., value projections frozen at the scaled initialization) shows a clear accuracy drop, the claim survives.","tokens_in":18751,"feed_emoji":"⚡","tokens_out":7282,"duration_ms":61596,"temperature":0.7,"pith_summary":"Scaling a pretrained model's rotary position embeddings (RoPE) to extend its context window disrupts the attention geometry learned during pretraining, and standard fixes such as continued pre-training consume hundreds of millions of tokens. LinearARD proposes that this damage can be repaired far more cheaply by treating the original model as a frozen teacher and forcing the scaled student to reproduce the teacher's internal row-wise attention-relation distributions — the softmax-normalized Q/Q, K/K, and V/V similarity matrices inside every attention head. The paper introduces an exact linear-memory kernel that computes the KL divergence between these distributions and its gradients without materializing any n×n matrix, removing the quadratic-memory barrier that previously limited attention distillation to short sequences. On three 7B-8B backbones extended to 32K, LinearARD recovers about 95% of native short-text accuracy using 4.25M training tokens — roughly 60x fewer than prior restoration baselines — while matching or exceeding their long-context RULER scores, especially at 32K. The point of the paper, if right, is that restoring a scaled model is mostly a geometry-repair problem: align the internal relational structure and the output-level behavior comes back.","feed_headline":"4.25M tokens restore scaled LLMs to 95% accuracy","feed_subtitle":"Aligning Q/Q, K/K, and V/V attention relations beats 256M-token baselines on long-context RULER — with a linear-memory kernel.","key_machinery":"The central object is the masked row-wise relation distribution R_m = softmax(X_m X_m^T/√d + M) for X ∈ {Q,K,V}; distillation aligns teacher and student rows by forward KL, L = (1/n) Σ_i D_KL(R_t(i,:) ∥ R_s(i,:)). The load-bearing identity is ∂L/∂Z_s(i,j) = (R_s(i,j) − R_t(i,j))/n, which lets each gradient entry be formed locally. The kernel exploits this in a two-pass tiled scheme: Phase 1 stores only the per-row log-sum-exp statistics LSE_m(i) = log Σ_k exp(Z_m(i,k)) (O(n) memory); Phase 2 recomputes masked logits tile by tile in on-chip memory, reconstructs R_s and R_t from the precomputed LSEs, accumulates the KL loss, and streams gradients to the student's Q/K/V projections. This makes","core_discovery":"The paper's central claim is that the short-context degradation caused by RoPE scaling is a fine-grained positional-geometry distortion inside attention, and that it can be corrected by distilling the row-wise distributions of three dense self-relation matrices — R_Q = softmax(QQ^T/√d + M), R_K = softmax(KK^T/√d + M), and R_V = softmax(VV^T/√d + M) — from the frozen native-RoPE teacher to the scaled student using forward KL. Because the gradient of this KL with respect to a student logit is simply R_s(i,j) − R_t(i,j), the loss can be accumulated tile by tile: precompute per-row log-sum-exp statistics, recompute logits in the backward pass, and reconstruct probabilities on-chip. The paper pro","pith_inferences":["Editorial inference: the V/V constraint is the least theoretically motivated piece — V/V never enters the attention logits — so its contribution likely comes from regularizing what features the value pathway carries. A direct test would be to run the method on a backbone where V/V is omitted; the paper's own ablation shows a small but real drop, so the prediction is that V/V matters most when valu","Editorial inference: the linear-memory KL kernel is a general-purpose primitive, not tied to QKV relations. It should apply to any row-wise distribution-matching objective — attention-map distillation, logit KL, or even non-attention softmax layers — so the method's scope may be broader than context restoration.","Editorial inference: the 60x token reduction suggests the dominant cost of restoration is geometric recalibration, not knowledge acquisition. If that is right, the same budget should transfer to larger scaling factors (64K, 128K) and to newer positional schemes; the paper does not test those regimes.","Editorial inference: because the teacher is evaluated only within its native context length, the method cannot teach the student behaviors that live beyond the teacher's range; the optional continued-pretraining stage is doing that work, so the two stages are complementary rather than interchangeable."],"forward_implications":["Context extension becomes roughly 60x cheaper: restoring short-text behavior needs 4.25M tokens instead of 256M, so long-context deployment no longer requires a large high-quality long-context corpus.","Attention-structure consistency is sufficient supervision: because output-logit divergence falls in tandem with the relation KL during training, explicit logit-level distillation is unnecessary for restoration.","Dense, exact distribution matching is now practical at long contexts, retiring the need for sparse or selective attention-supervision objectives that sacrifice fidelity.","The restoration transfers across backbones and scaling factors: 4x and 8x RoPE scaling on three 7B-8B models all recover ~95% of native short-text accuracy, with the largest long-context gains at 32K.","A short continued-pretraining stage remains necessary for long-context activation: ARD alone restores short-text performance but leaves RULER low (36.8), while adding 2M tokens of continued pre-training raises it to 63.2."],"fun_headline_variants":["60x fewer tokens: distill attention relations to restore scaled LLMs","Linear-memory distillation restores RoPE-scaled LLMs with 60x less data","Distill self-relations, not hidden states: restore scaled LLMs with 4.25M tokens","RoPE-scaled LLMs restored via attention self-relations: 60x fewer tokens","Exact KL attention distillation: 4.25M tokens beat 256M on LLM restoration"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that matching the row-wise softmax distributions of Q/Q, K/K, and V/V self-relations is a complete and sufficient supervision signal to restore the student's short-context behavior — in particular that the V/V constraint, which never appears in the attention logits, is necessary.","fun_headline_variants_meta":{"raw":{"variants":["60x fewer tokens: distill attention relations to restore scaled LLMs","Linear-memory distillation restores RoPE-scaled LLMs with 60x less data","Distill self-relations, not hidden states: restore scaled LLMs with 4.25M tokens","RoPE-scaled LLMs restored via attention self-relations: 60x fewer tokens","Exact KL attention distillation: 4.25M tokens beat 256M on LLM restoration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001114,"raw_usage":{"total_tokens":4523,"prompt_tokens":841,"completion_tokens":3682,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":3564}},"tokens_in":585,"tokens_out":3682,"duration_ms":23244,"temperature":1.0,"reasoning_tokens":3564,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T18:32:55.478578+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a new backbone/scaling pair not used in the paper (e.g., a 1B model with YaRN 4x) and run LinearARD with the V/V term ablated. If short-text recovery stays at the full-method level, the claimed holism of Q/Q-K/K-V/V alignment is wrong. Conversely, if a student whose Q/Q and K/K match the teacher but whose V/V is deliberately misaligned (e.g., value projections frozen at the scaled initialization) shows a clear accuracy drop, the claim survives.","supporting_citations":[],"review_version":1}