Pith. sign in

REVIEW 3 major objections 5 minor 26 references

Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A language model converted from full attention to KDA linear attention can look healthy on perplexity while sticking to option label 'A' on 81% of multiple-choice questions; the paper diagnoses this as a repairable interface injury.

desk verdict A genuinely useful four-permutation probe for linearization failure, well-documented and honest, but the headline mechanism claim (interface injury) rests on an uncontrolled repair stage. read the letter →

arxiv 2608.02689 v1 pith:SZIRWIBG submitted 2026-08-03 cs.CL cs.LG

Ronglong Bao This is my paper
classification cs.CLcs.LG
keywords linearattentionKDAconversionknowledgedistillationinterfaceinjurymultiple-choiceevaluationlabelstickingmodellinearizationC-Eval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what a large-scale architecture conversion actually damages, and finds that the standard metric stack — perplexity, hidden-state alignment, end-to-end KL — can look healthy while the model has lost the ability to map knowledge onto answer options. Converting 21 of 28 attention layers of a 0.6B model into KDA linear-attention layers on a single consumer GPU leaves the student predicting option A 81% of the time on a four-rotation multiple-choice diagnostic, with the correct-option margin negative. The paper's central claim is that this is an interface injury — a broken link between retained knowledge and the MCQ format — rather than uniform knowledge loss, and that a 1,000-step completion-only KL stage substantially repairs it, lifting C-Eval from 28.8% to 41.3% and halving label stickiness. The paper is careful to state that zero knowledge loss is not proven; the repair is offered as strong supporting evidence that most benchmark-relevant knowledge survived.

What carries the argument

The central object is the four-permutation diagnostic: each multiple-choice question is scored under all four cyclic rotations of its options, yielding a stickiness score (fraction of questions where the predicted label is identical under all rotations) and an average correct-option margin. This separates three failure modes: content-following (stickiness near zero, positive margin), knowledge loss (uniform failure), and label-sticking (high stickiness, collapsing margin), and places the student in the third. The repair mechanism is completion-only masked KL, in which the prompt contributes no loss and only tokens in the answer completion are supervised against teacher distributions; paired with teacher-verified, label-balanced MCQ permutations, it gives the student direct format supervision that pretraining-text KL lacks.

What would settle it

Run the same four-permutation protocol with a label-free condition in which the answer options are stripped of their letters and scored by content, or compare the Stage 3b repair against a format-only control that trains on randomly permuted options without teacher-verified content; if the student still ranks correct answer content at chance under label-free scoring, or if the C-Eval gain disappears when the repair data are not content-verified, then the deficit is knowledge loss rather than an interface injury.

Watch

Extended reading notes

Core claim

After surgery and distillation, the student matches its teacher closely on perplexity (valid CE gap +0.128 nats) yet scores 28.8% on C-Eval, near the 25% random baseline, while the teacher scores 50.6%. The paper's four-permutation diagnostic — rotating the four answer options of 161 clean questions while holding content fixed — shows the student predicts 'A' 81.06% of the time, keeps the same label across all four rotations on 106/161 questions, and has an average margin of -0.117 nats on the correct option. The paper reads this as label-sticking rather than content-following or uniform forgetting: the model has lost the interface that maps knowledge onto option labels. A 1,000-step completion-only KL stage on teacher-verified MCQ permutations, poetry, and translation QA lifts C-Eval to 41.31%, roughly halves stickiness, and turns the correct-option margin positive, at the cost of slightly worse general distribution fit (valid KL 0.160 to 0.196). The paper cautions that knowledge damage cannot be excluded and treats the repair as strong evidence, not proof, that most benchmark-relevant knowledge survived conversion.

Load-bearing premise

The paper's central diagnosis assumes that most benchmark-relevant knowledge survived the conversion and is merely blocked from being expressed; if that knowledge was actually damaged, the repair may just be teaching the model the multiple-choice format rather than restoring what was lost.

Editorial extensions

If this is right

  • Perplexity, KL, and hidden-state alignment are not sufficient evidence that a converted model has kept its task abilities; a directed behavioral diagnostic is needed.
  • The four-permutation protocol can be applied to any linearization or compression pipeline that is evaluated on multiple-choice benchmarks, not just this KDA conversion.
  • A short, format-targeted completion-only KL stage is an inexpensive repair: 1,000 steps on 8.2M tokens recover 12.48 C-Eval points and halve label stickiness.
  • Persona alignment through SFT, an identity booster, and one round of on-policy DPO can proceed after the repair without catastrophic forgetting, keeping C-Eval within noise (final 41.83% vs. 41.31%).
  • The repair costs general distribution fit: valid KL rises from 0.160 to 0.196, so the interface fix is not free.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the interface-injury framing generalizes, other distillation pipelines that omit format supervision may be silently under-reporting ability; a cheap check is to run a rotation-balanced MCQ probe before and after any format-targeted stage.
  • The diagnostic could be extended beyond multiple choice by rotating answer spans in generation tasks, which would test whether label-sticking is a general positional bias or specific to the option-letter format.
  • A label-free condition — asking the model to produce the answer content without option letters, or to rank contents after stripping labels — would settle whether the residual gap is interface or knowledge; the paper itself notes this missing control.
  • The FP32-master failure mode (bf16 swallowing small updates) is a plausible explanation for some frozen-weight divergence reports in other linearization recipes, though the paper only claims a matching symptom, not causation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper converts 21 of 28 layers of Qwen3-0.6B-Base into KDA linear-attention layers on a single consumer GPU, then uses layer-wise alignment and end-to-end KL distillation to produce a student that matches teacher perplexity (CE gap +0.128 nats) while scoring near chance on C-Eval (28.8%). A four-permutation MCQ diagnostic shows the student predicts option label 'A' 81.06% of the time, keeps the same label on all four rotations for 106/161 questions, and has a negative average correct-option margin. The paper calls this an 'interface injury'—a broken mapping from preserved knowledge to option labels—and claims that a 1,000-step completion-only KL stage on teacher-verified MCQs repairs the interface, lifting C-Eval to 41.31% and halving label stickiness. It further reports that persona SFT and one round of DPO preserve benchmark scores within noise, and it releases code, weights, recipes, and an audit trail.

Significance. If the central interpretive claim held, this would be a valuable practical and diagnostic contribution: it documents a failure mode of linearization that perplexity and KL cannot see, provides a cheap four-permutation probe for label-sticking, and offers a repair recipe. Strengths include the open release of code/weights/recipes, the explicit disclosure of the winner's-curse limitation from repeated ceval-valid selection, the careful forensic debugging of the conversion pipeline (e.g., false teacher-init, bf16 update swallowing), and the honest admission in Sections 4.2 and 5 that knowledge damage cannot be excluded. The observed label-sticking phenomenon itself is robust and worth reporting. However, the paper's stronger claim—that the diagnostic separates interface injury from knowledge loss and that the repair stage proves most benchmark-relevant knowledge survived—is not supported by the evidence as presented; the missing control and the unproven premise are load-bearing for the title and Contribution (i).

major comments (3)
  1. [§4.1–4.2, Contribution (i)] The four-permutation diagnostic does not separate interface injury from knowledge loss. The reported pattern—81.06% 'A' predictions, 106/161 questions with the same label across all rotations, and a −0.117-nat correct-option margin in Table 1—is also exactly what a model with partially damaged answer-content knowledge plus a positional prior would produce. The paper itself concedes this in §4.2 ('knowledge damage cannot be excluded') and §5 ('not as proof of zero knowledge loss'), yet the abstract and Contribution (i) assert that the diagnostic separates the two. Please either re-scope the claim to 'detects label-sticking' or add a control that demonstrates answer-content knowledge survives independently of the label interface, for example by scoring with label-free content probes or by comparing against a model with no position prior.
  2. [§5, Stage 3b repair] The repair evidence lacks a control arm. Training on 6,250 teacher-verified MCQs with balanced labels plus poetry/translation QA under completion-only KL for 1,000 steps could raise C-Eval from 28.83% to 41.31% and improve the 4-perm metrics in Table 2 by teaching MCQ-format heuristics or by re-fitting answer-content associations, independent of whether the target knowledge survived conversion. Please add a control arm matched for data quantity and distribution—e.g., the same MCQs with shuffled or permuted labels, or an equivalent volume of non-MCQ completion-only KL—and report whether 4-perm accuracy, margin, and stickiness improve only when labels are correctly mapped.
  3. [§9(i), Table 3] Because ceval-valid was used for 15+ checkpoint selections, the headline improvement from 28.83% to 41.31% is subject to winner's curse. The paper discloses this and locks a release ruler, but no final numbers on the locked ruler or on independent benchmarks such as MMLU/CMMLU are reported. Please report the locked-ruler and held-out results, or explicitly frame the ceval-valid numbers as selection results rather than test results.
minor comments (5)
  1. [§3.2 vs. §1] The overall CE after Stage 2 is given as '4.13' in the introduction and '4.125' in Section 3.2; please make the precision consistent.
  2. [§4.1, Eq. (3)] The stickiness statistic counts equal labels across all four rotations, but the text says the model 'sticks to option labels' using both the 81.06% 'A' share and 106/161 same-label questions; please clarify that the same-label count and the 'A' share are distinct statistics and that a same-label response is not necessarily 'A'.
  3. [§3.1] The description 'every 4th layer keeps its native GQA (layers 3, 7, 11, 15, 19, 23, 27, 0-indexed)' is clear but 'every 4th' with 0-indexing could be confusing; consider stating explicitly that the period is four layers starting from layer 3.
  4. [§5] The token budget for Stage 3b (1,000 steps × 8,192 tokens = 8.19M tokens) matches the text, but a per-component breakdown of the 8.2M tokens (MCQ permutations, poetry QA, translation QA) would help reproducibility.
  5. [§7] In the FP32-master failure-mode paragraph, adding the actual sub-ULP update magnitude (e.g., the ratio of update to bf16 epsilon) would make the 'silently swallowed' claim more concrete.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the diagnostic and repair are empirical observations, and the unproven knowledge-survival premise is a validity confound, not a circular step.

full rationale

This paper's derivation chain is self-contained and contains no step that reduces to its own inputs. The four-permutation diagnostic (Eqs. 3-4) is an original measurement whose inputs are the model's log-probabilities over rotated MCQ labels; the label-stickiness and margin values are reported, not fitted. The conclusion that the pattern constitutes an 'interface injury' is an interpretive inference, and the paper itself disclaims the stronger reading: Section 4.2 states 'knowledge damage cannot be excluded', and Section 5 says the repair is 'strong evidence that most benchmark-relevant knowledge survived conversion—not as proof of zero knowledge loss'. The repair stage is uncontrolled, so it cannot decisively rule out knowledge loss, but that is an underdetermination or validity problem, not circularity, because the repair outcome is new evidence rather than a premise already contained in the diagnosis. There are no self-citations carrying argumentative weight; citations to GenDistill and HALO are external prior work. The reuse of ceval-valid for checkpoint selection and then as the headline number is a disclosed benchmark-leakage issue ('the final score is not an untouched test'), but no parameter is fitted to it in a way that makes the reported improvement true by construction. Consequently, no circular step is exhibited, and the correct finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claim depends on a handful of hand-chosen hyperparameters (gate init, temperature, margin threshold, layer split, data mix), none of which are derived from theory. The 'interface' is a new explanatory construct, but it is anchored to measurable quantities. The paper is honest about the limits of its own assumptions.

free parameters (5)
  • KDA gate init decay-scale (g6) = exp(Alog) in (0.03, 0.3), dt_bias=0
    Selected from a six-scheme ablation by overall CE on the held-out ruler; near-total retention explodes the state and near-total forgetting scores worse than random.
  • KL temperature T = T=2
    Selected from a two-arm ablation (T=1 vs T=2) by C-Eval; the 1.8-point gap is not significance-tested, so T=2 is a directional choice that Stage 3b and later stages inherit.
  • Teacher-verified MCQ margin threshold = >=0.25 nats
    Chosen by hand to filter Stage 3b data; questions where the teacher is correct under all four rotations with min margin at least 0.25 nats. The threshold affects repair data quality and is not swept.
  • Hybrid layer ratio 21 KDA / 7 GQA = 21 of 28 layers, every 4th layer retained
    Design choice for the conversion layout; motivated by prior recipes (GenDistill, HALO) but not ablated here. The default knowledge-transfer scope depends on this split.
  • Stage 3b data composition = 6,250 MCQs x4 + 3,000 poetry + 3,000 translation
    Chosen by hand; no ablation of the contribution of each component to the C-Eval gain, so the attribution to format repair is not isolated.
assumptions (4)
  • domain assumption The KDA update rule (Eq. 1) and its reference implementation in fla kernels faithfully represent the architecture claimed
    Section 3.1 states the implementation matches Kimi Linear to about 1e-6 numerical error against chunk_kda; the paper relies on this to attribute effects to KDA linearization.
  • domain assumption Forward KL distillation (Eq. 2) is an appropriate transfer objective that preserves task knowledge
    The whole pipeline uses forward KL between teacher and student distributions; no evidence is offered that this objective does not itself induce label priors.
  • domain assumption The held-out ruler questions are genuinely disjoint from C-Eval, MMLU, CMMLU and the evaluation rulers
    Section 4.1 asserts zero exact and 5-gram overlap; this underpins the claim that the diagnostic measures a conversion effect rather than benchmark leakage.
  • ad hoc to paper Loss of knowledge would manifest as uniform failure rather than label stickiness
    Section 4.1 assumes the three behaviors (content-following, knowledge loss, label-sticking) separate cleanly; Section 4.2 admits label-free content scoring is absent, so the separation is not directly proven.
invented entities (1)
  • Option-label interface (interface injury) independent evidence
    purpose: Conceptual construct used to explain why the student has near-teacher perplexity but chance-level multiple-choice accuracy: knowledge is intact but the mapping from knowledge to option labels is broken.
    The construct has falsifiable handles: label stickiness measured by the four-permutation diagnostic and the positive margin after repair. It is not a physical entity, but it is a new explanatory component introduced by this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model." pith.science (2026). https://pith.science/paper/SZIRWIBG

@misc{pith2026260802689,
  author       = {Pith},
  title        = {Pith review of: Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SZIRWIBG}},
  note         = {Machine review of arXiv:2608.02689}
}
read the original abstract

We convert 21 of 28 full-attention layers of Qwen3-0.6B-Base into KDA (Kimi Delta Attention) linear-attention layers on a single consumer-grade GPU budget, and ask a simple question: what exactly does the conversion break? After surgery, hidden-state alignment and end-to-end KL distillation drive the student close to its teacher in perplexity, yet multiple-choice accuracy stays near random chance (25-29% vs. the teacher's 50.6% on C-Eval). Using a four-permutation diagnostic that rotates answer options while holding content fixed, we show the model sticks to option labels (predicting "A" 81% of the time; 106/161 questions keep the same label under all four rotations) rather than following answer content -- an interface injury that standard distillation metrics cannot see. A 1,000-step format-targeted completion-only KL stage repairs the interface (+12.48 points on C-Eval, label-stickiness roughly halved), after which persona SFT and one round of on-policy DPO preserve benchmark scores within noise. We release code, weights, recipes, and the full audit trail, and distill the engineering lessons -- including an FP32-master failure mode in which bf16 optimizer updates are silently swallowed -- that made convergence possible at this budget.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 19 canonical work pages

  1. [1]

    2026 , howpublished =

    When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models , author =. 2026 , howpublished =

    work page 2026

  2. [2]

    2026 , howpublished =

    Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts , author =. 2026 , howpublished =

    work page 2026

  3. [3]

    Gated Delta Networks: Improving

    Yang, Songlin and Kautz, Jan , journal =. Gated Delta Networks: Improving

    work page

  4. [4]

    2025 , howpublished =

    Kimi Linear: An Expressive, Efficient Attention Architecture , author =. 2025 , howpublished =

    work page 2025

  5. [5]

    Transformers are

    Katharopoulos, Angelos and Vyas, Apoorv and Pappas, Nikolaos and Fleuret, Fran. Transformers are. Proceedings of ICML , year =

    work page

  6. [6]

    2024 , howpublished =

    fla: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism , author =. 2024 , howpublished =

    work page 2024

  7. [7]

    2015 , howpublished =

    Distilling the Knowledge in a Neural Network , author =. 2015 , howpublished =

    work page 2015

  8. [8]

    Proceedings of NeurIPS , year =

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model , author =. Proceedings of NeurIPS , year =

    work page

Show all 26 references
  1. [9]

    2025 , howpublished =

    Qwen3 Technical Report , author =. 2025 , howpublished =

    2025

  2. [10]

    Huang, Yuzhen and Bai, Yuzhuo and Zhu, Zhihao and Zhang, Junlei and Zhang, Jinghan and Su, Tangjun and Liu, Junteng and Lv, Chuancheng and Zhang, Yikai and Lei, Jiayi and Fu, Yao and Sun, Maosong and He, Junxian , booktitle =

  3. [11]

    Proceedings of ICLR , year =

    Measuring Massive Multitask Understanding , author =. Proceedings of ICLR , year =

  4. [12]

    Li, Haonan and Zhang, Yixuan and Koto, Fajri and Yang, Yifei and Zhao, Hai and Gong, Yeyun and Duan, Nan and Baldwin, Timothy , year =

  5. [13]

    2024 , howpublished =

    A Framework for Few-Shot Language Model Evaluation , author =. 2024 , howpublished =

    2024

  6. [14]

    Yingfa Chen, Zhen Leng Thai, Zihan Zhou, Zhu Zhang, Xingyu Shen, Shuo Wang, Chaojun Xiao, Xu Han, and Zhiyuan Liu. 2026. Hybrid linear attention done right: Efficient distillation and effective architectures for extremely long contexts. arXiv:2601.22156

    2026

  7. [15]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  8. [16]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask understanding. In Proceedings of ICLR

    2021

  9. [17]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv:1503.02531

    2015 arXiv

  10. [18]

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-Eval : A multi-level multi-discipline chinese evaluation suite for foundation models. In Proceedings ...

    2023

  11. [19]

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran c ois Fleuret. 2020. Transformers are RNN s: Fast autoregressive transformers with linear attention. In Proceedings of ICML

    2020

  12. [20]

    Juan Gabriel Kostelec and Qinghai Guo. 2026. When perplexity lies: Generation-focused distillation of hybrid sequence models. arXiv:2603.26556v2. Huawei Zurich Research Center; ACS Lab, Huawei Technologies

    2026 arXiv

  13. [21]

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. CMMLU : Measuring massive multitask language understanding in chinese. arXiv:2306.09212

    2023 arXiv

  14. [22]

    Moonshot AI . 2025. Kimi linear: An expressive, efficient attention architecture. arXiv:2510.26692

    2025 arXiv

  15. [23]

    Qwen Team . 2025. Qwen3 technical report. arXiv:2505.09388

    2025 arXiv

  16. [24]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Proceedings of NeurIPS

    2023

  17. [25]

    Songlin Yang and Jan Kautz. 2024. Gated delta networks: Improving M amba2 with delta rule. arXiv preprint arXiv:2412.06464

    2024 arXiv

  18. [26]

    Songlin Yang and Yu Zhang. 2024. fla: A triton-based library for hardware-efficient implementations of linear attention mechanism. https://github.com/fla-org/flash-linear-attention

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.