{"id":"ba8e2f71-f530-4c71-a353-7c54767534e0","arxiv_id":"2505.15442","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Knowledge distillation improves small language models by up to 10% on reasoning tasks, but the claimed reasoning-fidelity paradox is not well supported by the BLEU-based metric used.","lead":"This paper tests how well knowledge distillation, training a small model to copy a large one, works across 0.5B to 7B language models on reasoning tasks. It reports large gains for small models and a mismatch between accuracy and copying the teacher's reasoning, but that mismatch is measured with a weak metric.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'fidelity paradox' rests on BLEU as a proxy for reasoning structure, which the paper's own error analysis refutes; the accuracy claims are unaffected.","rationale":"The reader's weakest assumption already identifies the BLEU issue, and I agree. I considered whether the more load-bearing concern is the absence of multiple seeds or the 'zero-shot' label (the SFT/KD training sets overlap with several evaluation tasks), but those affect the strength of the accuracy claims, not the paper's novel conceptual contribution. Even if the 10% gain shrank with more seeds, the paper would still report meaningful accuracy improvements; but if BLEU is not a reasoning-fidelity metric, the title claim collapses into a much weaker observation about surface generation. Because the accuracy findings are largely independent of BLEU, the appropriate outcome remains the reader's CONDITIONAL verdict: accept the empirical accuracy results while requiring either a semantic/step-level fidelity metric or a reframed claim. Hence no verdict change.","tokens_in":25862,"tokens_out":5206,"duration_ms":47565,"concrete_test":"Take 100 problems each from SVAMP and GSM8K. For each problem, sample two correct outputs from the same teacher model (e.g., Qwen-14B) with different temperatures or random seeds; compute BLEU between them. If the median same-teacher BLEU is close to the 0.13 value seen between two correct but differently worded solutions, then BLEU cannot distinguish 'different reasoning' from 'different wording', and the fidelity paradox is an artifact of the metric. A complementary check: have two annotators judge whether student and teacher reasoning chains are logically equivalent step-by-step on the Table 10 sample plus 50 fresh examples; if BLEU and annotator agreement diverge, the metric is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's most novel claim, the 'generalization vs fidelity paradox' in which students gain accuracy without preserving the teacher's structured decision-making, depends entirely on the BLEU-based 'reasoning fidelity' metric defined in Section 4. BLEU measures n-gram overlap between teacher and student strings; it does not measure reasoning validity or structure. The paper's own Table 10 demonstrates this: in the first two examples the student reaches the wrong final answer yet scores BLEU 0.77 and 0.79 with the teacher because the surface text is similar; in the third example both models reason correctly but via different valid routes and score BLEU 0.13. The validation against Sentence-BERT cosine similarity (r=0.97) does not rescue the metric because cosine similarity of sentence embeddings is likewise a surface-form similarity, not a logical-equivalence measure. Consequently, the reported low correlations between student accuracy and 'fidelity' (Table 5) and the framing 'KD does not always maintain structured decision-making processes' are not supported. What the data support is the much weaker statement that distillation does not force surface-level imitation. The accuracy-based findings (size-dependent gains, task-expertise effects) do not depend on BLEU and remain plausible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a large-scale empirical study of knowledge distillation (KD) for small language models. Using Qwen-2.5 (0.5B to 14B) and LLaMA-3 (1B to 8B) teachers and students, the authors fine-tune students with three KD methods (SeqKD, RevKD, GKD) that are compared against supervised fine-tuning, and evaluate zero-shot performance on mathematical reasoning, commonsense reasoning, and instruction-following tasks. The headline findings are that KD improves smaller students by up to about 10 absolute average points (with the largest gains for sub-1B models) while giving only marginal gains to 7B students; that teacher performance correlates weakly with student gains while teacher task expertise matters; and that there is an alleged 'generalization vs. fidelity paradox' in which students gain accuracy without preserving the teacher's reasoning. The paradox rests on a fidelity metric defined in Section 4 as the BLEU score between teacher and student generated reasoning outputs. The paper also reports ablations on logit noise and temperature smoothing. The accuracy-side findings are reported in detailed per-task tables (Tables 7 to 9).","tokens_in":25983,"tokens_out":11492,"duration_ms":88982,"significance":"If the accuracy findings hold, this is a useful large-scale descriptive benchmark: it covers two model families, three KD methods, and 19 tasks, it ships code, and it provides unusually detailed per-task tables. The descriptive pattern - larger gains for smaller students, diminishing returns at 7B, and the importance of teacher task adaptation (Figure 2, with up to 40% degradation from a task-unaware teacher) over raw teacher strength - is practically relevant and internally consistent with the tables. The noise and temperature ablations are well motivated and clearly reported. However, the paper's most distinctive claim, the generalization vs. fidelity paradox, is not supported by the evidence: the BLEU-based fidelity metric measures surface-text overlap, and the paper's own error analysis (Table 10) shows incorrect answers scoring high fidelity (0.77 and 0.79) while two correct solutions with different reasoning paths score 0.13. The validation against Sentence-BERT cosine similarity (r=0.97) only confirms agreement with another surface-form similarity measure. The accuracy contributions do not depend on this metric and remain plausible; the paradox framing does not.","major_comments":[{"comment":"The load-bearing premise of the fidelity paradox is that the BLEU score between teacher and student generated text measures 'reasoning fidelity' or 'structured decision-making.' Table 10 refutes this: in the first two examples the student reaches the wrong final answer yet scores BLEU 0.77 and 0.79 because the surface text is similar, while in the third example both models reason correctly through different valid paths yet score 0.13. Consequently, the weak correlations in Table 5 and the abstract claim that KD 'does not always maintain the structured decision-making processes of the teacher' are not supported; the data establish at most that KD does not enforce surface-level textual imitation. The reported r=0.97 agreement with Sentence-BERT cosine similarity does not rescue the metric, since cosine sentence similarity is likewise a surface-form measure. I recommend either validating the fidelity metric against an annotation of reasoning-step equivalence (e.g., whether the student's chain correctly and completely derives the final answer) or reframing the paradox claim in terms of surface/textual fidelity.","section":"Section 4 ('Measures for quantifying teacher-student agreement and fidelity') and Section 5 ('How does KD impair…"},{"comment":"The statistical reporting omits the sample sizes and units behind the t-tests, ANOVA, and Spearman correlations. In Table 2, the Spearman correlation between KD gains and teacher performance has at most three teacher sizes per student (3B/7B/14B for Qwen), and Table 3 correlates student performance with model size over four Qwen sizes or two LLaMA sizes; p-values of 0.0 at such N are not meaningful without stating N. No random seeds are reported, and the t-tests in Table 1 appear to pool a single training run per configuration. Because several headlines ('teacher performance has minimal impact', 'smaller models benefit more from KD') are inferential claims, the manuscript should state the number of independent observations, report variance across seeds, and temper the significance framing where N is tiny; the descriptive monotone trends in Figure 1 would still support the size-dependent claim.","section":"Tables 1-5; Section 5 ('Does KD depend on teacher performance?' and 'Does KD depend on student model size?')"},{"comment":"Several cells in Table 8 indicate collapsed or unstable runs that are inconsistent with the claim that KD consistently improves over SFT (Section 5, Figure 1): Qwen-1.5B distilled with RevKD from Qwen-3B scores 16.72 on Hellaswag and 50.60 on PiQA against SFT baselines of 63.90 and 76.93, and Qwen-3B distilled with GKD from Qwen-7B scores 11.67 on Hellaswag against 81.91 for SFT. These entries feed directly into the reported averages and into the t-tests of Table 1. The paper should report multiple seeds and variances, exclude or explain degenerate runs, or present medians; as reported, the reader cannot tell whether the averages describe typical behavior of the distillation methods.","section":"Table 8 (commonsense reasoning results)"}],"minor_comments":[{"comment":"The abstract says '14 complex reasoning tasks' while the Introduction and Section 4 state 19 tasks; please align the counts.","section":"Abstract and Introduction"},{"comment":"There are typographical artifacts in the camera text: 'SV AMP' and 'ANOV A' in table headers, and 'Adversial' in the Hellaswag description in Appendix A.","section":"Tables 1, 3, 4 and Appendix A"},{"comment":"The table heading says 'student performance and student model size' while the text says 'between KD improvement and student size'; state explicitly whether the correlated variable is the raw score or the gain over SFT.","section":"Table 3 and Section 5"},{"comment":"The 'up to 40%' degradation claim needs a specification of whether the drop is absolute or relative, and the figure should report the underlying values.","section":"Figure 2 and Section 5 ('Does KD depend on teacher performance?')"},{"comment":"Please state how per-instance BLEU is aggregated into the fidelity scores shown in Figures 4 and 9, and whether the same protocol is used for the instruction-following outputs that are scored with Rouge-L in Table 9.","section":"Section 4 and Appendix B"},{"comment":"The Limitation section asserts that 'the findings generalize well within these domains'; this is an unsupported empirical claim and should be qualified or removed.","section":"Limitation section"},{"comment":"The heading 'How does KD impair teacher-student agreement?' presumes a negative effect, while Figures 3, 8, and 10 mostly show agreement increasing after KD; a neutral heading would be more accurate.","section":"Section 5 subsection headings"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the fidelity metric: the paradox is the paper's headline novelty, and the paper's own Table 10 shows that BLEU cannot carry the reasoning-structure interpretation. A focused revision that validates the metric against reasoning-equivalence annotations or reframes the claim to surface-level fidelity would make the paper publishable, since the accuracy-side empirical contribution is solid and the descriptive size-dependent pattern is credible. I see no circularity problem: the Sengupta et al. (2023) citation appears in related work only, and the fidelity metric is a measurement choice rather than a fitted entity. One positioning note: the paper cites Stanton et al. (2021) and should explicitly connect its 'fidelity paradox' to that line of work to sharpen the novelty claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the accuracy results are a genuinely useful addition to the KD literature: a consistent scan of SeqKD, RevKD, and GKD across 0.5B-7B students on 19 reasoning and instruction tasks, showing up to 10% gains for sub-1B models and about 1.3% for 7B. These measurements are new and plausible. Second, the headline 'generalization vs fidelity paradox' does not survive contact with the paper's own error analysis. The fidelity metric is BLEU between teacher and student text, and Table 10 shows why that fails: two wrong answers with near-identical surface text score BLEU 0.77 and 0.79, while two correct answers that reason differently score 0.13. BLEU measures n-gram overlap, not reasoning structure. The validation against Sentence-BERT cosine similarity (r=0.97) does not rescue it, because cosine similarity of sentence embeddings is also surface-form similarity. So the claim that KD does not preserve the teacher's 'structured decision-making' is not supported. What the data support is the weaker, still interesting claim that KD improves accuracy without forcing surface-level imitation.\n\nThe accuracy half is the real contribution, and it is solid. The study is the first large-scale empirical and statistical scan of KD on LLMs in this setting. The size-dependent gains, the teacher-task-expertise finding, and the noise/temperature ablations are worth having. The code link is a plus. The statistical testing is serviceable but not rigorous: no multiple seeds, the t-test unit of analysis is not specified, and there is no multiple-comparison correction. There are also presentation issues: the abstract says 14 tasks, the body says 19, and Table 3 has a header mismatch. These are fixable.\n\nBottom line: the paper deserves a serious referee. The accuracy findings are a solid empirical contribution for anyone working on distillation for on-device deployment. The fidelity half needs either a real reasoning-fidelity metric (human evaluation or model-based step matching) or a reframed claim. If the authors do that, the paper is publishable; as is, the abstract overclaims.","headline":"Solid accuracy findings on KD for small LMs; the 'fidelity paradox' is BLEU overreach contradicted by the paper's own error analysis.","tokens_in":26612,"tokens_out":2723,"would_cite":true,"duration_ms":22446,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Knowledge distillation raises zero-shot accuracy of sub-1B language models by up to 10% (22% on some tasks) while leaving the fidelity of the teacher's reasoning largely untransferred — the paper's generalization-fidelity paradox.","keywords":["knowledge distillation","zero-shot generalization","reasoning fidelity","teacher-student agreement","language model compression","BLEU fidelity metric","reverse KL divergence","generalization-fidelity paradox"],"falsifier":"Score the same student-teacher output pairs on a reasoning-structure metric that ignores wording, such as human-annotated step equivalence or matched symbolic equations, and correlate that score with student accuracy. If structural fidelity correlates strongly with accuracy (say Spearman $\\rho > 0.5$) where BLEU showed near zero, the paradox is an artifact of the metric; if the near-zero correlation persists, the paradox stands on a stronger footing.","tokens_in":25587,"feed_emoji":"🧠","tokens_out":10484,"duration_ms":79930,"temperature":0.7,"pith_summary":"The paper claims that knowledge distillation (KD), which trains a small student model to imitate a larger teacher, is a strong upgrade for the smallest language models: in zero-shot evaluation on 19 reasoning and instruction-following tasks, sub-1B students gain up to 10% average accuracy and up to 22% on individual tasks, while 7B students gain only about 1.3%. It further claims that the quality of the teacher, measured by raw benchmark performance, matters little for student gains, but the teacher's expertise on the target task matters a great deal: a task-unaware teacher can cut student accuracy by up to 40%. The paper's headline finding is a paradox: distilled students get more answers right without reproducing the teacher's reasoning steps, since teacher-student agreement and reasoning fidelity correlate weakly or not at all with student accuracy. If these claims hold, KD is an unusually cheap way to improve small deployed models, but 'distilled reasoning' cannot be inferred from accuracy gains alone.","feed_headline":"Small language models gain up to 10% from distillation","feed_subtitle":"But students rarely copy the teacher's reasoning, so accuracy gains don't mean the teacher's skill transferred.","key_machinery":"The argument runs on a fixed distillation protocol: supervised fine-tuning plus three sequence-level KD losses — SeqKD (maximize student probability of the teacher's full output sequence, $-\\log P_S(y_T|x)$), RevKD (reverse KL divergence, $D_{\\text{KL}}(P_S(y|x) \\| P_T(y|x))$), and GKD (mix fixed teacher-generated data with the student's own on-policy generations under Jensen-Shannon divergence with $\\lambda = 0.5$, $\\beta = 0.5$). Two measurement instruments carry the paradox: teacher-student agreement, the fraction of matching top-1 predictions, and reasoning fidelity, the BLEU score between the teacher's and student's generated reasoning text, validated against sentence-embedding cosine similarity (Spearman 0.97). The ablations vary teacher-logit noise $\\sigma$ and softmax temperature $\\tau$, showing that moderate noise and $\\tau = 2$ help while $\\sigma = 5$ or $\\tau = 5$ collapse student accuracy.","core_discovery":"The central claim is empirical: across the Qwen-2.5 and LLaMA-3 families from 0.5B to 7B parameters, three knowledge-distillation methods (SeqKD, reverse-KL RevKD, and GKD) consistently beat supervised fine-tuning on mathematical reasoning, commonsense reasoning, and instruction-following benchmarks in zero-shot evaluation, with the benefits concentrated in the smallest models and a strong negative correlation (about $\\rho = -0.66$ on math) between student size and KD gain. The paper also claims that teacher task expertise, not teacher performance, drives distillation success, and that teacher-student fidelity, measured by BLEU between teacher and student reasoning outputs, is not tied to student accuracy, which it frames as a generalization-fidelity paradox: distillation transfers task effectiveness without reliably transferring the teacher's structured decision-making. Statistical comparisons (t-tests, ANOVA, Spearman rank correlations) across all teacher-student pairs, together with ablations on teacher-logit noise and temperature smoothing, form the evidence base.","pith_inferences":["If the near-zero fidelity-performance correlation is metric-driven, then a structure-aware fidelity measure (symbolic equation matching or token-level rationale alignment) could change the paradox's strength; re-running the analysis with such a metric would test whether BLEU is the reason fidelity looks disconnected from accuracy.","The strong negative size-benefit correlation hints at a capacity threshold where the student already has the target skill; a testable extension is to take the best distilled small checkpoint and re-distill it from an even stronger teacher to see whether gains compound or saturate.","The noise and temperature ablations imply that distillation success is governed by the signal-to-noise quality of the teacher's logits; one could test this directly by measuring how student accuracy tracks the mutual information between teacher logits and task labels.","The fidelity gap suggests an objective-level fix the paper does not test: adding a reasoning-fidelity penalty to the KD loss (matching intermediate steps, not just final tokens) might close the paradox, and measuring whether students keep their accuracy gains would settle whether the trade-off is intrinsic or an artifact of current losses."],"forward_implications":["Sub-1B models can gain roughly 10 points of zero-shot accuracy from distillation, making KD a practical upgrade path for edge and on-device deployment rather than a marginal trick.","Larger models (7B) show only about 1.3% average gains, so for them the compute spent on distillation needs separate justification.","Choice of KD method matters little for math and commonsense benchmarks, but reverse-KL RevKD is the most consistent method overall, especially on instruction-following tasks.","A teacher strong on general benchmarks but not fine-tuned on the target domain can actively harm the student (up to 40% degradation), so teacher selection should prioritize task expertise over raw strength.","Accuracy gains and teacher-student fidelity are separate axes: a high-scoring student may reason in a way that diverges from the teacher, so fidelity scores should not be used as a proxy for transferred reasoning."],"supporting_citations":[{"why":"Defines soft-target knowledge distillation, the base method that all three KD variants tested here extend.","marker":"(Hinton et al., 2015)"},{"why":"Introduces SeqKD, one of the three distilled methods whose post-distillation performance is compared.","marker":"(Kim and Rush, 2016)"},{"why":"Proposes reverse-KL MiniLLM distillation (RevKD), the method found most consistent across student sizes.","marker":"(Gu et al., 2024)"},{"why":"Supplies GKD, the on-policy distillation method that trains students on self-generated sequences.","marker":"(Agarwal et al., 2024)"},{"why":"Defines BLEU, the metric used to measure reasoning fidelity and to ground the generalization-fidelity paradox.","marker":"(Papineni et al., 2002)"},{"why":"Provides the teacher-student gap hypothesis that the paper replicates and extends to language models.","marker":"(Mirzadeh et al., 2020)"}],"fun_headline_variants":["Distillation boosts small LMs 10%, but fidelity trails","Small LMs shine in distillation, yet reasoning fidelity lags","Distillation's paradox: accuracy gains, but teacher reasoning lost","Distillation: smaller LMs gain, but fidelity breaks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that BLEU, a text-overlap score, validly measures whether the student has reproduced the teacher's reasoning steps, even though the paper's own error analysis shows two correct but differently reasoned answers scoring BLEU 0.13.","fun_headline_variants_meta":{"raw":{"variants":["Distillation boosts small LMs 10%, but fidelity trails","Small LMs shine in distillation, yet reasoning fidelity lags","Distillation's paradox: accuracy gains, but teacher reasoning lost","Distillation: smaller LMs gain, but fidelity breaks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000715,"raw_usage":{"total_tokens":3240,"prompt_tokens":999,"completion_tokens":2241,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":2171}},"tokens_in":615,"tokens_out":2241,"duration_ms":15471,"temperature":1.0,"reasoning_tokens":2171,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:17:13.029397+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score the same student-teacher output pairs on a reasoning-structure metric that ignores wording, such as human-annotated step equivalence or matched symbolic equations, and correlate that score with student accuracy. If structural fidelity correlates strongly with accuracy (say Spearman $\\rho > 0.5$) where BLEU showed near zero, the paradox is an artifact of the metric; if the near-zero correlation persists, the paradox stands on a stronger footing.","supporting_citations":[],"review_version":1}