{"id":"56d3cf6a-44e6-4325-af88-9545653daa3a","arxiv_id":"2504.20000","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Fine-tuning the teacher before knowledge distillation improves the distilled model most reliably when teacher and student share a vocabulary, and fine-tuning both gives the best scores overall.","lead":"This paper tests whether fine-tuning the teacher model, the student model, or both before knowledge distillation helps build a smaller telecom question-answering model. It finds that fine-tuning the teacher helps most when teacher and student share a vocabulary, and fine-tuning both usually gives the best scores.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's significance counts are unverifiable because Section 2.2 omits the pairing unit and sample size for the Wilcoxon tests on which every core SFT-improvement claim rests, and Table 3 itself has missing entries.","rationale":"The reader's weakest assumption is that the Wilcoxon signed-rank test is valid and correctly applied, with the unit of pairing and sample size unspecified. That is exactly the load-bearing concern I identify: the paper's central claims are statements about statistical significance, and Table 3, which is the only quantitative basis for those claims, cannot be checked without knowing what was paired and how many observations entered each test. The reader's conditional verdict is therefore appropriate: the paper should be accepted only if the statistical details and the underlying per-metric scores are provided. I also note a concrete internal inconsistency in the conclusion, where the Mistral teacher is said to show significant improvements despite Table 3 reporting zero rejections for Mistral on all SFT hypotheses, and an incompleteness in Table 3's H_Alg rows; these reinforce the need for a full audit of the reported significance counts but do not change the verdict from CONDITIONAL.","tokens_in":7902,"tokens_out":6419,"duration_ms":68866,"concrete_test":"Ask the authors for per-question metric scores or the code for Section 2.1/2.2, and rerun each Wilcoxon signed-rank comparison from Eq. (1)-(3): for each of the 14 metrics on the 597 TeleQuAD test pairs, form the paired differences between the two conditions (e.g., (TSFT,SB) vs (TB,SB)) and apply a two-sided Wilcoxon test at a stated alpha; count rejected nulls and compare to Table 3. If the pairing unit or the number of samples entering each test is not disclosed, or if the reproduced counts differ, the claims that SFT of teacher improves performance irrespective of algorithm and metrics, and the conclusion's sentence about Mistral, lose empirical support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2 defines hypotheses such as H^S_SFT: Perf(TSFT, SSFT) = Perf(TB, SSFT) and says they are tested with a Wilcoxon signed-rank test per metric, with results aggregated in Table 3. The paper never states the unit of paired observations. If pairing is per test question, each metric supplies 597 paired scores and the test is meaningful; if pairing is across the 14 metrics, the effective sample is n=14 scalars per condition and the test is not validly defined; if pairing is per run, the number of runs is never stated. No significance level, effect size, or correction for multiple comparisons is reported. Table 3 is internally incomplete: the H_Alg rows list only two values instead of four (Llama-V, Llama-D, Mistral-V, Mistral-D), so the '10 out of 14 and 14 out of 14' claim cannot be reconstructed. The conclusion also swaps the model families: it says 'when teacher is Mistral ... improvement ... same vocabulary is significant,' but Table 3 shows Mistral columns with 0 rejections for every SFT hypothesis. Because the abstract's central claim is phrased in terms of statistical significance, the entire argument is currently non-reproducible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether supervised fine-tuning (SFT) of the teacher, the student, or both, before knowledge distillation (KD) improves question-answering performance in the telecom domain. Using Llama-7B and Mistral-7B as teachers and TinyLlama-1.1B as student, the authors compare vanilla KD and Dual-Space KD (DSKD) under same-vocabulary and different-vocabulary conditions, evaluating outputs on 14 N-gram, embedding, and LLM-based metrics. Statistical significance is assessed through Wilcoxon signed-rank tests on hypotheses about SFT, vocabulary, and algorithm effects. The paper claims that SFT of the teacher improves distilled performance when teacher and student share a vocabulary, and that SFT of both generally helps, with significance depending on vocabulary.","tokens_in":8190,"tokens_out":3205,"duration_ms":29874,"significance":"If the results are reproducible, this is a useful empirical contribution: it systematically addresses a practical question—whether to domain-adapt the teacher, the student, or both before KD—and it evaluates a broader metric suite than is typical in KD papers. The experimental design covers a useful matrix of conditions (two teachers, two algorithms, two SFT choices for each of teacher and student). The paper also provides a concrete recommendation for practitioners: when SFT is infeasible, different-vocabulary DSKD is preferred. These findings would be valuable to the NLP/KD community, provided the statistical evidence is presented rigorously. The main weakness is that the statistical analysis, which underpins every significance claim, is under-specified and partly internally inconsistent.","major_comments":[{"comment":"The statistical test is not fully specified. The paper states that a Wilcoxon signed-rank test is performed per metric, but it never defines the unit of paired observations. A signed-rank test requires paired samples; the 14 metrics appear to be single aggregate values per condition, so the pairing cannot be across metrics. If the intended pairing is across the 597 test questions, that should be stated explicitly and the degrees of freedom reported. If the pairing is across runs, the number of runs is never given. Without this information, the rejection counts in Table 3 cannot be verified, and Table 3 is the sole basis for the significance-based conclusions in Sections 3.1, 3.2, and 3.5.","section":"Section 2.2 and Table 3"},{"comment":"The conclusion text contradicts the reported results. Section 4 states: \"when teacher is Mistral, training using SFT improves performance across vocabulary and algorithms; improvement with SFT of teacher and/or student when models have same vocabulary is significant.\" Mistral is the different-vocabulary teacher, and Table 3 shows 0 rejections out of 14 for all Mistral SFT hypotheses (H_S_train, H_T_train, H_T,S_train). The significant improvement is reported for the same-vocabulary Llama teacher. This misattribution affects the paper's central practical recommendation and must be corrected.","section":"Section 4"},{"comment":"Table 3 is incomplete for the algorithm hypotheses. The table has four columns (Llama-V, Llama-D, Mistral-V, Mistral-D), but the rows H_T,S_Alg and H_B_Alg list only two values each (\"0 1\" and \"10 14\", respectively). The text in Section 3.3.1 refers to \"10 out of 14 and 14 out of 14\" without clarifying which columns those numbers correspond to, so the reader cannot reconstruct the evidence for the DSKD-over-vanilla claim. All four entries should be reported.","section":"Table 3"},{"comment":"No multiple-comparison correction is applied. The paper performs 24 hypothesis definitions (12 H-Train, 8 H-SFT, 4 H-Algo), each tested on 14 metrics, for a total of 336 Wilcoxon tests. At α=0.05, one would expect about 0.7 false rejections per hypothesis over 14 metrics under the global null. The claim that \"13 out of 14\" metrics reject for H_T_train may partly reflect multiplicity. Adjusted p-values, a stated significance level, or an explicit justification for not correcting should be provided.","section":"Section 2.2"}],"minor_comments":[{"comment":"The caption refers to \"the heatmap in Fig. ??\", an unresolved cross-reference that should be fixed.","section":"Figure 4 caption"},{"comment":"The sentence \"We observe that training improves results Fig. 4\" is missing a preposition and a period; it should read \"training improves results in Fig. 4\".","section":"Section 3.1.2"},{"comment":"The phrase \"in-lieu of these gaps\" is non-standard; \"to address these gaps\" would be clearer.","section":"Section 1.1"},{"comment":"The training configuration reports LoRA rank, alpha, dropout, and learning rate, but the KD hyperparameters (e.g., distillation temperature, loss weighting, batch sizes, number of epochs for KD) are not reported, which limits reproducibility.","section":"Table 2"},{"comment":"No random seeds, number of runs, or confidence intervals are reported; the paper should clarify whether each metric value is a single run or an average over multiple runs.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a worthwhile practical question and the experimental matrix is reasonable, but the statistical reporting is not yet at the level required for a reliable contribution. The Wilcoxon test specification, Table 3 completeness, and the Section 4 contradiction are fixable; I would like to see a revised version with a clear description of the pairing, sample sizes, multiple-comparison handling, and corrected conclusion text before I can recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere is my read on the TeleQuAD KD paper. The core take-away is that the paper likely has a true and useful practical point: when the teacher and student share a vocabulary, fine-tuning the teacher before distillation helps, and fine-tuning both is fine but not necessary. That is a design rule practitioners can use. What the paper does not yet provide is a verifiable statistical backing for that rule.\n\nWhat is new: the comparison matrix—SFT of teacher-only, student-only, both, and neither, crossed with same/different vocab and vanilla KD vs DSKD, evaluated with 14 metrics across three metric groups—is, as far as I can tell, not covered by prior work. That is a legitimate empirical contribution. The experimental design is systematic and the metric coverage is a plus; for a narrow domain like telecom QA this is the kind of study that saves a team a lot of trial and error. I also appreciate that they separate n-gram, embedding, and LLM-judge metrics and check that trends agree across groups.\n\nThe soft spots are real and mostly in the statistical layer. The Wilcoxon signed-rank tests in Section 2.2 are defined syntactically but the pairing unit is never stated. Whether the test is per test question (n=597), per metric (n=14), or per run (unknown runs) changes the validity and power completely. No significance level, effect size, or multiple-comparison correction is reported. Table 3 is also internally incomplete—the H_Alg rows show only two values instead of four, so the '10 out of 14 and 14 out of 14' claim cannot be reconstructed. And the conclusion prose confuses the model families: it says Mistral in a sentence about same-vocabulary significance, when the table gives Mistral 0/14 rejections everywhere. That has to be a typo, but it should be fixed.\n\nI would not call any of this fatal. The qualitative trend is visible in the heatmaps and bar plots even without the tests. The concern is that the headline claims 'statistically significant' and that claim is currently non-reproducible. The paper is a solid empirical contribution for a workshop or applied venue, but the stats section needs to be rewritten from scratch or the significance language dropped.\n\nMy recommendation: engage with it. Send it to peer review. The reviewers can push for the statistical details and a corrected table/conclusion. I'd cite it in my own work if the test details get sorted; as posted, I would steer clear of citing the significance counts.","headline":"Useful empirical design rule for telecom KD, but the significance tests backing the headline are too under-specified to trust as reported.","tokens_in":8675,"tokens_out":6347,"would_cite":true,"duration_ms":52028,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning just the teacher improves distilled QA models when teacher and student share a vocabulary, across algorithms and metrics.","keywords":["knowledge distillation","domain adaptation","question answering","telecom","supervised fine-tuning","vocabulary mismatch","dual space knowledge distillation","LLM evaluation"],"falsifier":"Re-run the 16 distillation experiments but score every test question separately, then apply the Wilcoxon signed-rank test with per-question pairing and report the sample sizes; if the number of metrics rejecting the null hypothesis for same-vocabulary teacher SFT drops well below the reported 13 of 14, the paper's central claim fails. A complementary check is replication on a larger telecom QA set: if teacher-only SFT gains disappear, the claim is specific to this dataset rather than general.","tokens_in":7757,"feed_emoji":"📡","tokens_out":5255,"duration_ms":51724,"temperature":0.7,"pith_summary":"The paper asks, when distilling a large language model into a smaller one for telecom question-answering, which model should be domain-adapted first: the teacher, the student, or both? It runs a systematic comparison of 16 combinations: teacher and/or student supervised fine-tuning, same versus different vocabulary, and vanilla versus dual-space distillation, judged by 14 metrics in three families. The central finding is that fine-tuning the teacher alone produces statistically significant gains in the distilled model when teacher and student share a vocabulary, regardless of distillation algorithm or metric family. Fine-tuning both helps on average, but the statistical significance depends on whether the teacher's vocabulary matches the student's. When no fine-tuning is feasible, using a different-vocabulary teacher with dual-space distillation is the better choice.","feed_headline":"Fine-tune the teacher, not both, for better distilled QA","feed_subtitle":"On telecom QA, teacher-only SFT gives significant gains when vocabularies match; if SFT is impossible, dual-space KD wins.","key_machinery":"The argument is carried by a factorial experiment grid: four binary choices (teacher SFT on/off, student SFT on/off, same/different vocabulary, vanilla KD/DSKD) produce 16 distillation runs, each scored on 14 metrics. Statistical significance is assessed with the Wilcoxon signed-rank test, grouping metrics to test hypotheses about training the teacher, training the student, training both, and choosing an algorithm. The DSKD algorithm, which projects teacher and student token spaces into a shared space via cross-model attention, is the mechanism that makes different-vocabulary distillation comparable to same-vocabulary distillation.","core_discovery":"The paper establishes, on the TeleQuAD telecom QA dataset, that supervised fine-tuning (SFT) of the teacher model before knowledge distillation is sufficient to improve the distilled student's performance when teacher and student share the same vocabulary, and this holds across both vanilla KD and DSKD and across all 14 evaluation metrics. When both teacher and student are fine-tuned, average performance improves on all metrics, but the improvement is statistically significant only in the same-vocabulary setting; with a different-vocabulary teacher the gains are not statistically significant, possibly due to limited training data. When SFT is infeasible, a different-vocabulary teacher distilled with DSKD is statistically better than vanilla KD. The paper also finds that the three metric groups, n-gram, embedding-based, and oracle-LLM, show aligned trends, so they do not contradict one another.","pith_inferences":["The non-significant results for different-vocabulary teachers may be a power problem caused by the small TeleQuAD training set (2,385 QA pairs), not evidence that teacher SFT has no effect; a larger dataset could reverse that conclusion.","The same-vocabulary advantage suggests that vocabulary alignment, rather than model family size, is the key factor in deciding where to apply domain adaptation before distillation.","If the finding holds across other technical domains, a practical rule emerges: when compute is scarce, fine-tune the teacher only and keep the student's tokenizer aligned with the teacher's.","A direct extension would be to test whether the DSKD advantage without SFT persists when the teacher is much larger or a mixture-of-experts model, since the paper only used 7-billion-parameter teachers."],"forward_implications":["When teacher and student share a vocabulary, spending the fine-tuning budget on the teacher alone is enough; fine-tuning the student as well does not add statistically significant gains.","When both teacher and student are fine-tuned, the choice between vanilla KD and DSKD does not matter statistically, so practitioners can pick the simpler algorithm.","When fine-tuning is not possible, choosing a different-vocabulary teacher and using DSKD is statistically better than vanilla KD.","Because the three metric groups show the same trends, cheaper n-gram metrics can serve as a proxy for holistic evaluation in this setting.","The recipe transfers to any domain-specific QA task where a labeled dataset and a same-vocabulary teacher-student pair are available."],"supporting_citations":[{"why":"Supplies the TeleQuAD telecom QA dataset used for all SFT and distillation experiments.","marker":"[24]"},{"why":"Defines Dual Space KD, the algorithm that enables distillation between different-vocabulary teacher and student models.","marker":"[17]"},{"why":"Provides the KL-divergence distillation objective used as the vanilla KD baseline.","marker":"[16]"},{"why":"Supplies LoRA, the low-rank adaptation method used for all supervised fine-tuning of teacher and student.","marker":"[27]"},{"why":"Provides the Wilcoxon signed-rank test used to determine which performance differences are statistically significant.","marker":"[25]"},{"why":"Motivates the multi-faceted RAG metric evaluation for telecom QA that shapes the 14-metric comparison.","marker":"[3]"}],"fun_headline_variants":["Teacher-only SFT improves distilled QA when vocab matches","Dual-space KD wins for different-vocab teacher in QA distillation","Match vocabulary then fine-tune teacher only for distilled QA","Teacher-only fine-tuning beats both-tuned for same-vocab QA","If teacher SFT infeasible, dual-space KD beats vanilla"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Wilcoxon signed-rank test is valid and correctly applied, including a well-defined pairing unit and enough paired samples; the paper never states the unit of pairing or the number of samples entering each test, so the reported rejection counts could be unsupported if the test assumptions are violated.","fun_headline_variants_meta":{"raw":{"variants":["Teacher-only SFT improves distilled QA when vocab matches","Dual-space KD wins for different-vocab teacher in QA distillation","Match vocabulary then fine-tune teacher only for distilled QA","Teacher-only fine-tuning beats both-tuned for same-vocab QA","If teacher SFT infeasible, dual-space KD beats vanilla"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001368,"raw_usage":{"total_tokens":5550,"prompt_tokens":953,"completion_tokens":4597,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":4509}},"tokens_in":569,"tokens_out":4597,"duration_ms":32202,"temperature":1.0,"reasoning_tokens":4509,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:36:54.101327+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 16 distillation experiments but score every test question separately, then apply the Wilcoxon signed-rank test with per-question pairing and report the sample sizes; if the number of metrics rejecting the null hypothesis for same-vocabulary teacher SFT drops well below the reported 13 of 14, the paper's central claim fails. A complementary check is replication on a larger telecom QA set: if teacher-only SFT gains disappear, the claim is specific to this dataset rather than general.","supporting_citations":[{"cited_title":"TeleQuAD: A suite of question answering datasets for the telecom domain,","cited_arxiv_id":null,"evidence_quote":"Supplies the TeleQuAD telecom QA dataset used for all SFT and distillation experiments."},{"cited_title":"Dual-space knowledge distillation for large language models,","cited_arxiv_id":null,"evidence_quote":"Defines Dual Space KD, the algorithm that enables distillation between different-vocabulary teacher and student models."},{"cited_title":"Knowledge distillation from internal representa- tions,","cited_arxiv_id":null,"evidence_quote":"Provides the KL-divergence distillation objective used as the vanilla KD baseline."},{"cited_title":"A generalized wilcoxon test for comparing arbitrarily singly-censored samples,","cited_arxiv_id":null,"evidence_quote":"Provides the Wilcoxon signed-rank test used to determine which performance differences are statistically significant."},{"cited_title":"Evaluation of RAG metrics for question answering in the telecom domain,","cited_arxiv_id":null,"evidence_quote":"Motivates the multi-faceted RAG metric evaluation for telecom QA that shapes the 14-metric comparison."}],"review_version":1}