Pith. sign in

REVIEW 4 major objections 5 minor 27 references

Knowledge Distillation of Domain-adapted LLMs for Question-Answering in Telecom

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning just the teacher improves distilled QA models when teacher and student share a vocabulary, across algorithms and metrics.

desk verdict Useful empirical design rule for telecom KD, but the significance tests backing the headline are too under-specified to trust as reported. read the letter →

arxiv 2504.20000 v1 pith:DH4NKZRS submitted 2025-04-28 cs.CL cs.IRcs.LG

Rishika Sen , Sujoy Roychowdhury , Sumit Soman , H. G. Ranjani , Srikhetra Mohanty This is my paper · ORCID
classification cs.CLcs.IRcs.LG
keywords knowledgedistillationdomainadaptationquestionansweringtelecomsupervisedfine-tuningvocabularymismatchdualspaceLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks, when distilling a large language model into a smaller one for telecom question-answering, which model should be domain-adapted first: the teacher, the student, or both? It runs a systematic comparison of 16 combinations: teacher and/or student supervised fine-tuning, same versus different vocabulary, and vanilla versus dual-space distillation, judged by 14 metrics in three families. The central finding is that fine-tuning the teacher alone produces statistically significant gains in the distilled model when teacher and student share a vocabulary, regardless of distillation algorithm or metric family. Fine-tuning both helps on average, but the statistical significance depends on whether the teacher's vocabulary matches the student's. When no fine-tuning is feasible, using a different-vocabulary teacher with dual-space distillation is the better choice.

What carries the argument

The argument is carried by a factorial experiment grid: four binary choices (teacher SFT on/off, student SFT on/off, same/different vocabulary, vanilla KD/DSKD) produce 16 distillation runs, each scored on 14 metrics. Statistical significance is assessed with the Wilcoxon signed-rank test, grouping metrics to test hypotheses about training the teacher, training the student, training both, and choosing an algorithm. The DSKD algorithm, which projects teacher and student token spaces into a shared space via cross-model attention, is the mechanism that makes different-vocabulary distillation comparable to same-vocabulary distillation.

What would settle it

Re-run the 16 distillation experiments but score every test question separately, then apply the Wilcoxon signed-rank test with per-question pairing and report the sample sizes; if the number of metrics rejecting the null hypothesis for same-vocabulary teacher SFT drops well below the reported 13 of 14, the paper's central claim fails. A complementary check is replication on a larger telecom QA set: if teacher-only SFT gains disappear, the claim is specific to this dataset rather than general.

Watch

Extended reading notes

Core claim

The paper establishes, on the TeleQuAD telecom QA dataset, that supervised fine-tuning (SFT) of the teacher model before knowledge distillation is sufficient to improve the distilled student's performance when teacher and student share the same vocabulary, and this holds across both vanilla KD and DSKD and across all 14 evaluation metrics. When both teacher and student are fine-tuned, average performance improves on all metrics, but the improvement is statistically significant only in the same-vocabulary setting; with a different-vocabulary teacher the gains are not statistically significant, possibly due to limited training data. When SFT is infeasible, a different-vocabulary teacher distilled with DSKD is statistically better than vanilla KD. The paper also finds that the three metric groups, n-gram, embedding-based, and oracle-LLM, show aligned trends, so they do not contradict one another.

Load-bearing premise

The load-bearing premise is that the Wilcoxon signed-rank test is valid and correctly applied, including a well-defined pairing unit and enough paired samples; the paper never states the unit of pairing or the number of samples entering each test, so the reported rejection counts could be unsupported if the test assumptions are violated.

Editorial extensions

If this is right

  • When teacher and student share a vocabulary, spending the fine-tuning budget on the teacher alone is enough; fine-tuning the student as well does not add statistically significant gains.
  • When both teacher and student are fine-tuned, the choice between vanilla KD and DSKD does not matter statistically, so practitioners can pick the simpler algorithm.
  • When fine-tuning is not possible, choosing a different-vocabulary teacher and using DSKD is statistically better than vanilla KD.
  • Because the three metric groups show the same trends, cheaper n-gram metrics can serve as a proxy for holistic evaluation in this setting.
  • The recipe transfers to any domain-specific QA task where a labeled dataset and a same-vocabulary teacher-student pair are available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The non-significant results for different-vocabulary teachers may be a power problem caused by the small TeleQuAD training set (2,385 QA pairs), not evidence that teacher SFT has no effect; a larger dataset could reverse that conclusion.
  • The same-vocabulary advantage suggests that vocabulary alignment, rather than model family size, is the key factor in deciding where to apply domain adaptation before distillation.
  • If the finding holds across other technical domains, a practical rule emerges: when compute is scarce, fine-tune the teacher only and keep the student's tokenizer aligned with the teacher's.
  • A direct extension would be to test whether the DSKD advantage without SFT persists when the teacher is much larger or a mixture-of-experts model, since the paper only used 7-billion-parameter teachers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper investigates whether supervised fine-tuning (SFT) of the teacher, the student, or both, before knowledge distillation (KD) improves question-answering performance in the telecom domain. Using Llama-7B and Mistral-7B as teachers and TinyLlama-1.1B as student, the authors compare vanilla KD and Dual-Space KD (DSKD) under same-vocabulary and different-vocabulary conditions, evaluating outputs on 14 N-gram, embedding, and LLM-based metrics. Statistical significance is assessed through Wilcoxon signed-rank tests on hypotheses about SFT, vocabulary, and algorithm effects. The paper claims that SFT of the teacher improves distilled performance when teacher and student share a vocabulary, and that SFT of both generally helps, with significance depending on vocabulary.

Significance. If the results are reproducible, this is a useful empirical contribution: it systematically addresses a practical question—whether to domain-adapt the teacher, the student, or both before KD—and it evaluates a broader metric suite than is typical in KD papers. The experimental design covers a useful matrix of conditions (two teachers, two algorithms, two SFT choices for each of teacher and student). The paper also provides a concrete recommendation for practitioners: when SFT is infeasible, different-vocabulary DSKD is preferred. These findings would be valuable to the NLP/KD community, provided the statistical evidence is presented rigorously. The main weakness is that the statistical analysis, which underpins every significance claim, is under-specified and partly internally inconsistent.

major comments (4)
  1. [Section 2.2 and Table 3] The statistical test is not fully specified. The paper states that a Wilcoxon signed-rank test is performed per metric, but it never defines the unit of paired observations. A signed-rank test requires paired samples; the 14 metrics appear to be single aggregate values per condition, so the pairing cannot be across metrics. If the intended pairing is across the 597 test questions, that should be stated explicitly and the degrees of freedom reported. If the pairing is across runs, the number of runs is never given. Without this information, the rejection counts in Table 3 cannot be verified, and Table 3 is the sole basis for the significance-based conclusions in Sections 3.1, 3.2, and 3.5.
  2. [Section 4] The conclusion text contradicts the reported results. Section 4 states: "when teacher is Mistral, training using SFT improves performance across vocabulary and algorithms; improvement with SFT of teacher and/or student when models have same vocabulary is significant." Mistral is the different-vocabulary teacher, and Table 3 shows 0 rejections out of 14 for all Mistral SFT hypotheses (H_S_train, H_T_train, H_T,S_train). The significant improvement is reported for the same-vocabulary Llama teacher. This misattribution affects the paper's central practical recommendation and must be corrected.
  3. [Table 3] Table 3 is incomplete for the algorithm hypotheses. The table has four columns (Llama-V, Llama-D, Mistral-V, Mistral-D), but the rows H_T,S_Alg and H_B_Alg list only two values each ("0 1" and "10 14", respectively). The text in Section 3.3.1 refers to "10 out of 14 and 14 out of 14" without clarifying which columns those numbers correspond to, so the reader cannot reconstruct the evidence for the DSKD-over-vanilla claim. All four entries should be reported.
  4. [Section 2.2] No multiple-comparison correction is applied. The paper performs 24 hypothesis definitions (12 H-Train, 8 H-SFT, 4 H-Algo), each tested on 14 metrics, for a total of 336 Wilcoxon tests. At α=0.05, one would expect about 0.7 false rejections per hypothesis over 14 metrics under the global null. The claim that "13 out of 14" metrics reject for H_T_train may partly reflect multiplicity. Adjusted p-values, a stated significance level, or an explicit justification for not correcting should be provided.
minor comments (5)
  1. [Figure 4 caption] The caption refers to "the heatmap in Fig. ??", an unresolved cross-reference that should be fixed.
  2. [Section 3.1.2] The sentence "We observe that training improves results Fig. 4" is missing a preposition and a period; it should read "training improves results in Fig. 4".
  3. [Section 1.1] The phrase "in-lieu of these gaps" is non-standard; "to address these gaps" would be clearer.
  4. [Table 2] The training configuration reports LoRA rank, alpha, dropout, and learning rate, but the KD hyperparameters (e.g., distillation temperature, loss weighting, batch sizes, number of epochs for KD) are not reported, which limits reproducibility.
  5. [General] No random seeds, number of runs, or confidence intervals are reported; the paper should clarify whether each metric value is a single run or an average over multiple runs.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical study with external benchmarks; claims do not reduce to their inputs.

full rationale

This is an empirical study, not a derivation chain, so the circularity patterns do not apply. The quantities being compared (BLEU, ROUGE, BERTScore, RAGAs, cosine similarity) are external metrics defined independently of the paper's SFT choices; the hypotheses in Section 2.2 compare measured performance of trained models with direct, non-parametric Wilcoxon comparisons. No parameter is fitted to one subset of data and then 'predicted' on a closely related quantity: the SFT models are trained once and evaluated on the fixed TeleQuAD test split. The self-citations (TeleQuAD dataset [24], RAG evaluation [3], telecom LLM observations [1]) are used as data source and method background, not as a uniqueness theorem or a definition that forces the conclusion. The omission of pairing unit and sample size for the Wilcoxon tests is a reproducibility or correctness concern about Table 3, not an equivalence-by-construction between input and output; the claims would stand or fall on the data if properly reported. There is therefore no step where Eq. X equals Eq. Y by definition or where a fitted parameter is renamed a prediction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper is empirical, so no derivational free parameters exist; the hyperparameters listed are experimental choices. The central claims rest on assumptions about the statistical test, metric validity, dataset representativeness, and the correct implementation of DSKD.

free parameters (5)
  • LoRA rank = 256
    Chosen for SFT in Table 2; standard value, not fitted to the target result.
  • LoRA alpha = 8
    SFT hyperparameter from Table 2; hand-chosen.
  • LoRA dropout = 0.1
    SFT hyperparameter from Table 2; standard.
  • Learning rate = 0.001 with cosine decay
    SFT hyperparameter from Table 2; affects distillation outcome but is not fitted to conclusions.
  • Early stopping patience = 3 epochs
    SFT hyperparameter from Table 2; influences checkpoint selection.
assumptions (5)
  • standard math Wilcoxon signed-rank test is a valid test for the paired comparisons in the experimental design.
    Invoked in Section 2.2. The test requires paired observations; the paper does not specify the pairing unit (test questions or metric values), so validity is assumed.
  • domain assumption The 14 metrics (N-gram, embedding, and LLM-based) are reliable indicators of QA answer quality.
    Section 1.3 lists these metrics without validating them against human judgment for the telecom domain.
  • domain assumption TeleQuAD is representative of telecom domain QA tasks.
    Section 2.3: dataset from 3GPP Rel 15, only 2385 training pairs, so conclusions may not generalize.
  • domain assumption Llama-7b, Mistral-7b, and TinyLlama are representative of larger and smaller LLM families.
    Section 2.4 uses two 7B teachers and one 1.1B student; results may not transfer to other model families or sizes.
  • domain assumption DSKD implementation follows the original paper [17] and is correctly integrated.
    The paper describes DSKD briefly via cross-model attention; no code is supplied to verify correct implementation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Knowledge Distillation of Domain-adapted LLMs for Question-Answering in Telecom." pith.science (2026). https://pith.science/paper/DH4NKZRS

@misc{pith2026250420000,
  author       = {Pith},
  title        = {Pith review of: Knowledge Distillation of Domain-adapted LLMs for Question-Answering in Telecom},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DH4NKZRS}},
  note         = {Machine review of arXiv:2504.20000}
}
read the original abstract

Knowledge Distillation (KD) is one of the approaches to reduce the size of Large Language Models (LLMs). A LLM with smaller number of model parameters (student) is trained to mimic the performance of a LLM of a larger size (teacher model) on a specific task. For domain-specific tasks, it is not clear if teacher or student model, or both, must be considered for domain adaptation. In this work, we study this problem from perspective of telecom domain Question-Answering (QA) task. We systematically experiment with Supervised Fine-tuning (SFT) of teacher only, SFT of student only and SFT of both prior to KD. We design experiments to study the impact of vocabulary (same and different) and KD algorithms (vanilla KD and Dual Space KD, DSKD) on the distilled model. Multi-faceted evaluation of the distillation using 14 different metrics (N-gram, embedding and LLM-based metrics) is considered. Experimental results show that SFT of teacher improves performance of distilled model when both models have same vocabulary, irrespective of algorithm and metrics. Overall, SFT of both teacher and student results in better performance across all metrics, although the statistical significance of the same depends on the vocabulary of the teacher models.

Figures

Figures reproduced from arXiv: 2504.20000 by the authors.

Figure 1
Figure 1. A schematic representation of experiments consisting of the choice of SFT for teacher student, the choice of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Schematic representation of different choices based on which we conduct Hypothesis tests [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance on 14 metrics for various combinations of [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Group-wise average of performance metrics from the heatmap in Fig. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 15 canonical work pages

  1. [1]

    Observations on LLMs for telecom domain: capabilities and limitations,

    S. Soman and H. G. Ranjani, “Observations on LLMs for telecom domain: capabilities and limitations,” in Proceedings of the Third International Conference on AI-ML Systems, 2023, pp. 1–5

    work page 2023

  2. [2]

    Understanding telecom language through large language models,

    L. Bariah, H. Zou, Q. Zhao, B. Mouhouche, F. Bader, and M. Debbah, “Understanding telecom language through large language models,” in GLOBECOM 2023-2023 IEEE Global Communications Conference. IEEE, 2023, pp. 6542–6547

    work page 2023

  3. [3]

    Evaluation of RAG metrics for question answering in the telecom domain,

    S. Roychowdhury, S. Soman, H. G. Ranjani, N. Gunda, V . Chhabra, and S. K. Bala, “Evaluation of RAG metrics for question answering in the telecom domain,” in ICML 2024 Workshop on Foundation Models in the Wild , 2024. 8

    work page 2024

  4. [4]

    Using large language models to understand telecom standards,

    A. Karapantelakis, M. Thakur, A. Nikou, F. Moradi, C. Olrog, F. Gaim, H. Holm, D. D. Nimara, and V . Huang, “Using large language models to understand telecom standards,” in 2024 IEEE International Conference on Machine Learning for Communication and Networking (ICMLCN), 2024, pp. 440–446

    work page 2024

  5. [5]

    Telecomgpt: A framework to build telecom-specfic large language models,

    H. Zou, Q. Zhao, Y . Tian, L. Bariah, F. Bader, T. Lestable, and M. Debbah, “Telecomgpt: A framework to build telecom-specfic large language models,” arXiv preprint arXiv:2407.09424, 2024

    arXiv 2024

  6. [6]

    Telecom language models: Must they be large?

    N. Piovesan, A. De Domenico, and F. Ayed, “Telecom language models: Must they be large?” arXiv preprint arXiv:2403.04666, 2024

    arXiv 2024

  7. [7]

    Large language models for telecom: Forthcoming impact on the industry,

    A. Maatouk, N. Piovesan, F. Ayed, A. De Domenico, and M. Debbah, “Large language models for telecom: Forthcoming impact on the industry,”IEEE Communications Magazine, 2024

    work page 2024

  8. [8]

    It’s not just size that matters: Small language models are also few-shot learners,

    T. Schick and H. Sch ¨utze, “It’s not just size that matters: Small language models are also few-shot learners,” arXiv preprint arXiv:2009.07118, 2020

    arXiv 2009

Show all 27 references
  1. [9]

    Revisiting block-based quantisation: What is important for sub-8-bit llm inference?

    C. Zhang, J. Cheng, I. Shumailov, G. A. Constantinides, and Y . Zhao, “Revisiting block-based quantisation: What is important for sub-8-bit llm inference?” arXiv preprint arXiv:2310.05079, 2023

    2023 arXiv

  2. [10]

    Llm-pruner: On the structural pruning of large language models,

    X. Ma, G. Fang, and X. Wang, “Llm-pruner: On the structural pruning of large language models,” Advances in neural information processing systems, vol. 36, pp. 21 702–21 720, 2023

    2023

  3. [11]

    Knowledge distillation: A survey,

    J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,”International Journal of Computer Vision, vol. 129, no. 6, pp. 1789–1819, 2021

    2021

  4. [12]

    A survey on knowledge distillation of large language models,

    X. Xu, M. Li, C. Tao, T. Shen, R. Cheng, J. Li, C. Xu, D. Tao, and T. Zhou, “A survey on knowledge distillation of large language models,” arXiv preprint arXiv:2402.13116, 2024

    2024 arXiv

  5. [13]

    Distilling the knowledge in a neural network,

    G. E. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” ArXiv, vol. abs/1503.02531, 2015. [Online]. Available: https://api.semanticscholar.org/CorpusID:7200347

    2015 arXiv

  6. [14]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017

    2017

  7. [15]

    Knowledge distillation of russian language models with reduction of vocabulary,

    A. Kolesnikova, Y . Kuratov, V . Konovalov, and M. Burtsev, “Knowledge distillation of russian language models with reduction of vocabulary,”arXiv preprint arXiv:2205.02340, 2022

    2022 arXiv

  8. [16]

    Knowledge distillation from internal representa- tions,

    G. Aguilar, Y . Ling, Y . Zhang, B. Yao, X. Fan, and C. Guo, “Knowledge distillation from internal representa- tions,” in Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 05, 2020, pp. 7350–7357

    2020

  9. [17]

    Dual-space knowledge distillation for large language models,

    S. Zhang, X. Zhang, Z. Sun, Y . Chen, and J. Xu, “Dual-space knowledge distillation for large language models,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP),Miami Florida USA, 2024

    2024

  10. [18]

    Evalullm: Llm assisted evaluation of gen- erative outputs,

    M. Desmond, Z. Ashktorab, Q. Pan, C. Dugan, and J. M. Johnson, “Evalullm: Llm assisted evaluation of gen- erative outputs,” in Companion Proceedings of the 29th International Conference on Intelligent User Interfaces, 2024, pp. 30–32

    2024

  11. [19]

    On the evaluation of neural code summarization,

    E. Shi, Y . Wang, L. Du, J. Chen, S. Han, H. Zhang, D. Zhang, and H. Sun, “On the evaluation of neural code summarization,” in Proceedings of the 44th international conference on software engineering , 2022, pp. 1597– 1608

    2022

  12. [20]

    ROUGE: A package for automatic evaluation of summaries,

    C.-Y . Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81. [Online]. Available: https://aclanthology.org/W04-1013

    2004

  13. [21]

    Making monolingual sentence embeddings multilingual using knowledge distilla- tion,

    N. Reimers and I. Gurevych, “Making monolingual sentence embeddings multilingual using knowledge distilla- tion,” arXiv preprint arXiv:2004.09813, 2020

    2004 arXiv

  14. [22]

    Bertscore: Evaluating text generation with bert,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,” arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  15. [23]

    Ragas: Automated evaluation of retrieval augmented generation,

    S. Es, J. James, L. E. Anke, and S. Schockaert, “Ragas: Automated evaluation of retrieval augmented generation,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, 2024, pp. 150–158

    2024

  16. [24]

    TeleQuAD: A suite of question answering datasets for the telecom domain,

    F. Gebre, H. Holm, M. Gunnarsson, D. Nimara, J. Wei, V . Huang, A. Sharma, and H. G. Ranjani, “TeleQuAD: A suite of question answering datasets for the telecom domain,” 2025. [Online]. Available: https://github.com/EricssonResearch/TeleQuAD

    2025

  17. [25]

    A generalized wilcoxon test for comparing arbitrarily singly-censored samples,

    E. A. Gehan, “A generalized wilcoxon test for comparing arbitrarily singly-censored samples,” Biometrika, vol. 52, no. 1-2, pp. 203–224, 1965. 9

    1965

  18. [26]

    3GPP release 15,

    3GPP, “3GPP release 15,” https://www.3gpp.org/specifications-technologies/releases/release-15, 3GPP, Tech. Rep., 2019, accessed: 2024-05-19

    2019

  19. [27]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021. 10

    2021 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.