REVIEW 5 major objections 6 minor 13 references
PEFT of SLM for Telecommunications Customer Support: A Comparative Study of LoRA Configurations with Energy Consumption Analysis
T0 review · 5 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read The LoRA configuration with the highest validation loss produced the best chatbot responses, while the lowest-loss configuration ranked only 6th–7th—validation loss alone is a poor selector of conversational quality.
desk verdict A useful practitioner study with a striking loss-vs-quality divergence, but the headline claim is shakier than the abstract suggests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument rests on a comparison grid: 16 LoRA configurations that hold the base model, data, and training procedure fixed while varying learning rate, batch size, LoRA rank (16 vs 32), and the set of target modules (query/value; query/key/value/output; plus gate/up/down feed-forward layers). Low-Rank Adaptation (LoRA) adds trainable low-rank matrices to selected layers and freezes the rest. The divergence claim is produced by running the same two evaluation channels side by side—validation loss/perplexity on a 1,500-example held-out set, and complete rankings from two LLM-as-a-judge systems over nine prompts—with energy consumption measured as a third axis. The overfitting explanation is
What would settle it
A panel of telecom support specialists and customers rates the lowest-loss and highest-loss configurations on a fresh set of 50–100 held-out prompts, or a different LLM judge ranks them. If the highest-loss configuration no longer ranks above the lowest-loss one, the claimed divergence is refuted; equally, if the ordering flips under a new judge or prompt set, the result is shown to be judge- or prompt-dependent.
Extended reading notes
Core claim
On its own terms, the paper's central discovery is a divergence between quantitative and qualitative performance for fine-tuned conversational models. Among 16 LoRA configurations trained on the same 3-billion-parameter model and the same synthetic telecom-support dataset, the configuration with the highest validation loss (0.6807), which adapts only the query and value attention projections at rank 16, was ranked first by two independent LLM judges; the configuration with the lowest validation loss (0.5024), which adapts all seven attention and feed-forward modules, ranked 6th–7th. The paper interprets this as evidence that validation loss reflects token-level fit and can reward overfitting
Load-bearing premise
The load-bearing assumption is that the two LLM judges' rankings on nine prompts faithfully represent the quality real telecom customers perceive; the paper does not validate the judges against human raters and defers human evaluation to future work.
Editorial extensions
If this is right
- For practitioners selecting among fine-tuning runs, the lowest validation loss should not be the deciding factor; the paper recommends adding qualitative (human or LLM-judge) evaluation to configuration-selection pipelines.
- Adapting all seven attention and feed-forward modules with LoRA produces the biggest reduction in validation loss, but on a simple, short-example dataset this added capacity can overfit; attention-only adaptation may yield more natural responses.
- A LoRA rank of 16 was sufficient in this setting; rank 32 gave diminishing or negative returns on both loss and qualitative ranking.
- Training energy varied by roughly a factor of five across configurations; faster convergence can make configurations that touch more modules cheaper than ones that touch fewer.
- Validation loss and perplexity measure token-level fit and do not capture naturalness, appropriateness, or calibration of responses.
Reading between the lines
- If this divergence generalizes beyond the 9 evaluation prompts, it suggests that early stopping on validation loss—a standard practice—may actively select worse conversational models; this could be tested by varying the evaluation set while keeping the training procedure fixed.
- The combinatorial data-generation scheme (glossary terms × causes × contexts) is portable to other support domains, but the rank-reversal it produced in this study may not persist; replication with other base models and domains would clarify whether the divergence is a property of LoRA/telecom data or a general phenomenon.
- A natural next experiment is to use the two divergent configurations (worst-loss vs best-loss) and compare them on human-rated preference and on task-specific metrics such as resolution success, since LLM-judge rankings may not track actual problem-solving.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a systematic study of 16 LoRA configurations applied to Qwen2.5-3B for telecommunications customer support. A synthetic dataset of roughly 30,000 examples is generated by factorizing 52 domain terms across problem causes and usage contexts, then expanded with Gemini 2.0 Flash. Each configuration is evaluated quantitatively (validation loss, perplexity), energetically (GPU-only energy via nvidia-smi), and qualitatively via LLM-as-a-judge rankings from GPT-5.2 and Claude 4.5 Sonnet. The paper's central claim is a divergence between quantitative and qualitative performance: the configuration with the lowest validation loss (config 8, loss 0.5024) ranks only 6th–7th with both judges, while the configuration with the highest loss (config 2, loss 0.6807) ranks 1st with both judges. Secondary findings concern target-module selection dominating loss reduction over LoRA rank, and a 5× energy variation across configurations. The paper concludes that validation loss alone is insufficient for selecting among fine-tuning configurations for conversational deployment.
Significance. If the central divergence claim were convincingly established, the paper would be a useful cautionary data point for PEFT practitioners, showing that validation loss can disagree with perceived response quality and arguing for multi-metric evaluation. The paper has concrete strengths: a systematic configuration grid with controlled architecture and weight count, explicit GPU energy measurements, and a correctly computed and replicable Spearman rank correlation (ρ = 0.843, t = 6.07) for inter-judge agreement. The synthetic-data construction method is clearly described and potentially reusable. However, the main claim is currently supported only by uncalibrated LLM judges with a 9-prompt, single-run evaluation, and the paper's own data do not support the stronger 'consistently rank better' phrasing. The conclusion about validation loss being unreliable for human-perceived quality is therefore not yet established, though it is a plausible and testable hypothesis.
major comments (5)
- [§5.3, Table 5; also Abstract] The claim that 'configurations with higher validation loss consistently rank better in qualitative assessments' is not supported by the data in Table 5. Computing Spearman's rank correlation between validation-loss rank and GPT-5.2 rank across the 16 configurations gives ρ ≈ −0.28, a weak, non-significant relationship. The headline divergence rests almost entirely on the extreme pair: configuration 2 (worst loss, best qualitative rank) versus configuration 8 (best loss, middle qualitative rank). Many high-loss configurations rank poorly (e.g., configuration 15, loss 0.6694, ranks 14th; configuration 14, loss 0.6700, ranks 13th), and several mid-loss configurations rank near the top (configurations 3 and 4). The abstract and introduction should be rewritten to state the actual pattern: the best-loss configuration does not rank first, and the worst-loss configuration does, but there is no
- [§4.3, §6.1, Abstract] The qualitative evaluation uses GPT-5.2 and Claude 4.5 Sonnet as judges, with a single holistic ranking over only 9 handpicked prompts, and no per-prompt scores or sampling variance. The paper calls the judges 'human-aligned' in the abstract, but no human validation is presented; §6.1 explicitly defers human evaluation to future work. As a result, the central divergence claim is established only for LLM-judge preferences, not for human-perceived conversational quality. This is load-bearing because the paper's main conclusion is that validation loss is unreliable for selecting deployment-ready conversational systems. The authors should either add a human evaluation (even small-scale, e.g., a few expert raters) or substantially weaken the wording, for example by stating that 'validation loss diverges from LLM-judge rankings' and flagging the absence of human validation as a limitation in t
- [§5.3.2, Figures 1–2] The three hypotheses for the divergence (token-level over-optimization, memorization/generation flexibility, evaluation distribution shift) are presented as explanations, but none is directly tested. The discussion of a 'sharp drop' in training loss for configuration 8 and a 'gradual' decrease for configuration 2 relies on Figures 1 and 2, but the figures are not included in the manuscript text and no quantitative values or overfitting metrics (e.g., train-validation gap, response-length statistics, n-gram diversity) are reported. These paragraphs should be explicitly labeled as speculative, or supplemented with concrete evidence from the training curves and generations. Otherwise, the causal narrative risks over-interpreting a single seeded run.
- [§4.1, §4.3, §5.2] All experiments are single-run with a fixed seed (42), and the qualitative evaluation consists of one ranking per judge over one set of 9 prompts. Adjacent validation-loss values are very close (e.g., configuration 6 at 0.6228 vs. configuration 7 at 0.6235 in Table 2; configuration 1 at 0.6532 vs. configuration 3 at 0.6526), so rank orderings could easily change across seeds, batch orders, or judge calls. The paper reports no confidence intervals, bootstrap errors, or repeated-measure stability. This is particularly important for the quantitative claims about target-module dominance (§5.1.1) and for the divergence claim. At minimum, the authors should acknowledge this fragility and avoid language like 'consistently' and 'striking divergence' without uncertainty quantification.
- [§5.4, Figure 3] The energy analysis reports GPU-only power from nvidia-smi at 1-second intervals, with no repeated measurements. The '5× variation' comparison is confounded by learning rate and early stopping: configuration 10 (LR 2e-4, 7 modules) converges in 1h00m and uses 284 Wh, while configuration 1 (LR 1e-4, 2 modules) runs 2h46m and uses 861 Wh; this is an early-stopping artifact at least as much as a module-coverage effect. Additionally, the 'Combined Quality Score' used in Figure 3 is an ad hoc average of normalized GPT rank, normalized Claude rank, and normalized loss with equal weights; the choice of weights is arbitrary and can change the Pareto conclusions. The authors should present the energy data as descriptive measurements, not as a controlled comparative finding, and should either justify the combined-score weights or remove the metric.
minor comments (6)
- [Table 5] The row for configuration 2 reads '0.68071 1' — the validation loss and the two rank columns are not separated. It should be '0.6807 1 1'.
- [Table 4] The columns are labeled 'GPT 5.2' and 'Claude 4.5 Sonnet', but the rows pair the model at GPT rank i with the model at Claude rank i, and the d column mixes the two possible difference definitions. For example, row 'configuration 5 configuration 1' shows d=3, but configuration 5's GPT rank is 5 and Claude rank is 8 (d=−3), while configuration 1's GPT rank is 6 and Claude rank is 5 (d=1). The final ρ and t values are correct, but the table's intermediate columns are misleading and should be reorganized, e.g., by listing each model's two ranks explicitly.
- [Abstract / §1] The phrase 'human-aligned judges' is used without supporting evidence. Recommend 'LLM judges' or 'LLM-as-a-judge' throughout, or add a human-validation experiment that justifies the term.
- [§5.4, Figure 3] The definition of the combined quality score should state how each ranking is normalized (e.g., (17−rank)/16) and why equal weighting was chosen. As written, a reader cannot reproduce the y-axis values.
- [§3.1.1 / §3.1.3] The line '52×10×3 ≈ 1,560 distinct problem scenarios' treats the approximate counts of causes and contexts as exact. If the actual numbers differ, please report the exact counts. Also clarify whether all 1,560 scenarios are covered in the 30,000 generated examples or only a subset.
- [General] The manuscript does not include a data or code availability statement. Given the emphasis on reproducibility of energy measurements and the synthetic-data pipeline, a statement (even 'available on request') would be helpful.
Circularity Check
No circularity found: the paper's central divergence and energy results are empirical observations, not derived or fitted claims.
full rationale
The paper's main claims are empirical measurements: validation loss/perplexity from fine-tuning runs, LLM-judge rankings from independent GPT-5.2 and Claude 4.5 Sonnet evaluations, and energy consumption from nvidia-smi polling. The headline divergence between validation loss and qualitative ranking is read directly off Table 5; it is not computed from a fitted parameter or from an equation that embeds the conclusion. No configuration is selected using the qualitative rankings and then 'predicted' to rank well. The combined quality score in §5.4 is an ad hoc normalized index that mixes loss and judge rankings, but it is used only for the energy trade-off figure and is not load-bearing for the main divergence claim. The paper cites external prior work for LoRA, synthetic data, and LLM-as-a-judge; there is no load-bearing self-citation chain, and the cited LLM-as-a-judge literature is independent of the present authors. The absence of human validation of the LLM judges is a real limitation (and is honestly acknowledged in §6.1 as future work), but it is a validity concern, not circularity: the paper observes what two LLM judges prefer and claims a divergence from loss, without assuming the judges' alignment in the derivation. No step reduces to its own inputs by construction.
Assumptions & free parameters
free parameters (1)
- Combined quality score weights =
1/3 each for normalized GPT rank, normalized Claude rank, normalized loss
assumptions (4)
- standard math Spearman rank correlation and Student's t-test are valid for the 17-item ranking comparison.
- domain assumption The Gemini-generated synthetic dataset, built from 52 glossary terms × 10 causes × 3 contexts, adequately represents real telecom customer-support conversations.
- domain assumption LLM-as-a-judge rankings by GPT-5.2 and Claude 4.5 Sonnet approximate human-perceived response quality.
- domain assumption Retaining the checkpoint with the lowest validation loss is the right checkpoint for qualitative evaluation.
Cite this review
Pith. "Pith review of PEFT of SLM for Telecommunications Customer Support: A Comparative Study of LoRA Configurations with Energy Consumption Analysis." pith.science (2026). https://pith.science/paper/OBXJSGHN
@misc{pith2026260605176,
author = {Pith},
title = {Pith review of: PEFT of SLM for Telecommunications Customer Support: A Comparative Study of LoRA Configurations with Energy Consumption Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/OBXJSGHN}},
note = {Machine review of arXiv:2606.05176}
}
read the original abstract
While large language models (LLMs) show strong performance in natural language understanding and generation, their evaluation and adaptation to domain-specific constraints in telecommunications customer support remain limited. In addition, data sovereignty, regulatory constraints, and the handling of sensitive customer and network information complicate the use of externally hosted foundation models in this domain. We present a systematic study of parameter-efficient fine-tuning (PEFT) using Low-Rank Adaptation (LoRA) applied to Qwen2.5-3B to build a domain-specific conversational assistant. We introduce a combinatorial synthetic data generation approach based on a glossary of 52 industry-specific terms, producing approximately 30,000 training examples across 1,560 distinct problem scenarios via a generative pipeline powered by Gemini 2.0 Flash. We evaluate 16 LoRA configurations by varying hyperparameters and target modules. Our evaluation extends beyond standard metrics by incorporating energy consumption analysis and qualitative assessment using an LLM-as-a-judge framework with GPT-5.2 and Claude 4.5 Sonnet. Results show a clear divergence between quantitative and qualitative performance: models achieving the lowest validation loss do not necessarily obtain the best human-aligned rankings. The best validation loss (0.5024) ranks only 6th-7th in qualitative evaluation, while the worst loss (0.6807) ranks first according to both judges. This work contributes (1) a combinatorial method for synthetic dataset construction, (2) insights into the impact of target module selection for LoRA injection, (3) evidence that validation loss alone is insufficient for selecting fine-tuning configurations in conversational AI, and (4) an energy-performance trade-off analysis for sustainable LLM deployment.
Figures
Reference graph
Works this paper leans on
-
[4]
The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691,
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691,
-
[5]
Prefix-tuning: Optimizing continuous prompts for generation.arXiv preprint arXiv:2101.00190,
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation.arXiv preprint arXiv:2101.00190,
-
[6]
Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,
-
[9]
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Han- naneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions.arXiv preprint arXiv:2212.10560,
-
[10]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, R´ emi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing.Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45,
2020
-
[11]
Qwen2.5: A party of foundation models.arXiv preprint arXiv:2412.15115,
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2.5: A party of foundation models.arXiv preprint arXiv:2412.15115,
-
[12]
Adaptive budget allocation for parameter-efficient fine-tuning.arXiv preprint arXiv:2303.10512,
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning.arXiv preprint arXiv:2303.10512,
-
[13]
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.arXiv preprint arXiv:2306.05685,
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.arXiv preprint arXiv:2306.05685,
Show all 13 references
-
[2020]
Energy and policy considerations for deep learning in nlp.arXiv preprint arXiv:1906.02243,
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp.arXiv preprint arXiv:1906.02243,
1906 arXiv
-
[2021]
Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685,
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685,
-
[2022]
Carbon emissions and large neural network training.arXiv preprint arXiv:2104.10350,
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training.arXiv preprint arXiv:2104.10350,
-
[2023]
Transformer feed-forward layers are key- value memories.arXiv preprint arXiv:2012.14913,
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key- value memories.arXiv preprint arXiv:2012.14913,
2012 arXiv
-
[2024]
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer
Technical Report. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms.arXiv preprint arXiv:2305.14314,
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.