Human Grounded Evaluation of Large Language Models for Optical Network Automation

Carlos Natalino; Kiarash Rezaei; Omran Ayoub; Paolo Monti

REVIEW 4 major objections 6 minor 25 references

Human Grounded Evaluation of Large Language Models for Optical Network Automation

T0 review · 4 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper's central claim is that a reliable LLM ranking can be obtained by calibrating one judge LLM against a small human baseline and then automating the rest, producing a quality-efficiency score that puts a 12B model ahead of larger on

desk verdict A sensible human-calibrated LLM-judge pipeline with a useful QoT explanation case study, but the headline 12B result rests on an untested judge-generalization assumption. read the letter →

arxiv 2607.18068 v2 pith:GSJFHZ2K submitted 2026-07-20 cs.NI cs.AI

Kiarash Rezaei , Omran Ayoub , Paolo Monti , Carlos Natalino This is my paper

classification cs.NIcs.AI

keywords LLM-as-a-judgehuman-groundedevaluationquality-efficiencyscoreopticalnetworkautomationqualityoftransmissionexplanationexplainableAILLM

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HuGLEN is a staged evaluation pipeline that lets a network operator rank LLMs for an automation task using only a small sample of expert-rated outputs, rather than grading every generated response by hand. The pipeline asks candidate LLMs to turn SHAP explanations from a quality-of-transmission estimator into operator-friendly text, recruits four human experts to grade 40 such explanations, and then selects a second LLM as an automatic judge based on how well its scores agree with those experts. Once that judge is chosen, it scores all candidate outputs and combines quality with a parameter-count efficiency proxy into a single quality-efficiency score (QES). On this optical-network explanation task, the 12B-parameter Gemma3 model tops the QES ranking, which the authors read as evidence that mid-sized models offer the best quality-per-compute trade-off. A sympathetic reader would care because the approach promises reproducible, resource-aware LLM selection without a large labeling budget.

What carries the argument

The load-bearing mechanism is judge calibration by agreement, followed by QES aggregation. HuGLEN defines three human-centered metrics—correctness (binary), scope (binary), and usefulness (0–5)—and has four human experts score 40 explanations from DeepSeek-R1 (32B). Eight candidate LLM judges score the same explanations; Prometheus-7B-v2.0 shows the highest agreement with the experts across all three metrics and is therefore selected as the judge. It then scores all 700 candidate outputs; the scores are combined into a Composite Quality Index, normalized across models, and blended with a parameter-count efficiency factor into a single quality-efficiency score with a default weighting of 70%

What would settle it

Give the same four human experts a held-out set of explanations from every candidate LLM family, have them rate with the same rubric, and compare the selected judge's scores family by family; a material drop in agreement on any family would overturn the ranking and with it the 12B conclusion.

Watch

Extended reading notes

Core claim

On the paper's own terms, HuGLEN shows that an LLM's suitability for a network-automation task can be established by a two-stage procedure: calibrate one judge LLM against a small human baseline, then use that judge to score every candidate and rank them by a quality-efficiency score. In the demonstrated use case—translating SHAP values into explanations for optical QoT estimates—the authors find that Gemma3 (12B) achieves the highest QES, approximately 78, ahead of DeepSeek-R1 (32B) and Qwen3 (14B) despite being smaller, while 4B models lag by a wide margin in quality. The argument is that explanation quality saturates with model size and efficiency penalizes the largest models, so the swee

Load-bearing premise

The load-bearing premise is that the judge LLM selected on 40 explanations from a single model agrees with human raters just as well on all 700 outputs from the other six models.

Editorial extensions

If this is right

Operators can reduce human labeling to a small calibration set and still rank candidate LLMs reproducibly, because the judge-selection step anchors the automated scores to human judgment.
The QES ranking implies that mid-sized models around 10–15B parameters offer the best quality per unit of inference cost for this explanation task, and that the largest models deliver diminishing returns.
Because the framework is task-agnostic and uses standardized prompting, the same calibrate-then-automate procedure can be applied to other operator-facing LLM tasks without fine-tuning.
The QES weighting parameter lets an operator shift the choice toward quality or efficiency, supporting policy-driven deployment decisions rather than a single universal ranking.
The ranking is driven mainly by usefulness scores; correctness and scope are near-optimal for most models, so usefulness is where LLMs genuinely separate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not establish that the 12B winner generalizes beyond the judged sample; a natural extension is to validate the selected judge against per-family human ratings before trusting the QES ranking.
Its efficiency factor uses parameter count as a stand-in for inference cost; substituting measured latency or energy could reorder the mid-size results, especially between 12B and 14B models.
The binary correctness and scope metrics compress a lot of variation; a finer-grained error taxonomy might separate models that currently look tied at near-optimal.
The judge-selection sample is 40 explanations from one model; using a stratified sample across all candidate families would make the agreement scores more robust.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: HuGLEN is a clear, well-written pipeline for calibrating an LLM judge against a small human rating set and then ranking candidate LLMs with a quality-efficiency score. The QoT explanation use case is well chosen, and the Gemma3-12B result is a genuinely useful datapoint for optical-network QoT explanation. The thing to pay attention to is the judge-generalization assumption: the judge was calibrated on 40 explanations from DeepSeek-R1 (32B), then used to score 100 outputs from each of seven model families, with no human check on any non-DeepSeek output.

What's new: the specific composition—LLM-as-a-judge, human calibration, CQI/QES aggregation—is not groundbreaking, but applying that pipeline to XAI explanation translation for QoT estimation is new as far as I know. The QES formula is a simple weighted average with min-max normalization; that's fine, and the hand-set weights (alpha=0.7, equal metric weights) mean there is no circular fitting to the ranking. The paper is also honest about its simplifications: it uses prompt engineering rather than fine-tuning, and explicitly says parameter count is a proxy for inference cost that can be replaced with measured latency or energy.

The soft spots, in proportion: the judge-generalization issue is load-bearing and the paper doesn't address it. The authors even cite [18] showing LLM-as-a-judge can diverge from human experts in domain-knowledge tasks, so they know the risk. There are no error bars on the agreement scores or on the CQI/QES rankings, and the 'reduces human-labeling burden' claim has no baseline. The efficiency proxy (parameter count) is crude, though acknowledged. Code and artifacts are promised only upon acceptance, which makes independent verification impossible right now.

Is the central argument supported? The methodology is plausible and the pipeline is clearly specified, but the headline empirical claim (Gemma3-12B is the best quality-per-compute choice) depends entirely on the unvalidated judge agreement. That is an addressable limitation, not a fatal flaw. With a small human validation on a held-out sample from other model families (or at least per-family agreement statistics), the result would be much more convincing.

Who this is for: researchers and practitioners working on LLM evaluation for network automation, especially optical-network QoT explanation. It deserves a serious referee: the topic is timely, the writing is clear, and the limitations are of the kind that can be fixed with additional experiments. I would send it to peer review, and I would ask the authors to add at least a held-out human validation and some uncertainty quantification before final acceptance.

Referee Report

4 major / 6 minor

Summary. The paper introduces HuGLEN, a four-stage pipeline for evaluating LLMs in network-automation tasks using a human-calibrated LLM-as-a-judge and a quality-efficiency score (QES). The pipeline is demonstrated on a use case where candidate LLMs translate SHAP-based explanations of optical-network QoT predictions into operator-facing natural language. Human experts rate 40 explanations produced by a single representative model (DeepSeek-R1 32B); candidate judges are compared against these ratings; Prometheus-7B-v2.0 is selected; this judge then scores 100 outputs from each of seven candidate LLMs. The paper claims that a medium-sized model, Gemma3-12B, achieves the highest QES, balancing explanation quality and computational efficiency, and that HuGLEN reduces human-labeling burden while supporting consistent model selection.

Significance. If the claims are established, HuGLEN would be a useful, transparent addition to LLM evaluation in networking: the QES and CQI definitions are explicit, the default weights are hand-set rather than fitted to the data, and the pipeline is reproducible in principle. The choice to ground an automated judge in a small set of human ratings is sensible. However, the current evidence is not sufficient for the paper's general claims. The judge is calibrated on a single model family, no uncertainty or inter-rater reliability is reported, and the efficiency term in QES is a parameter-count proxy. These gaps directly affect the headline 12B-best result, so the contribution is significant only conditionally on additional validation.

major comments (4)

[Section IV, LLM Judge Selection] The judge is selected using 40 explanations generated by DeepSeek-R1 (32B) only, then applied to 100 outputs from each of seven model families. The paper provides no evidence that agreement measured on one model family transfers to other families. If Prometheus-7B-v2.0 has a family-correlated bias (e.g., toward a particular style, verbosity, or format), the per-model CQI differences in Fig. 6 and the resulting QES ranking in Fig. 7—including the claim that Gemma3-12B is best—could be artifacts of judge bias rather than explanation quality. This is load-bearing because the headline result depends entirely on automatic scores. Please add human ratings on outputs from at least two non-DeepSeek families, report per-family judge-human agreement, or provide other evidence that the judge does not systematically favor or penalize specific candidate families.
[Section IV, Figs. 6-7] No uncertainty quantification is provided. The human baseline is based on four experts and 40 samples, but no inter-rater reliability (e.g., Fleiss' kappa) is reported; the usefulness agreement is an inverted standard deviation without confidence intervals. The per-model CQI and QES values are point estimates without error bars, so it is unclear whether the differences among models in Figs. 6-7 are meaningful. Please report bootstrap or other confidence intervals for CQI and QES, per-sample variance, and an inter-rater reliability measure for the human ratings. Without this, the claim that HuGLEN supports 'consistent model selection' is not established.
[Section II-D2, Eqs. (3)-(5)] The efficiency factor eta_i = Pmax/P_i treats parameter count as a proxy for inference cost without measuring latency, energy, throughput, or monetary cost. The paper notes this proxy can be replaced, but the specific conclusion that 'a medium-sized LLM (12B parameters) achieves the highest QES' is derived from this proxy. The ranking in Fig. 7 is therefore not a demonstrated efficiency-quality trade-off. Please report sensitivity of the QES ranking to the default alpha=0.7 and to an alternative efficiency measure, or measure actual inference cost; otherwise, the efficiency claims should be explicitly scoped to parameter-count-based cost.
[Abstract and Conclusion] The claim that HuGLEN 'reduces the human-labeling burden' is not quantified against any baseline. The pipeline indeed uses only 40 human-rated explanations, but the paper does not compare this with full human evaluation or with any existing evaluation cost, nor does it state what burden reduction is claimed. Please either provide a concrete burden metric (e.g., number of human labels needed for a target agreement or confidence level) or explicitly limit the claim to 'uses fewer human labels than full human evaluation.'

minor comments (6)

[Section II-D2, Eq. (4)] Please clarify the normalization in Eq. (4): the denominator 'Pmax/Pmin - 1' is the range of eta; writing the expression as (eta_i - 1)/(Pmax/Pmin - 1) would avoid ambiguity.
[Fig. 3] The y-axis label is 'Agreement (%)' in (a) and (b) but 'Agreement (0 1)' in (c). Use consistent units and explain how the inverted standard deviation is normalized to 0-1.
[Section II-C] Typo: 'This procedure have' should be 'This procedure has'.
[Fig. 7] Inconsistent capitalization: 'Deepseek-R1' in the legend vs 'DeepSeek-R1' in the text.
[Section IV] The phrase 'we relied on a single LLM for this step' is ambiguous. Clarify that it refers to using a single source model to generate the 40 explanations, not a single human judge or a single judge model.
[Section IV] For reproducibility, please provide sampling details for the 40 randomly extracted explanations (e.g., random seed or procedure) and include an example of the prompt template in an appendix; the promise to release artifacts at acceptance is appreciated but not sufficient for reviewers to verify reproducibility now.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HuGLEN's QES ranking is produced by a separately calibrated judge and a hand-defined score, not forced by its inputs.

full rationale

The paper's derivation chain is self-contained and does not reduce to its inputs. CQI (Eq. 1) is a fixed weighted average of three judge-assigned metrics with equal default weights; QES (Eqs. 2-5) is a defined composite of min-max normalized CQI and a parameter-count efficiency proxy with a hand-set alpha=0.7. No parameter is fitted to the target ranking. The LLM judge (Prometheus-7B-v2.0) is selected by agreement with four human raters on 40 explanations from DeepSeek-R1 (32B), but those human labels enter only into judge selection and are not used to compute CQI/QES; the Gemma3-12B ranking is an output of the judge, not a restatement of the labels or of the judge-selection criterion. The efficiency term eta is a modeling proxy (parameter count), not a fitted value. Self-citations [12] and [13] provide the evaluation metrics and use-case setup, but they are not invoked as a uniqueness theorem and do not force the framework's choice of judge or weights. The absence of per-family human validation for the judge is a legitimate generalizability/validity limitation (the judge could be biased across model families), but it is not circularity by construction because the judge is not defined in terms of, or fitted to, the final ranking.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper introduces no physical entities. Its contribution is a metric (QES) and a pipeline (HuGLEN); the load-bearing choices are hand-set weights, a small single-source human calibration set, and the parameter-count efficiency proxy.

free parameters (3)

alpha (quality-efficiency trade-off) = 0.7 (default)
Eq. (5) defines QES as a weighted sum; alpha=0.7 is chosen by hand, not estimated. If alpha were lower, smaller models would rank higher; if higher, larger models would rank higher.
CQI metric weights = w1=w2=w3=1 (default)
Eq. (1) uses equal weights for correctness, scope, and usefulness. The reported ranking depends on this default; the paper says practitioners can customize weights.
Min-max normalization set = computed from the seven evaluated candidate models
Eqs. (2) and (4) normalize using only the evaluated model set; adding or removing a model changes every model's dCQI and eta-tilde, so QES values are relative to the candidate set.

assumptions (5)

domain assumption Four human experts' ratings on 40 explanations are a valid and reliable ground truth
Section IV: these ratings are used to select the LLM judge. No inter-rater reliability statistic is reported.
domain assumption Agreement between an LLM judge and human ratings on DeepSeek-R1-32B outputs generalizes to all candidate models
Section IV, LLM Judge Selection: Prometheus-7B is chosen on 40 explanations from one model and then used to score all seven model families.
domain assumption Parameter count is an adequate proxy for inference cost/efficiency
Eqs. (3)-(4) define efficiency as Pmax/Pi; the paper notes it can be replaced by measured cost, so the headline QES ranking rests on this proxy.
domain assumption Standardized prompts provide a fair comparison across LLMs
Section II-A: all models are evaluated under identical prompt conditions, but prompt sensitivity across model families is not controlled.
domain assumption SHAP values and interactions are the correct grounding for explanation quality
Section IV: XGBoost+SHAP outputs are embedded in prompts, and correctness/scope are judged against these outputs.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Human Grounded Evaluation of Large Language Models for Optical Network Automation." pith.science (2026). https://pith.science/paper/GSJFHZ2K

@misc{pith2026260718068,
  author       = {Pith},
  title        = {Pith review of: Human Grounded Evaluation of Large Language Models for Optical Network Automation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GSJFHZ2K}},
  note         = {Machine review of arXiv:2607.18068}
}

read the original abstract

Large language models (LLMs) are increasingly adopted for network automation, yet their output quality and inference cost can vary substantially across LLM families. We present HuGLEN, a stepwise evaluation pipeline that uses an LLM-as-a-judge together with a small set of expert ratings to enable scalable and reproducible comparison of candidate LLMs, and to rank them using a quality efficiency score (QES). We demonstrate HuGLEN for translating outputs from an explainable artificial intelligence (XAI) model for the optical network quality of transmission (QoT) estimation task into operator-friendly explanations. Our results show that a medium-sized LLM (12B parameters) achieves the highest QES, indicating the best trade-off between explanation quality and efficiency. Overall, HuGLEN reduces the human-labeling burden while supporting consistent model selection for operator-facing automation tasks.

Figures

Figures reproduced from arXiv: 2607.18068 by the authors.

**Figure 2.** Detailed workflow of the HuGLEN framework, ex [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Agreement scores between candidate LLM judges and human evaluators across the three predefined metrics for QoT [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Use case workflow (Task Definition stage). [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 6.** Figure 6: Comparative performance analysis of candidate LLMs [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

**Figure 7.** Figure 7: Quality efficiency score (QES) comparison for candi [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 2 linked inside Pith

[18]

Limitations of the LLM-as-a-judge approach for evaluating LLM outputs in expert knowledge tasks,

A. Szymanski, N. Ziems, H. A. Eicher-Miller, T. J.-J. Li, M. Jiang, and R. A. Metoyer, “Limitations of the LLM-as-a-judge approach for evaluating LLM outputs in expert knowledge tasks,” inProceedings of the 30th International Conference on Intelligent User Interfaces (IUI ’25). New York, NY , USA: Association for Computing Machinery, 2025, pp. 952–966

2025
[1]

Synergistic interplay of large language model and digital twin for autonomous optical networks: Field demonstrations,

Y . Song, Y . Zhang, A. Zhou, Y . Shi, S. Shen, X. Tang, J. Li, M. Zhang, and D. Wang, “Synergistic interplay of large language model and digital twin for autonomous optical networks: Field demonstrations,”IEEE Communications Magazine, vol. 63, no. 6, pp. 90–96, 2025

2025
[2]

First field trial of LLM-powered AI agent for lifecycle management of autonomous driving optical networks,

X. Liu, Q. Qiu, Y . Zhang, Y . Cheng, L. Yi, W. Hu, and Q. Zhuge, “First field trial of LLM-powered AI agent for lifecycle management of autonomous driving optical networks,” inOptical Fiber Communication Conference (OFC). Optica Publishing Group, 2025, p. Th1A.2

2025
[3]

Large language model-based wireless network design,

K. Qiu, S. Bakirtzis, I. Wassell, H. Song, J. Zhang, and K. Wang, “Large language model-based wireless network design,”IEEE Wireless Communications Letters, vol. 13, no. 12, pp. 3340–3344, 2024

2024
[4]

Deepform: Reasoning large language model for communication system formulation,

P. Wu, T. Wang, Y . Zhong, H. Zhang, Z. Wang, and F. Wang, “Deepform: Reasoning large language model for communication system formulation,” 2025. [Online]. Available: https://arxiv.org/abs/2506.08551

arXiv 2025
[5]

Netconfeval: Can llms facilitate network configuration?

C. Wang, M. Scazzariello, A. Farshin, S. Ferlin, D. Kosti ´c, and M. Chiesa, “Netconfeval: Can llms facilitate network configuration?” Proc. ACM Netw., vol. 2, no. CoNEXT2, Jun. 2024. [Online]. Available: https://doi.org/10.1145/3656296

doi:10.1145/3656296 2024
[6]

Telbench: A benchmark for evaluating telco-specific large language models,

S. Lee, D. Arya, S.-M. Cho, G.-e. Han, S. Hong, W. Jang, S. Lee, S. Park, S. Sek, I. Song, S. Yoon, and E. Davis, “Telbench: A benchmark for evaluating telco-specific large language models,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. Miami, Florida, US: Association for Computational Linguistic...

2024
[7]

On combining XAI and LLMs for trustworthy zero-touch network and service management in 6G,

A. Mekrache, M. Mekki, A. Ksentini, B. Brik, and C. Verikoukis, “On combining XAI and LLMs for trustworthy zero-touch network and service management in 6G,”IEEE Communications Magazine, vol. 63, no. 4, pp. 154–160, 2025

2025

Show all 25 references

[8]

How hungry is ai? benchmarking energy, water, and carbon footprint of llm inference,

N. Jegham, M. Abdelatti, L. Elmoubarki, and A. Hendawi, “How hungry is ai? benchmarking energy, water, and carbon footprint of llm inference,” 2025. [Online]. Available: https://arxiv.org/abs/2505.09598

2025
[9]

Energy considerations of large language model inference and efficiency optimizations,

J. Fernandez, C. Na, V . Tiwari, Y . Bisk, S. Luccioni, and E. Strubell, “Energy considerations of large language model inference and efficiency optimizations,” 2025

2025
[10]

Bleu: a method for automatic evaluation of machine translation,

K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” inProceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2002, pp. 311–318

2002
[11]

ROUGE: A package for automatic evaluation of summaries,

C.-Y . Lin, “ROUGE: A package for automatic evaluation of summaries,” inText Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81. [Online]. Available: https://aclanthology.org/W04-1013

2004
[12]

Natural language interpretability for ML- based QoT estimation via large language models,

O. Ayoub, S. Troia, C. Natalino, C. Rottondi, D. Andreoletti, F. Lelli, S. Giordano, and P. Monti, “Natural language interpretability for ML- based QoT estimation via large language models,”International Con- ference on Transparent Optical Networks (ICTON), p. Tu.C2.4, 2025

2025
[13]

Generative explainability for next-generation networks: LLM- augmented XAI with mutual feature interactions,

K. Rezaei, O. Ayoub, S. Troia, F. Lelli, P. Monti, and C. Natal- ino, “Generative explainability for next-generation networks: LLM- augmented XAI with mutual feature interactions,” in2025 21th Interna- tional Conference on Wireless and Mobile Computing, Networking and Communic...

2025
[14]

Judging LLM-as-a-judge with MT-bench and chatbot arena,

L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-bench and chatbot arena,” inProceedings of the 37th International Conference on Neural Information Proces...

2023
[15]

Tele-LLMs: A series of specialized large language models for telecommunications,

A. Maatouk, K. C. Ampudia, R. Ying, and L. Tassiulas, “Tele-LLMs: A series of specialized large language models for telecommunications,”
[16]

G- eval: Nlg evaluation using gpt-4 with better human alignment,

Y . Liu, D. Iter, Y . Xu, S. Wang, R. Xu, and C. Zhu, “G- eval: Nlg evaluation using gpt-4 with better human alignment,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics, Dec. 2023, ...

2023
[17]

Chatbot arena: An open platform for evaluating LLMs by human preference,

W.-L. Chiang, L. Zheng, Y . Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. Jordan, J. E. Gonzalez, and I. Stoica, “Chatbot arena: An open platform for evaluating LLMs by human preference,” inProceedings of the 41st International Conference on Machine Learning, v...

2024
[19]

Lora: Low-rank adaptation of large language models,

E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” 2021

2021
[20]

Beyond full fine-tuning: Harnessing the power of LoRA for multi-task instruction tuning,

C. Xin, Y . Lu, H. Lin, S. Zhou, H. Zhu, W. Wang, Z. Liu, X. Han, and L. Sun, “Beyond full fine-tuning: Harnessing the power of LoRA for multi-task instruction tuning,” inProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources an...

2024
[21]

ML-assisted QoT estimation: a dataset collection and data visualization for dataset quality evaluation,

G. Bergk, B. Shariati, P. Safari, and J. K. Fischer, “ML-assisted QoT estimation: a dataset collection and data visualization for dataset quality evaluation,”Journal of Optical Communications and Networking, vol. 14, no. 3, pp. 43–55, 2021

2021
[22]

A unified approach to interpreting model predictions,

S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” inAdvances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://papers.nips.cc/paper files/paper/2017/hash/ 8a20a8621978632d76c43dfd2...

2017
[23]

Prometheus 2: An open source language model specialized in evaluating other language models,

S. Kim, J. Suk, S. Longpre, B. Y . Lin, J. Shin, S. Welleck, G. Neubig, M. Lee, K. Lee, and M. Seo, “Prometheus 2: An open source language model specialized in evaluating other language models,” 2024

2024
[24]

Judging the judges: Evaluating alignment and vulnerabilities in LLMs-as-judges,

A. S. Thakur, K. Choudhary, V . S. Ramayapally, S. Vaidyanathan, and D. Hupkes, “Judging the judges: Evaluating alignment and vulnerabilities in LLMs-as-judges,” inProceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), K. Dhole and M. Clinciu, Eds. Vi...

2025
[2025]

Available: https://arxiv.org/abs/2409.05314

[Online]. Available: https://arxiv.org/abs/2409.05314

arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [18]

Limitations of the LLM-as-a-judge approach for evaluating LLM outputs in expert knowledge tasks,

A. Szymanski, N. Ziems, H. A. Eicher-Miller, T. J.-J. Li, M. Jiang, and R. A. Metoyer, “Limitations of the LLM-as-a-judge approach for evaluating LLM outputs in expert knowledge tasks,” inProceedings of the 30th International Conference on Intelligent User Interfaces (IUI ’25). New York, NY , USA: Association for Computing Machinery, 2025, pp. 952–966

2025

[2] [1]

Synergistic interplay of large language model and digital twin for autonomous optical networks: Field demonstrations,

Y . Song, Y . Zhang, A. Zhou, Y . Shi, S. Shen, X. Tang, J. Li, M. Zhang, and D. Wang, “Synergistic interplay of large language model and digital twin for autonomous optical networks: Field demonstrations,”IEEE Communications Magazine, vol. 63, no. 6, pp. 90–96, 2025

2025

[3] [2]

First field trial of LLM-powered AI agent for lifecycle management of autonomous driving optical networks,

X. Liu, Q. Qiu, Y . Zhang, Y . Cheng, L. Yi, W. Hu, and Q. Zhuge, “First field trial of LLM-powered AI agent for lifecycle management of autonomous driving optical networks,” inOptical Fiber Communication Conference (OFC). Optica Publishing Group, 2025, p. Th1A.2

2025

[4] [3]

Large language model-based wireless network design,

K. Qiu, S. Bakirtzis, I. Wassell, H. Song, J. Zhang, and K. Wang, “Large language model-based wireless network design,”IEEE Wireless Communications Letters, vol. 13, no. 12, pp. 3340–3344, 2024

2024

[5] [4]

Deepform: Reasoning large language model for communication system formulation,

P. Wu, T. Wang, Y . Zhong, H. Zhang, Z. Wang, and F. Wang, “Deepform: Reasoning large language model for communication system formulation,” 2025. [Online]. Available: https://arxiv.org/abs/2506.08551

arXiv 2025

[6] [5]

Netconfeval: Can llms facilitate network configuration?

C. Wang, M. Scazzariello, A. Farshin, S. Ferlin, D. Kosti ´c, and M. Chiesa, “Netconfeval: Can llms facilitate network configuration?” Proc. ACM Netw., vol. 2, no. CoNEXT2, Jun. 2024. [Online]. Available: https://doi.org/10.1145/3656296

doi:10.1145/3656296 2024

[7] [6]

Telbench: A benchmark for evaluating telco-specific large language models,

S. Lee, D. Arya, S.-M. Cho, G.-e. Han, S. Hong, W. Jang, S. Lee, S. Park, S. Sek, I. Song, S. Yoon, and E. Davis, “Telbench: A benchmark for evaluating telco-specific large language models,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. Miami, Florida, US: Association for Computational Linguistic...

2024

[8] [7]

On combining XAI and LLMs for trustworthy zero-touch network and service management in 6G,

A. Mekrache, M. Mekki, A. Ksentini, B. Brik, and C. Verikoukis, “On combining XAI and LLMs for trustworthy zero-touch network and service management in 6G,”IEEE Communications Magazine, vol. 63, no. 4, pp. 154–160, 2025

2025