Pith. sign in

REVIEW 3 major objections 6 minor 19 references

A multi-agent AI system cuts telecom troubleshooting time sixfold and improves accuracy by 10 percent.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 00:27 UTC pith:BCYHGUCK

load-bearing objection Credible industrial system paper whose headline speedup/accuracy numbers are not backed by any documented measurement methodology, and whose SLM evaluation is partly circular; the engineering is worth reading but the claims need major revision. the 3 major comments →

arxiv 2511.00651 v2 pith:BCYHGUCK submitted 2025-11-01 cs.AI cs.CLcs.ITcs.MAcs.NImath.IT

Leveraging Multi-Agent System (MAS) and Fine-Tuned Small Language Models (SLMs) for Automated Telecom Network Troubleshooting

Chenhua Shi , Bhavika Jalli , Gregor Macdonald , John Zou , Wanlu Lei , Mridul Jain , Joji Philip This is my paper
classification cs.AI cs.CLcs.ITcs.MAcs.NImath.IT
keywords multi-agent systemsmall language modelnetwork troubleshootingreinforcement fine-tuningretrieval-augmented generationknowledge graphradio access networkcore network
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Telecom network troubleshooting still leans on human experts manually correlating alarms, logs, and counters—slow, costly, and hard to scale. This paper argues that a multi-agent system, with a large language model coordinating specialized agents, can automate the whole loop, and that the hard part—generating trustworthy repair plans—can be handled by a small language model fine-tuned on internal troubleshooting documents. The authors report a sixfold reduction in mean troubleshooting time per node and a 10% accuracy gain in trials on radio access and core networks. They also show that the fine-tuned 8-billion-parameter model produces stepwise plans with specific counters and checks, at quality comparable to a much larger commercial model. If the claims hold, operators could shift from expert-driven triage to AI-assisted automation while keeping sensitive telemetry private and cutting reliance on expensive external AI services.

Core claim

The authors' central claim is that end-to-end automated troubleshooting can be assembled from an LLM orchestrator plus a small, domain-tuned 'solution planner' that converts retrieved troubleshooting knowledge into executable stepwise plans. The planner is fine-tuned in two stages—supervised training followed by reinforcement fine-tuning with rewards for completeness, relevance, and groundedness to the retrieved context—and is backed by a knowledge graph of performance, alarm, and configuration documents. In evaluations covering a RAN power-supply fault and a core-network PDU-session degradation, the system produced plans specifying the exact counters and alarms to check, localized faults (e

What carries the argument

The load-bearing mechanism is the agentic workflow, specifically the solution-planner agent: a small language model fine-tuned with supervised plus reinforcement fine-tuning, guided by reward functions that check whether answers are complete, relevant, and grounded in retrieved evidence. The planner is coupled to a knowledge-graph retrieval layer built from internal performance-management counters, fault alarms, and configuration documents, so every generated step references real network objects. An LLM orchestrator delegates to the planner, a data retriever, a root-cause analyzer, an executor, and a display agent in a reason-act loop, with a human-in-the-loop approval before execution. The

Load-bearing premise

The 6× speedup and 10% accuracy gain are computed against a human-engineer baseline that the paper never describes—how the human times were measured, how accuracy was scored, or whether the same fault cases were used for both the system and the humans.

What would settle it

Re-run the same set of fault cases with a panel of engineers and with the MAS, using identical stopping criteria and a pre-registered accuracy rubric; if the MAS does not show roughly a 6× time reduction and 10% accuracy advantage, the paper's headline claims are not supported.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Operators could turn first-line troubleshooting into an automated triage service, reserving human experts for edge cases, reducing operational cost and time-to-restore.
  • A fine-tuned small model can replace per-query calls to large external LLM APIs for plan generation, enabling on-premise deployment that keeps sensitive telemetry inside operator environments.
  • The reward design—completeness, relevancy, groundedness—provides a reusable recipe for grounding language models in other documentation-heavy operational domains.
  • Because the system produces stepwise plans with named counters and alarms, its outputs are inspectable by experts, which supports adoption in safety-conscious network operations.
  • The same architecture could extend beyond the two fault families tested (power failures and PDU-session degradation) to other alarm classes, provided retrieval documents exist.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The biggest unstated assumption is the human baseline: the reported 6×/10% figures would carry much more weight if the comparison were run on identical fault cases with a pre-registered accuracy rubric, rather than the summary figures presented.
  • The fine-tuning approach suggests a general template: for any regulated industry with proprietary operational manuals, a small model fine-tuned on those manuals and grounded by retrieval could reach parity with frontier LLMs at a fraction of the cost—testable by applying the same pipeline to, say, datacenter or grid operations.
  • A cost-per-incident analysis, comparing the MAS's compute and human-oversight costs against a fully human process, would make the business case concrete; the paper reports time and accuracy but not total cost or error-consequence severity.
  • One could test how the system degrades when retrieval documents are missing or outdated, since the groundedness reward assumes the knowledge graph reflects current network configurations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a multi-agent system (MAS) for automated telecom network troubleshooting, combining LLM-based orchestration with a fine-tuned small language model (SLM) solution planner. The SLM is fine-tuned via SFT followed by GRPO-based RFT, using RAGAS-style and format rewards, and is integrated with a HippoRAG knowledge graph built from proprietary troubleshooting documents. The system is evaluated in RAN and Core network domains, with claimed outcomes of a six-fold reduction in mean troubleshooting time per node and a 10% accuracy improvement over human engineers (Section IV.A.3, Figures 3–4). Additional experiments report reward improvements for the fine-tuned 8B model (Table I, Figures 5–6). The paper's conclusions rest on these two sets of quantitative claims.

Significance. If properly validated, the proposed architecture is a relevant contribution: it addresses cost, privacy, and scalability concerns in telecom operations by combining LLM orchestration with a fine-tuned SLM, and it incorporates practical details such as LoRA, GRPO, and multi-GPU RFT. The system design is plausible and the integration of a knowledge-graph RAG with an agentic workflow is of interest to the community. However, the current evidence is insufficient to support the paper's central claims. The human-baseline comparison is undocumented, and the SLM evaluation is circular because it uses the same reward functions employed during training as the evaluation metrics. Strengths of the manuscript include the detailed description of the MAS components, the reward design, and the practical engineering choices; these do not compensate for the missing rigorous evaluation.

major comments (3)
  1. [Section IV.A.3, Figures 3–4, Conclusion] The headline claim of 'six-fold reduction in mean troubleshooting time per node and 10% improvement in accuracy' is not supported by the described methodology. No human-baseline collection protocol is reported: the number and seniority of engineers, the fault set and domains, the phases included in 'troubleshooting time' (plan generation, SME approval, execution, RCA), whether the same alarm/log/PM data were presented to humans and the system, and the accuracy rubric are all unspecified. There are no trial counts, confidence intervals, or error bars. The figures are referenced but not described in terms of axes, units, or N. This is load-bearing because the paper's central conclusion rests on these comparative numbers, which cannot be reproduced or verified as written.
  2. [Section IV.B.3, Table I, Figures 5–6] The fine-tuning evaluation is circular. The metrics reported in Table I (RAGAS Rewards, Format Rewards, Total Rewards) are the same reward functions used as the training signal in the GRPO-based RFT described in Section IV.B.2. An improvement on the training objective is expected and does not demonstrate that the fine-tuned SLM produces better troubleshooting plans. The claimed reduction in standard deviation is also measured on these same reward signals. To support the conclusion that the SLM is 'domain-grounded' and 'actionable,' the authors need an independent evaluation—e.g., SME scoring of generated plans, exact-match with gold steps, or end-to-end task success on held-out faults.
  3. [Section IV.A.1–IV.A.2] The experimental description for the MAS is anecdotal. The text walks through only two example faults (Input Power Failure in RAN and PDU session degradation in Core) without specifying how many cases were run, how alarms/KPIs were selected, or how outputs were scored. Section IV.A.3 states benefits but provides no quantitative experimental design. This makes it impossible to judge robustness or generalizability. Please provide a systematic evaluation protocol, including dataset statistics, the exact metric definitions, and statistical significance.
minor comments (6)
  1. [References [18]] Reference [18] is cited for RAGAS but points to 'DecodingTrust' (Li et al.); the RAGAS paper (Es et al., arXiv:2309.15217) should be cited. Please check all references for accuracy.
  2. [References [2] and [10]] References [2] and [10] are duplicate entries for the same 'Generative AI' paper. Please consolidate.
  3. [Reference [14]] Reference [14] for HippoRAG appears to point to the follow-up 'From RAG to Memory' rather than the original HippoRAG paper. Clarify which algorithm/version is actually used.
  4. [Section IV.B.1] The sentence '1000 training steps executes faster and converges more smoothly' is grammatically awkward. Also, 'proved more effective' should be 'proved to be more effective'. Please revise for clarity.
  5. [Section IV.A.3] Define 'per node' and 'accuracy' explicitly. The current text leaves the units and the accuracy metric ambiguous; this is important for interpreting the headline numbers.
  6. [Figures 3 and 4] The figures are not included in the submitted text and no axes or captions are described. Please ensure they include error bars, sample sizes, and clear definitions of the metrics plotted.

Circularity Check

1 steps flagged

The fine-tuning evaluation metric is the training objective itself: Table I's RAGAS/format rewards are the exact GRPO reward functions, so the reported improvement is partly by construction; the 6x/10% baseline claim is undocumented but not circular.

specific steps
  1. fitted input called prediction [Section IV.B.2 (Reward Design for RFT) and Section IV.B.3 (RFT Results Comparison), Table I, Figures 5-6]
    "In our framework, we employ two categories of reward functions: (i) Format Rewards, and (ii) RAGAS-based Rewards customized for telecom troubleshooting. ... Table I summarizes the results for the 8B model, showing that, compared to its base version, the fine-tuned model achieves higher rewards and significantly lower standard deviation."

    The 'RAGAS Rewards' and 'Format Rewards' in Table I are the same reward signals used to train the model via GRPO in Section IV.B.2. A policy optimized to maximize a reward will, if training succeeds, increase that reward; reporting that increase as evidence that the fine-tuned model is better at generating troubleshooting plans is reporting the training objective rather than an independent evaluation. The improvement (RAGAS 3.44 to 5.19, Format 2.52 to 5.31, Total 5.96 to 10.51) is therefore partly by construction, unless evaluated on a held-out set with a different metric, which the paper does not establish.

full rationale

The paper's headline end-to-end claim (6x mean-time reduction, 10% accuracy improvement over human engineers, Section IV.A.3/Conclusion) is not itself circular by construction: it is an external comparison, but the paper gives no protocol for the human baseline (task set, timing, accuracy rubric, same fault cases), making it unverifiable rather than circular. The clear circularity is in the RFT evaluation: the same RAGAS/format scoring functions used as GRPO rewards are reported as performance metrics in Table I and Figures 5-6, so the demonstrated 'improvement' of the fine-tuned SLM is partially the optimizer fitting its own objective. This does not reduce the whole paper to a tautology—the MAS architecture, retrieval grounding, and time/accuracy comparison are independent in principle—but the quantitative evidence for the SLM component's quality is by construction. Self-citations ([13], [19]) supply tools and data but are not invoked as uniqueness theorems or ansatz justifications, so they are not load-bearing circularity under the rules.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The central claims rest on an undocumented human baseline, on reward scores that double as training objectives, and on the representativeness of two case studies. No independent physical or mathematical axioms are needed; these are domain assumptions about measurement validity.

free parameters (2)
  • RAGAS reward composition
    Hand-crafted reward design (format, completeness, relevancy, groundedness) is both the RFT objective and the reported evaluation metric; reward weights are not reported.
  • Fine-tuning hyperparameters
    LoRA rank, batch size, generations per step, and uniform chunk size are chosen empirically; changes (e.g., larger rank after chunking) are credited with speed/quality gains, but values are not specified.
axioms (3)
  • domain assumption Human-engineer baseline for time and accuracy is measured on comparable fault cases with a consistent accuracy definition.
    Central 6×/10% claim rests on Figures 3/4; no methodology for the baseline is given (Section IV.A.3).
  • domain assumption RAGAS-based reward scores are a valid proxy for real troubleshooting plan quality.
    Table I reports RAGAS/format rewards as evidence of improvement, but these same rewards guide RFT; no independent human evaluation or task-level metric is provided (Section IV.B.2).
  • domain assumption Two case studies (RAN power failure, Core PDU session degradation) generalize to heterogeneous telecom deployments.
    Experiments cover only two fault types; the introduction claims generalization across heterogeneous deployments (Sections I and IV.A).

pith-pipeline@v1.3.0-alltime-deepseek · 7717 in / 9722 out tokens · 102871 ms · 2026-08-04T00:27:53.271836+00:00 · methodology

0 comments
read the original abstract

Telecom networks are rapidly growing in scale and complexity, making effective management, operation, and optimization increasingly challenging. Although Artificial Intelligence (AI) has been applied to many telecom tasks, existing models are often narrow in scope, require large amounts of labeled data, and struggle to generalize across heterogeneous deployments. Consequently, network troubleshooting continues to rely heavily on Subject Matter Experts (SMEs) to manually correlate various data sources to identify root causes and corrective actions. To address these limitations, we propose a Multi-Agent System (MAS) that employs an agentic workflow, with Large Language Models (LLMs) coordinating multiple specialized tools for fully automated network troubleshooting. Once faults are detected by AI/ML-based monitors, the framework dynamically activates agents such as an orchestrator, solution planner, executor, data retriever, and root-cause analyzer to diagnose issues and recommend remediation strategies within a short time frame. A key component of this system is the solution planner, which generates appropriate remediation plans based on internal documentation. To enable this, we fine-tuned a Small Language Model (SLM) on proprietary troubleshooting documents to produce domain-grounded solution plans. Experimental results demonstrate that the proposed framework significantly accelerates troubleshooting automation across both Radio Access Network (RAN) and Core network domains.

Figures

Figures reproduced from arXiv: 2511.00651 by Bhavika Jalli, Chenhua Shi, Gregor MacDonald, John Zou, Joji Philip, Mridul Jain, Wanlu Lei.

Figure 2
Figure 2. Figure 2: RFT pipeline for fine-tuning a Small Language Model (SLM) as [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Autonomous Network Operations Agent Mean Time to Trou [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Autonomous Network Operations Agent Troubleshooting Accuracy. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Mean of Rewards on Training and Evaluation among Different [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Standard Deviation of Rewards on Training and Evaluation among [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of Responses from Base Model, Fine-Tuned Model, [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 1 canonical work pages

  1. [1]

    Telecom Foundation Models: Applications, Challenges, and Future Trends,

    T. Zanouda, M. Masoudi, F. G. Gebre, and M. Dohler, “Telecom Foundation Models: Applications, Challenges, and Future Trends,” arXiv:2408.03964v1, 2024. [Online]. Available: https://arxiv.org/abs/2408.03964v1

    Pith/arXiv arXiv 2024

  2. [3]

    On the Opportunities and Risks of Foundation Models,

    R. Bommasani, D. Hudson, E. Adeliet al., “On the Opportunities and Risks of Foundation Models,” 2021, arXiv:2108.07258. [Online]. Available: https://arxiv.org/abs/2108.07258

    Pith/arXiv arXiv 2021

  3. [4]

    Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,

    P. Lewis, E. Perez, A. Piktuset al, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” in *Advances in Neural Information Processing Systems*, vol. 33, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds. Curran Associates, Inc., 2020, pp. 9459– 9474

    2020

  4. [5]

    A survey on knowledge- oriented retrieval-augmented generation,

    M. Cheng, Y . Luo, J. Ouyanget al, “A survey on knowledge- oriented retrieval-augmented generation,” 2025. arXiv:2503.10677. [On- line]. Available: https://arxiv.org/abs/2503.10677

    Pith/arXiv arXiv 2025

  5. [6]

    ReAct: Synergizing Reasoning and Acting in Language Models,

    S. Yao, D. Yu, J. Zhaoet al., “ReAct: Synergizing Reasoning and Acting in Language Models,” in ICLR, 2023, arXiv:2210.03629. [Online]. Available: https://arxiv.org/abs/2210.03629

    Pith/arXiv arXiv 2023

  6. [7]

    Small Language Models are the Future of Agentic AI,

    P. Belcak, G. Heinrich, S. Diao,et al., “Small Language Models are the Future of Agentic AI,” arXiv:2506.02153, 2025. [Online]. Available: https://arxiv.org/abs/2506.02153

    Pith/arXiv arXiv 2025

  7. [8]

    Network troubleshoot- ing: Survey, taxonomy and challenges,

    V . Tong, H. A. Tran, S. Souihi, and A. Mellouk, “Network troubleshoot- ing: Survey, taxonomy and challenges,” inProc. IEEE SaCoNeT, 2018. doi: 10.1109/SaCoNeT.2018.8585610

    arXiv 2018

  8. [9]

    Harnessing Machine Learning for Predictive Trou- bleshooting in Telecom Networks,

    P. Singh, “Harnessing Machine Learning for Predictive Trou- bleshooting in Telecom Networks,” SSRN, 2025. [Online]. Available: http://dx.doi.org/10.2139/ssrn.5218808

    work page doi:10.2139/ssrn.5218808 2025

  9. [10]

    Generative AI,

    S. Feuerriegel, J. Hartmann, C. Janiesch, and P. Zschech, “Generative AI,” arXiv:2309.07930, 2023. [Online]. Available: https://arxiv.org/abs/2309.07930

    Pith/arXiv arXiv 2023

  10. [11]

    Large Language Models for Telecom- munications,

    T. Zhouet al., “Large Language Models for Telecom- munications,” arXiv:2405.10825, 2024. [Online]. Available: https://arxiv.org/abs/2405.10825

    Pith/arXiv arXiv 2024

  11. [12]

    Hypha: A distributed application framework for large- scale data management and AI model serving,

    Amun-AI, “Hypha: A distributed application framework for large- scale data management and AI model serving,” [Online]. Available: https://github.com/amun-ai/hypha

  12. [13]

    BioImage.IO Chat- bot: a community-driven AI assistant for integrative computational bioimaging,

    W. Lei, C. Fuster-Barcel ´o, G. Rederet al., “BioImage.IO Chat- bot: a community-driven AI assistant for integrative computational bioimaging,”Nature Methods, vol. 21, pp. 1368–1370, 2024. doi: 10.1038/s41592-024-02370-y

    doi:10.1038/s41592-024-02370-y 2024

  13. [14]

    From RAG to Memory: Non- Parametric Continual Learning for Large Language Models,

    B. J. Guti ´errez, Y . Shu, W. Qiet al, “From RAG to Memory: Non- Parametric Continual Learning for Large Language Models,” 2025, arXiv:2502.14802. [Online]. Available: https://arxiv.org/abs/2502.14802

    Pith/arXiv arXiv 2025

  14. [15]

    LoRA: Low- Rank Adaptation of Large Language Models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhuet al., “LoRA: Low- Rank Adaptation of Large Language Models,” arXiv:2106.09685, 2021. [Online]. Available: https://arxiv.org/abs/2106.09685

    Pith/arXiv arXiv 2021

  15. [17]

    Efficient Memory Manage- ment for Large Language Model Serving with PagedAttention,

    W. Kwon, Z. Li, S. Zhuanget al, “Efficient Memory Manage- ment for Large Language Model Serving with PagedAttention,” 2023, arXiv:2309.06180. [Online]. Available: https://arxiv.org/abs/2309.06180

    Pith/arXiv arXiv 2023

  16. [18]

    DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models,

    J. Turc, M. Lester, J. Ham, D. Zettlemoyer, and M. Lewis, “DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models,” 2023, arXiv:2309.15217. [Online]. Available: https://arxiv.org/abs/2309.15217

    Pith/arXiv arXiv 2023

  17. [19]

    Think Less, Label Bet- ter: Multi-Stage Domain-Grounded Synthetic Data Generation for Fine-Tuning Large Language Models in Telecommunications,

    C. Shi, G. Macdonald, B. Jalli,et al, “Think Less, Label Bet- ter: Multi-Stage Domain-Grounded Synthetic Data Generation for Fine-Tuning Large Language Models in Telecommunications,” 2025, arXiv:2509.25736. [Online]. Available: https://arxiv.org/abs/2509.25736

    arXiv 2025

  18. [20]

    ZeRO: Memory Optimizations Toward Training Trillion Parameter Models,

    S. Rajbhandari, J. Rasley, O. Ruwase, and Y . He, “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models,” 2019, arXiv:1910.02054, . [Online]. Available: https://arxiv.org/abs/1910.02054

    Pith/arXiv arXiv 2019

  19. [2024]

    Available: https://arxiv.org/abs/2402.03300

    [Online]. Available: https://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv