RCA Copilot: Transforming Network Data into Actionable Insights via Large Language Models

Alexander Shan; Jasleen Kaur; Rahul Singh; Raj Yavatkar; Tarun Banka; T. Sridhar

REVIEW 4 major objections 5 minor 17 references

RCA Copilot: Transforming Network Data into Actionable Insights via Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RCA Copilot claims that combining statistical causality ranking, retrieval of past incidents, and LLM reasoning produces human-engineer-level root-cause explanations and action steps for network faults without fine-tuning.

desk verdict Plausible engineering integration of statistical RCA with LLM prompting, but the evaluation metrics cannot support the 'human engineer-level' claim and the few-shot scores may be inflated by retrieval leakage. read the letter →

arxiv 2507.03224 v1 pith:HTTYBIHL submitted 2025-07-03 cs.NI

Alexander Shan , Jasleen Kaur , Rahul Singh , Tarun Banka , Raj Yavatkar , T. Sridhar This is my paper · ORCID

classification cs.NI

keywords rootcauseanalysisnetworkincidentmanagementlargelanguagemodelsfew-shotlearningretrieval-augmentedgenerationGrangercausalityPageRanktelemetry

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RCACopilot is a system that automates root cause analysis (RCA) of network incidents without fine-tuning a large language model. It tries to establish that combining a statistically ranked list of likely root causes with retrieved past incident reports and structured prompting lets an LLM produce explanations and action steps that are as useful as a human on-call engineer's, at least for the eight fault scenarios tested. The authors report high semantic-similarity scores against hand-written gold diagnoses in the few-shot setting, and argue this makes automated, interpretable RCA practical across cloud, data center, and on-premises networks. A reader should care because the design points to a way of updating operational knowledge by adding retrieval examples rather than retraining costly models.

What carries the argument

The load-bearing mechanism is the retrieval-augmented statistical prompting pipeline: (i) capture a JSON snapshot of the topology with node telemetry; (ii) run Granger causality on the anomalous time series, weight edges with Pearson correlation, and PageRank the resulting graph to get a top-$K$ root-cause list; (iii) embed the current diagnostic state into a vector database and prepend similar past incidents with their gold diagnoses and action steps as in-context examples; (iv) prompt a large language model with chain-of-thought and a mixture of hypotheses to produce a narrative diagnosis and node-specific action steps. The statistical list filters the input space, the retrieved examples supply domain knowledge, and the prompt structure enforces reasoning and consensus.

What would settle it

Apply the generated action steps to the fault-injected topology and check whether the application-layer anomaly clears, and also check whether a deliberately fluent but wrong explanation scores as high as the reported 0.81–0.95; if either fails, the central claim is not supported.

Watch

Extended reading notes

Core claim

The central claim is that a no-fine-tuning LLM pipeline can close the interpretability gap of statistical RCA: the system ingests topology snapshots and telemetry, ranks candidate causes with Granger causality and PageRank, retrieves similar past incidents to supply in-context examples, and then generates a narrative diagnosis with concrete action steps. In the paper's experiments across two topologies and eight fault types, the generated text scored between 0.81 and 0.95 (F1) against the authors' gold diagnoses in the few-shot setting, and the authors state that this demonstrated human engineer-level analysis. The same system with no examples scored markedly lower, which the authors read as evidence that the retrieved exemplars, not the model's parametric memory, carry the task-specific knowledge.

Load-bearing premise

The whole evaluation rests on the assumption that textual similarity to the authors' hand-written gold diagnosis and action steps measures whether the root cause was actually found and the fix would actually work; nothing in the experiments tests the actions on the live system.

Editorial extensions

If this is right

Instead of manual inspection of logs and telemetry, an operator can trigger on-demand analysis that returns a root-cause hypothesis, an explanation narrative, and node-specific action steps.
Domain knowledge can be updated by adding or curating retrieval examples, so the system can track emerging network issues without retraining a large model.
The interpretability of statistical RCA improves because the ranked candidate list is translated into a human-readable diagnosis rather than left as a black-box score.
If the few-shot generalization holds beyond the eight tested scenarios, a small corpus of past incidents may be enough to root-cause previously unseen fault types.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A test the paper does not run is to have independent engineers execute the action steps on a live fault-injected topology, which would separate explaining plausibly from resolving the incident.
If the architecture transfers, the same retrieval-plus-statistical pipeline could be applied to other structured diagnostic domains with time-series metrics and topology graphs, such as microservices or storage systems, without fine-tuning.
The large drop from few-shot to zero-shot performance suggests the retrieved examples, rather than the base model's knowledge, are doing most of the RCA work; a natural follow-up is to measure how explanation quality scales as the exemplar corpus grows.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a plausible engineering integration, but the evidence doesn't back the headline "human engineer-level" claim, and the few-shot results may be inflated by retrieval leakage. Worth a serious referee, but not as-is.

What's actually new: the combination of statistical RCA (Granger causality + PageRank) with retrieval-augmented few-shot prompting, chain-of-thought, and mixture-of-agents is not present in the cited work. The architecture is clearly described, and the fault-injection setup on two topologies is a reasonable way to generate labeled scenarios. The no-fine-tuning motivation is sensible.

Where it goes soft: the evaluation measures BERTScore and S-BERT similarity between the generated text and the authors' gold diagnosis/action steps. That cannot support a claim of human-level analysis—it measures textual proximity, not whether the root cause is right or the actions resolve the incident. A fluent but wrong explanation could score high. There are also no baselines (statistical RCA alone, fine-tuned LLM, human operator), no error bars, and only eight scenarios. The zero-shot-to-few-shot jump is large; one straightforward explanation is that the retrieval index contains gold answers for the same fault scenarios. Section IV-C says the retrieval states are "a different set of states" from the evaluation states, but the paper never states that query states are excluded from the index or that fault types are disjoint. That distinction is the load-bearing check for the "unseen scenarios" claim in VII-B, and it needs to be made explicit with a leave-one-fault-out split.

Minor issues: reference [9] is mis-cited (it points to a combinatorial auction paper, not Zhang and Chen's RCA work), and the "novel evaluation framework" is just two off-the-shelf similarity metrics.

Bottom line: the integration is timely for network operations, and the contamination concern is fixable. A serious editor should send this to review, with the expectation that the authors either run a clean leave-one-out evaluation or tone down the claim to "improves on statistical RCA alone in textual similarity." I wouldn't cite it in its current form.

Referee Report

4 major / 5 minor

Summary. The paper presents RCACopilot, a fault-diagnosis system that combines statistical root cause analysis (Granger causality plus PageRank on network telemetry) with an LLM prompted via retrieval-augmented few-shot examples. The system ingests network topology snapshots, metric data, and a statistical RCA ranked list, and is asked to output a root cause explanation and action steps. The authors evaluate on eight fault-injection scenarios drawn from two network topologies, comparing generated text to author-written gold diagnoses using BERTScore and S-BERT cosine similarity, in both few-shot and zero-shot settings. The paper claims that RCACopilot achieves human engineer-level analysis and generalizes to unseen scenarios with a small set of exemplars, without fine-tuning.

Significance. If the claims were fully validated, the contribution would be practically useful: a no-fine-tuning RCA assistant that integrates statistical causality scores with interpretable LLM reasoning, and that can be updated by adding retrieval examples, is a sensible architectural direction with real operations value. The paper also responsibly reports zero-shot comparisons and the statistical-RCA component is grounded in established methods. However, the current evidence is insufficient to support the central claims. The evaluation is small (eight cases, no variance), the metric measures textual similarity to the authors' own gold text rather than diagnostic correctness, and the retrieval setup may allow the gold answer to leak into the prompt. Because the load-bearing evaluation does not currently isolate the system's ability to identify and act on the true root cause, the significance of the reported numbers is unclear without substantial additional validation.

major comments (4)

[§IV-C vs §V-A] The retrieval index and the evaluation set appear to overlap. Section IV-C states that 'we collect eight graph topology states with faults' to build the retrieval database, with paired gold diagnoses and action steps, while Section V-A says the evaluation collects 'eight samples of graph topology states' across two topologies with 5 and 3 fault scenarios. The paper never states that the query state is excluded from the retrieval index, nor that the index contains different fault types or topologies than the evaluation set. If the index contains the query state, or another state from the same injected fault scenario, then the few-shot prompt can include the gold diagnosis and action steps, and the results would measure text reproduction rather than RCA generalization. This directly undermines the Section VII-B claim that 'a small set of few shot examples were good enough to root cause unseen scenarios.' The evaluation must use a leave-one-fault-scenario-out split, and the paper must state explicitly whether the query state is removed from the retrieval index.
[§V-B] The evaluation metric does not measure whether the root cause is correct or whether the action steps resolve the incident. BERTScore and S-BERT cosine similarity compare the full generated text to the authors' gold diagnosis and action steps; a fluent but wrong explanation can achieve high similarity to gold text, and no human validation or resolution test is provided. Moreover, the gold text itself is written by the authors (Section III-A) without any independent verification that it identifies the true root cause or that the listed actions actually mitigate the fault. The evaluation should include, at minimum, a structured correctness metric (e.g., whether the predicted root-cause node/layer matches the injected fault, and whether the proposed action targets that node), ideally alongside human judgment of diagnosis and action quality.
[§VII-B] The claim that 'RCACopilot demonstrated human engineer-level analysis' is not supported by the reported experiments. There is no comparison to human engineers on the same inputs, no measure of diagnostic accuracy beyond text similarity, and no evidence that the generated action steps, if executed, would resolve the incidents. With only eight scenarios and no repeated trials or confidence intervals, the claim is disproportionate to the evidence. The authors should either soften the claim to match what the data show (e.g., high textual similarity to gold diagnoses in a small set of scenarios) or add a human-study and/or resolution-verification component.
[§VI] The paper does not include a baseline that isolates the contribution of each component. While zero-shot LLM performance is reported, there is no baseline that uses only the statistical RCA ranked list (e.g., reporting how often the true root cause is in the top-K list for these eight scenarios) or only the LLM with raw diagnostic data but without the statistical RCA oracle. Such baselines are necessary to support the claim that the combination of statistical tests and LLM reasoning is what drives performance, rather than the statistical RCA list alone or the LLM's prior knowledge.

minor comments (5)

[§III-A] The sentence 'We configured are two different network topologies' contains a grammatical error and should read 'We configured two different network topologies.'
[References [9] and [6]] Reference [9] is cited as 'Zhang and Chen's work' on in-context learning, but it points to a combinatorial-auction paper (Duan et al., arXiv:2402.11904), which is unrelated. Similarly, Section IV-C attributes the few-shot result to 'Zhang et al. [6]', but reference [6] is a causal-inference survey by Pham, Ha, and Zhang. The citations should be corrected to the intended works on retrieval-based in-context learning.
[§III-B] The phrase 'In contrast to Zhang and Chen's work [9]' inherits the citation error described above; if the intended reference is the retrieval-based RCA work of Chen et al. [8], the contrast should be stated against that work.
[§V-A] The list of eight use cases mixes fault scenarios from two different topologies, but the paper does not state which use case belongs to which topology. This makes it difficult to assess whether the retrieval index and the evaluation set are separated by topology. A small table mapping each use case to its topology and injected fault would improve clarity.
[§III-A] The paper says gold diagnoses were assigned because faults were injected at controlled intervals, but it does not describe how the gold reasoning chains and action steps were authored or reviewed. A sentence on authoring process and any inter-author agreement would help readers calibrate the gold standard.

Circularity Check

1 steps flagged · score 6.0 of 10

Few-shot results may reduce to retrieval: the retrieval corpus and evaluation set share the same eight fault-scenario counts and no exclusion is stated, so high BERTScore/S-BERT can reflect copying gold text from the prompt rather than out-of-sample RCA.

fitted input called prediction [Sections IV-C, V-A, V-B and Table I; VII-B generalization claim]
"To construct our retrieval dataset, we use the same approach as in III-A, but on a different set of states: we collect eight graph topology states with faults that we have the ground truth causes for. These states have paired gold diagnoses and action steps written in text. ... For our setup, we collect eight samples of graph topology states across different times. In our setup, there are 2 different network topologies (Figure 1 and 2) , each with 5 and 3 unique fault scenarios respectively."

The retrieval corpus stores gold diagnoses and action steps for eight fault states, while the evaluation set is eight graph-topology samples from the same two topologies with the same 5+3 scenarios. 'Different set of states' may only mean different snapshots; the paper never states that the query state, or any state from the same injected fault, is excluded from the index. If such a state is indexed, its gold text is in the prompt, and Table I scores that output against the same gold text. High few-shot scores can then reflect copying the gold answer, not root-causing an unseen scenario, so the VII-B claim that few-shot examples were 'good enough to root cause unseen scenarios' is not established.

full rationale

Apart from the few-shot evaluation, the paper's chain is largely non-circular: the diagnostic inputs (telemetry, topology, statistical Granger/PageRank results) are independent of the generated text, injected faults give objective labels, and there are no load-bearing self-citations. The statistical RCA ranking is an oracle input, not a fitted target, though it does narrow the LLM's hypothesis space. The central circularity risk is the overlap between the retrieval corpus and the evaluation set: both are described as eight fault states over the same two topologies with the same 5+3 scenario split, and no leave-one-fault-scenario-out split is stated. If the index contains the queried state or a sibling state of the same fault, the gold diagnosis and action steps sit inside the prompt, so the few-shot BERTScore/S-BERT numbers can measure text reproduction. The Section VII-B anecdote about Transit Gateway blackholes with only Gateway-layer examples provides one piece of independent evidence, but it does not validate all eight scored use cases. Separately, reference [9] is cited as 'Zhang and Chen's work' but points to an unrelated combinatorial-auction paper; this is a citation error and a correctness risk, not a circularity. Overall, the few-shot prediction potentially reduces by construction to retrieving the gold text, giving a partial circularity score of 6.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The system is an engineering artifact; it introduces no new physical or theoretical entity. The only invented thing is the named pipeline itself, which requires evidence rather than independent falsifiability.

free parameters (3)

Statistical RCA top-K = K=5
Section IV-B says the prompt highlights the top K ranked nodes with K=5; chosen by hand and not swept or validated.
Number of retrieved few-shot examples = not specified
Section IV-C describes prepending retrieved examples but never states how many are used; performance depends on this choice.
Embedding model and retrieval similarity threshold = not specified
Section IV-C names a vector database (e.g., FAISS) but not the embedding model, indexing parameters, or similarity threshold.

assumptions (4)

domain assumption Granger causality and PageRank on the collected time series produce a ranking with the true root cause near the top.
Invoked in Section IV-B and assumed in Section II-A; the paper states the true root cause is top-five in its implementations but provides no standalone evaluation of this claim.
ad hoc to paper Semantic similarity to author-written gold text is a valid measure of RCA quality.
Section V-B defines the evaluation this way without validating against human judgment or whether the proposed action fixes the fault.
domain assumption The eight fault-injected states are representative of real customer incidents.
Section III-A says faults were injected to reflect realistic scenarios, but no real incident data is used.
domain assumption Retrieved few-shot examples for a test state do not come from the same fault scenario.
Section IV-C says the retrieval dataset is a 'different set of states', but overlap with the eight evaluation states is not demonstrated; leakage would invalidate few-shot results.

how reviews work

0 comments

Cite this review

Pith. "Pith review of RCA Copilot: Transforming Network Data into Actionable Insights via Large Language Models." pith.science (2026). https://pith.science/paper/HTTYBIHL

@misc{pith2026250703224,
  author       = {Pith},
  title        = {Pith review of: RCA Copilot: Transforming Network Data into Actionable Insights via Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HTTYBIHL}},
  note         = {Machine review of arXiv:2507.03224}
}

read the original abstract

Ensuring the reliability and availability of complex networked services demands effective root cause analysis (RCA) across cloud environments, data centers, and on-premises networks. Traditional RCA methods, which involve manual inspection of data sources such as logs and telemetry data, are often time-consuming and challenging for on-call engineers. While statistical inference methods have been employed to estimate the causality of network events, these approaches alone are similarly challenging and suffer from a lack of interpretability, making it difficult for engineers to understand the predictions made by black-box models. In this paper, we present RCACopilot, an advanced on-call system that combines statistical tests and large language model (LLM) reasoning to automate RCA across various network environments. RCACopilot gathers and synthesizes critical runtime diagnostic information, predicts the root cause of incidents, provides a clear explanatory narrative, and offers targeted action steps for engineers to resolve the issues. By utilizing LLM reasoning techniques and retrieval, RCACopilot delivers accurate and practical support for operators.

Figures

Figures reproduced from arXiv: 2507.03224 by the authors.

**Figure 2.** On-Prem AIML Workload Topology A state consists of two pieces of information: 1. Snapshot of the Topology graph: This captures all the nodes in the topology, their connectivity to other nodes, and their telemetry data. More information about this diagnostic information will be described in III-B [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** outlines the design of RCACopilot. When an anomaly is detected at the application layer (e.g. application node experiences latency), the workflow is triggered. First, diagnostic data sources are queried to reconstruct the state of the Network Graph Topology (node metrics, connectivity between nodes at each layer, etc.). Some data sources, such as those representing graph topology and application layer symptoms, must… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 10 canonical work pages

[9]

Automated Deterministic Auction Design with Objective Decomposition

Z. Duan et al., “Scalable virtual valuations combinatorial auction design by combining zeroth-order and first-order optimization method,” arXiv preprint arXiv:2402.11904, 2024

work page Pith review arXiv 2024
[1]

Microrca: Root cause localization of performance issues in microservices,

L. Wu, J. Tordsson, E. Elmroth, and O. Kao, “Microrca: Root cause localization of performance issues in microservices,” in NOMS 2020- 2020 IEEE/IFIP Network Operations and Management Symposium , pp. 1–9, IEEE, 2020

work page 2020
[2]

A comprehensive survey on root cause analysis in (micro) services: Methodologies, challenges, and trends,

T. Wang and G. Qi, “A comprehensive survey on root cause analysis in (micro) services: Methodologies, challenges, and trends,” arXiv preprint arXiv:2408.00803, 2024

arXiv 2024
[3]

Exploring llm-based agents for root cause analysis,

D. Roy et al., “Exploring llm-based agents for root cause analysis,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering , pp. 208–219, 2024

work page 2024
[4]

Gpt-4 technical report,

J. Achiam et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

arXiv 2023
[5]

Mining causality of network events in log data,

S. Kobayashi, K. Otomo, K. Fukuda, and H. Esaki, “Mining causality of network events in log data,” IEEE Transactions on Network and Service Management, vol. 15, no. 1, pp. 53–67, 2017

work page 2017
[6]

Root cause analysis for microservice system based on causal inference: How far are we?,

L. Pham, H. Ha, and H. Zhang, “Root cause analysis for microservice system based on causal inference: How far are we?,” arXiv preprint arXiv:2408.13729, 2024

arXiv 2024
[7]

Recommending root-cause and mitigation steps for cloud incidents using large language models,

T. Ahmed et al. , “Recommending root-cause and mitigation steps for cloud incidents using large language models,” in 2023 IEEE/ACM 45th Intl Conf on Software Engineering (ICSE) , pp. 1737–1749, IEEE, 2023

work page 2023

Show all 17 references

[8]

Automatic root cause analysis via large language models for cloud incidents,

Y . Chen et al. , “Automatic root cause analysis via large language models for cloud incidents,” in Proceedings of the Nineteenth European Conference on Computer Systems , pp. 674–688, 2024

2024
[10]

Granger causality

“Granger causality.” https://en.wikipedia.org/wiki/Grangercausality
[11]

Page rank

“Page rank.” https://en.wikipedia.org/wiki/PageRank
[12]

The faiss library,

M. Douze et al. , “The faiss library,” arXiv preprint arXiv:2401.08281 , 2024

2024 arXiv
[13]

Chain-of-thought prompting elicits reasoning in large language models,

J. Wei et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24824–24837, 2022

2022
[14]

Large language models are zero-shot reasoners,

T. Kojima et al. , “Large language models are zero-shot reasoners,” Advances in neural information processing systems , vol. 35, pp. 22199– 22213, 2022

2022
[15]

Mixture- of-agents enhances large language model capabilities,

J. Wang, J. Wang, B. Athiwaratkun, C. Zhang, and J. Zou, “Mixture- of-agents enhances large language model capabilities,” arXiv preprint arXiv:2406.04692, 2024

2024 arXiv
[16]

Bertscore: Evaluating text generation with bert,

T. Zhang et al., “Bertscore: Evaluating text generation with bert,” arXiv preprint arXiv:1904.09675, 2019

1904 arXiv
[17]

Sentence-bert: Sentence embeddings using siamese bert- networks,

N. Reimers, “Sentence-bert: Sentence embeddings using siamese bert- networks,” arXiv preprint arXiv:1908.10084 , 2019

1908 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [9]

Automated Deterministic Auction Design with Objective Decomposition

Z. Duan et al., “Scalable virtual valuations combinatorial auction design by combining zeroth-order and first-order optimization method,” arXiv preprint arXiv:2402.11904, 2024

work page Pith review arXiv 2024

[2] [1]

Microrca: Root cause localization of performance issues in microservices,

L. Wu, J. Tordsson, E. Elmroth, and O. Kao, “Microrca: Root cause localization of performance issues in microservices,” in NOMS 2020- 2020 IEEE/IFIP Network Operations and Management Symposium , pp. 1–9, IEEE, 2020

work page 2020

[3] [2]

A comprehensive survey on root cause analysis in (micro) services: Methodologies, challenges, and trends,

T. Wang and G. Qi, “A comprehensive survey on root cause analysis in (micro) services: Methodologies, challenges, and trends,” arXiv preprint arXiv:2408.00803, 2024

arXiv 2024

[4] [3]

Exploring llm-based agents for root cause analysis,

D. Roy et al., “Exploring llm-based agents for root cause analysis,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering , pp. 208–219, 2024

work page 2024

[5] [4]

Gpt-4 technical report,

J. Achiam et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

arXiv 2023

[6] [5]

Mining causality of network events in log data,

S. Kobayashi, K. Otomo, K. Fukuda, and H. Esaki, “Mining causality of network events in log data,” IEEE Transactions on Network and Service Management, vol. 15, no. 1, pp. 53–67, 2017

work page 2017

[7] [6]

Root cause analysis for microservice system based on causal inference: How far are we?,

L. Pham, H. Ha, and H. Zhang, “Root cause analysis for microservice system based on causal inference: How far are we?,” arXiv preprint arXiv:2408.13729, 2024

arXiv 2024

[8] [7]

Recommending root-cause and mitigation steps for cloud incidents using large language models,

T. Ahmed et al. , “Recommending root-cause and mitigation steps for cloud incidents using large language models,” in 2023 IEEE/ACM 45th Intl Conf on Software Engineering (ICSE) , pp. 1737–1749, IEEE, 2023

work page 2023