Pith. sign in

REVIEW 2 major objections 1 minor 1 cited by

LLM agents identify correct root causes but verify their causal paths to symptoms in only 61.5 percent of cases.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-07-01 07:02 UTC pith:766HXIZB

load-bearing objection The paper gives a new RCA benchmark with step-wise causal paths via PAVE and shows LLMs often spot a root service without tracing the actual chain, but the labeling protocol lacks reported validation. the 2 major comments →

arxiv 2606.27154 v2 pith:766HXIZB submitted 2026-06-25 cs.AI

OpenRCA 2.0: From Outcome Labels to Causal Process Supervision

Aoyang Fang , Yifan Yang , Jin'ao Shang , Qisheng Lu , Junjielung Xu , Rui Wang , Songhan Zhang , Yuzhong Zhang
show 2 more authors
Boxi Yu Pinjia He
This is my paper
classification cs.AI
keywords root cause analysisLLM agentscausal propagation pathsfault injectionprocess supervisionbenchmarkcausal annotations
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing root cause analysis datasets for LLMs label only the final root cause service without the propagation path that connects it to the observed symptom. This design reduces evaluation to pattern matching instead of testing whether agents can trace verified causal chains. The paper introduces the PAVE protocol, which reconstructs those paths by applying known fault injection interventions and verifying forward from cause to effect. The resulting OpenRCA 2.0 benchmark contains 500 cross-system instances with step-wise causal annotations. Across 11 frontier models, exact root-cause set recovery averages 20.7 percent while the gap between spotting a correct service and grounding it in a verified path reaches 14.5 points.

Core claim

By replacing outcome-only labels with step-wise causal process supervision, OpenRCA 2.0 shows that frontier LLMs recover the exact root-cause set in only 20.7 percent of cases on average. Agents locate at least one correct root-cause service in 76.0 percent of cases yet succeed in grounding that service inside a verified causal propagation path to the symptom in just 61.5 percent of cases. Outcome-only scoring therefore conceals the ungrounded diagnosis failure mode that step-wise annotations expose.

What carries the argument

The PAVE protocol, which reconstructs causal propagation paths by applying known fault-injection interventions and performing forward verification from cause to effect rather than backward inference from symptoms.

Load-bearing premise

Known interventions from fault injection can accurately reconstruct the true causal propagation paths without introducing labeling errors or missing links.

What would settle it

Independent measurement of causal links in a production system that shows systematic mismatches with the paths reconstructed by the PAVE protocol on the same injected faults.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Exact recovery of the full root-cause set succeeds in only 20.7 percent of cases across frontier models.
  • Agents reach a correct root-cause service in 76.0 percent of cases but ground that service in a verified causal path in only 61.5 percent.
  • Outcome-only evaluation masks the ungrounded diagnosis failure mode that step-wise labels reveal.
  • Causal process supervision is required to build trustworthy LLM-based root cause analysis agents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same gap between identifying an answer and verifying its causal justification may appear in other multi-step agent tasks that involve tracing effects through complex systems.
  • Benchmarks limited to outcome labels may systematically overestimate agent performance whenever the domain requires explicit causal chaining.
  • The forward-verification approach could be adapted to label propagation steps in domains such as software debugging or medical differential diagnosis.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper argues that existing RCA benchmarks label only root causes rather than causal propagation paths, reducing evaluation to pattern matching. It introduces the PAVE protocol, which uses forward verification via known fault-injection interventions to reconstruct step-wise causal paths, yielding the OpenRCA 2.0 benchmark of 500 cross-system instances with these annotations. Evaluation across 11 frontier LLMs reports 20.7% exact root-cause set recovery on average; relaxing the criterion shows 76.0% of agents identify at least one correct root-cause service while only 61.5% ground it in a verified causal path to the symptom.

Significance. If the PAVE annotations prove reliable, the distinction between service identification and path-grounded diagnosis identifies a concrete failure mode that outcome-only metrics obscure, providing a clearer signal for improving LLM agent reasoning and tool use in RCA. The work supplies the first publicly described step-wise causal benchmark in this domain.

major comments (2)
  1. [PAVE protocol description] PAVE protocol description: the central performance gap (76.0% vs 61.5%) rests on the claim that fault-injection interventions produce exhaustive, error-free causal paths; however, the manuscript provides no inter-annotator agreement, error analysis, or validation against confounding paths for the 500 instances, which directly undermines the reliability of the 'ungrounded diagnosis' metric.
  2. [Dataset construction] Dataset construction (500 instances): the abstract states the benchmark is 'cross-system' but supplies no criteria for instance selection, system diversity, or how interventions were chosen to ensure the reconstructed paths are representative rather than artifactual.
minor comments (1)
  1. [Abstract] The abstract reports aggregate percentages without per-model breakdowns or confidence intervals, which would help readers assess variability across the 11 LLMs.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments highlighting the importance of validating the PAVE protocol and clarifying dataset construction. We respond point by point below, defending the objective nature of our intervention-based annotations while agreeing to strengthen the manuscript where gaps exist.

read point-by-point responses

  1. Referee: [PAVE protocol description] PAVE protocol description: the central performance gap (76.0% vs 61.5%) rests on the claim that fault-injection interventions produce exhaustive, error-free causal paths; however, the manuscript provides no inter-annotator agreement, error analysis, or validation against confounding paths for the 500 instances, which directly undermines the reliability of the 'ungrounded diagnosis' metric.

    Authors: PAVE reconstructs paths via deterministic forward verification from known fault-injection interventions rather than subjective human labeling, rendering traditional inter-annotator agreement inapplicable. The paths are verifiable by construction through the interventions themselves. We nevertheless agree that an explicit error analysis and discussion of potential confounding paths would strengthen the reliability claim for the ungrounded diagnosis metric and will add this to the revised manuscript. revision: partial

  2. Referee: [Dataset construction] Dataset construction (500 instances): the abstract states the benchmark is 'cross-system' but supplies no criteria for instance selection, system diversity, or how interventions were chosen to ensure the reconstructed paths are representative rather than artifactual.

    Authors: We will expand the manuscript with a dedicated section detailing the instance selection criteria, quantitative measures of system diversity across the 500 cross-system cases, and the rationale for choosing interventions to ensure the causal paths reflect representative RCA scenarios rather than artifacts. revision: yes

Circularity Check

0 steps flagged

No circularity; empirical benchmark and evaluations are independent

full rationale

The paper introduces the PAVE labeling protocol and applies it to create OpenRCA 2.0, then reports empirical LLM performance metrics on that benchmark. No equations, derivations, fitted parameters renamed as predictions, or self-citation chains appear in the text. The reported figures (20.7%, 76.0%, 61.5%) are direct evaluation outcomes on the constructed dataset and do not reduce to the inputs by construction. The benchmark construction and the outcome measurements remain separate and externally falsifiable.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

Abstract-only information limits the ledger; the central claim depends on the validity of the fault-injection-based labeling protocol, which is asserted but not evidenced here.

axioms (1)
  • domain assumption Known interventions from fault injection accurately reconstruct causal propagation paths
    This underpins the PAVE protocol and the step-wise annotations in OpenRCA 2.0.
invented entities (1)
  • PAVE labeling protocol no independent evidence
    purpose: Step-wise causal path annotation for RCA datasets
    New method introduced to address the outcome-only limitation of prior datasets.

pith-pipeline@v0.9.1-grok · 5794 in / 1236 out tokens · 43452 ms · 2026-07-01T07:02:22.813166+00:00 · methodology

0 comments
read the original abstract

Root cause analysis (RCA) poses a holistic test of LLM agentic capabilities, such as long-context understanding, multi-step reasoning, and tool use. However, existing datasets suffer from a fundamental gap: they label only the root cause, not the propagation path connecting it to the observed symptom, which largely simplifies the task to naive pattern matching. To support rigorous evaluation, we introduce PAVE, a step-wise labeling protocol that leverages known interventions from fault injection to reconstruct causal propagation paths. The mechanism is forward verification: reasoning from cause to effect rather than inferring backward from symptoms. Applying PAVE yields OpenRCA 2.0 (500 instances), the first cross-system RCA benchmark with step-wise causal annotations for LLM agents. Across 11 frontier LLMs, recovering the exact root-cause set succeeds in only 20.7% of cases on average. To locate where this difficulty lies, we relax the criterion and find what we call the ungrounded diagnosis: agents identify at least one correct root-cause service in 76.0% of cases, but ground that service in a verified causal propagation path to the observed symptom in only 61.5%. Outcome-only evaluation hides this failure mode; step-wise causal ground truth is the missing piece for trustworthy LLM-based RCA agents.

Figures

Figures reproduced from arXiv: 2606.27154 by Aoyang Fang, Boxi Yu, Jin'ao Shang, Junjielung Xu, Pinjia He, Qisheng Lu, Rui Wang, Songhan Zhang, Yifan Yang, Yuzhong Zhang.

Figure 1
Figure 1. Figure 1: An ungrounded diagnosis on the running NetworkDelay failure in Seat ser￾vice. From the known intervention (top), PAVE reconstructs the verified causal path Seat→Travel→Order→Gateway (middle). The agent names the correct root cause but produces a graph that skips Travel (bottom): outcome-only evaluation scores this as a success, whereas process￾level evaluation surfaces the missing edge. into the agent’s pr… view at source ↗
Figure 2
Figure 2. Figure 2: The RCA setting. (a) A microservice system forms a two-layer dependency graph. A fault injected at Seat cascades along the RPC chain (Seat → Travel → Order) and may also propagate vertically through shared infrastructure (e.g., co-located pods). (b) The agent observes only telemetry (traces, metrics, logs) and must reason backward to the originating fault; the intervention itself is hidden. set Π∗ . Phase … view at source ↗
Figure 3
Figure 3. Figure 3: Coarse-to-Fine verification refines a cluttered observation graph into one verified causal chain. Running TrainTicket example: the true root cause is Seat, while Route is a benign service that appears anomalous due to background noise. Input. Every observed anomaly and every potential edge are kept, so the root cause cannot be picked out. Phase 1 (Structural Pruning). Edges marked × are removed because the… view at source ↗
Figure 4
Figure 4. Figure 4: Spurious causal reasoning case. (a) Ground truth shows fault propagation through inter [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures

    cs.AI 2026-07 conditional novelty 6.0

    The paper introduces a 41-mode taxonomy that assigns each agent failure to an interaction edge and a fault side, and shows LLM judges can reproduce the labels with Cohen's κ=0.76.

Reference graph

Works this paper leans on

37 extracted references · 37 canonical work pages · cited by 1 Pith paper · 2 internal anchors

  1. [1]

    Openrca: Can large language models locate the root cause of software failures? InThe Thirteenth International Conference on Learning Representations, 2025

    Junjielong Xu, Qinan Zhang, Zhiqing Zhong, Shilin He, Chaoyun Zhang, Qingwei Lin, Dan Pei, Pinjia He, Dongmei Zhang, and Qi Zhang. Openrca: Can large language models locate the root cause of software failures? InThe Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  2. [2]

    Aiopslab: A holistic framework to evaluate ai agents for enabling autonomous clouds

    Yinfang Chen, Manish Shetty, Gagan Somashekar, Minghua Ma, Yogesh Simmhan, Jonathan Mace, Chetan Bansal, Rujia Wang, and Saravan Rajmohan. Aiopslab: A holistic framework to evaluate ai agents for enabling autonomous clouds. InMLSys, 2025. URL https:// openreview.net/forum?id=3EXBLwGxtq

    work page 2025

  3. [3]

    Stratus: A multi-agent system for autonomous reliability engineering of modern clouds.arXiv preprint arXiv:2506.02009v2, 05 2025

    Yinfang Chen, Jiaqi Pan, Jackson Clark, Yiming Su, Noah Zheutlin, Bhavya Bhavya, Rohan Arora, Yu Deng, Saurabh Jha, and Tianyin Xu. Stratus: A multi-agent system for autonomous reliability engineering of modern clouds.arXiv preprint arXiv:2506.02009v2, 05 2025. URL https://arxiv.org/abs/2506.02009v2

    work page arXiv 2025

  4. [4]

    Thinkfl: Self-refining failure localization for microservice systems via reinforcement fine-tuning.ACM Transactions on Software Engineering and Methodology,

    Lingzhe Zhang, Yunpeng Zhai, Tong Jia, Chiming Duan, Siyu Yu, Jinyang Gao, Bolin Ding, Zhonghai Wu, and Ying Li. Thinkfl: Self-refining failure localization for microservice systems via reinforcement fine-tuning.ACM Transactions on Software Engineering and Methodology,

    work page

  5. [5]

    URLhttps://doi.org/10.1145/3789262

    doi: 10.1145/3789262. URLhttps://doi.org/10.1145/3789262

    work page doi:10.1145/3789262

  6. [6]

    Fault analysis and debugging of microservice systems: Industrial survey, benchmark system, and empirical study.IEEE Transactions on Software Engineering, 47(2):243–260, 2018

    Xiang Zhou, Xin Peng, Tao Xie, Jun Sun, Chao Ji, Wenhai Li, and Dan Ding. Fault analysis and debugging of microservice systems: Industrial survey, benchmark system, and empirical study.IEEE Transactions on Software Engineering, 47(2):243–260, 2018

    work page 2018

  7. [7]

    Rcaeval: A bench- mark for root cause analysis of microservice systems with telemetry data

    Luan Pham, Hongyu Zhang, Huong Ha, Flora Salim, and Xiuzhen Zhang. Rcaeval: A bench- mark for root cause analysis of microservice systems with telemetry data. InCompanion Proceedings of the ACM on Web Conference 2025, pages 777–780, 2025

    work page 2025

  8. [8]

    Rethinking the evaluation of microservice rca with a fault propagation-aware benchmark.arXiv preprint arXiv:2510.04711v2, 10 2025

    Aoyang Fang, Songhan Zhang, Yifan Yang, Haotong Wu, Junjielong Xu, Xuyang Wang, Rui Wang, Manyi Wang, Qisheng Lu, and Pinjia He. Rethinking the evaluation of microservice rca with a fault propagation-aware benchmark.arXiv preprint arXiv:2510.04711v2, 10 2025. URL https://arxiv.org/abs/2510.04711v2

    work page arXiv 2025

  9. [9]

    Cambridge university press, 2009

    Judea Pearl.Causality. Cambridge university press, 2009

    work page 2009

  10. [10]

    Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gain

    Wei Liu, Siya Qi, Yali Du, and Yulan He. Self-play only evolves when self-synthetic pipeline ensures learnable information gain.arXiv preprint arXiv:2603.02218v1, 02 2026. URL https://arxiv.org/abs/2603.02218v1

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  11. [11]

    https://github.com/delimitrou/DeathStarBench/tree/master,

    Deathstarbench. https://github.com/delimitrou/DeathStarBench/tree/master,

    work page

  12. [12]

    Accessed: 2026-02-10

    work page 2026

  13. [13]

    https://github.com/open-telemetry/opentelemetry-demo,

    Opentelemetry demo. https://github.com/open-telemetry/opentelemetry-demo,

    work page

  14. [14]

    Accessed: 2026-05-05

    work page 2026

  15. [15]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InThe eleventh international conference on learning representations, 2022

    work page 2022

  16. [16]

    Self-balancing agentic AI: Test-time diffusion and context engineering re-imagined for deep research.https://github.com/thinkdepthai/Deep_Research, 2025

    Paichun Lin. Self-balancing agentic AI: Test-time diffusion and context engineering re-imagined for deep research.https://github.com/thinkdepthai/Deep_Research, 2025

    work page 2025

  17. [17]

    Microrank: End-to-end latency issue localization with extended spectrum analysis in microservice environments

    Guangba Yu, Pengfei Chen, Hongyang Chen, Zijie Guan, Zicheng Huang, Linxiao Jing, Tianjun Weng, Xinmeng Sun, and Xiaoyun Li. Microrank: End-to-end latency issue localization with extended spectrum analysis in microservice environments. InProceedings of the Web Conference 2021, pages 3087–3098, 2021. 10

    work page 2021

  18. [18]

    Dynacausal: Dynamic causality-aware root cause analysis for distributed microservices.arXiv preprint arXiv:2510.22613v1, 10 2025

    Songhan Zhang, Aoyang Fang, Yifan Yang, Ruiyi Cheng, Xiaoying Tang, and Pinjia He. Dynacausal: Dynamic causality-aware root cause analysis for distributed microservices.arXiv preprint arXiv:2510.22613v1, 10 2025. URLhttps://arxiv.org/abs/2510.22613v1

    work page arXiv 2025

  19. [19]

    MIT press, 2000

    Peter Spirtes, Clark N Glymour, and Richard Scheines.Causation, prediction, and search. MIT press, 2000

    work page 2000

  20. [20]

    Root cause analysis of failures in microservices through causal discovery.Advances in Neural Information Processing Systems, 35:31158–31170, 2022

    Azam Ikram, Sarthak Chakraborty, Subrata Mitra, Shiv Saini, Saurabh Bagchi, and Murat Kocaoglu. Root cause analysis of failures in microservices through causal discovery.Advances in Neural Information Processing Systems, 35:31158–31170, 2022

    work page 2022

  21. [21]

    Root cause analysis of anomalies in multivariate time series through granger causal discovery

    Xiao Han, Saima Absar, Lu Zhang, and Shuhan Yuan. Root cause analysis of anomalies in multivariate time series through granger causal discovery. InThe Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  22. [22]

    Constructing large-scale real-world benchmark datasets for aiops.arXiv preprint arXiv:2208.03938v1, 08 2022

    Zeyan Li, Nengwen Zhao, Shenglin Zhang, Yongqian Sun, Pengfei Chen, Xidao Wen, Minghua Ma, and Dan Pei. Constructing large-scale real-world benchmark datasets for aiops.arXiv preprint arXiv:2208.03938v1, 08 2022. URLhttps://arxiv.org/abs/2208.03938v1

    work page arXiv 2022

  23. [23]

    Nezha: Interpretable fine-grained root causes analysis for microservices on multi-modal observability data

    Guangba Yu, Pengfei Chen, Yufeng Li, Hongyang Chen, Xiaoyun Li, and Zibin Zheng. Nezha: Interpretable fine-grained root causes analysis for microservices on multi-modal observability data. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 553–565, 2023

    work page 2023

  24. [24]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe Twelfth International Conference on Learning Representations, 2023

    work page 2023

  25. [25]

    Solving math word problems with process- and outcome-based feedback

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback.arXiv preprint arXiv:2211.14275v1, 11 2022. URL https://arxiv.org/abs/2211.14275v1

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  26. [26]

    Versaprm: Multi-domain process reward model via synthetic reasoning data

    Thomas Zeng, Shuibai Zhang, Shutong Wu, Christian Classen, Daewon Chae, Ethan Ewer, Minjae Lee, Heeju Kim, Wonjun Kang, Jackson Kunde, et al. Versaprm: Multi-domain process reward model via synthetic reasoning data. InForty-second International Conference on Machine Learning, 2025

    work page 2025

  27. [27]

    Dynamic and generalizable process reward modeling

    Zhangyue Yin, Qiushi Sun, Zhiyuan Zeng, Qinyuan Cheng, Xipeng Qiu, and Xuan-Jing Huang. Dynamic and generalizable process reward modeling. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4203–4233, 2025

    work page 2025

  28. [28]

    Applicable Levels

    Yibo Yan, Jiamin Su, Jianxiang He, Fangteng Fu, Xu Zheng, Yuanhuiyi Lyu, Kun Wang, Shen Wang, Qingsong Wen, and Xuming Hu. A survey of mathematical reasoning in the era of multimodal large language model: Benchmark, method & challenges. InFindings of the Association for Computational Linguistics: ACL 2025, pages 11798–11827, 2025. 11 Appendix organization...

    work page 2025

  29. [29]

    query_parquet_files: DuckDB SQL on parquets in this case dir

    work page

  30. [30]

    list_tables_in_directory: list parquets

    work page

  31. [31]

    get_schema: column types of a parquet

    work page

  32. [32]

    ## Hard limits •Tool-call budget: aim for∼50 calls; extend if the evidence genuinely warrants it

    think_tool: REQUIRED after each query; summarize, plan next step. ## Hard limits •Tool-call budget: aim for∼50 calls; extend if the evidence genuinely warrants it. Hard cap is 100, at which point the runtime forces a stop. •Spend the budget efficiently: list_tables_in_directory once, get_schema on the files you actually plan to query, then spend the rest ...

    work page

  33. [33]

    list_tables_in_directory to confirm the parquet files

    work page

  34. [34]

    get_schema on the relevant ones (start with abnormal_traces)

    work page

  35. [35]

    Diff abnormal vs normal: error rates, latency, status codes, log levels

    work page

  36. [36]

    Trace the call chain (parent_span_id→span_id) to find the earliest service whose own work, not its dependency’s, went wrong

    work page

  37. [37]

    Agent Contract

    Decide every root cause and every propagation edge. More than one root cause is possible; note each separately when evidence supports it. USER: RCA_ANAL YSIS_UP {incident_description} 24 F.4.2 Synthesis Phase SYSTEM: COMPRESS_FINDINGS_SP You are an RCA synthesizer. Today’s date is {date}. Your job: convert the investigation messages above into a single ST...

    work page