RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Data

Flora Salim; Hongyu Zhang; Huong Ha; Luan Pham; Xiuzhen Zhang

REVIEW 4 major objections 5 minor 3 cited by

RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Data

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RCAEval provides the first comprehensive open-source benchmark for root cause analysis in microservice systems, with 735 failure cases, 11 fault types, and 15 reproducible baselines.

desk verdict RCAEval's code-level dataset is genuinely new, but the paper never runs a baseline on it, so the 'comprehensive' claim is ahead of the evidence. read the letter →

arxiv 2412.17015 v5 pith:G6OM2EER submitted 2024-12-22 cs.SE

Luan Pham , Hongyu Zhang , Huong Ha , Flora Salim , Xiuzhen Zhang This is my paper · ORCID

classification cs.SE

keywords rootcauseanalysismicroservicesbenchmarktelemetrydatafaultinjectionevaluationframeworkmulti-sourcecode-levelfaults

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces RCAEval, a benchmark that standardizes how root cause analysis (RCA) of microservice systems is evaluated. It provides 735 failure cases across three microservice systems, covering 11 fault types including code-level faults, with metrics, logs, and traces for each case. The accompanying open-source evaluation framework offers 15 reproducible baselines and measures both coarse-grained (root cause service) and fine-grained (root cause indicator) performance using $ ext{AC@}k$ and $ ext{Avg@}k$. If the benchmark holds up, researchers can stop comparing RCA methods on incompatible datasets and instead evaluate new approaches against a common standard.

What carries the argument

The core mechanism is the benchmark itself: fault injection into three real microservice systems under load, multi-source telemetry collection (Prometheus metrics, logs, Jaeger traces), and annotation of ground-truth root cause service and indicator for each failure. The evaluation framework then ranks candidate root causes and scores them with $ ext{AC@}k$ and $ ext{Avg@}k$, making the benchmark a shared testbed for comparing RCA methods. The datasets are deliberately built from realistic open-source systems and fault types drawn from empirical studies of production failures.

What would settle it

Take a random sample of failure cases, have two or more independent DevOps engineers annotate the root cause service and indicator without seeing the original labels, and measure inter-annotator agreement; if agreement is substantially below the level needed for the $ ext{AC@}k$ metrics to be meaningful, the benchmark's ground truth is not robust. For code-level faults, one can directly check whether the faulty line identified by the stack trace matches the actual injected code modification, and a mismatch rate above a small threshold would invalidate the fine-grained labels.

Watch

Extended reading notes

Core claim

RCAEval claims to be the first comprehensive, open-source benchmark for root cause analysis of microservice systems. The paper assembles three datasets (RE1, RE2, RE3) collected from Online Boutique, Sock Shop, and Train Ticket, totaling 735 failure cases with 11 fault types: four resource faults (CPU, MEM, DISK, SOCKET), two network faults (DELAY, LOSS), and five code-level faults (incorrect parameter values, missing parameters, missing function calls, incorrect return values, missing exception handlers). Each case includes telemetry data (metrics, logs, traces) and an annotated root cause service and root cause indicator. The evaluation framework supplies 15 baseline methods spanning metric-based, trace-based, and multi-source approaches, with $ ext{AC@}k$ and $ ext{Avg@}k$ metrics for coarse- and fine-grained evaluation.

Load-bearing premise

The benchmark's results are only as trustworthy as the 735 root cause annotations, which were verified by a single DevOps engineer with five years of experience.

Editorial extensions

If this is right

Researchers can apply the 15 existing RCA baselines to the same 735 failure cases and compare their results directly, removing dataset mismatch as a confounder.
The inclusion of five code-level fault types enables evaluation of RCA methods that go beyond resource and network faults, which is new territory for the field.
The framework supports both coarse-grained and fine-grained RCA evaluation, so methods can be tested on localizing the root cause service and the specific root cause indicator.
Because the benchmark is open-source and installable via PyPI, new RCA methods can be plugged in and compared against a documented, reproducible set of baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the annotations prove reliable, RCAEval could become the default comparison point in the RCA field, much as standardized benchmarks have shaped other machine learning subfields; this trajectory is plausible but not claimed by the paper.
The code-level fault subset may reveal that log-based stack trace analysis is essential for fine-grained RCA, since metric-only methods cannot identify line-level bugs; the paper does not test this directly.
The three systems differ in scale (12 to 64 services), so the benchmark could be used to study how RCA methods scale with system complexity, an extension the authors did not investigate.
A natural testable extension is to use the framework to benchmark a newly proposed RCA method against the included baselines; the paper invites this use but does not demonstrate it.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the dataset may be the real contribution. RE3, with five code-level fault types on three systems plus logs and traces, is new to this literature, and the public framework with 15 baselines is a step toward consistent evaluation. But the paper overstates what it has demonstrated: the experiments never touch RE3, so the headline novelty is unexercised. That is not fatal in a 4-page companion, but it should be fixed in a full version or the claims trimmed.

What I like: the data collection is substantial—735 cases, three systems, 11 fault types, multi-source telemetry. The format is well documented, and the code is on GitHub with a PyPI package. The authors adapted several existing implementations and say they reproduced their reported results, which is more than most benchmark papers bother to do. PDiagnose was re-implemented because the original source is unavailable, which adds some risk, but it is a reasonable choice.

Where it is soft:

1. The code-level dataset, the main novelty over PyRCA, AIOps 2020, and their own ASE'24 work, is absent from Table 6. No baseline is run on RE3. The central claim of supporting code-level RCA is supported by data files and a GitHub demo, not by experimental evidence. If the adapters fail on RE3 or the labels are wrong, the 'comprehensive' claim evaporates.

2. Label verification is thin. Section 3.3 says a single DevOps engineer verified data quality. No inter-annotator agreement, no automated cross-check. For resource and network faults, ground truth is mostly determined by the injection method, so risk is modest. For code-level faults, where the root cause indicator is an interpretation of stack traces and error logs, the risk is real. The paper should either provide a second annotator or acknowledge this limitation explicitly.

3. The evaluation in Table 6 has no error bars or significance tests, and it covers only Train Ticket. That is fine for a preliminary result, but it does not support the word 'comprehensive' as used throughout.

I disagree with nothing in the stress-test note; it lands.

Who this is for: anyone building or comparing RCA methods for microservices. The resource could become a standard benchmark, which makes the paper worth citing even with its weaknesses. It deserves a proper referee—I would send it out, but I would ask for RE3 results or a narrowed scope, and at least a paragraph on label validation.

For your reading group: maybe, because the data is interesting but the paper is thin.

Referee Report

4 major / 5 minor

Summary. RCAEval is a benchmark paper for root cause analysis (RCA) of microservice systems. The authors contribute three datasets (RE1, RE2, RE3) totaling 735 failure cases collected from Online Boutique, Sock Shop, and Train Ticket, spanning 11 fault types in three categories: resource faults, network faults, and code-level faults. The datasets include multi-source telemetry data (metrics, logs, and traces), and the paper also releases an evaluation framework with 15 baselines and coarse-grained and fine-grained evaluation metrics (AC@k and Avg@k). The paper reports preliminary experiments on the RE2 Train Ticket subset for coarse-grained service ranking, and it positions RCAEval as the first comprehensive open-source benchmark for RCA in microservice systems.

Significance. If the dataset labels are correct and the code-level fault subset is usable by existing methods, RCAEval is a genuinely useful community resource: it is larger in fault-type coverage than most prior RCA evaluations, it includes logs and traces in addition to metrics, and it ships an open-source evaluation framework with reproducible baselines. The aggregate statistics are internally consistent (375 + 270 + 90 = 735 cases; 4 + 2 + 5 = 11 fault types), and the paper explicitly builds on and extends the authors' prior ASE'24 framework, which provides a degree of continuity. However, the central novelty of the benchmark is not yet validated in the paper: the RE3 code-level dataset is never used in the experimental evaluation, the fine-grained root-cause-indicator evaluation is not demonstrated, and the ground-truth labels rest on verification by a single DevOps engineer with no reported inter-annotator agreement or independent audit. The significance of the contribution is therefore conditional on these gaps being addressed.

major comments (4)

[Section 5, Table 6] The experimental section evaluates only coarse-grained service ranking on the RE2 Train Ticket subset; neither the RE3 code-level dataset nor the fine-grained root-cause-indicator level is exercised anywhere in the paper. Since the paper's novelty over prior benchmarks (Table 1) is precisely the code-level fault coverage and the multi-source combination, the claim that RCAEval provides a 'comprehensive' benchmark and evaluation framework is currently supported only by dataset statistics and a GitHub demonstration, not by experimental evidence that the provided baselines can consume RE3 and reproduce its labels. Please either add experiments on RE3 (at least on one system) with the provided baselines, or present a fine-grained evaluation on RE2, and then state the benchmark's scope accordingly.
[Section 3.2, Section 3.3] The quality of the entire benchmark depends on the correctness of the annotated root cause service and root cause indicator for each failure case, yet Section 3.3 states only that a single DevOps engineer with five years of experience verified the data. No inter-annotator agreement, automated consistency check, or independent audit is reported. For resource and network faults the indicator is largely determined by the injected fault type, but for code-level faults (Section 3.2) the indicator is derived from stack traces, error logs, or response codes, which is an interpretive judgment involving the faulty line of code. Please report a detailed labeling protocol, provide a sample of independently re-annotated cases with agreement statistics, and release any label-audit artifacts; without this, the baseline comparisons in Table 6 and any downstream benchmark conclusions rest on unvalidated labels.
[Section 5, Table 6] The reported baseline results are point estimates with no variance, no confidence intervals, and no significance tests. RE2 includes three repetitions per fault-service pair, and these repetitions appear to be collapsed into the AC@k and Avg@k numbers, making it impossible to tell whether differences such as TraceRCA versus multi-source BARO on DELAY or LOSS reflect real superiority or noise. Please report per-fault means with standard deviations (or per-repetition results) and, where feasible, a simple significance test; this is necessary for the benchmark to support fair method comparison.
[Section 5, Section 4.1] The description of the baseline set contains an inconsistency that should be corrected: the text says 'we select 11 baselines: 5 metric-based RCA methods (BARO, CausalRCA, CIRCA, MicroCause), 2 trace-based RCA methods, and 4 multi-source RCA methods', but the enumerated metric-based list has four names and Table 6 includes an additional metric RCD row and a separate multi-source RCD row. Please reconcile the count and the table rows, and clarify which configuration of each baseline is used (for example, whether metric RCD and multi-source RCD differ only in input data or also in algorithm).

minor comments (5)

[Section 4.1] For PDiagnose, the paper says the source code is unavailable and the authors implemented it following previous works; please state which commit or version of their implementation is used and whether its outputs were validated against any original result.
[Section 5, Table 6] The table header appears as 'Ag@5' in the reproduction of Table 6; please ensure the heading matches the 'Avg@5' metric defined in Section 4.2.
[Section 3.3] The load range is stated as 10–200 requests per second, but no rationale or sensitivity analysis is given; a brief statement of how the load was varied across services and repetition would improve reproducibility.
[References] References [6] and [15] appear to describe the same paper (Pham et al., ASE'24) with different titles; please deduplicate or clarify if they are distinct works.
[Section 3.4] The data format description gives sample rows for metrics, logs, and traces, but it does not state how timestamps across the three data sources are aligned or how the failure-injection period is marked; please add a precise alignment and windowing description.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: benchmark construction and baseline evaluation are empirically grounded; no derivation reduces to its inputs.

full rationale

This paper contributes datasets and an evaluation framework rather than a mathematical derivation, so no result is equivalent to its inputs by construction. The RE1 dataset and the evaluation framework are explicitly reused or extended from the authors' prior work [6], and the paper cites that prior work for claims about synthetic datasets and metric-based RCA; these are ordinary incremental self-citations with independent content, not load-bearing circularity, because the benchmark's central claims are supported by the described collection procedure (fault injection with stress-ng and tc, source-code modifications for code-level faults, telemetry gathered via Prometheus, cAdvisor, Istio, Vector, Loki, and Jaeger) and by the executed baseline experiments in Table 6. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' earlier papers, and no equation equates an output with an input. The RE3 code-level labels are determined by a stated labeling methodology (stack traces in logs, or error logs/response codes when stack traces are unavailable), which is a ground-truth construction choice rather than a circular reduction; the absence of RE3 results in the preliminary experiments is a coverage and validation gap, not evidence of circularity. The benchmark is self-contained against external systems (Online Boutique, Sock Shop, Train Ticket) and external baselines with reproduced implementations, so the honest finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The benchmark's validity depends on the fault injections being realistic, the telemetry stack capturing the induced symptoms, the root-cause labels being correct, and the three open-source systems being representative of production microservices. I list these as domain assumptions. The two hand-chosen collection settings (load range and repetition counts) are the only numerical choices that directly shape the dataset size; there are no fitted model parameters and no invented entities.

free parameters (2)

Request load range = 10 to 200 requests per second
Chosen by the authors for data collection in Section 3.3; affects perceived realism and symptom severity, but is not fitted to data.
Repetition counts per fault-service pair = 5 for RE1, 3 for RE2 and RE3
Reported in Section 3; determines the 735 case total and the diversity of the datasets, set by hand rather than derived.

assumptions (4)

domain assumption Fault injection using stress-ng, tc, and source-code modifications produces failures that resemble real-world microservice failures.
Section 3.2 describes how each fault type was injected; the paper does not validate against naturally occurring failure data, so representativeness is assumed.
domain assumption The telemetry data collected by Prometheus, cAdvisor, Istio, Vector, Loki, and Jaeger correctly capture the symptoms of the injected faults.
Section 3.3 describes the collection stack but provides no data-quality checks beyond one DevOps engineer's verification.
domain assumption The annotated root-cause service and root-cause indicator labels are correct.
Section 3.2 says code-level root causes are determined from stack traces, error logs, or response codes, and Section 3.3 says a single DevOps engineer verified the data; no inter-annotator agreement or automated validation is reported.
domain assumption The three open-source systems (Online Boutique, Sock Shop, Train Ticket) are representative enough of production microservices for benchmark conclusions to transfer.
Section 3.1 admits that no single system can fully capture real-world diversity; the paper relies on the community's use of these systems.

how reviews work

0 comments

Cite this review

Pith. "Pith review of RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Data." pith.science (2026). https://pith.science/paper/G6OM2EER

@misc{pith2026241217015,
  author       = {Pith},
  title        = {Pith review of: RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G6OM2EER}},
  note         = {Machine review of arXiv:2412.17015}
}

read the original abstract

Root cause analysis (RCA) for microservice systems has gained significant attention in recent years. However, there is still no standard benchmark that includes large-scale datasets and supports comprehensive evaluation environments. In this paper, we introduce RCAEval, an open-source benchmark that provides datasets and an evaluation environment for RCA in microservice systems. First, we introduce three comprehensive datasets comprising 735 failure cases collected from three microservice systems, covering various fault types observed in real-world failures. Second, we present a comprehensive evaluation framework that includes fifteen reproducible baselines covering a wide range of RCA approaches, with the ability to evaluate both coarse-grained and fine-grained RCA. We hope that this ready-to-use benchmark will enable researchers and practitioners to conduct extensive analysis and pave the way for robust new solutions for RCA of microservice systems.

Figures

Figures reproduced from arXiv: 2412.17015 by the authors.

**Figure 1.** Overview of the RCAEval benchmark. 2 Background & Related Work In microservice systems, failures refer to a service’s inability to perform its intended functions, while faults represent the underlying causes of such failures (e.g., memory leaks). Root cause analysis (RCA) seeks to pinpoint the root causes of failures by analyzing multi-source telemetry data (i.e., metrics, logs, and traces) [6, 15, 16]. A major lim… view at source ↗

**Figure 2.** Illustration of our data collection setup. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Safe Remediation as Risk-Constrained Intervention Decision in Microservice Systems
cs.AI 2026-07 conditional novelty 6.0 of 10

A risk-constrained CMDP with a 3D risk filter and adaptive escalation gate is reported to reduce false remediation by 39% while improving success by 2.5 points over a runbook baseline on a microservice benchmark.
TN-AutoRCA: Benchmark Construction and Agentic Framework for Self-Improving Alarm-Based Root Cause Analysis in Telecommunication Networks
cs.CL 2025-07 conditional novelty 6.0 of 10

A new 530-scenario benchmark for telecom alarm root cause analysis, plus an iterative agent that lifts F1 from 58.99% to 91.79% by repeatedly repairing its code against the benchmark.
Autonomic Microservice Management via Agentic AI and MAPE-K Integration
cs.SE 2025-06 conditional novelty 4.0 of 10

A conceptual framework integrating MAPE-K with agentic AI for autonomous microservice anomaly management, including a proposed autonomic threshold for human oversight, offered without empirical validation.

Reference graph

Works this paper leans on

19 extracted references · 16 canonical work pages · cited by 3 Pith papers

[1]

https://github.com/FudanSELab/train-ticket

2024. https://github.com/FudanSELab/train-ticket

work page 2024
[2]

https://github.com/GoogleCloudPlatform/microservices-demo

2024. https://github.com/GoogleCloudPlatform/microservices-demo

work page 2024
[3]

https://github.com/microservices-demo/microservices-demo

2024. https://github.com/microservices-demo/microservices-demo

work page 2024
[4]

Qian Cheng et al. 2023. Ai for it operations (aiops) on cloud platforms: Reviews, opportunities and challenges. arXiv:2304.04661 (2023)

arXiv 2023
[5]

Domenico Cotroneo et al. 2019. How bad can a bug get? an empirical analysis of software failures in the openstack cloud computing platform. In FSE’19

work page 2019
[6]

Pham et al. 2024. Root Cause Analysis for Microservice System based on Causal Inference: How Far Are We?. In IEEE/ACM ASE’24

work page 2024
[7]

Chuanjia Hou et al. 2021. Diagnosing performance issues in microservices with heterogeneous data source. In 2021 IEEE ISPA/BDCloud/SocialCom/SustainCom

work page 2021
[8]

Azam Ikram et al. 2022. Root Cause Analysis of Failures in Microservices through Causal Discovery. In NeurIPS’22

work page 2022

Show all 19 references

[9]

Cheryl Lee et al. 2023. Eadro: An End-to-End Troubleshooting Framework for Microservices on Multi-source Data. In ICSE’23

2023
[10]

Mingjie Li et al. 2022. Causal Inference-Based Root Cause Analysis for Online Service Systems with Intervention Recognition. In KDD’22

2022
[11]

Zeyan Li et al. 2021. Practical Root Cause Localization for Microservice Systems via Trace Analysis. In IWQOS’21

2021
[12]

Zeyan Li et al. 2022. Constructing large-scale real-world benchmark datasets for aiops. arXiv preprint arXiv:2208.03938 (2022)

2022 arXiv
[13]

Cheng-Ming Lin and et al. 2024. Root Cause Analysis in Microservice Using Neural Granger Causal Discovery. In AAAI’24

2024
[14]

Chenghao Liu et al. 2023. PyRCA: A Library for Metric-based Root Cause Analysis. arXiv preprint arXiv:2306.11417 (2023)

2023 arXiv
[15]

Luan Pham, Huong Ha, and Hongyu Zhang. 2024. Root Cause Analysis for Microservices based on Causal Inference: How Far Are We?. In ASE’24. IEEE

2024
[16]

Luan Pham et al. 2024. BARO: Robust Root Cause Analysis for Microservices via Multivariate Bayesian Online Change Point Detection. In FSE’24

2024
[17]

Guangba Yu et al. 2021. Microrank: End-to-end latency issue localization with extended spectrum analysis in microservice environments. In WWW’21

2021
[18]

Guangba Yu et al. 2023. Nezha: Interpretable Fine-Grained Root Causes Analysis for Microservices on Multi-modal Observability Data. In FSE’23

2023
[19]

Shenglin Zhang et al. 2023. Robust failure diagnosis of microservice system through multimodal data. TSC’23 (2023)

2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

https://github.com/FudanSELab/train-ticket

2024. https://github.com/FudanSELab/train-ticket

work page 2024

[2] [2]

https://github.com/GoogleCloudPlatform/microservices-demo

2024. https://github.com/GoogleCloudPlatform/microservices-demo

work page 2024

[3] [3]

https://github.com/microservices-demo/microservices-demo

2024. https://github.com/microservices-demo/microservices-demo

work page 2024

[4] [4]

Qian Cheng et al. 2023. Ai for it operations (aiops) on cloud platforms: Reviews, opportunities and challenges. arXiv:2304.04661 (2023)

arXiv 2023

[5] [5]

Domenico Cotroneo et al. 2019. How bad can a bug get? an empirical analysis of software failures in the openstack cloud computing platform. In FSE’19

work page 2019

[6] [6]

Pham et al. 2024. Root Cause Analysis for Microservice System based on Causal Inference: How Far Are We?. In IEEE/ACM ASE’24

work page 2024

[7] [7]

Chuanjia Hou et al. 2021. Diagnosing performance issues in microservices with heterogeneous data source. In 2021 IEEE ISPA/BDCloud/SocialCom/SustainCom

work page 2021

[8] [8]

Azam Ikram et al. 2022. Root Cause Analysis of Failures in Microservices through Causal Discovery. In NeurIPS’22

work page 2022