{"id":"26c79efd-4c85-4af8-971f-825359002d29","arxiv_id":"2412.17015","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RCAEval provides three telemetry datasets with 735 microservice failure cases and an evaluation framework with 15 baselines for metric-based, trace-based, and multi-source root cause analysis.","lead":"This paper introduces RCAEval, a benchmark for root cause analysis of microservice systems, with 735 failure cases across three systems and eleven fault types. It also provides an open-source evaluation framework with fifteen baseline methods, so researchers can compare RCA approaches on the same data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The code-level RE3 dataset—the paper's main novelty—is never evaluated, so the 'comprehensive benchmark' claim rests on an unexercised and unvalidated subset.","rationale":"The reader's weakest assumption (label correctness) is directionally right, but partly mitigated: for RE1/RE2 resource and network faults, the root-cause service is known from the fault-injection design (which service was perturbed) and the indicator is the fault-type metric, so label errors would mostly arise only if injection failed. The genuinely subjective and novel part is RE3 code-level labels, which are derived by interpreting logs/traces and verified by a single engineer. My more specific concern is that the paper never demonstrates any baseline on RE3, so the central 'comprehensive benchmark' claim is not backed by in-paper evidence. The concrete test above would settle this. I keep CONDITIONAL because the benchmark may well be sound; it needs either an artifact run or independent labeling to substantiate the code-level contribution.","tokens_in":7712,"tokens_out":4948,"duration_ms":47282,"concrete_test":"Run the released harness on all 30 RE3 Train Ticket code-level cases with at least one metric-based (BARO), one trace-based (TraceRCA), and one multi-source (PDiagnose) baseline, using the evaluation script in Section 4.2. Check (a) whether each baseline imports the RE3 CSV logs/traces without modification, (b) whether the reported AC@1/AC@3/Avg@5 are above chance, and (c) whether a second engineer independently labels a random sample of 30 RE3 cases from raw stack traces/logs with root-cause service and indicator, measuring agreement with the released labels. If baselines cannot run or agreement is low, the code-level novelty is unsubstantiated; if they run and agree, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that RCAEval is the first comprehensive benchmark covering 11 fault types, including new code-level faults (RE3), with 15 reproducible baselines. But Section 5 evaluates only 11 baselines on the RE2 Train Ticket dataset, coarse-grained service ranking; no baseline result is reported on RE3 or on fine-grained root-cause indicators. RE3 labels are especially delicate: Section 3.2 says code-level root causes are 'determined using the stack traces in logs' or error logs/response codes, and Section 3.3 says quality was verified by one DevOps engineer. Unlike resource/network faults, where the injected service and the metric are defined by the fault type, the RE3 ground-truth indicator is an interpretation of logs/traces, and no inter-annotator or independent check is reported. Because the paper's novelty over prior benchmarks (Table 1) is precisely the code-level/multi-source combination, the benchmark's central contribution is currently supported only by dataset statistics and a GitHub demo, not by evidence that existing baselines can consume RE3 and reproduce the labels. If the adapters or labels fail on RE3, the 'comprehensive benchmark' claim overstates what the resource actually provides.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"RCAEval is a benchmark paper for root cause analysis (RCA) of microservice systems. The authors contribute three datasets (RE1, RE2, RE3) totaling 735 failure cases collected from Online Boutique, Sock Shop, and Train Ticket, spanning 11 fault types in three categories: resource faults, network faults, and code-level faults. The datasets include multi-source telemetry data (metrics, logs, and traces), and the paper also releases an evaluation framework with 15 baselines and coarse-grained and fine-grained evaluation metrics (AC@k and Avg@k). The paper reports preliminary experiments on the RE2 Train Ticket subset for coarse-grained service ranking, and it positions RCAEval as the first comprehensive open-source benchmark for RCA in microservice systems.","tokens_in":7941,"tokens_out":4309,"duration_ms":40948,"significance":"If the dataset labels are correct and the code-level fault subset is usable by existing methods, RCAEval is a genuinely useful community resource: it is larger in fault-type coverage than most prior RCA evaluations, it includes logs and traces in addition to metrics, and it ships an open-source evaluation framework with reproducible baselines. The aggregate statistics are internally consistent (375 + 270 + 90 = 735 cases; 4 + 2 + 5 = 11 fault types), and the paper explicitly builds on and extends the authors' prior ASE'24 framework, which provides a degree of continuity. However, the central novelty of the benchmark is not yet validated in the paper: the RE3 code-level dataset is never used in the experimental evaluation, the fine-grained root-cause-indicator evaluation is not demonstrated, and the ground-truth labels rest on verification by a single DevOps engineer with no reported inter-annotator agreement or independent audit. The significance of the contribution is therefore conditional on these gaps being addressed.","major_comments":[{"comment":"The experimental section evaluates only coarse-grained service ranking on the RE2 Train Ticket subset; neither the RE3 code-level dataset nor the fine-grained root-cause-indicator level is exercised anywhere in the paper. Since the paper's novelty over prior benchmarks (Table 1) is precisely the code-level fault coverage and the multi-source combination, the claim that RCAEval provides a 'comprehensive' benchmark and evaluation framework is currently supported only by dataset statistics and a GitHub demonstration, not by experimental evidence that the provided baselines can consume RE3 and reproduce its labels. Please either add experiments on RE3 (at least on one system) with the provided baselines, or present a fine-grained evaluation on RE2, and then state the benchmark's scope accordingly.","section":"Section 5, Table 6"},{"comment":"The quality of the entire benchmark depends on the correctness of the annotated root cause service and root cause indicator for each failure case, yet Section 3.3 states only that a single DevOps engineer with five years of experience verified the data. No inter-annotator agreement, automated consistency check, or independent audit is reported. For resource and network faults the indicator is largely determined by the injected fault type, but for code-level faults (Section 3.2) the indicator is derived from stack traces, error logs, or response codes, which is an interpretive judgment involving the faulty line of code. Please report a detailed labeling protocol, provide a sample of independently re-annotated cases with agreement statistics, and release any label-audit artifacts; without this, the baseline comparisons in Table 6 and any downstream benchmark conclusions rest on unvalidated labels.","section":"Section 3.2, Section 3.3"},{"comment":"The reported baseline results are point estimates with no variance, no confidence intervals, and no significance tests. RE2 includes three repetitions per fault-service pair, and these repetitions appear to be collapsed into the AC@k and Avg@k numbers, making it impossible to tell whether differences such as TraceRCA versus multi-source BARO on DELAY or LOSS reflect real superiority or noise. Please report per-fault means with standard deviations (or per-repetition results) and, where feasible, a simple significance test; this is necessary for the benchmark to support fair method comparison.","section":"Section 5, Table 6"},{"comment":"The description of the baseline set contains an inconsistency that should be corrected: the text says 'we select 11 baselines: 5 metric-based RCA methods (BARO, CausalRCA, CIRCA, MicroCause), 2 trace-based RCA methods, and 4 multi-source RCA methods', but the enumerated metric-based list has four names and Table 6 includes an additional metric RCD row and a separate multi-source RCD row. Please reconcile the count and the table rows, and clarify which configuration of each baseline is used (for example, whether metric RCD and multi-source RCD differ only in input data or also in algorithm).","section":"Section 5, Section 4.1"}],"minor_comments":[{"comment":"For PDiagnose, the paper says the source code is unavailable and the authors implemented it following previous works; please state which commit or version of their implementation is used and whether its outputs were validated against any original result.","section":"Section 4.1"},{"comment":"The table header appears as 'Ag@5' in the reproduction of Table 6; please ensure the heading matches the 'Avg@5' metric defined in Section 4.2.","section":"Section 5, Table 6"},{"comment":"The load range is stated as 10–200 requests per second, but no rationale or sensitivity analysis is given; a brief statement of how the load was varied across services and repetition would improve reproducibility.","section":"Section 3.3"},{"comment":"References [6] and [15] appear to describe the same paper (Pham et al., ASE'24) with different titles; please deduplicate or clarify if they are distinct works.","section":"References"},{"comment":"The data format description gives sample rows for metrics, logs, and traces, but it does not state how timestamps across the three data sources are aligned or how the failure-injection period is marked; please add a precise alignment and windowing description.","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a dataset-and-benchmark resource submission to a Web conference companion. Its main engineering contribution is potentially valuable, but the current 4-page format is too compressed to validate the novel code-level subset, which is the main reason I recommend major revision rather than acceptance. Given the authors' prior work on the underlying framework, I see no reason to doubt the good faith of the collection effort; the missing validation is a completeness issue, not a circularity issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the dataset may be the real contribution. RE3, with five code-level fault types on three systems plus logs and traces, is new to this literature, and the public framework with 15 baselines is a step toward consistent evaluation. But the paper overstates what it has demonstrated: the experiments never touch RE3, so the headline novelty is unexercised. That is not fatal in a 4-page companion, but it should be fixed in a full version or the claims trimmed.\n\nWhat I like: the data collection is substantial—735 cases, three systems, 11 fault types, multi-source telemetry. The format is well documented, and the code is on GitHub with a PyPI package. The authors adapted several existing implementations and say they reproduced their reported results, which is more than most benchmark papers bother to do. PDiagnose was re-implemented because the original source is unavailable, which adds some risk, but it is a reasonable choice.\n\nWhere it is soft:\n\n1. The code-level dataset, the main novelty over PyRCA, AIOps 2020, and their own ASE'24 work, is absent from Table 6. No baseline is run on RE3. The central claim of supporting code-level RCA is supported by data files and a GitHub demo, not by experimental evidence. If the adapters fail on RE3 or the labels are wrong, the 'comprehensive' claim evaporates.\n\n2. Label verification is thin. Section 3.3 says a single DevOps engineer verified data quality. No inter-annotator agreement, no automated cross-check. For resource and network faults, ground truth is mostly determined by the injection method, so risk is modest. For code-level faults, where the root cause indicator is an interpretation of stack traces and error logs, the risk is real. The paper should either provide a second annotator or acknowledge this limitation explicitly.\n\n3. The evaluation in Table 6 has no error bars or significance tests, and it covers only Train Ticket. That is fine for a preliminary result, but it does not support the word 'comprehensive' as used throughout.\n\nI disagree with nothing in the stress-test note; it lands.\n\nWho this is for: anyone building or comparing RCA methods for microservices. The resource could become a standard benchmark, which makes the paper worth citing even with its weaknesses. It deserves a proper referee—I would send it out, but I would ask for RE3 results or a narrowed scope, and at least a paragraph on label validation.\n\nFor your reading group: maybe, because the data is interesting but the paper is thin.","headline":"RCAEval's code-level dataset is genuinely new, but the paper never runs a baseline on it, so the 'comprehensive' claim is ahead of the evidence.","tokens_in":8498,"tokens_out":2677,"would_cite":true,"duration_ms":25271,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RCAEval provides the first comprehensive open-source benchmark for root cause analysis in microservice systems, with 735 failure cases, 11 fault types, and 15 reproducible baselines.","keywords":["root cause analysis","microservices","benchmark","telemetry data","fault injection","evaluation framework","multi-source","code-level faults"],"falsifier":"Take a random sample of failure cases, have two or more independent DevOps engineers annotate the root cause service and indicator without seeing the original labels, and measure inter-annotator agreement; if agreement is substantially below the level needed for the $\text{AC@}k$ metrics to be meaningful, the benchmark's ground truth is not robust. For code-level faults, one can directly check whether the faulty line identified by the stack trace matches the actual injected code modification, and a mismatch rate above a small threshold would invalidate the fine-grained labels.","tokens_in":7521,"feed_emoji":"🔬","tokens_out":3996,"duration_ms":32539,"temperature":0.7,"pith_summary":"This paper introduces RCAEval, a benchmark that standardizes how root cause analysis (RCA) of microservice systems is evaluated. It provides 735 failure cases across three microservice systems, covering 11 fault types including code-level faults, with metrics, logs, and traces for each case. The accompanying open-source evaluation framework offers 15 reproducible baselines and measures both coarse-grained (root cause service) and fine-grained (root cause indicator) performance using $\text{AC@}k$ and $\text{Avg@}k$. If the benchmark holds up, researchers can stop comparing RCA methods on incompatible datasets and instead evaluate new approaches against a common standard.","feed_headline":"RCAEval: 735 failure cases and 15 baselines for microservice RCA","feed_subtitle":"Three systems, 11 fault types, and a reproducible framework let researchers compare RCA methods fairly.","key_machinery":"The core mechanism is the benchmark itself: fault injection into three real microservice systems under load, multi-source telemetry collection (Prometheus metrics, logs, Jaeger traces), and annotation of ground-truth root cause service and indicator for each failure. The evaluation framework then ranks candidate root causes and scores them with $\text{AC@}k$ and $\text{Avg@}k$, making the benchmark a shared testbed for comparing RCA methods. The datasets are deliberately built from realistic open-source systems and fault types drawn from empirical studies of production failures.","core_discovery":"RCAEval claims to be the first comprehensive, open-source benchmark for root cause analysis of microservice systems. The paper assembles three datasets (RE1, RE2, RE3) collected from Online Boutique, Sock Shop, and Train Ticket, totaling 735 failure cases with 11 fault types: four resource faults (CPU, MEM, DISK, SOCKET), two network faults (DELAY, LOSS), and five code-level faults (incorrect parameter values, missing parameters, missing function calls, incorrect return values, missing exception handlers). Each case includes telemetry data (metrics, logs, traces) and an annotated root cause service and root cause indicator. The evaluation framework supplies 15 baseline methods spanning metric-based, trace-based, and multi-source approaches, with $\text{AC@}k$ and $\text{Avg@}k$ metrics for coarse- and fine-grained evaluation.","pith_inferences":["If the annotations prove reliable, RCAEval could become the default comparison point in the RCA field, much as standardized benchmarks have shaped other machine learning subfields; this trajectory is plausible but not claimed by the paper.","The code-level fault subset may reveal that log-based stack trace analysis is essential for fine-grained RCA, since metric-only methods cannot identify line-level bugs; the paper does not test this directly.","The three systems differ in scale (12 to 64 services), so the benchmark could be used to study how RCA methods scale with system complexity, an extension the authors did not investigate.","A natural testable extension is to use the framework to benchmark a newly proposed RCA method against the included baselines; the paper invites this use but does not demonstrate it."],"forward_implications":["Researchers can apply the 15 existing RCA baselines to the same 735 failure cases and compare their results directly, removing dataset mismatch as a confounder.","The inclusion of five code-level fault types enables evaluation of RCA methods that go beyond resource and network faults, which is new territory for the field.","The framework supports both coarse-grained and fine-grained RCA evaluation, so methods can be tested on localizing the root cause service and the specific root cause indicator.","Because the benchmark is open-source and installable via PyPI, new RCA methods can be plugged in and compared against a documented, reproducible set of baselines."],"supporting_citations":[{"why":"Supplies the RE1 dataset and the initial metric-based evaluation framework that RCAEval extends.","marker":"[6]"},{"why":"The AIOps 2020 dataset with metrics and traces but no logs, used as a comparison point showing RCAEval's addition of logs.","marker":"[12]"},{"why":"PyRCA, a metric-only RCA library with synthetic datasets, serves as a prior resource that RCAEval surpasses in scope.","marker":"[14]"},{"why":"Empirical analysis of OpenStack software failures that motivates the five code-level fault types in RE3.","marker":"[5]"},{"why":"Provides a causal-discovery-based RCA method adapted as a metric-based baseline.","marker":"[8]"},{"why":"Causal inference based RCA method adapted as a baseline for metric and multi-source evaluation.","marker":"[10]"},{"why":"Trace-based RCA baseline (MicroRank) used in the evaluation framework.","marker":"[11]"},{"why":"BARO, a metric-based RCA method, is adapted as a baseline and is also the basis of the extended framework.","marker":"[16]"}],"fun_headline_variants":["RCAEval benchmark: 735 cases, 15 baselines for microservice RCA","First open-source RCA benchmark for microservices with 735 failures","735 real failures, 15 baselines: RCAEval sets the RCA standard","Microservice RCA benchmark: 3 systems, 11 fault types, 735 cases","RCAEval: open benchmark with 735 cases for fair RCA comparison"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's results are only as trustworthy as the 735 root cause annotations, which were verified by a single DevOps engineer with five years of experience.","fun_headline_variants_meta":{"raw":{"variants":["RCAEval benchmark: 735 cases, 15 baselines for microservice RCA","First open-source RCA benchmark for microservices with 735 failures","735 real failures, 15 baselines: RCAEval sets the RCA standard","Microservice RCA benchmark: 3 systems, 11 fault types, 735 cases","RCAEval: open benchmark with 735 cases for fair RCA comparison"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000256,"raw_usage":{"total_tokens":1538,"prompt_tokens":870,"completion_tokens":668,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":564}},"tokens_in":486,"tokens_out":668,"duration_ms":4725,"temperature":1.0,"reasoning_tokens":564,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:51:38.467416+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of failure cases, have two or more independent DevOps engineers annotate the root cause service and indicator without seeing the original labels, and measure inter-annotator agreement; if agreement is substantially below the level needed for the $\text{AC@}k$ metrics to be meaningful, the benchmark's ground truth is not robust. For code-level faults, one can directly check whether the faulty line identified by the stack trace matches the actual injected code modification, and a mismatch rate above a small threshold would invalidate the fine-grained labels.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RE1 dataset and the initial metric-based evaluation framework that RCAEval extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Empirical analysis of OpenStack software failures that motivates the five code-level fault types in RE3."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides a causal-discovery-based RCA method adapted as a metric-based baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Causal inference based RCA method adapted as a baseline for metric and multi-source evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Trace-based RCA baseline (MicroRank) used in the evaluation framework."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BARO, a metric-based RCA method, is adapted as a baseline and is also the basis of the extended framework."}],"review_version":1}