{"id":"73b712bb-3411-4757-8238-186500e37ace","arxiv_id":"2501.11545","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RADICE combines causal discovery, partial domain knowledge, entropy-based edge orientation, and correlation-based refinement to output root cause causal sub-graphs, and it outperformed four baselines in simulated and real industrial cases.","lead":"This paper introduces RADICE, a pipeline that learns causal relations between system metrics from monitoring time series, augments them with partial expert knowledge, and outputs a causal sub-graph of likely root causes for a performance anomaly. It matters for engineers who need to explain why a system metric dropped without requiring a complete causal model of the system.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline subgraph output is never scored: Table I measures root-cause set recall/precision, not correctness of the returned causal sub-graph, so the central claim lacks direct evidence.","rationale":"The reader correctly identified that incorrect or noisy domain knowledge is a risk, and I agree that no perturbation experiment exists. However, in my reading the more load-bearing gap is more direct: the paper's headline output is the causal sub-graph, yet the evaluation never scores that output. The root-cause set precision/recall in Table I could be high while the sub-graph is structurally wrong, because the sub-graph includes intermediate nodes, edge orientations, and path choices that are not captured by the root-cause metric. The simulated data includes a ground-truth causal graph, so the missing edge-level evaluation is feasible and would directly settle whether the claimed sub-graph is accurate. This concern is internal to the paper's evidence rather than an external robustness assumption, which is why I treat it as the single most load-bearing issue. The reader's concern about domain knowledge is real but secondary; it would become decisive only after the sub-graph itself is shown to be accurate under the idealized knowledge condition. My proposed test is exactly the missing comparison: quantify whether GRC matches the true affected sub-graph. If it does, the conditional verdict can move toward accept; if not, the headline claim needs re-scoping. Since the reader already issued a conditional verdict, I do not change the verdict category, but I would strengthen the condition to require sub-graph accuracy evaluation.","tokens_in":16728,"tokens_out":3911,"duration_ms":48407,"concrete_test":"In the existing simulated benchmark, compute edge-level precision/recall (or structural Hamming distance) between the returned GRC and the ground-truth sub-graph consisting of the true root cause, all causal descendants, and the performance metric, for all 200 runs (4 graph sizes x 50) used in Table I, and report mean edge F1 alongside the root-cause F1. If edge F1 is close to root-cause F1, the sub-graph claim is supported; if it is substantially lower, the central claim must be weakened or re-scoped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section III promise a 'root cause causal sub-graph showing the causal relations between the system components affected by the anomaly,' and Section III-E describes a dedicated subtraction phase that selects paths from candidate root causes to the performance metric. However, the quantitative evaluation in Section IV-B scores only root-cause sets: recall is the fraction of runs returning the correct root cause, and precision is the ratio of correct to returned root causes (Table I). No metric measures edge correctness, path correctness, or structural distance between the returned sub-graph GRC and the ground-truth sub-graph. The real-data use case (Section IV-C) shows qualitative graphs but has no ground truth and relies on expert confirmation. This is load-bearing because the unique output of RADICE is the sub-graph, not merely a root-cause list. Even with perfect root-cause identification, the sub-graph can contain wrong edges, missing intermediate nodes, or incorrect paths: the entropy-based orientation (Section III-C1) is a heuristic, and the path-selection rules in Section III-E (prefer paths through more candidates, then shortest paths) are not validated against causal correctness. Thus the paper's central claim about the output sub-graph is currently unsupported by the experiments, independent of the separate concern about robustness to incorrect domain knowledge.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RADICE, a causal-graph-based root-cause analysis method for system performance diagnostics. Given multivariate time series and a partial causal domain knowledge model, RADICE (i) discovers an initial causal graph with PCMCI+; (ii) enhances it by inserting domain-directed edges, using a domain node-level function to discard conflicting edges and orient undirected relations, and applying an entropy-based orientation strategy; (iii) refines the candidate root-cause set with an adjusted correlation score and graph refinement knowledge; and (iv) subtracts the graph to output a root cause causal sub-graph connecting root causes to the performance metric. Experiments on simulated data with four ground-truth graph sizes compare RADICE variants to CoFlux, ε-diagnosis, PCMCI+, and TCDF, and a real advertising-system use case illustrates the outputs. The paper claims RADICE outperforms the baselines and produces root cause causal sub-graphs.","tokens_in":16920,"tokens_out":4258,"duration_ms":44486,"significance":"If the claims are borne out, the main contribution is a practical hybrid: a causal-discovery pipeline that accepts partial expert knowledge rather than requiring a complete structural causal model, and that returns a causal sub-graph rather than a bare list of root causes. The algorithm is specified in enough detail to be reimplemented, the ablation with RADICE P indicates the adjusted correlation score adds value, and the simulated data are made available. However, the evaluation as presented does not yet substantiate the headline sub-graph claim, and several experimental design choices (threshold tuning on the test data, oracle-sourced domain knowledge, no sub-graph scoring) leave the quantitative results weaker than the text suggests.","major_comments":[{"comment":"The quantitative evaluation measures only root-cause set recall and precision, yet the abstract and Section III-E define the output as a root cause causal sub-graph G_RC. No metric assesses edge correctness, path correctness, or structural distance between G_RC and the ground-truth sub-graph. This is load-bearing because the subtraction phase (Section III-E) could return the correct root-cause set while containing wrong intermediate edges; the paper's central claim is therefore unsupported by the current experiments. I ask the authors to add sub-graph-level evaluation on the simulated data, for example edge/path precision and recall or structural Hamming distance.","section":"Section IV-B and Table I"},{"comment":"The minSim threshold is selected on the same simulated datasets that are then used to report the final recall/precision in Table I ('we set minSim = 0.5 in the remaining of our evaluation'), and CoFlux's threshold is also chosen as 'the best results' on the same data. This parameter selection on the test set can inflate the reported gains; it should be redone with a held-out validation split, or results should be reported across the full minSim range with confidence intervals.","section":"Section IV-B, Fig. 3 and Table I"},{"comment":"The domain knowledge is assumed correct and is drawn from the ground-truth graph in the simulations, but Algorithm 1 (lines 2-8) unconditionally inserts every domain directed edge and discards learned edges that conflict with the level function. There is no experiment with noisy, incorrect, or adversarially supplied domain edges or level assignments, so the claimed robustness to 'partial domain knowledge' from experts is not tested. Please add a sensitivity analysis that flips or randomizes a fraction of domain edges and level constraints and reports recall/precision of both root-cause sets and sub-graphs.","section":"Algorithm 1 and Section IV-B"},{"comment":"The graph refinement knowledge is never exercised in the simulated experiments because the paper states that correlation signs are unknown under randomized edge weights; it is only demonstrated qualitatively in the real-data case (Section IV-C). Since this is one of the four core phases and a claimed differentiator, its contribution to recall/precision is unmeasured. I ask for a simulated scenario with fixed edge signs so that refinement rules can be defined and ablated.","section":"Section IV-B and Section III-A2"}],"minor_comments":[{"comment":"The entropy orientation procedure is described but no ablation isolates its effect from the level-function orientation; adding RADICE with the level function but without entropy orientation would clarify the contribution of each enhancement step.","section":"Section III-C1"},{"comment":"The real-data graphs have no ground truth and rely on expert confirmation; this is appropriate as a use case, but it should be described as illustrative rather than as evidence of sub-graph correctness.","section":"Section IV-C and Fig. 4"},{"comment":"The table reports averages over 50 runs without standard deviations or significance tests, so it is difficult to judge whether differences such as 0.74 vs. 0.70 are meaningful; please add variance information.","section":"Table I"},{"comment":"The comparison to reference [20] states that its improvements 'only affect the computational complexity', but no direct comparison to [20] is run; this claim should be softened or supported.","section":"Section II"},{"comment":"There are several typographical and spacing issues, for example 'iflevel(u) =level(v)' in Section III-C1, 'the next session' in Section IV-A, and the acronym expansion 'causal gRAph baseD root cause analysIs for system performanCE diagnostic' in Section III; these should be corrected.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The algorithmic contribution is clear and the paper is likely fixable, but the evaluation gaps are real: the headline sub-graph output is not scored, minSim is tuned on the test data, and domain knowledge is only tested in oracle form. I would like to see the additional experiments described in the major comments before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my take. The paper is a clean, well-specified integration of known pieces: PCMCI+ for discovery, an entropy-based orientation, a partial causal knowledge model (directed edges plus node levels), an adjusted correlation filter, and graph subtraction to output a root-cause sub-graph. The genuinely new bit is the partial knowledge model used to enhance and repair a discovered graph, not merely to speed up PC, and that is a useful idea for industrial RCA where full causal models are unrealistic. The algorithm is described precisely, and the simulated data is public. That is real credit.\n\nThe soft spots are real but not fatal. The stress-test note is right: the evaluation scores root-cause recall and precision, not the correctness of the returned sub-graph. No edge accuracy, no path accuracy, no structural distance. So the abstract's claim about outputting a root-cause causal sub-graph that shows the causal relations is unsupported by the quantitative experiments. The real use case is qualitative and expert-confirmed, which is fine as a demonstration but not as evidence. Second, the domain knowledge in simulation is generated from the ground-truth graph, giving RADICE an oracle advantage; there is no test of robustness to wrong edges or levels, and Algorithm 1 inserts domain edges unconditionally, so a single bad edge can corrupt the output. Third, minSim is chosen on the same simulated datasets used in the headline table, so the reported numbers are partly post hoc. No error bars either.\n\nWho should read this? Researchers working on performance diagnosis or hybrid causal-discovery systems. It is a solid engineering paper with a clear method and a plausible advantage, but the evaluation needs work before the claims are accepted as established. If I were an editor, I would send it to review and ask for three things: a structural metric on the simulated sub-graphs, a perturbation analysis of the domain knowledge, and a note on how minSim is selected (or a sensitivity analysis across data splits).\n\nMy verdict: worth a serious referee, not a desk reject.","headline":"Useful hybrid RCA pipeline with a clear spec and real simulation gains, but the headline causal sub-graph output is never directly evaluated and the domain knowledge is only tested in an oracle setup.","tokens_in":17487,"tokens_out":2081,"would_cite":true,"duration_ms":26330,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that augmenting time-series causal discovery with partial expert knowledge and correlation-based filtering yields a root-cause causal sub-graph that outperforms correlation-only and discovery-only baselines in simulated…","keywords":["root cause analysis","system performance diagnostic","causal graph","causal discovery","causal domain knowledge","partial domain knowledge","adjusted correlation score","time series anomaly"],"falsifier":"In the paper's simulation setup, randomly corrupt a fraction of the supplied domain directed edges or reverse the node level ordering and rerun the method; if recall and precision do not fall relative to the no-knowledge variant, the claim that correct partial knowledge drives the improvement is falsified.","tokens_in":16498,"feed_emoji":"🔍","tokens_out":9178,"duration_ms":91369,"temperature":0.7,"pith_summary":"This paper tries to solve a practical diagnostic problem: when a system performance metric drops, which underlying component caused it, and through which causal path did the anomaly spread? Its answer, RADICE, is a pipeline that starts from a causal graph learned from monitoring data, sharpens it with partial domain knowledge (a few expert-directed edges and a node-level ordering), filters candidate metrics by an adjusted correlation score, and outputs a root-cause causal sub-graph connecting the root causes to the performance metric. The paper argues this matters because full expert causal models are usually unavailable, while correlation alone produces false root causes. In simulations over ground-truth graphs of 5 to 25 nodes, RADICE with partial knowledge attained the highest recall and precision among the compared methods, and a real advertising-system case study produced sub-graphs that domain experts confirmed.","feed_headline":"Root causes found via causal graphs plus partial expert hints","feed_subtitle":"RADICE combines time-series causal discovery, expert edges, and correlation filtering into one diagnostic sub-graph.","key_machinery":"The load-bearing mechanism is the causal domain knowledge model plus the four-phase pipeline. The knowledge model has three parts: domain directed edges, which are known instantaneous cause-effect pairs; a domain node level function level(·), which encodes higher-order causal ordering so that a metric cannot be caused by a higher-level metric; and graph refinement rules, which state necessary conditions such as the required correlation sign with the performance metric. The enhancement phase, formalized in Algorithm 1, relies on a constraint-based time-series causal discovery algorithm, PCMCI+, to produce the initial graph; it then inserts every domain edge, removes learned edges that violate level(·), orients undirected edges first by level(·) and then by an entropy-based orientation strategy that discards inconclusive or cycle-forming orientations. The refinement phase replaces plain correlation with an adjusted correlation score computed under smoothing and time-shift penalties, and the subtraction phase trims the enhanced graph to the paths connecting candidate root causes to the performance metric.","core_discovery":"The paper's central claim is that a root cause analysis algorithm can exploit partial, not complete, causal domain knowledge to build a usable root-cause causal sub-graph. RADICE takes a time-series dataset and a causal domain knowledge model, learns causal relations with a time-series causal discovery algorithm, then enhances the learned graph by inserting expert-directed edges without question, deleting learned edges that contradict the expert level ordering, and orienting remaining undirected edges with an entropy-based rule plus the level ordering. A refinement phase computes an adjusted correlation score, defined as the maximum Pearson correlation over moving-average smoothing and negative time-shift transformations minus penalties for the amount of transformation, and applies graph refinement rules to discard metrics that cannot be root causes. A subtraction phase keeps, for each surviving candidate, the causal path to the performance metric that passes through the most root-cause candidates, with ties broken by shortest path, yielding the root cause causal sub-graph. On simulated data with ground-truth graphs, supplying the level function and a fraction of true directed edges lifted recall and precision above all baselines, and the real-data case study reports expert-confirmed root causes.","pith_inferences":["Beyond the paper, a deliberately corrupted or reversed level ordering should produce worse results than no knowledge at all, because the enhancement step deletes any learned edge that conflicts with the ordering; a random-corruption robustness test would settle how much the method depends on knowledge quality.","The binary insert-or-delete treatment of domain edges could be replaced by probabilistic or confidence-weighted edges, letting low-confidence expert statements be overruled by strong data evidence; the paper lists probabilistic domain knowledge as future work, and such a variant is testable in the same simulations.","The adjusted-correlation phase already ranks candidate root causes by score, so the same score could drive a prioritized inspection list for engineers, not merely a binary filter.","The real-data evidence is two anomalies in one advertising system, so the method's generality across system types remains open; a natural next test is to apply RADICE to annotated outage data from different architectures."],"forward_implications":["A diagnostic system no longer needs a complete expert-defined causal graph; a few reliable directed edges and a level ordering can push root-cause recall and precision above correlation-based baselines.","The method outputs a causal sub-graph rather than only a list of root causes, so an engineer can see the intermediate components along which the anomaly propagated to the performance metric.","In the simulations, the largest gain came from the node level function, indicating that coarse causal-ordering information accounts for most of the improvement.","Even with no domain knowledge, the entropy-based orientation and adjusted-correlation refinement improve over the underlying PCMCI+ discovery algorithm on most tested graph sizes.","Recall and precision both drop on the largest simulated system (25 nodes), so scaling to larger systems will require additional pruning or hierarchical modeling."],"supporting_citations":[{"why":"Supplies PCMCI+, the base time-series causal discovery algorithm used to learn lagged and contemporaneous causal edges in the discovery phase.","marker":"[21]"},{"why":"Supplies the entropy-based orientation strategy used to orient remaining undirected causal edges in the enhancement phase.","marker":"[8]"},{"why":"Motivates the adjusted correlation score that combines Pearson correlation with data transformations in the refinement phase.","marker":"[4]"},{"why":"Provides CoFlux, a correlation-based root cause analysis baseline that RADICE is compared against.","marker":"[25]"},{"why":"Provides ε-diagnosis, a correlation and latency-based root cause analysis baseline that RADICE is compared against.","marker":"[22]"},{"why":"Provides TCDF, a neural-network causal discovery baseline that RADICE is compared against.","marker":"[14]"},{"why":"Supplies the causal-inference software used for the PCMCI+ implementation and for generating simulated time-series data.","marker":"[27]"}],"fun_headline_variants":["RADICE pinpoints root causes with causal graphs and partial expert hints","Causal graphs plus partial expert clues beat correlation for root cause","RADICE turns partial expert knowledge into a causal root cause sub-graph","Root cause analysis boosted by causal discovery and partial expert input"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the partial domain knowledge supplied by experts is correct: a wrong expert edge or a wrong level ordering is inserted or used to delete learned edges regardless of what the data shows.","fun_headline_variants_meta":{"raw":{"variants":["RADICE pinpoints root causes with causal graphs and partial expert hints","Causal graphs plus partial expert clues beat correlation for root cause","RADICE turns partial expert knowledge into a causal root cause sub-graph","Root cause analysis boosted by causal discovery and partial expert input"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00077,"raw_usage":{"total_tokens":3417,"prompt_tokens":961,"completion_tokens":2456,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":2381}},"tokens_in":577,"tokens_out":2456,"duration_ms":17955,"temperature":1.0,"reasoning_tokens":2381,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:08:10.691037+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In the paper's simulation setup, randomly corrupt a fraction of the supplied domain directed edges or reverse the node level ordering and rerun the method; if recall and precision do not fall relative to the no-knowledge variant, the claim that correct partial knowledge drives the improvement is falsified.","supporting_citations":[{"cited_title":"In: Conference on Uncertainty in Artificial Intelligence","cited_arxiv_id":null,"evidence_quote":"Supplies PCMCI+, the base time-series causal discovery algorithm used to learn lagged and contemporaneous causal edges in the discovery phase."},{"cited_title":"In: Proceedings of the AAAI Conference on Artificial Intelligence","cited_arxiv_id":null,"evidence_quote":"Supplies the entropy-based orientation strategy used to orient remaining undirected causal edges in the enhancement phase."},{"cited_title":"https://www.dynatrace.com/support/ help/platform/davis-ai/on-demand-analysis/causal-correlation-analysis, last accessed 1 Nov 2024","cited_arxiv_id":null,"evidence_quote":"Motivates the adjusted correlation score that combines Pearson correlation with data transformations in the refinement phase."},{"cited_title":"In: Proceedings of the International Symposium on Quality of Service","cited_arxiv_id":null,"evidence_quote":"Provides CoFlux, a correlation-based root cause analysis baseline that RADICE is compared against."},{"cited_title":"In: The World Wide Web Conference","cited_arxiv_id":null,"evidence_quote":"Provides ε-diagnosis, a correlation and latency-based root cause analysis baseline that RADICE is compared against."},{"cited_title":"Machine Learning and Knowledge Extraction 1(1), 19 (2019)","cited_arxiv_id":null,"evidence_quote":"Provides TCDF, a neural-network causal discovery baseline that RADICE is compared against."},{"cited_title":"https://github.com/ jakobrunge/tigramite, last accessed 8 Nov 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the causal-inference software used for the PCMCI+ implementation and for generating simulated time-series data."}],"review_version":1}