{"id":"49042319-ab14-4e44-9267-dda1fe55b673","arxiv_id":"2502.15728","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"BSODiag is an unsupervised framework that fuses alerts, incidents, and changes, mines historical failure correlations, and uses a random walk on an event graph to locate outage root causes and propagation paths.","lead":"This paper presents BSODiag, an automated system that finds the root cause of large-scale cloud server outages by combining alerts, incident tickets, and change records into a single diagnosis graph. The system reports both the most likely root cause and the failure propagation path, and its tests on Alibaba Cloud data show it outperforms existing diagnostic tools.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground truth for root causes and propagation paths is never described; if labels were built from the same expert rule tree, CMDB connectivity, and historical co-occurrence that BSODiag encodes, the reported PR@3/PCR gains could be by construction.","rationale":"Reader's verdict is CONDITIONAL and I agree with the overall assessment, but I want to sharpen the condition. The weakest link in the argument is not the choice of association rule mining per se; many industrial RCA systems can be useful even when 'causal' edges are only predictive. The load-bearing point is that the evaluation labels are unobservable and may share the exact structure that the method encodes. The paper states the metrics depend on rU and pU but never defines the labeling process. Since the method's weights are a product of historical conditional probability and CMDB overlap, any labeler who consults the same historical incidents, the same expert hierarchy, or the same CMDB will make the method look good. This is a testable concern, not a claim of misconduct: the authors may well have careful independent ground truth from OSE postmortems. The concrete test above would settle it. If independent labels are confirmed, the central claim should be accepted as a plausible industrial contribution; if not, the reported numbers should be treated as unverified. I also note the absence of error bars and the lack of released code/data, but those are secondary; the ground-truth provenance is the condition that determines whether the headline comparison has meaning. Therefore I keep the reader's CONDITIONAL verdict unchanged, while making the condition explicit.","tokens_in":18104,"tokens_out":3803,"duration_ms":37198,"concrete_test":"Obtain or request the label-construction protocol for rU and pU. The decisive check is whether the annotators of the 68/95 outage cases had access to the failure rule tree, the CMDB connectivity scores, or the historical co-occurrence statistics used in Section IV-B and Section IV-C1. If they did, the authors should re-evaluate on a held-out set labeled independently (e.g., postmortem reports written before BSODiag was run, with explicit prohibition on consulting the rule tree/CMDB), and report PR@3 and PCR on that set. A second, cheaper check: compute a baseline that predicts the path by greedily following the expert rule tree alone; if its PCR is close to BSODiag's 46.3%, the added value of the learned weights is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"BSODiag's headline numbers are computed against a 'true root cause' rU and a ground-truth propagation path pU (Section V-A3), but the paper never states how these labels were obtained. The only mention is that a 'large-scale testing platform' was built 'for outage data collection and verification' (Section V-A1); no annotation procedure, annotator qualifications, or exclusion rules are given. This matters because the edges and weights used to rank root causes and infer paths are built from the same ingredients a labeler would plausibly use: an expert-provided failure rule tree (Section IV-B), historical support/confidence co-occurrence (Section IV-B), and CMDB connectivity dist(ei, ej) (Section IV-C1). If rU and pU were produced by experts applying that same hierarchy and connectivity view, then PR@3 and PCR partly measure BSODiag's fidelity to its own priors rather than independent diagnostic accuracy. The concern is especially acute for PCR, since the predicted path is selected by maximizing cumulative propagation probability over exactly the edges whose weights are exp(conf)*dist; a ground truth path derived from the same rule tree and CMDB would make PCR high by construction. The paper also reports no error bars or per-case variance, so it is not possible to tell whether the 10.2%/3.7% margins over COT/FHM are robust or driven by a few cases. This is load-bearing because the central claim is an empirical superiority claim; without independent labels the comparison cannot be interpreted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BSODiag, an unsupervised framework for diagnosing batch server outages in large-scale cloud infrastructure. It integrates multi-source monitoring data (alerts, incidents, and changes), detects outage-related failures, mines historical failure correlations using Apriori with an expert-provided failure rule tree, and combines this historical knowledge with current CMDB connectivity to build an event cause graph. A multi-attribute random walk (MAPR) localizes root causes, and a propagation-probability-based path inference (PPI) reconstructs failure propagation paths. The method is evaluated on Alibaba Cloud data with a 2022 training/2023 testing split, reporting PR@3=87.5% and PCR=46.3%, exceeding several baselines. The core claim is that global spatio-temporal modeling of failures improves both root cause localization and path inference.","tokens_in":18447,"tokens_out":6467,"duration_ms":59247,"significance":"If the reported results are valid, BSODiag is a practically valuable contribution to cloud infrastructure diagnosis: it operates on coarse-grained operational data, provides interpretable root cause and propagation path outputs, and is reported to be efficient. The temporal split between the failure knowledge graph construction (2022) and evaluation (2023) is a genuine strength, reducing the risk of self-fit. The paper uses real industrial data and compares against a reasonable set of baselines. However, the evaluation has critical gaps that currently prevent the central empirical claim from being accepted: the ground-truth labeling process is not described, no statistical uncertainty is reported, and the path inference formula is not a proper propagation probability. These issues are load-bearing because the headline claims are empirical superiority numbers.","major_comments":[{"comment":"The procedure for obtaining the ground-truth root cause rU and propagation path pU is never described. Section V-A1 only states that a 'large-scale testing platform' was built 'for outage data collection and verification,' but no annotation protocol, annotator qualifications, or validation steps are given. This is a serious omission because the method's edges and weights are built from the expert-provided failure rule tree (Section IV-B), historical co-occurrence (Section IV-B), and CMDB connectivity dist(ei, ej) (Section IV-C1). If the ground truth was generated by experts using the same hierarchy and connectivity information, then PR@3 and PCR would partially measure the method's fidelity to its own priors rather than independent diagnostic accuracy. The paper must specify exactly how rU and pU were obtained for each of the 68 outage cases in Dall, what information was available to the labelers, and how labeler disagreements were resolved.","section":"Section V-A1, V-A3; Section IV-B, IV-C1"},{"comment":"All reported metrics are point estimates without error bars, confidence intervals, or significance tests. With 68 outage cases in Dall, the claim that BSODiag 'significantly outperforms' baselines is not supported statistically. For example, on Dall the PR@3 margin over COT is 10.2 percentage points, which could be driven by a small number of cases. The authors should report per-case results, bootstrap confidence intervals, or a paired significance test (e.g., Wilcoxon signed-rank) for PR@1, PR@2, PR@3, MAP, and PCR. This is necessary to substantiate the core empirical claim.","section":"Tables III and IV; Section V-A3"},{"comment":"The propagation path inference formula is not a valid propagation probability. The paper defines TransPr(pi) as the product of node personalization scores u_j along path pi, but these u_j are the converged random walk node scores, not transition probabilities. A path's propagation probability should be the product of edge weights (e.g., normalized wij) along the path. As written, the selected path maximizes a quantity with no stated probabilistic interpretation. The authors must redefine TransPr in terms of edge weights, or provide a justification for why the product of node personalization scores is a meaningful propagation probability. This is load-bearing because PCR is one of the two headline metrics.","section":"Section IV-C3, PPI equation"},{"comment":"The method depends on several free parameters that are set 'based on diagnostic experience' without any sensitivity analysis: support threshold α, time slot length δ, event merge window η, snapshot initial window [−L,−T], diagnosis window [−T,T'], and MAPR iteration count L. Since these parameters are hand-chosen rather than learned, the robustness of the reported gains is unknown. The authors should provide a sensitivity analysis for at least α, δ, and the window sizes, and report how performance varies across reasonable values. Without this, it is unclear whether the claimed improvements are parameter-sensitive.","section":"Section V-A1 and Section IV"},{"comment":"The ablation study is underspecified. For each ablation variant ('w/o FKG', 'w/o CMDB', 'w/o MAPR'), the exact modification to the pipeline is not described. For example, does removing FKG mean setting pij.conf to 1, or removing the historical edges entirely? Does removing CMDB mean setting dist(ei, ej) to 1? Does removing MAPR mean replacing the random walk with a simple ranking heuristic? Without these details, the reader cannot interpret the contribution of each component or rule out that the ablations are implemented in a way that unfairly degrades performance. The text should precisely define each ablation.","section":"Section V-D, Figure 7"}],"minor_comments":[{"comment":"The support threshold is applied as pij.count ≥ |Q2|·α, where |Q2| is the number of candidate pairs. Standard support is a frequency over the number of event groups, not over the number of unique items or pairs. Define the denominator explicitly; as written, the threshold is size-dependent in a way that does not match the support and confidence definitions in Section IV-B.","section":"Algorithm 1, lines 7 and 16"},{"comment":"The node personalization score is defined as ui = exp(−t)·dist(ei, eo), but the variable t is not defined. Specify whether t is the time distance to outage, and in which units.","section":"Section IV-C2, Eq. for ui"},{"comment":"The y-axis label 'PRC' should be 'PCR' for consistency with the text.","section":"Figure 6"},{"comment":"There is a typo in the opening paragraph: 'batch serves outage' should be 'batch servers outage'.","section":"Section IV-A"},{"comment":"The expert-provided failure rule tree and the change whitelist are key external inputs but are never described or exemplified. Please provide at least a small example of the rule tree structure and the whitelist criteria, or a reference to where they are defined.","section":"Section IV-B"},{"comment":"The efficiency analysis reports an average diagnosis time of 24.5 seconds, but no hardware specifications, software versions, or variance across cases are given. Adding these details will help readers assess the claim of lightweight operation.","section":"Section V-E"},{"comment":"The sum of outage cases in Didc (19) and Dnet (47) is 66, while Dall has 68. Please clarify whether some outage cases have multi-domain root causes that place them in both subsets, and explain the overlap.","section":"Table II"},{"comment":"The claim of being 'the first to study the batch servers outage diagnosis problem' should be tempered, given prior outage analysis work in large-scale production clouds (e.g., COT [4] and AirAlert [1]). Consider phrasing the novelty more narrowly, e.g., as the first to jointly model root cause and propagation path for batch server outages from multi-source coarse-grained data.","section":"Section I"}],"recommendation":"major_revision","confidential_remarks":"The paper comes from a strong industrial-academic collaboration and uses a real dataset with a clean training/evaluation temporal split. The main concern is that the evaluation may not measure independent diagnostic accuracy because the ground-truth labeling process is opaque. If the authors can provide a detailed annotation protocol and demonstrate that labels were generated without access to the same rule tree and CMDB information that the model uses, the paper could become a significant contribution. I would encourage the editor to request the missing ground-truth description and statistical reporting as mandatory revision items. The technical issue with the PPI formula (using node scores as path probabilities) also needs to be addressed directly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, read BSODiag. It's a credible industrial-system paper: first to treat batch servers outage as its own diagnosis problem, and it puts together alert/incident/change detection, Apriori-based correlation mining with an expert rule tree, and a random-walk root-cause ranker with path inference. That combination is new, and the year-split evaluation (2022 knowledge graph, 2023 test) means the headline numbers are not a self-fit. The gains over COT (PR@3 87.5% vs 77.3%) and FHM (PCR 46.3% vs 42.6%) are plausible for this class of method, and the ablation and timing numbers are consistent with the story. I'd give credit for the empirical study in Section II-C and for framing the troubleshooting need (propagation path, not just a single root cause).\n\nThe soft spots are real, and one is load-bearing. The paper never describes how the ground-truth root cause rU and propagation path pU were obtained. Section V-A1 only says a 'large-scale testing platform' was built for 'outage data collection and verification.' That matters because the predicted path is chosen by maximizing cumulative probability over edges whose weights are exp(conf)*dist, and both terms come from the same ingredients a labeler would plausibly use: the expert failure rule tree, historical co-occurrence, and CMDB connectivity. If the labels were built from that same view, then PCR is partly measuring fidelity to the method's own priors rather than independent diagnostic accuracy. PR@3 is less exposed, but it still depends on the same ground-truth construction. The stress-test note is right to flag this; the paper needs to explain the labeling procedure before the empirical claim can be fully interpreted.\n\nSecond-tier issues: no error bars or significance tests anywhere, so the 10.2% and 3.7% margins could be driven by a few cases. Several hyperparameters (alpha, delta, eta, windows, MAPR iterations) are set from 'diagnostic experience' with no sensitivity analysis. Dataset and code aren't available. These are common in industrial papers and don't sink it, but they cap the confidence level.\n\nBottom line: who is this for? Cloud reliability researchers and practitioners who want a deployable, unsupervised alternative to microservice-oriented RCA on coarse-grained data. It deserves a serious referee: the problem framing and system design are solid, and the empirical gap is fixable by describing ground-truth construction and adding variance estimates. I wouldn't desk-reject this.","headline":"Credible industrial-system paper with a real year-split evaluation, but the undisclosed ground-truth labeling makes the headline path-coverage numbers hard to interpret without more transparency.","tokens_in":18964,"tokens_out":2519,"would_cite":true,"duration_ms":24076,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BSODiag claims that batch-server outages can be diagnosed by merging coarse-grained alerts, incidents, and changes into a single event cause graph, scoring 87.5% top-3 root-cause recall and 46.3% propagation-path coverage on a two-year…","keywords":["batch servers outage","cloud infrastructure","root cause analysis","failure propagation path","multi-source monitoring data","failure correlation","event cause graph","unsupervised diagnosis"],"falsifier":"Compare BSODiag's predicted cause-effect edges against engineer-confirmed repair histories on the 95 collected outage cases: if pairs with high confidence and high device overlap are frequently common effects of a third failure or co-located coincidences, the edge weights are not measuring propagation. A sharper test is to replace the expert hierarchy filter with pure temporal ordering, so that earlier events cause later ones, and see whether PR@3 and PCR hold; equal performance would show the causal signal does not depend on the claimed expert mechanism.","tokens_in":17931,"feed_emoji":"⚡","tokens_out":10935,"duration_ms":93674,"temperature":0.7,"pith_summary":"Batch-server outages—whole clusters of servers failing at once—are the most severe failure class in cloud infrastructure because they make every upstream service unavailable, and today's diagnosis relies on engineers manually reconciling alerts, incident tickets, and change records. The paper tries to establish that this diagnosis can be automated end-to-end without labels: detect failures genuinely related to the outage from all three coarse-grained streams, merge them into events, build a single event cause graph that combines historical failure co-occurrence with current device connectivity, and rank both the root cause and the failure propagation path. On a two-year dataset from a large production cloud, BSODiag reports 87.5% root-cause-in-top-three and 46.3% propagation-path coverage, beating the strongest compared baseline by 10.2 and 3.7 percentage points respectively. If that holds, infrastructure teams gain a lightweight, interpretable alternative to a slow, multi-specialty manual investigation.","feed_headline":"Batch-server outage roots found 87.5% of the time at top-3","feed_subtitle":"Unsupervised framework merges alerts, incidents, and changes to rank causes and trace the failure path.","key_machinery":"The central object is the event cause graph $G_e$: a directed graph whose nodes are the outage-related events detected from alerts, incidents, and changes, and whose edges are candidate trigger links. The paper assigns each edge a causal strength $w_{ij} = \\exp(p_{ij}.conf) \\cdot \\mathrm{dist}(e_i, e_j)$, where $p_{ij}.conf$ is the confidence of the failure pair mined from historical data by an Apriori-based algorithm and filtered by an expert hierarchy tree, and $\\mathrm{dist}(e_i, e_j)$ is the fraction of $e_i$'s devices that are physically connected to or shared with $e_j$ in the configuration management database (CMDB). This single formula carries the global-perspective argument: it fuses historical failure knowledge with current device dependencies. A customized multi-attribute random walk (MAPR) over $G_e$, seeded by each event's time before the outage and its connectivity to the outage node, produces the root-cause ranking; then a propagation-probability inference (PPI) selects the path from the top-ranked root cause to the outage node with the largest product of node scores.","core_discovery":"BSODiag's central claim is that batch-server outages are diagnosable from the coarse-grained data a large cloud already records, provided the analysis is global rather than per-stream. Alerts alone flood and carry false positives; incidents alone omit minor failures; changes alone are optional high-risk actions—so the paper argues that genuine outage-related failures emerge only when all three are detected separately and then merged into events. The root cause, in turn, is not a single point but the head of a propagation process: the paper constructs an event cause graph whose directed edges encode how likely one event is to trigger another, combining the historical confidence mined from co-occurring failure pairs with the physical device-overlap ratio measured in the current outage, and then ranks nodes with a multi-attribute random walk. The same graph supports a propagation-probability search that returns the most likely path from the root cause to the outage node, which is what engineers actually inspect. The reported result is that this unsupervised pipeline outperforms rule-based, supervised, and prior failure-graph baselines across both root-cause localization and path inference.","pith_inferences":["Editorial extension: the edge-weight formula is not tied to any particular failure type, so the same event cause graph could be applied to other coarse-grained failure classes such as network partitions or power-chain faults by supplying the appropriate expert hierarchy tree and CMDB connectivity relation.","Editorial extension: because causal direction is only constrained by the expert tree rather than by temporal precedence within the outage window, a natural stress test is to reverse the hierarchy filter or add lag-correlation checks; the sensitivity of PR@3 would reveal how much of the causal claim is statistical versus expert-supplied.","Editorial extension: the headline accuracy figures come from one production environment with its own change whitelist, hierarchy tree, and support threshold, so porting BSODiag to another cloud would require re-tuning those components and the reported margins may not transfer unchanged.","Editorial extension: a stronger operational validation would track whether following the inferred propagation path actually prevents follow-on failures during repair, an outcome the paper mentions as motivation but does not measure."],"forward_implications":["Because no single monitoring stream is sufficient, dropping alert, incident, or change fusion should degrade root-cause recall; the framework's gains over single-stream baselines are a direct test of this claim.","Historical failure co-occurrence filtered by an expert hierarchy tree plus current device overlap is enough to estimate causal strength without labeled outage data, so the method can be deployed in environments where labeled diagnoses are scarce.","Root-cause ranking and propagation-path inference come from one graph, so engineers can move directly from what broke to which sequence of devices to inspect and repair.","The framework's lightweight design enables a single diagnosis in 24.5 seconds in the reported deployment, compared with roughly 30 minutes for manual analysis, which changes how quickly an outage response can start.","As historical failure data accumulate, the failure knowledge graph improves and diagnostic performance rises to a plateau, implying the system gets better with deployment time."],"supporting_citations":[{"why":"Supplies the strongest compared failure-graph baseline, built from incident correlation mining; outperforming it is the paper's main empirical claim.","marker":"[4]"},{"why":"Supplies one of the compared failure-graph baselines, learning causal relations between alert sequences and outages.","marker":"[1]"},{"why":"Supplies the Apriori association-rule algorithm used to mine high-frequency failure pairs from historical event data.","marker":"[23]"},{"why":"Cited as the SPOT outlier detector used to turn per-device alert time series into genuine outage-related failures.","marker":"[22]"},{"why":"Supports treating changes as high-risk failure triggers and motivates the whitelist-based change filter.","marker":"[15]"},{"why":"Establishes the alert false-positive and flooding problems that motivate the Alert2Event component.","marker":"[13]"},{"why":"Establishes the delayed-reporting and omission problems of incident data that motivate multi-source fusion.","marker":"[14]"},{"why":"Supplies the hierarchy-first rule baseline and the failure-rule-tree idea used to filter candidate causal pairs.","marker":"[24]"}],"fun_headline_variants":["Top-3 root cause found for 87.5% of batch-server outages","No labeled data needed to find batch-server outage root cause","Global failure graph pinpoints batch-server outage root cause","87.5% top-3 precision for batch-server outage root cause","Unsupervised AI ranks batch-server outage root causes in top 3"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a failure pair observed together in the same data center on the same day, kept only when it fits an expert-ranked hierarchy of failure types, and scaled by how much the devices overlap physically, is a true cause-effect relation rather than a coincidence or a shared external cause.","fun_headline_variants_meta":{"raw":{"variants":["Top-3 root cause found for 87.5% of batch-server outages","No labeled data needed to find batch-server outage root cause","Global failure graph pinpoints batch-server outage root cause","87.5% top-3 precision for batch-server outage root cause","Unsupervised AI ranks batch-server outage root causes in top 3"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001766,"raw_usage":{"total_tokens":7008,"prompt_tokens":1025,"completion_tokens":5983,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":5892}},"tokens_in":641,"tokens_out":5983,"duration_ms":37859,"temperature":1.0,"reasoning_tokens":5892,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T21:50:38.090967+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare BSODiag's predicted cause-effect edges against engineer-confirmed repair histories on the 95 collected outage cases: if pairs with high confidence and high device overlap are frequently common effects of a third failure or co-located coincidences, the edge weights are not measuring propagation. A sharper test is to replace the expert hierarchy filter with pure temporal ordering, so that earlier events cause later ones, and see whether PR@3 and PCR hold; equal performance would show the causal signal does not depend on the claimed expert mechanism.","supporting_citations":[{"cited_title":"Fast outage analysis of large-scale production clouds with service correlation mining,","cited_arxiv_id":null,"evidence_quote":"Supplies the strongest compared failure-graph baseline, built from incident correlation mining; outperforming it is the paper's main empirical claim."},{"cited_title":"Outage prediction and diagnosis for cloud service systems,","cited_arxiv_id":null,"evidence_quote":"Supplies one of the compared failure-graph baselines, learning causal relations between alert sequences and outages."},{"cited_title":"Detecting and localizing end-to-end performance degradation for cellular data services based on tcp loss ratio and round trip time,","cited_arxiv_id":null,"evidence_quote":"Cited as the SPOT outlier detector used to turn per-device alert time series into genuine outage-related failures."},{"cited_title":"Identifying root-cause changes for user-reported incidents in online service systems,","cited_arxiv_id":null,"evidence_quote":"Supports treating changes as high-risk failure triggers and motivates the whitelist-based change filter."},{"cited_title":"Understanding and handling alert storm for online service systems,","cited_arxiv_id":null,"evidence_quote":"Establishes the alert false-positive and flooding problems that motivate the Alert2Event component."},{"cited_title":"Towards intelligent incident management: why we need it and how we make it,","cited_arxiv_id":null,"evidence_quote":"Establishes the delayed-reporting and omission problems of incident data that motivate multi-source fusion."},{"cited_title":"Microhecl: High-efficient root cause localization in large-scale microservice systems,","cited_arxiv_id":null,"evidence_quote":"Supplies the hierarchy-first rule baseline and the failure-rule-tree idea used to filter candidate causal pairs."}],"review_version":1}