{"id":"4c1e1c26-4be8-4d38-85a0-d361bfffa9dd","arxiv_id":"2506.04569","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"KPIRoot+ improves on KPIRoot by adding decomposition-based anomaly detection and a trend-aware symbolic representation, achieving higher root-cause localization accuracy on three industrial datasets.","lead":"KPIRoot+ is a system for finding which virtual machine caused a cloud service slowdown, using a smarter way to detect anomalies and to compare KPI patterns. It claims to beat eight existing methods in accuracy and run in about eight seconds, and the authors say it has been running in production at a large cloud vendor since 2023.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The integrated score in Eq. 11 mixes an unbounded Granger F-statistic with a [0,1] Jaccard coefficient without normalization, so the claimed combine-similarity-and-causality ranking is not well-defined and may be dominated by the unnormalized causality term.","rationale":"I read the paper's central claim as: KPIRoot+ achieves higher root-cause localization accuracy than eight baselines and does so efficiently by combining similarity and causality after improved anomaly detection. The reader's weakest assumption was the additive STL/U-Net anomaly segment; I agree that is a genuine risk, but the sharpest load-bearing technical issue is the score fusion in Eq. 11. The paper itself constrains c_i to [0,1] yet sums a bounded Jaccard with an unbounded F-statistic; no normalization is specified, and lambda is tuned on the same test sets in RQ5. This creates an ill-defined objective whose ranking may depend on term scale rather than on the intended semantic combination. I also note the abstract's claim of reducing time cost by 34.7% is contradicted by RQ4, where KPIRoot+ takes about 8 seconds while KPIRoot takes about 5 seconds, but that is an efficiency-reporting issue, not the primary threat to the accuracy claim. A single re-computation with normalized F would settle whether Eq. 11 is actually load-bearing: if the headline metrics survive normalization, the concern is not fatal; if they shift, the reported improvements are not attributable to the proposed integration. Because this concern reinforces the existing CONDITIONAL verdict rather than overturning the paper, I keep the verdict unchanged. I mark partial agreement with the reader because the reader's rationale mentioned the score-combination issue but did not make it the weakest assumption.","tokens_in":21352,"tokens_out":7914,"duration_ms":101525,"concrete_test":"Recompute Table 3 on all three datasets with the Granger score normalized before Eq. 11, for example F' = (F - min(F)) / (max(F) - min(F)) or rank(F)/m, keeping lambda=0.9; also report F1, Hit@5, Hit@10, and NDCG@10 for Jaccard-only and F-only rankings. If the normalized-fusion results differ from the published table by more than 0.01 in any metric, or if the published ranking is nearly identical to the F-only ranking, then the unnormalized Eq. 11 is a load-bearing artifact of the reported accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim rests on the correlation score in Eq. 11: c_i = lambda * Jaccard(S_host, S_i) + (1-lambda) * F_vm_to_host, with lambda=0.9. Jaccard is bounded in [0,1], but F is an unbounded F-statistic from Eq. 10, so c_i is not in [0,1] as stated in Section 3.1, and the two terms are on incomparable scales. In the reported regime, where a root-cause VM's past values help predict the alarm KPI, F can be several times larger than 1, making 0.1*F comparable to or larger than the maximum 0.9*Jaccard contribution. The ranking of VMs can therefore be driven by the Granger term alone, or by whichever term happens to have the larger scale, rather than by the intended integration of similarity and causality. The ablation in Table 4 removes the ISAX or decomposition components but never removes or renormalizes the causality term, so it cannot detect this issue. RQ5 then tunes lambda on the same test sets, so the reported F1 and Hit@10 values may reflect a lambda chosen to mask the scale mismatch. If the final ranking is essentially the same as an F-only ranking, the paper's claim that similarity and causality are jointly exploited is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KPIRoot+, an extension of the earlier KPIRoot framework for root cause localization in large-scale cloud systems. KPIRoot+ first applies STL-based time series decomposition and a U-Net autoencoder to detect anomaly segments in the alarm KPI, then represents VM KPIs with an Improved SAX (ISAX) encoding, and finally combines Jaccard similarity and Granger causality into a per-VM correlation score. The authors evaluate the method on three industrial datasets from a large cloud provider, report anomaly detection F1 scores around 0.89–0.93, root cause localization F1 around 0.871–0.891 and Hit@10 around 0.934–0.967 across datasets, and describe a production deployment experience. The main claimed advantages are higher localization accuracy than eight baselines and an average execution time of about 8 seconds.","tokens_in":21609,"tokens_out":5481,"duration_ms":56495,"significance":"If the central claims hold, this is a practically valuable contribution: the problem is important, the evaluation uses real industrial KPI data, the baseline set is broad, and a production deployment is reported. The paper also ships a public code repository, which is a strength for reproducibility. The decomposition-plus-U-Net extension of KPIRoot is a reasonable direction, and the anomaly detection evaluation is a useful addition. However, several load-bearing issues currently prevent the quantitative claims from being accepted as stated: the abstract's efficiency claim is contradicted by the reported runtimes, the combined score in Eq. (11) mixes bounded and unbounded quantities, and hyperparameters are tuned on the same test datasets used for the headline results.","major_comments":[{"comment":"The correlation score is defined as c_i = λ * Jaccard(S_host, S_i) + (1−λ) * F_vm_to_host, but Section 3.1 states that c_i is in [0,1]. Jaccard is bounded in [0,1], whereas the Granger F-statistic in Eq. (10) is unbounded above. With λ=0.9, the term 0.1*F can be comparable to or larger than the maximum attainable similarity contribution, so the ranking is not a scale-invariant weighted combination of the two analyses. The ablation in Table 4 and the sensitivity study in RQ5 do not include a variant that removes or renormalizes the causality term, so the claim that similarity and causality are jointly exploited is not supported by the experiments. Please normalize the causality score (for example, to a p-value, a rank, or an empirical quantile) before combining it with the Jaccard term, and add an ablation that isolates the causality-only and normalized-combination variants.","section":"Section 3.1 and Eq. (11)"},{"comment":"The abstract claims 'reducing time cost by 34.7%', but the evaluation in RQ4 reports that KPIRoot+ takes 'around 8 seconds' while KPIRoot takes 'around 5 seconds' (Fig. 7 and the accompanying text), which is an increase in execution time, not a reduction. No comparison baseline or dataset subset that would yield a 34.7% reduction is specified. The efficiency contribution should be restated with an explicit comparison (which baseline, which datasets, what percentage) or removed.","section":"Abstract and Section 4.2.4"},{"comment":"The hyperparameters λ, w, and the U-Net configuration are selected on the same three datasets used to produce the headline results; Section 6.2 states 'we find the most suitable configurations based on the best results obtained in different parameters', and RQ5 varies λ and w on exactly the datasets reported in Tables 3–4. This means the reported F1, Hit@10, and NDCG values can be optimistically biased by test-set selection. The paper needs a validation protocol, such as a held-out dataset or nested cross-validation, before the quantitative claims can be taken at face value.","section":"Section 6.2 and RQ5"},{"comment":"The U-Net anomaly detection component is under-specified. The paper does not state the training window length, training set construction, number of epochs, anomaly threshold, or how the per-component outputs AE(τ_host), AE(s_host), and AE(r_host) are combined into the anomaly segment i_anomaly. Since the downstream similarity and causality computations depend on the segment boundaries t_s and t_e, the current description is insufficient for reproducibility and for assessing the risk that a mis-detected segment leads to incorrect root cause candidates.","section":"Section 3.3 and Algorithm 1, line 7"},{"comment":"The number of baselines is inconsistent: the abstract says 'eight state-of-the-art baselines', while the RQ2 text says 'seven baselines', and Table 3 lists eight methods including the earlier KPIRoot. The claimed improvement range of 2.9%–35.7% is ambiguous unless the set of external baselines and whether KPIRoot is counted are stated precisely.","section":"Section 1 and Section 4.2.2"}],"minor_comments":[{"comment":"The text says 'if a VM KPI undergoes an uprush and causes the alarm KPI to trigger alerts ... then Granger causality exists from the alarm KPI to the VM KPI'; the direction should be from the VM KPI to the alarm KPI, consistent with the F-statistic notation F_vm_to_host.","section":"Section 3.5"},{"comment":"The deployment date is inconsistent: the introduction says the system was deployed 'since Nov 2022', while the industrial experience section says 'since Aug 2023'. Please reconcile these dates.","section":"Section 1 and Section 5"},{"comment":"The ISAX symbol assignment in Eq. (6), s_i = alphabet_{2α − φ_i·l}, uses an index l whose meaning is not defined in this equation; it should be clarified whether l is the SAX breakpoint index or the lag parameter from Eq. (2).","section":"Section 3.4 and Eq. (6)"},{"comment":"The text says 'the scale of similarity is typically smaller than that of causality', but λ=0.9 gives the similarity term nine times the weight of the causality term; this sentence appears to contradict the chosen weighting and should be clarified to avoid confusion with the scale mismatch raised in Eq. (11).","section":"Section 4.2.5"},{"comment":"Eq. (3) uses the notation τ_t_host, s_t_host, and r_t_host, but the text later refers to 'seasonal' rather than 'periodic'; using consistent terminology for the seasonal component would improve readability.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is an extended version of an ISSRE publication, and the incremental novelty is acceptable for a journal if the technical issues are resolved. The main risk is that the reported improvements may partly reflect test-set hyperparameter selection, and the abstract's efficiency claim appears to contradict the body. These issues should be corrected before the quantitative claims are accepted. The deployment experience and the real-world datasets are valuable assets, but they do not by themselves resolve the methodological concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a straightforward, honest extension of the authors' earlier KPIRoot: STL decomposition plus a U-Net for anomaly detection, and an ISAX representation that adds slope signs to SAX. Neither is conceptually new, but the evaluation is solid in scope—three industrial datasets, eight baselines, and a new NDCG metric that speaks to practical usability. The deployment narrative from CloudH is plausible and genuinely useful for the AIOps community.\n\nThe soft spots are real and material. First, the abstract claims a 34.7% time cost reduction, but Figure 7 shows KPIRoot+ takes about 8 seconds versus KPIRoot's 5 seconds—that's an increase. Either the comparison is against a different baseline or the text is wrong; it needs fixing. Second, Section 6.2 admits hyperparameters are chosen based on the best results on the test datasets. That's tuning on the test set, which inflates the reported improvements over KPIRoot. It's not fatal, but it weakens the precision of the headline numbers.\n\nThe more serious issue is Eq. 11. The correlation score adds a Jaccard term bounded on [0,1] to an unbounded Granger F-statistic, with no normalization. The paper states c_i is in [0,1], which is simply false. With λ=0.9, the 0.1-weighted F term can easily dominate the 0.9-weighted Jaccard term, so the ranking may be driven by causality alone rather than a principled integration. The ablation study removes either the ISAX or the decomposition component, but never the causality term, so it cannot detect this scale mismatch. The sensitivity analysis then tunes λ on the test data, potentially masking the problem.\n\nOn the other hand, the paper does ship a code repository, and the production experience is a legitimate contribution. The anomaly detection results show a clear improvement over the threshold approach in the original KPIRoot, which is exactly what the industrial case studies motivate.\n\nWho is this for? Practitioners working on root cause localization in large cloud systems who need a fast, interpretable alternative to graph neural networks. It deserves a serious referee, but the authors need to normalize the two score components, fix the time claim, and provide a proper validation set for hyperparameter selection or release the full KPIRoot+ code. I would accept the paper conditional on these revisions.","headline":"Useful incremental extension of KPIRoot with credible deployment evidence, but the score combination in Eq. 11 mixes an unbounded F-statistic with a bounded Jaccard coefficient without normalization, and test-set tuning inflates the reported gains.","tokens_in":22209,"tokens_out":3246,"would_cite":true,"duration_ms":39448,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that KPIRoot+ can pinpoint root-cause KPIs in cloud systems at an F1 of 0.882 and Hit@10 of 0.946 while running in about 8 seconds, by combining STL/U-Net anomaly detection, Improved SAX similarity, and Granger causality.","keywords":["root cause localization","cloud monitoring KPIs","time series decomposition","anomaly detection","symbolic aggregate approximation","Granger causality","U-Net autoencoder","production deployment"],"falsifier":"Run the pipeline on a labeled dataset where the alarm KPI is generated by a multiplicative or level-dependent seasonal process, so that additive STL is misspecified, or where ground-truth anomaly intervals are provided by independent SRE annotation; if F1 and Hit@10 do not beat KPIRoot's results, or if the ranking changes substantially when the U-Net is retrained with different seeds, the central claim would be contradicted.","tokens_in":21117,"feed_emoji":"🔍","tokens_out":6817,"duration_ms":57430,"temperature":0.7,"pith_summary":"KPIRoot+ is the paper's upgraded answer to a practical question: when an aggregated alarm KPI in a large cloud cluster goes abnormal, which individual VM KPIs actually caused it? The paper argues that the previous KPIRoot design failed on two fronts—threshold-based detection missed seasonal and residual anomalies, and SAX downsampling flattened away trend shape—and that replacing both with STL decomposition plus a U-Net autoencoder, and with an Improved SAX that records slope direction, gives more accurate root cause ranking without losing real-time efficiency. A sympathetic reader cares because root cause localization in production clouds has to be both interpretable and fast; the paper reports F1 0.882 and Hit@10 0.946 with an average runtime around 8 seconds on three industrial datasets, and a production deployment that localized ten emerging issues. The claim is an engineering one: a lightweight, explainable pipeline can beat both correlation baselines and graph-neural-network methods at a fraction of their cost.","feed_headline":"KPIRoot+ finds cloud root causes at 88.2% F1 in 8 seconds","feed_subtitle":"STL/U-Net anomaly detection plus similarity and causality scoring beats eight baselines in production cloud data.","key_machinery":"The load-bearing mechanism is a pipeline of four modules. First, Seasonal-Trend decomposition using Loess (STL) splits the alarm KPI into trend, seasonality, and remainder components. Second, a U-Net autoencoder—a convolutional autoencoder with skip connections—is trained on sliding-window segments and flags anomalies in each component separately, so trend, seasonal, and residual anomalies can all start a root-cause investigation. Third, Improved SAX (ISAX) down-samples every KPI into symbols whose alphabet index is augmented by the sign of the segment slope, so two segments with the same mean but opposite directions no longer share a symbol; Jaccard similarity over these symbol sequences measures how closely a VM KPI tracks the alarm KPI. Fourth, Granger causality, implemented as an F-test on autoregressions with and without the VM KPI's lagged values, checks temporal precedence. The final correlation score is a weighted sum $\\lambda \\times \\text{Jaccard} + (1-\\lambda)\\times F$, with $\\lambda=0.9$ in the evaluation. The efficiency claim rests on the SAX/ISAX reduction: with representation length $w \\approx \\sqrt{n}$, similarity analysis costs $O(\\sqrt{n})$ and Granger causality costs $O(\\sqrt{n} q^3)$, which is how the pipeline keeps its runtime near eight seconds.","core_discovery":"At its core, the paper claims that the quality of root cause localization is gated by the quality of anomaly segmentation, and that poor segmentation—not poor correlation—was the main weakness of KPIRoot. It models the alarm KPI additively as trend plus seasonality plus residual, detects anomalies in each component with a U-Net autoencoder trained on sliding windows, and uses the union of those detections to define the anomaly interval. Within that interval, each VM KPI is compared with the alarm KPI using Jaccard similarity over an Improved SAX alphabet whose symbols encode both level and slope direction, and scored with a Granger causality F-statistic that checks whether the VM KPI's past values help forecast the alarm KPI. The two scores are combined into a single ranking. The author's central empirical claim is that this pipeline achieves an F1 of 0.882 and Hit@10 of 0.946, outperforming eight state-of-the-art baselines by 2.9% to 35.7% while taking about 8 seconds on average, and that in production it correctly localized ten emerging performance issues.","pith_inferences":["Beyond the paper: the U-Net's calibration and retraining stability are not measured, so an obvious extension is to test whether anomaly-segment boundaries shift across training seeds and how much that changes the final ranking.","Beyond the paper: the additive STL assumption is likely the fragile point, and a natural test is to compare against multiplicative or trend-free decompositions on KPI families with strong daily or weekly seasonality.","Beyond the paper: because ISAX preserves only the sign of the slope, not its magnitude, very fast versus slow ramps inside one window still collapse; encoding slope magnitude or second differences is a direct, untested refinement.","Beyond the paper: the same similarity-plus-causality scoring could in principle be lifted to microservice trace metrics, although the paper itself argues that dependency-graph methods are unsuitable in its VM-isolated scenario."],"forward_implications":["If correct, the anomaly segmentation stage becomes the highest-leverage place to invest in cloud root cause analysis, since both similarity and causality downstream inherit its errors.","A root-cause candidate list with Hit@10 around 0.94 and NDCG@10 between 0.79 and 0.84 means engineers only need to inspect the top ten KPIs, shortening mean-time-to-resolution.","The $O(\\sqrt{n})$ complexity claim implies the approach can scale to thousands of VM KPIs per cluster, not just the 21 to 803 VMs in the evaluated datasets.","The production deployment claim—ten issues localized with 100% accuracy and localization time cut from 30 minutes to 5 minutes—suggests the method generalizes to anomaly types beyond the training distribution.","Because the method never needs a service dependency graph, it is applicable to VM-isolated cloud clusters where call graphs are unavailable."],"supporting_citations":[{"why":"The predecessor KPIRoot framework whose two drawbacks—threshold-based detection and SAX averaging—this paper identifies and fixes; it also supplies the main baseline.","marker":"Gu et al (2024b)"},{"why":"Introduces SAX and the Gaussian breakpoint construction that ISAX extends with slope signs.","marker":"Lin et al (2003)"},{"why":"Provides the STL seasonal-trend decomposition used to split the alarm KPI into trend, seasonal, and residual components.","marker":"RB (1990)"},{"why":"Supplies the U-Net autoencoder architecture used to detect anomalies in each decomposed component.","marker":"Ronneberger et al (2015)"},{"why":"Formulates the Granger causality framework and F-test used to test whether VM KPIs temporally precede the alarm KPI.","marker":"Shojaie and Fox (2022)"},{"why":"AID is the DTW-based similarity baseline that motivates the similarity analysis and is one of the eight baselines.","marker":"Yang et al (2021)"},{"why":"LOUD is the graph-centrality baseline for cloud fault localization that KPIRoot+ must beat.","marker":"Mariani et al (2018)"},{"why":"CMMD is the graph attention network baseline whose cross-metric root cause localization KPIRoot+ compares against.","marker":"Yan et al (2022)"},{"why":"CloudScout is the correlation-based service dependency discovery baseline used in the evaluation.","marker":"Yin et al (2016)"}],"fun_headline_variants":["KPIRoot+ lifts root cause F1 to 88.2%","Cloud root cause found in 8s via KPIRoot+","KPIRoot+ cuts cloud root cause time 34.7%","KPIRoot+ beats 8 cloud RCA baselines","U-Net plus SAX plus causality: KPIRoot+"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the alarm KPI can be decomposed as trend plus seasonality plus residual, and that anomalies found by the U-Net in those components mark the true anomaly interval; if that decomposition or the trained detector mis-times the interval, every downstream similarity and causality score is computed over the wrong window.","fun_headline_variants_meta":{"raw":{"variants":["KPIRoot+ lifts root cause F1 to 88.2%","Cloud root cause found in 8s via KPIRoot+","KPIRoot+ cuts cloud root cause time 34.7%","KPIRoot+ beats 8 cloud RCA baselines","U-Net plus SAX plus causality: KPIRoot+"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1334,"prompt_tokens":976,"completion_tokens":358,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":266}},"tokens_in":592,"tokens_out":358,"duration_ms":18209,"temperature":1.0,"reasoning_tokens":266,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:39:34.064975+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline on a labeled dataset where the alarm KPI is generated by a multiplicative or level-dependent seasonal process, so that additive STL is misspecified, or where ground-truth anomaly intervals are provided by independent SRE annotation; if F1 and Hit@10 do not beat KPIRoot's results, or if the ranking changes substantially when the U-Net is retrained with different seeds, the central claim would be contradicted.","supporting_citations":[{"cited_title":"In: Proceedings of the 8th ACM SIGMOD workshop on Research issues in data mining and knowledge discovery, pp 2--11","cited_arxiv_id":null,"evidence_quote":"Introduces SAX and the Gaussian breakpoint construction that ISAX extends with slope signs."},{"cited_title":"Annual Review of Statistics and Its Application 9:289--319","cited_arxiv_id":null,"evidence_quote":"Formulates the Granger causality framework and F-test used to test whether VM KPIs temporally precede the alarm KPI."},{"cited_title":"In: 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), IEEE, pp 653--665","cited_arxiv_id":null,"evidence_quote":"AID is the DTW-based similarity baseline that motivates the similarity analysis and is one of the eight baselines."},{"cited_title":"In: 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), IEEE, pp 262--273","cited_arxiv_id":null,"evidence_quote":"LOUD is the graph-centrality baseline for cloud fault localization that KPIRoot+ must beat."},{"cited_title":"In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp 4310--4320","cited_arxiv_id":null,"evidence_quote":"CMMD is the graph attention network baseline whose cross-metric root cause localization KPIRoot+ compares against."},{"cited_title":"IEEE Transactions on Parallel and Distributed Systems 28(5):1271--1284","cited_arxiv_id":null,"evidence_quote":"CloudScout is the correlation-based service dependency discovery baseline used in the evaluation."}],"review_version":1}