pith. sign in

arxiv: 2605.21627 · v1 · pith:ARJFXBTVnew · submitted 2026-05-20 · 📊 stat.ME · stat.ML

Distribution-free root cause analysis

Rohan Hore , Aaditya Ramdas This is my paper

Pith reviewed 2026-05-22 08:57 UTC · model grok-4.3

classification 📊 stat.ME stat.ML
keywords root cause analysisconformal predictiondistribution-free inferencechange detectionmulti-stream dataconfidence setsp-values
0
0 comments X

The pith

Conformal p-values construct finite-sample valid confidence sets for the root-cause stream among multiple changing data streams.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a framework that identifies which of several independent data streams is the first to undergo a distributional change, without assuming any particular form for those distributions. It does so by turning scores computed before and after a candidate change point into conformal p-values and then forming a set of streams that must contain the true earliest one with guaranteed probability. A sympathetic reader cares because many monitoring applications in engineering, finance, and science involve multiple sensors or series whose change order reveals the underlying cause, yet classical methods require strong parametric assumptions that are rarely met. The work further proves that any other distribution-free root-cause procedure can be recast inside the same framework and that suitably chosen scores make the resulting sets asymptotically as small as possible.

Core claim

Leveraging conformal p-values, the authors propose Conformal Root Cause Analysis (CROC) which constructs finite-sample valid confidence sets for the root-cause index under the assumptions that the data streams are independent and that, within each stream, the pre- and post-change observations are exchangeably sampled from arbitrary unknown distributions. They establish a universality property showing that any distribution-free root-cause localization method can be represented within CROC, and under mild regularity conditions and principled score design the method yields asymptotically sharp confidence sets. The framework is extended to accommodate cross-stream dependence while preserving the

What carries the argument

Conformal Root Cause Analysis (CROC) framework, which converts pre- and post-change scores on each stream into p-values and assembles them into a finite-sample valid set for the index of the earliest-changing stream.

If this is right

  • Any finite collection of streams yields a non-empty confidence set that covers the true root-cause index with probability at least 1-alpha, for any sample size.
  • Existing distribution-free change-localization procedures can be embedded inside CROC without losing validity.
  • When scores are chosen to separate pre- and post-change distributions well, the size of the confidence set shrinks to one as the number of observations grows.
  • The same construction continues to deliver valid sets after a mild relaxation that permits limited cross-stream dependence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The method could be run sequentially on streaming data by updating the conformal p-values in an online fashion.
  • When streams exhibit slow drifts rather than abrupt changes, the exchangeability assumption inside each window may need to be replaced by a weaker local-exchangeability condition.
  • Combining CROC sets across multiple candidate change times could produce a joint inference procedure for both the root cause and its approximate timing.

Load-bearing premise

The data streams are independent, and within each stream the pre- and post-change observations are sampled exchangeably from arbitrary and unknown distributions.

What would settle it

Generate many finite-sample datasets from independent streams with a known earliest-changing stream; compute the CROC confidence set at nominal level 1-alpha and check whether the empirical coverage falls below 1-alpha.

Figures

Figures reproduced from arXiv: 2605.21627 by Aaditya Ramdas, Rohan Hore.

Figure 1
Figure 1. Figure 1: CROC root-cause p-values across candidate streams. Left: Setting 1 (moderate sepa￾ration). Right: Setting 2 (weak separation). The true root stream is highlighted in red, and the dotted line denotes α = 0.1. able. In particular, interactions across streams are often localized, allowing the streams to be partitioned into groups that are internally independent. This motivates a refinement of CONCH￾agg: by ap… view at source ↗
Figure 2
Figure 2. Figure 2: Left: Example images from each stream before and after the changepoint. Right: [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: CROC root-cause p-values for the multi-domain sentiment experiment. The streams correspond to books, DVD, electronics, and kitchen/housewares reviews. The black dotted line denotes the threshold α = 0.1. B CROC under structured cross-stream dependence Building on the baseline method of CONCH-agg framework (See Algorithm 3) introduced in Section 6 for handling arbitrary cross-stream dependence, we show how … view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of CONCH-agg and CROC-dep under structured cross-stream dependence with K = 6 streams. The second stream is the root stream, and dependence is introduced within pairs (1, 2), (3, 4), and (5, 6). The plot shows root-cause p-values across candidate streams. C Proofs C.1 Proving validity of CROC confidence sets C.2 Proof of Theorem 3.1 Fix t ∈ R. Observe that under the null H′ 0,t , π(X) d= X for a… view at source ↗
read the original abstract

We study distribution-free root cause analysis in multi-stream data, where an evolving underlying system is observed through multiple data streams that may each undergo distributional changes at unknown timepoints. In such settings, the stream exhibiting the earliest change provides a natural starting point for investigating the underlying cause, which we refer to as the root-cause index. Leveraging conformal $p$-values, we propose a novel framework, Conformal Root Cause Analysis (CROC), which constructs finite-sample valid confidence sets for the root-cause index under minimal assumptions: the data streams are independent, and within each stream the pre- and post-change observations are sampled exchangeably from arbitrary and unknown distributions. We further establish a universality property, showing that any distribution-free method for root cause localization can be represented within the CROC framework. In addition, under mild regularity conditions and principled score design, our method yields asymptotically sharp confidence sets that efficiently isolate the root cause. We further extend CROC to efficiently handle cross-stream dependence when present. Extensive simulations demonstrate accurate localization of the root stream, supporting our theoretical guarantees.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript proposes the Conformal Root Cause Analysis (CROC) framework, which applies conformal p-values to construct finite-sample valid confidence sets for the root-cause index (the stream with the earliest distributional change) in multi-stream data. The central assumptions are cross-stream independence and within-stream exchangeability of pre- and post-change observations drawn from arbitrary unknown distributions. Additional results include a universality property (any distribution-free root-cause method can be represented in CROC), asymptotic sharpness under mild regularity conditions with principled score design, an extension to cross-stream dependence, and simulation evidence of accurate localization.

Significance. If the finite-sample validity of the confidence sets is established under the stated minimal assumptions, the work would provide a useful distribution-free tool for root-cause localization in monitoring and fault-detection settings. The universality claim, if rigorously shown, would position CROC as a general wrapper; the asymptotic sharpness result would add efficiency guarantees. Simulation support is noted but secondary to the theoretical claims.

major comments (3)
  1. [Section on conformal p-value construction and validity proof] The finite-sample validity claim rests on each conformal p-value being super-uniform under the null that its candidate index is the true root cause. Because change times are unknown, any practical score or p-value construction necessarily involves a data-dependent search or ranking over possible split points within each stream. This selection step can destroy the exchangeability required for the conformal guarantee to hold marginally, even when the raw observations satisfy the stated assumptions. The manuscript must supply the explicit argument (likely in the proof of the main validity theorem) showing that super-uniformity is nevertheless preserved.
  2. [Section stating the universality property] The universality property asserts that any distribution-free root-cause localization method can be represented inside the CROC framework. This requires an explicit embedding or reduction argument that maps an arbitrary valid procedure into a choice of score function and conformal p-value within CROC; without it, the claim reduces to a restatement rather than a substantive unification.
  3. [Section on asymptotic sharpness] Asymptotic sharpness is claimed under 'mild regularity conditions and principled score design.' These conditions must be stated precisely (e.g., rates on the score functions or separation between change times), and it must be shown that they yield confidence sets that isolate the true root cause with probability approaching 1 at the optimal rate; otherwise the sharpness claim is not load-bearing for the efficiency guarantee.
minor comments (2)
  1. [Abstract] The abstract states that simulations 'demonstrate accurate localization'; adding a brief summary of the simulation design (number of streams, sample sizes per stream, types of distributional shifts, and performance metrics) would improve readability.
  2. [Notation and definitions] Notation for the root-cause index, candidate indices, and the resulting confidence sets should be introduced once and used consistently to avoid ambiguity when moving between the method and the theoretical statements.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the careful and constructive review. The comments highlight important points regarding the rigor of our theoretical claims. We respond to each major comment below and indicate the revisions we will make.

read point-by-point responses

  1. Referee: The finite-sample validity claim rests on each conformal p-value being super-uniform under the null that its candidate index is the true root cause. Because change times are unknown, any practical score or p-value construction necessarily involves a data-dependent search or ranking over possible split points within each stream. This selection step can destroy the exchangeability required for the conformal guarantee to hold marginally, even when the raw observations satisfy the stated assumptions. The manuscript must supply the explicit argument showing that super-uniformity is nevertheless preserved.

    Authors: We agree that an explicit argument is required to confirm preservation of super-uniformity after data-dependent split-point selection. The current proof of the main validity result (Theorem 3.1) invokes exchangeability within streams and independence across streams to establish marginal validity, but does not spell out the intermediate steps that show the selection does not introduce bias under the null. In the revision we will expand the proof with a dedicated lemma that demonstrates the conformal p-value remains super-uniform by exploiting the symmetry of the exchangeable pre- and post-change observations with respect to any fixed ranking rule. revision: yes

  2. Referee: The universality property asserts that any distribution-free root-cause localization method can be represented inside the CROC framework. This requires an explicit embedding or reduction argument that maps an arbitrary valid procedure into a choice of score function and conformal p-value within CROC; without it, the claim reduces to a restatement rather than a substantive unification.

    Authors: The universality claim is meant to position CROC as a general wrapper. To make the reduction explicit, we will add a new proposition that, for any distribution-free procedure outputting a valid root-cause index, constructs a score function whose conformal p-values recover the same decisions. The construction will map the arbitrary method's output directly into the thresholded p-value set of CROC, thereby showing that every such method is a special case of our framework. revision: yes

  3. Referee: Asymptotic sharpness is claimed under 'mild regularity conditions and principled score design.' These conditions must be stated precisely (e.g., rates on the score functions or separation between change times), and it must be shown that they yield confidence sets that isolate the true root cause with probability approaching 1 at the optimal rate; otherwise the sharpness claim is not load-bearing for the efficiency guarantee.

    Authors: We accept that the regularity conditions and the rate result need to be stated more precisely. In the revision we will replace the current informal statement with an explicit set of assumptions (minimum separation of change times of order log n / n and uniform consistency of the score functions at rate o(1/sqrt(n))). Under these conditions we will prove that the probability that the confidence set contains any stream other than the true root cause tends to zero at the optimal rate, thereby making the sharpness claim rigorous. revision: yes

Circularity Check

0 steps flagged

No significant circularity in CROC derivation chain

full rationale

The paper applies standard conformal p-value constructions to a new root-cause localization task under explicitly stated cross-stream independence and within-stream exchangeability assumptions. The finite-sample validity of the resulting confidence sets follows directly from the exchangeability property without any reduction of the target sets to fitted parameters or data-dependent selections that are redefined as predictions. The universality claim is a representation result showing that other distribution-free methods fit inside the framework, not a self-definitional equivalence. No load-bearing self-citation chains, ansatz smuggling, or renaming of known results appear in the derivation; the central guarantees rest on the minimal assumptions plus established conformal properties that are externally verifiable.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 1 invented entities

The central claim rests on conformal p-value construction plus the two stated minimal assumptions; no free parameters or new entities with independent evidence are described.

axioms (2)
  • domain assumption Data streams are independent
    Stated as a minimal assumption required for the validity of the confidence sets.
  • domain assumption Within each stream the pre- and post-change observations are sampled exchangeably from arbitrary and unknown distributions
    Enables distribution-free use of conformal p-values without parametric assumptions.
invented entities (1)
  • CROC framework no independent evidence
    purpose: Constructs finite-sample valid confidence sets for the root-cause index
    Newly proposed method that leverages conformal p-values for this task.

pith-pipeline@v0.9.0 · 5707 in / 1312 out tokens · 65907 ms · 2026-05-22T08:57:05.804201+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages · 2 internal anchors

  1. [1]

    Expert Systems with Applications , volume=

    Using Bayesian networks for root cause analysis in statistical process control , author=. Expert Systems with Applications , volume=. 2011 , publisher=

    work page 2011

  2. [2]

    The Annals of Statistics , volume=

    Testing for outliers with conformal p-values , author=. The Annals of Statistics , volume=. 2023 , publisher=

    work page 2023

  3. [3]

    2009 28th IEEE International Symposium on Reliable Distributed Systems , pages=

    A framework for distributed monitoring and root cause analysis for large ip networks , author=. 2009 28th IEEE International Symposium on Reliable Distributed Systems , pages=. 2009 , organization=

    work page 2009

  4. [4]

    Proceedings of the 45th annual meeting of the association of computational linguistics , pages=

    Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification , author=. Proceedings of the 45th annual meeting of the association of computational linguistics , pages=

    work page

  5. [5]

    Statistics & probability letters , volume=

    Universal residuals: A multivariate transformation , author=. Statistics & probability letters , volume=. 2007 , publisher=

    work page 2007

  6. [6]

    Computational Statistics & Data Analysis , volume=

    Bootstrap confidence intervals for multiple change points based on moving sum procedures , author=. Computational Statistics & Data Analysis , volume=. 2022 , publisher=

    work page 2022

  7. [7]

    arXiv preprint arXiv:2505.00292 , year=

    Offline changepoint localization using a matrix of conformal p-values , author=. arXiv preprint arXiv:2505.00292 , year=

    work page arXiv

  8. [8]

    IEEE signal processing magazine , volume=

    The mnist database of handwritten digit images for machine learning research [best of the web] , author=. IEEE signal processing magazine , volume=. 2012 , publisher=

    work page 2012

  9. [9]

    Journal of the Royal Statistical Society Series B: Statistical Methodology , volume=

    Multiscale change point inference , author=. Journal of the Royal Statistical Society Series B: Statistical Methodology , volume=. 2014 , publisher=

    work page 2014

  10. [10]

    arXiv preprint arXiv:2602.06267 , year=

    Conformal changepoint localization , author=. arXiv preprint arXiv:2602.06267 , year=

    work page arXiv

  11. [11]

    Journal of Machine Learning Research , volume=

    Selection by prediction with conformal p-values , author=. Journal of Machine Learning Research , volume=

    work page

  12. [12]

    Biometrika , volume=

    The likelihood ratio test for a change-point in simple linear regression , author=. Biometrika , volume=. 1989 , publisher=

    work page 1989

  13. [13]

    2005 , publisher=

    Testing statistical hypotheses , author=. 2005 , publisher=

    work page 2005

  14. [14]

    arXiv preprint arXiv:2503.23051 , year=

    Coca: Generative root cause analysis for distributed systems with code knowledge , author=. arXiv preprint arXiv:2503.23051 , year=

    work page arXiv

  15. [15]

    2017 IEEE 56th annual conference on decision and control (CDC) , pages=

    Data-driven root-cause analysis for distributed system anomalies , author=. 2017 IEEE 56th annual conference on decision and control (CDC) , pages=. 2017 , organization=

    work page 2017

  16. [16]

    Journal of the Royal Statistical Society Series B: Statistical Methodology , pages=

    Post-detection inference for sequential changepoint localization , author=. Journal of the Royal Statistical Society Series B: Statistical Methodology , pages=. 2026 , publisher=

    work page 2026

  17. [17]

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter , author=. arXiv preprint arXiv:1910.01108 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  18. [18]

    Survey on Models and Techniques for Root-Cause Analysis

    Survey on models and techniques for root-cause analysis , author=. arXiv preprint arXiv:1701.08546 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    The Annals of Statistics , volume=

    Optimal change-point detection and localization , author=. The Annals of Statistics , volume=. 2023 , publisher=

    work page 2023

  20. [20]

    Machine-learning applications of algorithmic randomness , author=

    work page

  21. [21]

    Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages=

    Root cause analysis for microservice systems via hierarchical reinforcement learning from human feedback , author=. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages=

    work page

  22. [22]

    arXiv preprint arXiv:2409.16829 , year=

    Conditional testing based on localized conformal p-values , author=. arXiv preprint arXiv:2409.16829 , year=

    work page arXiv

  23. [23]

    IEEE Transactions on Automation Science and Engineering , volume=

    Statistical estimation and testing for variation root-cause identification of multistage manufacturing processes , author=. IEEE Transactions on Automation Science and Engineering , volume=. 2004 , publisher=

    work page 2004