Distribution-free changepoint localization after sequential change detection

Aaditya Ramdas; Aytijhya Saha

REVIEW 2 major objections 4 minor 3 references

Distribution-free changepoint localization after sequential change detection

T0 review · 2 major / 4 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read After any sequential change detector stops, you can form a finite-sample confidence set for the changepoint with no distributional assumptions.

desk verdict First clean distribution-free post-detection localization with finite-sample conditional coverage and non-asymptotic size control; the math and experiments hold up. read the letter →

arxiv 2606.01256 v2 pith:PFHZLMYU submitted 2026-05-31 stat.ML cs.LGstat.ME

Aytijhya Saha , Aaditya Ramdas This is my paper

classification stat.MLcs.LGstat.ME MSC 62L1062G1562M99

keywords sequentialchangepointdetectionconformaltestmartingalesdistribution-freeinferencepost-detectionconfidencesetsexchangeabilityonlinecalibrators

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Once a sequential detector raises an alarm that a change in distribution has occurred, the usual procedures say nothing about when that change happened. This paper shows how to wrap any such detector with a conformal-test-martingale construction that produces a valid confidence set for the unknown changepoint. The only modelling assumptions required are exchangeability inside the pre-change segment and inside the post-change segment; the pre- and post-change distributions themselves may be completely unknown and may live on any space. Coverage is guaranteed conditionally on the detector having stopped after the true change, and non-asymptotic bounds show that the expected size of the set stays controlled—and can even remain bounded—when the detector’s delay is not too large. The method therefore turns any black-box sequential detector into a tool that both detects and localises changes without parametric models.

What carries the argument

Forward and backward conformal p-value sequences converted into test martingales by online-learned calibrators; a candidate time t is retained only when the corresponding martingale stays below the data-dependent threshold 1/(α r̂t), where r̂t is a distribution-free estimate of the null survival probability of the detector.

What would settle it

Run any conformal detector on a sequence with a known changepoint T, construct the reported two-sided set at nominal level 1−α, and check whether the empirical frequency of T belonging to the set, conditional on correct detection, falls materially below 1−α.

Watch

Extended reading notes

Core claim

Under only exchangeability of the pre- and post-change segments, the lower and upper conformal confidence sets Cα_low and Cβ_up, and their intersection, satisfy the finite-sample conditional coverage guarantee PF0,T,F1(T∈C∣τ≥T)≥1−α−β for an arbitrary detector whose stopping time is τ. Under mild regimes on the detection delay the conditional expected size of these sets remains O(1).

Load-bearing premise

The null distribution of the detector’s stopping time must either be free of the pre-change law or be estimable from an independent historical sample of that law, so that the Monte-Carlo threshold remains valid and independent of the conformal scores used for localisation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This paper finally gives a usable wrapper that turns any sequential detector into a distribution-free confidence set for the changepoint, with genuine finite-sample conditional coverage. That is new. Earlier post-detection work (Ding, Wu, and the authors’ own parametric paper) needed known or non-overlapping model classes; offline conformal localization does not survive data-dependent stopping. Here they invert forward and backward conformal e-martingales, calibrate the threshold with an independent null-survival estimator, and get PF(T in C | tau >= T) >= 1-alpha-beta under only segment-wise exchangeability.

The proofs are standard but carefully done: Ville for coverage, positive drift of order (k delta/(k+m))^2 plus McDiarmid with an O(m) variance proxy for the size bounds. The asymptotic sharpness (expected size O(1) or o(T) under logarithmic/exponential delay regimes) follows cleanly, and the online-learning extension with logarithmic regret is routine. Experiments on Gaussian mean shifts, MNIST digit changes, and the wine-quality stream look honest and match the theory; the lower sets are especially practical for quality-control isolation.

Soft spots are real but proportionate. You need a good estimator of P_infty(tau >= t) (Assumption 3 or the historical-sample construction in the appendix); that is automatic for conformal detectors and feasible otherwise, but it is an extra ingredient. Power still depends on the score function and the ONS+FLH hyperparameters, and the size bounds require the detector to wait long enough after the change. None of these break the coverage claims or the main contribution.

This is for anyone who already runs sequential detectors and wants honest uncertainty on when the change happened, plus the conformal-inference crowd. The citation pattern is clean; the concurrent parametric paper is used only for setup. I would bring it to reading group, cite it, and send it to peer review without hesitation. Solid, useful work.

Referee Report

2 major / 4 minor

Summary. The paper proposes a distribution-free post-detection procedure that, after any sequential changepoint detector stops at time τ, constructs lower, upper and two-sided confidence sets for the unknown changepoint T from the observed prefix alone. Using forward and backward conformal p-values converted to e-value martingales (with online-learned calibrators), the sets C_α_low, C_β_up and their intersection satisfy the finite-sample conditional coverage P(T ∈ C | τ ≥ T) ≥ 1-α-β under only segment-wise exchangeability (Theorems 1–2, Corollary 1). Non-asymptotic bounds on conditional expected size are derived via positive-drift calculations and McDiarmid concentration (Theorems 4, 6); under mild logarithmic/exponential delay regimes the expected size remains O(1) or o(T) (Theorems 3, 5, 7, 8–10). The method is detector-agnostic once an unbiased/negatively-biased estimator of the null survival function of τ is available, and is illustrated on Gaussian, MNIST and wine-quality data.

Significance. If the claims hold, the work supplies the first general distribution-free wrapper for sequential changepoint localization with valid post-detection coverage, decoupling detection from localization and removing the need for known, non-intersecting pre-/post-change model classes required by earlier parametric analyses. Finite-sample coverage via Ville on conformal e-martingales, explicit non-asymptotic size bounds that become uniformly bounded under natural delay regimes, and a practical online calibrator (ONS+FLH) constitute a clean and usable advance for streaming monitoring. The empirical sharpness on both synthetic and real data further supports practical utility. The contribution is therefore of clear interest to sequential analysis, conformal inference and distribution-free sequential decision making.

major comments (2)

Assumption 3 (or the historical-sample construction of Appendix C) is load-bearing for the Monte-Carlo estimator r̂_t that appears in the thresholds of (4)–(5). While Proposition 1 correctly shows that any conformal-p-value detector satisfies the assumption, the claim that the procedure is a wrapper around “any” detector is slightly overstated for non-conformal detectors that lack an independent historical F0 sample; the coverage proofs themselves remain valid once such an estimator is supplied, but the practical scope should be stated more carefully in the introduction and Section 2.4.
The size bounds (Theorems 3–10) condition on the stronger events {τ ≥ T + s log T} or {T ≤ τ ≤ T + exp(T/s)}, which are natural but stronger than the coverage conditioning {τ ≥ T}. Remark 2 explains the necessity, yet a short quantitative discussion of how often these events hold for standard ARL-controlled detectors (e.g., conformal CUSUM) would strengthen the asymptotic claims.

minor comments (4)

Abstract and first page: several small grammatical slips (“we proved … and demonstrate”, missing capital after period, “a valid post-detection coverage guarantee”). A light copy-edit would improve readability.
Section 2.5: the recommendation to use (monotone transforms of) the likelihood ratio is clear, but a one-sentence pointer to how the online estimation of pre-/post-change densities is performed in the experiments would help practitioners.
Figures 1–3: error bars and the precise definition of the plotted “average test statistic” are helpful; adding the numerical values of the realized set sizes next to the red markers would make the visual comparison with the theoretical O(1) claim more immediate.
Appendix C: the permutation estimator (15) is unbiased only for t ≤ m; a brief remark on the practical choice of historical sample size m relative to expected τ would be useful.

Circularity Check

1 steps flagged · score 1.0 of 10

No significant circularity: coverage and size bounds are derived from Ville, exchangeability and McDiarmid; only non-load-bearing self-citation for problem setup.

self citation load bearing [Section 1 (paragraph after Eq. 2) and Section 1.1]
"We follow the setup in Saha and Ramdas (2026); these authors studied this problem in a non-distribution-free setting... Saha and Ramdas (2026) also showed that for detection algorithms with a bounded average run length to false alarm... no post-detection confidence set C can satisfy an unconditional coverage guarantee."

The concurrent parametric paper is cited for the problem statement and the known impossibility result under ARL control. The citation is not load-bearing for any coverage or size claim of the present work (those rest on Ville + McDiarmid), so the circularity is minor and does not force the main theorems.

full rationale

The finite-sample conditional coverage (Theorems 1–2, Corollary 1) follows by applying Ville’s inequality to the conformal e-martingales constructed from exchangeable segments (Assumptions 1–2), after thresholding by an independent unbiased/negatively-biased estimator r̂_t of the null survival function; the equality P_{F0,T,F1}(τ≥T)=P_{F0,∞}(τ≥T) is immediate from F_{T-1}-measurability of {τ<T}. Size bounds (Theorems 3–10) rest on a strictly positive expected log-drift of order (kδ/(k+m_k))^{2} together with a McDiarmid variance proxy O(m_k) obtained from rank-sensitivity analysis of the conformal p-values; the logarithmic-regret extension for learned calibrators is standard online-convex-optimization bookkeeping and does not embed the target claim. The sole self-reference (Saha & Ramdas 2026) supplies only the abstract problem formulation and the known impossibility of unconditional coverage under ARL control; no numerical constant, uniqueness theorem or lemma from that paper is used inside any proof of the present distribution-free results. No parameter is fitted to data and then re-presented as a prediction, no ansatz is smuggled via citation, and the constructions are not definitional tautologies. The derivation is therefore self-contained against external martingale and concentration tools.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central coverage claim rests only on exchangeability inside each regime plus the existence of an unbiased/negatively-biased estimator of the null survival function of the detector; size claims add independence, a positive total-variation gap for the score, and logarithmic regret of the online calibrator. No free parameters are fitted to the target coverage probability; α is user-chosen. No new physical or statistical entities are postulated.

free parameters (2)

score function S (and S')
Any measurable score is valid for coverage; power (size) depends on the discriminative gap δ of the chosen score. In experiments the authors use identity, negative identity, or a pre-trained log-likelihood ratio; these are design choices, not fitted constants that enter the coverage proof.
online-learning hyperparameters of ONS+FLH (β, expert weights)
Affect only the regret constant C' that appears in the size bounds of Section 4; coverage remains valid for any sequence of calibrators with integral ≤ 1.

assumptions (5)

domain assumption Post-change observations XT, XT+1, … are exchangeable (Assumption 1)
Required for the forward conformal p-values to be i.i.d. Uniform under the null T = t, enabling the e-value martingale and Ville’s inequality (Theorem 1).
domain assumption Pre-change observations X1, …, XT-1 are exchangeable (Assumption 2)
Symmetric requirement for the backward martingale used in the upper confidence set (Theorem 2).
domain assumption Null distribution of the stopping time τ does not depend on F0 (Assumption 3), or an independent historical sample from F0 is available
Needed to construct the unbiased/negatively-biased estimator r̂_t that appears in the thresholds of C_low and C_up; without it the conditional coverage proof fails.
domain assumption Observations are independent and the score satisfies |P(S(Z)>S(Y))-1/2|=δ>0 (Assumptions 4–5)
Used only for the non-asymptotic and asymptotic size bounds (Theorems 3–10); coverage holds without them.
standard math Online calibrator achieves logarithmic adaptive regret (Assumption 9/10)
Standard guarantee for ONS+FLH on 1-exp-concave losses; converts the oracle size bounds into practical size bounds.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Distribution-free changepoint localization after sequential change detection." pith.science (2026). https://pith.science/paper/PFHZLMYU

@misc{pith2026260601256,
  author       = {Pith},
  title        = {Pith review of: Distribution-free changepoint localization after sequential change detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PFHZLMYU}},
  note         = {Machine review of arXiv:2606.01256}
}

read the original abstract

This paper introduces a distribution-free framework for constructing post-detection confidence sets for changepoints after stopping a sequential change detection procedure. It is well known that conformal test martingales can be used to sequentially detect changes in distribution, but by themselves provide no inference for the time at which a proclaimed change occurred. Past work on post-detection inference requires pre- and post-change classes of distributions to be known, but this paper accomplishes localization of the changepoint without any distributional assumptions. We establish finite-sample coverage guarantees (conditional on correct detection). We provide non-asymptotic bounds on the conditional expected size of the confidence sets. Under suitable asymptotic regimes, we prove that the conditional expected size of the confidence set remains uniformly bounded and demonstrate strong empirical performance on simulated and real data. To the best of our knowledge, this is the first general distribution-free framework for sequential changepoint localization with valid post-detection coverage.

Figures

Figures reproduced from arXiv: 2606.01256 by the authors.

**Figure 1.** Gaussian mean change: changepoint at T = 500, marked in black vertical lines. T = 400. We consider two different settings — (i) Observations before the changepoint are handwritten digit 3, while observations after the changepoint are handwritten digit 7. (ii) 80% of the observations before the changepoint are handwritten digit 3 (and the remaining 20% are handwritten digit 7), while 20% of the observations after the… view at source ↗

**Figure 2.** Test statistics for lower and upper confidence sets are plotted, and the 90% [PITH_FULL_IMAGE:figures/full_fig_p022_2.png] view at source ↗

**Figure 3.** Test statistic for lower confidence sets is plotted, and the 90% lower confidence [PITH_FULL_IMAGE:figures/full_fig_p022_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

3 extracted references

[1]

Ift≤m, thenE[ˆr t] =P F0,∞(τ≥t), which is an unbiased estimate
[2]

Since PF0,∞(τ≥t ) ≥ 0, the estimator is negatively biased (or unbiased if the probability is exactly zero)

If t > m , then E[ˆrt] = 0. Since PF0,∞(τ≥t ) ≥ 0, the estimator is negatively biased (or unbiased if the probability is exactly zero). In all cases,E[ˆrt]≤P F0,∞(τ≥t). Another asymptotic alternative is based on a bootstrap approximation. Draw B many bootstrap sequences {Y ∗j n }n, for j = 1 , .., B and run the change detector to compute 45 1(τ j ≥t), for...
[3]

Note that a larger A leads to a longer detection delay

is the conformal test martingale for the sequence X1,· · ·, X t. Note that a larger A leads to a longer detection delay. The average (of 50 independent runs) conditional size of the both-sided confidence sets ( (7) with α = 0.05) is reported in Tables 1 and 2. In Table 1, we consider relatively small values of A/T ratio and s. In this regime, the confiden...

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.

[1] [1]

Ift≤m, thenE[ˆr t] =P F0,∞(τ≥t), which is an unbiased estimate

[2] [2]

Since PF0,∞(τ≥t ) ≥ 0, the estimator is negatively biased (or unbiased if the probability is exactly zero)

If t > m , then E[ˆrt] = 0. Since PF0,∞(τ≥t ) ≥ 0, the estimator is negatively biased (or unbiased if the probability is exactly zero). In all cases,E[ˆrt]≤P F0,∞(τ≥t). Another asymptotic alternative is based on a bootstrap approximation. Draw B many bootstrap sequences {Y ∗j n }n, for j = 1 , .., B and run the change detector to compute 45 1(τ j ≥t), for...

[3] [3]

Note that a larger A leads to a longer detection delay

is the conformal test martingale for the sequence X1,· · ·, X t. Note that a larger A leads to a longer detection delay. The average (of 50 independent runs) conditional size of the both-sided confidence sets ( (7) with α = 0.05) is reported in Tables 1 and 2. In Table 1, we consider relatively small values of A/T ratio and s. In this regime, the confiden...