REVIEW 2 major objections 4 minor 1 cited by
Conformal changepoint localization
T0 review · 2 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read The paper proves that ranking each candidate changepoint by how often within-segment permutations beat its plausibility score yields a finite-sample confidence set that contains the true changepoint with probability at least 1−α, under no d
desk verdict CONCH gives a genuinely finite-sample, distribution-free confidence set for changepoint localization under exchangeability, and the paper deserves a serious referee despite a known gap between the learned-score theory and the experiments. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the split-permutation group Π_t, the set of permutations of the n indices that shuffle the left block among itself and the right block among itself without mixing across t. For any changepoint plausibility score S, the conformal p-value p_t is the rank of S_t(X) among the scores evaluated on all permuted data sets. This rank transform is what converts exchangeability of the two blocks into finite-sample coverage, and it is also what allows arbitrary black-box scores to be plugged in without altering validity.
What would settle it
Generate n observations with a changepoint where the pre-change block is an AR(1) process with nonzero autocorrelation and the post-change block is i.i.d.; run CONCH with a weighted-mean score over many replicates and check empirical coverage. If coverage falls materially below 1−α, the exchangeability assumption is load-bearing; if coverage remains at 1−α, the method tolerates more within-block dependence than Assumption 1 states.
Extended reading notes
Core claim
CONCH defines, for each candidate index t, a conformal p-value p_t as the fraction of all permutations that keep the left and right blocks intact for which the plausibility score at t is no larger than the score on the observed data. Theorem 3.1 shows that p_t is super-uniform under the null hypothesis that t is the true changepoint, so thresholding at α yields a distribution-free confidence set with coverage at least 1−α. Building on this, the paper proves a conformal Neyman–Pearson lemma: the score that maximizes power against a given alternative is the likelihood ratio between the two changepoint hypotheses, and any strictly increasing transform of this oracle score minimizes expected con
Load-bearing premise
The guarantee holds only if, under the null hypothesis that t is the true changepoint, the observations to the left of t are exchangeable among themselves, those to the right are exchangeable among themselves, and the two groups are independent.
Editorial extensions
If this is right
- For any score S and any α, the set {t : p_t > α} contains the true changepoint with probability at least 1−α whenever the pre- and post-change blocks are exchangeable and independent.
- The oracle likelihood-ratio score minimizes expected confidence-set length, so with a well-chosen score the confidence set can be a singleton or near-singleton.
- If a learned log-likelihood ratio is L2-consistent on data independent of the test sequence, the normalized confidence-set length converges to zero as the sample size grows.
- Any heuristic or model-based confidence set can be recalibrated into a distribution-free finite-sample-valid set by defining a score from membership in or distance to that set.
- Any distribution-free changepoint localization procedure can be represented as CONCH with an appropriate score, making CONCH the canonical class for such methods.
Reading between the lines
- Inference: Because validity is automatic for any score, the practical bottleneck is score design; investing in a distinguishing score (for example, classifier logits) is the main lever for set width, not further statistical machinery.
- Inference: When within-segment exchangeability fails but dependence has a known block structure, replacing full permutations with block permutations is a natural repair; the paper's proof structure suggests this route, though the paper itself does not establish a formal guarantee for it.
- Inference: The sharpness theory assumes the learned score is trained on separate data, while the experiments train on the same sequence; the authors flag this gap, and a useful test is whether in-sample training preserves both coverage and shrinking set length in practice.
- Inference: The universality result positions CONCH as an audit layer: any black-box segmentation algorithm's reported changepoint can be turned into a finite-sample-valid confidence set without re-deriving distributional theory.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CONCH, a conformal prediction framework for offline changepoint localization. Given any changepoint plausibility score S, it defines a permutation p-value p_t by comparing S_t(X) against S_t(π(X)) over the split-permutation group Π_t, and outputs the confidence set {t : p_t > α}. The central theoretical claim (Theorem 3.1) is finite-sample distribution-free coverage under exchangeability of the pre-change segment, exchangeability of the post-change segment, and independence between the two. The paper further contributes an oracle optimal score via a "conformal Neyman-Pearson lemma," asymptotic sharpness results for oracle and learned log-likelihood-ratio scores (Theorems 5.1 and 5.2), a universality theorem (Theorem 6.1), a calibration procedure for heuristic confidence sets, an extension to multiple changepoints, and experiments on synthetic and real image/text data.
Significance. If the sharpness results were rigorously established, this would be a substantial contribution: it would provide the first finite-sample distribution-free confidence sets for changepoint localization that are also asymptotically sharp, complementing the broader conformal-prediction toolbox. The universality theorem is elegant and gives a useful design principle. The coverage theorem itself is clean and correct, and the paper is transparent about its assumptions and limitations. However, the proof of asymptotic sharpness contains a false deterministic inequality (Lemma B.8) that invalidates the proofs of Theorems 5.1 and 5.2 as written. Since asymptotic sharpness is one of the paper's headline contributions, the manuscript currently overclaims relative to what is proven.
major comments (2)
- [Appendix B.3.4, Lemma B.8] Lemma B.8 claims p_{t,n} ≤ \tilde{p}_{t,n} deterministically for the CPP score (5.1). The inequality direction is reversed. For every π, S_t(π(X)) ≤ S^{(n)}_t(π(X)) because \hat{ξ}(π(X)) is the maximizer of the permuted prefix sum, while S_t(X) = S^{(n)}_t(X). Therefore {S^{(n)}_t(π(X)) ≤ S_t(X)} ⊆ {S_t(π(X)) ≤ S_t(X)}, giving \tilde{p}_{t,n} ≤ p_{t,n}. A concrete counterexample: take n=5, t=4, \hat{ξ}=2, and \hat{ℓ}-values (10,10,-100,100,-1). The permutation π=(1,3,4,2,5) gives S_4(π)=0 ≤ S_4(X)=0, so it contributes to p_4, but S^{(n)}_4(π)=110 > 0, so it does not contribute to \tilde{p}_4. Thus p_4 > \tilde{p}_4. The proofs of Theorem 5.1 (Step 1, Appendix B.3.2) and Theorem 5.2 (final inequality, Appendix B.3.3) rely on p_{t,n} ≤ \tilde{p}_{t,n}; with the inequality reversed, the stated arguments do not yield the claimed O_P(1) length or normalized-length convergence. The asymptotic
- [Section 5.2] Theorem 5.2 assumes that \hat{ℓ}_n is estimated on an independent dataset D'_n, as stated in (5.3) and the surrounding text. The experiments in Section 8 train the score on the same sequence D_n, so the formal sharpness guarantee does not apply to the headline experimental results. The authors acknowledge this gap and suggest a sample-splitting scheme. This is a limitation rather than an internal inconsistency, but it is load-bearing for the practical interpretation of the sharpness claim: a sample-splitting experiment or a clear statement that the reported widths are only heuristic would be needed to connect Theorem 5.2 to the empirical results.
minor comments (4)
- [Appendix B.3.3] Typo: 'Hnece' should be 'Hence'. Also in (5.2), the argument of \hat{ℓ}_n appears as x_s; it should be x_i.
- [Appendix B.1.2] Typo: 'fucntion' should be 'function'.
- [Appendix B.3.2] Typo: 'independet' should be 'independent'.
- [Section 4.1] The phrase 'which we call the second1 Conformal NP Lemma' contains a stray footnote marker '1'.
Circularity Check
No significant circularity: central coverage theorem is a direct exchangeability argument; acknowledged Theorem 5.2 gap is an assumption mismatch, not a circular step.
full rationale
The paper's central finite-sample guarantee (Theorem 3.1) is derived directly from Assumption 1: under H_{0,t}, X and any π(X) with π∈Π_t are equally distributed, so the conformal p-value p_t in (3.2) is super-uniform by Harrison's lemma. This argument does not fit any parameter, does not invoke the authors' prior results, and holds for any user-specified CPP score. The 'optimal score' results (Lemma 4.2, Theorem 4.3) are oracle benchmarks: they assume f0, f1, and ξ are known and use the classical Neyman-Pearson lemma; they are not predictions from fitted inputs. The universality result (Theorem 6.1) is a representation theorem - it constructs a CPP score from a given valid set and shows CONCH can reproduce it - but it is not used to establish coverage and is not a self-citation chain. The only notable caveat is an explicit limitation, quoted in Section 5.2: 'While Theorem 5.2 assumes that ℓ̂n is learned on data independent of Dn, CONCH would typically use ℓ̂n that is learned on Dn itself. Nevertheless, we expect that asymptotic sharpness should continue to hold provided the learned estimator is suitably stable... we leave it to future work.' This is an honest gap between the theorem's independence assumption and the same-data experiments, not a circular reduction. Self-citations to Dandapanthula and Ramdas (2025) are contextual or comparative, not load-bearing. Overall, the derivation is self-contained against the exchangeability assumption, and no step reduces by construction to its own input.
Assumptions & free parameters
free parameters (3)
- Weight function w_{t,i} in score (4.3)
- Monte Carlo permutation count M (Algorithm 4)
- Classifier architecture and training schedule (Section 8)
assumptions (5)
- domain assumption Exchangeability and independence of pre/post segments (Assumption 1)
- standard math Harrison (2012) Lemma 3
- standard math Neyman-Pearson lemma
- domain assumption Finite second moment of log-likelihood ratio and interior changepoint (Theorem 5.1)
- domain assumption L2 consistency of estimated log-likelihood ratio on independent data (5.3)
Cite this review
Pith. "Pith review of Conformal changepoint localization." pith.science (2026). https://pith.science/paper/MH3R6333
@misc{pith2026260206267,
author = {Pith},
title = {Pith review of: Conformal changepoint localization},
year = {2026},
howpublished = {\url{https://pith.science/paper/MH3R6333}},
note = {Machine review of arXiv:2602.06267}
}
read the original abstract
We study the problem of offline changepoint localization in a distribution-free setting. One observes a vector of data with a single changepoint, assuming that the data before and after the changepoint are i.i.d. (or more generally exchangeable) from arbitrary and unknown distributions. The goal is to produce a finite-sample confidence set for the index at which the change occurs without making any other assumptions. Existing methods often rely on parametric assumptions, tail conditions, or asymptotic approximations, or only produce point estimates. In contrast, our distribution-free algorithm, CONformal CHangepoint localization (CONCH), only leverages exchangeability arguments to construct confidence sets with finite sample coverage. By proving a conformal Neyman--Pearson lemma, we derive principled score functions that yield informative (small) sets. Moreover, with such score functions, the normalized length of the confidence set shrinks to zero under weak assumptions. We also establish a universality result showing that any distribution-free changepoint localization method must be an instance of CONCH. Experiments suggest that CONCH delivers precise confidence sets even in challenging settings involving images or text.
Figures
Figures from the paper (9 more)
Forward citations
Cited by 1 Pith paper
-
Conformal Changepoint Localization and Root Cause Analysis with Corrupted Observations
Weighted conformal changepoint localization (W-CONCH) and root-cause analysis (W-CROC) downweight likely-corrupted observations via classifier uncertainty, preserving coverage and shrinking confidence sets.
Reference graph
Works this paper leans on
-
[1]
Lemma B.8.Let{p t,n}be theCONCHp-values computed based on CPP score(5.1)
) (1 +ε 2,n/η+ 2 √ ε2,n/η) η ≥1−4η, 42 as required. Lemma B.8.Let{p t,n}be theCONCHp-values computed based on CPP score(5.1). Then, it holds that deterministically, pt,n ≤˜pt,n,fort∈[n−1]. Proof.We start with observing that the original scoreS t(X)and frozen scoreS (n) t (X)can be equivalently written as St(x) = t∑ i=1 ˆℓn(xi)− ˆξn(x)∑ i=1 ˆℓn(xi), S (...
2005
-
[2]
Given samplesX∈R n, observe that d ( P[r] X|XL,t,XR,t ) d ( P[t] X|XL,t,XR,t )(X)∝ d(P[r] X ) d(P[r] X ) = ∏ i≤rf0(Xi)∏ i>rf1(Xi) ∏ i≤tf0(Xi)∏ i>tf1(Xi) =s ⋆(X)−1
B.2.2 Proof of Lemma 4.2: second conformal NP lemma In the setup of Section 4.1, we consider the following hypothesis testing problem: H′ 0 :X|X L,t,XR,t∼P [t] X|XL,t,XR,t v.s.H ′ 1 :X|X L,t,XR,t∼P [r] X|XL,t,XR,t . Given samplesX∈R n, observe that d ( P[r] X|XL,t,XR,t ) d ( P[t] X|XL,t,XR,t )(X)∝ d(P[r] X ) d(P[r] X ) = ∏ i≤rf0(Xi)∏ i>rf1(Xi) ∏ i≤tf0(Xi)...
2005
-
[3]
Sanjit Dandapanthula and Aaditya Ramdas. Offline changepoint localization using a matrix of conformal p-values.arXiv preprint arXiv:2505.00292,
-
[4]
Theorem A.1.For anyt∈[n],p t defined in(A.1)is a validp-value underH 0,t
Similar toCONCH, any randomly sampled π∈Π t preserves pre-change and post-change exchangeability underH 0,t, thereby providing us with a validp-value˜pt, as we state in Theorem A.1 and prove in Appendix B.1. Theorem A.1.For anyt∈[n],p t defined in(A.1)is a validp-value underH 0,t. In particular, for anyα∈(0,1),P ξ(˜pξ≤α)≤α. Consequently,C CONCH-MC 1−α is ...
2012
-
[5]
In particular, we partition the index set into two disjoint folds I1 :={t∈[n] :todd},andI 2 :={t∈[n] :teven}. 44 Algorithm 5:CONCH-SEG-crossfit Input:(X t)n t=1 (data);S: ⋃ m∈NXm→R m (CPP score function); segmentation algorithmSEG Output:C CONCH-SEG-crossfit 1−α 1I 1←{t≤n:todd},I 2←{t≤n:teven}; 2C←∅; 3forr∈{1,2}do 4( ˆK(r), ˆξ(r) 1 ,..., ˆξ(r) ˆK(r))←SEG ...
2018
-
[6]
Post-detection inference for sequential changepoint localiza- tion.arXiv preprint arXiv:2502.06096,
Aytijhya Saha and Aaditya Ramdas. Post-detection inference for sequential changepoint localiza- tion.arXiv preprint arXiv:2502.06096,
-
[14]
Figure 6 displays thep-value distributions from both methods
In the Gaussian mean-shift setting described in Section 8.1.1, we compare our framework against the changepoint localization method of Dandapanthula and Ramdas [2025], which also constructs distribution-free confidence sets for changepoints using a matrix of conformalp-values. Figure 6 displays thep-value distributions from both methods. Their approach yi...
2025
-
[15]
C.2 Gaussian mean-shift: localization of multiple changepoints We consider a multiple-changepoint Gaussian mean-shift model to illustrate the performance of CONCH-SEG (Algorithm 3)
andCONCHunder the Gaussian mean-shift model. C.2 Gaussian mean-shift: localization of multiple changepoints We consider a multiple-changepoint Gaussian mean-shift model to illustrate the performance of CONCH-SEG (Algorithm 3). In particular, we generaten= 1500observations with true change- points atξ1 = 150,ξ 2 = 500,ξ 3 = 820, andξ4 = 1100, segment means...
2018
Show all 17 references
-
[16]
1”, and the latter observations are i.i.d. samples of the digit “7
withM= 300 permutations to obtain confidence sets. Whenδis small, the pre-change and post-change distributions are nearly indistinguishable. 49 Consequently, no method can sharply localize the changepoint, includingCONCHconfidence sets. Asδincreases, the two distributions beco...
2012
-
[400]
Specifically, we construct a sequence ofn= 1,000 observations with a changepoint atξ= 400: the pre-change distributionP0,ξ consists of i.i.d
Threshold ( = 0.05) Figure 10: p-values for digit shift detection in MNIST: from digit ‘1’ to digit ‘7’ atξ= 400 C.5 CIFAR100: classifier strength affects power of CONCH We simulate a class-shift scenario using the CIFAR-100 image dataset [Krizhevsky et al., 2009] to evaluateC...
2009
-
[1968]
Theoretical guarantees for change localization using conformal p-values.arXiv preprint arXiv:2510.08749,
Swapnaneel Bhattacharyya and Aaditya Ramdas. Theoretical guarantees for change localization using conformal p-values.arXiv preprint arXiv:2510.08749,
-
[1999]
Conditional testing based on localized conformal p-values.arXiv preprint arXiv:2409.16829,
Xiaoyang Wu, Lin Lu, Zhaojun Wang, and Changliang Zou. Conditional testing based on localized conformal p-values.arXiv preprint arXiv:2409.16829,
-
[2012]
Consistent kernel change-point detection under m- dependence for text segmentation.arXiv preprint arXiv:2510.03437,
Jairo Diaz-Rodriguez and Mumin Jia. Consistent kernel change-point detection under m- dependence for text segmentation.arXiv preprint arXiv:2510.03437,
-
[2014]
Damien Garreau and Sylvain Arlot
ISSN 00905364. Damien Garreau and Sylvain Arlot. Consistent change-point detection with kernels.arXiv preprint arXiv:1612.04740,
-
[2022]
Theoretical foundations of conformal prediction.arXiv preprint arXiv:2411.11824,
Anastasios N Angelopoulos, Rina Foygel Barber, and Stephen Bates. Theoretical foundations of conformal prediction.arXiv preprint arXiv:2411.11824,
-
[2023]
21 Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts
ISSN 2693-7166. 21 Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. InProceedings of the 2013 Conference on Empirical Methods in Natu...
2013
-
[2025]
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108,
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108,
1910 arXiv
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.