Pith. sign in

REVIEW 2 major objections 6 minor 16 references

Validity and efficiency of the conformal CUSUM procedure

T0 review · 2 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Proposition 1 shows that the conformal CUSUM based on the likelihood-ratio quantile betting function has exactly the same pre-change distribution as the classical CUSUM, so classical false-alarm thresholds transfer while only…

desk verdict Proposition 1 is a genuine distributional identity and the paper is honest about scope; the efficiency part is thin, but the validity result alone justifies a serious referee. read the letter →

arxiv 2412.03464 v1 pith:EKEPBWQS submitted 2024-12-04 math.ST cs.LGstat.TH

Vladimir Vovk , Ilia Nouretdinov , Alex Gammerman This is my paper · ORCID
classification math.STcs.LGstat.TH MSC 62L1060G4262G10
keywords conformalCUSUMchangedetectiontestmartingalebettingfunctionlikelihood-ratioNeyman-PearsonROCcurveexchangeabilitysequentialanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that the conformal CUSUM procedure can match the classical CUSUM in validity and, under a soft model, approximate its efficiency, while assuming only exchangeability of pre-change observations rather than knowledge of the pre-change distribution. Its central result is that a conformal test martingale using the likelihood ratio as nonconformity measure and a betting function equal to the inverse survival function of the likelihood ratio under $Q_0$ has exactly the same pre-change distribution as the likelihood-ratio martingale. Consequently, false-alarm thresholds calibrated for the classical CUSUM transfer to the conformal procedure. The paper also derives closed-form betting functions for Gaussian mean and variance changes via the Neyman–Pearson ROC curve and proves a Bernoulli-case efficiency bound, with experiments illustrating the behavior.

What carries the argument

The load-bearing object is the canonical asymptotically optimal (CAO) betting function $f(p)$, defined as the left-continuous inverse of the survival function of the likelihood ratio $L$ under the pre-change law $Q_0$ (equivalently, the upper quantile function of $L$). It is the exact increment map that converts uniform conformal p-values into variables with the law of $L$ under $Q_0$. The argument rests on two identities: conformal p-values are independent and uniform under exchangeability, and the pushforward of uniform by the inverse survival function is the law of $L$. Proposition 2 adds that $f$ is the derivative of the Neyman–Pearson ROC curve, which is how the paper computes $f$ in closed form for $N(0,1)\to N(\mu,1)$ and $N(0,1)\to N(0,\sigma^2)$.

What would settle it

Simulate, say, 10,000 pre-change observations from $Q_0 = \text{Bernoulli}(0.5)$ with post-change $Q_1 = \text{Bernoulli}(0.6)$, compute conformal p-values with the likelihood-ratio nonconformity measure and the CAO betting function $f(p) = 1.2$ for $p \le 0.5$, $0.8$ otherwise (using random tie-breaking), and test whether the empirical distribution of $f(p_n)$ equals the $Q_0$-law of $L$ (1.2 with probability 0.5, 0.8 with probability 0.5) by a Kolmogorov–Smirnov test. A significant mismatch would refute Proposition 1.

Watch

Extended reading notes

Core claim

The central claim is Proposition 1: for any pair $(Q_0, Q_1)$ of pre- and post-change distributions, the distribution of the likelihood ratio martingale under $Q_0^\infty$ coincides with the distribution of any asymptotically optimal conformal test martingale (CTM) under randomness. An asymptotically optimal CTM is built by taking the likelihood ratio $L = f_1/f_0$ as the nonconformity measure and betting according to the inverse survival function of $L$ under $Q_0$. Because conformal p-values are independent and uniform under exchangeability, the betting function $f$ maps them into exactly the $Q_0$-law of $L$, so the conformal CUSUM's pre-change behavior is distributionally identical to the standard CUSUM's. Consequently, false-alarm thresholds calibrated for the classical CUSUM carry over to the conformal CUSUM without assuming $Q_0$ or $Q_1$ is known—only exchangeability of the pre-change data is required. The paper also proves that the CAO betting function is the derivative of the Neyman–Pearson ROC curve, giving closed forms for Gaussian location and scale changes, and establishes a Bernoulli-case efficiency bound: for post-change run lengths of order $\sqrt{N_0}$, the log ratio of the LRM to the CAO CTM stays bounded with high probability.

Load-bearing premise

The asymptotically optimal betting function is defined from the true pre- and post-change distributions (the likelihood ratio and its quantile under $Q_0$), so the full distributional match of Proposition 1 and the efficiency results require those distributions to be specified exactly.

Editorial extensions

If this is right

  • The threshold $c$ for the conformal CUSUM can be set using classical CUSUM approximations (e.g., Gaussian ARL formulas) instead of separate conformal simulations, because the pre-change law is identical by Proposition 1.
  • The same reasoning gives the conformal version of the Shiryaev–Roberts procedure the same perfect validity, as the paper notes.
  • Closed-form CAO betting functions make the efficient conformal CUSUM directly implementable for Gaussian location and scale changes.
  • Theorem 5 provides a quantitative guarantee in the Bernoulli case: for post-change run lengths $N_1$ of order $\sqrt{N_0}$, the log ratio of likelihood-ratio martingale to CAO conformal martingale is bounded with probability $1-\epsilon$, so the conformal alarm is not much slower after a genuine change.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ROC-derivative characterization suggests the same construction works for any parametric change pair with a known power function (e.g., Poisson or exponential), where the optimal conformal betting function is simply $R'$—a recipe the paper does not spell out for non-Gaussian cases.
  • Proposition 1's exact validity relies on randomized tie-breaking in conformal p-values; measuring how the non-randomized version degrades in discrete settings such as the Bernoulli 0.1 to 0.9 case would quantify the practical cost of dropping randomization.
  • A testable extension is to use the CAO betting function with plug-in estimates of $Q_0$ from a short pre-change sample; the paper's soft-model stance suggests validity should degrade gracefully, but no guarantee is proven.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper studies the conformal CUSUM change-detection procedure within the 'soft model' (Burnaev–Wasserman) programme. The authors define the canonical asymptotically optimal (CAO) betting function f as a quantile inverse of the survival function of the likelihood ratio L = f1/f0 under an assumed pre-change law Q0. Proposition 1 claims that the distribution of any asymptotically optimal conformal test martingale under mere IID randomness coincides with the distribution of the likelihood ratio martingale under Q0^∞; the proof uses the uniformity and independence of randomized conformal p-values together with the quantile-transform property of f. Section 4 derives closed-form CAO betting functions for Gaussian location and scale changes and connects f to the derivative of the Neyman–Pearson ROC curve (Proposition 2). Section 5 reports simulations (Bernoulli and Gaussian changes) showing that the CAO conformal martingale tracks the likelihood ratio martingale closely after the changepoint, with validity boxplots confirming the distributional match before it, and with one example (Fig. 7) where the conformal e-procedure violates validity. Section 6 states Theorem 5, a finite-sample Bernoulli-case bound on the log-ratio of the likelihood ratio martingale to the CAO conformal test martingale after a changepoint, valid when the post-change window is O(√N0), with the proof deferred to Appendix A and a martingale multiplicative Chernoff bound proved in Appendix B.

Significance. The central observation, Proposition 1, is clean, correct, and genuinely useful: it transfers the pre-change distributional behavior of the oracle likelihood-ratio CUSUM to a fully nonparametric conformal procedure, so that thresholds calibrated in classical CUSUM analysis can in principle be carried over under the soft model, while the conservative validity bound E(τ_k − τ_{k−1}) ≥ c holds under exchangeability alone. The proof is elementary and verifiable: it rests on two well-established facts (randomized conformal p-values are iid uniform; any quantile inverse of a survival function composed with a uniform variable has the target law). The explicit formulas (9), (13), (17), (19), and (20) are correct, and Proposition 2 (f equals the derivative of the ROC curve) is a nice interpretive result. The experiments are honest and instructive, particularly the demonstration that the conformal e-procedure can violate validity (Fig. 7) while the conformal test martingale does not.

major comments (2)
  1. [Appendix A (proof of Theorem 5), Eq. (24)] The application of the martingale multiplicative Chernoff bound (Theorem 6) at Eq. (24) is not justified as written. Theorem 6 requires a constant μ such that Σ_{i=1}^{N} θ_i ≤ μ almost surely, where θ_i = P(ξ_i = 1 | F_{i-1}), but the proof supplies only the expectation bound E[A] ≤ N1δ + N1(N1+1)/(2N0) on the number of anomalous steps. The conditional anomaly probabilities are not almost surely bounded by δ + n/N0 with the stated constant: for example, when the pre-change count takes its upper-deviation value K0 = θ0N0 + δN0, the anomaly probability at the first post-change step is about 2δ for the case z_1 = 1, exceeding δ + 1/N0 for large N0, and on atypical pasts the conditional probability can be of order θ1. The proof must be reworked, e.g., by first establishing a high-probability bound on Σ θ_i and then applying a two-event Chernoff argument, or by proving a genuine almost-sure bound on the conditional probabilities with correctly tracked constants. Because Theorem 5 is the paper's only theoretical efficiency result, this gap is load-bearing.
  2. [Section 3 (paragraph after Proposition 1); Section 7] The scope of the distributional identity in Proposition 1 should be stated more carefully. The result says that any asymptotically optimal conformal test martingale under an arbitrary IID law behaves like the likelihood ratio martingale under the assumed Q0^∞, and the betting function f is built from the assumed pair (Q0, Q1) through L. Consequently, the recommendation to use 'the whole arsenal of the existing results in the standard theory of change detection' for choosing the threshold c is exact only when the assumed pair equals the true pre- and post-change distributions. Under misspecification, the pre-change behavior of the conformal CUSUM matches the oracle for the assumed (Q0, Q1), not for the true pair, and only the conservative bound (2) is guaranteed. The authors do flag the soft model at the outset, but the unqualified phrases 'perfectly valid' (conclusion) and the threshold-transfer claim are easy to over-read; an explicit caveat at both locations would make the practical reach of the main claim accurate.
minor comments (6)
  1. [Section 4, Eq. (12)] The symbol N is used for the standard normal distribution function without being defined, after Φ had been used for the same object; please use one symbol consistently throughout.
  2. [Section 3, Eq. (6)] The right-hand side 'qk · · ·+ qK' is missing the plus signs among the terms; it should read q_k + ··· + q_K.
  3. [Appendix B, proof of Theorem 6] The phrase 'assuming (ξn) is a martingale' is imprecise; the needed hypothesis is that E(ξn | F_{n−1}) = θn, not that the difference sequence itself is a martingale.
  4. [Introduction, first paragraph] The phrase 'Lorden's [7, 8] worst-case definition' cites both Lorden [7] and Moustakides [8] in a way that blurs which result is whose; consider writing 'Lorden's worst-case criterion [7], for which CUSUM is optimal [8]'.
  5. [Section 5, captions of Figures 5 and 6] The stray repeated text 'log LRM log CAO CTM log CeP' in the captions should be removed or formatted as a proper legend description.
  6. [Section 1, last paragraph] The sentence 'In this paper we will refer to the assumption that the observations are IID as the assumption of randomness' is grammatically tangled; a plainer phrasing such as 'we refer to the assumption that the observations are IID as the randomness assumption' would be clearer, and 'equivalent to that of exchangeability' should be completed, e.g., 'equivalent to exchangeability for observations taking values in a standard Borel space'.

Circularity Check

1 steps flagged · score 6.0 of 10

Proposition 1's distributional match is engineered via the quantile-transform definition of the betting function; efficiency results remain non-circular.

  1. self definitional [Section 3, definition of asymptotically optimal betting function and proof of Proposition 1]
    "An asymptotically optimal betting function f : [0,1] → [0,∞) for the pair (Q0,Q1) ... is defined by the condition Q0({z : L(z) > f(p)}) ≤ p ≤ Q0({z : L(z) ≥ f(p)}) ... In other words, it is defined to be an inverse function of the survival function F¯(t) := Q0({z : L(z) > t}) of the likelihood ratio L under Q0. ... The latter’s distribution is identical by definition and [2, Lemma A.23]."

    The betting function f is defined, via the inverse survival function, as the quantile transform that turns a uniform random variable into a variable with the Q0-distribution of L. Proposition 1's conclusion — that an asymptotically optimal CTM has increments distributed like the LRM under Q0 — is exactly this defining property once conformal p-values are known to be uniform. The proof even says 'identical by definition'. Thus the 'perfect validity' match with the oracle CUSUM is built into the construction rather than derived from independent first principles.

full rationale

The paper is largely self-contained against external benchmarks: Theorem 5 is a genuine finite-sample deviation bound, the conservative validity inequality (2) is standard conformal-martingale theory, and no parameter is fitted to data. The one definitional element is Proposition 1: because the CAO betting function is deliberately chosen as the inverse survival function of L under Q0, the statement that the CTM's distribution equals the LRM's distribution under Q0 is true by construction, modulo the uniform-p-value lemma. This reduces the force of the 'perfect validity' claim as an independent discovery, but it does not undermine the efficiency results or the practical validity guarantee. Since the central validity claim itself reduces to its own definition, the circularity score is moderate rather than zero.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No hidden free parameters are fitted to make the central theorem work; the mu, sigma, and theta_1 values are explicit soft-model inputs. The main construction deliberately mirrors the likelihood ratio, so the coincidence in Proposition 1 is partly by design. The finite-sample Bernoulli bound is the main non-tautological efficiency content.

free parameters (3)
  • Post-change mean mu for location-change CAO betting function = None (soft-model input); experiment uses 0.2
    Formula (9) or (17) defines f(p) using mu; the procedure must commit to a post-change distribution. No fitting to data occurs; misspecification degrades efficiency.
  • Post-change scale sigma for scale-change CAO betting function = None (soft-model input); experiments use 1.1 and 0.9
    Formulas (13), (14), (19), and (20) depend on sigma; chosen by the user as a soft-model input, not fitted to observations.
  • Bernoulli post-change parameter theta_1 = None (soft-model input); experiment uses 0.6
    The betting function in Theorem 5 and Figure 1 uses theta_1; the pre-change theta_0 is known from Q0. It is an assumed parameter of the soft model.
assumptions (4)
  • domain assumption Pre-change observations are IID (randomness), used for conformal validity.
    Invoked in Section 2 and via [15, Theorem 11.1] for p-value uniformity; required for Proposition 1 validity.
  • domain assumption The soft model specifies exact pre-change Q0 and post-change Q1 with positive densities, so the likelihood ratio L is well defined.
    Section 2 defines L=f1/f0; Section 3 assumes positive f0,f1; all CAO betting functions require this.
  • standard math Conformal p-values are independent and uniform under exchangeability ([15, Theorem 11.1]).
    Used in Proposition 1 to equate increment distributions; taken as a theorem from prior work.
  • ad hoc to paper The inverse-survival-function definition of f is the right optimality criterion.
    The CAO betting function is constructed so that f(U) has the distribution of L under Q0; this embeds the target LRM into the construction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Validity and efficiency of the conformal CUSUM procedure." pith.science (2026). https://pith.science/paper/EKEPBWQS

@misc{pith2026241203464,
  author       = {Pith},
  title        = {Pith review of: Validity and efficiency of the conformal CUSUM procedure},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EKEPBWQS}},
  note         = {Machine review of arXiv:2412.03464}
}
read the original abstract

In this paper we study the validity and efficiency of a conformal version of the CUSUM procedure for change detection both experimentally and theoretically.

Figures

Figures reproduced from arXiv: 2412.03464 by the authors.

Figure 1
Figure 1. Left panel: the likelihood ratio martingale, conformal e-pseudomartin [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Left panel: the three processes as in Fig. 1, but with a change from [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Left panel: the three processes as in Fig. 1, but with a change from [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Left panel: the three processes as in Fig. 1, but with a change from [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Left panel: the validity for the Bernoulli case. Right panel: the [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Left panel: the validity for the Gaussian case with an upward change [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Validity and lack thereof for a Bernoulli case, as described in text [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 15 canonical work pages

  1. [1]

    Nikiforov

    Mich` ele Basseville and Igor V. Nikiforov. Detection of Abrupt Changes: Theory and Application . Prentice-Hall, Englewood Cliffs, NJ, 1993

    work page 1993

  2. [2]

    Stochastic Finance: An Introduction in Discrete Time

    Hans F¨ ollmer and Alexander Schied. Stochastic Finance: An Introduction in Discrete Time . De Gruyter, Berlin, fourth edition, 2016

    work page 2016

  3. [3]

    Goel and S

    Amrit L. Goel and S. M. Wu. Determination of A. R. L. and a con- tour nomogram for Cusum charts to control normal mean. Technometrics, 13:221–230, 1971

    work page 1971

  4. [4]

    Chernoff bounds, and some applications: Lecture notes

    Michel Goemans. Chernoff bounds, and some applications: Lecture notes. Principles of Discrete Applied Mathematics, Massachusetts Institute of Technology, February 2015

    work page 2015

  5. [5]

    The multiplicative version of Azuma’s in- equality, with an application to contention analysis

    William Kuszmaul and Qi Qi. The multiplicative version of Azuma’s in- equality, with an application to contention analysis. Technical Report arXiv:2102.05077 [cs.DS], arXiv.org e-Print archive, February 2021

    arXiv 2021

  6. [6]

    Lehmann and Joseph P

    Erich L. Lehmann and Joseph P. Romano. Testing Statistical Hypotheses. Springer, Cham, fourth edition, 2022

    work page 2022

  7. [7]

    Procedures for reacting to a change in distribution

    Gary Lorden. Procedures for reacting to a change in distribution. Annals of Mathematical Statistics , 42:1897–1908, 1971

    work page 1908

  8. [8]

    Moustakides

    George V. Moustakides. Optimal stopping times for detecting a change in distribution. Annals of Statistics , 14:1379–1388, 1986. 15

    work page 1986

Show all 16 references
  1. [9]

    Ewan S. Page. Continuous inspection schemes. Biometrika, 41:100–115, 1954

    1954

  2. [10]

    Vincent Poor and Olympia Hadjiliadis

    H. Vincent Poor and Olympia Hadjiliadis. Quickest Detection. Cambridge University Press, Cambridge, 2009

    2009

  3. [11]

    Decision theoretic optimality of the CUSUM procedure

    Ya’acov Ritov. Decision theoretic optimality of the CUSUM procedure. Annals of Statistics , 18:1464–1469, 1990

    1990

  4. [12]

    Game-Theoretic Foundations for Proba- bility and Finance

    Glenn Shafer and Vladimir Vovk. Game-Theoretic Foundations for Proba- bility and Finance . Wiley, Hoboken, NJ, 2019

    2019

  5. [13]

    CRC Press, Boca Raton, FL, 2015

    Alexander Tartakovsky, Igor Nikiforov, and Mich` ele Basseville.Sequential Analysis: Hypothesis Testing and Changepoint Detection. CRC Press, Boca Raton, FL, 2015

    2015

  6. [14]

    Inductive conformal martingales for change-point detection

    Denis Volkhonskiy, Evgeny Burnaev, Ilia Nouretdinov, Alex Gammerman, and Vladimir Vovk. Inductive conformal martingales for change-point detection. Proceedings of Machine Learning Research , 60:132–153, 2017. COPA 2017

    2017

  7. [15]

    Algorithmic Learning in a Random World

    Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World . Springer, Cham, second edition, 2022

    2022

  8. [16]

    ≤” in (29) follows from the conditionPN i=1 θi ≤ µ a.s. The first inequality “ ≤

    Vladimir Vovk, Ilia Nouretdinov, and Alex Gammerman. Conformal e-test- ing. Technical Report arXiv:2006.02329v2 [math.ST], arXiv.org e-Print archive, November 2024. A Proof of Theorem 5 To compare conformal test martingales with likelihood ratio martingales, we will use the ma...

    2006 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.