Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that weighted-conformal p-values are IID uniform under a null that allows modeled covariate shift, so test martingales built from them can adapt to benign shifts while keeping anytime-valid alarm control.

desk verdict Solid method paper with a real anytime-validity gap between the clean martingale theory and the implemented adaptive procedure; worth a serious referee but needs a joint treatment of the trigger. read the letter →

arxiv 2505.04608 v4 pith:RCRPA4VY submitted 2025-05-07 cs.LG cs.AIstat.ML

Drew Prinster , Xing Han , Anqi Liu , Suchi Saria This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords weightedconformalpredictiontestmartingalesanytime-validinferenceonlinechangepointdetectiondistributionshiftmonitoringcovariateadaptationconceptAIdeployment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to give deployed machine-learning systems a monitor that raises an alarm only when a distribution shift is genuinely harmful, while silently adapting to benign shifts. It generalizes conformal test martingales by weighting each calibration score with a likelihood ratio, and proves that the resulting online p-values are IID uniform under a null that allows modeled covariate shift. Consequently the weighted martingale inherits anytime-valid false-alarm control, and the practical WATCH system detects concept shifts or extreme covariate shifts and tells them apart. If correct, this removes a central limitation of prior conformal monitoring: alarms no longer fire for every deviation from exchangeability.

What carries the argument

The central object is the weighted-conformal p-value, a sum over calibration scores with likelihood-based weights $\tilde{w}_i$ replacing the uniform $1/(n+1)$ weights of a standard conformal p-value. The key identity is Theorem 3.1: when the weights equal the oracle weights under the null, the online p-value sequence is IID uniform on $[0,1]$. Feeding that sequence into a product of betting functions yields a nonnegative martingale, and a parallel input-only conformal test martingale triggers adaptation and helps diagnose whether an alarm is a concept shift or an extreme covariate shift.

What would settle it

Repeatedly simulate a long stream of data from a known discrete distribution, compute oracle weighted-conformal p-values exactly using the true joint probabilities of Eq. (10), and feed them through a fixed betting strategy; if the empirical p-value distribution deviates from IID Uniform[0,1], or if the fraction of runs whose martingale crosses threshold $c=100$ exceeds $1/100$, the paper's central theorem and anytime-valid guarantee are false, since this simulation has no weight-estimation error to mask the issue.

Watch

Extended reading notes

Core claim

Under a null hypothesis that either exchangeability holds or the covariate distribution shifts exactly as modeled by the estimated density-ratio weights while the label distribution stays invariant, the sequence of online weighted-conformal p-values is independent and uniform on $[0,1]$. Therefore a test martingale constructed by betting on those p-values is a nonnegative martingale under that null, and Ville's inequality gives $P(\text{alarm ever raised}) \le 1/c$ for any threshold $c$. This shifts the monitored null from 'the data distribution is identical over time' to 'the data distribution changes only in the anticipated way,' so a monitor can adapt to benign covariate shifts without sacrificing false-alarm control.

Load-bearing premise

The false-alarm guarantee holds only when the estimated density-ratio weights exactly match the true covariate shift and the calibration set is large enough that its score distribution exactly matches the source distribution, with the adaptation trigger treated as fixed rather than data-dependent.

Editorial extensions

If this is right

  • A deployment monitor can adapt its conformal prediction sets to a mild covariate shift and keep target coverage without alarming, because the weighted p-values remain uniform under the modeled shift.
  • Any violation of the modeled null, such as a concept shift or an unmodeled extreme covariate shift, will eventually inflate the weighted martingale, with probability of ever crossing threshold $c$ bounded by $1/c$.
  • The same construction extends to any null hypothesis the user can express through oracle weights, not just covariate shift, as stated by the general Theorem 3.1.
  • Running a Shiryaev-Roberts procedure on the weighted martingale gives average run length at least $c$, providing a stagewise alternative to anytime-valid alarms.
  • Running a standard input-only X-CTM alongside the weighted martingale enables root-cause diagnosis: both alarm indicates extreme covariate shift, only the weighted martingale alarms indicates concept shift, and neither alarms while the X-CTM is active indicates a successfully adapted benign shift.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not quantify how the false-alarm guarantee degrades when density-ratio estimates are only approximately correct; a natural extension is a robustness theorem bounding alarm inflation as a function of weight-estimation error.
  • Because the X-CTM runs on inputs alone, the same machinery could provide a label-free early-warning layer that flags when the input stream has left the region where the estimated weights are trustworthy, before any labels arrive.
  • One could close the monitoring loop by triggering retraining at the alarm and reinitializing the martingale with the updated model; this paper stops at detection and diagnosis.
  • The alarm threshold $c$ could be tuned against deployment-specific costs of false versus missed alarms, treating the martingale value as a continuous evidence scale rather than a binary trigger.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces weighted-conformal test martingales (WCTMs), built from sequences of online weighted-conformal p-values, as a generalization of conformal test martingales for testing nonparametric null hypotheses beyond exchangeability. The main theoretical result (Theorem 3.1) states that online weighted-conformal p-values are IID uniform under a null hypothesis H0(f-hat) that includes the accuracy of the user's modeling assumptions; Proposition 3.2 then gives anytime-valid false-alarm control via Ville's inequality, and Proposition 3.3 gives average-run-length control for a Shiryaev-Roberts variant. The proposed WATCH framework uses a secondary X-CTM to trigger adaptation to mild covariate shifts via online density-ratio estimation, raises alarms for extreme covariate shifts or concept shifts, and diagnoses the root cause. Experiments on tabular and image datasets compare WCTMs against standard CTMs and Podkopaev-Ramdas baselines in terms of detection delay, false alarms, coverage, and runtime.

Significance. If the guarantees hold as stated, the paper makes a substantial contribution to sequential nonparametric testing and AI monitoring: it extends conformal test martingales to composite nulls that accommodate modeled shifts, it provides a principled betting-based construction that adapts online, and it offers root-cause diagnostics. The paper is also commendable for shipping code, for running extensive experiments on multiple real datasets, and for giving a self-contained proof of the oracle-weight validity result. The appendix is unusually candid in acknowledging the finite-calibration idealization and the non-independence of the practical p-values; however, these acknowledgments reveal that the headline false-alarm guarantee for the implemented adaptive procedure is not actually established by the theorems.

major comments (3)
  1. [Section 3.6 and Proposition 3.2] The anytime-valid false-alarm guarantee is not transferred to the adaptive WATCH procedure. Proposition 3.2 assumes a WCTM built from weighted-conformal p-values that are IID uniform under a fixed null H0(f-hat), but the implementation in Section 3.6 makes the adaptation time tad a data-dependent stopping time: the secondary X-CTM triggers adaptation the first time its wealth crosses the adaptation threshold. The null H0(cs) in Eq. (23) and the modified null in Eq. (28) both treat tad as fixed, so they do not cover the event that the X-CTM falsely triggers under the no-harmful-shift null. Once a false trigger occurs, the main WCTM switches from uniform weights to estimated density-ratio weights bw(t)(x) from Eq. (17), and the resulting p-values need not be uniform. No joint anytime-valid bound over the trigger and the subsequent WCTM is supplied, so the claim that WATCH controls false alarms is not established for the implemented adaptive procedure. Please either prove a joint guarantee over the trigger and the WCTM, or state the conditional nature of the guarantee precisely and provide null simulations at the advertised 1/c rate.
  2. [Appendix B.1, Lemma B.1] The proof of Lemma B.1 assumes that the nonconformity score function is bijective, stating that 'because the score function bS is bijective, this further implies that Vt = vσ(t) = vi'. This is false for the scores used in the experiments, such as absolute residuals |y - bµ(x)| and one-minus-softmax scores, which map many data points to the same score value. When the score function is non-injective, the event {Vt = vi} does not identify a single data index, and the oracle weights in Eq. (19) do not equal the probabilities of the observed scores under the permutation distribution. The random tie-breaking with un+1 does not repair the proof as written because the argument needs injectivity to condition on the test score. Please provide a version of Lemma B.1 that handles non-injective score functions, for example by defining oracle weights on score atoms or via a randomized score-injection argument, or state exact validity only for injective scores and give a separate treatment of ties.
  3. [Appendix C, Eqs. (26)-(28)] The practical WCTM's false-alarm guarantee rests on the idealization bF_{[n+tad-1]}^V = F_V, i.e., that the empirical CDF of the calibration scores equals the true source score distribution, which the paper itself notes is 'of course true in the limit n → ∞'. For finite n this equality fails, so the implemented method does not achieve exact anytime-valid false-alarm control under H0(cs); it achieves control only in an idealized asymptotic regime. In addition, the permutation-restriction approximation in Eq. (26) means the p-values in Eq. (27) are not independent across times, as the paper acknowledges in the paragraph preceding Eq. (28), so Theorem 3.1 does not directly apply to the sequence actually used in WATCH. Please either modify the method to restore the requisite independence and prove the claimed guarantees for the modified method, or explicitly label the headline false-alarm statements as approximations that are empirically validated rather than as consequences of Theorem 3.1.
minor comments (5)
  1. [Section 5] The text contains several typos: 'deplyed' should be 'deployed', and in Section 3.2 'eactly accurate' should be 'exactly accurate'.
  2. [Figure 2 caption] The caption contains typos: 'ans Shiryaev-Roberts paths' should be 'and Shiryaev-Roberts paths', and 'simuluate' should be 'simulate'.
  3. [Section 3.6, Eq. (17)] The notation 'bw(t(Xi)' appears to be missing a closing parenthesis; it should presumably read 'bw(t)(Xi)'.
  4. [Figure 4] The legend item 'X-CTM (Adapt Criterion)' is unclear; please specify that the X-CTM's wealth crossing the adaptation threshold triggers the adaptation phase.
  5. [Section 4.3] The paper does not report the actual alarm threshold c used for the WCTM and SR-WCTM methods, nor the adaptation threshold for the X-CTM; please report these values so the anytime-valid claims can be checked against the stated rates.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: WCTM false-alarm control follows from Ville's inequality once IID-uniform p-values are established; the adaptive-trigger and finite-calibration gaps are scope limitations, not construction-equivalence.

full rationale

The paper's central derivation is not circular. Theorem 3.1 (Eq. 12) is proven self-contained in Appendix B (Lemma B.1 and Eqs. 18-22) by a time-reversal/permutation argument generalizing Vovk et al. (2003) and Tibshirani et al. (2019); it shows that weighted-conformal p-values are IID uniform under a null H0(fhat) that includes exact accuracy of the estimated oracle weights. Proposition 3.2 (Eq. 14) then follows by Ville's inequality, with the betting strategy depending only on past p-values. No fitted parameter is used to force the p-values to look uniform, and the empirical p-value histograms are checks of whether the null holds, not inputs to the guarantee. The self-citations to Prinster et al. (2024) are for exposition of oracle weights (Eq. 10) and are not load-bearing: the underlying analysis is attributed to Tibshirani et al. (2019), and the theorem's proof is given in the appendix rather than imported. No uniqueness theorem from the authors is invoked to forbid alternative constructions. The limitations flagged by the skeptic are real scope gaps, not circularity. Appendix C explicitly concedes that Theorem 3.1 applies directly only at t = tad for the non-resampled procedure and patches the gap by strengthening the null to H0(cs) AND bF = FV (Eq. 28). The data-dependent trigger time tad (Sec. 3.6) is outside this fixed-tad null, so the anytime-valid guarantee is not established for the fully adaptive procedure; this is an unproven-validity issue, not an equivalence-by-construction. The finite-calibration idealization and the omitted full pseudocode (Appendix F) are additional reproducibility and validity caveats, but they do not make the derivation circular.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claim rests on standard martingale theory plus three strong domain assumptions: exact density-ratio estimates, an effectively infinite calibration set, and a fixed adaptation time. These are realistic only asymptotically or in ideal settings, so they are the main source of fragility.

free parameters (3)
  • Online density-ratio estimator bw(t)(x) = 3-layer MLP trained online to distinguish source vs target data
    Used to compute weights in Eq. (17); Theorem 3.1 validity requires bw(t) to be exactly the true density ratio under the null (Eq. 13/23).
  • X-CTM adaptation threshold = not specified in text
    Determines when adaptation begins (tad); the theory treats tad as fixed while the algorithm selects it online (Sec. 3.6, App. C).
  • Composite Jumper betting parameters = J in {1e-4, 1e-3, 1e-2, 1e-1, 1}
    Standard betting strategy from Vovk et al. 2022; affects detection speed but not validity.
assumptions (6)
  • standard math Ville's inequality for nonnegative martingales
    Used for anytime-valid false alarm control in Prop 3.2.
  • domain assumption The joint density fZ exists (Radon-Nikodym derivative) and the oracle weights in Eq. (10) are well-defined
    Needed for the proof of Theorem 3.1 (Appendix B).
  • domain assumption Covariate shift factorization: Y|X invariant, X marginal shifts with density ratio bw
    Defines the null H0(cs) in Eq. (13) and the factorization in Eq. (24).
  • ad hoc to paper The per-point nonconformity score function is bijective
    Stated in Lemma B.1 proof; false for typical scores like absolute residual, and the proof's handling of ties is not fully detailed.
  • ad hoc to paper Calibration set is effectively infinite, bF_V = F_V
    Added in Eq. (28) to restore independence for practical WCTM p-values computed after tad.
  • ad hoc to paper The adaptation time tad is fixed in advance or not data-dependent
    The X-CTM trigger makes tad random; no theory covers this selection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales." pith.science (2026). https://pith.science/paper/RCRPA4VY

@misc{pith2026250504608,
  author       = {Pith},
  title        = {Pith review of: WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RCRPA4VY}},
  note         = {Machine review of arXiv:2505.04608}
}
read the original abstract

Responsibly deploying artificial intelligence (AI) / machine learning (ML) systems in high-stakes settings arguably requires not only proof of system reliability, but also continual, post-deployment monitoring to quickly detect and address any unsafe behavior. Methods for nonparametric sequential testing -- especially conformal test martingales (CTMs) and anytime-valid inference -- offer promising tools for this monitoring task. However, existing approaches are restricted to monitoring limited hypothesis classes or ``alarm criteria'' (e.g., detecting data shifts that violate certain exchangeability or IID assumptions), do not allow for online adaptation in response to shifts, and/or cannot diagnose the cause of degradation or alarm. In this paper, we address these limitations by proposing a weighted generalization of conformal test martingales (WCTMs), which lay a theoretical foundation for online monitoring for any unexpected changepoints in the data distribution while controlling false-alarms. For practical applications, we propose specific WCTM algorithms that adapt online to mild covariate shifts (in the marginal input distribution), quickly detect harmful shifts, and diagnose those harmful shifts as concept shifts (in the conditional label distribution) or extreme (out-of-support) covariate shifts that cannot be easily adapted to. On real-world datasets, we demonstrate improved performance relative to state-of-the-art baselines.

Figures

Figures reproduced from arXiv: 2505.04608 by the authors.

Figure 1
Figure 1. Each column represents a data shift scenario: the top row is a simulated shift example and the bottom row shows WATCH’s response, averaged over 20 random seeds. WATCH raises an alarm to retrain the AI/ML once the WCTM (blue) exceeds its alarm threshold; meanwhile, an X-CTM (gray)—a standard CTM that only depends on inputs X, and thus only detects covariate shifts—dynamically initiates the WCTM’s adaptation phase and… view at source ↗
Figure 2
Figure 2. Tabular data results for the benign covariate shift setting to evaluate the adaptation ability of proposed WCTM methods (blue); all values are averaged over 200 random seeds. Training and calibration sets were sampled uniformly at random (with 1/3 of the total data used for training and calibration each), while post-changepoint test-set datapoints were bias-sampled from the remaining holdout data with probability pr… view at source ↗
Figure 3
Figure 3. Example martingale trajectories of WCTM and CTM on CIFAR-10-C with increasing levels of corruption. WCTM adapt to milder shifts to avoid unnecessary alarms while still detecting severe shifts. All plots are averaged from 5 random experiments [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Results for root-cause analysis with a WCTM (blue) and a secondary [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Ablation study illustrating synthetic data example of WATCH performance for different magnitudes of covariate [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: Ablation study on density-ratio estimator for synthetic data. [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: Ablation study on betting function [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 8
Figure 8. Figure 8: Results on CIFAR-10 with various corruption types, all at the highest severity level. WCTM reacts more quickly [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: The results supplement Figure [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CUSUM-Shaped Inference-Time Monitoring and Targeted Re-Decoding for Quantized Small Language Model Reasoning

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A monitoring-guided rollback controller produced a small and statistically uncertain accuracy gain on first-seen MATH-500 problems, and a larger exploratory gain on a historical-coverage set contaminated by manual thr...

Reference graph

Works this paper leans on

86 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    E., Sridharan, A., Soleimani, H., Zhan, A., Rawat, N., Johnson, L., Hager, D

    Adams, R., Henry, K. E., Sridharan, A., Soleimani, H., Zhan, A., Rawat, N., Johnson, L., Hager, D. N., Cosgrove, S. E., Markowski, A., et al. (2022). Prospective, multi-site study of patient outcomes after implementation of the trews machine learning-based early warning system for sepsis. Nature medicine , 28(7):1455--1460

    2022

  3. [3]

    Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Man \'e , D. (2016). Concrete problems in ai safety. arXiv preprint arXiv:1606.06565

    arXiv 2016

  4. [4]

    N., Barber, R

    Angelopoulos, A. N., Barber, R. F., and Bates, S. (2024). Theoretical foundations of conformal prediction. arXiv preprint arXiv:2411.11824

    arXiv 2024

  5. [5]

    Bar, Y., Shaer, S., and Romano, Y. (2024). Protected test-time adaptation via online entropy matching: A betting approach. arXiv preprint arXiv:2408.07511

    work page Pith review arXiv 2024

  6. [6]

    F., Candes, E

    Barber, R. F., Candes, E. J., Ramdas, A., and Tibshirani, R. J. (2023). Conformal prediction beyond exchangeability. The Annals of Statistics , 51(2):816--845

    2023

  7. [7]

    Barber, R. F. and Tibshirani, R. J. (2025). Unifying different theories of conformal prediction. arXiv preprint arXiv:2504.02292

    arXiv 2025

  8. [8]

    Bashari, M., Epstein, A., Romano, Y., and Sesia, M. (2023). Derandomized novelty detection with fdr control via conformal e-values. Advances in Neural Information Processing Systems , 36:65585--65596

    2023

Show all 86 references
  1. [9]

    Bates, S., Cand \`e s, E., Lei, L., Romano, Y., and Sesia, M. (2023). Testing for outliers with conformal p-values. The Annals of Statistics , 51(1):149--178

    2023

  2. [10]

    B., Wang, Y., Barber, R

    Berrett, T. B., Wang, Y., Barber, R. F., and Samworth, R. J. (2020). The conditional permutation test for independence while controlling for confounders. Journal of the Royal Statistical Society Series B: Statistical Methodology , 82(1):175--197

    2020

  3. [11]

    W., Cohen, S

    Cohen, J. W., Cohen, S. B., and Banthin, J. S. (2009). The medical expenditure panel survey: a national information resource to support healthcare cost research and inform policy and practice. Medical care , 47(7\_Supplement\_1):S44--S50

    2009

  4. [12]

    and Papadopoulos, H

    Eliades, C. and Papadopoulos, H. (2022). A betting function for addressing concept drift with conformal martingales. In Conformal and Probabilistic Prediction with Applications , pages 219--238. PMLR

    2022

  5. [13]

    and Papadopoulos, H

    Eliades, C. and Papadopoulos, H. (2023). A conformal martingales ensemble approach for addressing concept drift. In Conformal and Probabilistic Prediction with Applications , pages 328--346. PMLR

    2023

  6. [14]

    Fanaee-T, H. (2013). Bike Sharing . UCI Machine Learning Repository. DOI : https://doi.org/10.24432/C5W894

    2013 doi

  7. [15]

    N., Listgarten, J., and Jordan, M

    Fannjiang, C., Bates, S., Angelopoulos, A. N., Listgarten, J., and Jordan, M. I. (2022). Conformal prediction under feedback covariate shift for biomolecular design. Proceedings of the National Academy of Sciences , 119(43):e2204569119

    2022

  8. [16]

    Farinhas, A., Zerva, C., Ulmer, D., and Martins, A. F. (2023). Non-exchangeable conformal risk control. arXiv preprint arXiv:2310.01262

    2023 arXiv

  9. [17]

    Fedorova, V., Gammerman, A., Nouretdinov, I., and Vovk, V. (2012). Plug-in martingales for testing exchangeability on-line. arXiv preprint arXiv:1204.3251

    2012 arXiv

  10. [18]

    and Romano, Y

    Feldman, S. and Romano, Y. (2024). Robust conformal prediction using privileged information. arXiv preprint arXiv:2406.05405

    2024 arXiv

  11. [19]

    V., Malenica, I., Bishara, A., Hubbard, A

    Feng, J., Phillips, R. V., Malenica, I., Bishara, A., Hubbard, A. E., Celi, L. A., and Pirracchio, R. (2022). Clinical artificial intelligence quality improvement: towards continual monitoring and updating of ai algorithms in healthcare. NPJ digital medicine , 5(1):66

    2022

  12. [20]

    Feng, J., Xia, F., Singh, K., and Pirracchio, R. (2025). Not all clinical ai monitoring systems are created equal: Review and recommendations. NEJM AI , 2(2):AIra2400657

    2025

  13. [21]

    G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I

    Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., and Saria, S. (2021). The clinician and dataset shift in artificial intelligence. New England Journal of Medicine , 385(3):283--286

    2021

  14. [22]

    and Ramdas, A

    Fischer, L. and Ramdas, A. (2025). Sequential monte carlo testing by betting. Journal of the Royal Statistical Society Series B: Statistical Methodology , page qkaf014

    2025

  15. [23]

    J., Ramdas, A., and Tibshirani, R

    Foygel Barber, R., Candes, E. J., Ramdas, A., and Tibshirani, R. J. (2021). The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA , 10(2):455--482

    2021

  16. [24]

    Gauthier, E., Bach, F., and Jordan, M. I. (2025). E-values expand the scope of conformal prediction. arXiv preprint arXiv:2503.13050

    2025 arXiv

  17. [25]

    Hamidieh, K. (2018). Superconductivty Data . UCI Machine Learning Repository. DOI : https://doi.org/10.24432/C53P47

    2018 doi

  18. [26]

    He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770--778

    2016

  19. [27]

    and Dietterich, T

    Hendrycks, D. and Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations

    2019

  20. [28]

    Hindy, A., Luo, R., Banerjee, S., Kuck, J., Schmerling, E., and Pavone, M. (2024). Diagnostic runtime monitoring with martingales. arXiv preprint arXiv:2407.21748

    2024 arXiv

  21. [29]

    Ho, S.-S. (2005). A martingale framework for concept change detection in time-varying data streams. In Proceedings of the 22nd international conference on Machine learning , pages 321--327

    2005

  22. [30]

    Ho, S.-S., Schofield, M., Sun, B., Snouffer, J., and Kirschner, J. (2019). A martingale-based approach for flight behavior anomaly detection. In 2019 20th IEEE International Conference on Mobile Data Management (MDM) , pages 43--52. IEEE

    2019

  23. [31]

    I Amoukou, S., Bewley, T., Mishra, S., Lecue, F., Magazzeni, D., and Veloso, M. (2024). Sequential harmful shift detection without labels. Advances in Neural Information Processing Systems , 37:129279--129302

    2024

  24. [32]

    and Cand \`e s, E

    Jin, Y. and Cand \`e s, E. J. (2023). Model-free selective inference under covariate shift via weighted conformal p-values. arXiv preprint arXiv:2307.09291

    2023 arXiv

  25. [33]

    and Ren, Z

    Lee, Y. and Ren, Z. (2025). Selection from hierarchical data with conformal e-values. arXiv preprint arXiv:2501.02514

    2025 arXiv

  26. [34]

    and Gilmer, J

    Mu, N. and Gilmer, J. (2019). Mnist-c: A robustness benchmark for computer vision. arXiv preprint arXiv:1906.02337

    2019 arXiv

  27. [35]

    and Janson, L

    Nair, Y. and Janson, L. (2023). Randomization tests for adaptively collected data. arXiv preprint arXiv:2301.05365

    2023 arXiv

  28. [36]

    Page, E. S. (1954). Continuous inspection schemes. Biometrika , 41(1/2):100--115

    1954

  29. [37]

    Papadopoulos, H. (2008). Inductive conformal prediction: Theory and application to neural networks. In Tools in artificial intelligence . Citeseer

    2008

  30. [38]

    Papadopoulos, H., Proedrou, K., Vovk, V., and Gammerman, A. (2002). Inductive confidence machines for regression. In Machine learning: ECML 2002: 13th European conference on machine learning Helsinki, Finland, August 19--23, 2002 proceedings 13 , pages 345--356. Springer

    2002

  31. [39]

    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011). Scikit-learn: Machine learning in python. the Journal of machine Learning research , 12:2825--2830

    2011

  32. [40]

    and Ramdas, A

    Podkopaev, A. and Ramdas, A. (2021a). Distribution-free uncertainty quantification for classification under label shift. In Uncertainty in artificial intelligence , pages 844--853. PMLR

    2021

  33. [41]

    and Ramdas, A

    Podkopaev, A. and Ramdas, A. (2021b). Tracking the risk of a deployed model and detecting harmful distribution shifts. arXiv preprint arXiv:2110.06177

    2021 arXiv

  34. [42]

    and Ramdas, A

    Podkopaev, A. and Ramdas, A. (2023). Sequential predictive two-sample and independence testing. Advances in neural information processing systems , 36:53275--53307

    2023

  35. [43]

    Prinster, D., Liu, A., and Saria, S. (2022). Jaws: Auditing predictive uncertainty under covariate shift. Advances in Neural Information Processing Systems , 35:35907--35920

    2022

  36. [44]

    Prinster, D., Saria, S., and Liu, A. (2023). Jaws-x: addressing efficiency bottlenecks of conformal prediction under standard and feedback covariate shift. In International Conference on Machine Learning , pages 28167--28190. PMLR

    2023

  37. [45]

    D., Liu, A., and Saria, S

    Prinster, D., Stanton, S. D., Liu, A., and Saria, S. (2024). Conformal validity guarantees exist for any data distribution (and how to find them). Forty-first International Conference on Machine Learning

    2024

  38. [46]

    Ramdas, A., Gr \"u nwald, P., Vovk, V., and Shafer, G. (2023). Game-theoretic statistics and safe anytime-valid inference. Statistical Science , 38(4):576--601

    2023

  39. [47]

    Ramdas, A., Ruf, J., Larsson, M., and Koolen, W. M. (2022). Testing exchangeability: Fork-convexity, supermartingales and e-processes. International Journal of Approximate Reasoning , 141:83--109

    2022

  40. [48]

    Roberts, S. (1966). A comparison of some control chart procedures. Technometrics , 8(3):411--430

    1966

  41. [49]

    Romano, Y., Sesia, M., and Candes, E. (2020). Classification with valid and adaptive coverage. Advances in neural information processing systems , 33:3581--3591

    2020

  42. [50]

    and Ramdas, A

    Saha, A. and Ramdas, A. (2024). Testing exchangeability by pairwise betting. In International Conference on Artificial Intelligence and Statistics , pages 4915--4923. PMLR

    2024

  43. [51]

    Shaer, S., Maman, G., and Romano, Y. (2023). Model-x sequential testing for conditional independence via testing by betting. In International Conference on Artificial Intelligence and Statistics , pages 2054--2086. PMLR

    2023

  44. [52]

    Shafer, G. (2021). Testing by betting: A strategy for statistical and scientific communication. Journal of the Royal Statistical Society Series A: Statistics in Society , 184(2):407--431

    2021

  45. [53]

    and Vovk, V

    Shafer, G. and Vovk, V. (2019). Game-theoretic foundations for probability and finance , volume 455. John Wiley & Sons

    2019

  46. [54]

    and Ramdas, A

    Shekhar, S. and Ramdas, A. (2023a). Nonparametric two-sample testing by betting. IEEE Transactions on Information Theory , 70(2):1178--1203

    2023

  47. [55]

    and Ramdas, A

    Shekhar, S. and Ramdas, A. (2023b). Reducing sequential change detection to sequential estimation. arXiv preprint arXiv:2309.09111

    2023 arXiv

  48. [56]

    and Ramdas, A

    Shekhar, S. and Ramdas, A. (2023c). Sequential changepoint detection via backward confidence sequences. In International Conference on Machine Learning , pages 30908--30930. PMLR

    2023

  49. [57]

    Shimodaira, H. (2000). Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference , 90(2):227--244

    2000

  50. [58]

    Shin, J., Ramdas, A., and Rinaldo, A. (2022). E-detectors: A nonparametric framework for sequential change detection. arXiv preprint arXiv:2203.03532

    2022 arXiv

  51. [59]

    Shiryaev, A. N. (1963). On optimum methods in quickest detection problems. Theory of Probability & Its Applications , 8(1):22--46

    1963

  52. [60]

    Shiryaev, A. N. (2016). Probability-1 , volume 95. Springer

    2016

  53. [61]

    Stanton, S., Maddox, W., and Wilson, A. G. (2023). Bayesian optimization with conformal prediction sets. In International Conference on Artificial Intelligence and Statistics , pages 959--986. PMLR

    2023

  54. [62]

    Sugiyama, M., Krauledat, M., and M \"u ller, K.-R. (2007). Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research , 8(5)

    2007

  55. [63]

    Tartakovsky, A., Nikiforov, I., and Basseville, M. (2014). Sequential analysis: Hypothesis testing and changepoint detection . CRC press

    2014

  56. [64]

    and Sulam, J

    Teneggi, J. and Sulam, J. (2024). Testing semantic importance via betting. Advances in Neural Information Processing Systems , 37:76450--76499

    2024

  57. [65]

    J., Foygel Barber, R., Candes, E., and Ramdas, A

    Tibshirani, R. J., Foygel Barber, R., Candes, E., and Ramdas, A. (2019). Conformal prediction under covariate shift. Advances in neural information processing systems , 32

    2019

  58. [66]

    Veeravalli, V. V. and Banerjee, T. (2014). Quickest change detection. In Academic press library in signal processing , volume 3, pages 209--255. Elsevier

    2014

  59. [67]

    Ville, J. (1939). Etude critique de la notion de collectif . Gauthier-Villars Paris

    1939

  60. [68]

    Volkhonskiy, D., Burnaev, E., Nouretdinov, I., Gammerman, A., and Vovk, V. (2017). Inductive conformal martingales for change-point detection. In Conformal and Probabilistic Prediction and Applications , pages 132--153. PMLR

    2017

  61. [69]

    Vovk, V. (2002). On-line confidence machines are well-calibrated. In The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings. , pages 187--196. IEEE

    2002

  62. [70]

    Vovk, V. (2003). Well-calibrated predictions from online compression models. In Algorithmic Learning Theory: 14th International Conference, ALT 2003, Sapporo, Japan, October 17-19, 2003. Proceedings 14 , pages 268--282. Springer

    2003

  63. [71]

    Vovk, V. (2020). Testing for concept shift online. arXiv preprint arXiv:2012.14246

    2020 arXiv

  64. [72]

    Vovk, V. (2021). Testing randomness online. Statistical Science , 36(4):595--611

    2021

  65. [73]

    Vovk, V. (2023). The power of forgetting in statistical hypothesis testing. In Conformal and Probabilistic Prediction with Applications , pages 347--366. PMLR

    2023

  66. [74]

    Vovk, V., Gammerman, A., and Shafer, G. (2005). Algorithmic learning in a random world , volume 29. Springer

    2005

  67. [75]

    Vovk, V., Gammerman, A., and Shafer, G. (2022). Algorithmic Learning in a Random World . Springer Nature

    2022

  68. [76]

    Vovk, V., Nouretdinov, I., and Gammerman, A. (2003). Testing exchangeability on-line. In Proceedings of the 20th international conference on machine learning (ICML-03) , pages 768--775

    2003

  69. [77]

    Vovk, V., Petej, I., Nouretdinov, I., Ahlberg, E., Carlsson, L., and Gammerman, A. (2021). Retrain or not retrain: Conformal test martingales for change-point detection. In Conformal and Probabilistic Prediction and Applications , pages 191--210. PMLR

    2021

  70. [78]

    and Wang, R

    Vovk, V. and Wang, R. (2021). E-values: Calibration, combination and applications. The Annals of Statistics , 49(3):1736--1754

    2021

  71. [79]

    and Wang, R

    Vovk, V. and Wang, R. (2023). Confidence and discoveries with e-values. Statistical Science , 38(2):329--354

    2023

  72. [80]

    Wald, A. (1945). Sequential tests of statistical hypotheses. In Breakthroughs in statistics: Foundations and basic theory , pages 117--186. Springer

    1945

  73. [81]

    Wasserman, L., Ramdas, A., and Balakrishnan, S. (2020). Universal inference. Proceedings of the National Academy of Sciences , 117(29):16880--16890

    2020

  74. [82]

    and Ramdas, A

    Waudby-Smith, I. and Ramdas, A. (2024). Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society Series B: Statistical Methodology , 86(1):1--27

    2024

  75. [83]

    Xie, L., Zou, S., Xie, Y., and Veeravalli, V. V. (2021). Sequential (quickest) change detection: Classical results and new directions. IEEE Journal on Selected Areas in Information Theory , 2(2):494--514

    2021

  76. [84]

    and Xie, Y

    Xu, C. and Xie, Y. (2021). Conformal prediction interval for dynamic time-series. In International Conference on Machine Learning , pages 11559--11569. PMLR

    2021

  77. [85]

    K., and Tchetgen Tchetgen, E

    Yang, Y., Kuchibhotla, A. K., and Tchetgen Tchetgen, E. (2024). Doubly robust calibration of prediction sets under covariate shift. Journal of the Royal Statistical Society Series B: Statistical Methodology , page qkae009

    2024

  78. [86]

    Zhang, Y.-J., Zhang, Z.-Y., Zhao, P., and Sugiyama, M. (2024). Adapting to continuous covariate shift via online density ratio estimation. Advances in Neural Information Processing Systems , 36

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.