Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

On Continuous Monitoring of Risk Violations under Unknown Shift

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A betting-style monitor can flag a deployed model's risk violations in real time under unknown distribution shift, while keeping the false-alarm rate bounded by a user-set level.

desk verdict Solid testing-by-betting risk monitor with a real gap between the stated unconditional target and the proven conditional guarantee. read the letter →

arxiv 2506.16416 v1 pith:DFXNWPX3 submitted 2025-06-19 stat.ML cs.LG

Alexander Timans , Rajeev Verma , Eric Nalisnick , Christian A. Naesseth This is my paper · ORCID
classification stat.MLcs.LG MSC 62L1060G42
keywords riskmonitoringdistributionshiftsequentialhypothesistestingbybettingtestsupermartingalefalsealarmcontrolanytime-validinferenceboundedfunctions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a monitoring system can watch an evolving data stream and raise an alarm whenever a deployed model's bounded risk exceeds a tolerance level, even when the distribution shifts in unknown ways. The approach casts risk monitoring as a sequential hypothesis test built on 'testing by betting': each candidate threshold runs a wealth process that multiplies small bets against the null hypothesis that the risk is under control, and a threshold is flagged when its wealth crosses a preset boundary. The central guarantee is finite-sample false-alarm control: a threshold whose risk is genuinely under control is falsely flagged with probability at most a user-set $\delta$. If correct, this gives deployers a way to keep statistical safety assurances valid under arbitrary drift, which static risk-control methods cannot do.

What carries the argument

The load-bearing object is the wealth process $M_t(\psi) = \prod_{i=1}^t (1 + \lambda_i (z_i - \epsilon))$, a non-negative test supermartingale that starts at 1 and is driven by the centred losses $z_i - \epsilon$ scaled by a predictable betting rate $\lambda_i \in [0, 1/\epsilon)$. Its role is to convert a stream of bounded losses into accumulating evidence against the 'risk controlled' null; when the losses' conditional mean exceeds $\epsilon$, the process drifts upward, and crossing the boundary $1/\delta$ triggers rejection. The guarantees rest on three standard facts used by the paper: the supermartingale property under the null, Ville's inequality converting it into the time-uniform false-alarm bound, and the growth-rate-optimal (GRO) choice of betting rate to make evidence accumulate as fast as possible. The companion object is the confidence set $C^\psi_t$, the set of thresholds whose wealth has not yet crossed the boundary.

What would settle it

Simulate a stream of bounded losses whose marginal mean at every time is at most $\epsilon$ but whose conditional mean given the recent past exceeds $\epsilon$, for instance losses that alternate predictably (high after low) with the low runs making up the majority, and run the monitor against the stated target $R_t(\psi)$; if the wealth process crosses $1/\delta$ with empirical frequency above $\delta$ for small nominal $\delta$, the guarantee does not protect the quantity the paper names as its monitoring target.

Watch

Extended reading notes

Core claim

On the paper's own terms, the contribution is a test-supermartingale-based monitor for time-dependent risks. For each threshold $\psi$ governing a decision mechanism, the process $M_t(\psi) = \prod_{i=1}^t (1 + \lambda_i (z_i - \epsilon))$ accumulates evidence against the null hypothesis $H_0(\psi): E_{P_t}[z_t \mid \mathcal{F}_{t-1}] \le \epsilon$ for all times, with the betting rate $\lambda_t$ chosen predictably (depending only on past losses) and capped at $1/\epsilon$ to keep the wealth non-negative. Lemma 4.2 gives that under the null the wealth crosses the rejection boundary $1/\delta$ with probability at most $\delta$, which the paper interprets as control of the false alarm rate, and Lemma 4.3 gives that a persistently violating threshold is eventually rejected with probability one. The set of thresholds not yet rejected, $C^\psi_t = \{\psi : M_t(\psi) < 1/\delta\}$, is the live set of 'still trustworthy' settings, and its shrinkage signals that the model's reliability is deteriorating. Empirical results on outlier detection and set prediction under stepwise, immediate, and natural temporal shifts show the method detecting violations with controlled false alarms, in contrast to a running-risk estimator that flags safe thresholds.

Load-bearing premise

The false-alarm guarantee is proved for the conditional mean of the loss given the past, $E_{P_t}[z_t \mid \mathcal{F}_{t-1}] \le \epsilon$, but the quantity the paper says it is monitoring is the unconditional risk $R_t(\psi) = E_{P_t}[z_t]$; the two coincide only under an independence condition that is mentioned in an appendix remark rather than assumed in the main theorem.

Editorial extensions

If this is right

  • A threshold whose conditional risk stays at or below the tolerance $\epsilon$ is falsely flagged with probability at most $\delta$ over the entire monitoring run, so the false-alarm budget is respected without any assumption on the shift mechanism.
  • A threshold whose risk persistently exceeds $\epsilon$ is eventually rejected with probability one, giving the monitor asymptotic completeness (power one).
  • Detection delay grows roughly as $(\log(1/\delta) + T)/(\lambda \mu)$, so stronger violations, more aggressive betting, and smaller $\delta$ all speed detection, while shifts that arrive late are inherently harder to catch because earlier evidence must be overcome.
  • The live set of still-trustworthy thresholds $C^\psi_t$ shrinks as shift intensity grows, providing a real-time picture of how much of the threshold grid remains reliable and when a model update is warranted.
  • Across outlier-detection and set-prediction tasks under stepwise, immediate, and natural temporal shifts, the wealth process matches the fastest baselines while keeping false alarms at zero in the reported runs, unlike the running-risk estimator, which flags safe thresholds.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the false-alarm guarantee is per-threshold across the whole run, an operator who each day selects the least conservative threshold still inside the confidence set faces a multiple-testing problem; a separate correction step, which the paper sets aside, would be needed before adaptive selection inherits the $\delta$ guarantee.
  • The conditional-versus-unconditional gap in the null hypothesis is the natural stress point: a stream of positively autocorrelated bounded losses whose marginal mean stays below $\epsilon$ while the conditional mean given the past sits above $\epsilon$ would show whether the advertised false-alarm control protects the risk quantity an operator actually observes.
  • The shrinkage rate of the confidence set could be turned into an operational 'retrain now' signal; calibrating what shrink rate warrants an alert, as a function of $\epsilon$ and $\delta$, is a testable design question the paper leaves open.
  • The same machinery could monitor other bounded, online-observed quantities of a deployed system, such as calibration error or subgroup-specific error rates, since the proof only uses boundedness, predictability of the betting rate, and the supermartingale structure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a sequential-testing-by-betting framework for continuously monitoring whether a bounded risk of a deployed decision rule remains below a user-specified level ε in an evolving data stream. For each threshold ψ, the loss process z_t is monitored through a multiplicative wealth process M_t(ψ)=∏_{i=1}^t (1+λ_i(z_i−ε)), with a predictable betting rate λ_i; a threshold is flagged as violating when M_t(ψ) crosses 1/δ. The set of non-violating thresholds is reported as C_ψ^t={ψ:M_t(ψ)<1/δ}. The theoretical section claims a false-alarm guarantee by applying Ville's inequality to a test supermartingale (Lemma 4.2), asymptotic consistency under persistent violation (Lemma 4.3), and an approximate detection-delay bound (Proposition 4.5). Experiments on outlier detection (CIFAR-10/SVHN mixtures) and set prediction (FMoW and Naval propulsion) compare the wealth process with running-risk, summation, and Empirical-Bernstein baselines, reporting controlled false-alarm rates and competitive detection delays.

Significance. If the main guarantee were established for the paper's stated target, the framework would be a useful and broadly applicable contribution: it offers finite-sample, anytime-valid control of false alarms when monitoring a time-varying risk under unknown distribution shift, with essentially no distributional assumptions beyond boundedness and predictable betting rates. The paper gives credit to the underlying machinery (Waudby-Smith and Ramdas, Ville's inequality, GRO betting rates), and it provides open-source code and experiments on real datasets with multiple baselines. The principal advertised strength—a finite-sample false-alarm guarantee—does hold for the conditional null E[z_t|F_{t-1}]≤ε, via a correct use of Ville's inequality. However, the manuscript does not establish this guarantee for the unconditional risk R_t(ψ)=E[z_t] that it defines as the monitoring target, and the consistency and detection-delay statements are supported by heuristic arguments rather than complete proofs. These gaps bear directly on the paper's central claims.

major comments (3)
  1. [§2–§4, Eq. (2)–(3), Lemma 4.2] The monitoring target is defined as the unconditional risk R_t(ψ)=E_{P_t}[z_t] (Eq. 2), but all theoretical guarantees—Lemma 4.1, Lemma 4.2, Lemma 4.3, and the ψ-CS construction in Eq. (5)—are stated under the conditional null E_{P_t}[z_t|F_{t-1}]≤ε (Eq. 3). Under arbitrary temporal dependence the two quantities need not coincide: a threshold may satisfy R_t(ψ)≤ε for all t while E[z_t|F_{t-1}]>ε on some histories, causing M_t(ψ) to cross 1/δ and raise a false alarm relative to the stated target. Lemma 4.2 does not bound the probability of this event because its null hypothesis is not satisfied by such a stream. The appendix remark in §A.3 acknowledges this gap, stating that an independence assumption on samples drawn from P_t and P_t' 'can simplify' the hypothesis pair so that E[z_t|F_{t-1}]=R_t(ψ), but this assumption is not part of the main problem statement and is not imposed in the theoretical section. The advertised false-alarm guarantee for the defined risk R_t is therefore not proven; the guarantee holds only for a different, conditional quantity. The authors should either re-define the monitoring target as the conditional risk E[z_t|F_{t-1}], or add an explicit, verifiable condition under which the conditional and unconditional means agree and check that the experiments satisfy it.
  2. [§A.4, proof of Lemma 4.3] The proof of asymptotic consistency is not rigorous as written. The step 'Assuming a small, non-zero betting rate λ_t we use the approximation log(1+x)≈x' replaces an inequality with an uncontrolled approximation, so the subsequent Cesàro-mean argument does not constitute a proof. The statement lim inf_t (1/t)∑_{i=1}^t A_i ≥ lim inf_t A_t = λ·μ additionally requires a positive uniform lower bound on the conditional expected log-increments under the alternative and on λ_t; the definition of μ as an infimum over {E[z_t−ε|F_{t-1}]>0} does not guarantee that such a bound exists. The boundedness claim |v_t−A_t|≤λ_t is also asserted without derivation. Since Lemma 4.3 underlies the claim that persistently violating thresholds are eventually detected with probability one, this proof needs to be made fully rigorous under transparent conditions or replaced by a citation to a standard power-one result (e.g., Darling and Robbins, 1968) with the assumptions explicitly matched.
  3. [§4, Proposition 4.5 and §A.4 proof] The proposed detection-delay characterization (τ(ψ)−τ*(ψ)) ≈ O((log(1/δ)+T)/(λ·μ)) is heuristic rather than a proven worst-case bound. The proof in §A.4 explicitly assumes a sharp changepoint T, a constant violation intensity μ, a fixed betting rate λ, and uses the log(1+x)≈x approximation to pass from the multiplicative wealth process to the summation process; it also defines 'worst-case' through a loose Azuma-Hoeffding bound in the summation case without accounting for the martingale fluctuation term in the multiplicative case. The notation '≈O' is not a well-defined asymptotic statement. The proposition should be either restated as a heuristic (and labeled as such in the contributions) or converted into a theorem with precise assumptions, a rigorous proof, and a clear definition of the worst case.
minor comments (6)
  1. [§3] In the sentence describing the forecaster's guess, 'minimize the mean squared prediction error ..., thei best guess' contains a typo; it should read 'the best guess'.
  2. [Figure 4 caption] The caption states 'detection delays τ(ψ)−τ*(ψ) across all ψ∈H', but the threshold set is denoted Ψ elsewhere; this is inconsistent notation.
  3. [§6, Experimental text] The text says the gap between running risk and wealth process 'remains fairy narrow' in the Naval propulsion experiment; this should be 'fairly narrow'.
  4. [Appendix C] The phrase 'which are negligable for sufficiently large streams' contains a typo; it should be 'negligible'.
  5. [§7 Discussion] The paragraph describing limitations says 'we are inherently limited in our safety assurances by the unpredictability of any occuring shift'; 'occuring' should be 'occurring'.
  6. [§3.1] The claim 'we do not impose sample independence' is in tension with the conditional-versus-unconditional gap noted above; the paper should explicitly discuss how the experiments handle this distinction when reporting false-alarm rates against the true unconditional risk.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reasoning: the false-alarm guarantee follows from Ville's inequality applied to a verifiable test supermartingale, with external betting-rate results used only as design choices.

full rationale

The paper's central guarantee is not derived from its own conclusions. Lemma 4.1 verifies, from the definition of the multiplicative wealth process in Eq. 4, that under the conditional null E_{P_t}[z_t | F_{t-1}] ≤ ε the process is a non-negative supermartingale: the proof computes E_{P_t}[M_t(ψ)|F_{t-1}] = M_{t-1}(ψ)(1 + λ_t E_{P_t}[z_t - ε | F_{t-1}]) ≤ M_{t-1}(ψ), using only boundedness of z, predictability of λ_t, and the null. Lemma 4.2 is then an immediate application of Ville's inequality, an external standard result, not a claim imported from the authors' own prior work. Lemma 4.3 is a power-one consistency argument with its own proof in Appendix A.4, and Proposition 4.5 is explicitly presented as a characterization with a simplified changepoint proof rather than a fitted claim. The GRO betting-rate choice is taken from Waudby-Smith and Ramdas (2024) as a design recommendation; it is a predictable function of past losses, not fitted to the experimental outcomes, and the false-alarm guarantee holds for any predictable λ_t in [0, 1/ε). There is no self-citation chain carrying the load: the cited Ville inequality and test-supermartingale machinery are external and parameter-free given the stated assumptions. One substantive concern is flagged but is not circular: the problem statement targets the unconditional risk R_t(ψ) = E_{P_t}[z_t] in Eq. 2, while all guarantees are stated for the conditional null E_{P_t}[z_t | F_{t-1}] ≤ ε in Eq. 3; the appendix Remark in A.3 explicitly acknowledges that an independence assumption is needed to equate the two. That is an assumption-match gap relevant to correctness, not a derivation whose output is equivalent to its input by construction. Therefore no circularity is present.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The theoretical guarantee is built on standard martingale inequalities plus the bounded-loss assumption. The paper introduces no fitted constants. The main hidden assumptions are the conditional-mean interpretation of the target risk and extra conditions for the consistency and delay proofs.

free parameters (5)
  • Risk level epsilon in experiments = 0.1
    User-specified safety threshold; not fitted. The method's guarantees are valid for any epsilon in (0,1).
  • Confidence level delta in experiments = 0.1
    User-specified false-alarm tolerance; not fitted.
  • Sliding window size S = None, 200, 50, 10 across tables
    Hand-chosen tracking hyperparameter; affects detection delay and false-alarm sensitivity, not the Ville guarantee.
  • Batch size B = 1, 10, 50
    Hand-chosen; averages evidence within a time step and reduces variance and delay.
  • Burn-in steps t_burn = floor(100/B)
    Chosen to stabilize running estimates before testing; an implementation choice.
assumptions (6)
  • domain assumption Loss is bounded: l in [0,1], so risk R_t(psi) in [0,1] and z_t - epsilon in [-epsilon, 1-epsilon].
    Stated in Section 2 as the key restriction; needed for lambda_t in [0,1/epsilon) to keep wealth nonnegative and for Ville's inequality.
  • domain assumption The stream is governed by time-dependent conditional distributions P_t such that E[z_t|F_{t-1}] is well-defined under a single joint measure.
    Required for the martingale calculation in Lemma 4.1; the paper does not specify the probability space beyond a filtration in A.1.
  • standard math Ville's inequality for nonnegative supermartingales.
    Used to prove Lemma 4.2; cited to Ville 1939 and not reproved.
  • standard math Azuma-Hoeffding inequality for bounded martingale difference sequences.
    Used in Lemma 4.3 and Prop 4.5 proofs in A.4.
  • ad hoc to paper Under the alternative, an eventually positive and bounded betting rate exists, and the approximation log(1+x) is approximately x is accurate enough for the Cesaro argument.
    Lemma 4.3's statement omits this condition; the proof in A.4 invokes a small nonzero lambda and takes an approximation, so consistency is proven only under extra conditions.
  • ad hoc to paper Proposition 4.5 assumes a sharp changepoint at T with constant violation intensity mu greater than 0 after T, and uses the log-linear approximation for small bets.
    The proposition is a heuristic delay characterization, not a finite-sample bound; the setting is explicit in A.4 but absent from the main statement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Continuous Monitoring of Risk Violations under Unknown Shift." pith.science (2026). https://pith.science/paper/DFXNWPX3

@misc{pith2026250616416,
  author       = {Pith},
  title        = {Pith review of: On Continuous Monitoring of Risk Violations under Unknown Shift},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DFXNWPX3}},
  note         = {Machine review of arXiv:2506.16416}
}
read the original abstract

Machine learning systems deployed in the real world must operate under dynamic and often unpredictable distribution shifts. This challenges the validity of statistical safety assurances on the system's risk established beforehand. Common risk control frameworks rely on fixed assumptions and lack mechanisms to continuously monitor deployment reliability. In this work, we propose a general framework for the real-time monitoring of risk violations in evolving data streams. Leveraging the 'testing by betting' paradigm, we propose a sequential hypothesis testing procedure to detect violations of bounded risks associated with the model's decision-making mechanism, while ensuring control on the false alarm rate. Our method operates under minimal assumptions on the nature of encountered shifts, rendering it broadly applicable. We illustrate the effectiveness of our approach by monitoring risks in outlier detection and set prediction under a variety of shifts.

Figures

Figures reproduced from arXiv: 2506.16416 by the authors.

Figure 1
Figure 1. We consider an evolving data stream t = 1, . . . , T susceptible to distribution shifts, i.e. observations are drawn from a time-dependent distribution Pt at each step. A predictor ˆfψ is equipped with a decision-making mechanism governed by a threshold ψ (e.g. on outlier flagging). At deployment, we monitor each candidate ψ ∈ Ψ using a sequential testing process Mt(ψ) which collects evidence for or against risk vio… view at source ↗
Figure 2
Figure 2. Results for outlier detection with a stepwise shift (§ 6.1). From left to right: Visuals of the growing risk and wealth process behaviour with respective rejection thresholds ϵ and 1/δ, for a single threshold candidate (here ψ = 0.50); the behaviour of the valid threshold set ψ-CS (Eq. 5), which eventually shrinks to zero signalling a model update; and the empirical distributions of detection delays τ (ψ) − τ∗(ψ) ac… view at source ↗
Figure 3
Figure 3. Results for set prediction with a temporal shift on FMoW (§ 6.2). From left to right: Visuals of the growing risk and wealth process behaviour with respective rejection thresholds ϵ and 1/δ, for a single threshold candidate (here ψ = 0.08); the behaviour of the valid threshold set ψ-CS (Eq. 5), which eventually tends to zero signalling a model update; and the empirical distributions of detection delays τ (ψ) − τ∗(ψ)… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Results for set prediction with a temporal shift on Naval propulsion (§ 6.2). From left to right: Visuals of the growing risk and wealth process behaviour with respective rejection thresholds ϵ and 1/δ, for a single threshold candidate (here ψ = 0.005); the behaviour o…
Figure 5
Figure 5. Figure 5: Results for outlier detection with no shift (§ 6.1). From left to right: Visuals of the steady risk and wealth process behaviour with respective rejection thresholds ϵ and 1/δ, for a single threshold candidate (here ψ = 0.20); the behaviour of the valid threshold set ψ…
Figure 6
Figure 6. Figure 6: Results for outlier detection with an immediate shift early on (§ 6.1). From left to right: Visuals of the strongly growing risk and wealth process behaviour with respective rejection thresholds ϵ and 1/δ, for a single threshold candidate (here ψ = 0.90); the behaviour…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing

    cs.LG 2025-12 conditional novelty 6.0 of 10

    A density-ratio e-process wrapper converts black-box verifier scores into sequential decisions that control the false-alarm rate for agent trajectories, with empirical gains in early stopping.

Reference graph

Works this paper leans on

60 extracted references · 54 canonical work pages · cited by 1 Pith paper

  1. [1]

    Sequential H armful S hift D etection W ithout L abels

    Salim I Amoukou, Tom Bewley, Saumitra Mishra, Freddy Lecue, Daniele Magazzeni, and Manuela Veloso. Sequential H armful S hift D etection W ithout L abels. Neural Information Processing Systems, 2024

    work page 2024

  2. [2]

    Prediction-powered inference

    Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I Jordan, and Tijana Zrnic. Prediction-powered inference. Science, 2023 a

    work page 2023

  3. [3]

    Conformal P rediction: A G entle I ntroduction

    Anastasios N Angelopoulos, Stephen Bates, et al. Conformal P rediction: A G entle I ntroduction. Foundations and Trends in Machine Learning , 2023 b

    work page 2023

  4. [4]

    Conformal R isk C ontrol

    Anastasios N Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal R isk C ontrol. International Conference on Learning Representations, 2024 a

    work page 2024

  5. [5]

    Angelopoulos, Stephen Bates, Emmanuel J

    Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Cand \`e s, Michael I. Jordan, and Lihua Lei. Learn T hen T est: C alibrating P redictive A lgorithms to A chieve R isk C ontrol. The Annals of Applied Statistics, 2025

    work page 2025

  6. [6]

    Online conformal prediction with decaying step sizes

    Anastasios Nikolas Angelopoulos, Rina Barber, and Stephen Bates. Online conformal prediction with decaying step sizes. International Conference on Machine Learning, 2024 b

    work page 2024

  7. [7]

    A greement-on-the-line: P redicting the performance of neural networks under distribution shift

    Christina Baek, Yiding Jiang, Aditi Raghunathan, and J Zico Kolter. A greement-on-the-line: P redicting the performance of neural networks under distribution shift. Advances in Neural Information Processing Systems, 2022

    work page 2022

  8. [8]

    Protected T est- T ime A daptation via O nline E ntropy M atching: A B etting A pproach

    Yarin Bar, Shalev Shaer, and Yaniv Romano. Protected T est- T ime A daptation via O nline E ntropy M atching: A B etting A pproach. Neural Information Processing Systems, 2024

    work page 2024

Show all 60 references
  1. [9]

    Conformal prediction beyond exchangeability

    Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, 2023

    2023

  2. [10]

    Distribution-free, R isk-controlling P rediction S ets

    Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. Distribution-free, R isk-controlling P rediction S ets. Journal of the ACM, 2021 a

    2021

  3. [11]

    Cand \`e s, Lihua Lei, Yaniv Romano, and Matteo Sesia

    Stephen Bates, Emmanuel J. Cand \`e s, Lihua Lei, Yaniv Romano, and Matteo Sesia. Testing for outliers with conformal p-values. The Annals of Statistics, 2021 b

    2021

  4. [12]

    F unctional M ap of the W orld

    Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. F unctional M ap of the W orld. Proceedings of the Conference on Computer Vision and Pattern Recognition, 2018

    2018

  5. [13]

    C ondition-based maintenance of naval propulsion systems with supervised data analysis

    Francesca Cipollini, Luca Oneto, Andrea Coraddu, Alan John Murphy, and Davide Anguita. C ondition-based maintenance of naval propulsion systems with supervised data analysis. Ocean Engineering, 2018

    2018

  6. [14]

    M ultiple testing in multi-stream sequential change detection

    Sanjit Dandapanthula and Aaditya Ramdas. M ultiple testing in multi-stream sequential change detection. arXiv Preprint (arXiv:2501.04130), 2025

    2025 arXiv

  7. [15]

    S ome nonparametric sequential tests with power one

    Donald A Darling and Herbert Robbins. S ome nonparametric sequential tests with power one. Proceedings of the National Academy of Sciences, 1968

    1968

  8. [16]

    T esting the mean and variance by e-processes

    Yixuan Fan, Zhanyi Jiao, and Ruodu Wang. T esting the mean and variance by e-processes. Biometrika, 2025

    2025

  9. [17]

    I s out-of-distribution detection learnable? Advances in Neural Information Processing Systems, 2022

    Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. I s out-of-distribution detection learnable? Advances in Neural Information Processing Systems, 2022

    2022

  10. [18]

    Achieving R isk C ontrol in O nline L earning S ettings

    Shai Feldman, Liran Ringel, Stephen Bates, and Yaniv Romano. Achieving R isk C ontrol in O nline L earning S ettings. Transactions on Machine Learning Research, 2023

    2023

  11. [19]

    Conformal prediction: A unified review of theory and new challenges

    Matteo Fontana, Gianluca Zeni, and Simone Vantini. Conformal prediction: A unified review of theory and new challenges. Bernoulli, 2023

    2023

  12. [20]

    Cand \`e s

    Isaac Gibbs and Emmanuel J. Cand \`e s. Adaptive C onformal I nference U nder D istribution S hift. Neural Information Processing Systems, 2021

    2021

  13. [21]

    Safe testing

    Peter Gr \"u nwald, Rianne de Heide, and Wouter Koolen. Safe testing. Journal of the Royal Statistical Society B, 2024

    2024

  14. [22]

    L ocalized conformal prediction: A generalized inference framework for conformal prediction

    Leying Guan. L ocalized conformal prediction: A generalized inference framework for conformal prediction. Biometrika, 2023

    2023

  15. [23]

    D istribution-free binary classification: prediction sets, confidence intervals and calibration

    Chirag Gupta, Aleksandr Podkopaev, and Aaditya Ramdas. D istribution-free binary classification: prediction sets, confidence intervals and calibration. Advances in Neural Information Processing Systems, 2020

    2020

  16. [24]

    Probability Inequalities for Sums of Bounded Random Variables

    Wassily Hoeffding. Probability Inequalities for Sums of Bounded Random Variables. The Collected Works of Wassily Hoeffding, 1994

    1994

  17. [25]

    W ilds: A benchmark of in-the-wild distribution shifts

    Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. W ilds: A benchmark of in-the-wild distribution shifts. International Conference on Machine Learning, 2021

    2021

  18. [26]

    L earning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. L earning multiple layers of features from tiny images. Tech Report, 2009

    2009

  19. [27]

    I nductive conformal anomaly detection for sequential detection of anomalous sub-trajectories

    Rikard Laxhammar and G \"o ran Falkman. I nductive conformal anomaly detection for sequential detection of anomalous sub-trajectories. Annals of Mathematics and Artificial Intelligence, 2015

    2015

  20. [28]

    A comprehensive survey on test-time adaptation under distribution shifts

    Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribution shifts. International Journal of Computer Vision, 2024

    2024

  21. [29]

    Online distribution shift detection via recency prediction

    Rachel Luo, Rohan Sinha, Yixiao Sun, Ali Hindy, Shengjia Zhao, Silvio Savarese, Edward Schmerling, and Marco Pavone. Online distribution shift detection via recency prediction. International Conference on Robotics and Automation, 2024

    2024

  22. [30]

    R eading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. R eading digits in natural images with unsupervised feature learning. NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011

    2011

  23. [31]

    Naesseth, and Patrick Forr \' e

    Teodora Pandeva, Tim Bakker, Christian A. Naesseth, and Patrick Forr \' e . E- V aluating C lassifier T wo- S ample T ests. Transactions on Machine Learning Research, 2024 a

    2024

  24. [32]

    Deep anytime-valid hypothesis testing

    Teodora Pandeva, Patrick Forr \' e , Aaditya Ramdas, and Shubhanshu Shekhar. Deep anytime-valid hypothesis testing. International Conference on Artificial Intelligence and Statistics, 2024 b

    2024

  25. [33]

    Distribution-free uncertainty quantification for classification under label shift

    Aleksandr Podkopaev and Aaditya Ramdas. Distribution-free uncertainty quantification for classification under label shift. Uncertainty in Artificial Intelligence, 2021

    2021

  26. [34]

    Tracking the risk of a deployed model and detecting harmful distribution shifts

    Aleksandr Podkopaev and Aaditya Ramdas. Tracking the risk of a deployed model and detecting harmful distribution shifts. International Conference on Learning Representations, 2022

    2022

  27. [35]

    Sequential P redictive T wo- S ample and I ndependence T esting

    Aleksandr Podkopaev and Aaditya Ramdas. Sequential P redictive T wo- S ample and I ndependence T esting. Advances in Neural Information Processing Systems, 2023

    2023

  28. [36]

    Conformal V alidity G uarantees E xist for A ny D ata D istribution (and H ow to F ind T hem)

    Drew Prinster, Samuel Don Stanton, Anqi Liu, and Suchi Saria. Conformal V alidity G uarantees E xist for A ny D ata D istribution (and H ow to F ind T hem). International Conference on Machine Learning, 2024

    2024

  29. [37]

    H ypothesis T esting with E -values

    Aaditya Ramdas and Ruodu Wang. H ypothesis T esting with E -values. arXiv Preprint (arXiv:2410.23614), 2024

    2024 arXiv

  30. [38]

    Game- T heoretic S tatistics and S afe A nytime- V alid I nference

    Aaditya Ramdas, Peter Gr \"u nwald, Vladimir Vovk, and Glenn Shafer. Game- T heoretic S tatistics and S afe A nytime- V alid I nference. Statistical Science, 2023

    2023

  31. [39]

    ( A lmost) P rovable E rror B ounds U nder D istribution S hift via D isagreement D iscrepancy

    Elan Rosenfeld and Saurabh Garg. ( A lmost) P rovable E rror B ounds U nder D istribution S hift via D isagreement D iscrepancy. Advances in Neural Information Processing Systems, 2023

    2023

  32. [40]

    T esting exchangeability by pairwise betting

    Aytijhya Saha and Aaditya Ramdas. T esting exchangeability by pairwise betting. International Conference on Artificial Intelligence and Statistics, 2024

    2024

  33. [41]

    Nonparametric T wo- S ample T esting by B etting

    Shubhanshu Shekhar and Aaditya Ramdas. Nonparametric T wo- S ample T esting by B etting. IEEE Transactions on Information Theory, 2021

    2021

  34. [42]

    On the near-optimality of betting confidence sets for bounded means

    Shubhanshu Shekhar and Aaditya Ramdas. On the near-optimality of betting confidence sets for bounded means. arXiv Preprint (arXiv:2310.01547), 2023 a

    2023 arXiv

  35. [43]

    Sequential C hangepoint D etection via B ackward C onfidence S equences

    Shubhanshu Shekhar and Aaditya Ramdas. Sequential C hangepoint D etection via B ackward C onfidence S equences. International Conference on Machine Learning, 2023 b

    2023

  36. [44]

    Reducing sequential change detection to sequential estimation

    Shubhanshu Shekhar and Aaditya Ramdas. Reducing sequential change detection to sequential estimation. International Conference on Machine Learning, 2024

    2024

  37. [45]

    E-detectors: A N onparametric F ramework for S equential C hange D etection

    Jaehyeok Shin, Aaditya Ramdas, and Alessandro Rinaldo. E-detectors: A N onparametric F ramework for S equential C hange D etection. The New England Journal of Statistics in Data Science, 2023

    2023

  38. [46]

    Online A daptive A nomaly T hresholding with C onfidence S equences

    Sophia Huiwen Sun, Abishek Sankararaman, and Balakrishnan Murali Narayanaswamy. Online A daptive A nomaly T hresholding with C onfidence S equences. International Conference on Machine Learning, 2024

    2024

  39. [47]

    Conformal P rediction U nder C ovariate S hift

    Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal P rediction U nder C ovariate S hift. Advances in Neural Information Processing Systems, 2019

    2019

  40. [48]

    When accurate prediction models yield harmful self-fulfilling prophecies

    Wouter AC van Amsterdam, Nan van Geloven, Jesse H Krijthe, Rajesh Ranganath, and Giovanni Cin \'a . When accurate prediction models yield harmful self-fulfilling prophecies. Patterns, 2025

    2025

  41. [49]

    E tude critique de la notion de collectif

    Jean Ville. E tude critique de la notion de collectif . Gauthier-Villars Paris, 1939

    1939

  42. [50]

    Taming F alse P ositives in O ut-of- D istribution D etection with H uman F eedback

    Harit Vishwakarma, Heguang Lin, and Ramya Korlakai Vinayak. Taming F alse P ositives in O ut-of- D istribution D etection with H uman F eedback. International Conference on Artificial Intelligence and Statistics, 2024

    2024

  43. [51]

    Inductive conformal martingales for change-point detection

    Denis Volkhonskiy, Evgeny Burnaev, Ilia Nouretdinov, Alexander Gammerman, and Vladimir Vovk. Inductive conformal martingales for change-point detection. Conformal and Probabilistic Prediction and Applications, 2017

    2017

  44. [52]

    T esting R andomness O nline

    Vladimir Vovk. T esting R andomness O nline. Statistical Science, 2021

    2021

  45. [53]

    Retrain or not retrain: C onformal test martingales for change-point detection

    Vladimir Vovk, Ivan Petej, Ilia Nouretdinov, Ernst Ahlberg, Lars Carlsson, and Alex Gammerman. Retrain or not retrain: C onformal test martingales for change-point detection. Conformal and Probabilistic Prediction and Applications, 2021

    2021

  46. [54]

    A C omprehensive S urvey of C ontinual L earning: T heory, M ethod and A pplication

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A C omprehensive S urvey of C ontinual L earning: T heory, M ethod and A pplication. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  47. [55]

    E stimating means of bounded random variables by betting

    Ian Waudby-Smith and Aaditya Ramdas. E stimating means of bounded random variables by betting. Journal of the Royal Statistical Society Series B: Statistical Methodology, 2024

    2024

  48. [56]

    O nline multiple testing with e-values

    Ziyu Xu and Aaditya Ramdas. O nline multiple testing with e-values. International Conference on Artificial Intelligence and Statistics, 2024

    2024

  49. [57]

    Active, anytime-valid risk controlling prediction sets

    Ziyu Xu, Nikos Karampatziakis, and Paul Mineiro. Active, anytime-valid risk controlling prediction sets. Neural Information Processing Systems, 2024

    2024

  50. [58]

    W ild- T ime: A benchmark of in-the-wild distribution shift over time

    Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei W Koh, and Chelsea Finn. W ild- T ime: A benchmark of in-the-wild distribution shift over time. Advances in Neural Information Processing Systems, 2022

    2022

  51. [59]

    A daptive conformal predictions for time series

    Margaux Zaffran, Olivier F \'e ron, Yannig Goude, Julie Josse, and Aymeric Dieuleveut. A daptive conformal predictions for time series. International Conference on Machine Learning, 2022

    2022

  52. [60]

    Adaptive L earn-then- T est: S tatistically V alid and E fficient H yperparameter S election

    Matteo Zecchin and Osvaldo Simeone. Adaptive L earn-then- T est: S tatistically V alid and E fficient H yperparameter S election. International Conference on Machine Learning, 2025

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.