{"id":"4b0e2764-30ec-47ec-9078-c6f49708731d","arxiv_id":"2505.04608","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"WCTMs generalize conformal test martingales to test non-exchangeability nulls, enabling adaptation to mild covariate shifts, fast detection of harmful shifts, and root-cause diagnosis.","lead":"This paper introduces weighted-conformal test martingales, a method that monitors AI systems after deployment and raises alarms when data shifts become harmful. It also adapts to mild, benign shifts and diagnoses whether a problem comes from changed inputs or changed input-output relationships.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The X-CTM adaptation trigger is a data-dependent stopping time outside the fixed-tad null of Prop. 3.2; a false trigger can make WCTM p-values non-uniform and inflate the false-alarm rate.","rationale":"The paper's headline contribution is a false-alarm-controlled adaptive monitor, not just a theoretical martingale construction. The proof of Prop. 3.2 is the linchpin: if the p-values fed to the betting process are not IID uniform under the null, Ville's inequality does not apply. The implementation changes the data-generating process under which those p-values are defined because the adaptation start is itself chosen by a separate test martingale. This is not a disagreement with the authors' preferred assumption; it is an internal mismatch between the theorem's fixed-tad null and the algorithm's data-dependent tad. The concern is concretely testable: the pure-null experiment isolates the trigger's effect. If the empirical false-alarm rate is controlled, the concern is resolved and the paper's practical claim stands; if not, the authors need either a joint e-process over the trigger and the WCTM, or a protocol (e.g., a fixed adaptation schedule or a much more stringent trigger threshold with explicit false-alarm accounting). The finite-calibration assumption in Eq. (28) is a further caveat, but the trigger problem is more fundamental because it can produce false alarms even in the infinite-calibration limit. The reader's CONDITIONAL verdict remains appropriate; no rejection is warranted because the theoretical WCTM framework is coherent and the practical issue is identifiable and addressable.","tokens_in":35027,"tokens_out":11714,"duration_ms":131517,"concrete_test":"Run the Sec. 3.6 WATCH algorithm under the pure null (all post-deployment data IID from the source distribution, no covariate or concept shift) using the published X-CTM adaptation threshold and WCTM alarm threshold c (e.g., X-CTM threshold 20, c=100) over many seeds, and record whether the WCTM ever exceeds c. Repeat the same null with tad fixed at a predetermined time (e.g., 500) instead of X-CTM-triggered. If the adaptive-trigger version's empirical false-alarm rate is materially above 1/c while the fixed-tad version is not, the missing joint guarantee is a real failure of the central false-alarm claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing gap is the transfer of Proposition 3.2 to the adaptive WATCH procedure. Proposition 3.2 (Eq. 14) assumes a WCTM built from p-values that are IID uniform under a fixed null H0(fhat). The implementation in Sec. 3.6 does not meet this condition: the adaptation time tad is not a fixed constant but the stopping time of the secondary X-CTM (first time that X-CTM crosses the adaptation threshold). Under the natural 'no harmful shift' null, the X-CTM can falsely trigger with probability at most 1/threshold; once this happens the main WCTM switches from uniform weights to estimated density-ratio weights bw(t) (Eq. 17) that are not the true identity weights. The resulting p-values need not be uniform, and the WCTM can grow to an alarm. Appendix C's null H0(cs) (Eq. 23) fixes tad and therefore does not cover this event; no joint bound over the trigger and the subsequent WCTM is supplied. The finite-calibration idealization bF_V=F_V (Eq. 28) is a second, separate gap; but even with an infinite exact calibration set, the data-dependent trigger alone can invalidate the anytime-valid false-alarm claim for the implemented procedure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces weighted-conformal test martingales (WCTMs), built from sequences of online weighted-conformal p-values, as a generalization of conformal test martingales for testing nonparametric null hypotheses beyond exchangeability. The main theoretical result (Theorem 3.1) states that online weighted-conformal p-values are IID uniform under a null hypothesis H0(f-hat) that includes the accuracy of the user's modeling assumptions; Proposition 3.2 then gives anytime-valid false-alarm control via Ville's inequality, and Proposition 3.3 gives average-run-length control for a Shiryaev-Roberts variant. The proposed WATCH framework uses a secondary X-CTM to trigger adaptation to mild covariate shifts via online density-ratio estimation, raises alarms for extreme covariate shifts or concept shifts, and diagnoses the root cause. Experiments on tabular and image datasets compare WCTMs against standard CTMs and Podkopaev-Ramdas baselines in terms of detection delay, false alarms, coverage, and runtime.","tokens_in":35360,"tokens_out":4284,"duration_ms":45118,"significance":"If the guarantees hold as stated, the paper makes a substantial contribution to sequential nonparametric testing and AI monitoring: it extends conformal test martingales to composite nulls that accommodate modeled shifts, it provides a principled betting-based construction that adapts online, and it offers root-cause diagnostics. The paper is also commendable for shipping code, for running extensive experiments on multiple real datasets, and for giving a self-contained proof of the oracle-weight validity result. The appendix is unusually candid in acknowledging the finite-calibration idealization and the non-independence of the practical p-values; however, these acknowledgments reveal that the headline false-alarm guarantee for the implemented adaptive procedure is not actually established by the theorems.","major_comments":[{"comment":"The anytime-valid false-alarm guarantee is not transferred to the adaptive WATCH procedure. Proposition 3.2 assumes a WCTM built from weighted-conformal p-values that are IID uniform under a fixed null H0(f-hat), but the implementation in Section 3.6 makes the adaptation time tad a data-dependent stopping time: the secondary X-CTM triggers adaptation the first time its wealth crosses the adaptation threshold. The null H0(cs) in Eq. (23) and the modified null in Eq. (28) both treat tad as fixed, so they do not cover the event that the X-CTM falsely triggers under the no-harmful-shift null. Once a false trigger occurs, the main WCTM switches from uniform weights to estimated density-ratio weights bw(t)(x) from Eq. (17), and the resulting p-values need not be uniform. No joint anytime-valid bound over the trigger and the subsequent WCTM is supplied, so the claim that WATCH controls false alarms is not established for the implemented adaptive procedure. Please either prove a joint guarantee over the trigger and the WCTM, or state the conditional nature of the guarantee precisely and provide null simulations at the advertised 1/c rate.","section":"Section 3.6 and Proposition 3.2"},{"comment":"The proof of Lemma B.1 assumes that the nonconformity score function is bijective, stating that 'because the score function bS is bijective, this further implies that Vt = vσ(t) = vi'. This is false for the scores used in the experiments, such as absolute residuals |y - bµ(x)| and one-minus-softmax scores, which map many data points to the same score value. When the score function is non-injective, the event {Vt = vi} does not identify a single data index, and the oracle weights in Eq. (19) do not equal the probabilities of the observed scores under the permutation distribution. The random tie-breaking with un+1 does not repair the proof as written because the argument needs injectivity to condition on the test score. Please provide a version of Lemma B.1 that handles non-injective score functions, for example by defining oracle weights on score atoms or via a randomized score-injection argument, or state exact validity only for injective scores and give a separate treatment of ties.","section":"Appendix B.1, Lemma B.1"},{"comment":"The practical WCTM's false-alarm guarantee rests on the idealization bF_{[n+tad-1]}^V = F_V, i.e., that the empirical CDF of the calibration scores equals the true source score distribution, which the paper itself notes is 'of course true in the limit n → ∞'. For finite n this equality fails, so the implemented method does not achieve exact anytime-valid false-alarm control under H0(cs); it achieves control only in an idealized asymptotic regime. In addition, the permutation-restriction approximation in Eq. (26) means the p-values in Eq. (27) are not independent across times, as the paper acknowledges in the paragraph preceding Eq. (28), so Theorem 3.1 does not directly apply to the sequence actually used in WATCH. Please either modify the method to restore the requisite independence and prove the claimed guarantees for the modified method, or explicitly label the headline false-alarm statements as approximations that are empirically validated rather than as consequences of Theorem 3.1.","section":"Appendix C, Eqs. (26)-(28)"}],"minor_comments":[{"comment":"The text contains several typos: 'deplyed' should be 'deployed', and in Section 3.2 'eactly accurate' should be 'exactly accurate'.","section":"Section 5"},{"comment":"The caption contains typos: 'ans Shiryaev-Roberts paths' should be 'and Shiryaev-Roberts paths', and 'simuluate' should be 'simulate'.","section":"Figure 2 caption"},{"comment":"The notation 'bw(t(Xi)' appears to be missing a closing parenthesis; it should presumably read 'bw(t)(Xi)'.","section":"Section 3.6, Eq. (17)"},{"comment":"The legend item 'X-CTM (Adapt Criterion)' is unclear; please specify that the X-CTM's wealth crossing the adaptation threshold triggers the adaptation phase.","section":"Figure 4"},{"comment":"The paper does not report the actual alarm threshold c used for the WCTM and SR-WCTM methods, nor the adaptation threshold for the X-CTM; please report these values so the anytime-valid claims can be checked against the stated rates.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about several of its idealizations, but the gap between the theory and the adaptive implementation is load-bearing: the headline false-alarm control for WATCH is not a theorem as written. The issues are fixable within the scope of the paper if the authors either supply a joint guarantee over the adaptation trigger and the subsequent WCTM or reframe the practical guarantees as conditional and empirically validated. The work is otherwise novel and appropriate for this venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"WATCH is a genuine contribution to post-deployment monitoring. The new piece is Theorem 3.1: sequences of online weighted-conformal p-values are IID uniform under a null that includes the estimated density-ratio weights, so you can build martingales beyond exchangeability. That's not in the existing CTM or weighted-CP literature, and the proof sketch is plausible. The practical setup—a secondary X-CTM that triggers adaptation, plus root-cause diagnosis—is useful, and the experiments are thorough, multi-dataset, with code and reasonable baselines.\n\nWhere it gets soft: the anytime-valid false-alarm guarantee in Prop 3.2 applies to a WCTM with fixed weights and a fixed null. The implemented WATCH procedure does not meet that condition, because the adaptation time tad is chosen by the X-CTM—a data-dependent stopping time. Appendix C fixes tad in the null H0(cs), so it doesn't cover the event that the X-CTM fires falsely under the null and then the WCTM uses estimated density-ratio weights that may not be the correct weights. The resulting p-values need not be uniform, and Ville's inequality no longer controls the alarm. This is the same point the stress-test note makes, and it lands: the stated false-alarm guarantee does not cover the implemented method. The paper also flags the finite-calibration idealization bF_V=F_V, but that's a separate issue.\n\nMinor: Lemma B.1 assumes the nonconformity score is bijective. Most scores aren't; ties need a careful argument. And the appendix promises full pseudocode but only gives partial algorithms—the authors say they'll update, but that should be part of the revision.\n\nOn balance, this is not a circular or incoherent paper. The core theorem is proved, and the experiments are credible. The gap is real but fixable: the authors need a joint treatment of the trigger (e.g., a union bound over the trigger event, or a modified null that covers the trigger), or they need to state the guarantee as conditional on the trigger and show empirical false-alarm rates.\n\nFor whom: anyone building practical monitoring systems for deployed ML, and conformal-prediction theorists. It deserves a serious referee; I'd send it out and ask for the trigger gap to be addressed. If they can close it, this is a strong method paper.","headline":"Solid method paper with a real anytime-validity gap between the clean martingale theory and the implemented adaptive procedure; worth a serious referee but needs a joint treatment of the trigger.","tokens_in":65,"tokens_out":2023,"would_cite":true,"duration_ms":73362,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that weighted-conformal p-values are IID uniform under a null that allows modeled covariate shift, so test martingales built from them can adapt to benign shifts while keeping anytime-valid alarm control.","keywords":["weighted conformal prediction","conformal test martingales","anytime-valid inference","online changepoint detection","distribution shift monitoring","covariate shift adaptation","concept shift detection","AI deployment monitoring"],"falsifier":"Repeatedly simulate a long stream of data from a known discrete distribution, compute oracle weighted-conformal p-values exactly using the true joint probabilities of Eq. (10), and feed them through a fixed betting strategy; if the empirical p-value distribution deviates from IID Uniform[0,1], or if the fraction of runs whose martingale crosses threshold $c=100$ exceeds $1/100$, the paper's central theorem and anytime-valid guarantee are false, since this simulation has no weight-estimation error to mask the issue.","tokens_in":1710,"feed_emoji":"🔔","tokens_out":2708,"duration_ms":87884,"temperature":0.7,"pith_summary":"This paper tries to give deployed machine-learning systems a monitor that raises an alarm only when a distribution shift is genuinely harmful, while silently adapting to benign shifts. It generalizes conformal test martingales by weighting each calibration score with a likelihood ratio, and proves that the resulting online p-values are IID uniform under a null that allows modeled covariate shift. Consequently the weighted martingale inherits anytime-valid false-alarm control, and the practical WATCH system detects concept shifts or extreme covariate shifts and tells them apart. If correct, this removes a central limitation of prior conformal monitoring: alarms no longer fire for every deviation from exchangeability.","feed_headline":"Monitor adapts to benign shifts and flags harmful ones","feed_subtitle":"Weighted-conformal p-values stay uniform under modeled covariate shift, keeping false alarms in check while the monitor adapts.","key_machinery":"The central object is the weighted-conformal p-value, a sum over calibration scores with likelihood-based weights $\\tilde{w}_i$ replacing the uniform $1/(n+1)$ weights of a standard conformal p-value. The key identity is Theorem 3.1: when the weights equal the oracle weights under the null, the online p-value sequence is IID uniform on $[0,1]$. Feeding that sequence into a product of betting functions yields a nonnegative martingale, and a parallel input-only conformal test martingale triggers adaptation and helps diagnose whether an alarm is a concept shift or an extreme covariate shift.","core_discovery":"Under a null hypothesis that either exchangeability holds or the covariate distribution shifts exactly as modeled by the estimated density-ratio weights while the label distribution stays invariant, the sequence of online weighted-conformal p-values is independent and uniform on $[0,1]$. Therefore a test martingale constructed by betting on those p-values is a nonnegative martingale under that null, and Ville's inequality gives $P(\\text{alarm ever raised}) \\le 1/c$ for any threshold $c$. This shifts the monitored null from 'the data distribution is identical over time' to 'the data distribution changes only in the anticipated way,' so a monitor can adapt to benign covariate shifts without sacrificing false-alarm control.","pith_inferences":["The paper does not quantify how the false-alarm guarantee degrades when density-ratio estimates are only approximately correct; a natural extension is a robustness theorem bounding alarm inflation as a function of weight-estimation error.","Because the X-CTM runs on inputs alone, the same machinery could provide a label-free early-warning layer that flags when the input stream has left the region where the estimated weights are trustworthy, before any labels arrive.","One could close the monitoring loop by triggering retraining at the alarm and reinitializing the martingale with the updated model; this paper stops at detection and diagnosis.","The alarm threshold $c$ could be tuned against deployment-specific costs of false versus missed alarms, treating the martingale value as a continuous evidence scale rather than a binary trigger."],"forward_implications":["A deployment monitor can adapt its conformal prediction sets to a mild covariate shift and keep target coverage without alarming, because the weighted p-values remain uniform under the modeled shift.","Any violation of the modeled null, such as a concept shift or an unmodeled extreme covariate shift, will eventually inflate the weighted martingale, with probability of ever crossing threshold $c$ bounded by $1/c$.","The same construction extends to any null hypothesis the user can express through oracle weights, not just covariate shift, as stated by the general Theorem 3.1.","Running a Shiryaev-Roberts procedure on the weighted martingale gives average run length at least $c$, providing a stagewise alternative to anytime-valid alarms.","Running a standard input-only X-CTM alongside the weighted martingale enables root-cause diagnosis: both alarm indicates extreme covariate shift, only the weighted martingale alarms indicates concept shift, and neither alarms while the X-CTM is active indicates a successfully adapted benign shift."],"supporting_citations":[{"why":"Defines conformal test martingales as betting martingales on IID uniform p-values and gives the Shiryaev-Roberts average-run-length result the paper reuses.","marker":"Vovk (2021)"},{"why":"Supplies the reversed-time induction argument that the paper generalizes to prove weighted-conformal p-values are IID uniform.","marker":"Vovk et al. (2003)"},{"why":"Introduces weighted conformal prediction under covariate shift with density-ratio weights, the template for the paper's practical weights.","marker":"Tibshirani et al. (2019)"},{"why":"Provides the oracle-weights view of conformal validity for arbitrary joint distributions that Theorem 3.1 builds on.","marker":"Prinster et al. (2024)"},{"why":"Gives the martingale inequality that converts the martingale property into anytime-valid false-alarm control.","marker":"Ville (1939)"},{"why":"Proposes conformal test martingales for retraining decisions and the nearest-neighbor X-CTM used for adaptation triggers and diagnosis.","marker":"Vovk et al. (2021)"},{"why":"Supplies the risk-tracking sequential testing and changepoint baselines that the paper compares detection speed and runtime against.","marker":"Podkopaev and Ramdas (2021b)"},{"why":"Defines the conformal p-value construction and the composite jumper betting strategy used in implementations.","marker":"Vovk et al. (2022)"},{"why":"Inspires the online density-ratio estimation via a probabilistic classifier used to update weights after adaptation begins.","marker":"Zhang et al. (2024)"}],"fun_headline_variants":["Weighted-conformal martingales: adapt and flag harmful shifts","AI monitors that adapt to benign drift, detect harmful change","Adaptive AI monitors: detect harmful shifts, tolerate benign drift","Weighted-conformal martingales adapt to covariate shifts, flag harmful ones","Uniform p-values under covariate shift keep AI monitoring false-alarm free"],"cache_read_input_tokens":38016,"weakest_assumption_plain":"The false-alarm guarantee holds only when the estimated density-ratio weights exactly match the true covariate shift and the calibration set is large enough that its score distribution exactly matches the source distribution, with the adaptation trigger treated as fixed rather than data-dependent.","fun_headline_variants_meta":{"raw":{"variants":["Weighted-conformal martingales: adapt and flag harmful shifts","AI monitors that adapt to benign drift, detect harmful change","Adaptive AI monitors: detect harmful shifts, tolerate benign drift","Weighted-conformal martingales adapt to covariate shifts, flag harmful ones","Uniform p-values under covariate shift keep AI monitoring false-alarm free"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001864,"raw_usage":{"total_tokens":7307,"prompt_tokens":926,"completion_tokens":6381,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":6289}},"tokens_in":542,"tokens_out":6381,"duration_ms":45191,"temperature":1.0,"reasoning_tokens":6289,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:24:08.749290+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeatedly simulate a long stream of data from a known discrete distribution, compute oracle weighted-conformal p-values exactly using the true joint probabilities of Eq. (10), and feed them through a fixed betting strategy; if the empirical p-value distribution deviates from IID Uniform[0,1], or if the fraction of runs whose martingale crosses threshold $c=100$ exceeds $1/100$, the paper's central theorem and anytime-valid guarantee are false, since this simulation has no weight-estimation error to mask the issue.","supporting_citations":[{"cited_title":"D., Liu, A., and Saria, S","cited_arxiv_id":null,"evidence_quote":"Provides the oracle-weights view of conformal validity for arbitrary joint distributions that Theorem 3.1 builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the conformal p-value construction and the composite jumper betting strategy used in implementations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Inspires the online density-ratio estimation via a probabilistic classifier used to update weights after adaptation begins."}],"review_version":1}