REVIEW 3 major objections 5 minor 19 references
Adaptive OOD detectors self-poison according to a sharp threshold law: the slope of the admission kernel acts as a reproduction number, and once it reaches one the detector's memory bank fills with in-distribution points and collapses.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 08:25 UTC pith:5WIVGQ7R
load-bearing objection Real phenomenon, useful theory, but the 'prediction' is self-consistency and the complete-poisoning arithmetic doesn't match the fits. the 3 major comments →
Self-Poisoning in Adaptive Out-of-Distribution Detection: A Sharp-Threshold Theory and Certified Label-Free Calibration
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Adaptive OOD detection obeys a reinforced stochastic recursion: the impurity ρ_t of the adaptive dictionary updates as ρ_t = ρ_{t-1} + γ_t(q(ρ_{t-1}) − ρ_{t-1} + noise), where q(ρ) is the admission kernel, the expected fraction of a batch's admissions that are wrong given current impurity. For affine kernels, the slope b = q′(ρ) acts as a reproduction number: b < 1 gives a benign equilibrium ρ* = a/(1−b); b ≥ 1 gives complete poisoning, ρ_t → 1. Across 96 settings, the kernel is affine with slope ≈0.947, so the class is near-critical by design and the predicted critical contamination rate lands on the empirical collapse knee. Gating admission through a frozen reserve (requiring a fixed e-val
What carries the argument
The central object is the admission kernel q(ρ), the expected fraction of a batch's admitted points that are wrong when bank impurity is ρ, and the mean-field map h(ρ) = q(ρ) − ρ that drives the urn recursion. Its slope b = q′(ρ) works like a reproduction number R₀: it decides whether the equilibrium is benign or fully poisoned. The fixing machinery is a severed-feedback gate: conformal e-values computed only against a frozen reserve, with every admission required to clear a fixed e-value threshold, so no admission evidence ever reads the bank it feeds. For calibration, the key device is a Storey-type contamination upper bound used to correct the quantile level 1 − α(1 − π̂_up) with DKW slac
Load-bearing premise
The load-bearing premise is that a detector's false-admission probability is fully captured by a single scalar impurity through a Lipschitz (and in practice affine) kernel — plus, for the calibration guarantee, that drift-induced p-value sub-uniformity outweighs OOD leakage above the tuning level; if either fails on a new domain, the predicted threshold and certificate do not transfer.
What would settle it
Measure the admission kernel on a new deployment with heterogeneous poison clusters and a different feature encoder: if the false-admission proportion is not described by an affine function of scalar impurity (R² well below 0.99), or if the fitted slope leaves the measured 0.93–0.96 band while the detector still collapses (or enters the predicted subcritical regime while the detector survives), the sharp-threshold transfer fails. Concretely, run one bursty low-π stream with an admission fraction q = 0.2 and record per-batch (ρ_{t-1}, w_t/a_t); a non-affine q(ρ) or a threshold mismatch at the p
If this is right
- Ungated OOD-dictionary detectors in bursty low-contamination streams are predicted and observed to collapse: impurity exceeds 0.9 and AUROC drops by up to 0.163 below the detector's own frozen baseline.
- The critical contamination rate πc is computable from two measured kernel coefficients a(π) and b(π), and the paper's predicted πc matches the empirical knee on all 96 settings within a pre-registered half-decade tolerance.
- A certified admission gate on a frozen reserve removes the phase transition: wrong-admission intensity is capped independently of dictionary state, contamination rate, and even an adaptive adversary, and the gated detector holds FPR near the nominal level.
- No label-free procedure can simultaneously control FPR under benign drift and retain full power under tail-like contamination; the closed-form ceiling κ = α/(π + α(1−π)) is matched in shape by CDC, which certifies all tested drift-affected cells at mean FPR ≈0.04 while retaining about two-thirds of oracle TPR.
Where Pith is reading between the lines
- If the near-universal slope ≈0.947 is a signature of the fixed-fraction, bank-relative admission protocol rather than of any particular encoder, then any detector built on that protocol inherits near-critical fragility; the safe design is to separate the evidence channel from the object it updates, not to tune admission fractions.
- A testable extension suggested by the theory: increase burst length or burst-phase autocorrelation and measure whether the per-π kernel slope rises; if it does, the critical contamination rate is time-structure-dependent and the i.i.d.-window certificate would need the block-dependent slack in practice.
- The impossibility ceiling can be read operationally: a deployment must choose explicitly whether to sacrifice FPR validity or detection power on confusable tails; CDC takes the conservative branch, and any method that claims both without labels is logically excluded.
- The paper's own loose dependent-window bound suggests a concrete improvement: a block-bootstrap quantile certificate matched to measured burst lengths could close the worst-case slack while preserving validity.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies test-time adaptive OOD detectors that update a memory bank or dictionary from an unlabelled stream. It models bank impurity as a generalized Pólya urn and proves (Theorem 3) almost-sure convergence to the stable zeros of a scalar mean field h(ρ)=q(ρ)-ρ, where q is the false-admission kernel. For affine kernels q(ρ)=a+bρ, the slope b is interpreted as a reproduction number: b<1 gives an equilibrium ρ*=a/(1-b) and b≥1 gives complete poisoning (Corollary 4). The paper reports that across 96 settings and four encoder families the measured kernel is affine with slope just below one (b≈0.947), so the detector class is 'near-critical by design'; it also reports a predicted collapse threshold matching the empirical knee on 96/96 settings. A gated admission protocol, WARDEN, is proved to sever the feedback loop and bound wrong-admission intensity independently of contamination and even under an adaptive adversary (Lemma 8, Theorem 9, Corollary 10). For the complementary static-calibration failure under drift, the paper proposes CDC, a label-free quantile-correction procedure with finite-sample FPR certificates (Theorems 11-13), and proves a two-world indistinguishability theorem (Theorem 15) giving a worst-case power ceiling κ=α/(π+α(1-π)) for any label-free procedure. Extensive experiments (9,264 streaming cells plus a one-shot held-out replication) support the empirical claims.
Significance. If the central claims hold, the paper is a substantial contribution: it gives a predictive, testable theory of self-poisoning in adaptive OOD detection, a certified design that provably removes the collapse, and a clean impossibility boundary for label-free calibration. The urn/stochastic-approximation treatment is appropriate, the proof sketches are coherent, and the paper is unusually careful about what it does not claim (e.g., no AUROC improvement from adaptation or from the dictionary channel). The empirical program is extensive and includes a held-out one-shot replication, which is a real strength. The impossibility theorem with the explicit κ ceiling is elegant and appears correct. However, the load-bearing empirical validation of the kernel assumption and of the predicted threshold is weaker than the presentation suggests, and one internal numerical claim is contradicted by the paper's own fitted coefficients. These issues are fixable but require substantial revision.
major comments (3)
- [Section 7 / Finding 2; Section 8(i)] The validation of Assumption 1 is not sufficient for the role it plays. The binned regression of false-admission proportion on ρ (Fig. 3, R²≥0.996) estimates one projection of the conditional law, but Assumption 1 requires E[w_t/a_t | F_{t-1}] = q(ρ_{t-1}) with no other dependence. In bursty streams, ρ_{t-1} is confounded with burst phase, batch composition, and the finer state of the bank; Section 8(i) concedes that 'richer state (e.g. cluster structure of the poison)' is unmodeled. A high-R² affine fit of the projection is consistent with the one-dimensional mean field but does not establish it, so the reproduction number b, the predicted critical rate π_c, and the 'near-critical by design' claim are not transportable to a new domain without re-deriving or directly testing the conditional-expectation form. I suggest a concrete diagnostic: include burst-phase indicators or a small set o
- [Section 4.2 and Appendix D, measured kernel table] The statement 'every measured setting has R0<1 but a≥1−b, so ρ*=1' is contradicted by the paper's own pooled fits. The Appendix D table gives (a,b): vision (0.045,0.947), vision-foundation (0.046,0.947), document (0.066,0.944), text (0.045,0.947), all (0.048,0.947). For the first three of these and for the aggregate, a<1−b (e.g., 0.045<0.053, 0.048<0.053), so Corollary 4 predicts ρ*=a/(1−b)≈0.85–0.91, not complete poisoning. Only the document family satisfies a≥1−b. The claim that the complete-poisoning branch is the one operating in all settings is therefore not supported by the reported coefficients. This is a load-bearing numerical inconsistency for the 'complete poisoning' version of the collapse claim, though the predicted equilibria are still high and consistent with impurity >0.9. Please correct the inequality, report per-setting (a,b) if that is what the sentence refers to, and r
- [Section 7 / Finding 2 and Section 4.2, operational π_c] The 96/96 match between predicted and empirical π_c is presented as a confirmation of the sharp-threshold theory, but as described it is a self-consistency check. The π-conditional coefficients (a(π),b(π)) are fitted on the same streaming runs whose empirical knees are then located; fitting a curve and reading off its own implied crossing does not provide out-of-sample evidence for the threshold law. To make the threshold claim load-bearing, please split the data: estimate the kernel on development streams (or on the exploration seeds) and predict the collapse knee on the held-out seeds/streams, and report the tolerance and the number of settings for which the knee is identifiable. If the match is only in-sample, say so explicitly and rephrase 'predicted threshold matches' accordingly.
minor comments (5)
- [Section 5, Theorem 13] The 'deterministic interval' [τ*, F^{-1}_{mix}(1−α(1−π̂_up)+2ε_n)] actually depends on the random π̂_up. Please rephrase as 'fixed interval determined by the realized π̂_up' or state the high-probability event on which the density bound is used.
- [Section 4.2] The phrase 'pre-registered half-decade tolerance' for the π_c match is vague. If there is a preregistration document, cite it; otherwise use a less loaded term such as 'prespecified tolerance' and define it precisely.
- [Section 5, CDC algorithm] In Step 2, the notation F̂^{-1}_{mix}(p) is undefined for p>1. The text says to set τ̂=+∞, but the definition of the empirical quantile should be stated for all p∈[0,1] explicitly.
- [Abstract/Introduction] The abstract's 'complete possibility/impossibility characterization' overstates the scope given that the theory relies on Assumption 1 and the CDC adaptive tier relies on the unverifiable condition (A2). The body is appropriately nuanced; the abstract and introduction should be toned down to match.
- [Appendix D.2] The adversarial stress test is described as 'gray-box' and 'feature-space mimicry'; the caption and text should make clear this is not a gradient-based input-space attack and that the proof of Corollary 10, while unconditional, is not a statement about empirical attack optimization. The limitation is already in Section 8(vi), but the appendix could be more explicit.
Circularity Check
The 'predicted threshold' is a self-consistency check on the fitted kernel; the severed-feedback and impossibility theorems are independent.
specific steps
-
fitted input called prediction
[Section 4.2 (operational critical rate) and Section 7, Finding 2]
"The operational critical rate, the quantity we measure, is πc :=inf{π:ρ ∗(π)≥1/2}, which under Corollary 4 is determined by the measured kernel coefficients. ... Empirically the affine form of Assumption 1 is not merely convenient. The measured kernel fits with R2 ≥0.996 across all four encoder families with slope R0 ≈0.947 just below critical (Section 7), and ρ ∗ =a/(1−b) sits at the saturation boundary. ... The predicted πc lands within the pre-registered half-decade tolerance of the empirical knee on 96/96 knee-settings."
The 'prediction' of πc is computed from the per-π affine kernel parameters (a(π), b(π)) that were fitted on the very same streaming runs used to locate the empirical knee. The predicted equilibrium ρ*=a/(1−b) is likewise evaluated against the same admission events from which q(ρ)=a+bρ was estimated. Thus the 96/96 agreement is a model-consistency / fixed-point check, not an out-of-sample falsification: if the fitted curve describes the conditional mean, then solving q(ρ)=ρ will approximately reproduce the average impurity of the same data. This is pattern 2, fitted input renamed as prediction. The underlying conditional theorem (Theorem 3 / Corollary 4) is not circular, but the paper's headline empirical validation of the 'sharp threshold' is.
full rationale
The central dynamical theorem is a conditional mathematical result: given Assumption 1 (a scalar Lipschitz kernel q(ρ) and predictable admission sizes), the recursion is a generalized Pólya urn and converges to the zeros of h(ρ)=q(ρ)−ρ. This part is self-contained and not circular. Similarly, Theorem 9 and Corollary 10 derive the severed-feedback guarantee from external e-BH validity and exchangeability, and Theorem 15 is a genuine two-world indistinguishability construction with a closed-form ceiling; none of these reduce to fitted inputs or self-citations. The circularity is confined to how the theory is empirically validated: the 'predicted threshold' πc and the 'near-critical by design' slope R0≈0.947 are read off the same kernel fits and the same runs whose collapse they are said to predict. This is a self-consistency check, and the paper's own Section 8 limitation—'Assumption 1 posits a one-dimensional state. The affine fit is excellent empirically but the reduction is a modelling step, and richer state (e.g. cluster structure of the poison) is future work'—further undercuts transportability of the fitted reproduction number to new domains. The score reflects this partial circularity in the headline threshold prediction, while recognizing that the severed-feedback certificates and the impossibility theorem stand on independent grounds.
Axiom & Free-Parameter Ledger
free parameters (5)
- admission kernel intercept a =
0.045–0.066 pooled; per-π 0.94→0.02
- admission kernel slope b = R0 =
0.944–0.947 pooled; per-π 0.03→0.90; per-setting [0.929,0.959]
- e-value calibration exponent a_ev =
0.1
- e-BH level δ =
0.1
- Storey parameter λ =
1/2
axioms (8)
- domain assumption True-ID test points are exchangeable with the frozen reserve under a predictable scorer (Prop. 7, Lemma 8, Cor. 10).
- domain assumption Calibration windows are i.i.d. draws from Fmix (Theorem 11); Proposition 14 relaxes to independent L-blocks with effective sample size n/L.
- domain assumption Condition (A2): (1−π)ε_dr(R) ≥ π κ_λ(R) for the adaptive tier of CDC (Theorem 12).
- domain assumption Test-ID scores stochastically dominate train-ID scores (Theorem 12b).
- domain assumption Assumption 2: noise floor σ0²>0, differentiability near unstable zeros, admissions in a positive fraction of batches.
- standard math Background stochastic-approximation theorems (Benaïm 1999 ODE method; Pemantle 2007) for generalized Pólya urns.
- domain assumption Saturating-kernel regularity conditions (i)–(v) in Proposition 5 (sigmoidal, steep, non-degenerate tangencies).
- domain assumption Feature whitening is fit on train-ID only; settings are screened to base separability in [0.62,0.995] AUROC.
read the original abstract
Test-time adaptive out-of-distribution (OOD) detectors update a memory bank from the unlabelled stream. We show this adaptation obeys a provable dynamical law. Modelling bank impurity as a generalized P\'olya urn, we prove almost-sure convergence to a mean-field equilibrium whose slope acts as a reproduction number. Below one, impurity stays benign. Above one, the bank is fully poisoned and the detector collapses. The measured admission kernel is affine ($R^2 \ge 0.996$) with slope just below one in every encoder family (a protocol signature), so this detector class is near-critical by design, and across 96 settings the predicted threshold matches the empirical collapse, where ungated dictionaries lose up to $0.163$ AUROC. We then prove that a certified admission gate, reading only a frozen reserve, severs the feedback loop and removes the transition at every contamination rate, even adversarially, while controlling false positives label-free. For the complementary static-calibration failure under drift we give CDC, which restores nominal FPR label-free on all tested drift-affected cells. Finally we prove a two-world impossibility theorem. Drift and contamination are indistinguishable without labels, forcing a closed-form power ceiling our procedure approaches. Together these give a complete possibility/impossibility characterization of label-free adaptive OOD detection.
Figures
Reference graph
Works this paper leans on
-
[3]
arXiv:2502.14604. E. Dohmatob, Y . Feng, A. Subramonian, and J. Kempe. Strong model collapse. InInternational Conference on Learning Representations (ICLR),
-
[4]
arXiv:2410.04840. X. Du, Z. Fang, I. Diakonikolas, and Y . Li. How does unlabeled data provably help out-of- distribution detection? InInternational Conference on Learning Representations (ICLR),
-
[5]
arXiv:2402.03502. Z. Fang, Y . Li, J. Lu, J. Dong, B. Han, and F. Liu. Is out-of-distribution detection learnable? In Advances in Neural Information Processing Systems (NeurIPS),
-
[7]
arXiv:2404.01413. P. J. Huber. A robust version of the probability ratio test.The Annals of Mathematical Statistics, 36 (6):1753–1758,
-
[8]
PMLR 162; arXiv:2202.03299. M. Kloft and P. Laskov. Security analysis of online centroid anomaly detection.Journal of Machine Learning Research, 13:3681–3724,
-
[9]
arXiv:2501.02514. 27 VISHNUBINDUBALACHANDRAN A. Marandon, L. Lei, D. Mary, and E. Roquain. Adaptive novelty detection with false discovery rate guarantee.The Annals of Statistics, 52(1):157–183,
-
[12]
arXiv:2410.04682. Y . Sun, Y . Ming, X. Zhu, and Y . Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning (ICML), pages 20827–20840,
-
[13]
PMLR 162; arXiv:2204.06507. C. Wang. When does trimming help conformal prediction? a retained-law diagnostic under calibration contamination,
-
[15]
arXiv:2502.08539; to appear, Statistics & Probability Letters. R. Wang and A. Ramdas. False discovery rate control with e-values.Journal of the Royal Statistical Society, Series B, 84(3):822–852,
-
[16]
arXiv:2503.10468. F. F. Yilmaz and R. Heckel. Test-time recalibration of conformal predictors under distribution shift based on unlabeled examples,
-
[17]
arXiv:2210.04166. J. Zhang, J. Yang, P. Wang, H. Wang, Y . Lin, H. Zhang, Y . Sun, X. Du, Y . Li, Z. Liu, Y . Chen, and H. Li. OpenOOD v1.5: Enhanced benchmark for out-of-distribution detection.Journal of Data-centric Machine Learning Research (DMLR),
-
[18]
arXiv:2306.09301. Y . Zhang, X. Wang, T. Zhou, K. Yuan, Z. Zhang, L. Wang, R. Jin, and T. Tan. Model-free test-time adaptation for out-of-distribution detection,
-
[19]
arXiv:2311.16420. 28
-
[2010]
arXiv:1002.3716. I. Shumailov, Z. Shumaylov, Y . Zhao, N. Papernot, R. Anderson, and Y . Gal. AI models collapse when trained on recursively generated data.Nature, 631:755–759,
-
[2022]
arXiv:2210.14707. M. Gerstgrasser, R. Schaeffer, A. Dey, R. Rafailov, H. Sleight, J. Hughes, T. Korbak, R. Agrawal, D. Pai, A. Gromov, D. A. Roberts, D. Yang, D. L. Donoho, and S. Koyejo. Is model col- lapse inevitable? breaking the curse of recursion by accumulating real and synthetic data,
-
[2023]
arXiv:2306.05401. H. Renlund. Generalized P ´olya urns via stochastic approximation,
-
[2024]
arXiv:2310.00429. G. Blanchard, G. Lee, and C. Scott. Semi-supervised novelty detection.Journal of Machine Learning Research, 11(99):2973–3009,
-
[2025]
PMLR 267; arXiv:2502.04807. S. Bates, E. Cand`es, L. Lei, Y . Romano, and M. Sesia. Testing for outliers with conformal p-values. The Annals of Statistics, 51(1):149–178,
-
[2026]
arXiv:2605.06204. H. Wang, S. Dandapanthula, and A. Ramdas. Anytime-valid FDR control with the stopped e-BH procedure,
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.