Pith. sign in

REVIEW 3 major objections 4 minor 42 references

Online Conformal Selection with Accept-to-Reject Changes

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A method for online candidate selection that makes selections irreversible while provably controlling the false discovery rate at every timestep.

desk verdict A plausible and genuinely new combination of conformal selection and online BH that deserves a careful referee; the FDR proof is in the appendix and there is a small notation bug to fix. read the letter →

arxiv 2508.13838 v1 pith:RYSUZANP submitted 2025-08-19 stat.ML cs.LG

Kangdao Liu , Huajun Xi , Chi-Man Vong , Hongxin Wei This is my paper · ORCID
classification stat.MLcs.LG
keywords conformalselectiononlineFDRcontrolaccept-to-rejectchangesBenjamini-Hochbergfalsediscoveryratep-valuesirreversibledecisionsmultivariate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper addresses online candidate selection in which a selection, once made, is irreversible: a candidate not yet selected can be reconsidered later, but an accepted candidate cannot later be dropped. It extends conformal selection, a distribution-free framework for choosing candidates with false-discovery-rate (FDR) control, to this Accept-to-Reject Changes setting. The proposed procedure, OCS-ARC, forms conformal p-values from a monotone non-conformity score and feeds them into the online Benjamini-Hochberg algorithm. The central result is that under i.i.d. calibration and test data, or under a stated conditional exchangeability condition, the expected fraction of wrong selections stays at or below the nominal level q at every timestep, and the selection sets are nested so that earlier picks are never reversed.

What carries the argument

The load-bearing object is the online Benjamini-Hochberg procedure applied to conformal p-values. Online BH produces a nested sequence of selection sets: at time t it forms the threshold k*_t = max{k in [t] : sum_{j<=t} 1{p_j <= k · q · γ_j} >= k} and selects R_t = {j <= t : p_j <= k*_t · q · γ_j}; because these sets only grow as t increases, the Accept-to-Reject Changes property holds. The monotone non-conformity score (V(x,y) <= V(x,y') whenever y <= y') is what keeps the practical conformal p-value super-uniform even though the true response is replaced by the threshold c_t; without this monotonicity, the FDR proof collapses.

What would settle it

Under i.i.d. data with a monotone score, compute the empirical FDR after many replicates of OCS-ARC at a fixed timestep with q=0.1; if the empirical FDR exceeds 0.1 by more than simulation noise, Theorem 6 is false, whereas a drift test would only show the limitation the authors already flag in Remark 8.

Watch

Extended reading notes

Core claim

The central claim is that online conformal selection with irreversible decisions can be achieved by combining conformal p-values with the online Benjamini-Hochberg (BH) procedure. For any monotone non-conformity score function, the practical conformal p-value p_t, computed with the threshold c_t in place of the unobserved response, remains stochastically dominated by the uniform distribution. Theorem 6 then shows that OCS-ARC controls FDR_t <= q at any timestep under i.i.d. data, and Theorem 7 extends the guarantee to a conditional exchangeability condition. The proof does not require the positive-regression-dependence condition usually assumed by online BH, because it exploits the marginal super-uniformity of conformal p-values. The paper also claims a multivariate extension, mOCS-ARC, for regional monotone scores.

Load-bearing premise

The entire FDR guarantee rests on the calibration set and the incoming test samples being exchangeable (in particular, i.i.d.); if the test stream drifts, the conformal p-values stop being super-uniform and the false discovery rate can exceed the nominal level.

Editorial extensions

If this is right

  • Because the selection regions are nested, a screening pipeline that follows OCS-ARC never has to retract an earlier selection.
  • At any timestep t, the expected fraction of wrong selections among those made so far stays at or below q, so rolling decisions can be quoted with the same FDR guarantee.
  • The guarantee does not require feedback on whether past selections were correct, allowing use in settings where outcomes are delayed or never observed.
  • In the paper's experiments, OCS-ARC selects far more true candidates than the simple online Bonferroni rule while keeping FDR below the target.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to feed weighted or adaptive conformal p-values into the same nested thresholding scheme to cope with covariate shift, a direction the paper does not take.
  • The decay sequence γ_t controls a power-versus-conservatism trade-off; treating it as a tunable design parameter for specific arrival patterns is a plausible follow-up.
  • If outcome feedback eventually arrives, the same ARC-selection rule could double as an online monitoring certificate for a deployed process, though the paper assumes no feedback.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The manuscript proposes Online Conformal Selection with Accept-to-Reject Changes (OCS-ARC), a sequential candidate-selection procedure in which the selected set is required to be nested over time: previously selected items are never deselected, while non-selected items may later be selected. The method computes conformal p-values from a fixed calibration set using a monotone non-conformity score and then applies the online Benjamini-Hochberg procedure of Fischer, Xu, and Ramdas (2024). The main theoretical results are Theorem 5 (ARC property), Theorem 6 (FDR_t <= q at every timestep for i.i.d. calibration and test data), Theorem 7 (an exchangeability-based extension), and Theorem 11 (extension to multivariate responses). Experiments cover synthetic regression data, recruitment screening, drug property prediction, and LLM-based question answering, comparing OCS-ARC with an online Bonferroni baseline.

Significance. If the FDR guarantee is correct, the contribution is useful and timely: it is the first extension of conformal selection to an online Accept-to-Reject Changes setting, and the algorithm is simple enough to combine with any pre-trained black-box predictor. The ARC property itself follows from a clean monotonicity argument, and the experimental evaluation is broad, including LLM decision-making. The method is also transparent: the gamma_t sequence is a user-specified input and the calibration set is fixed in advance, so the main claim is not fitted to the data. The central obstacle is that the FDR proof is not present in the manuscript text provided for review and the printed algorithm contains an undefined alpha in the threshold definition; these issues must be resolved before the main claim can be accepted.

major comments (3)
  1. [Algorithm 1, line 5] The threshold k*_t is defined using a symbol alpha that appears nowhere in the Require list, in Theorem 6, or in the surrounding text, while line 6 and Theorem 6 use the nominal level q. If alpha is intended to equal q, the algorithm must state this; otherwise the printed selection rule is not the rule for which FDR control is claimed, and the ARC monotonicity argument would need to be re-examined under two different levels. This is a load-bearing inconsistency and must be fixed.
  2. [Theorem 6 and Remark 9] The main text states Theorem 6 but gives no proof, referring the reader to Appendix C.2. Because Remark 9 explicitly acknowledges that conformal p-values do not satisfy the conditional superuniformity or PRDS conditions under which the standard online BH theorem is usually analyzed, the proof that online BH controls FDR for these dependent conformal p-values is the central technical claim of the paper. The manuscript as provided does not contain that proof, so the main guarantee cannot be verified. The appendix should be included in the reviewed version, or the proof should be summarized in the main text.
  3. [Theorem 7] The exchangeability condition is stated as exchangeability of {V_1,...,V_n,V_{n+t}} conditional on {V_hat_{n+t'} : t' != t}, with an additional no-ties assumption on an undefined sequence {V_{t'}}. It is unclear what the conditioning sigma-algebra is, whether the conditioning object is the estimated non-conformity score at other test times, and how such a condition arises from a natural data-generating process. As written, the theorem cannot be checked and its scope is unclear; please define all symbols and give a concrete example where the condition holds.
minor comments (4)
  1. [Algorithm 1, line 4] The construction of p_t as in Eq. (2) requires drawing the tie-randomization variable U_t; Algorithm 1 should state this explicitly so that the procedure is fully reproducible.
  2. [Experiments, Baseline] The OB baseline is defined only with the CLIP score, but the text later says that OCS-ARC outperforms OB 'under all scores'; the comparison is fair for CLIP but should be clarified or extended for RES.
  3. [Figure 1 and Section 'Method'] The terms 'Reject-to-Accept Changes' and 'Accept-to-Reject Changes' are easy to confuse; since the figure counts the prohibited selected-to-deselected transitions, a sentence explicitly connecting 'reject' to 'selection' in the caption would improve readability.
  4. [Problem Setup] The notation for the predefined threshold is inconsistent: c_i appears in the formal definitions of FDR and power, while c_t is used in the method description; please unify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the FDR guarantee is an unverified but non-circular application of external conformal p-value and online BH results.

full rationale

The paper's central claim (Theorem 6) is that OCS-ARC controls FDR_t <= q under i.i.d. calibration and test data. Nothing in the visible derivation makes this claim true by construction: the conformal p-values come from the external conformal selection framework of Jin and Candes, the online BH rule comes from Fischer, Xu, and Ramdas, and the user-specified sequence {gamma_t} is an input rather than a fitted parameter. The experimental decay coefficient r=0.99 appears only in experiments and does not enter the theorem. The authors' own prior works are cited only in related-work and background contexts, not as the load-bearing justification for the FDR control, so those self-citations are not circular. The explicit limitation in Remark 8 that the guarantee does not hold under distribution shift is an honest scope restriction, not a hidden assumption that smuggles in the conclusion. Two concerns remain, but they are verification/correctness concerns rather than circularity: the proof of Theorem 6 is delegated to Appendix C.2, which is not available in the provided text, and Algorithm 1 line 5 uses an undefined alpha while line 6 and Theorem 6 use q. Even if alpha must equal q for the theorem to apply to the printed algorithm, that is an internal consistency gap, not a case of a fitted parameter being renamed a prediction or a target result being assumed by definition. No circular step can be exhibited from the quoted equations, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard conformal inference and online multiple testing results, plus the exchangeability/stationarity assumption and the monotonicity of the score function. No free parameters are fitted to data in the theoretical result; the decay coefficient r affects only the experimental instantiation of the gamma sequence, not the validity of the theorem.

assumptions (5)
  • domain assumption The calibration and test data are i.i.d. (or exchangeable), so the conformal scores are interchangeable.
    Theorem 6/7 state this; if violated (distribution shift), conformal p-values are not super-uniform and FDR control is not guaranteed (Remark 8).
  • domain assumption The non-conformity score function V is monotone in y (or regional monotone for multivariate).
    Definition 4 requires V(X,y) <= V(X,y') for y <= y', which makes the practical p-value conservative under the null.
  • domain assumption The gamma sequence satisfies sum_{t=1}^T gamma_t <= 1.
    Required by the online BH procedure (Algorithm 1); the experiments use gamma_t = r^t (1-r)/r which sums to <=1.
  • domain assumption The calibration set is fixed and independent of the test data.
    Assumed in the problem setup (Section Method); the p-values are computed from D_cal only.
  • standard math Prior results that conformal p-values are super-uniform under the null and online BH controls FDR under independence (or PRDS).
    The proof builds on Bates et al. 2023, Jin and Candes 2023b, and Fischer et al. 2024.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Online Conformal Selection with Accept-to-Reject Changes." pith.science (2026). https://pith.science/paper/RYSUZANP

@misc{pith2026250813838,
  author       = {Pith},
  title        = {Pith review of: Online Conformal Selection with Accept-to-Reject Changes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RYSUZANP}},
  note         = {Machine review of arXiv:2508.13838}
}
read the original abstract

Selecting a subset of promising candidates from a large pool is crucial across various scientific and real-world applications. Conformal selection offers a distribution-free and model-agnostic framework for candidate selection with uncertainty quantification. While effective in offline settings, its application to online scenarios, where data arrives sequentially, poses challenges. Notably, conformal selection permits the deselection of previously selected candidates, which is incompatible with applications requiring irreversible selection decisions. This limitation is particularly evident in resource-intensive sequential processes, such as drug discovery, where advancing a compound to subsequent stages renders reversal impractical. To address this issue, we extend conformal selection to an online Accept-to-Reject Changes (ARC) procedure: non-selected data points can be reconsidered for selection later, and once a candidate is selected, the decision is irreversible. Specifically, we propose a novel conformal selection method, Online Conformal Selection with Accept-to-Reject Changes (dubbed OCS-ARC), which incorporates online Benjamini-Hochberg procedure into the candidate selection process. We provide theoretical guarantees that OCS-ARC controls the false discovery rate (FDR) at or below the nominal level at any timestep under both i.i.d. and exchangeable data assumptions. Additionally, we theoretically show that our approach naturally extends to multivariate response settings. Extensive experiments on synthetic and real-world datasets demonstrate that OCS-ARC significantly improves selection power over the baseline while maintaining valid FDR control across all examined timesteps.

Figures

Figures reproduced from arXiv: 2508.13838 by the authors.

Figure 1
Figure 1. Comparison of conformal selection (CS) with OCS-ARC. This experiment measures the total number of reject-to-accept samples for both methods on 500 sequen￾tially arriving data points from a synthetic dataset across three models. Results are averaged over 30 independent runs. An ARC procedure allows non-selected data points to be reconsidered for selection later, and once a point is selected, the decision is final and… view at source ↗
Figure 2
Figure 2. Synthetic data experiments with varying noise levels across different data-generating processes. The target FDR is set to 0.10, and gradient boosting is used to fit the regression model across all configurations. Subplots (a) and (b) present results for simulation setting 1 (CLIP, RES, and OB, respectively), while subplots (c) and (d) correspond to simulation setting 2. Experiments We conduct a comprehensive evaluat… view at source ↗
Figure 3
Figure 3. Results for candidate screening in recruitment. The target FDR is set to 0.20, and gradient boosting is used to fit the classifier. FDPt and Powert are reported over 100 independent runs at timesteps 100, 200 and 300. the nominal level 0.1, whereas the baseline exhibits a larger deviation from it. Among the two non-conformity score func￾tions, CLIP consistently demonstrates a higher realized FDRt across all settings… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Results for the application of OCS-ARC to question answering with LLMs on the TriviaQA Dataset. Experiments are conducted using LLaMA-2-13B-chat, with the target FDR varied from 0 to 0.80 in fixed steps over 25 increments. FDPt and Powert are reported over 100 independ…
Figure 5
Figure 5. Figure 5: Results of parameter sensitivity analyses, including (a) the effect of varying the decay coefficient r, and (b) the effect of changing the size of the calibration set. The target FDR is fixed at 0.10, and gradient boosting is used to fit the regression model. Parameter…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 30 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    N.; Barber, R.; and Bates, S

    Angelopoulos, A. N.; Barber, R.; and Bates, S. 2024. Online conformal prediction with decaying step sizes. In Forty-first International Conference on Machine Learning

    work page 2024

  4. [4]

    N.; Bates, S.; Jordan, M

    Angelopoulos, A. N.; Bates, S.; Jordan, M. I.; and Malik, J. 2021. Uncertainty Sets for Image Classifiers using Conformal Prediction. In 9th International Conference on Learning Representations

    work page 2021

  5. [5]

    Bai, T.; Zhao, Y.; Yu, X.; and Yang, A. Y. 2025. Multivariate conformal selection. International Conference of Machine Learning

    work page 2025

  6. [6]

    Bao, Y.; Huo, Y.; Ren, H.; and Zou, C. 2024 a . Cap: A general algorithm for online selective conformal prediction with fcr control. arXiv preprint arXiv:2403.07728

    arXiv 2024

  7. [7]

    Bao, Y.; Huo, Y.; Ren, H.; and Zou, C. 2024 b . Selective conformal inference with false coverage-statement rate control. Biometrika

    work page 2024

  8. [8]

    Bates, S.; Cand \`e s, E.; Lei, L.; Romano, Y.; and Sesia, M. 2023. Testing for outliers with conformal p-values. The Annals of Statistics

    2023

Show all 42 references
  1. [9]

    Benjamini, Y.; and Hochberg, Y. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological)

    1995

  2. [10]

    Benjamini, Y.; and Yekutieli, D. 2001. The control of the false discovery rate in multiple testing under dependency. Annals of statistics

    2001

  3. [11]

    Bhatnagar, A.; Wang, H.; Xiong, C.; and Bai, Y. 2023. Improved Online Conformal Prediction via Strongly Adaptive Online Learning. In International Conference on Machine Learning

    2023

  4. [12]

    El Kharoua, R. 2024. Predicting Hiring Decisions in Recruitment Data. Retrieved July 25, 2025, from https://www.kaggle.com/datasets/rabieelkharoua/predicting-hiring-decisions-in-recruitment-data

    2024

  5. [13]

    Fischer, L.; Bofill Roig, M.; and Brannath, W. 2024. The online closure principle. The Annals of Statistics

    2024

  6. [14]

    Fischer, L.; Xu, Z.; and Ramdas, A. 2024. Online generalizations of the e-BH and BH procedure. arXiv preprint arXiv:2407.20683

    2024

  7. [15]

    P.; and Stine, R

    Foster, D. P.; and Stine, R. A. 2008. -investing: a procedure for sequential control of expected false discoveries. Journal of the Royal Statistical Society Series B: Statistical Methodology

    2008

  8. [16]

    Gibbs, I.; and Candes, E. 2021. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems

    2021

  9. [17]

    Gui, Y.; Jin, Y.; and Ren, Z. 2024. Conformal Alignment: Knowing When to Trust Foundation Models with Guarantees. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  10. [18]

    Heaslip, E. 2022. Ai tools for talent acquisition to help you hire. https://vervoe. com/ai-tools-for-talent-acquisition/

    2022

  11. [19]

    Hu, X.; and Lei, J. 2024. A two-sample conditional distribution test using conformal prediction and weighted rank sum. Journal of the American Statistical Association, 119(546): 1136--1154

    2024

  12. [20]

    Huang, J.; Xi, H.; Zhang, L.; Yao, H.; Qiu, Y.; and Wei, H. 2024. Conformal Prediction for Deep Classifier via Label Ranking. In Forty-first International Conference on Machine Learning

    2024

  13. [21]

    Javanmard, A.; and Montanari, A. 2018. Online rules for control of false discovery rate and false discovery exceedance. The Annals of statistics

    2018

  14. [22]

    Jin, Y.; and Cand \`e s, E. J. 2023 a . Model-free selective inference under covariate shift via weighted conformal p-values. arXiv preprint arXiv:2307.09291

    2023 arXiv

  15. [23]

    Jin, Y.; and Cand \`e s, E. J. 2023 b . Selection by prediction with conformal p-values. Journal of Machine Learning Research

    2023

  16. [24]

    Jin, Y.; and Ren, Z. 2025. Confidence on the focal: Conformal prediction with selection-conditional coverage. Journal of the Royal Statistical Society Series B: Statistical Methodology

    2025

  17. [25]

    S.; and Zettlemoyer, L

    Joshi, M.; Choi, E.; Weld, D. S.; and Zettlemoyer, L. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551

    2017 arXiv

  18. [26]

    Lei, J.; and Wasserman, L. 2014. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Society Series B: Statistical Methodology

    2014

  19. [27]

    Liu, K.; Sun, T.; Zeng, H.; Zhang, Y.; Pun, C.-M.; and Vong, C.-M. 2025. Spatial-aware conformal prediction for trustworthy hyperspectral image classification. IEEE Transactions on Circuits and Systems for Video Technology

    2025

  20. [28]

    Papadopoulos, H.; Proedrou, K.; Vovk, V.; and Gammerman, A. 2002. Inductive Confidence Machines for Regression. In Elomaa, T.; Mannila, H.; and Toivonen, H., eds., Machine Learning: ECML 2002, 13th European Conference on Machine Learning

    2002

  21. [29]

    Reddy, S.; Chen, D.; and Manning, C. D. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7: 249--266

    2019

  22. [30]

    Romano, Y.; Sesia, M.; and Cand \` e s, E. J. 2020. Classification with Valid and Adaptive Coverage. In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Proces...

    2020

  23. [31]

    Sale, Y.; and Ramdas, A. 2025. Online Selective Conformal Prediction: Errors and Solutions. arXiv preprint arXiv:2503.16809

    2025 arXiv

  24. [32]

    P.; McMasters, D

    Sheridan, R. P.; McMasters, D. R.; Voigt, J. H.; and Wildey, M. J. 2015. eCounterscreening: using QSAR predictions to prioritize testing for off-target activities and setting the balance between benefit and risk. Journal of Chemical Information and Modeling

    2015

  25. [33]

    Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi \`e re, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  26. [34]

    Vovk, V.; Gammerman, A.; and Saunders, C. 1999. Machine-learning applications of algorithmic randomness

    1999

  27. [35]

    Vovk, V.; Gammerman, A.; and Shafer, G. 2005. Algorithmic learning in a random world. Springer

    2005

  28. [36]

    Xi, H.; Huang, J.; Liu, K.; Feng, L.; and Wei, H. 2025 a . Does confidence calibration improve conformal prediction? Transactions on Machine Learning Research

    2025

  29. [37]

    Xi, H.; Liu, K.; Zeng, H.; Sun, W.; and Wei, H. 2025 b . Robust online conformal prediction under uniform label noise. arXiv preprint arXiv:2501.18363

    2025

  30. [38]

    Xu, Z.; and Ramdas, A. 2024. Online multiple testing with e-values. In International Conference on Artificial Intelligence and Statistics. PMLR

    2024

  31. [39]

    Zeng, H.; Liu, K.; Jing, B.; and Wei, H. 2025. Parametric scaling law of tuning bias in conformal prediction. International Conference of Machine Learning

    2025

  32. [40]

    Zhang, S.; Zhou, C.; Liu, Y.; Zhang, P.; Lin, X.; and Pan, S. 2025. Conformal Anomaly Detection in Event Sequences. In Forty-second International Conference on Machine Learning

    2025

  33. [41]

    Zhang, Y.; Jiang, H.; Ren, H.; Zou, C.; and Dou, D. 2022. AutoMS: automatic model selection for novelty detection with error rate control. Advances in Neural Information Processing Systems, 35: 19917--19929

    2022

  34. [42]

    Zhou, X.; Zeng, H.; Xia, X.; Jing, B.; and Wei, H. 2025. Semi-Supervised Conformal Prediction With Unlabeled Nonconformity Score. arXiv preprint arXiv:2505.21147

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.