{"id":"5068efd4-0579-4577-8caf-c8ac7976975e","arxiv_id":"2508.13838","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"OCS-ARC is the first conformal selection method for online Accept-to-Reject Changes settings, controlling FDR at every timestep by feeding conformal p-values into online Benjamini-Hochberg.","lead":"A new algorithm, OCS-ARC, adapts conformal selection to online settings where once a candidate is selected, the decision cannot be reversed, while still controlling the false discovery rate. It combines conformal p-values with an online Benjamini-Hochberg procedure and shows FDR control at every timestep for i.i.d. or exchangeable data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The FDR claim rests on an unverifiable appendix and on Algorithm 1's undefined α; unless α=q, the theorem does not apply to the printed algorithm.","rationale":"The reader's weakest assumption was distribution shift; Remark 8 already concedes this, so I do not treat it as the central flaw. My concern is different: the proof of the main theorem is absent from the visible text, and the algorithm has an undefined α. The method is otherwise credible: conditional on the calibration scores, conformal p-values are independent and satisfy P(H0_t, p_t≤u|cal)≤u; a leave-one-out weighted-BH argument with ∑γ_j≤1 then plausibly gives FDR≤q. Thus I do not see a fatal mathematical error, and I would keep the reader's CONDITIONAL verdict rather than reject. The concern is load-bearing because if α≠q or the appendix proof does not use the conditional joint-validity bound, the central FDR claim fails for the algorithm as written.","tokens_in":12400,"tokens_out":36825,"duration_ms":439443,"concrete_test":"Locate Appendix C.2 and re-derive Theorem 6's proof. Confirm it conditions on the calibration scores, uses the same q in both the k* condition and the selection rule (i.e., α=q in Algorithm 1), and establishes the weighted-BH bound E[V/R|cal]≤q∑γ_j using only joint superuniformity. Independently, rerun the Setting 1 synthetic experiment at q=0.1 with Algorithm 1 as printed but with α=0.05 in line 5; if FDR_t rises above q at t=600, the printed algorithm is not the one whose guarantee is proved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is Theorem 6: OCS-ARC controls FDR_t≤q for i.i.d. data. The main text gives no proof; Appendix C.2 is cited but cannot be inspected. The claimed novelty (Remark 9) is precisely that online BH normally needs conditional superuniformity and PRDS, while conformal p-values only give marginal joint validity P(p_t≤α, t∈H0_t)≤α and are dependent through the calibration set. Whether the weighted-BH leave-one-out bound survives this is the entire question, and the visible text does not settle it. In addition, Algorithm 1 line 5 computes k* with an undefined α, while line 6 and Theorem 6 use the nominal level q. If α is not q, the threshold in the k* condition differs from the selection threshold, so the FDR guarantee does not follow for the algorithm as printed. Remark 8 correctly disclaims distribution shift, so that is a scope limitation, not a contradiction; the unresolved issues are the absent proof and the α/q inconsistency.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Online Conformal Selection with Accept-to-Reject Changes (OCS-ARC), a sequential candidate-selection procedure in which the selected set is required to be nested over time: previously selected items are never deselected, while non-selected items may later be selected. The method computes conformal p-values from a fixed calibration set using a monotone non-conformity score and then applies the online Benjamini-Hochberg procedure of Fischer, Xu, and Ramdas (2024). The main theoretical results are Theorem 5 (ARC property), Theorem 6 (FDR_t <= q at every timestep for i.i.d. calibration and test data), Theorem 7 (an exchangeability-based extension), and Theorem 11 (extension to multivariate responses). Experiments cover synthetic regression data, recruitment screening, drug property prediction, and LLM-based question answering, comparing OCS-ARC with an online Bonferroni baseline.","tokens_in":12551,"tokens_out":9560,"duration_ms":101593,"significance":"If the FDR guarantee is correct, the contribution is useful and timely: it is the first extension of conformal selection to an online Accept-to-Reject Changes setting, and the algorithm is simple enough to combine with any pre-trained black-box predictor. The ARC property itself follows from a clean monotonicity argument, and the experimental evaluation is broad, including LLM decision-making. The method is also transparent: the gamma_t sequence is a user-specified input and the calibration set is fixed in advance, so the main claim is not fitted to the data. The central obstacle is that the FDR proof is not present in the manuscript text provided for review and the printed algorithm contains an undefined alpha in the threshold definition; these issues must be resolved before the main claim can be accepted.","major_comments":[{"comment":"The threshold k*_t is defined using a symbol alpha that appears nowhere in the Require list, in Theorem 6, or in the surrounding text, while line 6 and Theorem 6 use the nominal level q. If alpha is intended to equal q, the algorithm must state this; otherwise the printed selection rule is not the rule for which FDR control is claimed, and the ARC monotonicity argument would need to be re-examined under two different levels. This is a load-bearing inconsistency and must be fixed.","section":"Algorithm 1, line 5"},{"comment":"The main text states Theorem 6 but gives no proof, referring the reader to Appendix C.2. Because Remark 9 explicitly acknowledges that conformal p-values do not satisfy the conditional superuniformity or PRDS conditions under which the standard online BH theorem is usually analyzed, the proof that online BH controls FDR for these dependent conformal p-values is the central technical claim of the paper. The manuscript as provided does not contain that proof, so the main guarantee cannot be verified. The appendix should be included in the reviewed version, or the proof should be summarized in the main text.","section":"Theorem 6 and Remark 9"},{"comment":"The exchangeability condition is stated as exchangeability of {V_1,...,V_n,V_{n+t}} conditional on {V_hat_{n+t'} : t' != t}, with an additional no-ties assumption on an undefined sequence {V_{t'}}. It is unclear what the conditioning sigma-algebra is, whether the conditioning object is the estimated non-conformity score at other test times, and how such a condition arises from a natural data-generating process. As written, the theorem cannot be checked and its scope is unclear; please define all symbols and give a concrete example where the condition holds.","section":"Theorem 7"}],"minor_comments":[{"comment":"The construction of p_t as in Eq. (2) requires drawing the tie-randomization variable U_t; Algorithm 1 should state this explicitly so that the procedure is fully reproducible.","section":"Algorithm 1, line 4"},{"comment":"The OB baseline is defined only with the CLIP score, but the text later says that OCS-ARC outperforms OB 'under all scores'; the comparison is fair for CLIP but should be clarified or extended for RES.","section":"Experiments, Baseline"},{"comment":"The terms 'Reject-to-Accept Changes' and 'Accept-to-Reject Changes' are easy to confuse; since the figure counts the prohibited selected-to-deselected transitions, a sentence explicitly connecting 'reject' to 'selection' in the caption would improve readability.","section":"Figure 1 and Section 'Method'"},{"comment":"The notation for the predefined threshold is inconsistent: c_i appears in the formal definitions of FDR and power, while c_t is used in the method description; please unify.","section":"Problem Setup"}],"recommendation":"major_revision","confidential_remarks":"The main unresolved point is whether the deferred FDR proof actually handles the dependence structure of conformal p-values within the online BH procedure. The alpha/q inconsistency in Algorithm 1 is likely a typo, but it is load-bearing as printed. I would also like the authors to clarify, in the revision, how their weighted leave-one-out argument compares with the existing conformal BH proofs of Jin and Candes (2023) and with the online BH results of Fischer et al. (2024), so that the claimed novelty is precisely delineated. If the appendix proof is supplied and correct, the paper is publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper makes conformal selection monotone in time by feeding conformal p-values into the online BH procedure. That combination is new, it is directly relevant to applications like drug discovery and hiring where you cannot un-select, and the ARC property follows quickly from monotonicity of the selection threshold. The empirical work is fine: the baseline is a Bonferroni-style rule, and the power gain is large while FDR stays controlled. The sensitivity analysis is a nice touch.\n\nThe soft spots are real but manageable. The central claim, Theorem 6, says FDR_t ≤ q for i.i.d. data, and Theorem 7 extends to an exchangeability condition. Both proofs live in Appendix C, which is not part of this version, so I cannot verify the key step: online BH normally wants conditional superuniformity under the null and something like PRDS, while conformal p-values give only marginal validity P(p_t ≤ α, t ∈ H0_t) ≤ α and are dependent through the calibration set. The paper says a conditional-on-calibration argument handles this; that is plausible, but it is exactly the part that needs a referee's eyes. Do not take the guarantee on faith.\n\nThere is also a concrete typo in Algorithm 1: line 5 computes k*_t using an α that is never defined, while line 6 and Theorem 6 use q. If α ≠ q, the theorem does not apply to the algorithm as printed. That is trivial to fix but should be caught before publication.\n\nCredit where due: Remark 8 is an honest scope note—no overclaiming on distribution shift—and the writing is clear. The citation pattern is fine; the prior work on offline conformal selection and online BH is cited properly, and the novelty claim about being first in the online ARC setting looks correct.\n\nWho is this for? Anyone working on online FDR control or conformal selection. It is a subfield contribution, not a paradigm shift, but the problem is well motivated and the method is simple enough to be adopted. I would send it to a serious referee: the open question is purely whether the deferred proof is correct. If the appendix holds up, this is a solid paper; if not, the method remains a plausible heuristic but the main selling point disappears.","headline":"A plausible and genuinely new combination of conformal selection and online BH that deserves a careful referee; the FDR proof is in the appendix and there is a small notation bug to fix.","tokens_in":13121,"tokens_out":2911,"would_cite":true,"duration_ms":31298,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A method for online candidate selection that makes selections irreversible while provably controlling the false discovery rate at every timestep.","keywords":["conformal selection","online FDR control","accept-to-reject changes","online Benjamini-Hochberg","false discovery rate","conformal p-values","irreversible decisions","multivariate conformal selection"],"falsifier":"Under i.i.d. data with a monotone score, compute the empirical FDR after many replicates of OCS-ARC at a fixed timestep with q=0.1; if the empirical FDR exceeds 0.1 by more than simulation noise, Theorem 6 is false, whereas a drift test would only show the limitation the authors already flag in Remark 8.","tokens_in":12167,"feed_emoji":"🎯","tokens_out":6698,"duration_ms":62652,"temperature":0.7,"pith_summary":"This paper addresses online candidate selection in which a selection, once made, is irreversible: a candidate not yet selected can be reconsidered later, but an accepted candidate cannot later be dropped. It extends conformal selection, a distribution-free framework for choosing candidates with false-discovery-rate (FDR) control, to this Accept-to-Reject Changes setting. The proposed procedure, OCS-ARC, forms conformal p-values from a monotone non-conformity score and feeds them into the online Benjamini-Hochberg algorithm. The central result is that under i.i.d. calibration and test data, or under a stated conditional exchangeability condition, the expected fraction of wrong selections stays at or below the nominal level q at every timestep, and the selection sets are nested so that earlier picks are never reversed.","feed_headline":"A new online method keeps FDR capped and selections irreversible","feed_subtitle":"Once a candidate is picked it stays picked, while the false discovery rate stays at or below the target.","key_machinery":"The load-bearing object is the online Benjamini-Hochberg procedure applied to conformal p-values. Online BH produces a nested sequence of selection sets: at time t it forms the threshold k*_t = max{k in [t] : sum_{j<=t} 1{p_j <= k · q · γ_j} >= k} and selects R_t = {j <= t : p_j <= k*_t · q · γ_j}; because these sets only grow as t increases, the Accept-to-Reject Changes property holds. The monotone non-conformity score (V(x,y) <= V(x,y') whenever y <= y') is what keeps the practical conformal p-value super-uniform even though the true response is replaced by the threshold c_t; without this monotonicity, the FDR proof collapses.","core_discovery":"The central claim is that online conformal selection with irreversible decisions can be achieved by combining conformal p-values with the online Benjamini-Hochberg (BH) procedure. For any monotone non-conformity score function, the practical conformal p-value p_t, computed with the threshold c_t in place of the unobserved response, remains stochastically dominated by the uniform distribution. Theorem 6 then shows that OCS-ARC controls FDR_t <= q at any timestep under i.i.d. data, and Theorem 7 extends the guarantee to a conditional exchangeability condition. The proof does not require the positive-regression-dependence condition usually assumed by online BH, because it exploits the marginal super-uniformity of conformal p-values. The paper also claims a multivariate extension, mOCS-ARC, for regional monotone scores.","pith_inferences":["A natural extension is to feed weighted or adaptive conformal p-values into the same nested thresholding scheme to cope with covariate shift, a direction the paper does not take.","The decay sequence γ_t controls a power-versus-conservatism trade-off; treating it as a tunable design parameter for specific arrival patterns is a plausible follow-up.","If outcome feedback eventually arrives, the same ARC-selection rule could double as an online monitoring certificate for a deployed process, though the paper assumes no feedback."],"forward_implications":["Because the selection regions are nested, a screening pipeline that follows OCS-ARC never has to retract an earlier selection.","At any timestep t, the expected fraction of wrong selections among those made so far stays at or below q, so rolling decisions can be quoted with the same FDR guarantee.","The guarantee does not require feedback on whether past selections were correct, allowing use in settings where outcomes are delayed or never observed.","In the paper's experiments, OCS-ARC selects far more true candidates than the simple online Bonferroni rule while keeping FDR below the target."],"supporting_citations":[{"why":"Supplies conformal selection with conformal p-values and the monotone non-conformity score framework that OCS-ARC builds on.","marker":"Jin and Cand`es 2023b"},{"why":"Provides the online BH procedure and the γ_t sequence that OCS-ARC reuses for thresholding and nesting.","marker":"Fischer, Xu, and Ramdas 2024"},{"why":"Establishes the super-uniformity of conformal p-values that is the basis of the FDR proof.","marker":"Bates et al. 2023"},{"why":"Defines FDR and the BH procedure, the target guarantee that the online procedure extends.","marker":"Benjamini and Hochberg 1995"},{"why":"Introduces regional monotonicity for multivariate conformal selection, used to extend OCS-ARC to multivariate responses.","marker":"Bai et al. 2025"}],"fun_headline_variants":["Irreversible picks? OCS-ARC keeps FDR in check","Online conformal selection: lock in picks, cap false discoveries","OCS-ARC: irreversible selections, FDR always in check","Making selection irreversible: OCS-ARC controls FDR online","No take-backs in selection: OCS-ARC keeps FDR at target"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire FDR guarantee rests on the calibration set and the incoming test samples being exchangeable (in particular, i.i.d.); if the test stream drifts, the conformal p-values stop being super-uniform and the false discovery rate can exceed the nominal level.","fun_headline_variants_meta":{"raw":{"variants":["Irreversible picks? OCS-ARC keeps FDR in check","Online conformal selection: lock in picks, cap false discoveries","OCS-ARC: irreversible selections, FDR always in check","Making selection irreversible: OCS-ARC controls FDR online","No take-backs in selection: OCS-ARC keeps FDR at target"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001036,"raw_usage":{"total_tokens":4374,"prompt_tokens":974,"completion_tokens":3400,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":3307}},"tokens_in":590,"tokens_out":3400,"duration_ms":24588,"temperature":1.0,"reasoning_tokens":3307,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:11:43.882544+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Under i.i.d. data with a monotone score, compute the empirical FDR after many replicates of OCS-ARC at a fixed timestep with q=0.1; if the empirical FDR exceeds 0.1 by more than simulation noise, Theorem 6 is false, whereas a drift test would only show the limitation the authors already flag in Remark 8.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines FDR and the BH procedure, the target guarantee that the online procedure extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces regional monotonicity for multivariate conformal selection, used to extend OCS-ARC to multivariate responses."}],"review_version":2}