Pith. sign in

REVIEW 9 minor 1 cited by

ACS: An interactive framework for conformal selection

T0 review · 0 major / 9 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves that an analyst can adaptively reorder candidates, refit prediction models, and incorporate new labels during a selection procedure while the finite-sample false discovery rate stays below any preset level.

desk verdict ACS cleanly generalizes conformal selection to adaptive, human-in-the-loop selection with finite-sample FDR control, and the proof holds up; worth sending to serious review. read the letter →

arxiv 2507.15825 v1 pith:O4XVZZYK submitted 2025-07-21 stat.ME cs.LGstat.ML

Yu Gui , Ying Jin , Yash Nair , Zhimei Ren This is my paper
classification stat.MEcs.LGstat.ML MSC 62F0362G1062J1562L10
keywords adaptiveconformalselectionfalsediscoveryratemodel-freeinferenceinteractivedataanalysismultipletestingsequentialscreeningmartingale
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a screening procedure can be made interactive without losing error control. Its method, adaptive conformal selection (ACS), screens candidates one by one from least to most promising, and at every step it lets the analyst change the ordering rule—by refitting the model, switching models, emphasizing diversity, or using newly labeled data—provided the rule draws only on the information released by a fixed protocol. The protocol hides the labels and test/membership status of the unscreened candidates except through counts, which is what keeps the procedure honest. The paper proves a finite-sample guarantee: under joint exchangeability of labeled and test candidates with their property sets, the expected fraction of false discoveries in the final selection is below the chosen level $\alpha$. This matters because real screening workflows—drug discovery, output filtering from large language models, hiring—are naturally interactive, while previous conformal selection required the entire analysis plan to be fixed in advance.

What carries the argument

The load-bearing object is the filtration $\mathcal{F}_\ell$ defined in equation (1), which says exactly what the analyst may see when choosing $\pi(\ell+1)$: the full data and membership flags of screened units and of all non-null labeled units, the covariates of every unscreened unit, and only the counts $|N_\ell^-|$ and $|P_\ell|$ of unscreened null labeled and test units. Requiring the next screen index to be measurable with respect to this filtration prevents double-dipping while still permitting model updates and preference changes. The stopping rule then uses the FDP estimate $\widehat{\mathrm{FDP}}(\ell)$ from equation (2), and the proof shows that $M_\ell = |P_\ell^-|/(1+|N_\ell^-|)$ is a super-martingale with respect to a slightly richer filtration, so that optional stopping gives $\mathbb{E}[M_T] \le \mathbb{E}[M_k]$ and hence FDR below $\alpha$.

What would settle it

Enumerate every ordering rule that is measurable with respect to the filtration for a tiny exchangeable dataset, for example two labeled and two test units with a two-point outcome distribution, and compute the exact FDR by exhaustive averaging over all permutations; if any rule yields FDR above $\alpha$, the super-martingale bound fails.

Watch

Extended reading notes

Core claim

The central claim is Theorem 1: under Assumption 1, for any fixed $\alpha \in (0,1)$, the output of ACS applied at level $\alpha$ controls the false discovery rate (FDR) below $\alpha$. The procedure is sequential: after randomly assigning $k$ labeled units to initialize a model, it screens the least promising remaining unit, updates the ordering of all still-unscreened units using only the information in the filtration, and stops the first time the estimate $\widehat{\mathrm{FDP}}(\ell) = \frac{m}{n-k+1}\frac{1+|N_\ell^-|}{|P_\ell|\vee 1}$ falls to $\alpha$. The final selection is the set of unscreened test units at that stopping time. Because the ordering update is allowed to use a rich but controlled set of revealed information, the same guarantee covers model refitting, adaptive model selection, diversity-aware scoring, and the incorporation of labels revealed on screened test units.

Load-bearing premise

The load-bearing premise is Assumption 1: the labeled and test candidates, together with their property sets, are jointly exchangeable, and the analyst uses only the information that the protocol releases at each step.

Editorial extensions

If this is right

  • A screening team can refit or switch prediction models after each screened unit and still report FDR below the nominal level.
  • Analysts can re-prioritize the remaining candidates midstream, for example toward chemical diversity, without switching to asymptotic approximations.
  • Labels obtained from screened, unselected test units can be fed back into later model fits, which matches the workflow in LLM-output filtering with human review.
  • Every conformal selection application inherits the guarantee, because fixing the ordering in advance is a special case of the adaptive rule.
  • The power advantage over one-shot selection is largest when labeled data are scarce or signal is weak, because each screened unit becomes usable training material.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same information-flow device should generalize to other stopping rules, such as a target selected-set size or a budget of screens, but the FDR bound would have to be re-derived for each new stopping time.
  • Although the paper's guarantee is stated for property sets like Y not in C, the proof only uses exchangeability and the filtration, so outlier detection with a different null set is a natural transfer, a direction the paper mentions in its discussion.
  • Since ACS controls FDR even when the model is refit on screened data, it offers a template for adaptive data analysis tasks beyond selection, such as sequential experimentation with interim looks.
  • The paper leaves open how to choose the model-update frequency and the diversity weight; the simulations suggest these are tunable knobs with no effect on validity, only on power.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 9 minor

Summary. This paper proposes adaptive conformal selection (ACS), a sequential screening framework that generalizes conformal selection (Jin and Candès, 2023b) to allow adaptive, human-in-the-loop decision making while preserving finite-sample FDR control. Under an exchangeability assumption, the analyst screens units one at a time, and may reorder the remaining units, refit or select prediction models, optimize diversity-aware scores, or incorporate newly available labels, as long as the screening rule is measurable with respect to a restricted filtration. The central result (Theorem 1) shows that the FDR of the final selection set is bounded by alpha. The proof is based on a martingale argument with a carefully defined filtration and two lemmas. The paper provides three instantiations, extensive simulations, and real-data applications in LLM deployment and drug discovery. Code is available.

Significance. Assuming Theorem 1 holds (and my reading of Appendices A.1, B.1, and B.2 indicates it does), the paper is a significant contribution to model-free selective inference. It provides a principled answer to the question of how much adaptivity can be allowed in conformal selection without losing error control, and it demonstrates practical gains in power and diversity in both simulations and real applications. The martingale technique is elegant, the filtration restriction is a useful methodological idea, and the paper is strengthened by reproducible code, extensive simulation studies (1,000 replications across multiple settings), and real-data analyses in two distinct domains.

minor comments (9)
  1. [Section 4.3 and Appendix A.1] The claim that the proof of Theorem 1 extends to a 'slightly richer filtration' is not substantiated: the richer filtration is never explicitly defined, and Appendix A.1 does not address it. Please add a formal definition of the enriched filtration and a short argument (or a remark after the proof of Theorem 1) showing that Lemma 1 and the supermartingale argument remain valid when labels of screened test units are incorporated.
  2. [Appendix A.1, Eq. (6)] The second term on the right-hand side of Eq. (6) divides by |N^-_ell|, which can be zero. The proof should state that the term is interpreted as 0 when |N^-_ell| = 0, or treat this case separately.
  3. [Appendix B.1, proof of Lemma 1] In the displayed probability, the denominator writes 'P^+_ell = \bar P_ell' but should be 'P^+_ell = \bar P^+_ell'.
  4. [Section 5.4] The text says 'as introduced in Section 5.4' but the diversity-aware selection is introduced in Section 4.2.
  5. [Section 4.1] In the sentence 'one may the model every L steps', the verb 'update' is missing.
  6. [Section 5.4] There is a duplicate definite article in 'we adopt the the RBF kernel'.
  7. [Appendix C.2] The baseline 'CS-SCR' appears to be a typo for 'CS-SVR'.
  8. [Section 8] The typo 'flexibity' should be 'flexibility'.
  9. [Section 4.2.1] The statement that the solution to (4) thresholded to [0,1] is the 'working' selection rule would benefit from a brief explanation of why the thresholding preserves the ordering interpretation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the FDR bound is derived from Assumption 1 and the filtration restriction via a self-contained optional-stopping argument.

full rationale

Theorem 1 is proved from first principles rather than assumed or fitted. Appendix A.1 defines the supermartingale M_l = |P^-_l|/(1+|N^-_l|), verifies the conditional-expectation calculation using Lemma 1 (proved from Assumption 1's exchangeability and the F_l-measurability of the screening rule) and Lemma 2 (a hypergeometric bound), and then invokes optional stopping at T. No parameter is fitted to data and later renamed a prediction; the FDP estimator in Eq. (2) is a design choice that the proof controls, not an input that forces the theorem. The self-citations to Nair et al. (2025) and Gui et al. (2024) are contextual rather than load-bearing: the Nair martingale is mentioned as related work, but the proof in this paper is carried out in the appendix, so the cited result is not doing the work. The only under-specified passage is Section 4.3, where the claim that screened-test labels are covered by 'a slightly richer filtration' is not accompanied by an explicit construction in Appendix A.1; this is a completeness gap in exposition, not a circular reduction. Accordingly, the derivation chain is self-contained and no circular step is identified.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The FDR guarantee rests on exchangeability (Assumption 1) and on the analyst respecting the restricted filtration F_l; the free parameters are user-specified tuning choices that do not affect FDR control. No new entities are invented.

free parameters (5)
  • alpha = user-specified (e.g., 0.1)
    Target FDR level; a fixed input to the algorithm, not fitted to data.
  • lambda = 0.3, 0.4, 0.5, 0.8 in experiments
    User-specified trade-off between power and diversity in Sections 4.2 and 5.4; does not affect FDR control.
  • L (model update frequency) = 10 or 20
    Hyperparameter controlling how often the model is refit; chosen by the user for computational reasons.
  • sigma0 in RBF kernel = 5
    Kernel bandwidth for the similarity measure in diversity experiments; user-specified.
  • k (initial training size) = n in most experiments; varied in {60,80,100} in Appendix C.4
    Initial model training sample size; user-specified; robustness is studied.
assumptions (4)
  • domain assumption Assumption 1: {(X_i, Y_i, C_i)} jointly exchangeable across i, and property sets C_i are observed.
    Section 2.1; required for conformal validity and for the martingale exchangeability argument in the proof of Theorem 1.
  • ad hoc to paper The analyst's ordering rule is measurable with respect to the restricted filtration F_l (Eq. 1), and the initial random permutation makes the membership indicators A_i independent of the data.
    Section 3.2; the protocol that hides the identities of unscreened null labeled units is essential to Lemma 1. Without this restriction, the super-martingale property fails.
  • standard math Optional stopping theorem for nonnegative super-martingales.
    Appendix A.1; standard result applied to the bounded stopping time T.
  • domain assumption Positive definiteness of the matrix Theta in Proposition 1.
    Section 4.2.1; required for the closed-form solution of the diversity optimization; if it fails, numerical optimization would be needed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ACS: An interactive framework for conformal selection." pith.science (2026). https://pith.science/paper/O4XVZZYK

@misc{pith2026250715825,
  author       = {Pith},
  title        = {Pith review of: ACS: An interactive framework for conformal selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O4XVZZYK}},
  note         = {Machine review of arXiv:2507.15825}
}
read the original abstract

This paper presents adaptive conformal selection (ACS), an interactive framework for model-free selection with guaranteed error control. Building on conformal selection (Jin and Cand\`es, 2023b), ACS generalizes the approach to support human-in-the-loop adaptive data analysis. Under the ACS framework, we can partially reuse the data to boost the selection power, make decisions on the fly while exploring the data, and incorporate new information or preferences as they arise. The key to ACS is a carefully designed principle that controls the information available for decision making, allowing the data analyst to explore the data adaptively while maintaining rigorous control of the false discovery rate (FDR). Based on the ACS framework, we provide concrete selection algorithms for various goals, including model update/selection, diversified selection, and incorporating newly available labeled data. The effectiveness of ACS is demonstrated through extensive numerical simulations and real-data applications in large language model (LLM) deployment and drug discovery.

Figures

Figures reproduced from arXiv: 2507.15825 by the authors.

Figure 1
Figure 1. (a) The ROC curves of the (selected) prediction model at the [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. An illustration of the CS procedure as a sequential screening process, where the plus (resp. mi [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. An illustration of the ACS procedure, where the plus (resp. minus) signs correspond to the [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (31 more)
Figure 4
Figure 4. Figure 4: Realized power (left) and FDR (right) of ACS and CS for various noise levels [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Realized power (top) and FDR (bottom) of ACS with adaptive model selection, the naive [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: plots the ES-Power curves for the three methods. The realized ES of diversity-aware ACS is substantially lower than the other two methods at each power level, demonstrating the overall im￾provement in diversity [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Realized power (left) and FDR (right) as a function of the noise level [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Power (left) and FDR (right) of ACS with new labels (ACS-aug), ACS without new labels [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Realized power (left) and FDR (right) of ACS with new labels (ACS-aug), ACS without new [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Realized power (left) and FDR (right) of ACS with adaptive model selection, the naive [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Realized power (left) and FDR (right) of ACS with new labels (ACS-aug), ACS without new [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Realized power (left) and FDR (right) of ACS with adaptive model selection, the naive [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: ES of ACS (blue) compared to CS (red) for various quantile values [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Power of ACS (blue) compared to CS (red) for various quantile values [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Realized power (left) and FDR (right) of ACS and CS under simulation Setting 2. The other [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]
Figure 16
Figure 16. Figure 16: Realized power (left) and FDR (right) of ACS and CS under simulation Setting 3. The other [PITH_FULL_IMAGE:figures/full_fig_p028_16.png]
Figure 17
Figure 17. Figure 17: Realized power (left) and FDR (right) of ACS and CS under simulation Setting 4. The other [PITH_FULL_IMAGE:figures/full_fig_p028_17.png]
Figure 18
Figure 18. Figure 18: Realized power (left) and FDR (right) of ACS and CS under simulation Setting 5. The other [PITH_FULL_IMAGE:figures/full_fig_p028_18.png]
Figure 19
Figure 19. Figure 19: Realized power (left) and FDR (right) of ACS with adaptive model selection, the naive [PITH_FULL_IMAGE:figures/full_fig_p029_19.png]
Figure 20
Figure 20. Figure 20: Realized power (left) and FDR (right) of ACS with adaptive model selection, the naive [PITH_FULL_IMAGE:figures/full_fig_p029_20.png]
Figure 21
Figure 21. Figure 21: Realized power (left) and FDR (right) of ACS with adaptive model selection, the naive [PITH_FULL_IMAGE:figures/full_fig_p029_21.png]
Figure 22
Figure 22. Figure 22: Realized power (left) and FDR (right) of ACS with adaptive model selection, the naive [PITH_FULL_IMAGE:figures/full_fig_p030_22.png]
Figure 23
Figure 23. Figure 23: ES as a function of c Power for diversity-aware ACS with \ λ = 0.4, ACS, and CS under Setting 1. The other details are the same as in [PITH_FULL_IMAGE:figures/full_fig_p030_23.png]
Figure 24
Figure 24. Figure 24: Realized power (left) and FDR (right) of diversity-aware ACS with [PITH_FULL_IMAGE:figures/full_fig_p030_24.png]
Figure 25
Figure 25. Figure 25: ES as a function of c Power for diversity-aware ACS with \ λ = 0.5, ACS, and CS. The other details are the same as in [PITH_FULL_IMAGE:figures/full_fig_p031_25.png]
Figure 26
Figure 26. Figure 26: Realized power (left) and FDR (right) of diversity-aware ACS with [PITH_FULL_IMAGE:figures/full_fig_p031_26.png]
Figure 27
Figure 27. Figure 27: Realized power (left) and FDR (right) of ACS with [PITH_FULL_IMAGE:figures/full_fig_p032_27.png]
Figure 28
Figure 28. Figure 28: Realized power (left) and FDR (right) of ACS with new labels (ACS-aug), ACS without [PITH_FULL_IMAGE:figures/full_fig_p033_28.png]
Figure 29
Figure 29. Figure 29: Realized power (left) and FDR (right) of ACS with new labels (ACS-aug), ACS without new [PITH_FULL_IMAGE:figures/full_fig_p033_29.png]
Figure 30
Figure 30. Figure 30: Realized power (left) and FDR (right) of ACS with new labels (ACS-aug), ACS without new [PITH_FULL_IMAGE:figures/full_fig_p034_30.png]
Figure 31
Figure 31. Figure 31: Realized power (left) and FDR (right) of ACS with adaptive model selection, the naive method, [PITH_FULL_IMAGE:figures/full_fig_p034_31.png]
Figure 32
Figure 32. Figure 32: Realized power (left) and FDR (right) of ACS with adaptive model selection, the naive [PITH_FULL_IMAGE:figures/full_fig_p035_32.png]
Figure 33
Figure 33. Figure 33: Realized power (left) and FDR (right) of ACS with adaptive model selection, the naive [PITH_FULL_IMAGE:figures/full_fig_p035_33.png]
Figure 34
Figure 34. Figure 34: FDR of ACS (blue) compared to CS (red) for various quantile values [PITH_FULL_IMAGE:figures/full_fig_p035_34.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Robust Conformalized Selection with Noisy Responses

    stat.ML 2026-07 accept novelty 7.0 of 10

    RCS uses class-conditioned reweighting of noisy calibration data to control the false discovery rate in conformalized selection tasks, with asymptotic guarantees and empirical gains over prior methods.

Reference graph

Works this paper leans on

55 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [1]

    Alexander, S. P. H., Fabbro, D., Kelly, E., Mathie, A. A., Peters, J. A., Veale, E. L., Armstrong, J. F., Faccenda, E., Harding, S. D., Davies, J. A., Beuve, A., Brouckaert, P., Bryant, C., Burnett, J. C., Farndale, R. W., Friebe, A., Garthwaite, J., Hobbs, A. J., Jarvis, G. E., Koesling, D., Kuhn, M., MacEwan, D., Monie, T. P., Potter, L. R., Russwurm, M...

    work page 2023

  2. [2]

    and Jin, Y

    Bai, T. and Jin, Y. (2024). Optimized conformal selection: Powerful selective inference after conformity score optimization. arXiv preprint arXiv:2411.17983

    arXiv 2024

  3. [3]

    Bai, T., Tang, P., Xu, Y., Svetnik, V., Khalili, A., Yu, X., and Yang, A. (2024). Conformal selection for efficient and accurate compound screening in drug discovery

    work page 2024

  4. [4]

    Bajusz, D., R \'a cz, A., and H \'e berger, K. (2015). Why is tanimoto index an appropriate choice for fingerprint-based similarity calculations? Journal of cheminformatics , 7:1--13

    work page 2015

  5. [5]

    Balakrishnan, B. D. A. R. S. and Wasserman, L. (2019). Interactive martingale tests for the global null. arXiv preprint arXiv:1909.07339

    work page Pith review arXiv 2019

  6. [6]

    Bashari, M., Epstein, A., Romano, Y., and Sesia, M. (2024). Derandomized novelty detection with fdr control via conformal e-values. Advances in Neural Information Processing Systems , 36

    work page 2024

  7. [7]

    Bashari, M., Sesia, M., and Romano, Y. (2025). Robust conformal outlier detection under contaminated reference data. arXiv preprint arXiv:2502.04807

    work page Pith review arXiv 2025

  8. [8]

    Bates, S., Cand \`e s, E., Lei, L., Romano, Y., and Sesia, M. (2023). Testing for outliers with conformal p-values. The Annals of Statistics , 51(1):149--178

    work page 2023

Show all 55 references
  1. [9]

    and Hochberg, Y

    Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological) , 57(1):289--300

    1995

  2. [10]

    and Fithian, W

    Chao, P. and Fithian, W. (2021). Adapt-gmm: Powerful and robust covariate-assisted multiple testing. arXiv preprint arXiv:2106.15812

    2021 arXiv

  3. [11]

    Duan, B., Ramdas, A., and Wasserman, L. (2020). Familywise error rate control by interactive unmasking. In International Conference on Machine Learning , pages 2720--2729. PMLR

    2020

  4. [12]

    Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., and Roth, A. L. (2015). Preserving statistical validity in adaptive data analysis. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing , pages 117--126

    2015

  5. [13]

    Gui, Y., Jin, Y., and Ren, Z. (2024). Conformal alignment: Knowing when to trust foundation models with guarantees. arXiv preprint arXiv:2405.10301

    2024 arXiv

  6. [14]

    Huang, Z. (2007). Drug discovery research: new frontiers in the post-genomic era . John Wiley & Sons

    2007

  7. [15]

    Huo, Y., Lu, L., Ren, H., and Zou, C. (2024). Real-time selection under general constraints via predictive inference. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  8. [16]

    and Cand \`e s, E

    Jin, Y. and Cand \`e s, E. J. (2023a). Model-free selective inference under covariate shift via weighted conformal p-values. arXiv preprint arXiv:2307.09291

    2023 arXiv

  9. [17]

    and Cand \`e s, E

    Jin, Y. and Cand \`e s, E. J. (2023b). Selection by prediction with conformal p-values. Journal of Machine Learning Research , 24(244):1--41

    2023

  10. [18]

    E., Pollard, T

    Johnson, A. E., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-y., Mark, R. G., and Horng, S. (2019). Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data , 6(1):317

    2019

  11. [19]

    S., and Zettlemoyer, L

    Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. (2017). Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551

    2017 arXiv

  12. [20]

    Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022). Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

    2022 arXiv

  13. [21]

    Kuhn, L., Gal, Y., and Farquhar, S. (2023). Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664

    2023 arXiv

  14. [22]

    and Di Giovanni, C

    Lavecchia, A. and Di Giovanni, C. (2013). Virtual screening strategies in drug discovery: a critical review. Current medicinal chemistry , 20(23):2839--2860

    2013

  15. [23]

    Lee, J., Popov, I., and Ren, Z. (2025a). Full-conformal novelty detection: A powerful and non-random approach. arXiv preprint arXiv:2501.02703

    2025 arXiv

  16. [24]

    Lee, Y., Dobriban, E., and Tchetgen, E. T. (2025b). Finding distributions that differ, with false discovery rate control. arXiv preprint arXiv:2505.13769

    2025

  17. [25]

    and Ren, Z

    Lee, Y. and Ren, Z. (2025). Selection from hierarchical data with conformal e-values. arXiv preprint arXiv:2501.02514

    2025 arXiv

  18. [26]

    Lei, J., Rinaldo, A., and Wasserman, L. (2015). A conformal prediction approach to explore functional data. Annals of Mathematics and Artificial Intelligence , 74:29--43

    2015

  19. [27]

    and Fithian, W

    Lei, L. and Fithian, W. (2018). Adapt: an interactive procedure for multiple testing with side information. Journal of the Royal Statistical Society Series B: Statistical Methodology , 80(4):649--679

    2018

  20. [28]

    Lei, L., Ramdas, A., and Fithian, W. (2021). A general interactive framework for false discovery rate control under structural constraints. Biometrika , 108(2):253--267

    2021

  21. [29]

    Liang, Z., Sesia, M., and Sun, W. (2024). Integrative conformal p-values for out-of-distribution testing with labelled outliers. Journal of the Royal Statistical Society Series B: Statistical Methodology , 86(3):671--693

    2024

  22. [30]

    Lin, C.-Y. (2004). Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74--81

    2004

  23. [31]

    Lin, Z., Trivedi, S., and Sun, J. (2023). Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint arXiv:2305.19187

    2023 arXiv

  24. [32]

    Marandon, A., Lei, L., Mary, D., and Roquain, E. (2024). Adaptive novelty detection with false discovery rate guarantee. The Annals of Statistics , 52(1):157--183

    2024

  25. [33]

    and Roquain, E

    Mary, D. and Roquain, E. (2022). Semi-supervised multiple testing. Electronic Journal of Statistics , 16(2):4926--4981

    2022

  26. [34]

    Nair, Y., Jin, Y., Yang, J., and Candes, E. (2025). Diversifying conformal selections. arXiv preprint arXiv:2506.16229

    2025

  27. [35]

    Papadopoulos, H., Proedrou, K., Vovk, V., and Gammerman, A. (2002). Inductive confidence machines for regression. In Machine learning: ECML 2002: 13th European conference on machine learning Helsinki, Finland, August 19--23, 2002 proceedings 13 , pages 345--356. Springer

    2002

  28. [36]

    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011). Scikit-learn: Machine learning in python. the Journal of machine Learning research , 12:2825--2830

    2011

  29. [37]

    Reddy, S., Chen, D., and Manning, C. D. (2019). Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics , 7:249--266

    2019

  30. [38]

    and Cand \`e s, E

    Ren, Z. and Cand \`e s, E. (2023). Knockoffs with side information. The Annals of Applied Statistics , 17(2):1152--1174

    2023

  31. [39]

    M., Beeson, T., and Small, D

    Roy, S., Bogomolov, M., Heller, R., Claridge, A. M., Beeson, T., and Small, D. S. (2025). Exploration, confirmation, and replication in the same observational study: A two team cross-screening approach to studying the effect of unwanted pregnancy on mothers' later life outcome...

    2025

  32. [40]

    J., Kammeyer-Mueller, J

    Sajjadiani, S., Sojourner, A. J., Kammeyer-Mueller, J. D., and Mykerezi, E. (2019). Using machine learning to translate applicant work history into predictors of performance and turnover. Journal of Applied Psychology , 104(10):1207

    2019

  33. [41]

    Smit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A., and Lungren, M. (2004). Chexbert: Combining automatic labelers and expert annotations for accurate radiology report labeling using bert. arxiv [cscl]. published online april 20, 2020

    2004

  34. [42]

    Tanimoto, T. T. (1958). Elementary mathematical theory of classification and prediction

    1958

  35. [43]

    and Taylor, J

    Tian, X. and Taylor, J. (2018). Selective inference with a randomized response. The Annals of Statistics , 46(2):679--710

    2018

  36. [44]

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  37. [45]

    Vovk, V., Gammerman, A., and Shafer, G. (2005). Algorithmic learning in a random world , volume 29. Springer

    2005

  38. [46]

    Weinstein, A., Barber, R., and Candes, E. (2017). A power and prediction analysis for knockoffs with lasso statistics. arXiv preprint arXiv:1712.06465

    2017 arXiv

  39. [47]

    Wu, X., Huo, Y., Ren, H., and Zou, C. (2024). Optimal subsampling via predictive inference. Journal of the American Statistical Association , 119(548):2844--2856

    2024

  40. [48]

    Xie, R., Barber, R., and Candes, E. (2024). Boosted conformal prediction intervals. Advances in Neural Information Processing Systems , 37:71868--71899

    2024

  41. [49]

    Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., and Hooi, B. (2023). Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063

    2023 arXiv

  42. [50]

    and Ramdas, A

    Xu, Z. and Ramdas, A. (2024). Online multiple testing with e-values. In International Conference on Artificial Intelligence and Statistics , pages 3997--4005. PMLR

    2024

  43. [51]

    Yang, C.-Y., Lei, L., Ho, N., and Fithian, W. (2021). Bonus: Multiple multivariate testing with a data-adaptivetest statistic. arXiv preprint arXiv:2106.15743

    2021 arXiv

  44. [52]

    Yang, Y., Ma, Z., Nie, F., Chang, X., and Hauptmann, A. G. (2015). Multi-class active learning by uncertainty sampling with diversity maximization. International Journal of Computer Vision , 113:113--127

    2015

  45. [53]

    V., et al

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2023). Opt: Open pre-trained transformer language models, 2022. URL https://arxiv. org/abs/2205.01068 , 3:19--0

    2023 arXiv

  46. [54]

    Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. (2021). Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning , pages 12697--12706. PMLR

    2021

  47. [55]

    Zhong, P., Gong, Z., Li, S., and Sch \"o nlieb, C.-B. (2017). Learning to diversify deep belief networks for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing , 55(6):3516--3530

    2017

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.