Pith. sign in

REVIEW 2 major objections 5 minor 70 references

When May a Model Replace the Experiment? Audits, Licenses, and the Price of Trust in Surrogate-Driven Design

T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A surrogate may propose candidates freely, but every certified conclusion must rest on true evaluations alone—a separation sufficient for safety with no surrogate assumptions and necessary, since treating predictions as measurements…

desk verdict A serious, careful paper on when surrogates can substitute for expensive evaluations; the optimality claim is slightly oversold but the core theory and experiments are solid and peer-review-worthy. read the letter →

arxiv 2608.01378 v1 pith:4MFDUZN4 submitted 2026-08-02 cs.LG

Shuangxiu (Max) Ma , Wenhe (Zachary) Zhao This is my paper · ORCID
classification cs.LG
keywords surrogate-drivendesigncertificationselection-awareauditingseparationprincipleselectiontaxoraclecomplexityrankpreservationlicenses
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Design campaigns in chemistry, materials, and machine learning share a bottleneck: true evaluation is expensive, and surrogates that predict it are increasingly used to grade candidates and feed their own predictions back into the search. This paper asks when that substitution is safe, what a certificate of safety must cost, and when it pays. Its central answer is an architectural rule: surrogate predictions may propose candidates and train models without restriction, but every certified conclusion—intervals, eliminations, stopping rules, reported values—must be computed from true evaluations alone. The paper proves this separation is sufficient with no assumptions on the surrogate and necessary, because admitting predictions as measurements opens a deterministic self-confirmation failure; it also proves that accuracy metrics cannot anchor trust, that trust must be purchased through selection-aware audits, and that audited surrogates cut certified oracle cost for generative search but never for fixed-menu selection.

What carries the argument

The load-bearing object is the two-pipe information architecture: an unrestricted proposal pipe that may consume surrogate outputs, pseudo-labels, and arbitrary retraining, and a certification pipe that consumes oracle evaluations only. It is made concrete by a $\delta$-admissible confidence system built on a pre-sampled oracle noise array, so the coverage event lives on the array and pseudo-labels never touch it; that is what makes validity uniform over surrogate behavior. The companion mechanism is the selection-aware 'champion' audit, which draws a pool, lets the surrogate select its champion, and verifies only that champion with the oracle; this samples the deployment workflow rather than the candidate distribution, turning a passive audit's $\Theta(\min(1,Np))$ risk into a directly certified deployed failure rate at $O(\rho^{-1}\log(1/\delta))$ oracle calls. The cost dichotomy is carried by a quantile license for generative search, which certifies designs at a quality bar for $O(1)$ oracle evaluations each once the audit passes, against an $\Omega(1/p_v)$ oracle-only baseline.

What would settle it

Run the paper's contaminated-certification construction on a two-candidate noiseless task: the surrogate's predictions are reversed, accepted pseudo-labels enter the certification intervals with the standing of measurements, and no oracle is consulted. Theorem 5 predicts the wrong candidate is certified with probability one once the intervals narrow below half the value gap; if any contaminated protocol of this class certifies correctly, or if the self-confirming intervals fail to close, the necessity half of the separation principle is refuted.

Watch

Extended reading notes

Core claim

The paper's core discovery is a separation principle with matching necessity. Any design loop may let the surrogate do whatever it likes on the proposal side—select candidates, train on pseudo-labels, retrain freely—provided the certification side computes its confidence intervals, eliminations, stopping rules, and reported values from oracle evaluations alone. Under that architecture, with probability at least $1-\delta$ every certified conclusion is correct uniformly over all surrogate behaviors, including surrogates constructed adversarially with knowledge of the true objective. The converse is a deterministic failure construction: if accepted pseudo-labels enter certification statistics with the standing of measurements, there is a two-candidate task and a surrogate for which the protocol certifies the wrong candidate with probability one at zero oracle cost, its confidence generated entirely by the model's agreement with itself. Around this core the paper establishes supporting results: $R^2$ is neither necessary nor sufficient for decision quality, and screening $N$ candidates inflates expected over-prediction at the champion by at most $2\sigma\sqrt{\ln N}$ with a matching lower bound; the exact criterion for a model to act as an oracle is rank preservation on pairs separated by more than a tolerance, not accuracy; the only audit design that certifies deployed selection quality at a profit is selection-aware champion auditing, which is optimal among all audit designs up to a log factor; and for costs there is a strict-improvement dichotomy—certified fixed-menu selection never benefits from pseudo-labels in the tabular model, while certified generative search is cheaper by a factor $\Theta(M/\log(1/\rho))$ conditional on a passing audit. Across 432 surrogate fits on three ground-truthed tasks, the audit statistic tracks deployed regret at Spearman correlation 0.80–0.99 while $R^2$ falls as low as 0.33, and audited screening reduces certified oracle cost by a measured factor of 25.

Load-bearing premise

The guarantees cover one decision made once—choosing from a fixed pool or producing one generated design—with candidates drawn independently from a fixed distribution and the surrogate's outputs not reused to drive later learning. If a campaign lets the model's own predictions steer a multi-step search, the separation guarantee, the license, and the audit-cost floor do not transfer.

Editorial extensions

If this is right

  • If the separation principle is right, then reporting $R^2$, MSE, or any marginal accuracy metric as evidence of deployment reliability is unsafe: those metrics are structurally blind to the quality of the single selected candidate, and screening adds a quantified selection tax of order $\sqrt{\ln N}$ on expected over-prediction.
  • Any campaign that issues confidence intervals, eliminations, stopping decisions, or reported values must compute them from true evaluations alone; doing so preserves validity even if the surrogate is adversarially wrong, at the price of wasting oracle budget but never corrupting a certificate.
  • Certification of a surrogate must be performed by an audit that imitates deployment—draw pools, let the surrogate pick champions, verify only champions—because passive random-sample audits can both reject safe surrogates and approve dangerously aligned ones.
  • In generative search toward a quality bar, a surrogate holding a selection-aware license cuts certified oracle cost by roughly the screening factor (measured 25× on a standard architecture benchmark), while in tabular fixed-menu selection no use of pseudo-labels can reduce the certified cost.
  • Licenses bind to one trained model and one candidate distribution; retraining or distribution drift voids them, and renewal costs the same audit budget as the original, so re-auditing rather than transfer is the operational rule.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the separation principle transfers to multi-step design loops, the paper's own boundary suggests that any campaign reusing model outputs to drive learning—replay buffers, policy improvement, model-based rollouts—should be treated as unlicensed until per-step certificates exist; the paper leaves credit assignment open.
  • The per-fit bimodality documented here (identical recipes with near-identical audited rates deploying at failure 0 versus 1) implies that model-class reputation and training-regime reputation cannot substitute for per-fit auditing in practice, a conclusion that extends beyond the three tested tasks.
  • The selection-tax bound yields a cheap engineering heuristic the paper does not spell out: for large screens, discount any surrogate-reported champion value by roughly $\sigma\sqrt{2\ln N}$ unless a license says otherwise, since even an unbiased surrogate over-promises at the top by that amount.
  • The cost dichotomy suggests a practical division of labor for design campaigns: exhaustively certify fixed menus by direct oracle elimination, but invest in selection-aware audits for generative campaigns; the budget-plane results imply the training-label/audit-round split is the main strategic dial.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper develops a formal framework for deciding when a machine-learned surrogate may replace a costly oracle in design campaigns. Its main theoretical contributions are: (i) an accuracy--decision separation showing that R^2-type metrics are structurally blind to argmax regret, with matching upper and lower bounds on the resulting 'selection tax'; (ii) a separation principle stating that certification statistics must be computed solely from oracle evaluations while proposal machinery is unrestricted (Theorem 4), together with a deterministic self-confirmation failure for contaminated protocols (Theorem 5); (iii) oracle-cost bounds for certified elimination search (Theorems 6--7); (iv) an ordinal admissibility criterion with an audit--deployment gap and selection-aware audits (Theorems 8--11); (v) a universal lower bound on audit query complexity for the ordinal/regret guarantee (Theorem 14 and SI Theorem S7.7); and (vi) a strict-improvement dichotomy for generative versus fixed-menu search (Theorem 12 and SI Theorems S7.1--S7.3). The claims are supported by experiments on three exhaustively ground-truthed tasks (17,353 real training outcomes), 432 surrogate fits, and an extensive open-source verification suite.

Significance. Should the results stand, this is a substantial contribution: it converts 'trust in surrogates' from an accuracy heuristic into a measurable, license-based protocol with formal guarantees, and it provides a query-complexity lower bound over arbitrary audit designs for the ordinal/regret guarantee. The paper's main strengths are the clean pre-sampled-array argument behind the oracle-only invariance theorem, the explicit self-confirmation construction, the tight Theta(N) versus Theta(N^2) audit-statistic gap, the coupling-based universal lower bound, and the unusually complete reproducibility apparatus (ground-truth tables, committed code, numerical verification of each theorem). The experiments sensibly distinguish calibration of the audit statistic from independent confirmation, and the collapse of R^2 as a deployment diagnostic on the combinatorial benchmark is a concrete, falsifiable observation. The principal weakness is public-facing overstatement of the optimality result's scope, which is correctable and is detailed below.

major comments (2)
  1. [Abstract; Theorem 14; SI Thm. S7.7] The abstract states that selection-aware audits are 'provably optimal in query complexity--no audit design of any kind can certify deployed selection quality with asymptotically fewer oracle queries.' As the SI itself makes precise, SI Theorem S7.7 proves an Omega(N/rho_0) lower bound only for the ordinal/regret event FAIL = {max_i f(X_i) - f(hat x) > epsilon}, under a nontriviality/completeness condition. The paper's own value-floor license (SI Theorem S7.2, Corollary S7.6) certifies the one-sided deployed guarantee P(f(champion) < v_hat) <= 4rho with O(rho^{-1} log(1/delta)) oracle calls, independent of N. Remark 15 and Corollary S7.6 explicitly acknowledge that value licenses escape the N-floor because they certify a different, weaker object. The formal theorems are internally consistent, but the unqualified headline claim is false as written and should be amended to 'for the ordinal/regret guarantee' in the abstract, in the Theorem 14 statement, and in the concluding 'price of trust' narrative. Because this optimality claim is a keystone contribution, the scope correction is necessary before publication.
  2. [Theorem 12(b); SI Cor. S7.3; SI Thm. S7.2] The claimed Theta(M/log(1/rho)) separation is presented as 'conditional on the audit passing,' but the proof in Theorem S7.2 splits the guarantee into two distinct high-probability events: the validity event P(f(champion) < v_hat) <= 4rho holds with probability at least 1 - delta, while the high-level event v_hat >= q_{1 - log(1/rho)/M} - epsilon (which is what makes the screening factor meaningful) holds only with probability at least 1 - delta^{2/3} under the stated Ta >= (2/rho) log(1/delta). The combined success probability is therefore at least 1 - delta - delta^{2/3}, not 1 - delta. This is fixable by setting Ta >= (3/rho) log(1/delta) or by explicitly defining 'audit passing' as the conjunction of both events, but as written the probability accounting in the dichotomy's cost claim is incomplete.
minor comments (5)
  1. [Theorem 1; SI Thm. S3.1] The main-text statement of Theorem 1 omits the qualifications in the SI: the near-perfect-R^2 construction requires even N with N >= 4(1+beta)^2/epsilon. Please add the qualification or explicitly defer to the SI.
  2. [Theorem 7; SI Thm. S5.5] The main-text margin condition says 'the fraction of candidates within u of optimal is bounded by Cu^alpha,' while the SI uses #{x : 0 < Delta_x <= u} <= N min(1, C u^alpha). The factor N and the min(1,.) should be reflected in the main text for consistency.
  3. [Section 4.3; SI S8.2] The sentence 'high correlation is calibration, not clairvoyance' is an important and honest caveat. Consider moving it to the start of Section 4.3 so that readers do not mistake the 0.89--0.99 correlations for independent confirmation of the theory.
  4. [Section 5; SI Cor. S5.7] The final-paragraph statement that the policy-selection form 'follows from the pool theorems' should cite Corollary S5.7 by name; the main text currently refers only to 'SI Section S5.1', which is less precise.
  5. [Equation (1); SI Prop. S8.1] In Equation (1), the notation N qbar_delta(events, T_a) uses the word 'events' as a variable; define it in the main text as the number of regret events K_T, as in SI Proposition S8.1, to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: theoretical results derive from explicit assumptions with no fitted inputs, and the empirical audit calibration is explicitly labeled as calibration rather than prediction.

full rationale

The derivation chain is self-contained. Theorems 4 and 5 prove the separation principle from a delta-admissible oracle-only confidence system and an explicit deterministic counterexample; no fitted parameter or self-citation carries the argument. The audit statistic q_sel is constructed as a Monte Carlo probe of the champion-inversion rate, and the paper explicitly disclaims clairvoyance: Section 4.3 states that 'high correlation is calibration, not clairvoyance,' so the empirical tracking claim is an honest measurability result, not a fitted quantity renamed as a prediction. The optimality claim in Theorem 14 / SI Thm. S7.7 is proven for the ordinal/regret failure event FAIL = {max_i f(X_i)-f(hat x) > epsilon} under a stated completeness condition, while the value-floor license of SI Thm. S7.2 certifies a strictly weaker one-sided object; Remark 15 and Corollary S7.6 explicitly distinguish these, so the abstract's unqualified phrasing is a scope-overbreadth concern, not a circular reduction. The paper also carefully separates known results (LUCB/UGapE, quantile bandits, conformal selection, optimizer's curse) from its claimed contributions in Section S9, and none of its load-bearing steps relies on an unverified self-citation. No equation is defined in terms of the conclusion it is used to prove, and no parameter fitted to a subset is later reported as an independent prediction.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No fitted free parameters; the tolerances epsilon, N, M, rho, delta, and audit depth Ta are user-set inputs to the framework, not parameters tuned to make the theorems pass. Formal results assume queries yield f(x) plus mean-zero sigma-sub-Gaussian noise from a pre-sampled array, and deployment is a single argmax over i.i.d. pools from a fixed D. The alpha-margin condition and the tabular/i.i.d.-marks model are explicit structural assumptions. No new physical entities are introduced.

assumptions (6)
  • domain assumption Oracle queries return f(x) plus eta with eta mean-zero sigma-sub-Gaussian, independent across queries, formally from a pre-sampled array.
    Used in Lemma S4.2 and Theorem S5.4 to construct delta-admissible intervals; if oracle noise is biased or dependent, the interval coverage argument collapses.
  • domain assumption Deployment and audit pools are i.i.d. draws from a fixed candidate distribution D; the decision is a single argmax over the pool.
    The audit statistics qsel and qinv and the lower bounds of Theorems S7.4 and S7.7 certify exactly this protocol; sequential policy-improvement is explicitly left open in Section 5 and Remark S5.9.
  • domain assumption Alpha-margin condition: the fraction of candidates within u of optimal is bounded by N min(1, C u^alpha).
    Theorem S5.5's epsilon-independent cost regime for alpha greater than 2 depends on this bound; it is an assumed property of the task, not a consequence.
  • domain assumption Tabular / i.i.d.-marks information model for the strict-improvement dichotomy: candidates carry no shared structure usable by the surrogate.
    Theorem S7.1's Omega(1/pv) lower bound and Theorem 12(a)'s no-improvement result are stated in this model; structured linear and RKHS classes are left open.
  • domain assumption The audit lower bound assumes a sound auditor plus a completeness condition: it must pass a designated perfectly safe instance with probability at least 1/2.
    The Omega(N/rho0) bound in Theorem S7.7 is stated for such nontrivial auditors; without this condition an auditor that rejects everything trivially avoids the lower bound.
  • standard math Standard concentration inequalities and change-of-measure arguments, including Hoeffding, Chernoff, and Pinsker.
    Background used throughout the SI; not claimed as contributions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When May a Model Replace the Experiment? Audits, Licenses, and the Price of Trust in Surrogate-Driven Design." pith.science (2026). https://pith.science/paper/4MFDUZN4

@misc{pith2026260801378,
  author       = {Pith},
  title        = {Pith review of: When May a Model Replace the Experiment? Audits, Licenses, and the Price of Trust in Surrogate-Driven Design},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4MFDUZN4}},
  note         = {Machine review of arXiv:2608.01378}
}
read the original abstract

Design campaigns in chemistry, materials science, and machine learning share a bottleneck: determining how good a candidate truly is requires an expensive evaluation - an experiment, a first-principles simulation, or a full training run. Machine-learning surrogates that predict these outcomes are increasingly used not only to propose candidates but to grade them, and even to feed their own predictions back into the search as though they were measurements. Through mathematical analysis validated on three exhaustively ground-truthed design tasks, we establish when this practice is safe, what any certificate of safety must cost, and when the substitution provably pays. Predictive accuracy cannot anchor trust: near-perfect R^2 is compatible with worst-possible selections, and screening N candidates inflates the over-prediction at the selected candidate by a quantifiable "selection tax" with matching upper and lower bounds. Safety follows instead from an architectural rule - predictions may propose and train without restriction, but every certified conclusion must rest on true evaluations - which is sufficient with no assumptions on the surrogate, and necessary, since admitting predictions into certification with the standing of measurements opens a deterministic self-confirmation failure mode. We derive the minimal criterion under which a model may act as an oracle (rank preservation, not accuracy), show that trust must be purchased through selection-aware audits that are optimal in query complexity, and prove a dichotomy fixing when audited surrogates cut certified evaluation cost. Across 432 surrogate fits over six task-regime conditions, the audit statistic tracks deployed search performance at Spearman rank correlation 0.80-0.99, while the rank correlation of R^2 with deployed regret falls as low as 0.33; audited screening reduces certified oracle cost by a measured factor of 25.

Figures

Figures reproduced from arXiv: 2608.01378 by the authors.

Figure 1
Figure 1. The certified surrogate-use framework. Information flows through two pipes with distinct [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The selection tax measured on Task 1 (hyperparameter optimization; oracle = actual [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Oracle-grounded recommendation quality versus cumulative oracle cost (Task 1; 20 [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The budget plane: strongest earnable certificate over (training labels, audit calls), license [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: (a) Certified deployed bound versus training labels for four labeling policies (hard landscape; [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: (a) Damage map: strongest certificate under controlled corruption (fraction of candidates [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 62 canonical work pages

  1. [1]

    Conservative Objective Models for Effective Offline Model-Based Optimization.Proc

    Trabucco, B.; Kumar, A.; Geng, X.; Levine, S. Conservative Objective Models for Effective Offline Model-Based Optimization.Proc. 38th Int. Conf. Machine Learning (ICML)2021

    work page 2021

  2. [2]

    Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization.Proc

    Trabucco, B.; Geng, X.; Kumar, A.; Levine, S. Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization.Proc. 39th Int. Conf. Machine Learning (ICML)2022

    work page 2022

  3. [3]

    How Powerful Are Performance Predictors in Neural Architecture Search?Adv

    White, C.; Zela, A.; Ru, B.; Liu, Y.; Hutter, F. How Powerful Are Performance Predictors in Neural Architecture Search?Adv. Neural Inf. Process. Syst. (NeurIPS)2021

    work page 2021

  4. [4]

    Scaling Laws for Reward Model Overoptimization.Proc

    Gao, L.; Schulman, J.; Hilton, J. Scaling Laws for Reward Model Overoptimization.Proc. 40th Int. Conf. Machine Learning (ICML)2023

    work page 2023

  5. [5]

    Inference Scaling Flaws: The Limits of LLM Resampling with Imperfect Verifiers.arXiv2024, 2411.17501

    Stroebl, B.; Kapoor, S.; Narayanan, A. Inference Scaling Flaws: The Limits of LLM Resampling with Imperfect Verifiers.arXiv2024, 2411.17501

    arXiv

  6. [6]

    E.; Winkler, R

    Smith, J. E.; Winkler, R. L. The Optimizer’s Curse: Skepticism and Postdecision Surprise in Decision Analysis.Manage. Sci.2006,52, 311–322

    work page 2006

  7. [7]

    C.; Clapp, R

    Capen, E. C.; Clapp, R. V.; Campbell, W. M. Competitive Bidding in High-Risk Situations.J. Pet. Technol.1971,23, 641–653

    work page 1971

  8. [8]

    NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search

    Dong, X.; Yang, Y. NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search. Int. Conf. Learning Representations (ICLR)2020

    work page 2020

Show all 70 references
  1. [9]

    Offline Model-Based Optimization by Learning to Rank.Int

    Lyu, Y.; et al. Offline Model-Based Optimization by Learning to Rank.Int. Conf. Learning Representa- tions (ICLR)2025

    2025

  2. [10]

    On the Complexity of Best-Arm Identification in Multi-Armed Bandit Models.J

    Kaufmann, E.; Capp´ e, O.; Garivier, A. On the Complexity of Best-Arm Identification in Multi-Armed Bandit Models.J. Mach. Learn. Res.2016,17, 1–42

    2016

  3. [11]

    Verification Based Solution for Structured MAB Problems.Adv

    Karnin, Z. Verification Based Solution for Structured MAB Problems.Adv. Neural Inf. Process. Syst. (NeurIPS)2016

    2016

  4. [12]

    Jin, Y.; Cand` es, E. J. Selection by Prediction with Conformal p-Values.J. Mach. Learn. Res.2023,24, 1–41

    2023

  5. [13]

    Confidence on the Focal: Conformal Prediction with Selection-Conditional Coverage.J

    Jin, Y.; Ren, Z. Confidence on the Focal: Conformal Prediction with Selection-Conditional Coverage.J. R. Stat. Soc. B2025,87, 1239–1263

  6. [14]

    N.; Listgarten, J.; Jordan, M

    Fannjiang, C.; Bates, S.; Angelopoulos, A. N.; Listgarten, J.; Jordan, M. I. Conformal Prediction under Feedback Covariate Shift for Biomolecular Design.Proc. Natl. Acad. Sci. U.S.A.2022,119, e2204569119

    2022

  7. [15]

    R.; Kalyanakrishnan, S

    Chaudhuri, A. R.; Kalyanakrishnan, S. PAC Identification of a Bandit Arm Relative to a Reward Quantile.Proc. 31st AAAI Conf. Artificial Intelligence2017

  8. [16]

    Pure Exploration in Infinitely-Armed Bandit Models with Fixed-Confidence.Proc

    Aziz, M.; Anderton, J.; Kaufmann, E.; Aslam, J. Pure Exploration in Infinitely-Armed Bandit Models with Fixed-Confidence.Proc. Algorithmic Learning Theory (ALT)2018

    2018

  9. [17]

    Multi-Fidelity Gaussian Process Bandit Optimisation.J

    Kandasamy, K.; Dasarathy, G.; Oliva, J.; Schneider, J.; P´ oczos, B. Multi-Fidelity Gaussian Process Bandit Optimisation.J. Artif. Intell. Res.2019,66, 151–196

    2019

  10. [18]

    B.; Wagner, D

    Stark, P. B.; Wagner, D. A. Evidence-Based Elections.IEEE Secur. Priv.2012,10, 33–41

    2012

  11. [19]

    S.; Aykol, M.; Cheon, G.; Cubuk, E

    Merchant, A.; Batzner, S.; Schoenholz, S. S.; Aykol, M.; Cheon, G.; Cubuk, E. D. Scaling Deep Learning for Materials Discovery.Nature2023,624, 80–85. 44

  12. [20]

    J.; Rendy, B.; Fei, Y.; et al

    Szymanski, N. J.; Rendy, B.; Fei, Y.; et al. An Autonomous Laboratory for the Accelerated Synthesis of Novel Materials.Nature2023,624, 86–91; Author Correction2026

  13. [21]

    B.; Bhatt, P.; Schoop, L

    Leeman, J.; Liu, Y.; Stiles, J.; Lee, S. B.; Bhatt, P.; Schoop, L. M.; Palgrave, R. G. Challenges in High-Throughput Inorganic Materials Prediction and Autonomous Synthesis.PRX Energy2024,3, 011002

  14. [22]

    K.; Seshadri, R

    Cheetham, A. K.; Seshadri, R. Artificial Intelligence Driving Materials Discovery? Perspective on the Article: Scaling Deep Learning for Materials Discovery.Chem. Mater.2024,36, 3490–3495

    2024

  15. [23]

    Riebesell, J.; Goodall, R. E. A.; Benner, P.; Chiang, Y.; Deng, B.; et al. A Framework to Evaluate Machine Learning Crystal Stability Predictions.Nat. Mach. Intell.2025,7; DOI 10.1038/s42256-025- 01055-1

    2025 doi

  16. [24]

    Overtuning in Hyperparameter Optimization.Proc

    Schneider, L.; Bischl, B.; Feurer, M. Overtuning in Hyperparameter Optimization.Proc. AutoML Conf. 2025

    2025

  17. [25]

    Optimizer’s Information Criterion: Dissecting and Correcting Bias in Data-Driven Optimization.arXiv2023, 2306.10081

    Iyengar, G.; Lam, H.; Wang, T. Optimizer’s Information Criterion: Dissecting and Correcting Bias in Data-Driven Optimization.arXiv2023, 2306.10081

    arXiv

  18. [26]

    Efficient Estimation and Correction of Selection-Induced Bias with Order Statistics.Stat

    McLatchie, Y.; Vehtari, A. Efficient Estimation and Correction of Selection-Induced Bias with Order Statistics.Stat. Comput.2024,34, 132

    2024

  19. [27]

    M.; Restelli, M

    Poiani, R.; Degenne, R.; Kaufmann, E.; Metelli, A. M.; Restelli, M. Optimal Multi-Fidelity Best-Arm Identification.Adv. Neural Inf. Process. Syst. (NeurIPS)2024

    2024

  20. [28]

    Best Arm Identification with LLM Judges and Limited Human Audits.arXiv2026, 2601.21471

    Ao, R.; Chen, H.; Gao, S.; Li, H.; Simchi-Levi, D. Best Arm Identification with LLM Judges and Limited Human Audits.arXiv2026, 2601.21471

  21. [29]

    Best-Arm Identification with Generative Proxy.arXiv2026, 2607.06879

    Ma, T.; Qin, H.; Zhu, R.; Zuo, J. Best-Arm Identification with Generative Proxy.arXiv2026, 2607.06879

    arXiv

  22. [30]

    Abbasi-Yadkori, D

    Y. Abbasi-Yadkori, D. P´ al, C. Szepesv´ ari. Improved algorithms for linear stochastic bandits. NeurIPS 2011

    2011

  23. [31]

    Beirami et al

    A. Beirami et al. Theoretical guarantees on the best-of-n alignment policy. ICML 2025. arXiv:2401.01879

    2025 arXiv

  24. [32]

    R. Berk, L. Brown, A. Buja, K. Zhang, L. Zhao. Valid post-selection inference. Ann. Statist. 41(2), 2013

    2013

  25. [33]

    Bertsekas, J

    D. Bertsekas, J. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, 1996

    1996

  26. [34]

    Boucheron, G

    S. Boucheron, G. Lugosi, P. Massart. Concentration Inequalities. Oxford UP, 2013

    2013

  27. [35]

    Cameron, J

    C. Cameron, J. Hartford, T. Lundy, K. Leyton-Brown. The perils of learning before optimizing. AAAI 2022

    2022

  28. [36]

    Castro, R

    R. Castro, R. Nowak. Minimax bounds for active learning. IEEE Trans. IT 54(5), 2008

    2008

  29. [37]

    S. R. Chowdhury, A. Gopalan. On kernelized multi-armed bandits. ICML 2017

    2017

  30. [38]

    predict, then optimize

    A. Elmachtoub, P. Grigas. Smart “predict, then optimize”. Management Science 68(1), 2022

    2022

  31. [39]

    Even-Dar, S

    E. Even-Dar, S. Mannor, Y. Mansour. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. JMLR 7, 2006

    2006

  32. [40]

    A. M. Farahmand. Action-gap phenomenon in reinforcement learning. NeurIPS 2011

    2011

  33. [41]

    Gabillon, M

    V. Gabillon, M. Ghavamzadeh, A. Lazaric. Best arm identification: a unified approach to fixed budget and fixed confidence. NeurIPS 2012. 45

    2012

  34. [42]

    Howard, A

    S. Howard, A. Ramdas, J. McAuliffe, J. Sekhon. Time-uniform, nonparametric, nonasymptotic confidence sequences. Ann. Statist. 49(2), 2021

    2021

  35. [43]

    Jamieson, M

    K. Jamieson, M. Malloy, R. Nowak, S. Bubeck. lil’UCB: an optimal exploration algorithm for multi-armed bandits. COLT 2014

    2014

  36. [44]

    Jamieson, R

    K. Jamieson, R. Nowak. Best-arm identification algorithms for multi-armed bandits in the fixed confidence setting. CISS 2014

    2014

  37. [45]

    K. Ji, J. He, T. Yang, Q. Zhang, Q. Gu. Reinforcement learning from human feedback with active queries. 2024. arXiv:2402.09401

    2024 arXiv

  38. [46]

    Y. Jin, Z. Yang, Z. Wang. Is pessimism provably efficient for offline RL? ICML 2021

    2021

  39. [47]

    Kalyanakrishnan, A

    S. Kalyanakrishnan, A. Tewari, P. Auer, P. Stone. PAC subset selection in stochastic multi-armed bandits. ICML 2012

    2012

  40. [48]

    Kaufmann, W

    E. Kaufmann, W. Koolen. Mixture martingales revisited with applications to sequential tests and confidence intervals. JMLR 22, 2021

    2021

  41. [49]

    Kearns, S

    M. Kearns, S. Singh. Near-optimal reinforcement learning in polynomial time. Machine Learning 49, 2002

    2002

  42. [50]

    Lattimore, C

    T. Lattimore, C. Szepesv´ ari. Bandit Algorithms. Cambridge UP, 2020

    2020

  43. [51]

    Lyu et al

    Y. Lyu et al. On the learnability of offline model-based optimization: a ranking perspective. 2026. arXiv:2603.04000

    2026

  44. [52]

    Mannor, J

    S. Mannor, J. Tsitsiklis. The sample complexity of exploration in the multi-armed bandit problem. JMLR 5, 2004

    2004

  45. [53]

    Maron, A

    O. Maron, A. Moore. Hoeffding races: accelerating model selection search. NeurIPS 1993

    1993

  46. [54]

    Perchet, P

    V. Perchet, P. Rigollet. The multi-armed bandit problem with covariates. Ann. Statist. 41(2), 2013

    2013

  47. [55]

    Sekhari, K

    A. Sekhari, K. Sridharan, W. Sun, R. Wu. Selective sampling and imitation learning via online regression. NeurIPS 2023

    2023

  48. [56]

    Simchowitz, K

    M. Simchowitz, K. Jamieson. Non-asymptotic gap-dependent regret bounds for tabular MDPs. NeurIPS 2019

    2019

  49. [57]

    Singh, R

    S. Singh, R. Yee. An upper bound on the loss from approximate optimal-value functions. Machine Learning 16, 1994

    1994

  50. [58]

    Srivastava

    M. Srivastava. Fundamental limits of black-box safety evaluation. 2026. arXiv:2602.16984

    2026

  51. [59]

    Wang et al

    Z. Wang et al. When self-belief misleads: active label acquisition for RLVR (CARE). 2025. arXiv:2605.25864

    2025 arXiv

  52. [60]

    Williams, C

    J. Williams, C. Sheahan, A. Raymond, T. Korbak. Predicting LLM safety before release by simulating deployment. 2026. arXiv:2607.07184

    2026 arXiv

  53. [61]

    Angelopoulos, S

    A. Angelopoulos, S. Bates, E. Cand` es, M. I. Jordan, L. Lei. Learn then test: calibrating predictive algorithms to achieve risk control. Ann. Appl. Statist., 2025. arXiv:2110.01052

    2025 arXiv

  54. [62]

    Anthony, S

    L. Anthony, S. Nobari. Certifying technology-assisted review: audit lower bounds for screening recall

  55. [63]

    Chen, T.-Y

    W. Chen, T.-Y. Liu, Y. Lan, Z. Ma, H. Li. Ranking measures and loss functions in learning to rank. NeurIPS 2009. 46

    2009

  56. [64]

    Geifman, R

    Y. Geifman, R. El-Yaniv. Selective classification for deep neural networks. NeurIPS 2017

    2017

  57. [65]

    Katz-Samuels, K

    J. Katz-Samuels, K. Jamieson. The true sample complexity of identifying good arms. AISTATS 2020

    2020

  58. [66]

    S. Choi. Conformal candidate certification for offline model-based optimization. 2026. arXiv:2606.15217

    2026

  59. [67]

    W. Ji, Y. Pan, R. Zhu, L. Lei. Multi-armed bandits with machine-learning-generated surrogate rewards

  60. [68]

    Bakshi, L.-A

    S. Bakshi, L.-A. Gao, Z. Gao, S. Panigrahi. Flexible inference for winners with conditional validity. 2026. arXiv:2607.18545

    2026 arXiv

  61. [69]

    Adrian, J

    M. Adrian, J. A. Soloff, R. Willett. Stabilizing black-box model selection with the inflated argmax. TMLR 2025. arXiv:2410.18268

    2025

  62. [70]

    Kossen, S

    J. Kossen, S. Farquhar, Y. Gal, T. Rainforth. Active testing: sample-efficient model evaluation. ICML 2021. 47

    2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.