Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Cost-Optimal Active AI Model Evaluation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper derives a cost-optimal active sampling rule for AI evaluation: query the expensive rater with probability proportional to the square root of the cheap rater's conditional error, capped at 1.

desk verdict Solid theory for cost-optimal evaluation budgets; the active policy's practical edge depends on uncertainty estimates the paper itself shows are imperfect. read the letter →

arxiv 2506.07949 v1 pith:TIBGYRW2 submitted 2025-06-09 cs.LG

Anastasios N. Angelopoulos , Jacob Eisenstein , Jonathan Berant , Alekh Agarwal , Adam Fisch This is my paper · ORCID
classification cs.LG MSC 62K0562F12
keywords activeevaluationcost-optimalsamplingprediction-poweredinferenceweakratersmodelunbiasedestimationheteroskedasticerrorsannotationbudget
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how to estimate the mean quality score of an AI system when two rating sources are available: a cheap, often inaccurate automated rater and an expensive, accurate rater, with a fixed total annotation budget. It claims that the optimal allocation is not a fixed ratio of expensive to cheap ratings but an input-dependent rule that calls the expensive rater with probability proportional to the square root of the cheap rater's conditional error, capped at 1. Under i.i.d. sampling, the paper proves that this rule minimizes the mean squared error of an unbiased estimator of the strong-rater mean, and that the optimal fixed-rate policy is the special case where the cheap rater's error does not vary with the input. On real LLM-evaluation tasks, the policies reach the same estimation precision as always using the expensive rater at 40--50% of the budget in the best settings, with the largest gains when example difficulty is highly variable.

What carries the argument

The machinery is the inverse-propensity estimator $\hat{\theta}_T = \frac{1}{T}\sum_{t=1}^T \left[G_t + (H_t-G_t)\xi_t / \pi_t(X_t)\right]$, where $\xi_t$ indicates whether the strong label was queried. Its variance decomposes as $\mathrm{Error}_T(\pi)=\frac{1}{T}[\mathrm{Var}(H)-\mathbb{E}[(H-G)^2]+\mathbb{E}[(H-G)^2/\pi(X)]]$, and its expected cost is $T(c_h\mathbb{E}[\pi(X)]+c_g)$. Minimizing error for a fixed budget turns the problem into a convex optimization over the sampling rule; the square-root allocation is the resulting stationary condition, and the cap at 1 handles the constraint $\pi(x)\le 1$. The threshold $\tau^*$ is found by a one-dimensional search because the clipped objective is not convex in the threshold.

What would settle it

Take a real evaluation task with full strong labels, compute the true $u(x)=\mathbb{E}[(H-G)^2\mid X=x]$ on a held-out set, and compare it with the proxy $G(1-G)$ used by the paper; if the rank correlation between the two is near zero, the estimated active policy should have nearly the same squared error per budget as the fixed-rate policy, contradicting the paper's empirical claim that active sampling delivers large savings whenever example difficulty is heterogeneous.

Watch

Extended reading notes

Core claim

The central claim is that the annotation policy minimizing estimation error under a cost budget has an explicit square-root form. With $H$ the strong rating, $G$ the weak rating, costs $c_h > c_g$, and conditional error $u(x)=\mathbb{E}[(H-G)^2\mid X=x]$, the optimal active rule is $\pi_{\mathrm{active}}(x)=\min(\gamma^* / \sqrt{u(x)}, 1)$, where $\gamma^*$ is set by the cost ratio and the distribution of $u(X)$, and inputs with $\sqrt{u(x)}$ above a threshold are always sent to the strong rater. The companion result for policies that cannot depend on $x$ gives the optimal fixed sampling rate $\pi_{\mathrm{random}}(x)=\sqrt{(c_g/c_h)\,\mathbb{E}[(H-G)^2]/(\mathrm{Var}(H)-\mathbb{E}[(H-G)^2])}$, with a boundary at 1 when the weak rater is too inaccurate to help. The paper shows the active rule reduces to the fixed rule when $u(x)$ is constant, and it characterizes empirically that the benefit of active over random sampling grows with the variance of $u(X)$ and shrinks as the weak rater's average error or relative cost grows.

Load-bearing premise

The load-bearing assumption is that the evaluator has access to the conditional mean squared error of the weak rater, $u(x)=\mathbb{E}[(H-G)^2\mid X=x]$, for each input; in practice this quantity is replaced by proxies such as $G(1-G)$ or model confidence, and the paper's own oracle-headroom experiments show that when those proxies are poor the active policy's budget advantage over random sampling largely disappears.

Editorial extensions

If this is right

  • An evaluation pipeline that learns or transfers $u(x)$ can meet a target precision with roughly half the budget of a strong-rater-only evaluation, and more when the required precision is high.
  • The optimal weak-rater query rate is determined by the cost ratio and the error distribution, so budgets can be planned without trial-and-error.
  • Active policies are most valuable when example difficulty is heterogeneous; on homogeneous tasks, the fixed-rate policy captures most of the gain.
  • If the weak rater's average error exceeds the variance of the target rating by a cost-dependent margin, the optimal policy is to ignore the weak rater and query the strong rater always.
  • The framework extends beyond means to convex M-estimators and to choosing which inputs to sample, so the same allocation logic applies to estimating other evaluation targets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If conditional error $u(x)$ can be learned online from a small burn-in set, the active rule suggests a natural adaptive protocol for continually adding new models to an existing benchmark; the paper's main results treat fixed policies.
  • The same square-root allocation should apply whenever a cheap surrogate signal is combined with a gold label, such as A/B testing with surrogate metrics, so the result is not specific to AI evaluation.
  • The observed gap between the estimated and oracle policies points to uncertainty calibration, not just average accuracy, as the property of autoraters that determines active-sampling gains.
  • A practical diagnostic suggested by the theory is to estimate $\mathrm{Var}(u(X))$ on a labeled subsample; if it is near zero, use the fixed-rate policy and spend no engineering effort on active sampling.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper develops a framework for cost-optimal hybrid evaluation, in which an unbiased estimator of the mean of an expensive strong-rater signal H is built from cheap weak-rater observations G plus selectively queried H labels. The annotation policy π(x) controls the probability of querying H as a function of the input. The main theoretical results are Proposition 1, giving the optimal constant sampling rate, and Proposition 2, giving the optimal covariate-dependent rule π_active(x) = min(γ*/√u(x), 1) where u(x) = E[(H−G)^2 | X=x]. The policies minimize mean squared error subject to a cost budget. Synthetic experiments characterize the role of MSE(H,G), Var(U), and the cost ratio; real-data experiments on Chatbot Arena, AQA, ImageNet, and Seahorse compare the estimated random and active policies against a strong-rater-only baseline, with an oracle active policy added to quantify headroom. The paper is candid that the estimated uncertainty u(x) is 'far from perfect' and that the oracle 'often promises significant headroom,' and it positions better uncertainty estimation as future work.

Significance. The theoretical contribution is genuine and useful: it extends active statistical inference to a cost-constrained setting, correctly optimizes the sampling ratio rather than taking it as fixed, and handles the [0,1] constraint that prior work ignored. The closed-form solutions are simple enough to be of practical value, and the synthetic experiments give an informative map of when active sampling helps. The real-data section, however, does not establish the abstract's headline claim that the policies achieve 'far lower total annotation budget' in realistic settings: with the plug-in uncertainty estimator G(1−G), the active policy is close to random on AQA and homogeneous Chatbot Arena, and similar on ImageNet and Seahorse; only the constructed easy/hard split shows a large active benefit. The paper's own oracle results show that the failure is due to inaccurate u(x), not to the theory. The theoretical results are machine-checkable in the sense that the derivations are detailed and mostly rigorous, though one proof step in Proposition 1 appears to have a sign error. Overall, this is a solid theoretical paper whose empirical claims need recalibration.

major comments (4)
  1. [Abstract, Section 4.3, Appendix D.2] The abstract and Section 4.3 claim that the proposed policies can achieve the same precision at a 'far lower total annotation budget' than standard evaluation. This is only demonstrated for the estimated random policy and, for the active policy, only on the constructed easy/hard split of Chatbot Arena. On AQA and homogeneous Chatbot Arena (Figure 3), the estimated active policy nearly coincides with the random policy; the same holds for ImageNet and Seahorse in Appendix D.2 (Figure 5). Meanwhile the oracle active policy, which uses the true |h(x)−g(x)|^2, shows large headroom. The paper should either present real-data results with uncertainty estimates that are demonstrably accurate (e.g., a learned regressor for the squared error, or verbalized LLM confidence with calibration checks), or substantially temper the abstract and conclusion so that the practical benefit is claimed only for the fixed-rate policy.
  2. [Section 4.2, Proposition 2] The plug-in u(x)=G(1−G) equals E[(H−G)^2|X=x] only when G(x)=P(H=1|X=x) is perfectly calibrated. The paper applies Platt scaling on the transfer or burn-in set, but reports no calibration diagnostics (reliability diagrams, expected calibration error, or a test of whether E[(H−G)^2|X=x] is actually close to G(1−G) on held-out data). Because accurate u(x) is the load-bearing condition for Proposition 2, the empirical comparison against the oracle is not a valid test of the active policy's practical value unless calibration is verified. Adding calibration analyses and comparing alternative uncertainty estimators would directly address the observed gap between estimated and oracle active policies.
  3. [Appendix C.1, Proof of Proposition 1] The comparison between the interior critical point and the boundary value p*=1 has a sign error. The proof states that the inequality 0 < (c_h(Var(H)−E)−c_g E)^2 'cannot hold' under the condition (c_g+c_h)E < c_h Var(H), but the squared quantity is strictly positive in that regime so the inequality does hold, implying the interior point is better. This is the opposite of what the proof concludes. The final formula in Proposition 1 is consistent with choosing the interior point under that condition, so the error is likely a typo, but the proof needs to be corrected because it currently contradicts its own conclusion.
  4. [Appendix B.5, Propositions 8 and 9] The variance-inflation bounds in Proposition 8 and Corollary 9 are conditioned on E[1/π_tilde − 1/π*] ≤ δ. The real-data experiments imply this δ is large in practice, because otherwise the estimated active policy would perform close to the oracle. The paper should state this limitation explicitly next to the bounds, rather than presenting them as a robustness guarantee that could be read as covering the experimental regime. Otherwise readers may overestimate the safety of plug-in uncertainty estimates.
minor comments (4)
  1. [Appendix B.1] In the derivation of Var(Δπ), the expression 'Δπ = G + (H−G) 2 ξ/π(X)' appears to contain a typo: the '(H−G)' term should not be squared, since Eq. (1) defines Δt = G_t + (H_t−G_t) ξ_t/π_t(X_t). Please correct the exponent.
  2. [Sections 3.2 and 3.3] The text says the Beta distribution is 'satified' by U∼Beta(κμ, κ(1−μ)) and similarly 'satified' in the Gaussian construction; these should be 'satisfied'. There are a few similar typographical errors (e.g., 'doe not depend' in Section 3.1).
  3. [Section 3.1, ErrorRatio definition] The convention for ErrorRatio(π1,π2) treats π_base specially by discarding c_g in the denominator. This is explained, but the definition as written could mislead readers who compare it with the general formula; a brief note that the baseline is the estimator with π≡1 and cost c_h per sample would help.
  4. [Figure 2 caption and Section 4.3] In the text, the easy/hard split is described as constructed from the bottom 25% and top 25% of Gemma's uncertainty estimates, but it is not immediately clear whether the 'top' and 'bottom' are with respect to u(x)=G(1−G) or the underlying model softmax. Clarifying this would aid reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the optimal annotation policies are derived self-contained from the estimator's exact MSE, and the paper's oracle/limitation discussion is honest rather than a disguised fit.

full rationale

The derivation chain starts from the unbiased estimator (1), computes the exact mean squared error and cost in (2), and then solves the stated constrained optimization problems. Proposition 1 is obtained by substituting the budget-determined stopping time T_stop = B/(c_h p + c_g) and minimizing the resulting convex objective in p (Appendix C.1). Proposition 2 is obtained from the same objective, with a Lagrangian analysis over the discrete space X, yielding the clipped policy pi_active(x)=min(gamma*/sqrt(u(x)),1) and a one-dimensional search over the threshold tau (Appendix C.2). These results are not assumed or fitted; they are minimizers of the paper's own error/cost objective. The empirical section uses plug-in estimates such as u(x)=G(1-G), which the paper explicitly notes is equal to the target conditional MSE only under the calibration condition g(x)=P(H=1|X=x), and it acknowledges that the estimated u(x) is 'far from perfect' and that the oracle policy is only an upper-bound headroom. The easy/hard split is a constructed diagnostic for heteroskedasticity and is labeled as partly manipulated; it is not presented as a general prediction. Self-citations to PPI and active inference provide background methods and a baseline estimator, but the optimality proofs do not reduce to those citations: the balance equations and boundary cases are carried out in the paper. No prediction or derived quantity is, by construction, equal to its input, and no fitted parameter is renamed as a prediction. The paper's own limitation statements about cold start and imperfect uncertainty estimates are correctly framed as practical challenges, not as evidence of circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The theoretical contribution is essentially parameter-free: the optimal policies are derived from the data distribution. The empirical application introduces plug-in estimates for the unknown distributional quantities, and the practical gains depend on the accuracy of these estimates. No new physical or theoretical entities are postulated.

free parameters (2)
  • policy parameters (gamma*, tau*) for active policy = estimated via burn-in (first 200 examples) or transferred from related dataset
    The theoretically optimal active policy depends on Var(H), MSE(H,G), and the distribution of U; in practice these are replaced by plug-in estimates (Section 4), so the empirical performance depends on these fitted values.
  • conditional uncertainty function u(x) = chosen as G(1-G) or model confidence
    The active policy requires u(x)=E[(H-G)^2|X=x], which is unknown; experiments substitute heuristics that may be miscalibrated, affecting the active policy's advantage.
assumptions (5)
  • standard math X_1,...,X_T are i.i.d. from P_X; H and G have finite second moments.
    Used throughout Section 2 to derive error and cost expressions.
  • domain assumption The cost of querying H is c_h and G is c_g with c_h > c_g > 0; every example is always rated by G at cost c_g.
    The cost model in Eq. (3) and all policy derivations assume G is queried on every example; if G can be skipped, the optimization changes.
  • domain assumption For Proposition 2, X is discrete and the policy must lie in (0,1] for every x.
    The proof uses discreteness to derive the Lagrangian coordinate-wise; the positivity avoids infinite variance.
  • domain assumption The uncertainty estimate u(x)=E[(H-G)^2|X=x] is available or can be accurately approximated.
    The active policy's optimality and empirical benefit rely on this; Section 4 acknowledges it is imperfect and the oracle headroom shows the gap.
  • domain assumption The sampling indicator xi_t is independent of H_t and G_t given X_t, and G_t is sampled from P_G|X and H_t from P_H|X.
    This conditional independence is implicit in the sequential sampling protocol and is required for the estimator (1) to be unbiased.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cost-Optimal Active AI Model Evaluation." pith.science (2026). https://pith.science/paper/TIBGYRW2

@misc{pith2026250607949,
  author       = {Pith},
  title        = {Pith review of: Cost-Optimal Active AI Model Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TIBGYRW2}},
  note         = {Machine review of arXiv:2506.07949}
}
read the original abstract

The development lifecycle of generative AI systems requires continual evaluation, data acquisition, and annotation, which is costly in both resources and time. In practice, rapid iteration often makes it necessary to rely on synthetic annotation data because of the low cost, despite the potential for substantial bias. In this paper, we develop novel, cost-aware methods for actively balancing the use of a cheap, but often inaccurate, weak rater -- such as a model-based autorater that is designed to automatically assess the quality of generated content -- with a more expensive, but also more accurate, strong rater alternative such as a human. More specifically, the goal of our approach is to produce a low variance, unbiased estimate of the mean of the target "strong" rating, subject to some total annotation budget. Building on recent work in active and prediction-powered statistical inference, we derive a family of cost-optimal policies for allocating a given annotation budget between weak and strong raters so as to maximize statistical efficiency. Using synthetic and real-world data, we empirically characterize the conditions under which these policies yield improvements over prior methods. We find that, especially in tasks where there is high variability in the difficulty of examples, our policies can achieve the same estimation precision at a far lower total annotation budget than standard evaluation methods.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Prediction-Powered Active Testing

    stat.ML 2026-07 accept novelty 6.0 of 10

    PPAT residualizes losses via a prediction-powered control variate inside LURE, yielding lower-variance unbiased risk estimates, tailored acquisition, and asymptotic CIs that cover with fewer labels.

  2. CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion

    cs.LG 2026-07 accept novelty 6.0 of 10

    CollabEval turns model evaluation into low-rank matrix completion and uses the imputations as control variates to cut CI width and MSE at fixed annotation budget while preserving unbiasedness.

Reference graph

Works this paper leans on

37 extracted references · 16 canonical work pages · cited by 2 Pith papers

  1. [1]

    Prediction-powered inference.Science, 382(6671):669–674, 2023

    Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I Jordan, and Tijana Zrnic. Prediction-powered inference.Science, 382(6671):669–674, 2023

    2023

  2. [2]

    PPI++: Efficient prediction- powered inference.arXiv preprint arXiv:2311.01453, 2023

    Anastasios N Angelopoulos, John C Duchi, and Tijana Zrnic. PPI++: Efficient prediction- powered inference.arXiv preprint arXiv:2311.01453, 2023

    arXiv 2023

  3. [3]

    Bernd Bohnet, Vinh Q. Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, Tom Kwiatkowski, Ji Ma, Jianmo Ni, Lierni Sestorain Saralegui, Tal Schuster, William W. Cohen, Michael Collins, Dipanjan Das, Donald Metzler, Slav Petrov, and Kellie Webster. Attribute...

    arXiv 2023

  4. [4]

    AutoEval done right: Using synthetic data for model evaluation.arXiv preprint arXiv:2403.07008, 2024

    Pierre Boyeau, Anastasios N Angelopoulos, Nir Yosef, Jitendra Malik, and Michael I Jordan. AutoEval done right: Using synthetic data for model evaluation.arXiv preprint arXiv:2403.07008, 2024

    arXiv 2024

  5. [5]

    The price of debiasing automatic metrics in natural language evalaution

    Arun Chaganty, Stephen Mussmann, and Percy Liang. The price of debiasing automatic metrics in natural language evalaution. In Iryna Gurevych and Yusuke Miyao, editors,Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643–653, Melbourne, Australia, July 2018. Association for Computational...

    doi:10.18653/v1/p18-1060 2018

  6. [6]

    Prediction- powered ranking of large language models

    Ivi Chatzi, Eleni Straitouri, Suhas Thejaswi, and Manuel Gomez Rodriguez. Prediction- powered ranking of large language models. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Infor- mation Processing Systems, volume 37, pages 113096–113133. Curran Associates, Inc.,

    work page

  7. [7]

    Double/debiased machine learning for treatment and structural parameters.The Econometrics Journal, 21(1):C1–C68, 01 2018

    Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters.The Econometrics Journal, 21(1):C1–C68, 01 2018. ISSN 1368-4221. doi: 10. 1111/ectj.12097. URLhttps://doi.org/10.1111/ectj.12097

    doi:10.1111/ectj.12097 2018

  8. [8]

    Gonzalez, and Ion Stoica

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference.arXiv preprint: 2403.04132, 2024

    arXiv 2024

Show all 37 references
  1. [9]

    SEA- HORSE: A multilingual, multifaceted dataset for summarization evaluation

    Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, Vi- taly Nikolaev, Thibault Sellam, Aditya Siddhant, Dipanjan Das, and Ankur Parikh. SEA- HORSE: A multilingual, multifaceted dataset for summarization evaluation. In Houda Bouamor, Juan Pino, a...

    2023 doi

  2. [10]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  3. [11]

    Dorner, Vivian Yvonne Nastl, and Moritz Hardt

    Florian E. Dorner, Vivian Yvonne Nastl, and Moritz Hardt. Limits to scalable evaluation at the frontier: LLM as judge won’t beat twice the data. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=NO6Tv6QcDs

    2025

  4. [12]

    Using imperfect surrogates for downstream inference: Design-based supervised learning for social science applications of large language models

    Naoki Egami, Musashi Hinck, Brandon Stewart, and Hanying Wei. Using imperfect surrogates for downstream inference: Design-based supervised learning for social science applications of large language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine,...

    2023

  5. [13]

    Alex Hofer, Bhuwan Dhingra, Amir Globerson, and William W

    Adam Fisch, Joshua Maynez, R. Alex Hofer, Bhuwan Dhingra, Amir Globerson, and William W. Cohen. Stratified prediction-powered inference for effective hybrid evaluation of language models. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL...

    2024

  6. [14]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024

    Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URLhttps://arxiv.org/abs/2403.05530

    2024 arXiv

  7. [15]

    Gemma 3 technical report.arXiv preprint: arXiv 2503.19786, 2025

    Gemma Team. Gemma 3 technical report.arXiv preprint: arXiv 2503.19786, 2025. URL https://arxiv.org/abs/2503.19786

    2025 arXiv

  8. [16]

    Can uncon- fident llm annotations be used for confident conclusions?arXiv preprint arXiv:2408.15204, 2024

    Kristina Gligorić, Tijana Zrnic, Cinoo Lee, Emmanuel J Candès, and Dan Jurafsky. Can uncon- fident llm annotations be used for confident conclusions?arXiv preprint arXiv:2408.15204, 2024. 12 Cost-Optimal Active AI Model Evaluation

    2024 arXiv

  9. [17]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  10. [18]

    True: Re-evaluating factual consistency evaluation.arXiv preprint: 2204.04991, 2022

    Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. True: Re-evaluating factual consistency evaluation.arXiv preprint: 2204.04991, 2022

    2022 arXiv

  11. [19]

    Trust or escalate: LLM judges with provable guarantees for human agreement

    Jaehun Jung, Faeze Brahman, and Yejin Choi. Trust or escalate: LLM judges with provable guarantees for human agreement. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=UHPnqSTBPO

    2025

  12. [20]

    Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221, 2022

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221, 2022

    2022 arXiv

  13. [21]

    Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods

    John C Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. InAdvances in large margin classifiers, pages 61–74. MIT Press, 1999

    1999

  14. [22]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of Machine Learning Research, 21(140):1–67, 2020. URL http://...

    2020

  15. [23]

    B. D. Ripley.Stochastic simulation. John Wiley & Sons, Inc., New York, NY, USA, 1987. ISBN 0-471-81884-4

    1987

  16. [24]

    Robins and Andrea Rotnitzky

    James M. Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data.Journal of the American Statistical Association, 90(429):122–129,

  17. [25]

    ARES: An automated evaluation framework for retrieval-augmented generation systems

    Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. ARES: An automated evaluation framework for retrieval-augmented generation systems. In Kevin Duh, Helena Gomez, and Steven Bethard, editors,Proceedings of the 2024 Conference of the North American Chapter of ...

    2024 doi

  18. [26]

    Judging thejudges: Evaluatingalignmentand vulnerabilitiesin llms-as-judges

    Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and DieuwkeHupkes. Judging thejudges: Evaluatingalignmentand vulnerabilitiesin llms-as-judges. ArXiv preprint: arXiv 2406.12624, 2025. URLhttps://arxiv.org/abs/2406.12624

    2025 arXiv

  19. [27]

    (Anastasios Athanasios) Tsiatis.Semiparametric theory and missing data

    Anastasios A. (Anastasios Athanasios) Tsiatis.Semiparametric theory and missing data. Springer series in statistics. Springer, New York, 2006. ISBN 9780387373454

    2006

  20. [28]

    Cambridge university press, 2000

    Aad W Van der Vaart.Asymptotic statistics, volume 3. Cambridge university press, 2000

    2000

  21. [29]

    Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps:// openreview.net...

    2024

  22. [30]

    mt5: A massively multilingual pre-trained text-to-text transformer

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020

    2010 arXiv

  23. [31]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. InThirty-seventh Conference on Neural Inform...

    2023

  24. [32]

    Active statistical inference.arXiv preprint arXiv:2403.03208, 2024

    Tijana Zrnic and Emmanuel J Candès. Active statistical inference.arXiv preprint arXiv:2403.03208, 2024

    2024 arXiv

  25. [33]

    𝐺+(𝐻−𝐺) 𝜉 𝜋(𝑋) 2# −(𝜃∗)2 =𝔼 𝐺2 +𝔼

    Tijana Zrnic and Emmanuel J. Candès. Cross-prediction-powered inference.Proceedings of the National Academy of Sciences, 121(15):e2322083121, 2024. doi: 10.1073/pnas.2322083121. URLhttps://www.pnas.org/doi/abs/10.1073/pnas.2322083121. 14 Cost-Optimal Active AI Model Evaluation...

    2024 doi

  26. [36]

    The variance,Var(Δ𝜋), of each active incrementΔ𝜋, whereΔ𝜋 =𝐺+(𝐻−𝐺) 𝜉 𝜋(𝑋)

  27. [37]

    attribution to the source document

    The average sample size at which the estimator runs out of budget,𝑇stop = 𝐵 𝑐ℎ𝔼[𝜋(𝑋)]+𝑐 𝑔 . In this section, we provide some additional theoretical analysis on the first factor, i.e., the increase in Var(Δ𝜋) due to the mispecification error of anestimatedactive policy, while n...

  28. [1995]

    URLhttp://www.jstor.org/stable/2291135

    ISSN 01621459. URLhttp://www.jstor.org/stable/2291135

  29. [2024]

    URL https://proceedings.neurips.cc/paper_files/paper/2024/file/ cd47cd67caa87f5b1944e00f6781598f-Paper-Conference.pdf

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.