Pith. sign in

REVIEW 4 major objections 7 minor 52 references

With a small budget of human labels, BACON converts biased AI judge scores into calibrated item-level scores and population-level estimates with valid confidence intervals.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

BACON calibrates multiple AI judges against a small human-labeled sample, then uses cross-fitted outcome models and augmented estimating equations to produce calibrated summary estimates and item-level surrogate scores.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection A solid, honest translation of prediction-powered inference into a practical LLM-judge calibration protocol; the small-budget CI guarantee is asserted rather than checked, but the headline mean-estimation results hold. the 4 major comments →

arxiv 2607.16239 v1 pith:PKZQWL6E submitted 2026-06-26 cs.LG stat.AP

BACON: Budgeted Human Calibration for Modeling and Evaluation with Multiple AI Judges

Lei Shi , Anlan Zhang , Rita Lyu , Zhengmian Hu , Tong Yu , David Arbour , Avi Feller , Saayan Mitra
show 1 more author
Ritwik Sinha
This is my paper
classification cs.LG stat.AP
keywords AI judgeshuman calibrationaugmented estimating equationscross-fittingbudgeted labelingitem scoringsummary estimationevaluation bias
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BACON claims that a few hundred human labels suffice to turn a panel of biased AI judges into calibrated, statistically grounded evaluation tools. The method treats AI judge scores, their uncertainty statistics, and item embeddings as auxiliary features, fits a cross-fitted outcome model to a small human sample, and then uses an augmented estimating-equation estimator for population-level summaries. The central promise: the resulting estimates are consistent for the human measurement and come with confidence intervals that attain nominal coverage, while item-level surrogates beat raw AI scores and human-only baselines on accuracy and ranking. The paper demonstrates this across essay grading, machine translation quality, and web-design perception. If true, this decouples the scale of AI evaluation from the cost of human labels without inheriting AI biases.

Core claim

The paper's central claim is that the augmented estimating-equation (AEE) estimator, combined with cross-fitted outcome models, is consistent for human-assessment summary statistics and yields valid confidence intervals; and that the same outcome model provides calibrated item-level surrogate scores that improve accuracy and ranking consistency over raw AI outputs and human-label-only baselines. The authors treat AI judges not as ground truth but as measurements: human labels anchor the estimator, while AI-derived signals improve efficiency. The AEE estimator defines the target summary statistic as the solution to an estimating equation, then corrects the unlabeled mass with an inverse-prope

What carries the argument

The load-bearing object is the augmented estimating-equation (AEE) estimator: for a target like a population mean or quantile, it solves an equation that combines an inverse-propensity-weighted residual from the labeled subset with a full-coverage regression adjustment (or conditional distribution adjustment for quantiles). Cross-fitting produces honest out-of-fold predictions so the residual is not artificially small, and a sandwich variance estimator yields confidence intervals. Outcome models are chosen to match the score pattern—ridge for real-valued scores, a hurdle model for zero-inflated non-negative scores, proportional odds for ordinal scores, and multinomial regression for categori

Load-bearing premise

The confidence intervals are only valid if the estimated nuisance functions—the labeling propensity and the regression adjustments—are accurate enough that their errors wash out before the sampling noise does; the paper assumes this without demonstrating it for all its outcome models.

What would settle it

Take a dataset with known human labels and a deliberately uninformative AI judge (e.g., random scores), then run BACON's AEE estimator for the population mean at a 5% sampling budget with a flexible outcome model; if the empirical coverage of the nominal 95% intervals falls noticeably below 95% across many replications, the orthogonality assumption is not met. Alternatively, compute the Gateaux derivative of the mean and quantile estimating equations with respect to the regression adjustment; a nonzero derivative at the truth would invalidate the claimed o_p(N^{-1/2}) property.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • At a 5% human sampling budget, the full hybrid outcome model achieves non-trivial out-of-fold R2 and Spearman correlation on PERSUADE and MQM, meaning modest annotations already yield useful surrogate scores.
  • The AEE estimator stays approximately unbiased and maintains Bonferroni-corrected 95% CI coverage across MQM budgets, while raw AI judge averages are biased.
  • System-level rankings of machine translation systems recover the true order with high Spearman and Kendall correlations using only small human samples.
  • On WebDesign, calibrated outcome models beat uncalibrated VLM scores by a wide margin, showing the calibration layer is essential when AI judges operate on a different scale than human raters.
  • Better outcome models directly translate into lower estimator variance, so investing in AI-judge features and embeddings pays off in both item-level and aggregate accuracy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the AEE estimator's consistency does not rely on the outcome model being correct, a practitioner could in principle use any collection of AI judges—even a weakly predictive one—and still obtain unbiased summary estimates as long as the human sample is drawn by design and the orthogonality condition holds.
  • The appendix's variance decomposition suggests that adaptive sampling proportional to the outcome model's conditional error could further halve MSE under heteroskedasticity; that extension is not part of the main empirical validation and would need separate testing on real data.
  • Item-level surrogate scores are explicitly not unbiased per item; a safe deployment would reserve them for triage and ranking while routing close calls or high-stakes items to human review.
  • The same estimating-equation machinery could be applied to other estimands like tail quantiles or system-level differences; verifying CI coverage for those would be a natural next check.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes BACON, a four-stage evaluation pipeline that combines budgeted human labels with multiple AI-judge outputs as auxiliary measurements. For every item it constructs features from judge scores, token-level uncertainty statistics, and contextual embeddings; then it samples a small human-labeled subset, fits a cross-fitted outcome model (ridge, hurdle, ordinal, or multinomial depending on the outcome type), and uses the resulting predictions in two modes: (i) augmented estimating-equation (AEE) estimation of population summary metrics such as means and quantiles with sandwich confidence intervals, and (ii) item-level surrogate scoring and ranking. The paper reports experiments on PERSUADE essay scoring, WMT 2020 MQM machine-translation evaluation, and WebDesign screenshot scoring, claiming that the calibrated outcome model improves predictive accuracy and ranking consistency while the AEE estimator reduces bias and variance relative to raw AI outputs and human-label-only baselines. The theoretical framework is presented in Appendix C.1, and an adaptive-sampling extension is described in Appendix D.

Significance. If the claims hold, BACON offers a practically useful and statistically principled way to combine scarce human labels with abundant AI-judge outputs, with the important caveat that human labels remain the calibration anchor. The paper explicitly acknowledges that the mean estimator is mathematically equivalent to prediction-powered inference / regression-adjusted survey estimation, which is honest and appropriate. The cross-fitted AEE estimator for the population mean under uniform sampling with known propensity is on solid theoretical ground, and the empirical results on realistic multilingual and multi-modal datasets, together with the released code, are valuable. The main significance would be in operationalizing PPI-style estimation for AI evaluation with concrete feature recipes and model choices, and in demonstrating gains on real evaluation tasks. However, the broader claim of valid confidence intervals for general estimands (quantiles, adaptive sampling) and the small-budget coverage behavior are not yet established.

major comments (4)
  1. [Appendix C.1, Theorem 1 and Assumption 4] This is load-bearing because the central promise of 'valid confidence intervals' for the general AEE estimator is not supported without it.
  2. [Section 3.3 and Figures 4a, 5, 7] If coverage is omitted because it is undercoverage, that would directly undermine the main claim; if it is omitted because of space, a brief statement or an appendix table would suffice.
  3. [Appendix D.3.2 (Adaptive sampling simulation)] This is load-bearing for the adaptive-sampling extension, which is presented as a contribution in Section 2.2 and Appendix D.
  4. [Section 2.3, estimating equations for quantiles] This matters because the abstract and introduction advertise quantiles as one of the supported estimands, and the current proof does not cover them.
minor comments (7)
  1. [Section 2.2, Stage 3] The sentence 'We leave an detailed discussion of adaptive sampling strategies to Appendix D' contains a typo ('an' should be 'a detailed'). Also, the pipeline diagram uses 'bf(e i, ℓi, ui)' with inconsistent italic formatting.
  2. [Section 3.1, Embedding Features] The PCA dimensionality for embeddings is described as 'tunable' but the actual value used in the experiments is not reported. Since the number of principal components is a free parameter, please state the chosen values for each dataset or include a sensitivity analysis.
  3. [Section 3.3, MQM simulation] The paper mentions 'winsorised LLM score features' but does not define the winsorisation threshold or the precise transformation. Please specify this in the appendix or supplement.
  4. [Section 3.3, Figure 5] The description says 'All methods remain approximately unbiased throughout (max |bias|<0.02)' but the figure is small; please consider adding error bars or numeric values, and clarify what the reported coverage is over (e.g., across systems, across Monte Carlo trials).
  5. [Appendix C.2, Proof of Theorem 1] The proof uses both n and N without defining the relationship; the expansion writes r_n = o_p(n^{-1/2}) while the theorem statement uses N. This should be harmonized.
  6. [Section 3.3] The phrase 'Besides, all calibrated methods maintain Bonferroni-corrected CI coverage at or above the nominal 95% level throughout' appears without a citation to a figure or table. Please refer explicitly to the panel in Figure 5.
  7. [Section 1.2] The contribution list says 'decoupling estimation accuracy from the quality of the outcome model.' This is true for bias under known propensity, but variance still depends strongly on outcome-model quality. Consider rephrasing to 'decoupling the bias of summary estimates from the quality of the outcome model' to avoid overstatement.

Circularity Check

0 steps flagged

No significant circularity: the human-label anchor and cross-fitting keep predictions external to the estimand.

full rationale

BACON's central derivation is self-contained against its own inputs. Human labels are collected on a sampled subset and used both to fit the cross-fitted outcome model and to form the augmented estimating-equation (AEE) estimator; the target is the population-level human summary statistic, not the fitted model's own output. The paper explicitly acknowledges that for the mean the estimator is not new: 'For the special case of estimating a population mean, our estimator is mathematically equivalent to a PPI / regression-adjusted estimator with a learned prediction function.' Cross-fitting is used precisely so that residual-based bias correction reflects genuine generalization error rather than in-sample fit. No fitted parameter is renamed as a prediction of the same quantity, no self-citation is load-bearing, and no uniqueness theorem is imported from the authors' prior work. The main theoretical weakness—Theorem 1's Assumption 4 is asserted rather than verified for the quantile/hurdle equations—is a correctness gap, not a circular reduction. The Appendix D adaptive-sampling experiment reuses pilot residuals to fit propensities and then includes the same pilot labels in the final AEE estimator; this is a sample-reuse/validity concern about the adaptive extension, but it does not make the estimator definitionally equivalent to its inputs. Thus no step in the derivation chain reduces by construction to its own input.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

BACON's consistency is anchored by design in the headline experiments (uniform sampling, known propensity). The general validity claim relies on unverified nuisance conditions, and the efficiency claims depend on feature/model choices that are not fully specified. No new entities are introduced.

free parameters (5)
  • PCA dimensionality for embeddings = tunable, unspecified
    Section 3.1 says PCA retains a tunable number of principal components; this affects outcome-model performance and efficiency claims.
  • Ridge penalty = not reported
    Ridge regression is used in multiple outcome models (Table 1, Table 3), but the penalty value or selection criterion is not given.
  • Winsorisation threshold for MQM LLM scores = not reported
    Section 3.3 uses 'winsorised LLM score features' in simulations without specifying the threshold.
  • K-fold cross-fitting K = 5
    Section 2.3 sets K=5 to balance bias and variance; reasonable but a modeling choice.
  • Hurdle vs ridge and sqrt vs log variance encoding = selected per dataset
    The sqrt-variance encoding and hurdle model were chosen after observing the log-encoding floor at -18.4 produce pathological R2; this is data-dependent model selection (Appendix A.1).
axioms (4)
  • domain assumption Human labels are missing at random conditional on features X under the sampling design, with positivity π(X)≥ε.
    Required for AEE consistency (Theorem 1, Assumption 2). Holds by construction for the uniform sampling used in main experiments, but is an assumption for the adaptive sampling extension.
  • ad hoc to paper Nuisance estimators satisfy Neyman orthogonality and have o_p(N^{-1/2}) first-order impact.
    Theorem 1 Assumption 4 asserts this; Appendix C.2 gives only a generic Z-estimation sketch and does not verify it for the quantile, hurdle, or estimated-propensity AEE moments.
  • standard math Z-estimation asymptotics: uniform LLN and CLT for the score functions near the truth.
    Theorem 1 Assumptions 3 and 5 are standard but are not checked for the flexible ML nuisance classes used in the experiments.
  • domain assumption AI judge scores, uncertainty features, and embeddings provide full-coverage auxiliary features for every item.
    Stage 1 assumes full coverage; if any item lacks judge responses or embeddings, the surrogate scoring and imputation break.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of BACON: Budgeted Human Calibration for Modeling and Evaluation with Multiple AI Judges." pith.science (2026). https://pith.science/paper/PKZQWL6E

@misc{pith2026260716239,
  author       = {Pith},
  title        = {Pith review of: BACON: Budgeted Human Calibration for Modeling and Evaluation with Multiple AI Judges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PKZQWL6E}},
  note         = {Machine review of arXiv:2607.16239}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

AI judges offer a scalable, low-cost alternative to human evaluation, but their outputs can be biased relative to human preferences and highly item-dependent, varying across judges, tasks, and domains. When uncalibrated AI evaluations are used for model ranking, item scoring, or population-level quality reporting, these biases can directly distort downstream decisions. We propose BACON, a four-stage pipeline that combines budgeted human calibration with multiple AI-judge outputs to produce more accurate annotations. BACON constructs full-coverage auxiliary features for every item, including multi-judge scores, token-level uncertainty statistics, and contextual embeddings. It then collects human labels for a small sampled subset and trains a cross-fitted outcome model to generate calibrated item-level surrogate predictions. These predictions support two use cases: population-level estimation of summary metrics, such as means or quantiles, using an augmented estimating-equation estimator with valid confidence intervals; and individual-level surrogate scoring for item ranking and annotation. BACON treats AI judges as auxiliary measurements rather than ground truth: human labels provide the calibration anchor, while AI-derived signals improve efficiency. Across diverse tasks, domains, and labeling budgets, BACON improves predictive accuracy and ranking consistency, and reduces bias and variance relative to raw AI outputs and purely human-label-based methods. These results show that BACON offers a practical, statistically grounded framework for scalable evaluation with limited human annotation.

Figures

Figures reproduced from arXiv: 2607.16239 by Anlan Zhang, Avi Feller, David Arbour, Lei Shi, Rita Lyu, Ritwik Sinha, Saayan Mitra, Tong Yu, Zhengmian Hu.

Figure 1
Figure 1. Figure 1: Score distribution comparisons between human raters and AI judges for PERSUADE and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: WebDesign Universities score distributions for six perceptual outcomes (scores [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: BACON evaluation pipeline. Stage 1 prepares inputs (context features and AI jury scores) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: PERSUADE experimental results. Top: bias, standard deviation, and RMSE as a function [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: MQM AEE estimator quality vs. sampling budget [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: MQM system ranking quality vs. sampling budget [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: WebDesign (Universities) AEE estimator quality: bias, SD, and RMSE vs. sampling budget [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: WebDesign (Universities) outcome model quality vs. sampling budget [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Synthetic simulation: bias and MSE of three population-mean estimators as a function of [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Sensitivity to pilot rate π with fixed budget η = 0.40 and λ = 1.5. Left: bias; right: MSE (log scale). 400 trials per setting [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Sensitivity to total budget η with fixed pilot rate π = 0.15 and λ = 1.5. Left: bias; right: MSE (log scale). 400 trials per setting. Varying π (fixed η = 0.40) [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    and Tanno, Ryutaro and Schwaighofer, Anton and Tezcan, Kerem C

    Bernhardt, Mélanie and Castro, Daniel C. and Tanno, Ryutaro and Schwaighofer, Anton and Tezcan, Kerem C. and Monteiro, Miguel and Bannur, Shruthi and Lungren, Matthew P. and Nori, Aditya and Glocker, Ben and Alvarez-Valle, Javier and Oktay, Ozan , month = mar, year =. Active label cleaning for improved dataset quality under resource constraints , volume =...

    doi:10.1038/s41467-022-28818-3

  2. [2]

    doi:10.48550/arXiv.2210.06812 , abstract =

    Goh, Hui Wen and Tkachenko, Ulyana and Mueller, Jonas , month = jan, year =. doi:10.48550/arXiv.2210.06812 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2210.06812

  3. [3]

    Kim, Hoyoung and Hwang, Sehyun and Kwak, Suha and Ok, Jungseul , month = jun, year =. Active. doi:10.48550/arXiv.2403.10820 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.10820

  4. [4]

    Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels

    Yun, Sangdoo and Oh, Seong Joon and Heo, Byeongho and Han, Dongyoon and Choe, Junsuk and Chun, Sanghyuk , month = jul, year =. Re-labeling. doi:10.48550/arXiv.2101.05022 , abstract =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2101.05022

  5. [5]

    and Athalye, Anish and Mueller, Jonas , month = nov, year =

    Northcutt, Curtis G. and Athalye, Anish and Mueller, Jonas , month = nov, year =. Pervasive. doi:10.48550/arXiv.2103.14749 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2103.14749

  6. [6]

    and Jiang, Lu and Chuang, Isaac L

    Northcutt, Curtis G. and Jiang, Lu and Chuang, Isaac L. , month = aug, year =. Confident. doi:10.48550/arXiv.1911.00068 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.1911.00068 1911

  7. [7]

    International Journal of Human-Computer Studies , volume=

    The effect of prototypicality on webpage aesthetics, usability, and trustworthiness , author=. International Journal of Human-Computer Studies , volume=. 2023 , publisher=

    2023

  8. [8]

    Goswami, Mononito and Sanil, Vedant and Choudhry, Arjun and Srinivasan, Arvind and Udompanyawit, Chalisa and Dubrawski, Artur , file =

  9. [9]

    2016 , pages =

    Proceedings of the VLDB Endowment , author =. 2016 , pages =. doi:10.14778/2994509.2994514 , abstract =

    arXiv 2016

  10. [10]

    Applied Statistics , author =

    Maximum. Applied Statistics , author =. 1979 , pages =. doi:10.2307/2346806 , abstract =

    doi:10.2307/2346806 1979

  11. [11]

    , month = mar, year =

    Dubois, Yann and Galambosi, Balázs and Liang, Percy and Hashimoto, Tatsunori B. , month = mar, year =. Length-. doi:10.48550/arXiv.2404.04475 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.04475

  12. [12]

    Prometheus 2:

    Kim, Seungone and Suk, Juyoung and Longpre, Shayne and Lin, Bill Yuchen and Shin, Jamin and Welleck, Sean and Neubig, Graham and Lee, Moontae and Lee, Kyungjae and Seo, Minjoon , month = dec, year =. Prometheus 2:. doi:10.48550/arXiv.2405.01535 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.01535

  13. [13]

    Kossen, Jannik and Farquhar, Sebastian and Gal, Yarin and Rainforth, Tom , file =. Active

  14. [14]

    2002 , publisher=

    Generalized estimating equations , author=. 2002 , publisher=

    2002

  15. [15]

    arXiv preprint arXiv:2507.01372 , year=

    Active Measurement: Efficient Estimation at Scale , author=. arXiv preprint arXiv:2507.01372 , year=

    arXiv

  16. [16]

    arXiv preprint arXiv:2511.08991 , year=

    Robust Sampling for Active Statistical Inference , author=. arXiv preprint arXiv:2511.08991 , year=

    arXiv

  17. [17]

    Fu, Yichao and Wang, Xuewei and Tian, Yuandong and Zhao, Jiawei , month = aug, year =. Deep. doi:10.48550/arXiv.2508.15260 , abstract =

    Pith/arXiv arXiv doi:10.48550/arxiv.2508.15260

  18. [18]

    Assessing Writing , volume=

    A large-scale corpus for assessing written argumentation: PERSUADE 2.0 , author=. Assessing Writing , volume=. 2024 , publisher=

    2024

  19. [19]

    Transactions of the Association for Computational Linguistics , volume=

    Experts, errors, and context: A large-scale study of human evaluation for machine translation , author=. Transactions of the Association for Computational Linguistics , volume=. 2021 , publisher=

    2021

  20. [20]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Sigmoid loss for language image pre-training , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

  21. [21]

    The Annals of Applied Statistics , volume=

    More power to you: Using machine learning to augment human coding for more efficient inference in text-based randomized trials , author=. The Annals of Applied Statistics , volume=. 2025 , publisher=

    2025

  22. [22]

    The Econometrics Journal , volume=

    Double/debiased machine learning for treatment and structural parameters , author=. The Econometrics Journal , volume=. 2018 , publisher=

    2018

  23. [23]

    Science , volume=

    Prediction-Powered Inference , author=. Science , volume=. 2023 , publisher=

    2023

  24. [24]

    and Duchi, John C

    Angelopoulos, Anastasios N. and Duchi, John C. and Zrnic, Tijana , journal=

  25. [25]

    Proceedings of the National Academy of Sciences , volume=

    Cross-Prediction-Powered Inference , author=. Proceedings of the National Academy of Sciences , volume=. 2024 , publisher=

    2024

  26. [26]

    1992 , publisher=

    Model Assisted Survey Sampling , author=. 1992 , publisher=

    1992

  27. [27]

    Biometrika , volume=

    Some Results on Generalized Difference Estimation and Generalized Regression Estimation for Finite Populations , author=. Biometrika , volume=. 1976 , publisher=

    1976

  28. [28]

    The Annals of Statistics , volume=

    Semi-supervised Inference: General Theory and Estimation of Means , author=. The Annals of Statistics , volume=. 2019 , publisher=

    2019

  29. [29]

    Journal of the Royal Statistical Society: Series B , volume=

    Semi-supervised Inference for Explained Variance in High-Dimensional Linear Regression and Its Applications , author=. Journal of the Royal Statistical Society: Series B , volume=. 2020 , publisher=

    2020

  30. [30]

    and Zhang, Hao and Gonzalez, Joseph E

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , booktitle=. Judging

  31. [31]

    Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang , booktitle=

  32. [32]

    arXiv preprint arXiv:2305.17926 , year=

    Large Language Models are Not Fair Evaluators , author=. arXiv preprint arXiv:2305.17926 , year=

    Pith/arXiv arXiv

  33. [33]

    Li, Haitao and Dong, Qian and Chen, Junjie and Su, Huawei and Hui, Yiran and Shi, Yue and Fang, Shiyu and Zhu, Xiaohui and Liu, Qingyao and Liu, Yiqun , journal=

  34. [34]

    A Survey on

    Gu, Jiawei and Jiang, Xuhui and Shi, Zhengshan and Tan, Hexiang and Zhai, Xuehao and Xu, Chengjin and Li, Wei and Shen, Lei and Zhu, Shengjie and Cheng, Fei and Ma, Jian , journal=. A Survey on

  35. [35]

    Dubois, Yann and Li, Xuechen and Taori, Rohan and Zhang, Tianyi and Gulrajani, Ishaan and Ba, Jimmy and Guestrin, Carlos and Liang, Percy and Hashimoto, Tatsunori B. , year=

  36. [36]

    Proceedings of the 41st International Conference on Machine Learning , year=

    Improving Factuality and Reasoning in Language Models through Multiagent Debate , author=. Proceedings of the 41st International Conference on Machine Learning , year=

  37. [37]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , year=

    Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , year=

    2024

  38. [38]

    Chan, Chi-Min and Chen, Weize and Su, Yusheng and Yu, Jianxuan and Xue, Wei and Zhang, Shanghang and Fu, Jie and Liu, Zhiyuan , journal=

  39. [39]

    International Conference on Learning Representations , year=

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author=. International Conference on Learning Representations , year=

  40. [40]

    Replacing Judges with Juries: Evaluating

    Verga, Pat and Hofst. Replacing Judges with Juries: Evaluating. arXiv preprint arXiv:2404.18796 , year=

    Pith/arXiv arXiv

  41. [41]

    Enhancing

    Qian, Yuqing and Zhang, Shenghua and Zhou, Yupeng and Balakrishnan, Anusha and Li, Jian and Jauhar, Sujay Kumar and Kannan, Anand and Tian, Ran , journal=. Enhancing

  42. [42]

    Zhu, Lianghui and Wang, Xinggang and Wang, Xinlong , booktitle=

  43. [43]

    International Conference on Learning Representations , year=

    Prometheus: Inducing Fine-grained Evaluation Capability in Language Models , author=. International Conference on Learning Representations , year=

  44. [44]

    arXiv preprint arXiv:2405.01535 , year=

    Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models , author=. arXiv preprint arXiv:2405.01535 , year=

    Pith/arXiv arXiv

  45. [45]

    Wang, Yidong and Yu, Zhuohao and Zeng, Zhengran and Yang, Linyi and Wang, Cunxiang and Chen, Hao and Jiang, Chaoya and Xie, Rui and Wang, Jindong and Xie, Xing and Ye, Wei and Zhang, Shikun and Zhang, Yue , booktitle=

  46. [46]

    arXiv preprint arXiv:2310.05470 , year=

    Generative Judge for Evaluating Alignment , author=. arXiv preprint arXiv:2310.05470 , year=

    Pith/arXiv arXiv

  47. [47]

    An Empirical Study of

    Huang, Hui and Bu, Xinrun and Zhou, Haoran and Li, Ke and Fan, Xinyu and Guo, Yan and Fei, Hao and He, Qingyu and Li, Jing and Mi, Fei , booktitle=. An Empirical Study of

  48. [48]

    The Computational Geometry Algorithms Library , author =

  49. [49]

    Menelaos Karavelas , subtitle =

  50. [50]

    The Computational Geometry Algorithms Library , subtitle =

    Menelaos Karavelas , editor =. The Computational Geometry Algorithms Library , subtitle =

  51. [51]

    The Parmap library , author =

  52. [52]

    Christopher Anderson and Sophia Drossopoulou , title =

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.