{"id":"ddff0f51-6265-4bed-9d63-faaa85dadb39","arxiv_id":"2607.16239","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"BACON calibrates multiple AI judges against a small human-labeled sample, then uses cross-fitted outcome models and augmented estimating equations to produce calibrated summary estimates and item-level surrogate scores.","lead":"This paper proposes a protocol — BACON — that combines scores from several AI judges with a small budget of human labels to estimate human-aligned evaluation metrics and item-level scores. The statistical engine is mostly standard semi-supervised/PPI estimation, but the concrete feature recipes and cross-domain tests make it a useful addition to AI-evaluation workflows.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CI validity in the small-budget regime is asserted but not demonstrated; Theorem 1's Assumption 4 is left unverified.","rationale":"I read the paper as a practical, well-executed application of prediction-powered inference to AI-judge calibration. The empirical results for mean estimation under uniform sampling are credible: the AEE estimator is unbiased in simulations, variance reduction tracks outcome-model quality, and MQM coverage is reported. The reader's weakest-assumption identification is essentially correct: Theorem 1's Assumption 4 is asserted rather than proved, and the quantile/adaptive cases lack verification. However, for the mean with known propensity, Neyman orthogonality is immediate and the main experiments likely satisfy the needed conditions. The more concrete soft spot is that CI coverage is never shown for PERSUADE and WebDesign, the two datasets where the smallest budgets yield n = 200 or fewer. Since the central promise is 'valid confidence intervals' under limited human labels, the absence of coverage evidence in those regimes is the most load-bearing concern. The recommended verdict remains CONDITIONAL: the paper's main empirical claims are probably sound, but the general CI guarantee and the small-sample behavior need explicit support before full acceptance. My disagreement with the reader is only partial: I would place at least as much weight on the empirical coverage gap as on the unproved Assumption 4, because the latter is likely fixable with a short argument for the mean case, while the former is a direct check that has not been reported.","tokens_in":16744,"tokens_out":28844,"duration_ms":309477,"concrete_test":"Re-run the existing Monte Carlo protocol on PERSUADE and WebDesign at η ∈ {0.05, 0.10, 0.20} and compute the empirical coverage of the 95% sandwich CIs (with Monte Carlo standard errors). If coverage at η = 0.05 is below 90% on either dataset, the 'valid confidence intervals' claim is not supported in the small-budget regime; if coverage is at or above nominal, the omission is a reporting gap rather than a substantive flaw.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires the AEE estimator to provide valid confidence intervals at the small human budgets BACON targets (η = 0.05–0.30). The paper reports Bonferroni-corrected CI coverage only for MQM, where the labeled sample is large (n ≈ 700 at η = 0.05). For PERSUADE (n ≈ 230 at η = 0.05) and WebDesign (n ≈ 30–40 at η = 0.05), Figures 4a and 7 show bias, SD, and RMSE but no coverage. The asymptotic guarantee rests on Theorem 1, Assumption 4: nuisance estimators must have a first-order effect o_p(N^{-1/2}). The paper asserts this “holds, for example, under Neyman orthogonality together with standard consistency rates” but never verifies orthogonality for the quantile estimating equation, nor gives rates for the cross-fitted ridge/ordinal/hurdle models, and the adaptive-sampling extension in Appendix D estimates propensities without addressing the additional nuisance uncertainty. For the mean with known propensity, the estimating equation is algebraically orthogonal to the regression adjustment, so the concern is concentrated in (i) finite-sample coverage of the sandwich variance in the smaller datasets and (ii) the quantile and estimated-propensity generalizations, where the stated condition is not checked. If the sandwich CIs undercover at small η, the headline promise of valid uncertainty quantification fails exactly in the resource-constrained setting BACON targets.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BACON, a four-stage evaluation pipeline that combines budgeted human labels with multiple AI-judge outputs as auxiliary measurements. For every item it constructs features from judge scores, token-level uncertainty statistics, and contextual embeddings; then it samples a small human-labeled subset, fits a cross-fitted outcome model (ridge, hurdle, ordinal, or multinomial depending on the outcome type), and uses the resulting predictions in two modes: (i) augmented estimating-equation (AEE) estimation of population summary metrics such as means and quantiles with sandwich confidence intervals, and (ii) item-level surrogate scoring and ranking. The paper reports experiments on PERSUADE essay scoring, WMT 2020 MQM machine-translation evaluation, and WebDesign screenshot scoring, claiming that the calibrated outcome model improves predictive accuracy and ranking consistency while the AEE estimator reduces bias and variance relative to raw AI outputs and human-label-only baselines. The theoretical framework is presented in Appendix C.1, and an adaptive-sampling extension is described in Appendix D.","tokens_in":17111,"tokens_out":5356,"duration_ms":61644,"significance":"If the claims hold, BACON offers a practically useful and statistically principled way to combine scarce human labels with abundant AI-judge outputs, with the important caveat that human labels remain the calibration anchor. The paper explicitly acknowledges that the mean estimator is mathematically equivalent to prediction-powered inference / regression-adjusted survey estimation, which is honest and appropriate. The cross-fitted AEE estimator for the population mean under uniform sampling with known propensity is on solid theoretical ground, and the empirical results on realistic multilingual and multi-modal datasets, together with the released code, are valuable. The main significance would be in operationalizing PPI-style estimation for AI evaluation with concrete feature recipes and model choices, and in demonstrating gains on real evaluation tasks. However, the broader claim of valid confidence intervals for general estimands (quantiles, adaptive sampling) and the small-budget coverage behavior are not yet established.","major_comments":[{"comment":"This is load-bearing because the central promise of 'valid confidence intervals' for the general AEE estimator is not supported without it.","section":"Appendix C.1, Theorem 1 and Assumption 4"},{"comment":"If coverage is omitted because it is undercoverage, that would directly undermine the main claim; if it is omitted because of space, a brief statement or an appendix table would suffice.","section":"Section 3.3 and Figures 4a, 5, 7"},{"comment":"This is load-bearing for the adaptive-sampling extension, which is presented as a contribution in Section 2.2 and Appendix D.","section":"Appendix D.3.2 (Adaptive sampling simulation)"},{"comment":"This matters because the abstract and introduction advertise quantiles as one of the supported estimands, and the current proof does not cover them.","section":"Section 2.3, estimating equations for quantiles"}],"minor_comments":[{"comment":"The sentence 'We leave an detailed discussion of adaptive sampling strategies to Appendix D' contains a typo ('an' should be 'a detailed'). Also, the pipeline diagram uses 'bf(e i, ℓi, ui)' with inconsistent italic formatting.","section":"Section 2.2, Stage 3"},{"comment":"The PCA dimensionality for embeddings is described as 'tunable' but the actual value used in the experiments is not reported. Since the number of principal components is a free parameter, please state the chosen values for each dataset or include a sensitivity analysis.","section":"Section 3.1, Embedding Features"},{"comment":"The paper mentions 'winsorised LLM score features' but does not define the winsorisation threshold or the precise transformation. Please specify this in the appendix or supplement.","section":"Section 3.3, MQM simulation"},{"comment":"The description says 'All methods remain approximately unbiased throughout (max |bias|<0.02)' but the figure is small; please consider adding error bars or numeric values, and clarify what the reported coverage is over (e.g., across systems, across Monte Carlo trials).","section":"Section 3.3, Figure 5"},{"comment":"The proof uses both n and N without defining the relationship; the expansion writes r_n = o_p(n^{-1/2}) while the theorem statement uses N. This should be harmonized.","section":"Appendix C.2, Proof of Theorem 1"},{"comment":"The phrase 'Besides, all calibrated methods maintain Bonferroni-corrected CI coverage at or above the nominal 95% level throughout' appears without a citation to a figure or table. Please refer explicitly to the panel in Figure 5.","section":"Section 3.3"},{"comment":"The contribution list says 'decoupling estimation accuracy from the quality of the outcome model.' This is true for bias under known propensity, but variance still depends strongly on outcome-model quality. Consider rephrasing to 'decoupling the bias of summary estimates from the quality of the outcome model' to avoid overstatement.","section":"Section 1.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of a CS/ML venue and the central estimator is well-grounded for the uniform-propensity mean case, but the general theoretical claims and the adaptive-sampling results need to be tightened. The main issues are (1) unverified Neyman orthogonality / rates for quantiles and flexible nuisance fits, (2) missing CI coverage for PERSUADE and WebDesign, and (3) an apparent circularity in the Appendix D synthetic study. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection. The authors should also consider moving the adaptive-sampling material into the main text only after the cross-fitting issue is resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Serdar—read this if you work on LLM-as-judge evaluation. The statistical core is not new: for population means, the augmented estimating-equation estimator is the standard regression-adjusted / PPI estimator with cross-fitting, and the authors say so plainly. What is new is the protocol layer: how to build auxiliary features from multiple judge scores, token-level uncertainty, and embeddings; practical outcome-model recipes for ordinal, zero-inflated, and categorical outcomes; and a three-domain empirical study. That combination is useful and largely well executed. The experiments use Monte Carlo trials with known propensity and cross-fitting, and the results are convincing: calibrated outcome models reduce bias and variance relative to raw AI averages and human-only baselines, and the ablations show each feature source contributes. The WebDesign scale-mismatch result—raw VLM scores giving R² ≈ −9.7—makes the calibration case concretely. The soft spots are concentrated in the general theory rather than the main experiments. Theorem 1's Assumption 4 requires the nuisance estimators to have first-order effects o_p(N^{-1/2}), and the paper asserts this follows from Neyman orthogonality “for example” without verifying orthogonality for the quantile estimating equation or the hurdle model, and without rates for the ridge/ordinal/hurdle fits. For the mean with uniform sampling and known propensity, the estimating equation is algebraically orthogonal to the regression adjustment, so the headline results are on solid ground. But the broader claim of “valid confidence intervals” for all estimands and for adaptive sampling is not demonstrated. The small-budget regime is also under-tested: CI coverage is reported for MQM, where the labeled sample is large, but not for PERSUADE or WebDesign at η=0.05, where the labeled sample is only ~230 and ~30–40. That is exactly the regime BACON targets. One further, minor concern: the adaptive-sampling synthetic study in Appendix D fits the propensity model on pilot residuals and then includes those same pilot labels in the final estimator; the variance does not appear to account for that dependence. It does not affect the main results, but it should be addressed if the appendix is kept. Overall, a fair, honest paper with real practical value. It deserves serious peer review; the main theoretical gap is addressable either by verifying orthogonality for the general estimating equations or by restricting the CI guarantee to the known-propensity mean case, and by adding coverage results in the small-budget settings.","headline":"A solid, honest translation of prediction-powered inference into a practical LLM-judge calibration protocol; the small-budget CI guarantee is asserted rather than checked, but the headline mean-estimation results hold.","tokens_in":713,"tokens_out":860,"would_cite":true,"duration_ms":32929,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"With a small budget of human labels, BACON converts biased AI judge scores into calibrated item-level scores and population-level estimates with valid confidence intervals.","keywords":["AI judges","human calibration","augmented estimating equations","cross-fitting","budgeted labeling","item scoring","summary estimation","evaluation bias"],"falsifier":"Take a dataset with known human labels and a deliberately uninformative AI judge (e.g., random scores), then run BACON's AEE estimator for the population mean at a 5% sampling budget with a flexible outcome model; if the empirical coverage of the nominal 95% intervals falls noticeably below 95% across many replications, the orthogonality assumption is not met. Alternatively, compute the Gateaux derivative of the mean and quantile estimating equations with respect to the regression adjustment; a nonzero derivative at the truth would invalidate the claimed o_p(N^{-1/2}) property.","tokens_in":16634,"feed_emoji":"🎯","tokens_out":5626,"duration_ms":55562,"temperature":0.7,"pith_summary":"BACON claims that a few hundred human labels suffice to turn a panel of biased AI judges into calibrated, statistically grounded evaluation tools. The method treats AI judge scores, their uncertainty statistics, and item embeddings as auxiliary features, fits a cross-fitted outcome model to a small human sample, and then uses an augmented estimating-equation estimator for population-level summaries. The central promise: the resulting estimates are consistent for the human measurement and come with confidence intervals that attain nominal coverage, while item-level surrogates beat raw AI scores and human-only baselines on accuracy and ranking. The paper demonstrates this across essay grading, machine translation quality, and web-design perception. If true, this decouples the scale of AI evaluation from the cost of human labels without inheriting AI biases.","feed_headline":"Small human sample turns biased AI judges into valid estimators","feed_subtitle":"A few hundred human labels turn AI judge outputs into unbiased summaries and reliable rankings.","key_machinery":"The load-bearing object is the augmented estimating-equation (AEE) estimator: for a target like a population mean or quantile, it solves an equation that combines an inverse-propensity-weighted residual from the labeled subset with a full-coverage regression adjustment (or conditional distribution adjustment for quantiles). Cross-fitting produces honest out-of-fold predictions so the residual is not artificially small, and a sandwich variance estimator yields confidence intervals. Outcome models are chosen to match the score pattern—ridge for real-valued scores, a hurdle model for zero-inflated non-negative scores, proportional odds for ordinal scores, and multinomial regression for categori","core_discovery":"The paper's central claim is that the augmented estimating-equation (AEE) estimator, combined with cross-fitted outcome models, is consistent for human-assessment summary statistics and yields valid confidence intervals; and that the same outcome model provides calibrated item-level surrogate scores that improve accuracy and ranking consistency over raw AI outputs and human-label-only baselines. The authors treat AI judges not as ground truth but as measurements: human labels anchor the estimator, while AI-derived signals improve efficiency. The AEE estimator defines the target summary statistic as the solution to an estimating equation, then corrects the unlabeled mass with an inverse-prope","pith_inferences":["Because the AEE estimator's consistency does not rely on the outcome model being correct, a practitioner could in principle use any collection of AI judges—even a weakly predictive one—and still obtain unbiased summary estimates as long as the human sample is drawn by design and the orthogonality condition holds.","The appendix's variance decomposition suggests that adaptive sampling proportional to the outcome model's conditional error could further halve MSE under heteroskedasticity; that extension is not part of the main empirical validation and would need separate testing on real data.","Item-level surrogate scores are explicitly not unbiased per item; a safe deployment would reserve them for triage and ranking while routing close calls or high-stakes items to human review.","The same estimating-equation machinery could be applied to other estimands like tail quantiles or system-level differences; verifying CI coverage for those would be a natural next check."],"forward_implications":["At a 5% human sampling budget, the full hybrid outcome model achieves non-trivial out-of-fold R2 and Spearman correlation on PERSUADE and MQM, meaning modest annotations already yield useful surrogate scores.","The AEE estimator stays approximately unbiased and maintains Bonferroni-corrected 95% CI coverage across MQM budgets, while raw AI judge averages are biased.","System-level rankings of machine translation systems recover the true order with high Spearman and Kendall correlations using only small human samples.","On WebDesign, calibrated outcome models beat uncalibrated VLM scores by a wide margin, showing the calibration layer is essential when AI judges operate on a different scale than human raters.","Better outcome models directly translate into lower estimator variance, so investing in AI-judge features and embeddings pays off in both item-level and aggregate accuracy."],"fun_headline_variants":["Fixing AI judge bias with a handful of human labels","Budgeted human calibration turns AI judges into valid estimators","Small human budget corrects AI judge biases for rankings","Human-anchored AI judges produce unbiased metrics","Calibrate multiple AI judges with minimal human labels"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The confidence intervals are only valid if the estimated nuisance functions—the labeling propensity and the regression adjustments—are accurate enough that their errors wash out before the sampling noise does; the paper assumes this without demonstrating it for all its outcome models.","fun_headline_variants_meta":{"raw":{"variants":["Fixing AI judge bias with a handful of human labels","Budgeted human calibration turns AI judges into valid estimators","Small human budget corrects AI judge biases for rankings","Human-anchored AI judges produce unbiased metrics","Calibrate multiple AI judges with minimal human labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1272,"prompt_tokens":751,"completion_tokens":521,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":445}},"tokens_in":495,"tokens_out":521,"duration_ms":6359,"temperature":1.0,"reasoning_tokens":445,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T09:54:08.125298+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a dataset with known human labels and a deliberately uninformative AI judge (e.g., random scores), then run BACON's AEE estimator for the population mean at a 5% sampling budget with a flexible outcome model; if the empirical coverage of the nominal 95% intervals falls noticeably below 95% across many replications, the orthogonality assumption is not met. Alternatively, compute the Gateaux derivative of the mean and quantile estimating equations with respect to the regression adjustment; a nonzero derivative at the truth would invalidate the claimed o_p(N^{-1/2}) property.","supporting_citations":[],"review_version":1}