{"id":"3d59bd7c-d947-4592-86ab-8a8e7cdbf34d","arxiv_id":"2506.07949","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper derives budget-optimal rules for mixing cheap weak raters and expensive strong raters so that the mean rating is estimated with minimum variance, and shows large cost savings when example difficulty varies.","lead":"This paper derives rules for deciding when to pay for expensive human or strong-model ratings versus relying on cheap automated ratings, so that an unbiased quality estimate is as accurate as possible under a fixed budget. It matters because AI evaluation is currently expensive, and the new policies can reach the same precision at a fraction of the cost when examples vary in difficulty.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central practical claim that active sampling materially cuts evaluation budget requires an accurate conditional uncertainty u(x); the paper's own real-data results show the plug-in u is far from oracle, so the advantage largely vanishes outside a constructed easy/hard split.","rationale":"I agree with the reader's weakest-assumption: the theoretical derivation is the paper's strength, and the practical payoff is gated by u(x). The paper itself flags this in Section 4.3 and Appendix B.5, so this is not an internal inconsistency; it is an unverified condition for the central empirical claim. The synthetic results in Section 3 show the active advantage grows with Var(U), but real-data figures show only modest active-over-random gains except in the easy/hard split, which is selected using the very uncertainty estimate whose quality is at issue. A concrete calibration-split experiment would settle whether the gap to the oracle is due to u-estimation error or to lack of exploitable heteroskedasticity. Because the theoretical results and the fixed-random-rate policy remain useful even if active's extra gain is fragile, the reader's CONDITIONAL verdict is appropriate; my read does not change it.","tokens_in":32481,"tokens_out":17279,"duration_ms":216762,"concrete_test":"Re-run the AQA and ImageNet burn-in experiments (Section 4.3, Appendix D.2) with u(x) estimated from a held-out calibration set of 500 strong labels (e.g., isotonic calibration of G(1-G) against |H-G|^2, or a simple model of squared error), and compare the active-to-random error ratio at budget B=500 against the reported oracle. If the calibrated active policy does not approach the oracle's mean effective budget, then the bottleneck is not merely estimation noise; if it does, the paper should demonstrate this with a calibration budget that is actually available before evaluation, rather than full-data oracle u. Report both the rank correlation between predicted and true u and the resulting cost savings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition is that u(x)=E[(H-G)^2|X=x] in Proposition 2 can be estimated well enough that the active policy sorts inputs by true difficulty. In Section 4, u is replaced by G(1-G) or model confidence; this equals the target only under perfect calibration of G. The paper's own experiments undercut the practical claim: in Section 4.3, the estimated active policy is close to random on AQA and homogeneous Chatbot Arena, and Appendix D.2 shows the same on ImageNet and Seahorse, while the oracle (true |h-g|^2 on the full dataset) shows large headroom. The text explicitly says 'the estimated u(x) is also far from perfect' and that the oracle 'often promises significant headroom.' The theory in Appendix B.5 (Proposition 8, Corollary 9) only bounds variance inflation when E[1/pi_tilde-1/pi*]<=delta; the experiments indicate that in practice this delta is large, so the bound does not rescue the empirical benefit. The one real-data setting where active clearly beats random is the easy/hard split, which is constructed by selecting on the same G(1-G) ranking whose quality is in question. Thus the theoretical optimality is not the weak point; the empirical claim of 'far lower total annotation budget' for active evaluation is not established with realistically available uncertainty estimates.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a framework for cost-optimal hybrid evaluation, in which an unbiased estimator of the mean of an expensive strong-rater signal H is built from cheap weak-rater observations G plus selectively queried H labels. The annotation policy π(x) controls the probability of querying H as a function of the input. The main theoretical results are Proposition 1, giving the optimal constant sampling rate, and Proposition 2, giving the optimal covariate-dependent rule π_active(x) = min(γ*/√u(x), 1) where u(x) = E[(H−G)^2 | X=x]. The policies minimize mean squared error subject to a cost budget. Synthetic experiments characterize the role of MSE(H,G), Var(U), and the cost ratio; real-data experiments on Chatbot Arena, AQA, ImageNet, and Seahorse compare the estimated random and active policies against a strong-rater-only baseline, with an oracle active policy added to quantify headroom. The paper is candid that the estimated uncertainty u(x) is 'far from perfect' and that the oracle 'often promises significant headroom,' and it positions better uncertainty estimation as future work.","tokens_in":32798,"tokens_out":7212,"duration_ms":91780,"significance":"The theoretical contribution is genuine and useful: it extends active statistical inference to a cost-constrained setting, correctly optimizes the sampling ratio rather than taking it as fixed, and handles the [0,1] constraint that prior work ignored. The closed-form solutions are simple enough to be of practical value, and the synthetic experiments give an informative map of when active sampling helps. The real-data section, however, does not establish the abstract's headline claim that the policies achieve 'far lower total annotation budget' in realistic settings: with the plug-in uncertainty estimator G(1−G), the active policy is close to random on AQA and homogeneous Chatbot Arena, and similar on ImageNet and Seahorse; only the constructed easy/hard split shows a large active benefit. The paper's own oracle results show that the failure is due to inaccurate u(x), not to the theory. The theoretical results are machine-checkable in the sense that the derivations are detailed and mostly rigorous, though one proof step in Proposition 1 appears to have a sign error. Overall, this is a solid theoretical paper whose empirical claims need recalibration.","major_comments":[{"comment":"The abstract and Section 4.3 claim that the proposed policies can achieve the same precision at a 'far lower total annotation budget' than standard evaluation. This is only demonstrated for the estimated random policy and, for the active policy, only on the constructed easy/hard split of Chatbot Arena. On AQA and homogeneous Chatbot Arena (Figure 3), the estimated active policy nearly coincides with the random policy; the same holds for ImageNet and Seahorse in Appendix D.2 (Figure 5). Meanwhile the oracle active policy, which uses the true |h(x)−g(x)|^2, shows large headroom. The paper should either present real-data results with uncertainty estimates that are demonstrably accurate (e.g., a learned regressor for the squared error, or verbalized LLM confidence with calibration checks), or substantially temper the abstract and conclusion so that the practical benefit is claimed only for the fixed-rate policy.","section":"Abstract, Section 4.3, Appendix D.2"},{"comment":"The plug-in u(x)=G(1−G) equals E[(H−G)^2|X=x] only when G(x)=P(H=1|X=x) is perfectly calibrated. The paper applies Platt scaling on the transfer or burn-in set, but reports no calibration diagnostics (reliability diagrams, expected calibration error, or a test of whether E[(H−G)^2|X=x] is actually close to G(1−G) on held-out data). Because accurate u(x) is the load-bearing condition for Proposition 2, the empirical comparison against the oracle is not a valid test of the active policy's practical value unless calibration is verified. Adding calibration analyses and comparing alternative uncertainty estimators would directly address the observed gap between estimated and oracle active policies.","section":"Section 4.2, Proposition 2"},{"comment":"The comparison between the interior critical point and the boundary value p*=1 has a sign error. The proof states that the inequality 0 < (c_h(Var(H)−E)−c_g E)^2 'cannot hold' under the condition (c_g+c_h)E < c_h Var(H), but the squared quantity is strictly positive in that regime so the inequality does hold, implying the interior point is better. This is the opposite of what the proof concludes. The final formula in Proposition 1 is consistent with choosing the interior point under that condition, so the error is likely a typo, but the proof needs to be corrected because it currently contradicts its own conclusion.","section":"Appendix C.1, Proof of Proposition 1"},{"comment":"The variance-inflation bounds in Proposition 8 and Corollary 9 are conditioned on E[1/π_tilde − 1/π*] ≤ δ. The real-data experiments imply this δ is large in practice, because otherwise the estimated active policy would perform close to the oracle. The paper should state this limitation explicitly next to the bounds, rather than presenting them as a robustness guarantee that could be read as covering the experimental regime. Otherwise readers may overestimate the safety of plug-in uncertainty estimates.","section":"Appendix B.5, Propositions 8 and 9"}],"minor_comments":[{"comment":"In the derivation of Var(Δπ), the expression 'Δπ = G + (H−G) 2 ξ/π(X)' appears to contain a typo: the '(H−G)' term should not be squared, since Eq. (1) defines Δt = G_t + (H_t−G_t) ξ_t/π_t(X_t). Please correct the exponent.","section":"Appendix B.1"},{"comment":"The text says the Beta distribution is 'satified' by U∼Beta(κμ, κ(1−μ)) and similarly 'satified' in the Gaussian construction; these should be 'satisfied'. There are a few similar typographical errors (e.g., 'doe not depend' in Section 3.1).","section":"Sections 3.2 and 3.3"},{"comment":"The convention for ErrorRatio(π1,π2) treats π_base specially by discarding c_g in the denominator. This is explained, but the definition as written could mislead readers who compare it with the general formula; a brief note that the baseline is the estimator with π≡1 and cost c_h per sample would help.","section":"Section 3.1, ErrorRatio definition"},{"comment":"In the text, the easy/hard split is described as constructed from the bottom 25% and top 25% of Gemma's uncertainty estimates, but it is not immediately clear whether the 'top' and 'bottom' are with respect to u(x)=G(1−G) or the underlying model softmax. Clarifying this would aid reproducibility.","section":"Figure 2 caption and Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a solid theoretical contribution and is unusually honest about its empirical limitations, but the abstract and conclusion overstate the practical benefit of the active policy. The proof error in Proposition 1 is easily corrected but needs attention. The main question for the editor is whether the journal is willing to accept a paper whose real-data validation shows the active policy only beating random on a constructed split; I think the theory stands, but the empirical claims must be recalibrated or supplemented with better uncertainty estimates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real contribution, not a paradigm shift. The theoretical core is the cost-ratio optimization of n/N, which prior active inference left fixed. Propositions 1 and 2 are correct as far as I can tell, and the clipping fix for the [0,1] constraint is a genuine improvement over Zrnic-Candes. The extension to M-estimators in Appendix B is a nice bonus. The paper is also honest about its biggest weakness: the optimal active policy requires u(x) = E[(H-G)^2|X=x], and the experiments show that plug-ins like G(1-G) don't capture the oracle headroom. The authors say this openly in Section 4.3 and the conclusion.\n\nWhat's actually new: optimizing the budget split (n/N) under cost constraints, the clipped active policy, and the empirical characterization across four plus two datasets. The synthetic experiments are well designed and cleanly show when active beats random: high Var(U) and low c_g/c_h. The real-data results are more muted. Active gives a clear win on the constructed easy/hard split, but on homogeneous AQA and Arena, active is close to random; the oracle shows large headroom. That is exactly the stress-test concern, and the paper's own text confirms it. The random policy, however, delivers consistent gains over baseline in all the real-data settings. So the fixed-rate contribution is robust; the active policy's practical advantage remains conditional on having a good uncertainty model.\n\nSoft spots: no code released, which makes it hard to verify the more intricate proofs and the exact experimental pipeline. The proofs in the appendix are sketched but plausible; the reader's note about a typo in Proposition 1's proof (where the boundary comparison is handled) is minor and doesn't affect the statement. The finite-sample discrete-time result (Proposition 4) is not implemented, so the main results rely on the relaxed integer constraint, which is fine but worth flagging. The paper also samples with replacement from finite datasets, which sidesteps the fixed-population budget issue; they acknowledge this.\n\nCitation pattern is legitimate: they build on their own PPI work and Zrnic-Candes, and the novelty is clear. No invented entities. Overall, a solid methods paper with honest limitations. It deserves a serious referee; the main revision ask should be code release and more discussion of when the active policy's plug-in u can be trusted.","headline":"Solid theory for cost-optimal evaluation budgets; the active policy's practical edge depends on uncertainty estimates the paper itself shows are imperfect.","tokens_in":33302,"tokens_out":1671,"would_cite":true,"duration_ms":20178,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62K05","62F12"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper derives a cost-optimal active sampling rule for AI evaluation: query the expensive rater with probability proportional to the square root of the cheap rater's conditional error, capped at 1.","keywords":["active evaluation","cost-optimal sampling","prediction-powered inference","weak raters","model evaluation","unbiased estimation","heteroskedastic errors","annotation budget"],"falsifier":"Take a real evaluation task with full strong labels, compute the true $u(x)=\\mathbb{E}[(H-G)^2\\mid X=x]$ on a held-out set, and compare it with the proxy $G(1-G)$ used by the paper; if the rank correlation between the two is near zero, the estimated active policy should have nearly the same squared error per budget as the fixed-rate policy, contradicting the paper's empirical claim that active sampling delivers large savings whenever example difficulty is heterogeneous.","tokens_in":32325,"feed_emoji":"🎯","tokens_out":10665,"duration_ms":116355,"temperature":0.7,"pith_summary":"This paper asks how to estimate the mean quality score of an AI system when two rating sources are available: a cheap, often inaccurate automated rater and an expensive, accurate rater, with a fixed total annotation budget. It claims that the optimal allocation is not a fixed ratio of expensive to cheap ratings but an input-dependent rule that calls the expensive rater with probability proportional to the square root of the cheap rater's conditional error, capped at 1. Under i.i.d. sampling, the paper proves that this rule minimizes the mean squared error of an unbiased estimator of the strong-rater mean, and that the optimal fixed-rate policy is the special case where the cheap rater's error does not vary with the input. On real LLM-evaluation tasks, the policies reach the same estimation precision as always using the expensive rater at 40--50% of the budget in the best settings, with the largest gains when example difficulty is highly variable.","feed_headline":"Active sampling cuts AI evaluation budgets by up to half","feed_subtitle":"Optimal policies spend expensive labels where cheap model raters are unreliable, preserving accuracy at half budget.","key_machinery":"The machinery is the inverse-propensity estimator $\\hat{\\theta}_T = \\frac{1}{T}\\sum_{t=1}^T \\left[G_t + (H_t-G_t)\\xi_t / \\pi_t(X_t)\\right]$, where $\\xi_t$ indicates whether the strong label was queried. Its variance decomposes as $\\mathrm{Error}_T(\\pi)=\\frac{1}{T}[\\mathrm{Var}(H)-\\mathbb{E}[(H-G)^2]+\\mathbb{E}[(H-G)^2/\\pi(X)]]$, and its expected cost is $T(c_h\\mathbb{E}[\\pi(X)]+c_g)$. Minimizing error for a fixed budget turns the problem into a convex optimization over the sampling rule; the square-root allocation is the resulting stationary condition, and the cap at 1 handles the constraint $\\pi(x)\\le 1$. The threshold $\\tau^*$ is found by a one-dimensional search because the clipped objective is not convex in the threshold.","core_discovery":"The central claim is that the annotation policy minimizing estimation error under a cost budget has an explicit square-root form. With $H$ the strong rating, $G$ the weak rating, costs $c_h > c_g$, and conditional error $u(x)=\\mathbb{E}[(H-G)^2\\mid X=x]$, the optimal active rule is $\\pi_{\\mathrm{active}}(x)=\\min(\\gamma^* / \\sqrt{u(x)}, 1)$, where $\\gamma^*$ is set by the cost ratio and the distribution of $u(X)$, and inputs with $\\sqrt{u(x)}$ above a threshold are always sent to the strong rater. The companion result for policies that cannot depend on $x$ gives the optimal fixed sampling rate $\\pi_{\\mathrm{random}}(x)=\\sqrt{(c_g/c_h)\\,\\mathbb{E}[(H-G)^2]/(\\mathrm{Var}(H)-\\mathbb{E}[(H-G)^2])}$, with a boundary at 1 when the weak rater is too inaccurate to help. The paper shows the active rule reduces to the fixed rule when $u(x)$ is constant, and it characterizes empirically that the benefit of active over random sampling grows with the variance of $u(X)$ and shrinks as the weak rater's average error or relative cost grows.","pith_inferences":["If conditional error $u(x)$ can be learned online from a small burn-in set, the active rule suggests a natural adaptive protocol for continually adding new models to an existing benchmark; the paper's main results treat fixed policies.","The same square-root allocation should apply whenever a cheap surrogate signal is combined with a gold label, such as A/B testing with surrogate metrics, so the result is not specific to AI evaluation.","The observed gap between the estimated and oracle policies points to uncertainty calibration, not just average accuracy, as the property of autoraters that determines active-sampling gains.","A practical diagnostic suggested by the theory is to estimate $\\mathrm{Var}(u(X))$ on a labeled subsample; if it is near zero, use the fixed-rate policy and spend no engineering effort on active sampling."],"forward_implications":["An evaluation pipeline that learns or transfers $u(x)$ can meet a target precision with roughly half the budget of a strong-rater-only evaluation, and more when the required precision is high.","The optimal weak-rater query rate is determined by the cost ratio and the error distribution, so budgets can be planned without trial-and-error.","Active policies are most valuable when example difficulty is heterogeneous; on homogeneous tasks, the fixed-rate policy captures most of the gain.","If the weak rater's average error exceeds the variance of the target rating by a cost-dependent margin, the optimal policy is to ignore the weak rater and query the strong rater always.","The framework extends beyond means to convex M-estimators and to choosing which inputs to sample, so the same allocation logic applies to estimating other evaluation targets."],"supporting_citations":[{"why":"Supplies the sequential active-inference estimator in Eq. (1) and the fixed-ratio problem that the paper's cost optimization generalizes.","marker":"[32]"},{"why":"Establishes prediction-powered inference, the unbiased combination of trusted and predicted labels on which the estimator is built.","marker":"[1]"},{"why":"Provides power tuning, applied empirically after sampling to reduce estimator variance.","marker":"[2]"},{"why":"Provides the AQA retrieval-QA dataset with human attribution labels used as a second real-world test bed.","marker":"[3]"},{"why":"Provides the Chatbot Arena comparison data for the main real-world win-rate estimation experiments.","marker":"[8]"},{"why":"Prior active-sampling method that fixes the number of expensive labels in advance; the paper's cost-optimal policies remove that restriction.","marker":"[16]"}],"fun_headline_variants":["Active policies halve cost of AI model evaluation","Optimal rater mix cuts evaluation costs sharply","Cost-optimal active sampling for accurate AI eval","AI evaluation cheaper with smart weak-strong rater mix","Square-root rule slashes AI annotation budgets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the evaluator has access to the conditional mean squared error of the weak rater, $u(x)=\\mathbb{E}[(H-G)^2\\mid X=x]$, for each input; in practice this quantity is replaced by proxies such as $G(1-G)$ or model confidence, and the paper's own oracle-headroom experiments show that when those proxies are poor the active policy's budget advantage over random sampling largely disappears.","fun_headline_variants_meta":{"raw":{"variants":["Active policies halve cost of AI model evaluation","Optimal rater mix cuts evaluation costs sharply","Cost-optimal active sampling for accurate AI eval","AI evaluation cheaper with smart weak-strong rater mix","Square-root rule slashes AI annotation budgets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000418,"raw_usage":{"total_tokens":2191,"prompt_tokens":1023,"completion_tokens":1168,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":1096}},"tokens_in":639,"tokens_out":1168,"duration_ms":9493,"temperature":1.0,"reasoning_tokens":1096,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:21:44.432225+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a real evaluation task with full strong labels, compute the true $u(x)=\\mathbb{E}[(H-G)^2\\mid X=x]$ on a held-out set, and compare it with the proxy $G(1-G)$ used by the paper; if the rank correlation between the two is near zero, the estimated active policy should have nearly the same squared error per budget as the fixed-rate policy, contradicting the paper's empirical claim that active sampling delivers large savings whenever example difficulty is heterogeneous.","supporting_citations":[{"cited_title":"Can Unconfident LLM Annotations Be Used for Confident Conclusions?","cited_arxiv_id":"2408.15204","evidence_quote":"Prior active-sampling method that fixes the number of expensive labels in advance; the paper's cost-optimal policies remove that restriction."}],"review_version":1}