{"id":"bd034c07-8d55-4599-95ae-03520ffb3037","arxiv_id":"2608.03437","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Rank-weighted adaptive sampling of which model to score next recovers top-model rankings from WMT human evaluation data with less budget than uniform evaluation.","lead":"The paper proposes bandit-style adaptive sampling that spends more human evaluation effort on top-ranked models, and shows in simulations on WMT translation data that it recovers top-model orderings with less budget than uniform evaluation. Smart generalists should read it because evaluation cost is a real bottleneck in machine translation and LLM benchmarking, and this is a concrete proposal for spending annotation budget where it matters.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3's proof optimizes a variance proxy and its Lagrange step is algebraically inconsistent, so √ω optimality for weighted Kendall tau is unproven.","rationale":"The reader and I identify the same load-bearing weakness: Theorem 3, the paper's main theoretical contribution, is not actually proven. The algebra in the Lagrange step is inconsistent with the stated conclusion, and the proof minimizes a variance proxy rather than the weighted Kendall tau objective of Eq. 6. The paper's own Appendix D concedes that a key assumption of the theorem (homoscedasticity) is violated in the real data, and the correlated-arms framing (Lemma 2) contradicts the independence assumption in Eq. 40. These are correctness risks, not stylistic objections: if Theorem 3 is removed, the paper still has an interesting empirical study and a plausible heuristic, but the abstract's claim of proving optimality is unsupported. Because the issues are addressable through a corrected theorem, a proper derivation for the weighted Kendall tau objective, and an explicit empirical check of the √ω rule, the current CONDITIONAL verdict is appropriate rather than REJECT. I also note the introduction's unsupported 95%-at-40%-budget claim, but the theorem is the more load-bearing problem because it underpins the paper's theoretical selling point and its practitioner recommendation to sample according to √ω.","tokens_in":24872,"tokens_out":5893,"duration_ms":55539,"concrete_test":"Settle the proof-level concern by independently re-deriving the Lagrange solution for the two possible surrogates: for L = Σ_m ω_m·σ²/|R_m| the optimal allocation is |R_m| ∝ √ω_m; for L = Σ_m ω_m²·σ²/|R_m| the optimal allocation is |R_m| ∝ ω_m. Then settle the objective-level concern by simulating K = 20 Gaussian arms with known means and common variance, objective weights ω_m = 1/rank², and a fixed budget; compute expected weighted Kendall tau under the √ω, ω, and uniform allocations. If √ω does not maximize expected τ_ω, then Theorem 3 is not establishing optimality for the stated weighted-ranking objective.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that rank-based sampling is provably optimal rests on Theorem 3. The proof does not establish the stated result. It replaces the weighted Kendall tau objective (Eq. 6), whose pairwise terms are weighted by ω_i·ω_j, with the surrogate loss L = Σ_m ω_m²·σ²/|R_m| (Eq. 43) and never shows that minimizing this surrogate minimizes expected τ_ω. Within that surrogate, the Lagrange derivation is inconsistent: stationarity (Eq. 45) gives −ω_m²·σ²/|R_m|² + λ = 0, hence |R_m| ∝ ω_m, while Eq. 48 abruptly uses Σ_k √ω_k in the denominator and the text concludes |R_m| ∝ √ω_m. The conclusion is only true if the 'importance weight' in the loss is ω_m rather than ω_m²; but then the theorem statement's ω_m and the application ω_m = 1/rank² require the former, and the proof's loss should be Σ_m ω_m·σ²/|R_m|. Either way, the proof as written does not prove the theorem for the paper's stated objective. In addition, the independence and homoscedasticity assumptions (Eq. 40) are contradicted by the paper's own correlated-arms model (Lemma 2, Eq. 29) and by Figure 11. The empirical improvement of rank sampling over uniform may survive, but it cannot be explained by Theorem 3 as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a bandit-style framework for allocating human evaluation effort across competing NLP models, with the goal of recovering top-model rankings under a fixed budget. It formalizes evaluation allocation as best-arm identification, proves NP-hardness of the oracle allocation problem, and proposes several sampling policies, including rank-weighted sampling and a confusion-minimization policy. The main theoretical claim is Theorem 3, which states that sampling a model with probability proportional to the square root of its objective weight is optimal for minimizing weighted ranking uncertainty. The empirical section simulates these policies on 290k WMT human evaluations across 37 campaigns, reporting that rank-based sampling and confusion minimization improve weighted Kendall tau over uniform allocation and approach the greedy oracle.","tokens_in":25151,"tokens_out":4334,"duration_ms":40893,"significance":"If the theoretical and empirical claims held, the paper would offer a practical recipe for reducing the cost of large-scale human evaluation: focus annotation effort on competitive models while preserving top-k ranking fidelity. The empirical study is substantial and well-grounded: 100-seed simulations on real WMT data, comparison against a greedy oracle, ablation over item ordering, and synthetic robustness checks. The paper also ships code, which aids reproducibility. However, the central optimality theorem is not established by the proof as written, and the headline 95%/40% claim in the introduction is unsupported by the reported experiments. The practical heuristic value of rank-based sampling may survive, but the paper's core theoretical contribution currently does not.","major_comments":[{"comment":"The Lagrange step in the proof of Theorem 3 is algebraically inconsistent. Stationarity in Eq. (45) gives |R_m|^2 = ω_m²σ²/λ, hence |R_m| = ω_m σ/√λ. Substituting into the budget constraint yields |R_m|* = B·ω_m / Σ_k ω_k, i.e. proportional to ω_m, not √ω_m. The conclusion |R_m| ∝ √ω_m only follows if the loss in Eq. (43) were Σ_m ω_m σ²/|R_m| rather than Σ_m ω_m² σ²/|R_m|. As written, the proof does not prove the stated theorem.","section":"Appendix B, Theorem 3, Eqs. (43)-(49)"},{"comment":"The proof replaces the weighted Kendall tau objective of Eq. (6) with the surrogate loss L = Σ_m ω_m² σ²/|R_m| without establishing that minimizing this surrogate minimizes the expected value of τ_ω. Weighted Kendall tau is a nonlinear, pairwise function of the estimated means, so reducing individual variances need not reduce the expected number of weighted inversions. Without an explicit bound or monotonicity argument connecting L to E[τ_ω], the optimality claim concerns only a variance proxy, not the stated ranking objective.","section":"Section 3.2 and Appendix B, Theorem 3"},{"comment":"The assumptions behind Theorem 3 — independent estimators with common variance σ² across models — are contradicted by the paper's own correlated-arms model (Lemma 2, Eq. (29)) and by the empirical heteroscedasticity reported in Figure 11. The paper concedes this in Section 4.2 when it introduces Confusion Minimization specifically to address the violation. Consequently, Theorem 3 cannot justify the claimed optimality of inverse-rank sampling for the WMT experiments; at best the algorithm is an empirically motivated heuristic under those conditions.","section":"Appendix B, Eq. (40); Appendix D, Fig. 11; Section 4.2"},{"comment":"The introduction claims that the ordering of the top 3 among 20 translation models can be reliably (p=95%) retrieved with only 40% of the budget, but no figure or table reports this quantity. Figure 5 shows average weighted Kendall tau, which is not a retrieval probability, and no top-3 hit-rate result appears elsewhere in the provided text. This headline claim should either be removed or supported with the corresponding experiment, e.g. top-3 retrieval frequency over the 100 seeds.","section":"Section 1 and Section 5"}],"minor_comments":[{"comment":"The word 'concider' should be 'consider'; similar typographical errors appear elsewhere (e.g. 'Bolzmann' in Section 4.1 and Table 2, 'Rearrranging' in Eq. (46), 'Experimnents' in Limitations, 'cummulative' in Section 4.2).","section":"Section 3.4"},{"comment":"The proof of Lemma 1 contains a circular equation: it writes Var(μ̂_repeat) = Var(μ̂_repeat) + Cov(r1,r2), which is not a comparison. The argument should directly show Var(μ̂_repeat) = Var(μ̂_distinct) + positive covariance terms.","section":"Appendix B, Lemma 1"},{"comment":"The caption says numbers in the legend show areas under the curves, but no numerical AUC values are visible in the provided figure. Please clarify whether these are omitted or intended to appear in the legend.","section":"Figure 5"},{"comment":"The rows 'Greedy oracle rank1' and 'Greedy oracle rank top 3' are not defined in the text; please specify which objective each oracle variant optimizes.","section":"Table 2"},{"comment":"The statement that the greedy objective is not submodular and can be arbitrarily worse than optimum is important and should be moved into the main text rather than relegated to a footnote.","section":"Section 4.3, footnote 5"},{"comment":"The text says 'Numbers in legend show areas under curves' but also refers to 'Other algorithms are included in Appendix C.1'; consider adding explicit pointers to Table 2 when discussing Figure 5 and Table 1.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The algebraic error in Theorem 3 is load-bearing: the proof derives |R_m| ∝ ω_m from a loss that squares the weights, while the theorem statement and the application require |R_m| ∝ √ω_m. If the authors cannot repair the theorem or explicitly reframe it as a surrogate-variance result, the optimality claim should be withdrawn and the paper repositioned as an empirical study of allocation heuristics. Given the strength of the simulation study and the practical relevance, I would not reject outright, but the current theoretical contribution cannot stand as written."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical core of this paper deserves to be taken seriously. Running rank-based adaptive allocation on 290k WMT human evaluations is a sensible and reproducible way to ask whether we can spend less annotation budget on inferior models and still recover the top of the ranking. The results in Figures 5-6 and Tables 1-3 are plausible: rank sampling and confusion minimization beat uniform sampling under the weighted Kendall tau objective, and they approach the greedy oracle. That is a useful, concrete finding for benchmark organizers, and the paper ships code and uses public WMT data, which makes it verifiable. The synthetic extensions are also a good check on the limits of the WMT simulations. The problem is the central theorem. Theorem 3 states that sampling according to sqrt(omega_m) minimizes weighted ranking uncertainty. The proof does not establish that. The Lagrange step is internally inconsistent: if the surrogate loss is sum omega_m^2 sigma^2 / |R_m|, stationarity gives |R_m| proportional to omega_m, not sqrt(omega_m). The proof then quietly uses a denominator sum of sqrt(omega_k) and concludes sqrt(omega_m). Getting sqrt(omega_m) requires a surrogate loss linear in omega_m, not quadratic. The proof also never shows that minimizing this variance proxy minimizes the expected weighted Kendall tau from the paper's own Equation 6. So as written, Theorem 3 is not proved. The assumptions behind the theorem are also in tension with the paper's own model. Lemma 2 explicitly says items share difficulty, so model estimators are correlated; and Appendix D Figure 11 shows variance decreases for stronger models, contradicting the common-sigma assumption. The authors acknowledge this and introduce confusion minimization to cope, which is honest and reasonable, but it means the optimality theorem cannot explain why rank sampling works in their own setting. The practical algorithm may still be right, but the proof is not. There are smaller soft spots worth naming. The introduction's claim that top-3 among 20 models can be recovered with 95% probability at 40% budget is not tied to any figure or table. Table 2 shows the gains over uniform largely shrink for unweighted Kendall's tau, which is relevant to practitioners who care about full rankings. The warmup and exploration hyperparameters are numerous, though the main trends look stable. Who should read this? Evaluation researchers, WMT/IWSLT organizers, and anyone allocating human annotation budget for model comparison. It is a useful empirical study with a broken theoretical badge. I would send it to peer review rather than desk reject, with the expectation that the authors fix Theorem 3, connect or remove the 95%/40% claim, and discuss metric dependence. With those repairs it is publishable; without them, the theoretical selling point is unsupported but the empirical contribution likely survives.","headline":"The empirical allocation result is likely real, but Theorem 3's optimality proof has an algebra error, so the paper is a solid empirical contribution with an unproven headline theorem.","tokens_in":746,"tokens_out":1827,"would_cite":true,"duration_ms":38055,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that when the goal is a top-weighted model ranking, the optimal way to spend a fixed evaluation budget is to sample models in proportion to the square root of their rank-based importance, which with inverse-squared-rank…","keywords":["human evaluation","best-arm identification","multi-armed bandit","adaptive evaluation allocation","model ranking","rank-weighted correlation","translation evaluation","inverse-rank sampling"],"falsifier":"Compute the per-model evaluation variance on a benchmark and solve the Lagrangian minimization with those individual variances; if the resulting optimal allocation differs materially from inverse-rank sampling on the same data, then the theorem's recommendation is not the optimum once the equal-variance premise is relaxed.","tokens_in":24558,"feed_emoji":"🎯","tokens_out":10072,"duration_ms":83911,"temperature":0.7,"pith_summary":"Human evaluation of many competing models is expensive because standard protocols annotate every model on every item. The paper reframes the choice of which model to evaluate next as a best-arm identification problem in a multi-armed bandit, where pulling an arm means spending one annotation on that model. Its central theoretical result is that, under a Gaussian noise model, sampling models proportionally to the square root of their rank-weight minimizes weighted ranking uncertainty; with the weights the paper recommends, this reduces to sampling inversely to the model's current estimated rank. The same framework proves consistency of such policies as the budget grows, and shows that the unrestricted optimization is NP-hard, so the paper turns to adaptive heuristics. Simulation on large-scale translation evaluation data indicates that inverse-rank sampling recovers top-model rankings more accurately than uniform sampling and approaches an oracle that sees all scores.","feed_headline":"Sampling by inverse rank provably optimizes top-model ranking","feed_subtitle":"Focusing human annotation on close competitors recovers top-model rankings with a fraction of the usual budget.","key_machinery":"The load-bearing identity is the variance-allocation formula: with sample means of variance $\\sigma^2/|R_m|$, the weighted variance loss $\\sum_m \\omega_m^2 \\sigma^2/|R_m|$ is minimized under a budget by setting $|R_m|$ proportional to $\\omega_m$ (the square root of the importance weight). The paper proves this by Lagrangian differentiation (Theorem 3). The other piece of machinery is the weighted rank correlation $\\tau_\\omega$, a modified rank-correlation score in which disagreements involving high-weight models count more, which lets the paper define the objective as 'get the top of the leaderboard right' and choose $\\omega_m = 1/\\text{rank}^2_m$. The additive noise model $r_{x,m} = q_m + d_x + \\varepsilon_{x,m}$ then supports a linear estimator that removes bias when evaluation items are ordered by difficulty rather than at random.","core_discovery":"The paper's central claim is that evaluation effort should be allocated according to the square root of the importance weight assigned to each model's rank position, and that for weights $\\omega_m = 1/\\text{rank}^2_m$ this means sampling the next model to evaluate proportionally to $1/\\text{rank}_m$. This is established as Theorem 3: assuming the estimated mean score of each model is an independent Gaussian with variance $\\sigma^2/|R_m|$, the Lagrange-optimal allocation that minimizes the weighted sum $\\sum_m \\omega_m^2 \\sigma^2/|R_m|$ under a fixed budget is $|R_m| \\propto \\omega_m$. Since uniform sampling is recovered when all weights are equal, the theorem places the common baseline as the flat-weight case of the same rule. The paper further proves that any policy that never lets a model's selection probability drop below a positive constant will converge to the true ranking (Theorem 2), and that exact optimal allocation remains NP-hard even with oracle access (Theorem 1). Because the equal-variance assumption fails on real translation data, the paper adds a confusion-minimization algorithm that uses per-model variance estimates to maximize pairwise ranking certainty.","pith_inferences":["The inverse-rank rule implies an expected evaluation count that falls off as $1/\\text{rank}$; a direct extension would be a closed-form budget estimator that predicts, before annotation starts, how much budget is needed to retrieve the top-$k$ set at a target probability.","Because the optimality proof assumes equal variance across models while the paper's own data show better models have lower variance, a variance-aware analogue (allocating $|R_m| \\propto \\sigma_m \\omega_m$) would be the natural next theorem and could outperform $1/\\text{rank}$ sampling on real data.","The introductory 95%-at-40%-budget claim is framed as a retrieval probability, while the reported experiments use averaged weighted rank correlation; a dedicated experiment measuring exact top-3 retrieval probability would directly test the practical promise.","The additive quality-plus-difficulty model assumes a single quality per model; in settings where a model's relative quality varies by item type (e.g., language pairs), the same bandit policy could be run per cluster of items to produce specialized rankings."],"forward_implications":["With inverse-rank sampling, annotation campaigns can concentrate human effort on the few models that compete for the top spots, cutting the budget needed to identify the best model.","Uniform sampling, the current baseline, is exactly the optimal policy when all rank positions are weighted equally, so the framework generalizes rather than replaces standard practice.","The consistency theorem guarantees that adaptive allocation eventually recovers the true ranking as the budget grows, so the method is safe to use at any scale.","The same allocation logic transfers to automatic evaluation where each model-output judgment costs compute or API calls rather than annotator time.","Dynamic model selection can be combined with smart item ordering (e.g., difficulty-prioritized items) when score estimates are corrected with an additive difficulty model."],"supporting_citations":[{"why":"Frames fixed-budget best-arm identification, the bandit setting the paper adapts to model ranking.","marker":"Audibert and Bubeck, 2010"},{"why":"Introduces correlated-arms bandits, motivating the paper's treatment of shared evaluation items across models.","marker":"Gupta et al., 2021"},{"why":"Defines the weighted rank-correlation objective $\\tau_\\omega$ that the allocation policies are designed to optimize.","marker":"Vigna, 2015"},{"why":"Supplies the additive model of observed score as model quality plus item difficulty plus noise, used for the linear estimator.","marker":"Cronbach et al., 1972"},{"why":"Provides the large-scale translation evaluation data across multiple campaigns used in the simulation study.","marker":"Kocmi et al., 2025"},{"why":"Describes the human evaluation protocol whose scores serve as ground-truth annotations in the simulations.","marker":"Freitag et al., 2021"},{"why":"Supplies the item-selection perspective on evaluation efficiency that the paper merges with model-selection policies.","marker":"Zouhar et al., 2025"}],"fun_headline_variants":["Adaptive rank-based sampling optimizes model eval budget","Provably optimal allocation for ranking top models","Focus eval on top contenders with inverse-rank sampling","Sampling proportional to inverse rank provably beats uniform"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The optimality result depends on the assumption that every model's estimated mean score is an independent Gaussian with the same variance, so that shifting samples toward lower-ranked models provably buys the most ranking certainty; the paper's own measurements show per-model variances differ, which is why the authors introduce a separate confusion-minimization heuristic.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive rank-based sampling optimizes model eval budget","Provably optimal allocation for ranking top models","Focus eval on top contenders with inverse-rank sampling","Sampling proportional to inverse rank provably beats uniform"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1463,"prompt_tokens":916,"completion_tokens":547,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":487}},"tokens_in":532,"tokens_out":547,"duration_ms":6050,"temperature":1.0,"reasoning_tokens":487,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:51:15.636490+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the per-model evaluation variance on a benchmark and solve the Lagrangian minimization with those individual variances; if the resulting optimal allocation differs materially from inverse-rank sampling on the same data, then the theorem's recommendation is not the optimum once the equal-variance premise is relaxed.","supporting_citations":[],"review_version":2}