{"id":"0bac7a43-63cd-466b-8286-2e296f46a935","arxiv_id":"2601.21816","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A debiased, nonparametric estimator (DMLRank) yields efficient ranking scores and valid confidence intervals for LLM leaderboards from preference data, covering Bradley-Terry, Borda, and rank centrality.","lead":"This paper introduces a debiased machine-learning framework, DMLRank, for ranking LLMs from pairwise preference data with valid confidence intervals. It defines a broad family of ranking scores, handles ties and auto-raters, and derives cost-optimal data-collection policies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (15)'s confidence ellipsoid inverts a covariance that is singular for BT projection and rank centrality, and for Borda under the paper's default binary/ternary weights; the stated chi-square_d region cannot be valid.","rationale":"The reader's weakest-assumption pick, strict positivity, is a real limitation and is honestly acknowledged in Appendix B; it is a standard identifiability condition rather than an internal error. The more load-bearing issue is the singular covariance in Eq. (15): it is exact for BT projection and rank centrality in every experiment, and for Borda under the default binary/ternary weights used in the synthetic and MT-Bench experiments. This does not invalidate Theorem 5.1 or the coordinatewise simultaneous intervals in Appendix H, so the main contribution is salvageable. However, the paper's stated ellipsoidal confidence region is not valid as written and must be replaced or qualified. Since the appropriate verdict is still CONDITIONAL, the reader's verdict does not need to change, though the condition should now explicitly include fixing Eq. (15).","tokens_in":36076,"tokens_out":17845,"duration_ms":209464,"concrete_test":"Take the paper's synthetic C=3 (win/loss/tie) setup with default Borda weights and K=3. Compute the empirical covariance bSigma from Eq. (14) for Borda, BT, and rank centrality. Verify that its smallest eigenvalue is at numerical zero and that 1^T bphi_i = 0 for every i. Then construct the confidence region using a generalized inverse and the chi-square_{K-1} quantile, and compare coverage with the chi-square_K version in Eq. (15): the chi-square_K region will not achieve nominal coverage, while the generalized-inverse version should. Also re-derive 1^T phi=0 analytically from Eqs. (3), (5), (6), and (9) to confirm the singularity is exact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The confidence-region claim in Eq. (15) is internally inconsistent for the paper's own headline GARS. For the BT projection, F(mu) always lies in the zero-sum subspace (by construction through H), so 1^T F(mu)=0. For rank centrality, 1^T F(mu)=1 because F is a stationary distribution. For Borda with the default binary or ternary weights, a direct calculation gives 1^T F(mu)=K/2. In all these cases, differentiating 1^T F(mu)=const gives 1^T J_jk(mu)=0 for every pair (j,k), so every EIF evaluation satisfies 1^T phi(O,eta,theta)=0 exactly, not just asymptotically. Hence Sigma=E[phi phi^T] satisfies Sigma*1=0, so rank(Sigma)<=K-1 and Sigma is singular for K>=2. Therefore Sigma^{-1} in Eq. (15) does not exist. Using a finite-sample bSigma that is numerically invertible does not restore validity: the limiting distribution is degenerate on a hyperplane, and the correct chi-square reference has rank(K-1), not K. Appendix H's coordinatewise intervals avoid this inversion and are a viable fix, but the manuscript presents Eq. (15) as the main confidence region and repeats it in Appendix H.3. This is a mathematical defect in the stated inference procedure, not merely a violation of an external assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DMLRank, a nonparametric framework for estimating and comparing LLM ranking scores from pairwise preference data. The target is defined as a generalized average ranking score (GARS), θ = E[F(μ(X))], where F is a known functional of the contextual preference probabilities; this is shown to cover Bradley–Terry-type projections, Borda/win-rate scores, and Rank Centrality, including ties and black-box judges. The main theoretical contribution is Theorem 5.1, which derives the efficient influence function for GARS and proposes a cross-fitted one-step debiased estimator with claimed asymptotic efficiency and normality. Theorem 6.2 gives an A-optimal labeling policy under cost constraints. Experiments on synthetic data and Chatbot Arena/MT-Bench compare the debiased estimator with plug-in counterparts and random data acquisition.","tokens_in":36480,"tokens_out":10653,"duration_ms":123572,"significance":"If the inference procedure is correct, the paper is a substantial contribution: it unifies several ranking functionals under one semiparametric framework, permits flexible ML nuisance estimation, and derives a principled data-collection policy. The EIF derivation in Theorem 5.1 is a standard one-step/DML argument and is broadly correct; the A-optimal policy derivation under Assumption B.1 is also clean. The code is provided, and the synthetic experiments directly compare debiased and plug-in estimators, which is appropriate for this literature. However, the paper's stated confidence-region result in Eq. (15) is invalid for its own headline GARS examples because the asymptotic covariance is singular; the coordinatewise intervals in Appendix H.2 avoid the problem and appear viable. Because valid confidence intervals are the central claimed output for leaderboards, this defect must be fixed before the paper can be accepted.","major_comments":[{"comment":"The confidence ellipsoid inverts Σ, but for the paper's own headline GARS functionals Σ is singular. For the BT projection, F(µ) lies in the zero-sum subspace, so 1^T F(µ)=0; for Rank Centrality, 1^T F(µ)=1; for Borda with the default binary/ternary weights, 1^T F(µ)=K/2. Differentiating any of these constant-sum identities gives 1^T J_jk(µ)=0 for every pair (j,k), so every EIF evaluation satisfies 1^T φ(O,η,θ)=0 exactly. Hence Σ1=0, Σ is singular, and Σ^{-1} in Eq. (15) does not exist. Using a finite-sample bΣ that is numerically invertible does not restore the chi-square_d calibration; the limiting distribution is degenerate on a hyperplane and the correct reference has rank K−1. The Appendix H.2 coordinatewise Gaussian-max intervals avoid this inversion and are a viable fix, but Eq. (15) is presented as the main confidence region and repeated in H.3. Please replace Eq. (15) with a gen","section":"Section 5.2, Eq. (15); Appendix H.3"},{"comment":"The real-world Chatbot Arena experiment uses n=32,980 comparisons over K=20 models, so many model pairs are likely never observed together. Positivity — π_jk(x)>0 for every pair j≠k and every x — is asserted but not checked. The debiasing term in Eq. (12) divides by πhat_jk(x_i); small estimated propensities can make the correction unstable, and if the true propensity is zero the target is not identifiable from observed data alone. The manuscript notes positivity can be ensured by design, but the real-data experiment is not designed this way. Please report diagnostics for the fitted πhat (minimum, quantiles, proportion of pairs below a threshold), or rerun the analysis on the support where positivity approximately holds, and discuss sensitivity to truncation. This is needed to assess whether the real-world confidence intervals are credible.","section":"Section 7.2, Assumption B.1"}],"minor_comments":[{"comment":"Typo: 'fit models fit models bµ(−v)' should read 'fit models bµ(−v)'.","section":"Section 5.3"},{"comment":"The symbol α is used both for the generic scaling matrix in Eq. (8) and for the positivity threshold in Section 6. These should be distinguished to avoid confusion.","section":"Section 5.1, Eq. (8)"},{"comment":"'Simulteaneous' should be 'Simultaneous' in the appendix headings.","section":"Appendices H and K"},{"comment":"For Rank Centrality, the differentiability and Lipschitz conditions in Assumption A2 are not verified for cases where row sums of T(µ(x)) approach zero or the matrix in Eq. (6) is nearly singular. Please state explicit regularity conditions or describe a regularized version of the map F used in experiments.","section":"Section 4.2, Eq. (6)"},{"comment":"The Gaussian-max simultaneous intervals rely on simulating from N(0,bR). If bR is singular, this is still implementable, but the paper should state that the simulation can use any PSD square root of bR and that the max-absolute quantile remains valid for the marginals of a degenerate Gaussian.","section":"Appendix H.2"}],"recommendation":"major_revision","confidential_remarks":"The singular-covariance issue in Eq. (15) is a genuine mathematical defect in the paper's stated inference procedure, but it is localized: the coordinatewise intervals in Appendix H.2 appear to be a viable fix, and the central DML/EIF derivation is sound. If the authors correct the confidence-region statement and add positivity diagnostics for the real data, the paper would be a solid contribution. The paper's unification of BT, Borda, and Rank Centrality under one semiparametric framework is genuinely useful and sufficiently distinct from prior PPI-based ranking work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the core contribution is real: GARS unifies BT projection, Borda, and rank centrality into one nonparametric estimand θ=E[F(μ(X))], and the one-step debiased estimator with cross-fitting, judge-as-features, and the A-optimal labeling policy are all useful, documented, and backed by released code. The EIF derivations in Appendix D are correct for the three examples, and Theorem 5.1 is standard DML that holds up under the stated assumptions.\n\nSecond, the paper's main confidence-ellipsoid claim, Eq. (15), is mathematically wrong for its own headline examples. For BT projection, rank centrality, and Borda (with the default weights), there is a nonzero vector v such that v^T F(μ) is constant in μ — the all-ones vector works: 1^T F = 0, 1, or K/2. Hence the EIF satisfies v^T φ = 0 identically, Σ = E[φφ^T] is singular (rank ≤ K-1), and Σ^{-1} in Eq. (15) does not exist. The stated chi-square_d ellipsoid is not an asymptotic 1-α confidence region. This is not an external assumption violation; it is an internal inconsistency in the inference procedure. The coordinatewise simultaneous intervals in Appendix H avoid the inversion and are valid, and the experiments use those, so the main empirical claims survive. But the authors need to remove or fix Eq. (15) — e.g., use a generalized inverse with a chi-square on the rank, or explicitly restrict the ellipsoid to the identified subspace. I checked the constants; the stress-test note is correct.\n\nThe other soft spots are minor. The BT coverage in Table 1 is ~0.85-0.90, which is not 'close to 0.95'; the text overstates it. And the positivity assumption π_jk(x)>0 for all pairs is a real practical caveat for leaderboard data where many pairs are never compared; the paper mentions truncation and design-based assurance, but the real-data experiments don't guarantee it. Those are caveats, not fatal.\n\nOverall: this deserves a serious referee. The framework is substantive, the derivations are mostly careful, and the defect in Eq. (15) is fixable. I would accept with major revision conditional on fixing the ellipsoid and the coverage reporting.","headline":"GARS debiased ranking is a genuine, useful contribution, but Eq. (15)'s confidence ellipsoid inverts a singular covariance for the paper's own examples and must be fixed before publication.","tokens_in":36942,"tokens_out":4843,"would_cite":true,"duration_ms":49949,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLM rankings can be defined nonparametrically and estimated by a one-step debiased estimator that is asymptotically efficient, giving valid confidence intervals without any parametric model such as Bradley-Terry.","keywords":["debiased machine learning","efficient influence function","LLM leaderboards","preference data","Bradley-Terry model","rank centrality","Borda scores","optimal data acquisition"],"falsifier":"Two concrete checks would settle the practical scope of the claim. (1) Build a simulation where one model pair has true selection propensity zero, or where selection depends on the unobserved outcome: the inverse-propensity weights S_jk/π̂_jk either diverge or carry selection bias, and nominal 95% intervals should collapse. (2) Fit nuisance functions that converge at exactly the n^{-1/4} boundary in a high-dimensional setting: the product-rate remainder is then O_p(n^{-1/2}) rather than o_p(n^{-1/2}), so the claimed coverage is expected to degrade. A third, policy-level check: spend the same b","tokens_in":35993,"feed_emoji":"🏆","tokens_out":12131,"duration_ms":113554,"temperature":0.7,"pith_summary":"This paper sets out to show that ranking large language models from pairwise preference data can be done without committing to a parametric model, while still getting statistically efficient estimates and valid confidence intervals. The authors define the ranking target directly as a generalized average ranking score — the average of a smooth functional of context-dependent preference probabilities — which covers Bradley-Terry scores, Borda win rates, and rank centrality as special cases, including ties. They derive the efficient influence function for this target and prove that a one-step debiased estimator is asymptotically efficient and normal, so valid intervals survive even when the preference probabilities are learned by black-box machine learning or borrowed from a pre-trained judge. They further show how the same variance formula yields an A-optimal labeling policy that spends a limited annotation budget on the most informative model pairs. If correct, this replaces misspecification-prone parametric leaderboards with a model-agnostic pipeline that also tells practitioners which comparisons to collect.","feed_headline":"One debiased estimator gives LLM rankings valid confidence intervals","feed_subtitle":"A single correction term yields efficient scores and honest error bars for Bradley-Terry, Borda, and rank centrality.","key_machinery":"Generalized average ranking scores (GARS) — the estimand θ = E[F(μ(X))], where F is any differentiable functional of context-dependent preference probabilities μ, covering Bradley-Terry projections, Borda win rates, and rank-centrality stationary distributions. The load-bearing object is the efficient influence function (EIF) for θ: the plug-in score F(μ(X)) − θ plus a debiasing term that weights each labeled pair's residual Y_jk − μ_jk(X) by the inverse selection propensity 1/π_jk(X) and by the Jacobian J_jk(μ(X)), which measures how strongly that pair's preference probability affects the target. Cross-fitted nuisance estimates make the first-order bias vanish, the EIF gives the semiparamet","core_discovery":"Ranking an LLM from preference data need not require assuming a Bradley-Terry or any other parametric model. The paper's central claim is that the ranking target can be defined as θ = E[F(μ(X))], the average of a known differentiable functional F of the context-dependent preference probabilities μ(X), and that for any such F the efficient influence function has one template: F(μ(X)) − θ plus, summed over model pairs, the inverse-propensity-weighted residual S_jk/π_jk(X) · J_jk(μ(X))(Y_jk − μ_jk(X)), with J_jk the Jacobian of F. The one-step estimator built from this influence function is asymptotically efficient and normal, giving simultaneously valid confidence intervals for Borda, Bradley-","pith_inferences":["Editorial extension: the positivity assumption is the practical crux. Public leaderboards routinely contain model pairs that are never compared; if the true propensity for a pair is zero, the inverse-probability weights are undefined and θ is not identifiable from the observed data alone. Deploying the method on existing archives requires assuming the missingness is benign or re-collecting data wi","Editorial extension: the EIF template is not special to LLMs — any covariate-dependent pairwise-comparison task (sports ratings, search-engine side-by-side evaluation, recommender preference panels) fits the same framework, since F is arbitrary and the Jacobian carries all target-specific information. The paper never makes this claim.","Editorial extension: a natural live test would be to collect data under the A-optimal policy and compare realized confidence-interval widths against the predicted efficiency bound; the theory predicts the gap stays tight when μ is estimated well, and the plug-in versus debiased width gap on real data quantifies the cost of ignoring plug-in bias."],"forward_implications":["Leaderboard builders can get simultaneously valid confidence intervals for ranking scores even when preference probabilities are estimated with gradient-boosted trees, neural networks, or other black-box models — no parametric link function required.","One template covers the three ranking targets used in practice — Bradley-Terry projection scores, Borda win rates, and rank centrality — and the weighted-category extension handles ties and multi-way labels like 'both responses are bad'.","External judges (LLM-as-a-judge, auto-raters) can be folded into nuisance estimation as extra features: high-quality judges shrink finite-sample error, while poor judges are ignored, without breaking coverage.","Under BT model misspecification, the debiased projection estimator stays near-unbiased, whereas a restricted parametric estimator degrades — so the framework is a safer default for real preference data.","The A-optimal labeling policy (Theorem 6.2) allocates a fixed annotation budget toward model pairs that are intrinsically noisy and influential for the target score, and empirically beats uniformly random collection across all tested GARS."],"fun_headline_variants":["One debiased estimator ranks LLMs with valid uncertainty","Nonparametric LLM ranking with efficient confidence intervals","LLM leaderboards without restrictive models, valid error bars","Debiased ML gives efficient LLM scores, ties included","Unified influence function for LLM ranking and CIs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is 'positivity plus missing-at-random': every model pair must have a strictly positive probability of being compared in every context, and which pairs get labeled must not depend on the unobserved preference — if a pair is never compared, the inverse-propensity debiasing weights are undefined and the ranking target is not identifiable from the data alone.","fun_headline_variants_meta":{"raw":{"variants":["One debiased estimator ranks LLMs with valid uncertainty","Nonparametric LLM ranking with efficient confidence intervals","LLM leaderboards without restrictive models, valid error bars","Debiased ML gives efficient LLM scores, ties included","Unified influence function for LLM ranking and CIs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000385,"raw_usage":{"total_tokens":1884,"prompt_tokens":766,"completion_tokens":1118,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":1038}},"tokens_in":510,"tokens_out":1118,"duration_ms":10301,"temperature":1.0,"reasoning_tokens":1038,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T06:46:13.852310+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Two concrete checks would settle the practical scope of the claim. (1) Build a simulation where one model pair has true selection propensity zero, or where selection depends on the unobserved outcome: the inverse-propensity weights S_jk/π̂_jk either diverge or carry selection bias, and nominal 95% intervals should collapse. (2) Fit nuisance functions that converge at exactly the n^{-1/4} boundary in a high-dimensional setting: the product-rate remainder is then O_p(n^{-1/2}) rather than o_p(n^{-1/2}), so the claimed coverage is expected to degrade. A third, policy-level check: spend the same b","supporting_citations":[],"review_version":1}