Nonparametric LLM Evaluation from Preference Data

Athiya Deviyani; Dennis Frauen; Mihaela van der Schaar; Stefan Feuerriegel

REVIEW 2 major objections 5 minor 1 cited by

This paper claims that LLM rankings can be defined nonparametrically and estimated by a one-step debiased estimator that is asymptotically efficient, giving valid confidence intervals without any parametric model such as Bradley-Terry.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:46 UTC pith:DNA765IE

load-bearing objection GARS debiased ranking is a genuine, useful contribution, but Eq. (15)'s confidence ellipsoid inverts a singular covariance for the paper's own examples and must be fixed before publication. the 2 major comments →

arxiv 2601.21816 v2 pith:DNA765IE submitted 2026-01-29 cs.LG

Nonparametric LLM Evaluation from Preference Data

Dennis Frauen , Athiya Deviyani , Mihaela van der Schaar , Stefan Feuerriegel This is my paper

classification cs.LG

keywords debiased machine learningefficient influence functionLLM leaderboardspreference dataBradley-Terry modelrank centralityBorda scoresoptimal data acquisition

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper sets out to show that ranking large language models from pairwise preference data can be done without committing to a parametric model, while still getting statistically efficient estimates and valid confidence intervals. The authors define the ranking target directly as a generalized average ranking score — the average of a smooth functional of context-dependent preference probabilities — which covers Bradley-Terry scores, Borda win rates, and rank centrality as special cases, including ties. They derive the efficient influence function for this target and prove that a one-step debiased estimator is asymptotically efficient and normal, so valid intervals survive even when the preference probabilities are learned by black-box machine learning or borrowed from a pre-trained judge. They further show how the same variance formula yields an A-optimal labeling policy that spends a limited annotation budget on the most informative model pairs. If correct, this replaces misspecification-prone parametric leaderboards with a model-agnostic pipeline that also tells practitioners which comparisons to collect.

Core claim

Ranking an LLM from preference data need not require assuming a Bradley-Terry or any other parametric model. The paper's central claim is that the ranking target can be defined as θ = E[F(μ(X))], the average of a known differentiable functional F of the context-dependent preference probabilities μ(X), and that for any such F the efficient influence function has one template: F(μ(X)) − θ plus, summed over model pairs, the inverse-propensity-weighted residual S_jk/π_jk(X) · J_jk(μ(X))(Y_jk − μ_jk(X)), with J_jk the Jacobian of F. The one-step estimator built from this influence function is asymptotically efficient and normal, giving simultaneously valid confidence intervals for Borda, Bradley-

What carries the argument

Generalized average ranking scores (GARS) — the estimand θ = E[F(μ(X))], where F is any differentiable functional of context-dependent preference probabilities μ, covering Bradley-Terry projections, Borda win rates, and rank-centrality stationary distributions. The load-bearing object is the efficient influence function (EIF) for θ: the plug-in score F(μ(X)) − θ plus a debiasing term that weights each labeled pair's residual Y_jk − μ_jk(X) by the inverse selection propensity 1/π_jk(X) and by the Jacobian J_jk(μ(X)), which measures how strongly that pair's preference probability affects the target. Cross-fitted nuisance estimates make the first-order bias vanish, the EIF gives the semiparamet

Load-bearing premise

The load-bearing premise is 'positivity plus missing-at-random': every model pair must have a strictly positive probability of being compared in every context, and which pairs get labeled must not depend on the unobserved preference — if a pair is never compared, the inverse-propensity debiasing weights are undefined and the ranking target is not identifiable from the data alone.

What would settle it

Two concrete checks would settle the practical scope of the claim. (1) Build a simulation where one model pair has true selection propensity zero, or where selection depends on the unobserved outcome: the inverse-propensity weights S_jk/π̂_jk either diverge or carry selection bias, and nominal 95% intervals should collapse. (2) Fit nuisance functions that converge at exactly the n^{-1/4} boundary in a high-dimensional setting: the product-rate remainder is then O_p(n^{-1/2}) rather than o_p(n^{-1/2}), so the claimed coverage is expected to degrade. A third, policy-level check: spend the same b

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Leaderboard builders can get simultaneously valid confidence intervals for ranking scores even when preference probabilities are estimated with gradient-boosted trees, neural networks, or other black-box models — no parametric link function required.
One template covers the three ranking targets used in practice — Bradley-Terry projection scores, Borda win rates, and rank centrality — and the weighted-category extension handles ties and multi-way labels like 'both responses are bad'.
External judges (LLM-as-a-judge, auto-raters) can be folded into nuisance estimation as extra features: high-quality judges shrink finite-sample error, while poor judges are ignored, without breaking coverage.
Under BT model misspecification, the debiased projection estimator stays near-unbiased, whereas a restricted parametric estimator degrades — so the framework is a safer default for real preference data.
The A-optimal labeling policy (Theorem 6.2) allocates a fixed annotation budget toward model pairs that are intrinsically noisy and influential for the target score, and empirically beats uniformly random collection across all tested GARS.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Editorial extension: the positivity assumption is the practical crux. Public leaderboards routinely contain model pairs that are never compared; if the true propensity for a pair is zero, the inverse-probability weights are undefined and θ is not identifiable from the observed data alone. Deploying the method on existing archives requires assuming the missingness is benign or re-collecting data wi
Editorial extension: the EIF template is not special to LLMs — any covariate-dependent pairwise-comparison task (sports ratings, search-engine side-by-side evaluation, recommender preference panels) fits the same framework, since F is arbitrary and the Jacobian carries all target-specific information. The paper never makes this claim.
Editorial extension: a natural live test would be to collect data under the A-optimal policy and compare realized confidence-interval widths against the predicted efficiency bound; the theory predicts the gap stays tight when μ is estimated well, and the plug-in versus debiased width gap on real data quantifies the cost of ignoring plug-in bias.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

GARS debiased ranking is a genuine, useful contribution, but Eq. (15)'s confidence ellipsoid inverts a singular covariance for the paper's own examples and must be fixed before publication.

read the letter

Two things to know. First, the core contribution is real: GARS unifies BT projection, Borda, and rank centrality into one nonparametric estimand θ=E[F(μ(X))], and the one-step debiased estimator with cross-fitting, judge-as-features, and the A-optimal labeling policy are all useful, documented, and backed by released code. The EIF derivations in Appendix D are correct for the three examples, and Theorem 5.1 is standard DML that holds up under the stated assumptions.

Second, the paper's main confidence-ellipsoid claim, Eq. (15), is mathematically wrong for its own headline examples. For BT projection, rank centrality, and Borda (with the default weights), there is a nonzero vector v such that v^T F(μ) is constant in μ — the all-ones vector works: 1^T F = 0, 1, or K/2. Hence the EIF satisfies v^T φ = 0 identically, Σ = E[φφ^T] is singular (rank ≤ K-1), and Σ^{-1} in Eq. (15) does not exist. The stated chi-square_d ellipsoid is not an asymptotic 1-α confidence region. This is not an external assumption violation; it is an internal inconsistency in the inference procedure. The coordinatewise simultaneous intervals in Appendix H avoid the inversion and are valid, and the experiments use those, so the main empirical claims survive. But the authors need to remove or fix Eq. (15) — e.g., use a generalized inverse with a chi-square on the rank, or explicitly restrict the ellipsoid to the identified subspace. I checked the constants; the stress-test note is correct.

The other soft spots are minor. The BT coverage in Table 1 is ~0.85-0.90, which is not 'close to 0.95'; the text overstates it. And the positivity assumption π_jk(x)>0 for all pairs is a real practical caveat for leaderboard data where many pairs are never compared; the paper mentions truncation and design-based assurance, but the real-data experiments don't guarantee it. Those are caveats, not fatal.

Overall: this deserves a serious referee. The framework is substantive, the derivations are mostly careful, and the defect in Eq. (15) is fixable. I would accept with major revision conditional on fixing the ellipsoid and the coverage reporting.

Referee Report

2 major / 5 minor

Summary. The paper proposes DMLRank, a nonparametric framework for estimating and comparing LLM ranking scores from pairwise preference data. The target is defined as a generalized average ranking score (GARS), θ = E[F(μ(X))], where F is a known functional of the contextual preference probabilities; this is shown to cover Bradley–Terry-type projections, Borda/win-rate scores, and Rank Centrality, including ties and black-box judges. The main theoretical contribution is Theorem 5.1, which derives the efficient influence function for GARS and proposes a cross-fitted one-step debiased estimator with claimed asymptotic efficiency and normality. Theorem 6.2 gives an A-optimal labeling policy under cost constraints. Experiments on synthetic data and Chatbot Arena/MT-Bench compare the debiased estimator with plug-in counterparts and random data acquisition.

Significance. If the inference procedure is correct, the paper is a substantial contribution: it unifies several ranking functionals under one semiparametric framework, permits flexible ML nuisance estimation, and derives a principled data-collection policy. The EIF derivation in Theorem 5.1 is a standard one-step/DML argument and is broadly correct; the A-optimal policy derivation under Assumption B.1 is also clean. The code is provided, and the synthetic experiments directly compare debiased and plug-in estimators, which is appropriate for this literature. However, the paper's stated confidence-region result in Eq. (15) is invalid for its own headline GARS examples because the asymptotic covariance is singular; the coordinatewise intervals in Appendix H.2 avoid the problem and appear viable. Because valid confidence intervals are the central claimed output for leaderboards, this defect must be fixed before the paper can be accepted.

major comments (2)

[Section 5.2, Eq. (15); Appendix H.3] The confidence ellipsoid inverts Σ, but for the paper's own headline GARS functionals Σ is singular. For the BT projection, F(µ) lies in the zero-sum subspace, so 1^T F(µ)=0; for Rank Centrality, 1^T F(µ)=1; for Borda with the default binary/ternary weights, 1^T F(µ)=K/2. Differentiating any of these constant-sum identities gives 1^T J_jk(µ)=0 for every pair (j,k), so every EIF evaluation satisfies 1^T φ(O,η,θ)=0 exactly. Hence Σ1=0, Σ is singular, and Σ^{-1} in Eq. (15) does not exist. Using a finite-sample bΣ that is numerically invertible does not restore the chi-square_d calibration; the limiting distribution is degenerate on a hyperplane and the correct reference has rank K−1. The Appendix H.2 coordinatewise Gaussian-max intervals avoid this inversion and are a viable fix, but Eq. (15) is presented as the main confidence region and repeated in H.3. Please replace Eq. (15) with a gen
[Section 7.2, Assumption B.1] The real-world Chatbot Arena experiment uses n=32,980 comparisons over K=20 models, so many model pairs are likely never observed together. Positivity — π_jk(x)>0 for every pair j≠k and every x — is asserted but not checked. The debiasing term in Eq. (12) divides by πhat_jk(x_i); small estimated propensities can make the correction unstable, and if the true propensity is zero the target is not identifiable from observed data alone. The manuscript notes positivity can be ensured by design, but the real-data experiment is not designed this way. Please report diagnostics for the fitted πhat (minimum, quantiles, proportion of pairs below a threshold), or rerun the analysis on the support where positivity approximately holds, and discuss sensitivity to truncation. This is needed to assess whether the real-world confidence intervals are credible.

minor comments (5)

[Section 5.3] Typo: 'fit models fit models bµ(−v)' should read 'fit models bµ(−v)'.
[Section 5.1, Eq. (8)] The symbol α is used both for the generic scaling matrix in Eq. (8) and for the positivity threshold in Section 6. These should be distinguished to avoid confusion.
[Appendices H and K] 'Simulteaneous' should be 'Simultaneous' in the appendix headings.
[Section 4.2, Eq. (6)] For Rank Centrality, the differentiability and Lipschitz conditions in Assumption A2 are not verified for cases where row sums of T(µ(x)) approach zero or the matrix in Eq. (6) is nearly singular. Please state explicit regularity conditions or describe a regularized version of the map F used in experiments.
[Appendix H.2] The Gaussian-max simultaneous intervals rely on simulating from N(0,bR). If bR is singular, this is still implementable, but the paper should state that the simulation can use any PSD square root of bR and that the max-absolute quantile remains valid for the marginals of a degenerate Gaussian.

Circularity Check

0 steps flagged

No significant circularity: the estimand, EIF, debiased estimator, and acquisition policy are derived from first principles and are not self-referential.

full rationale

The paper's central derivation is self-contained and does not reduce to its inputs. GARS is defined explicitly as θ=E[F(μ(X))] for a user-specified functional F, and the examples (BT projection, Borda, Rank Centrality) are presented as special cases with derivations that follow from their definitions; the statement that the BT projection recovers the BT scores when the BT model holds is a mathematical identity (Eq. (3) equals r(x) under Eq. (1)), not a fitted prediction. The efficient influence function in Theorem 5.1 is derived from first principles via pathwise differentiation and the standard MAR/positivity factorization, and the one-step estimator is exactly the EIF plug-in; no parameter is fitted to a subset of data and then used to 'predict' the same quantity. The A-optimal labeling policy minimizes the semiparametric variance bound Σ(π) by construction of A-optimality, so its optimality is definitional rather than circular, and the closed-form rule in Theorem 6.2 follows from a separable convex optimization problem. There are no load-bearing self-citations and no imported uniqueness theorems from the authors' prior work. The manuscript honestly notes that positivity and MAR are needed for identifiability, and the singular-covariance concern about Eq. (15) raised by a skeptical reading is a mathematical validity issue, not an instance of circular reasoning. Overall, the derivation chain is independent and non-circular.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central inferential claims rest on standard semiparametric assumptions (positivity, MAR, n^{-1/4} nuisance rates). The optimal-acquisition claim additionally assumes independent selection. No ad hoc entities are introduced. User-chosen weights/costs/budget define the estimand and design but are not fitted.

free parameters (3)

GARS category weights w^(1), w^(2) = user-specified (e.g., tie=1/2, both-bad=0)
Define which categories count toward each item's score; changing them changes the estimand. Not fitted to data but chosen by hand.
Positivity threshold α = user-selected > 0
In optimal acquisition, lower bound on selection probabilities; chosen by hand to ensure feasibility and positivity. Affects variance but not the EIF itself.
Budget β and pair costs c_jk = user-specified
Constraints in optimal data acquisition; decide how many labels can be afforded and pair-specific labeling costs. Not fitted to data.

axioms (5)

domain assumption Positivity: π_jk(x)>0 for all pairs j≠k and x in the support of X
Required for inverse-propensity terms and identifiability; stated in Appendix B.1. Real data may violate this when pairs are never compared.
domain assumption Missing at random: Y⊥⊥S | X
Allows µ_jkc(x)=P(Y_jkc=1|X=x,S_jk=1) and the EIF derivation; Appendix B.1.
domain assumption DML rate conditions: cross-fitted nuisance estimators achieve o_p(n^{-1/4}) L2 rates and product rate o_p(n^{-1/2})
Needed for asymptotic normality and efficiency of the one-step estimator; assumptions A1-A4 in Appendix B.1. Not verified empirically.
standard math F is Fréchet differentiable with locally Lipschitz Jacobian
Ensures Taylor expansion and remainder control in the proof of Theorem 5.1; Appendix B.1 (A2). Holds for Borda/BT on compact support after truncation, but rank centrality requires matrix invertibility.
domain assumption Assumption B.1: independent Bernoulli selection across pairs for optimal design
Used to make the trace objective separable in Theorem 6.2; only needed for optimality of the acquisition policy, not for EIF validity.

pith-pipeline@v1.3.0-alltime-deepseek · 35776 in / 24042 out tokens · 242565 ms · 2026-08-03T06:46:13.852310+00:00 · methodology

0 comments

read the original abstract

Evaluating the performance of large language models (LLMs) from human preference data is crucial for obtaining LLM leaderboards. However, many existing approaches either rely on restrictive parametric assumptions or lack valid uncertainty quantification when flexible machine learning methods are used. In this paper, we propose a nonparametric statistical framework, called DMLRank, for comparing and ranking LLMs from preference data using debiased machine learning (DML). For this, we introduce generalized average ranking scores (GARS), which generalize commonly used ranking models, including the Bradley-Terry model or PageRank/ Rank centrality, with complex human responses such as ties. DMLRank comes with the following advantages: (i)~It produces statistically efficient estimates of GARS ranking scores. (ii) It naturally allows the incorporation of black-box machine learning methods for estimation. (iii) It can be combined with pre-trained LLM evaluators (e.g., using LLM-as-a-judge). (iv) It suggests optimal policies for collecting preference data under budget constraints. We demonstrate these advantages both theoretically and empirically using both synthetic and real-world preference datasets. In summary, our framework provides practitioners with powerful, state-of-the-art methods for comparing or ranking LLMs for leaderboards.

Figures

Figures reproduced from arXiv: 2601.21816 by Athiya Deviyani, Dennis Frauen, Mihaela van der Schaar, Stefan Feuerriegel.

**Figure 1.** Figure 1: Overview of preference-based LLM evaluation. In this paper, we propose a nonparametric statistical framework called DMLRANK to compare and rank LLMs from preference data while addressing the above limitations. Our key idea is to define the ranking target directly as a functional of the contextual preference probabilities, leading to generalized average ranking scores (GARS). GARS naturally accommodate d… view at source ↗

**Figure 2.** Figure 2: Overview of statistical inference for GARS. Stage 1 : Nuisance estimation with cross-fitting. We need estimates µˆ of the preference probabilities µ and, if unknown, estimates πˆ selection propensities π. In some scenarios (e.g., when designing preference data as in Sec. 6), π may be known in which case we can set πˆ = π. As it is standard in semiparametric inference, we use cross-fitting to obtain µˆ an… view at source ↗

**Figure 3.** Figure 3: Results for Chatbot Arena preference data. Shown: estimated ranking scores for different GARS functionals (Borda, BT, and rank centrality) and estimators (debiased and plugin). Changes in ranking are indicated in red and green (for debiased estimators as compared to debiased Borda scores, for plugin estimators as compared to the corresponding debiased estimator). We report 95% simultaneous confidence inter… view at source ↗

**Figure 4.** Figure 4: Synthetic experimental results. (a) Relative ranking error normalized by the no-judge baseline (mean and 95% CIs over n = 100 runs). (b) BT-projection estimation error (mean and 95% CIs over 30 runs). performs random collection across all GARS types, thus confirming our result from Theorem 6.2 [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

**Figure 5.** Figure 5: Results for MT-Bench preference data. Shown: estimated ranking scores for different GARS functionals (Borda, BT, and rank centrality) and estimators (debiased and plugin). Changes in ranking are indicated in red and green (for debiased estimators as compared to debiased Borda scores, for plugin estimators as compared to the corresponding debiased estimator). We report 95% simultaneous confidence intervals,… view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

ORTHOBO: Orthogonal Bayesian Hyperparameter Optimization
cs.LG 2026-05 unverdicted novelty 5.0

OrthoBO introduces an orthogonal acquisition estimator subtracting an optimally weighted score-function control variate to reduce Monte Carlo variance, preserve the acquisition target, and improve ranking stability in...

Reference graph

Works this paper leans on

4 extracted references · 1 linked inside Pith · cited by 1 Pith paper

[1]

While the estimated average cost deviates fromβby more thanε out: (a) Setλ←(λ lo +λ hi)/2

Outer loop (budget via bisection in λ≥0 ).Initialize 0 =λ lo < λhi large enough that the budget is satisfied. While the estimated average cost deviates fromβby more thanε out: (a) Setλ←(λ lo +λ hi)/2. (b)Per-context water-filling.For eachx i: 28 Nonparametric LLM Evaluation from Preference Data i. Define the decreasing function gi(ν) = X j<k max n α, s Wj...

2007
[2]

: (a) ComputeΣ(π (t))(e.g., via the population formula if available, or via an empirical estimate using samples{x i}n i=1)

Fort= 0,1,2, . . .: (a) ComputeΣ(π (t))(e.g., via the population formula if available, or via an empirical estimate using samples{x i}n i=1). (b) Update ˜π(t+1) jk (x) = clip[α,1] vuut tr Σ(π(t))−1 Mjk (x) λ cjk withλchosen by one-dimensional bisection so that the budget constraint holds. (c) Setπ (t+1) ←˜π(t+1) and stop when the iterates stabilize. 30 No...

2023
[3]

Initializeπ (0) (e.g., the A-optimal policy from Theorem 6.2)
[2007]

j wins” vs. “k wins

ISBN 978-0-19-929659-0. Bickel, P. J., Klaassen, C. A. J., Ritov, Y ., and Wellner, JohnA.Efficient and Adaptive Estimation for Semipara- metric Models. Springer New York, 1 edition, 1998. ISBN 978-0-387-98473-5. Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4):324–345, 195...

Pith/arXiv arXiv 1998

[1] [1]

While the estimated average cost deviates fromβby more thanε out: (a) Setλ←(λ lo +λ hi)/2

Outer loop (budget via bisection in λ≥0 ).Initialize 0 =λ lo < λhi large enough that the budget is satisfied. While the estimated average cost deviates fromβby more thanε out: (a) Setλ←(λ lo +λ hi)/2. (b)Per-context water-filling.For eachx i: 28 Nonparametric LLM Evaluation from Preference Data i. Define the decreasing function gi(ν) = X j<k max n α, s Wj...

2007

[2] [2]

: (a) ComputeΣ(π (t))(e.g., via the population formula if available, or via an empirical estimate using samples{x i}n i=1)

Fort= 0,1,2, . . .: (a) ComputeΣ(π (t))(e.g., via the population formula if available, or via an empirical estimate using samples{x i}n i=1). (b) Update ˜π(t+1) jk (x) = clip[α,1] vuut tr Σ(π(t))−1 Mjk (x) λ cjk withλchosen by one-dimensional bisection so that the budget constraint holds. (c) Setπ (t+1) ←˜π(t+1) and stop when the iterates stabilize. 30 No...

2023

[3] [3]

Initializeπ (0) (e.g., the A-optimal policy from Theorem 6.2)

[4] [2007]

j wins” vs. “k wins

ISBN 978-0-19-929659-0. Bickel, P. J., Klaassen, C. A. J., Ritov, Y ., and Wellner, JohnA.Efficient and Adaptive Estimation for Semipara- metric Models. Springer New York, 1 edition, 1998. ISBN 978-0-387-98473-5. Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4):324–345, 195...

Pith/arXiv arXiv 1998