Pith. sign in

REVIEW 4 major objections 5 minor 11 references

Winrate, the standard LLM benchmark, incentivizes model homogenization and lower consumer welfare; a value-weighted winrate provably reverses this.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 20:24 UTC pith:7CAGTTFF

load-bearing objection The homogenization result and the weighted-winrate fix are real and new; the paper needs a scope check and a model of noisy values before I'd buy the 'LLM evaluation' framing. the 4 major comments →

arxiv 2602.23293 v2 pith:7CAGTTFF submitted 2026-02-26 cs.GT cs.CY

Impacts of Aggregation on Model Diversity and Consumer Utility

Kate Donahue , Manish Raghavan This is my paper
classification cs.GT cs.CY MSC 91A1091B03
keywords model marketplacewinrateweighted winrateBradley-Terry-Luceconsumer welfaremodel diversitybenchmark incentivesLLM evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks what a healthy marketplace of AI models looks like when users and routers pick models noisily, and how to get producers to create such a marketplace. It models aggregation with the Bradley-Terry-Luce choice rule and shows that winrate—the probability that a model is selected—gives producers an incentive to spread their strengths evenly across tasks, a homogenization that hurts consumers. It then proposes weighted winrate, which rewards a model by its response value times selection probability, and proves that under the same conditions this metric encourages full specialization and strictly raises consumer welfare. Along the way it shows that even without strategic producers, noisy aggregation makes consumer welfare non-monotone: improving a model or adding a better one can reduce total utility. The results are illustrated on a 13-task LLM benchmark.

Core claim

The central claim is that the choice of evaluation metric shapes the entire model ecosystem through producer incentives. With Bradley-Terry-Luce aggregation, a producer who maximizes average winrate in response to a large enough value budget equalizes winrate across tasks, producing homogeneous models; yet consumer welfare is maximized by concentrating value on a single task. Weighted winrate—defined as the probability a model is selected multiplied by the value of its response—aligns the two: the producer's best response is to specialize fully, and the resulting consumer welfare strictly beats the winrate equilibrium whenever the winrate-equalizing condition of Theorem 3 holds. For model re

What carries the argument

The argument is carried by the Bradley-Terry-Luce (BTL) choice model, the workhorse probabilistic choice rule in which selection probability is exp(v/β) over the sum of exp(v/β). Winrate is simply a model's selection probability; weighted winrate is that probability multiplied by the response's value v. The key structural results are Theorem 3, which uses a log-sum-exp threshold to show that winrate best responses homogenize, and Lemma 9, which shows the weighted-winrate best response concentrates all value on a single task. The proof machinery also uses the observation (Lemma 1) that the change in welfare from adding a model is a 'swish'-shaped function of value differences.

Load-bearing premise

The welfare-improving claims for weighted winrate require a trusted scalar value v_{m,t} for each model-task pair; the paper assumes such values are available and correct, and the conclusion acknowledges that if user-provided measures are too expensive or noisy, weighted winrate may not deliver its guarantees.

What would settle it

Construct or observe a setting with two tasks where the entrant's value allocation under weighted winrate is not fully concentrated: for instance, introduce zero-mean noise on v_{m,t} and show that the best response spreads value across tasks, or empirically perturb MT-Bench scores and check whether weighted winrate's welfare advantage over winrate disappears under modest noise.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If leaderboards keep using winrate, stronger entrants will keep homogenizing; the paper shows this is not an accident of any one benchmark but a property of any separable choice function.
  • Adopting weighted winrate in public leaderboards would give producers a direct incentive to specialize, increasing the diversity of available models and the utility of users who aggregate over them.
  • The non-monotonicity results imply that 'improvements' to individual models can backfire under noisy routing, so evaluation of model updates must account for the selection context, not just the model in isolation.
  • Weighted winrate has a known gap: it leaves producers indifferent between returning value-0 and abstaining, and in the worst case this can still hurt consumers; this is the paper's stated open direction.
  • Under bounded task maxima (as on real benchmarks), the specialization results generalize to a greedy allocation rule (Lemma 11).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The same homogenizing pressure should apply to router-based leaderboards like the Max system that aggregate many models: any ranking metric based on selection frequency will push entrants toward the average, not toward complementary strengths.
  • Editorial inference: A natural testable extension is to deploy weighted winrate on a public leaderboard using coarse human value ratings (1-5 stars) and measure whether the distribution of model strengths across tasks becomes more heterogeneous over time.
  • Editorial inference: The theory suggests a calibration condition for the mechanism: the value signal must be accurate enough that the log-sum-exp ordering of tasks is preserved; when value noise is comparable to the gap between tasks, the welfare gain may vanish or reverse.
  • Editorial inference: The non-monotonicity results connect to AI oversight concerns: model homogenization may be driven not only by training data but by the evaluation metric itself, implying that oversight interventions could operate at benchmark design.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies a marketplace of AI models with discrete tasks, scalar per-task values v_{m,t}, BTL choice-based aggregation, consumer welfare equal to expected selected value, and producers maximizing average winrate. It first shows that BTL and other separable choice functions are non-monotone in value, and that monotone aggregation and guaranteed benefits from entry are mutually exclusive under substitutability/anonymity. It then analyzes two producer actions: model creation and model replacement. For model creation, Theorem 3 states that a sufficiently high-value entrant maximizing winrate equalizes winrate across tasks (homogenization), while Lemma 9 states that the proposed weighted-winrate mechanism induces full specialization on one task, and Theorem 4 claims strictly higher consumer welfare under the Theorem 3 regime. For model replacement with N=2, Lemmas 13–15 and Theorem 5 develop analogous local-derivative results. The empirical section uses MT-Bench-101 to illustrate the predicted specialization patterns. Proofs are in the appendix and code is provided.

Significance. If established, the paper's central mechanism-design point is valuable: standard winrate leaderboards can push producers toward homogenization under noisy BTL routing, while a value-weighted selection metric can improve specialization incentives. The theoretical calculus is mostly explicit, the proofs are organized in appendices, and the release of code is a strength. However, the headline claim is conditional on exact, known per-task response values; on large total value V for entry and N=2/local-derivative regimes for replacement; and on the BTL model. The abstract states the results more universally than the theorems support. The experiments are illustrative rather than a quantitative test. Conditional on these caveats, the contribution is a useful addition to the literature on benchmark incentives and model diversity.

major comments (4)
  1. [§2.3, Definition 5; §8] Weighted winrate is defined as p_m(v_t)·v_{m,t}, and all welfare theorems assume the planner and producers know the exact scalar value v_{m,t} for every model-task response. In the LLM-evaluation setting the abstract targets, such values are not directly observable and would need to be elicited or inferred; the paper's own conclusion concedes that 'if user-provided measures of value are sufficiently expensive or noisy... other mechanisms could improve upon weighted winrate,' but no noise or elicitation model is provided. Without such a model, Theorem 4's strict welfare improvement is not robust to perturbing the inputs to Definition 5: producers would best-respond to a different objective, and the comparison can fail even when all other assumptions hold. This is a load-bearing applicability gap for the central claim, not an internal inconsistency. The abstract and the theorem statements
  2. [§5.2, Theorem 3 / Appendix C proof] The proof of Theorem 3 in Appendix C establishes only a pairwise improvement. For a task k with v_k < β log E_k, a pigeonhole argument produces a task j whose combined budget with k exceeds β(log E_j + log E_k), and then Lemma 21 shows that the pair's total winrate could be increased by equalizing winrate across tasks j and k. This demonstrates that a non-equalizing allocation is not pairwise optimal, but it does not prove that the global maximizer equalizes winrate across all T tasks, which is exactly what the theorem states and what Theorem 4 invokes. A rigorous proof would need a global optimality condition (e.g., Lagrange equations showing p_t(1−p_t) is constant across tasks) or an explicit convergence argument for repeated pairwise equalization. As written, the load-bearing theorem is not fully established.
  3. [§5.3, Lemma 10] The statement of Lemma 10 says that the welfare suboptimality of weighted winrate is bounded by u(v_{i*})−u(v_{j*}), where i* maximizes p_{N+1}(v_i⊕V) and j* is the task with smallest current utility u(v_{j*}). The proof instead bounds the gap by u(v_2)−u(v_1), where task 2 is the weighted-winrate pick and task 1 is the consumer-welfare-optimal task. There is no step showing that the consumer-optimal task has the smallest current utility, and no use is made of j* in the proof. Under the stated definitions the claimed bound does not follow; the lemma should either be corrected to use the consumer-optimal task in the bound, or removed/relegated to a less prominent claim.
  4. [Abstract / Theorems 3–5] The abstract and introduction state the results as universal: winrate 'can incentivize model creators to homogenize for both types of model changes, reducing consumer welfare,' and weighted winrate 'provably improves incentives for producers to specialize and increases consumer welfare.' The actual theorems are substantially more conditional: Theorem 3 requires V ≥ 2Tβ·max_j log(Σ_i exp(v_{j,i}/β)); the model-replacement results are for N=2 and for instantaneous derivative-based changes; and Theorem 5 explicitly excludes the non-monotone regime where one player's improvement reduces welfare on both tasks (Observation 5, Table 1). The abstract should carry the main qualifications; otherwise readers will infer a stronger mechanism-design guarantee than is proven.
minor comments (5)
  1. [§7, Table 2 / Figure 5] The experiments choose V=120 and β=1 precisely to 'demonstrate patterns from our theoretical results,' and the text says the values are 'picked so as to demonstrate.' This is illustrative rather than a test of the theory. Please label it as such and, ideally, add robustness plots over V and β or a random subset of tasks.
  2. [Theorem 4 proof] In the final chain of inequalities, a weak inequality is written as '>2·T·β·max...' where '≥' is intended; the subsequent conclusion that V ≥ 2T·max_j u(v_j) then carries a minor strength-of-inequality mismatch. This is easy to fix but should be corrected for precision.
  3. [§8] There is a typo in the heading: 'F uture directions' should be 'Future directions.'
  4. [Definition 7 / Theorem 1] In Definition 7, the condition '∂f/∂v > 0 in at least one place' is used in the proof of Theorem 1 at a constructed v_1. The phrase 'at least one place' should be clarified to mean 'at the value v_1 used in the construction,' since the proof does not show monotonicity fails at every point.
  5. [Notation] The paper alternates between 'winrate' and 'total winrate' (Assumption 2 vs. Theorem 3) and between 'averaged across tasks' and 'summed across tasks.' Since additive constants do not change the argmax but do affect statements like Lemma 8, a single convention and a sentence clarifying that the argmax is unchanged would help.

Circularity Check

0 steps flagged

No significant circularity: the welfare and incentive theorems are proved from stated assumptions, and weighted winrate's alignment with consumer welfare is a design choice, not an identity that does the proof work.

full rationale

The paper's central results are derived in-line from explicitly stated assumptions rather than from fitted constants or load-bearing self-citations. The BTL choice model (Definition 3) and the producer winrate objective (Assumption 2) are assumptions, and the homogenization results (Theorem 3 and Lemmas 8, 13) are proven from them. The proposed mechanism, weighted winrate (Definition 5), is indeed the per-model term of the consumer-welfare sum in Equation (1), so it is intentionally aligned with welfare. However, this is not a circular reduction: the producer's objective p_m(v_t)·v_{m,t} is not equal to the change in consumer welfare, which Lemma 19 shows is p_{N+1}(v_i⊕V)·(V − u(v_i)). Lemmas 7, 9, and 10 explicitly characterize the residual gap between maximizing consumer welfare, weighted winrate, and winrate, and Theorem 4 is a nontrivial comparison of the two resulting allocations. The Section 8 caveat about noisy or expensive value elicitation is an applicability limitation, not a circular step, since the theorems are conditional on the stated value model. Experiments use MT-Bench-101 values to instantiate the model rather than fitting parameters that are then reported as predictions. No self-citation is load-bearing: Kleinberg and Raghavan (2021) is cited only as motivation for model homogeneity, and the proofs do not rely on it. Therefore no step meets the standard of reducing to its own inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

No new physical or ontological entities are introduced. 'Weighted winrate' is a proposed metric/mechanism, not an entity with a falsifiable handle outside the paper; its viability depends on the scalar-value elicitation assumption. The free parameters are β and V, plus the illustrative c in the monotonicity construction.

free parameters (3)
  • BTL temperature β = 1 in experiments; theoretical variable
    Controls selection noise; results such as the monotone-region bound depend on it, and the experiments set β=1 without a robustness sweep.
  • Total value budget V for a new model = 120 in the model-creation experiment
    V is a theoretical parameter; in Section 7 it is explicitly chosen so that winrate falls in the homogenization regime, making the illustration exhibit the predicted patterns.
  • Constant c in the monotone choice-function example = arbitrary positive constant
    Used in Definition 12/Lemma 5 to construct a monotone aggregation function; any c>0 works, so it is not fitted to data.
axioms (6)
  • domain assumption Consumer welfare is expected value of the selected response under BTL aggregation (Assumption 1).
    Defines the welfare objective; if users optimize a different criterion, the welfare conclusions shift.
  • domain assumption Producers maximize average winrate, defined as BTL selection probability (Assumption 2).
    This is the load-bearing premise for the homogenization results; it is not derived from first principles and may not match real producer objectives.
  • domain assumption Model responses have scalar, non-negative values v_{m,t} (or expected values) that are known or elicitable.
    Weighted winrate requires multiplying selection probability by a numeric value; the paper concedes in Section 8 that acquiring accurate values is hard.
  • domain assumption BTL choice with a fixed temperature β governs noisy user/routing selection.
    The producer-incentive theorems rely on BTL log-sum-exp structure; Theorem 1 only generalizes the non-monotonicity to separable choice functions, not the incentive results.
  • domain assumption In model creation, a producer can allocate arbitrary non-negative value across tasks subject to a total-value cap V.
    Assumes linear effort allocation with no diminishing returns except the cap; the scaling-law variant is handled only in Lemma 11.
  • domain assumption In model replacement, only local/instantaneous changes by exactly two producers are considered.
    Section 6 restricts to N=2 and first-order derivatives; no equilibrium or dynamic analysis is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 36966 in / 22861 out tokens · 200501 ms · 2026-08-02T20:24:54.908172+00:00 · methodology

0 comments
read the original abstract

Consider a marketplace of AI tools, each with slightly different strengths and weaknesses. By picking the right model for the task at hand, a user can do better than simply using the same model for everything. Routers operate under a similar principle, where sophisticated model selection can increase overall performance. However, aggregation is often noisy, reflecting imperfect user choices or routing decisions. This leads to two main questions: first, what does a "healthy marketplace" of models look like for maximizing consumer utility? Secondly, how can we incentivize producers to create such models? We show that winrate, a standard benchmark in LLM evaluation, can incentivize model creators to homogenize for both types of model changes, reducing consumer welfare. We propose a new mechanism, weighted winrate, which rewards models for answers that are higher quality, and show that it provably improves incentives for producers to specialize and increases consumer welfare. We conclude by exploring the impact of our theoretical results in empirical benchmark datasets and discussing implications for benchmark design.

Figures

Figures reproduced from arXiv: 2602.23293 by Kate Donahue, Manish Raghavan.

Figure 1
Figure 1. Figure 1: Plot of Equation 3 for a single task, with [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Figure demonstrating the bound in Lemma 6, where there is a set of values [1 [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Average value of combinations of models (restricted to top 10 models for computational reasons). [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Fraction of derivatives of v(S) that are negative [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Given allocations in Table 2, the change in consumer welfare. The prior aggregate value of the [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Each point represents a pair of models in (Bai et al., 2024). The [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Performance of different models on MT-Bench-101 (Bai et al., 2024) [PITH_FULL_IMAGE:figures/full_fig_p022_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Relative ordering of tasks that different models would choose to improve in, ordered by first [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Relative ordering of regimes that different models would choose to improve in, ordered by first [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Given a setting with two players with values [PITH_FULL_IMAGE:figures/full_fig_p024_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

11 extracted references · 2 linked inside Pith

  1. [1]

    IfBhas weakly higher total value thanA, then both models{A, B}together have higher consumer welfare than justAfor all temperaturesβ

  2. [2]

    Even ifBhas strictly lower total value thanA, it is still possible that both models{A, B}together can have higher consumer welfare than justAfor someβ <∞. Proof.1. This comes directly by observing that X t∈[T] exp((vt,b −v t,a)/β) exp((vt,b −v t,a)/β) + 1 ·(v t,b −v t,a)≥ X t∈[T] 0.5·(v t,b −v t,a)≥0 where the last condition holds if P t∈T vt,b ≥ P t∈T vt,a

  3. [3]

    9),p i(⃗ v)−pi(⃗ v′)≥0 for alli̸=j

    By substitutability (Def. 9),p i(⃗ v)−pi(⃗ v′)≥0 for alli̸=j. Considering the inequality from Lemma 17, we know thatu(⃗ v′)≥u(⃗ v) exactly whenever: X i∈⃗ v,i̸=j (pi(⃗ v)−pi(⃗ v′))·(v ′ j −v i)>(v j −v ′ j)·p j(⃗ v) (6) For the lefthand side, property 2 tells us thatv ′ j −v i is negative always and property 3 tells us that pi(⃗ v)−pi(⃗ v′)≥0. Note that t...

  4. [5]

    Consider a setting with exactly two tasksT= 2, and model A has values [6,8], whereas model B has values [7,5]

    This can be proved by example. Consider a setting with exactly two tasksT= 2, and model A has values [6,8], whereas model B has values [7,5]. Note that while model B has lower total value, its relative strengths are anti-correlated with model A. If choice-based aggregation is “strong” enough, adding modelB could help. Specifically: •BTL aggregation with n...

  5. [6]

    By assumption,⃗ vj < ⃗ v′ j and so⃗ vj −⃗ v′ j <0

  6. [7]

    By assumption, for all otheri̸=j, we have⃗ v ′ j < ⃗ vi, and so⃗ v′ j −⃗ vi <0

  7. [9]

    remainder

    + 0· X t̸=1 0·p c(⃗ v′ t)> X t∈[T] ⃗ vc,t ·p c(⃗ vt) X t∈[T] ⃗ vc,t ·p c(⃗ v′ 1)> X t∈[T] ⃗ vc,t ·p c(⃗ vt) (RewritingV c) X t∈[T] ⃗ vc,t ·(p c(⃗ v′ 1)−p c(⃗ vt))>0 34 This last inequality holds because ifp c(⃗ v′ 1)−p c(⃗ vt) for anyt∈[T] by design: pc(⃗ v′ 1)≥p c(⃗ vt) exp(V /β) exp(V /β) +E1 ≥ exp(vct/β) exp(vct/β) +E t (Et = P i̸=c,i∈[N] exp(⃗ vit/β))...

  8. [10]

    (By Lemma 24): Both players get higher winrate in picking their task:p b(⃗ v2)> pb(⃗ v1) andp a(⃗ v1)> pa(⃗ v2)

  9. [11]

    Combining both facts, this must mean that it has winrate<0.5 on both tasks, and the other agent has winrate>0.5 on both tasks

    (By Lemma 23): At least one agent has winrate<0.5 on the task that it picks. Combining both facts, this must mean that it has winrate<0.5 on both tasks, and the other agent has winrate>0.5 on both tasks. WLOG, assume that it is agentAthat does worse. Note that this implies ⃗ vbi > ⃗ vai fori= 1,2. When is the case where we getworsetotal sum of derivatives...

  10. [2014]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Diverse randomized agents vote to win.Advances in Neural Information Processing Systems27 (2014). Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361(2020). Michael Kearns, Aaron Roth, and ...

    Pith/arXiv arXiv 2014

  11. [2025]

    arXiv preprint arXiv:2503.10657(2025)

    Routereval: A comprehensive benchmark for routing llms to explore model-level scaling up in llms. arXiv preprint arXiv:2503.10657(2025). Meena Jagadeesan, Michael Jordan, Jacob Steinhardt, and Nika Haghtalab. 2023. Improved bayes risk can yield reduced social welfare under competition.Advances in Neural Information Processing Systems36 (2023), 66940–66952...

    Pith/arXiv arXiv 2025