Pith. sign in

REVIEW 3 major objections 5 minor 54 references

Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Using every LLM as a judge can beat single-judge benchmarks at ranking models by human preference.

desk verdict All-models-as-judges evaluation is a promising idea with a strong headline correlation, but the sub-quadratic complexity claim is wrong as stated and the hyperparameter tuning leaks the target. read the letter →

arxiv 2505.12808 v1 pith:TBY2NLQG submitted 2025-05-19 cs.CL cs.LG

Yanbin Yin , Kun Zhou , Zhen Wang , Xiangdong Zhang , Yifei Shao , Shibo Hao , Yi Gu , Jieyuan Liu
show 6 more authors
Somanshu Singla Tianyang Liu Eric P. Xing Zhengzhong Liu Haojian Jin Zhiting Hu
This is my paper · ORCID
classification cs.CLcs.LG
keywords decentralizedevaluationLLM-as-a-judgecollectiveintelligenceEloratingpairwisecomparisoncoarse-to-finerankingrepresentativequestionselectionChatbotArenacorrelation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that language-model evaluation can be made both more human-aligned and more scalable by letting every evaluated model serve as a judge. Instead of relying on one strong judge or a small committee, De-Arena runs democratic pairwise comparisons in which the full pool of models votes, weights each judge's vote by its own estimated skill, and inserts new models via binary search plus local reranking. On 66 LLMs this automated pipeline reaches a 0.974 Spearman correlation with the human-judged Chatbot Arena overall ranking, higher than any benchmark it compares against, while keeping comparison counts sub-quadratic. If the claim holds, model creators could obtain trustworthy, fine-grained leaderboards for thousands of models and new dimensions without expensive human annotation or the biases of a single authority judge.

What carries the argument

The load-bearing mechanism is a three-part pipeline. First, pairwise comparison: every model in the pool votes on which of two outputs wins, and a model never judges its own output. Second, coarse-to-fine incremental ranking: a new model is inserted by binary search against the current ranking, then re-ranked inside a small sliding window of neighbours, which concentrates judge effort on ambiguous pairs and keeps complexity near $O(k n \log n)$ rather than $O(n^3 k)$. Third, Elo score generation with judge weighting and style control: after comparisons, Elo scores are fit by logistic regression, each judge's normalized Elo is used as its weight in the loss, and output styles (length, headers, lists, bold) are controlled to reduce style-driven bias. A separate ranking-based question-selection step picks the questions whose per-question rankings most agree with the average ranking, so new evaluation dimensions can be built automatically.

What would settle it

Run De-Arena on the same 66 models with different initial Elo values or with judge weights frozen at their first iteration; if the final rankings shift materially, for example by more than a few adjacent swaps among close models, the claim that the weighting loop reliably improves human agreement is falsified. The paper's insertion-order shuffles show low variance, but they do not test initialization sensitivity.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the 'wisdom of the crowd' transfers to LLM evaluation: a decentralized system where all evaluated models judge each other produces rankings closer to human preference than single-judge or small-committee automated benchmarks, and it does so without exhaustively comparing every pair. The paper reports Spearman correlations with Chatbot Arena of 0.957, 0.967, and 0.974 for test sets of 15, 30, and 66 LLMs in the overall dimension, with correlation improving as the judge pool grows. It further reports that weighting judge votes by the judges' own Elo ratings adds a small but consistent gain (for example, 0.957 versus 0.949 on MT-Bench), and that its question-selection method outperforms random, perplexity-based, anchor-point, and single-strong-judge selection.

Load-bearing premise

The ranking rests on the assumption that the iterative loop of re-estimating Elo scores and re-weighting each judge by its own Elo converges to a stable, unbiased ranking; the paper gives no proof of convergence or uniqueness for this fixed point.

Editorial extensions

If this is right

  • Automatic leaderboards can be updated incrementally: adding a new model costs comparisons proportional to the log of the list size plus a small window, rather than a full re-ranking of all pairs.
  • Fine-grained dimensions such as algebra, geometry, probability, social reasoning, chemistry, biology, physics, and general chat can be constructed automatically from open-source question collections, removing the bottleneck of human-curated data.
  • As more models participate, agreement with human preference should continue to rise, since the paper observes monotone improvement as the judge pool grows from 8 to 16 to 26 models.
  • Because judge weights come from the models' own Elo scores, the method adapts to a growing pool without needing a fixed authority judge, which avoids the self-preference bias seen when a single model judges itself and its relatives.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not pursue is using the same ranking pipeline to build arbitrary new dimensions from scratch: any question collection could be standardized to open-ended format and scored by ranking consistency, so the barrier to creating a bespoke leaderboard drops to dataset collection.
  • If judge weighting is doing real work, then a testable consequence is that the method's advantage over unweighted voting should grow as the pool includes larger ability gaps; the paper's reported gap is small, so this may be a modest effect.
  • The democratic design also suggests a governance interpretation: rankings become a fixed point of a social-choice process rather than a decree of one model, which could be probed by checking whether excluding any single judge materially changes the ranking.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes De-Arena, a fully automatic LLM evaluation framework in which every model under evaluation also serves as a judge. The method combines coarse-to-fine incremental ranking (binary search followed by local in-window reranking), Elo-based score aggregation with adaptively weighted judges, and automatic representative question selection. The authors evaluate De-Arena on 66 LLMs across nine fine-grained dimensions and report Spearman correlations with the human-judged Chatbot Arena leaderboard up to 0.974 overall, outperforming 14 compared benchmarks including MixEval, WildBench, and Auto Arena. Additional experiments examine single-judge bias, group bias, insertion-order stability, question selection, style control, and cost.

Significance. If the reported correlations are robust, De-Arena is a significant contribution to automatic LLM evaluation: it replaces one or a few authority judges with collective voting by all evaluated models, thereby addressing a recognized source of judge bias, and it includes an automatic question-selection mechanism for building new evaluation dimensions. The paper's strengths include a broad comparison against 14 benchmarks, experiments with 66 models, nine fine-grained dimensions, stability studies under shuffled insertion order, and the promise of public code and data. The empirical comparisons and ablations are useful regardless of the theoretical framing. However, the central scalability claim and the independence of the reported headline correlations both need additional scrutiny before the contribution can be accepted at face value.

major comments (3)
  1. [Abstract; §3.1; Figure 2b] The claimed sub-quadratic complexity is not supported by the described algorithm. Section 3.1 states that the coarse-grained binary search has time complexity O(kn log n), and the abstract advertises "sub-quadratic complexity" and efficiency at scale. In the algorithm, each binary-search comparison uses all other models in the list as judges over k questions, so one insertion costs Θ(k t log t) judge calls for the binary search plus additional Θ(k t) judge calls for the fine-grained window reranking. Summing over t = 6..n insertions gives Θ(k n^2 log n) total judge calls. Since Figure 2b measures cost in "average judge counts," judge calls are the natural cost metric, and the total is not sub-quadratic. If the intended claim is only about the number of model-pair comparisons, that count is O(n log n), but the paper's own cost metric counts judge calls. The manuscript should state the per-insertion complexity explicitly and either correct or qualify the total-complexity claim; as written, the scalability headline is unsupported.
  2. [Appendix B, Tables 9 and 10] The key hyperparameters (window size and base model number) are selected by maximizing the same Chatbot Arena Spearman correlation that is reported as the main result in Table 2. For example, Table 10 chooses base model number 6 because it yields the highest MT-Bench and Math correlations, and Table 9 chooses window size 1 for the same reason. This is a form of selection on the evaluation target, so the reported correlations (including 0.957, 0.962, and the 0.974 in Table 2) are optimistic estimates of the method's agreement with human judgments. The authors should provide a validation split, report performance across a range of hyperparameter choices instead of only the selected maximum, or otherwise show that the headline numbers are not artifacts of this tuning.
  3. [§3.1, Score Generation and Style Control; Table 7] The judge-weighting scheme creates an unexamined feedback loop. The paper assigns each judge a weight proportional to its normalized Elo score and updates both the Elo scores and the weights iteratively. Because the Elo scores are themselves estimated from judge-weighted comparisons, this is a fixed-point iteration, but the paper offers no convergence, uniqueness, or bias analysis. A plausible failure mode is that a model with inflated Elo (for example, due to self-voting or family bias) receives higher weight and thereby further inflates its own Elo. Table 7 only compares final correlations with and without weights; it does not test whether the iterative process converges or whether the resulting fixed point is stable. The authors should report convergence diagnostics (e.g., trajectories of weights and Elo scores across iterations) or otherwise address the stability and fairness of this loop.
minor comments (5)
  1. [Abstract] The phrase "across extensive experiments across 66 LLMs" contains a duplicated preposition; rephrase for clarity.
  2. [§3.1] The sentence "The time complexity of this binary search is O(kn log n)" is ambiguous: it should specify whether this is the cost per insertion or the total cost over all insertions, since the surrounding text compares it with the total O(n^3 k) cost of full pairwise evaluation.
  3. [§4, Table 3] The text says the single-judge variants include GPT-4o-2024-08-06, but Table 3 lists only LLaMA-3-70B, Gemma-2-27B, Qwen2-70B, and De-Arena; the table should either include GPT-4o or the text should be corrected.
  4. [§4.2, Table 4 and Figure 2b] The cost metric "number of judges" is not precisely defined. Please clarify whether it counts pairwise votes, judge calls, or generated tokens, since this affects how the cost comparison in Figure 2b should be interpreted.
  5. [Appendix B, Tables 9 and 10] The hyperparameter choices differ by very small correlation margins (e.g., 0.957 vs. 0.955 for window size 3 in MT-Bench), yet no confidence intervals or significance tests are reported; the authors should indicate whether these differences are meaningful.

Circularity Check

2 steps flagged · score 4.0 of 10

Main Chatbot-Arena correlation is external and not tautological, but the judge-weighting loop is self-definitional and key hyperparameters are tuned on the same target metric, making part of the reported performance in-sample.

  1. self definitional [Section 3.1, 'Score Generation and Style Control' (Eq. 1 and following paragraph)]
    "we consider that the reliability of different LLMs as judges varies. Therefore, we introduce weights in the loss function. Our rationale is that an LLM with a higher Elo score is more likely to be a qualified judge; hence, we utilize the normalized Elo score as the weight in the loss function. Furthermore, whenever the Elo scores are updated, we dynamically adjust each model’s weight based on its new score."

    The judge weight is defined as the normalized Elo score, while the Elo score is itself estimated (Eq. 1, via weighted logistic regression) from the votes of those same judges. Thus 'qualified judge' is defined by the very rating the judge-weighted loop is trying to produce; the loop is a self-referential fixed point with no convergence, uniqueness, or fairness proof. The paper's ablation (Table 7) does compare the loop against no weights using the external Chatbot Arena target, so the main correlation is not tautological, but the weighting mechanism itself is circular by construction.

  2. fitted input called prediction [Appendix B, 'Hyper-parameter Tuning' (Tables 9-10), versus Table 2]
    "In De-Arena, the window size and base model number are two hyper-parameters that control the cost of in-window reranking and the initial ranking list, respectively. Here, we study their best settings by varying them in [1, 2, 3] and [3, 6, 9, 12], respectively. ... setting the window size to 1 can lead to the fewest judge counts, and also achieve a good correlation score ... using 6 base models can achieve the best performance."

    The hyperparameters are selected by maximizing the Spearman correlation with Chatbot Arena, and the same correlation is then reported as De-Arena's headline performance (Table 2: 0.974 at 66 LLMs). No held-out set of human judgments is used, so the reported correlation is an in-sample, tuned number rather than an out-of-sample prediction. It is not forced by construction because only a few discrete parameter choices are searched, but it is a partial, minor circularity in the evaluation claim.

full rationale

The paper's central empirical claim is that De-Arena's ranking correlates up to 0.974 with Chatbot Arena human judgments. That target is external and independent, so the headline correlation is not a tautology and the main derivation chain does not reduce to its inputs. However, two internal feedback loops weaken the claim. First, the judge-weighting scheme sets each judge's weight equal to its normalized Elo score, while those Elo scores are computed from the same judges' weighted votes; 'qualified judge' is therefore defined by the very quantity being estimated. Second, the window size and base-model count are tuned on the same Chatbot-Arena Spearman correlation that is then reported as the method's success, and no held-out human-judgment split is presented. The advertised sub-quadratic complexity is also not established, since the binary search uses all existing models as judges per comparison, giving Θ(k n^2 log n) total judge calls when summed over insertions, but this is a complexity-analysis error rather than a circularity. Overall, the central human-correlation result remains independently meaningful, so the circularity score is moderate rather than high.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. The free parameters are hyperparameters tuned on the target benchmark (window size, base model number) plus manual choices for question count and selection-model count. The key domain axioms are the wisdom-of-crowds assumption and the validity of Chatbot Arena as ground truth. The most notable ad-hoc assumption is the Elo-as-judge-quality weighting loop, which is internally self-referential.

free parameters (4)
  • window_size = 1
    Chosen from {1,2,3} by maximizing Chatbot Arena correlation in Appendix B, Table 9.
  • base_model_number = 6
    Chosen from {3,6,9,12} by maximizing Chatbot Arena correlation in Appendix B, Table 10.
  • selected_question_count = 32 out of 80 for MT-Bench
    The number of representative questions selected in the question-selection experiment (Figure 5b) is a manual choice; no sensitivity analysis is given.
  • number_of_models_for_question_selection = 15
    Appendix I states 15 open-source models are used in the data selection process to reduce time cost; this choice is not varied or justified.
assumptions (5)
  • domain assumption Collective judgments of diverse LLMs approximate human preferences on open-ended questions.
    The entire De-Arena design rests on this wisdom-of-crowds premise, stated in Section 2 and used throughout the method.
  • domain assumption Spearman correlation with the Chatbot Arena leaderboard is a valid gold-standard measure of evaluation quality.
    Section 4 and Appendix H use this as the primary success metric; no alternative ground truth is considered.
  • ad hoc to paper A judge LLM's Elo score is a valid proxy for its judging quality.
    Section 3.1 introduces judge weights proportional to normalized Elo scores, a self-referential assumption not derived from external evidence.
  • domain assumption The pool of 66 LLMs is diverse enough to avoid correlated group biases.
    Section 4.1 and Appendix C acknowledge that group biases could arise if models share training data; the paper assumes the chosen set is sufficiently diverse.
  • standard math Standard mathematical tools: binary search correctness, Elo update formula, Spearman rank correlation.
    Used in Sections 3.1 and 3.2 without proof, which is appropriate for standard results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models." pith.science (2026). https://pith.science/paper/TBY2NLQG

@misc{pith2026250512808,
  author       = {Pith},
  title        = {Pith review of: Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TBY2NLQG}},
  note         = {Machine review of arXiv:2505.12808}
}
read the original abstract

The recent explosion of large language models (LLMs), each with its own general or specialized strengths, makes scalable, reliable benchmarking more urgent than ever. Standard practices nowadays face fundamental trade-offs: closed-ended question-based benchmarks (eg MMLU) struggle with saturation as newer models emerge, while crowd-sourced leaderboards (eg Chatbot Arena) rely on costly and slow human judges. Recently, automated methods (eg LLM-as-a-judge) shed light on the scalability, but risk bias by relying on one or a few "authority" models. To tackle these issues, we propose Decentralized Arena (dearena), a fully automated framework leveraging collective intelligence from all LLMs to evaluate each other. It mitigates single-model judge bias by democratic, pairwise evaluation, and remains efficient at scale through two key components: (1) a coarse-to-fine ranking algorithm for fast incremental insertion of new models with sub-quadratic complexity, and (2) an automatic question selection strategy for the construction of new evaluation dimensions. Across extensive experiments across 66 LLMs, dearena attains up to 97% correlation with human judgements, while significantly reducing the cost. Our code and data will be publicly released on https://github.com/maitrix-org/de-arena.

Figures

Figures reproduced from arXiv: 2505.12808 by the authors.

Figure 1
Figure 1. The overview of our method, consisting of the automatic representative question selection [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. (a) Spearman correlation between different LLM benchmarks in the overall dimension. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Spearman correlation with Chatbot Arena across varying judge model number. Robustness against Potential Group Biases. Since De￾Arena leverages the collective intelligence of LLMs to judge each other, it effectively mitigates single-judge biases. To further assess its robustness and potential group biases, we varied the number of judge models across three settings: 8, 16, and 26. For the 8 and 16 settings, we randoml… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: (a) De-Arena’s mean (blue curve) and variance (shaded area) of Chatbot Arena correlation [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: (a) Binary search ranking differences between binary search and ground truth in four [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Correlations between the ranking results from different dimensions in De-Arena and [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Win-rate distribution map for the evaluated LLMs using De-Arena. [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 15 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  3. [3]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  4. [4]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    arXiv 2024

  5. [5]

    Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models

    Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  6. [6]

    Pixiu: A large language model, instruction data and evaluation benchmark for finance

    Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. Pixiu: A large language model, instruction data and evaluation benchmark for finance. arXiv preprint arXiv:2306.05443, 2023

    arXiv 2023

  7. [7]

    Evaluating the text-to-sql capabilities of large language models

    Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. Evaluating the text-to-sql capabilities of large language models. arXiv preprint arXiv:2204.00498, 2022

    arXiv 2022

  8. [8]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132, 2024

    arXiv 2024

Show all 54 references
  1. [9]

    Large language models for software engineering: A systematic literature review

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology, 33(8):1–79, 2024

    2024

  2. [10]

    Galactica: A large language model for science

    Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022

    2022 arXiv

  3. [11]

    Large language models for automatic equation discovery of nonlinear dynamics

    Mengge Du, Yuntian Chen, Zhongzheng Wang, Longfeng Nie, and Dongxiao Zhang. Large language models for automatic equation discovery of nonlinear dynamics. Physics of Fluids, 36(9), 2024

    2024

  4. [12]

    Large language models in medicine.Nature medicine, 29(8):1930–1940, 2023

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine.Nature medicine, 29(8):1930–1940, 2023

    1930

  5. [13]

    Towards understanding sycophancy in language models

    Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548, 2023

    arXiv

  6. [14]

    this is a problem, don’t you agree?

    Stephanie Schoch, Diyi Yang, and Yangfeng Ji. “this is a problem, don’t you agree?” framing and bias in human evaluation for natural language generation. In Proceedings of the 1st Workshop on Evaluating NLG Evaluation, pages 10–16, 2020

    2020

  7. [15]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024. 10

    2024 arXiv

  8. [16]

    Alpacaeval: An automatic evaluator of instruction-following models, 2023

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models, 2023

    2023

  9. [17]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  10. [18]

    Llm evaluators recognize and favor their own generations

    Arjun Panickssery, Samuel Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems, 37:68772–68802, 2024

    2024

  11. [19]

    The role of collective intelligence in crowdsourcing innovation

    Juho Salminen et al. The role of collective intelligence in crowdsourcing innovation. 2015

    2015

  12. [20]

    The Wisdom of Crowds: Why the Many Are Smarter than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations

    James Surowiecki. The Wisdom of Crowds: Why the Many Are Smarter than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations. Doubleday, New York, 2004

    2004

  13. [21]

    Great models think alike and this undermines ai oversight

    Shashwat Goel, Joschka Struber, Ilze Amanda Auzina, Karuna K Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, and Jonas Geiping. Great models think alike and this undermines ai oversight. arXiv preprint arXiv:2502.04313, 2025

    2025 arXiv

  14. [22]

    Mover- score: Text generation evaluating with contextualized embeddings and earth mover distance

    Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. Mover- score: Text generation evaluating with contextualized embeddings and earth mover distance. arXiv preprint arXiv:1909.02622, 2019

    1909 arXiv

  15. [23]

    Bertscore: Evaluating text generation with bert

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  16. [24]

    Bartscore: Evaluating generated text as text generation

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263–27277, 2021

    2021

  17. [25]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024

    2024

  18. [26]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024

    2024

  19. [27]

    Sadler, Wei-Lun Chao, and Yu Su

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023

    2023

  20. [28]

    Openai o1 system card, 2024

    OpenAI. Openai o1 system card, 2024

    2024

  21. [29]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024

    2024 arXiv

  22. [30]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024

    2024

  23. [31]

    Rethinking pragmatics in large language models: Towards open-ended evaluation and preference tuning

    Shengguang Wu, Shusheng Yang, Zhenglun Chen, and Qi Su. Rethinking pragmatics in large language models: Towards open-ended evaluation and preference tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22583–22599, 2024

    2024

  24. [32]

    NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark

    Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association...

    2023

  25. [33]

    Generative judge for evaluating alignment

    Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470, 2023. 11

    2023 arXiv

  26. [34]

    Prometheus 2: An open source lan- guage model specialized in evaluating other language models

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Gra- ham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source lan- guage model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535, 2024

    2024 arXiv

  27. [35]

    Verbosity bias in preference labeling by large language models

    Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. Verbosity bias in preference labeling by large language models. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023

    2023

  28. [36]

    Prd: Peer rank and discussion improve large language model based evaluations

    Ruosen Li, Teerth Patel, and Xinya Du. Prd: Peer rank and discussion improve large language model based evaluations. arXiv preprint arXiv:2307.02762, 2023

    2023 arXiv

  29. [37]

    Auto arena of llms: Automating llm evaluations with agent peer-battles and committee discussions

    Ruochen Zhao, Wenxuan Zhang, Yew Ken Chia, Deli Zhao, and Lidong Bing. Auto arena of llms: Automating llm evaluations with agent peer-battles and committee discussions. arXiv preprint arXiv:2405.20267, 2024

    2024 arXiv

  30. [38]

    A bayesian approach towards crowdsourcing the truths from llms

    Peiran Yao, Jerin George Mathew, Shehraj Singh, Donatella Firmani, and Denilson Barbosa. A bayesian approach towards crowdsourcing the truths from llms. In NeurIPS 2024 Workshop on Bayesian Decision-making and Uncertainty, 2024

    2024

  31. [39]

    A multiagent approach for collective decision making in knowledge management

    Imène Brigui-Chtioui and Inès Saad. A multiagent approach for collective decision making in knowledge management. Group Decision and Negotiation, 20:19–37, 2011

    2011

  32. [40]

    Swarm intelligence: A review of algorithms

    Amrita Chakraborty and Arpan Kumar Kar. Swarm intelligence: A review of algorithms. Nature-inspired computing and optimization: Theory and applications, pages 475–494, 2017

    2017

  33. [41]

    Swarm creativity: Competitive advantage through collaborative innovation networks

    Peter A Gloor. Swarm creativity: Competitive advantage through collaborative innovation networks. Oxford University Press, 2006

    2006

  34. [42]

    Binary search algorithm

    Anthony Lin. Binary search algorithm. WikiJournal of Science, 2(1):1–13, 2019

    2019

  35. [43]

    The proposed uscf rating system, its development, theory, and applications

    Arpad E Elo. The proposed uscf rating system, its development, theory, and applications. Chess life, 22(8):242–247, 1967

    1967

  36. [44]

    Opencompass: A universal evaluation platform for foundation models

    OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass, 2023

    2023

  37. [45]

    Patil, Ion Stoica, and Joseph E

    Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard.https://gorilla.cs.berkeley. edu/blogs/8_berkeley_function_calling_leaderboard.html, 2024

    2024

  38. [46]

    Holistic evaluation of language models

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022

    2022 arXiv

  39. [47]

    Livebench: A challenging, contamination-free llm benchmark

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314, 2024

    2024 arXiv

  40. [48]

    Eq-bench: An emotional intelligence benchmark for large language models

    Samuel J Paech. Eq-bench: An emotional intelligence benchmark for large language models. arXiv preprint arXiv:2312.06281, 2023

    2023 arXiv

  41. [49]

    Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures

    Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures. arXiv preprint arXiv:2406.06565, 2024

    2024 arXiv

  42. [50]

    Open llm leaderboard v2

    Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_ llm_leaderboard, 2024

    2024

  43. [51]

    The biggen bench: A principled benchmark for fine-grained evaluation of language models with language models.arXiv preprint arXiv:2406.05761, 2024

    Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, et al. The biggen bench: A principled benchmark for fine-grained evaluation of language models with language models.arXiv preprint arXiv:24...

    2024 arXiv

  44. [52]

    Wildbench: Benchmarking llms with challenging tasks from real users in the wild

    Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. arXiv preprint arXiv:2406.04770, 2024

    2024 arXiv

  45. [53]

    Anchor points: Benchmarking models with much fewer examples

    Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela. Anchor points: Benchmarking models with much fewer examples. In Yvette Graham and Matthew Purver, editors,Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Vol...

    2024

  46. [54]

    authority

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. 13 Table 8: Style control ablation study results across different dimensions. Each ...

    2009 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.