REVIEW 29 cited by
WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
We introduce WildBench, an automated evaluation framework designed to benchmark large language models (LLMs) using challenging, real-world user queries. WildBench consists of 1,024 tasks carefully selected from over one million human-chatbot conversation logs. For automated evaluation with WildBench, we have developed two metrics, WB-Reward and WB-Score, which are computable using advanced LLMs such as GPT-4-turbo. WildBench evaluation uses task-specific checklists to evaluate model outputs systematically and provides structured explanations that justify the scores and comparisons, resulting in more reliable and interpretable automatic judgments. WB-Reward employs fine-grained pairwise comparisons between model responses, generating five potential outcomes: much better, slightly better, slightly worse, much worse, or a tie. Unlike previous evaluations that employed a single baseline model, we selected three baseline models at varying performance levels to ensure a comprehensive pairwise evaluation. Additionally, we propose a simple method to mitigate length bias, by converting outcomes of ``slightly better/worse'' to ``tie'' if the winner response exceeds the loser one by more than $K$ characters. WB-Score evaluates the quality of model outputs individually, making it a fast and cost-efficient evaluation metric. WildBench results demonstrate a strong correlation with the human-voted Elo ratings from Chatbot Arena on hard tasks. Specifically, WB-Reward achieves a Pearson correlation of 0.98 with top-ranking models. Additionally, WB-Score reaches 0.95, surpassing both ArenaHard's 0.91 and AlpacaEval2.0's 0.89 for length-controlled win rates, as well as the 0.87 for regular win rates.
Forward citations
Cited by 29 Pith papers
-
Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoning
On a new 615-question business-case benchmark graded by AI against instructor rubrics, frontier LLMs score 87-88% partial credit but complete only about half the questions.
-
Self-Preference Bias in Rubric-Based Evaluation of Large Language Models
Self-preference bias persists in rubric-based LLM evaluation even with fully objective, programmatically verifiable rubrics, and can shift subjective medical-chat scores by up to ~10 points.
-
SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks
An arena of 4,357 blind votes ranks 16 LLMs on spreadsheet generation and finds wins track formatting and structure, not formula sophistication.
-
UQ: Assessing Language Models on Unsolved Questions
Unsolved Stack Exchange questions can serve as a dynamic benchmark: the best tested model passes validator screening on only 15% of 500 questions.
-
MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Space
MIG greedily selects instruction-tuning data by maximizing a concave information measure over a label graph, and with 5% of Tulu3 data it matches or exceeds full-data SFT performance.
-
Copilot Arena: A Platform for Code LLM Evaluation in the Wild
An in-IDE pairwise-preference platform for code LLMs reveals that real developer choices rank models differently than static coding benchmarks.
-
FinReportBench: Measuring and Improving Institution-Grade Financial Report Generation
FinReportBench is a fine-grained, expert-grounded benchmark for institution-grade LLM financial report generation, and its skill-evolution method improves G1 and G2 scores across model families.
-
Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation
A new benchmark (IG-Bench) reveals that LLM-based scientists fail at compositional lineage reasoning, with the best system reaching only 27.3% exact accuracy.
-
Response drift across frontier large language models
All ten tested frontier LLMs deviate substantially from expert-validated references, with eight models forming an indistinguishable ~78–81% fidelity ceiling while Claude and Gemini reach ~47–49%.
-
A global log for medical AI
MedLog defines a nine-field, syslog-style event log for clinical AI, intended to support real-world surveillance and auditing; the four-deployment validation claimed in the abstract is absent from the body.
-
UserBench: An Interactive Gym Environment for User-Centric Agents
A new multi-turn agent benchmark shows that current LLMs elicit fewer than 30% of user preferences and reach full intent alignment only about 20% of the time.
-
CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards
A 7B judge model trained with verifiable reward signals and a margin contrastive loss matches the judgment accuracy of models tens of times larger, and a new benchmark JudgerBenchV2 standardizes judge evaluation.
-
DocTalk: Scalable Graph-based Dialogue Synthesis for Enhancing LLM Conversational Capabilities
A graph-based pipeline converts related Wikipedia documents into 730k long multi-turn dialogues, and continued pre-training on them improves a 7B LLM's context memory and understanding by up to 40% relative to baselines.
-
Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models
A fully automatic LLM evaluation framework where all evaluated models serve as judges for one another reaches 97% Spearman correlation with human preference rankings while keeping cost sub-quadratic.
-
R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation
R-Bench is a new bilingual, multidisciplinary, graduate-level reasoning benchmark on which top AI models score 69% (text) and 53% (multimodal), well below saturation.
-
What's the Difference? Supporting Users in Identifying the Effects of Prompt and Model Changes Through Token Patterns
Token-pattern mining can surface systematic differences between groups of LLM outputs, helping users identify effects of prompt and model changes.
-
SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia
SeaExam and SeaBench are locally sourced benchmarks for Indonesian, Thai, and Vietnamese that show larger cross-model performance differences than translated English benchmarks, though the evidence for 'better discrim...
-
Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons
A fine-tuned 14B LLM judge, trained with scenario-based prompts and controlled instruction generation, approaches GPT-4's human-agreement performance, and the paper documents why scaling distillation data can fail.
-
R.I.P.: Better Models by Survival of the Fittest Prompts
RIP filters preference-optimization training data by keeping prompts whose rejected responses are high quality and whose chosen/rejected reward gap is small, yielding consistent benchmark gains over unfiltered data.
-
HREF: Human Response-Guided Evaluation of Instruction Following in Language Models
Adding human-written reference responses as context improves LLM-as-a-judge agreement with human preferences by up to 3.2%, yielding a new private task-centric benchmark, HREF.
-
Challenges in Trustworthy Human Evaluation of Chatbots
Open chatbot leaderboards like Chatbot Arena can have their model rankings moved by several positions with only 10% low-quality or adversarial votes.
-
Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains
Relay-Bench, a 30-problem benchmark of chained multi-domain tasks with encoded prompts, resists saturation: the best tested model, GPT-5.5, scores 43.3% Pass@1.
-
InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training
Rubric-based incremental RL with LLM-generated case-specific checklists lifts Qwen3-4B's HealthBench-Hard score from 7.0 to 27.5 with 2k samples, and improves InfoBench instruction-following from 42.0 to 82.9.
-
Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yet
Large AI judges agree only weakly to moderately with human raters when ranking the harmfulness of smaller AI models' outputs, and the three small models differ in how often they produce harmful content.
-
Yi-Lightning Technical Report
Yi-Lightning, a new MoE LLM from 01.AI, claims competitive human-preference rankings and modest benchmark gains over prior models.
-
OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries
DR.INFO, a vendor-built RAG clinical assistant, is reported to beat frontier LLMs on OpenAI's HealthBench, but the paper's own scores contradict its abstract.
-
Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings
DEEVO evolves better LLM prompts by debating outputs and selecting survivors with Elo ratings, without requiring labeled data or a hand-written fitness function.
-
Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks
A survey that frames the core problem of LLM evaluation as 'evaluation generalization': finite test sets cannot scale with unbounded model capabilities, and proposes two transitions in evaluation design.
-
BgGPT 1.0: Extending English-centric LLMs to other languages
Continually pretraining Gemma-2 on a curated Bulgarian corpus and merging with instruction-tuned models yields open Bulgarian-English models that beat larger open models on Bulgarian benchmarks.
Discussion (0). Continue with ORCID to comment.