REVIEW 5 cited by
Investigating Non-Transitivity in LLM-as-a-Judge
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Investigating Non-Transitivity in LLM-as-a-Judge
read the original abstract
Automatic evaluation methods based on large language models (LLMs) are emerging as the standard tool for assessing the instruction-following abilities of LLM-based agents. The most common method in this paradigm, pairwise comparisons with a baseline model, critically depends on the assumption of transitive preferences. However, the validity of this assumption remains largely unexplored. In this study, we investigate the presence of non-transitivity within the AlpacaEval framework and analyze its effects on model rankings. We find that LLM judges exhibit non-transitive preferences, leading to rankings that are sensitive to the choice of the baseline model. To mitigate this issue, we show that round-robin tournaments combined with Bradley-Terry models of preference can produce more reliable rankings. Notably, our method increases both the Spearman correlation and the Kendall correlation with Chatbot Arena (95.0% -> 96.4% and 82.1% -> 86.3% respectively). To address the computational cost of round-robin tournaments, we propose Swiss-Wise Iterative Matchmaking (Swim) tournaments, using a dynamic matching strategy to capture the benefits of round-robin tournaments while maintaining computational efficiency.
Forward citations
Cited by 5 Pith papers
-
Generalized Priority-Aware Shapley Value
GPASV extends priority-aware Shapley values to arbitrary directed weighted graphs with penalization of violations and applies it to LLM ensemble valuation on the cyclic Chatbot Arena preference graph.
-
Rank, Don't Generate: Statement-level Ranking for Explainable Recommendation
The work reframes explainable recommendation as statement-level ranking, introduces the StaR benchmark from Amazon reviews, and finds popularity baselines outperforming SOTA models in item-level personalized ranking.
-
PIPBench: A Profile-Inclusive Framework for Personalized Image Generation Evaluation
PIPBench is a profile-inclusive benchmark with 1,369 test cases from 251 real users and synthetic agents that evaluates personalized image generation methods, revealing that current approaches struggle to jointly inte...
-
Rank, Don't Generate: Statement-level Ranking for Explainable Recommendation
Statement-level ranking with an LLM-extracted, paraphrase-clustered Amazon benchmark shows popularity baselines often beat SOTA models in item-level personalized explanation ranking.
-
Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs
A prompt perturbation approach builds comparison graphs from LLM judgments, filters inconsistent cycles or ties, and aggregates more reliable rankings.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.