REVIEW 26 cited by
Verbosity Bias in Preference Labeling by Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
In recent years, Large Language Models (LLMs) have witnessed a remarkable surge in prevalence, altering the landscape of natural language processing and machine learning. One key factor in improving the performance of LLMs is alignment with humans achieved with Reinforcement Learning from Human Feedback (RLHF), as for many LLMs such as GPT-4, Bard, etc. In addition, recent studies are investigating the replacement of human feedback with feedback from other LLMs named Reinforcement Learning from AI Feedback (RLAIF). We examine the biases that come along with evaluating LLMs with other LLMs and take a closer look into verbosity bias -- a bias where LLMs sometimes prefer more verbose answers even if they have similar qualities. We see that in our problem setting, GPT-4 prefers longer answers more than humans. We also propose a metric to measure this bias.
Forward citations
Cited by 26 Pith papers
-
Does Multi-Agent Debate Improve AI Feedback on Research Papers?
Authors of economics meta-analyses found a single-pass AI report more useful than two multi-agent debate tools that cost up to thirty times more to run.
-
AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation
With prompt engineering (audio concatenation plus in-context examples), large audio models rank speech synthesis systems in line with human preferences, reaching up to 0.91 Spearman correlation.
-
JuStRank: Benchmarking LLM Judges for System Ranking
JuStRank ranks AI judges by how well their aggregated scores reproduce the Chatbot Arena human system ranking, revealing that judge realization and bias, not just model size, determine ranking quality.
-
Surfacing the Unsaid: CUE-Bench for Affective Stance in Chinese Discourse
CUE-Bench provides 51,823 Chinese discourse instances annotated with a nine-way Affective Stance defined by explicit-implicit polarity, plus pragmatic intent and fine-grained emotion labels.
-
Beyond a Single Judge: The Evidence-Grounded, Social-Weighted Persona Panel for Generative UI Evaluation
Evidence-grounded persona panels with bounded-confidence deliberation raise GenUI judge–human correlation from 0.716 to 0.922, mostly from persona grounding rather than multi-prompt averaging.
-
Test-Time Scaling via Error Localization
TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.
-
When Can You Debias an LLM Judge? Identifiability Limits, a Test, and Designs for Top-k Ranking
A bias-aware Bradley-Terry judge cannot identify the quality/bias split from comparisons alone; only prior assumptions, trusted anchors, or paired renderings can supply it.
-
Calibrating MLLM-as-a-judge via Multimodal Bayesian Prompt Ensembles
A cluster-conditioned Bayesian prompt ensemble improves the calibration and accuracy of multimodal LLM judges for text-to-image preference evaluation.
-
Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
A regression controlling for human-rated quality detects positive self- and family-bias in several LLM judges, including GPT-4o and Claude 3.5 Sonnet.
-
Evaluating the Use of LLMs for Documentation to Code Traceability
LLMs identify documentation-to-code trace links with F1 up to 80.4%, outperforming TF-IDF, BM25, and CodeBERT, but their explanations and chain reconstructions need human oversight.
-
Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models
MPO uses a meta reward model to continuously rewrite the reward model's evaluation prompt during PPO training, and the resulting models beat static-prompt RLAIF baselines on four tasks.
-
Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation
Pairwise LLM judgments flip in about 35% of cases when a stylistic distractor is added, versus only 9% for absolute scores, so pointwise scoring is more robust to manipulation.
-
Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering
Output-based LLM-as-a-judge methods using large LLMs achieve near-human correlation with human scores for code translation and generation, but not for code summarization or pairwise comparisons.
-
A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls
A rubric-plus-question-answering LLM framework for literary translation evaluation beats traditional MT metrics but still trails human agreement, especially on Korean honorifics.
-
Describe Now: User-Driven Audio Description for Blind and Low Vision Individuals
On-demand, user-activated AI audio descriptions give blind and low vision viewers control over timing and detail of video descriptions, but they increase cognitive load and are preferred more for instructional than en...
-
Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
A conceptual framework classifies human feedback to RL agents along nine dimensions and seven quality criteria, unifying human-centered, interface-centered, and model-centered design perspectives.
-
When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
Judge upgrades are not interchangeable: only Qwen3 1.7B→4B yields robust adjacent gains, MiniMax adjacent releases do not, and stronger judges reduce but do not remove bias or correlated jury errors.
-
SCOPE: Selective Conformal Optimized Pairwise LLM Judging
A conformal calibration method (SCOPE) plus a bidirectional entropy score (BPE) lets LLM pairwise judges abstain selectively while keeping accepted-set error below a user-specified bound.
-
Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective
The optimal reward for KL-regularized LLM alignment is a threshold function—reward B above a prompt-dependent cutoff, 0 below—which can be estimated from base-model samples and integrated into decoding-time alignment.
-
How Reliable is Multilingual LLM-as-a-Judge?
LLM-as-a-Judge is inconsistent across languages: five models grading the same parallel ground-truth content in 25 languages agreed poorly (Fleiss' Kappa around 0.2 on average), and the proposed majority-vote ensemble ...
-
From First Draft to Final Insight: A Multi-Agent Approach for Feedback Generation
A generate-evaluate-regenerate loop with GPT-4o raised rubric scores for feedback on 208 quiz responses, but the second-round evaluation was done by the same model that rewrote the feedback, and the abstract misreport...
-
Engineering AI Judge Systems
A constitution-based, search-driven development framework for AI judge systems improves judged accuracy by up to 6.2% on commit message generation, with about 58% of general principles reused across five languages.
-
Conversational AI for Rapid Scientific Prototyping: A Case Study on ESA's ELOPE Competition
One engineer paired with ChatGPT and reached second place in ESA's ELOPE competition in about one week of work; the paper draws best-practice lessons from that experience.
-
Exploring Modularity of Agentic Systems for Drug Discovery
On 26 chemistry questions, swapping the LLM, agent type, or prompt in an LLM agent changes its scores so much that the system cannot be treated as modular.
-
Potential and Perils of Large Language Models as Judges of Unstructured Textual Data
LLM judges show only fair-to-moderate agreement with human raters on thematic summary alignment and consistently over-rate alignment compared to humans.
-
A Systematic Review on the Evaluation of Large Language Models in Theory of Mind Tasks
A review of 58 papers finds that large language models pass some Theory of Mind tests but remain brittle and fall short of human performance.
Discussion (0). Continue with ORCID to comment.