REVIEW 34 cited by
A Long Way to Go: Investigating Length Correlations in RLHF
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Great success has been reported using Reinforcement Learning from Human Feedback (RLHF) to align large language models, with open preference datasets enabling wider experimentation, particularly for "helpfulness" in tasks like dialogue and web question answering. Alongside these improvements, however, RLHF also often drives models to produce longer outputs. This paper demonstrates, on three diverse settings, that optimizing for response length is, much more than previously thought, a significant factor behind RLHF. Studying the strategies RL optimization uses to maximize reward, we find improvements in reward to largely be driven by increasing response length, instead of other features. Indeed, we find that even a purely length-based reward reproduces most downstream RLHF improvements over supervised fine-tuned models. Testing a comprehensive set of length-countering interventions, we identify the dominant source of these biases to be reward models, which, by studying training dynamics, we find are non-robust and easily influenced by length biases in preference data.
Forward citations
Cited by 34 Pith papers
-
More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges
Self-play against reference-free LLM judges drives judge pass rates to 0.94 while true accuracy stays at 0.20, a reward-hacking basin that transfers across judge families and is prevented only by requiring the judge t...
-
Attention Limited Reward Learning
Limited attention makes pairwise preference labels non-identifiable for reward, can reverse Bradley-Terry rankings, and bounds learning by attended information rather than raw label count.
-
Preference learning made easy: Everything should be understood through win rate
Under two axioms (preference-consistency and prevalence-consistency), the only distribution-grounded evaluation for preference learning is h-win rate, and most popular alignment methods can be classified by whether th...
-
Copilot Arena: A Platform for Code LLM Evaluation in the Wild
An in-IDE pairwise-preference platform for code LLMs reveals that real developer choices rank models differently than static coding benchmarks.
-
Disentangling Length Bias In Preference Learning Via Response-Conditioned Modeling
Training reward models and DPO policies on response-conditioned preference pairs, where a length constraint is added to or withheld from the same prompt-response pair, reduces length bias and improves length instructi...
-
Style over Substance: A Shortcut Audit of Emotion-Description Preference Evaluation
A content-blind probe using only description length and generator identity matches finetuned 7B multimodal judges on EmoPrefer, showing benchmark scores are reachable without video grounding.
-
GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs
A dimension-corrected overlap statistic shows that transient excursions into pretrained principal subspaces precede LLM rollout-RL degradation, and a hard bilateral projection method (GCPO) that prevents such excursio...
-
RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists
RepoProbe uses GitHub Discussions and checklist-based verification to measure repository-level code understanding in LLMs, finding a persistent gap between fluent explanations and evidence-grounded correctness plus a ...
-
SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation
SMOPD trains one reward-specialized teacher per reward and merges them via token-level online distillation into a balanced student, reporting gains over GDPO on tool-calling format compliance and safety alignment.
-
Grounded Chess Reasoning in Language Models via Master Distillation
Master Distillation turns opaque chess-engine search into natural-language CoT, lifting a 4B model (C1) from near-zero to 48.1% accuracy that beats its teacher and most larger LMs.
-
Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling
BNRM learns rewards as sparse non-negative factor combinations and shows reduced length bias and better OOD alignment than standard BT reward models.
-
AALC: Large Language Model Efficient Reasoning via Adaptive Accuracy-Length Control
AALC, an accuracy-aware length reward, reduces reasoning-model response length by over 50% on math benchmarks while preserving or improving accuracy.
-
Med-U1: Incentivizing Unified Medical Reasoning in LLMs via Large-scale Reinforcement Learning
A pure reinforcement learning recipe with mixed rule-based rewards and length control improves Qwen2.5-based models across diverse medical QA formats.
-
Debiasing Online Preference Learning via Preference Feature Preservation
PFP preserves the distribution of human preference features during online preference learning by mapping each new instruction to a feature set, converting it to system prompts, and training with DPO.
-
Multi-Domain Explainability of Preferences
A concept-discovery plus hierarchical regression pipeline explains human, LLM-judge, and reward-model preferences at local and global levels across eight domains.
-
MPO: Multilingual Safety Alignment via Reward Gap Optimization
MPO reduces jailbreak success in multilingual LLMs by regressing target-language reward gaps onto the English reward gap, outperforming DPO and related methods while preserving utility.
-
Bias Fitting to Mitigate Length Bias of Reward Model in RLHF
FiMi-RM reduces length bias in RLHF reward models by fitting a nonlinear length-reward curve and training the reward model to be uncorrelated with that fitted curve.
-
Scalable Chain of Thoughts via Elastic Reasoning
Training reasoning models with a fixed two-part token budget, thinking plus solution, makes them robust to truncated thinking and more concise overall.
-
A RAG-Based Multi-Agent LLM System for Natural Hazard Resilience and Adaptation
WildfireGPT, a multi-agent RAG system with user profiling, outperforms ChatClimate and Perplexity AI in location-specific wildfire data analysis and evidence-based recommendations across ten expert case studies.
-
Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models
Forcing a self-rewarding LLM's generative judge and implicit DPO reward to agree on preference labels substantially improves its AlpacaEval 2.0 alignment score.
-
Online Preference Alignment for Language Models via Count-based Exploration
COPO adds a pseudo-count exploration bonus to DPO-style preference optimization and reports improved online RLHF results on Zephyr and Llama-3.
-
When Can Proxies Improve the Sample Complexity of Preference Learning?
Under four structural conditions linking proxy and true preference policies, the true policy is a low-dimensional adapter of the proxy policy, reducing the number of true preference samples needed.
-
UAlign: Leveraging Uncertainty Estimations for Factuality Alignment on Large Language Models
UAlign improves LLM factuality alignment by adding predicted confidence and semantic entropy as input features to prompts and the reward model, helping the model answer known questions and refuse unknown ones.
-
Self-Generated Critiques Boost Reward Modeling for Language Models
Self-generated, label-filtered critiques jointly trained with reward prediction improve preference accuracy and data efficiency without a stronger teacher model.
-
Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
A conceptual framework classifies human feedback to RL agents along nine dimensions and seven quality criteria, unifying human-centered, interface-centered, and model-centered design perspectives.
-
Establishing Reliability Metrics for Reward Models in Large Language Models
RETA measures reward model reliability as the average oracle quality of the top η fraction of ranked responses, divided by the average response quality.
-
Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment
Causal reward modeling applies MMD regularization to RLHF reward models to make reward scores statistically independent of spurious features, reducing measured length, sycophancy, concept, and demographic biases in ex...
-
Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models
Applying a utility-inspired, threshold-based transformation to individual rewards before summing them improved the harmlessness of an RLHF-trained 2B language model without reducing helpfulness.
-
CoT-based Synthesizer: Enhancing LLM Performance through Answer Synthesis
A small CoT-based synthesizer model trained on candidate-response analysis improves LLM reasoning accuracy, including cases where all sampled candidate answers are incorrect.
-
Interpreting Language Reward Models via Contrastive Explanations
Reward model preferences can be explained by generating counterfactual and semifactual answer variations along 15 hand-picked evaluation attributes and measuring which attribute changes flip the model's preference.
-
Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.
-
Meta-Thinking in LLMs via Multi-Agent Reinforcement Learning: A Survey
The paper surveys existing work on LLM meta-thinking and argues that multi-agent reinforcement learning is a promising missing ingredient for building self-correcting language models.
-
Open Problems in Machine Unlearning for AI Safety
Machine unlearning is not a comprehensive solution for AI safety; it is best suited to data removal, while capability control faces fundamental and unresolved challenges.
-
Reinforcement Learning from Human Feedback
The book introduces the origins, mathematical setup, and optimization stages of RLHF including reward modeling, reinforcement learning, rejection sampling, and direct alignment algorithms.
Discussion (0). Continue with ORCID to comment.