REVIEW 27 cited by
Critique-out-Loud Reward Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Traditionally, reward models used for reinforcement learning from human feedback (RLHF) are trained to directly predict preference scores without leveraging the generation capabilities of the underlying large language model (LLM). This limits the capabilities of reward models as they must reason implicitly about the quality of a response, i.e., preference modeling must be performed in a single forward pass through the model. To enable reward models to reason explicitly about the quality of a response, we introduce Critique-out-Loud (CLoud) reward models. CLoud reward models operate by first generating a natural language critique of the assistant's response that is then used to predict a scalar reward for the quality of the response. We demonstrate the success of CLoud reward models for both Llama-3-8B and 70B base models: compared to classic reward models CLoud reward models improve pairwise preference classification accuracy on RewardBench by 4.65 and 5.84 percentage points for the 8B and 70B base models respectively. Furthermore, CLoud reward models lead to a Pareto improvement for win rate on ArenaHard when used as the scoring model for Best-of-N. Finally, we explore how to exploit the dynamic inference compute capabilities of CLoud reward models by performing self-consistency decoding for reward prediction.
Forward citations
Cited by 27 Pith papers
-
Learning Latent Reasoning Traces for Scalar Reward Models End-to-End
LatentRM treats reasoning traces as discrete latent variables and trains a generator end-to-end so that the scalar reward model's likelihood of the true preference ranking is maximized, outperforming scalar, generativ...
-
LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks
Training a language model by distilling a coach's written experiential knowledge beats training on a scalar rubric score for open-ended tasks, with better out-of-distribution transfer.
-
RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback
RefCritic uses reinforcement learning with two rule-based rewards, judgment accuracy and refinement accuracy, to train long-chain-of-thought critic models that improve LLM math reasoning and detect erroneous steps.
-
Breaking the Myth: Can Small Models Infer Postconditions Too?
Fine-tuning Qwen2.5-Coder-7B on 1.5K reasoning examples yields postcondition generation close to GPT-4o and better than 32B open models on Defects4J.
-
Unlocking Recursive Thinking of LLMs: Alignment via Refinement
An offline alignment pipeline using reward-filtered self-refinement data and long chain-of-thought SFT raises an 8B model's AlpacaEval 2 win rate from 25.0% to 51.0% with roughly 14k training examples.
-
RewardAnything: Generalizable Principle-Following Reward Models
RewardAnything follows natural-language reward principles at inference time and, with the new RABench benchmark, demonstrates that principle-conditioned listwise training beats fixed-preference reward models on held-o...
-
Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability
DataRubrics introduces a structured ten-dimension rubric with an LLM-as-a-judge pipeline to automatically assess dataset quality, positioned as a more measurable alternative to descriptive datasheets for conference review.
-
Socratic-PRMBench: Benchmarking Process Reward Models with Systematic Reasoning Patterns
Socratic-PRMBench tests process reward models on deliberately flawed math reasoning across six reasoning patterns and reports that current PRMs score below LLM critics.
-
Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectiveness
A systematic benchmark shows multi-agent debate's value depends on task difficulty, model scale, and agent diversity: limited for math unless problems are hard or models weak, but useful for safety when agents are diverse.
-
Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
A generative reward model trained with long chain-of-thought and rule-based RL outperforms standard and vertically scaled reward baselines on RM-Bench and RewardBench.
-
Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization
Reward-model benchmarks become more predictive when they use multiple diverse, distribution-matched responses, and overoptimization should be a diagnostic tool, not a benchmark target.
-
J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judge
J1-7B, a judge LLM trained with supervised fine-tuning and reinforcement learning, improves when forced to reflect with 'wait' tokens, and the scaling ability emerges during the RL phase.
-
AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset
A three-part recipe, built from a 540K-problem open dataset, long reasoning distillations, tool-integrated reasoning, and GenSelect, produces open-weight models that top most math-reasoning baselines and win AIMO-2.
-
LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Information
LongDPO uses MCTS-collected, critique-refined stepwise preference pairs with step-level DPO to improve long-form generation quality and length adherence.
-
Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
A controlled, multi-family study shows the relative generation-verification gap grows with pretraining flops for stable verification methods, and iterative self-improvement saturates quickly.
-
Self-Generated Critiques Boost Reward Modeling for Language Models
Self-generated, label-filtered critiques jointly trained with reward prediction improve preference accuracy and data efficiency without a stronger teacher model.
-
Large Reasoning Models are not thinking straight: on the unreliability of thinking trajectories
Large reasoning models frequently ignore explicitly injected ground-truth solutions mid-trajectory and continue to overthink, often ending on incorrect answers.
-
ReasonGRM: Enhancing Generative Reward Models through Large Reasoning Models
ReasonGRM uses a likelihood-based metric R* to select correct, high-confidence reasoning paths for supervised fine-tuning and then fine-tunes on hard cases with GRPO, reaching an average score of 83.3 across RewardBen...
-
GFRIEND: Generative Few-shot Reward Inference through EfficieNt DPO
GFRIEND generates chain-of-thought preference judgments, scores them by perplexity, and uses weighted multi-level preference optimization so a reward model trained on 3,000 samples rivals models trained on much larger...
-
CyberV: Cybernetics for Test-time Scaling in Video Understanding
A training-free test-time feedback loop, using attention drift to select key frames, improves video MLLM accuracy, with the largest gains on knowledge-heavy VideoMMMU.
-
Generative RLHF-V: Learning Principles from Multi-modal Human Preference
A reinforcement-learned multimodal judge with grouped pairwise scoring improves vision-language model alignment on seven benchmarks.
-
Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
A trained critique model providing step-level feedback improves an LLM actor's mathematical reasoning at test time and during self-improvement, with gains concentrated on hard problems.
-
Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.
-
Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models
This survey organizes LLM synthetic data research around quality, diversity, and complexity, claiming quality mainly helps in-distribution generalization, diversity mainly helps out-of-distribution generalization, and...
-
Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
A single-author survey that taxonomizes reward-model design and learning strategies across training, inference, and post-inference for LLMs.
-
Reinforcement Learning from Human Feedback
The book introduces the origins, mathematical setup, and optimization stages of RLHF including reward modeling, reinforcement learning, rejection sampling, and direct alignment algorithms.
-
A Survey on Large Language Models for Mathematical Reasoning
Recent advances in LLM mathematical reasoning are organized into comprehension and generation phases, covering methods from prompting to test-time scaling.
Discussion (0). Continue with ORCID to comment.