Pith. sign in

REVIEW 12 cited by

Reward Reasoning Model

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2505.14674 v1 pith:CH2JSDHC submitted 2025-05-20 cs.CL

Reward Reasoning Model

Jiaxin Guo , Zewen Chi , Li Dong , Qingxiu Dong , Xun Wu , Shaohan Huang , Furu Wei This is my paper
classification cs.CL
keywords rewardreasoningrrmsmodelscomputetest-timemodelperformance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reward models play a critical role in guiding large language models toward outputs that align with human expectations. However, an open challenge remains in effectively utilizing test-time compute to enhance reward model performance. In this work, we introduce Reward Reasoning Models (RRMs), which are specifically designed to execute a deliberate reasoning process before generating final rewards. Through chain-of-thought reasoning, RRMs leverage additional test-time compute for complex queries where appropriate rewards are not immediately apparent. To develop RRMs, we implement a reinforcement learning framework that fosters self-evolved reward reasoning capabilities without requiring explicit reasoning traces as training data. Experimental results demonstrate that RRMs achieve superior performance on reward modeling benchmarks across diverse domains. Notably, we show that RRMs can adaptively exploit test-time compute to further improve reward accuracy. The pretrained reward reasoning models are available at https://huggingface.co/Reward-Reasoning.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Counsel: A Meta-Evaluation Dataset for Agentic Tasks

    cs.AI 2026-06 unverdicted novelty 7.0

    Counsel is a new dataset of LLM-generated process critiques on agent benchmarks paired with human labels on error location and reasoning quality, achieving 0.78 Krippendorff alpha.

  2. TAPAS: Throughput-adaptive Perception for Autonomous Systems

    cs.LG 2026-07 conditional novelty 6.0

    An RL-based scheduler chooses CPU/GPU/DLA clusters for each perception model to meet scene-derived frame-rate targets, reporting large energy savings, though the frame-rate targets are defined by a hand-tuned entropy mapping.

  3. Biological Reasoning-Informed Regression for Interpretable Regulatory DNA Activity Prediction

    q-bio.GN 2026-06 unverdicted novelty 6.0

    R3LM trains LLMs via two-stage reasoning-then-regression on a new dataset CRE-ReasonBench with mechanistic traces, achieving SOTA enhancer activity prediction across three cell types with interpretable outputs.

  4. AI Can Learn Scientific Taste

    cs.CL 2026-03 conditional novelty 6.0

    Reinforcement learning on citation-preference pairs teaches a model to predict which papers will be cited more and to propose ideas that LLM judges rate as likely to be cited more—but "taste" here means citation impact.

  5. OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning

    cs.CL 2025-10 reject novelty 6.0

    A tool-augmented reward model trained with GRPO on 27K synthetic pairs beats existing reward models on long-form QA judgment and improves downstream alignment.

  6. PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling

    cs.LG 2025-10 unverdicted novelty 6.0

    PaTaRM converts pairwise preference data into pointwise reward signals via a novel PAR mechanism and task-adaptive rubrics, reporting 8.7% gains on RewardBench/RMBench and 13.6% relative RLHF improvement.

  7. Learning to Refine: Self-Refinement of Parallel Reasoning in LLMs

    cs.LG 2025-08 conditional novelty 6.0

    GSR jointly trains LLMs to generate candidate solutions and refine a superior final answer from them, achieving state-of-the-art performance on five mathematical benchmarks while transferring across model scales.

  8. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains

    cs.LG 2025-07 unverdicted novelty 6.0

    RaR uses aggregated rubric feedback as rewards in on-policy RL, delivering up to 31% relative gains on HealthBench and 7% on GPQA-Diamond versus direct Likert LLM-as-judge baselines.

  9. Trust Region On-Policy Distillation

    cs.LG 2026-05 unverdicted novelty 5.0

    TrOPD stabilizes on-policy distillation for LLMs with trust-region learning, outlier estimation, and off-policy guidance, outperforming prior OPD methods on reasoning and code benchmarks.

  10. VRPRM: Process Reward Modeling via Visual Reasoning

    cs.LG 2025-08 unverdicted novelty 5.0

    VRPRM combines visual reasoning with a two-stage SFT-plus-RL strategy to deliver higher-quality process reward modeling using far less annotated data than prior non-thinking PRMs.

  11. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

  12. A Survey of Reinforcement Learning for Large Reasoning Models

    cs.CL 2025-09 accept novelty 3.0

    A survey compiling RL methods, challenges, data resources, and applications for enhancing reasoning in large language models and large reasoning models since DeepSeek-R1.