LLMs collapse advice into a single supportive persona; Inverse-Process Distillation restores human-like persona diversity, yet raters still prefer the collapsed default.
Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards
9 Pith papers cite this work. Polarity classification is still indexing.
9
Pith papers citing it
abstract
Reinforcement learning with verifiable rewards (RLVR) has enabled large language models (LLMs) to achieve remarkable breakthroughs in reasoning tasks with objective ground-truth answers, such as mathematics and code generation. However, a significant gap remains for non-verifiable tasks, like creative writing and open-ended dialogue, where quality assessment is inherently subjective and lacks definitive references. Existing approaches for these domains often rely on scalar reward models trained with human preferences, which suffer from limited generalization and are prone to reward hacking, such as over-explanation and length bias. In this work, we propose a unified RLVR-based training paradigm that bridges the gap between non-verifiable tasks and verifiable rewards. We introduce a writing-principle-based pairwise Generative Reward Model (GenRM) and a novel Bootstrapped Relative Policy Optimization (BRPO) algorithm. The pairwise writing GenRM leverages self-principled critique to transform subjective assessments into reliable, verifiable rewards, while BRPO enables dynamic, reference-free pairwise comparison by leveraging a bootstrapped response as temporary reference from within group rollouts during RL training. Our approach empowers LLMs to develop robust writing capabilities without supervised fine-tuning, as demonstrated by Writing-Zero, which shows consistent improvement and strong resistance to reward hacking compared to scalar reward baselines. Furthermore, our method achieves competitive results on both in-house and open-source writing benchmarks. Our findings suggest the potential to unify rule-based, reference-based, and reference-free reward modeling under the RLVR framework, thus paving the way for a comprehensive and scalable RL training paradigm applicable across all language tasks.
years
2026 9representative citing papers
CapRL++ applies reinforcement learning with verifiable rewards to dense image and video captioning by scoring captions via the accuracy of a vision-free LLM answering MCQs from the caption alone.
PReMISE discovers and audits rubric sets for LLM judges, finding no existing source meets all reliability, preference-fit, and robustness criteria simultaneously while showing two repair methods improve accuracy and reduce exploitability.
StoryReward, trained on a new 100k story preference dataset, sets state-of-the-art performance on the introduced StoryRMB benchmark for aligning LLM stories with human preferences.
WEval and WRL introduce fine-grained benchmarking and requirement-selective sample construction for training writing reward models, yielding substantial gains on writing benchmarks with strong generalization.
Small LMs reach 77.1% accuracy at comparative forecasting of research idea success on benchmarks after supervised fine-tuning, with RLVR yielding interpretable reasoning at 71.35%.
CoNL lets LLMs self-improve on non-verifiable tasks by rewarding critiques that produce better solutions in multi-agent conversations, jointly optimizing generation and judging without external feedback.
OPERA uses perplexity dynamics as intrinsic rewards for RL alignment on open-ended tasks, synthesizes a 20k trajectory dataset via guiding words and log-prob rollouts, and sets new SOTA on Qwen3-8B matching some proprietary models.
citing papers explorer
-
Diagnosing and Repairing Persona Collapse in LLM Advice
LLMs collapse advice into a single supportive persona; Inverse-Process Distillation restores human-like persona diversity, yet raters still prefer the collapsed default.
-
CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioning
CapRL++ applies reinforcement learning with verifiable rewards to dense image and video captioning by scoring captions via the accuracy of a vision-free LLM answering MCQs from the caption alone.
-
PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges
PReMISE discovers and audits rubric sets for LLM judges, finding no existing source meets all reliability, preference-fit, and robustness criteria simultaneously while showing two repair methods improve accuracy and reduce exploitability.
-
StoryAlign: Evaluating and Training Reward Models for Story Generation
StoryReward, trained on a new 100k story preference dataset, sets state-of-the-art performance on the introduced StoryRMB benchmark for aligning LLM stories with human preferences.
-
From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks
WEval and WRL introduce fine-grained benchmarking and requirement-selective sample construction for training writing reward models, yielding substantial gains on writing benchmarks with strong generalization.
-
Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation
Small LMs reach 77.1% accuracy at comparative forecasting of research idea success on benchmarks after supervised fine-tuning, with RLVR yielding interpretable reasoning at 71.35%.
-
Conversation for Non-verifiable Learning: Self-Evolving LLMs through Meta-Evaluation
CoNL lets LLMs self-improve on non-verifiable tasks by rewarding critiques that produce better solutions in multi-agent conversations, jointly optimizing generation and judging without external feedback.
-
OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning
OPERA uses perplexity dynamics as intrinsic rewards for RL alignment on open-ended tasks, synthesizes a 20k trajectory dataset via guiding words and log-prob rollouts, and sets new SOTA on Qwen3-8B matching some proprietary models.
- Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning