REVIEW 14 cited by
What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Reinforcement learning (RL) is pivotal for enabling large language models (LLMs) to generate long chains of thought (CoT) for complex tasks like math and reasoning. However, Proximal Policy Optimization (PPO), effective in many RL scenarios, fails in long CoT tasks. This paper identifies that value initialization bias and reward signal decay are the root causes of PPO's failure. We propose Value-Calibrated PPO (VC-PPO) to address these issues. In VC-PPO, the value model is pretrained to tackle initialization bias, and the Generalized Advantage Estimation (GAE) computation is decoupled between the actor and critic to mitigate reward signal decay. Experiments on the American Invitational Mathematics Examination (AIME) show that VC-PPO significantly boosts PPO performance. Ablation studies show that techniques in VC-PPO are essential in enhancing PPO for long CoT tasks.
Forward citations
Cited by 14 Pith papers
-
Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles
Enigmata's synthetic puzzles with verifiable rewards lift a 32B model to 32.8% on ARC-AGI, above o3-mini-high and o1, and give small apparent gains on math and STEM when added to Seed1.5-Thinking.
-
CVPO: Enhancing LLM Reinforcement Learning Reasoning via Value-Variance Adaptation and Dynamic Curriculum Learning
CVPO adds value-variance weighting and dynamic curriculum weighting to value-based RL for LLM math reasoning, reporting gains over VAPO on five benchmarks.
-
Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization
Learning prefix-conditioned success probabilities from outcome labels yields more reliable process rewards than sequence log-ratio PRMs and improves RL when used for candidate-token TD updates.
-
Boosting RL-Based Visual Reasoning with Selective Adversarial Entropy Intervention
Selectively attacking the visual input with a gradient of response entropy during GRPO sampling increases policy entropy and improves visual reasoning accuracy by roughly 1–2 points.
-
Sticker-TTS: Learn to Utilize Historical Experience with a Sticker-driven Test-Time Scaling Framework
Sticker-TTS uses three collaborating models to distill and reuse compact summaries (stickers) of past reasoning attempts, improving math benchmark accuracy at a claimed equivalent inference cost.
-
PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier
A new multi-turn reinforcement learning framework trains a single LLM to both solve math problems and verify its own solutions, revising only when its verifier finds a mistake.
-
Learning from Environmental Feedback: Credit Assignment across Multiple Timescales for Agentic Reinforcement Learning
EFCA combines immediate environment feedback and recent state-history signals into a return reweighting that improves step-level credit assignment for LLM agents in ALFWorld and WebShop.
-
Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models
Cross-fitted residualization of recurring teacher–reference disagreement lets offline OPD work when the SFT generator and distillation teacher differ, beating uncorrected Lightning OPD on math and code.
-
CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents
CompactionRL trains LLM agents to generate context summaries during RL rollouts, enabling long-horizon task completion under fixed context budgets with consistent gains on SWE-bench Verified and Terminal-Bench 2.0.
-
Stabilizing Policy Optimization via Logits Convexity
LCO replaces PPO-style policy gradients with regression toward the advantage-derived optimal logits/policy, restoring logits-level convexity and yielding more stable LLM RL training.
-
Why Does Reasoning Length Converge? Unveiling the Underfitting-Overfitting Trade-off in Chain-of-Thought
LLM chain-of-thought length converges to an optimum because of an underfitting-overfitting tradeoff, formalized in a continuous reasoning-space framework and tested with RL.
-
MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization
MiroMind-M1 open-sources a two-stage SFT plus RLVR recipe with a new context-aware multi-stage policy optimization (CAMPO) that claims competitive AIME24, AIME25, and MATH500 scores among Qwen-2.5-based models.
-
QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning
QwenLong-L1-32B, trained with warm-up SFT plus curriculum-based long-context RL, reaches 70.7 average accuracy on seven DocQA benchmarks, though the evaluation gives it longer context than its proprietary baselines.
-
Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Models
A survey-style position paper claims that reinforcement fine-tuning powers reasoning in multimodal LLMs, summarizing over a hundred recent works and proposing five future research directions.
Discussion (0). Continue with ORCID to comment.