REVIEW 19 cited by
First Return, Entropy-Eliciting Explore
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Reinforcement Learning from Verifiable Rewards (RLVR) improves the reasoning abilities of Large Language Models (LLMs) but it struggles with unstable exploration. We propose FR3E (First Return, Entropy-Eliciting Explore), a structured exploration framework that identifies high-uncertainty decision points in reasoning trajectories and performs targeted rollouts to construct semantically grounded intermediate feedback. Our method provides targeted guidance without relying on dense supervision. Empirical results on mathematical reasoning benchmarks(AIME24) show that FR3E promotes more stable training, produces longer and more coherent responses, and increases the proportion of fully correct trajectories. These results highlight the framework's effectiveness in improving LLM reasoning through more robust and structured exploration.
Forward citations
Cited by 19 Pith papers
-
OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation
OrderGrad supplies unbiased likelihood-ratio and reparameterization gradient estimators for finite-sample L-statistics by applying a rank-based reward transformation usable in standard policy-gradient updates.
-
The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Credits
The cancellation hypothesis shows how rollout-level rewards produce token-level credit assignment in critic-free RL through cancellation of opposing signals on shared tokens, with empirical support and batching interv...
-
From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning
SFT supplies entangled compositional traces of atomic skills and routing modules; RL identifies those modules and enables recombination on novel compositions outside the SFT support.
-
Optimizing What Policies Learn From: Recoverability-aware Rollout Intervention Learning
RAIL learns rollout interventions from realized recoverability gains via an online contextual bandit and improves GRPO-based agentic post-training under limited rollout budgets.
-
What are Key Factors for Updates in RL for LLM Reasoning?
Theoretical analysis of RLVR update dynamics leads to ACPO, an adaptive clipping method that outperforms DAPO and CISPO on reasoning benchmarks with 3B and 7B models.
-
From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning
Introduces a hierarchical latent selection model showing SFT supplies raw module materials in compound traces while RL decomposes them to identify atomic modules and enable recombination for new reasoning configurations.
-
Confidence Laundering in Agent Systems: Why Uncertainty Needs a Latent Carrier
Agent systems lose uncertainty at decision handoffs, causing downstream over-trust; the paper proposes latent uncertainty as a carrier to preserve pre-commitment fragility across interfaces.
-
On Advantage Estimates for Max@K Policy Gradients
Proposes MaxPO using a Leave-Two-Out baseline for centered unbiased advantages in max@K policy gradients, with a unified derivation of finite-batch estimators.
-
Reasoning Can Be Restored by Correcting a Few Decision Tokens
Reasoning gaps between base LLMs and LRMs concentrate on ~8% of early planning tokens; intervening with the reasoning model only at high-disagreement positions recovers performance.
-
Selective Off-Policy Reference Tuning with Plan Guidance
SORT turns all-wrong prompts into selective learning signals by weighting tokens more predictable under plan guidance from reference solutions, improving over GRPO on reasoning benchmarks especially for weaker models.
-
Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts
HINT boosts LLM reasoning RL by injecting teacher-generated heuristic hints only on all-failed rollouts, keeping hints out of the policy-optimization prompt, and monitoring guidance quality with a new Affinity metric.
-
Outcome-based Exploration for LLM Reasoning
Outcome-based exploration bonuses (UCB-Con and Batch) improve pass@1 and pass@32 for LLM math reasoning while slowing diversity collapse, supported by a bandit model with a strong generalization assumption.
-
Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learning
A new RL reward-shaping method that rewards low confidence on hard problems and high confidence on easy ones improves LLM math reasoning over a GRPO baseline.
-
Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization
CPO uses the log-ratio of reference-guided to vanilla token probabilities as a correctness signal for per-token advantage shaping in RLVR, beating entropy-based methods on math and generalization benchmarks.
-
Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning
Survey mapping RL techniques onto LLM training and highlighting gaps in value-based, off-policy, and bootstrapping methods.
-
Trust Region On-Policy Distillation
TrOPD stabilizes on-policy distillation for LLMs with trust-region learning, outlier estimation, and off-policy guidance, outperforming prior OPD methods on reasoning and code benchmarks.
-
Selective Off-Policy Reference Tuning with Plan Guidance
SORT converts all-failed reasoning prompts into selective, structure-aware training signals by weighting tokens according to how much a reference-derived plan increases their probability.
-
How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors
IMAX trains soft prefixes with an InfoMax reward to drive diverse exploration in RLVR, yielding up to 11.60% gains in Pass@4 over standard RLVR across model scales.
-
Self-Reflective Generation at Test Time
SRGen improves LLM math reasoning by detecting high-entropy tokens and injecting a small corrected vector into the hidden state at those points during decoding, without training.
Discussion (0). Sign in to comment.