Pith. sign in

REVIEW 4 major objections 4 minor 23 cited by

Outcome-based RL for LLM reasoning degrades answer diversity even on training questions the model has never solved, via a transfer from solved to unsolved questions; UCB-style bonuses over final answers improve accuracy while mitigating the

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Outcome-based exploration bonuses (UCB-Con and Batch) improve pass@1 and pass@32 for LLM math reasoning while slowing diversity collapse, supported by a bandit model with a strong generalization assumption.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Useful empirical study of diversity collapse with modest but real gains; theory is conditional on an unmeasured learner-capability assumption. the 4 major comments →

arxiv 2509.06941 v1 pith:TRBKOHBT submitted 2025-09-08 cs.LG cs.CL

Outcome-based Exploration for LLM Reasoning

Yuda Song , Julia Kempe , Remi Munos This is my paper
classification cs.LG cs.CL MSC 68Q3262L05
keywords diversity collapseoutcome-based explorationLLM reasoningreinforcement learningUCBpass@kGRPOoutcome-based bandits
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the standard recipe for improving LLM math reasoning — reinforcement learning that rewards only the final answer — destroys generation diversity even before deployment, and that the damage can be repaired cheaply by exploring in the space of final answers rather than the space of reasoning traces. Treating RL training as a sampling process, the authors compare what an RL run has produced on the training questions against what the base model would produce with the same sampling budget, and find that RL ends up with fewer distinct answers, even on questions it has never solved. They call this the transfer of diversity degradation: once the model concentrates on the correct answers of solved questions, the collapse propagates to unsolved ones through the shared policy. Because verifiable reasoning tasks admit only a handful of distinct final answers, they propose outcome-based exploration — UCB-style bonuses attached to answers — and two concrete algorithms, UCB-Con and Batch, that improve test accuracy on competition benchmarks while preserving pass@k diversity. A companion bandit analysis shows that outcome-level exploration enjoys regret that scales with the number of answers, not the number of traces, provided training on one trace generalizes to other traces with the same answer.

Core claim

The paper's central finding is that diversity collapse under outcome-based RL is a training-set phenomenon with a cross-question mechanism: as training proceeds, the policy concentrates probability on correct answers of questions it has already solved, and because the policy is shared, this concentration transfers to unsolved questions, where the model now explores fewer distinct answers than the base model would at equal sampling cost. The paper's second finding is that the damage is addressable at the outcome level, because the final-answer space of a reasoning task is small — on average under 50 distinct answers per question. Adding the bonus b_ucb(x,a) = min(1, 1/sqrt(N(x,a))) to the GRP

What carries the argument

Four objects carry the argument. (1) RL-as-sampling: each epoch of n trajectories per question is treated as a sample draw, so the RL run's cumulative pass@k and diff@k curves can be overlaid on base-model sampling with k = nT epochs; this is what exposes the transfer of diversity degradation. (2) The outcome space: for verifiable tasks the final answer acts as a coarse-grained label of the reasoning trace, and the paper measures that per-question answer supports stay under roughly 50, making outcome-space exploration tractable. (3) The outcome-based bonus: b_ucb(x,a) = min(1, 1/sqrt(N(x,a))) over final answers, inserted into the GRPO advantage, with variants UCB-Mean (batch-mean baseline) a

Load-bearing premise

The load-bearing premise is Assumption A.3 (soft generalization): after seeing one reasoning trace that ends in a particular final answer, the model can route back to that answer and can remove a substantial fraction of all other traces ending in the same answer from future exploration — an assumed property of how the model generalizes that the paper does not measure; if that fraction is zero, the theoretical regret benefit disappears.

What would settle it

Hold out a set of training questions and freeze the policy's updates on them (zero gradient) while training normally on the rest; if the frozen copy's diff@k on those questions does not fall below the base model's at equal sampling budget, the claimed transfer of diversity degradation is not caused by cross-question propagation and the paper's mechanism is wrong. Separately, measure the soft-generalization fraction rho_o directly: train on a single trace ending in answer o and compare the probability lift on other traces ending in o against traces ending in other answers; if the lift is near z

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • UCB-Con and Batch both beat GRPO in pass@1 and pass@32 on MATH-500, AIME, and AMC23 with Llama-3.1-8B and Qwen-2.5-7B, and they mitigate the late-training overoptimization that makes vanilla RL's final checkpoint worse than its early one.
  • The exploration bonus attaches to final answers, so it is agnostic to the underlying RL algorithm and can be layered onto any verifiable-reward training pipeline.
  • Under soft generalization, outcome-level UCB converts a problem as hard as a K-armed bandit into one with O(sqrt(mT log T)) regret: the effective difficulty of RL reasoning post-training is set by the number of distinct answers, not the number of reasoning traces.
  • Batch exploration produces higher per-generation entropy on incorrect traces and more distinct answers per batch, making it the better choice when test-time pass@k at large k is the deployment goal.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the transfer mechanism is real, exploration effort should be concentrated on questions the policy has not solved — a direct test would apply outcome bonuses only below a per-question solve threshold and compare with the paper's global bonuses.
  • The soft-generalization fraction rho_o that Theorem A.4 depends on is measurable: train on a single trace with answer o and measure the probability lift on other traces with answer o versus traces with other answers; models with a larger measured lift should show proportionally larger gains from outcome-based exploration.
  • The bandit view suggests a data-curation lever the paper does not explore: because regret scales with the answer count m, rebalancing training data to keep per-question answer diversity high should improve RL sample efficiency independently of the bonus.
  • Batch exploration is defined purely at the batch level, so its penalty could be applied at inference time when sampling many answers for test-time scaling, without any retraining.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies diversity collapse in outcome-based RL for LLM reasoning. It frames RL post-training as a sampling process and reports that RL reduces effective diversity on the training set relative to sampling from the base model, including on unsolved questions (the 'transfer of diversity degradation'), and that final-answer spaces are sufficiently small to be tractable. It then proposes outcome-based exploration bonuses: historical UCB-style variants (UCB, UCB-Mean, UCB-Con) and a batch-level repetition penalty (Batch). Experiments with GRPO on Llama-3.1-8B-Instruct and Qwen-2.5-7B-Base on MATH/DAPO splits report improved pass@1 and pass@k over vanilla RL while mitigating diversity collapse. The paper also introduces an outcome-based bandit abstraction and proves a lower bound (Theorem A.1) and conditional upper bounds (Theorems A.3 and A.4).

Significance. The empirical core is plausible and useful: the proposed bonuses are simple, algorithm-agnostic, and the evaluation covers two models, several datasets, three seeds, and Appendix D error bars. The paper honestly acknowledges limitations (verifiable domains, single-turn tasks). The lower bound in Theorem A.1 is a clean negative result, and the empirical finding that RL can reduce training-set diversity on a per-sample basis is noteworthy. However, the formal upper-bound contribution is currently not established as written: the proof of Theorem A.4 has a gap, and the key generalization assumption is unmeasured and not implemented by the proposed algorithms. The 'transfer' explanation in Section 2.2 is also presented with causal language despite only correlational evidence. The paper's central empirical claims are defensible, but the theoretical and explanatory claims need substantial reworking.

major comments (4)
  1. [Appendix A.5, Theorem A.4 and Algorithm 3] The proof decomposes regret at the first hit of the optimal outcome tau_star and then applies Lemma A.4 to the tail. But Algorithm 3 does not enter outcome-level UCB until all m outcomes have been discovered; the tail from tau_star to tau_disc still consists of fresh draws from U, not pulls of representatives. Lemma A.4 does not apply to those rounds, and no bound on tau_disc is given under Assumption A.3. Concretely, if m=2, s_o* = K/2, and rho_o=0 for all o, then after hitting the optimal outcome the algorithm must still discover the rare outcome, which requires about K fresh draws and incurs about K*Delta regret, while the claimed bound is O(2 + sqrt(T log T)), independent of K. The reduction to an m-armed bandit therefore does not follow from the stated algorithm and proof. Additionally, the pseudocode does not remove the drawn arm A_t from U in the discovery phase, unlike Algorithms
  2. [Assumption A.3 and Section 3] Theorem A.4's improvement over the K-arm lower bound is entirely driven by Assumption A.3: the learner can route perfectly to a previously observed outcome and exclude a fraction rho_o of its outcome class from fresh probes. The paper itself calls this assumption 'rather unconventional' and notes it is an assumption on the learner. Yet no experiment measures rho_o, and the proposed UCB-Con and Batch algorithms (Section 3) do not implement routing or exclusion; they only add count-based bonuses. If rho_o=0, the first term becomes K/s_o*, recovering the K-arm regime. The main text's informal Theorem 4.2 is even stronger, relying on Assumption A.2 (strong generalization). The conclusion that outcome-based exploration is 'principled' is therefore conditional on an unverified capability that is not used by the proposed methods. The authors should either measure/justify rho_o from LLM data, or
  3. [Section 2.2, transfer of diversity degradation] The paper claims that reduced diversity on solved questions 'propagates' to unsolved questions, and calls this a central finding. The evidence is that RL training yields lower diff@k on unsolved questions than sampling from the base model. Since RL updates one shared network, unsolved-question behavior can change through many mechanisms unrelated to collapse on solved questions (e.g., general reasoning drift, reward hacking, format shift). No intervention isolates the proposed causal chain. The sentence 'The observed diversity degradation can therefore be explained as follows' overstates the evidence. Please reframe this as a hypothesis consistent with the observations, or add a controlled experiment (e.g., training only on solved questions and measuring unsolved-question diversity).
  4. [Table 3 and Table 4] Table 3 reports pass@1 and pass@32 'at the best checkpoint over three random seeds' without specifying how the best checkpoint is selected. If the selection is based on the test metrics being reported, the comparison is optimistic and can favor the method with more variance. The final-checkpoint results in Table 4 show materially smaller or different effects (e.g., UCB-Con on Qwen DAPO pass@32: 0.642 in Table 3 vs. 0.589 final; Batch final pass@32 is often the best). Please report a fixed protocol (e.g., validation-based checkpoint selection, or average over a fixed training schedule) and apply the same criterion to all methods.
minor comments (4)
  1. [Table 5] The caption of Table 5 appears to be copied from Table 2: it says 'Comparison of different exploration strategies based on the number of different answers sampled in a batch,' but the table lists hyperparameters. Please correct the caption.
  2. [Section 4.2 / Theorem 4.2] The main text states Theorem 4.2 under 'Assumption Assumption A.2' (strong generalization), while the more realistic soft model appears only in the appendix. Please make explicit in the main text which assumption the informal claim uses, since the paper's algorithms do not implement either assumption.
  3. [Figure 1 caption] The parenthetical 'except Qwen-2.5-7B-Base with UCB-Con on pass@1 on the early checkpoint due to exploration, but it has much higher pass@32 rate' is grammatically awkward and should be split into clearer sentences.
  4. [Section 2.2 notation] The notation A^{pi_RL}(x) and A^{pi_base}(x) is not formally defined before use; please define the answer-support notation in the preliminaries.

Circularity Check

0 steps flagged

No circularity: the paper's central empirical findings are self-contained, and its theoretical claim is an explicitly conditional result rather than a reduction to its own inputs.

full rationale

The paper's main contributions are empirical: it documents diversity collapse during RL training (Section 2.2, Figure 2) and proposes UCB-Con and Batch exploration, evaluating them against the GRPO baseline on external benchmarks (MATH-500, AIME, AMC) with Llama and Qwen models. These results stand on their own and are not constructed from the paper's own assumptions. The theoretical analysis in Appendix A is explicitly conditional: Theorem A.4 derives a regret bound under Assumption A.3 (soft generalization), which the paper itself labels 'rather unconventional' and 'an assumption on the learner instead of the environment' (Appendix A.5). This is a stated modeling assumption, not a hidden circular input, and the paper is transparent that if rho_o = 0 the bound degrades toward the lower bound of Theorem A.1. The absence of direct validation of Assumption A.3 is a limitation of the theory's applicability, not a circular derivation. Self-citations (Song et al. 2024b, Tang and Munos 2025, Arnal et al. 2025) are contextual and do not carry the load of the paper's main claims. No equation is reused as a prediction, no fitted parameter is renamed as a discovery, and no uniqueness theorem from prior work is imported to force the proposed methods. The only notable gap is that the theoretical justification is not directly connected to the exact implemented algorithms, but that is a scope limitation, not circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The paper's empirical contributions rest on domain assumptions about the verifiable-reward setting and on an unverified causal explanation for the transfer effect. The theoretical contribution (Section 4.2, Appendix A) rests primarily on Assumption A.3, a learner-capability assumption that is not independently grounded. No new physical or ontological entities are introduced.

free parameters (2)
  • UCB bonus coefficient c = 0.1 (Llama-3.1-8B), 0.2 (Qwen-2.5-7B)
    Scales the exploration bonus in Eq. (2); chosen per model without a described selection procedure; exploration strength directly affects the trade-off between training and test performance.
  • UCB-Con constant baseline b0 = 1.0 (easy dataset), 0.5 (medium dataset)
    Controls the sign and magnitude of the exploration signal in UCB-Con; tuned per dataset (Section E); the paper notes b0=0.5 means an answer visited fewer than 4 times gets a positive signal.
axioms (5)
  • domain assumption Rewards depend only on the final answer and are checked by math_verify equivalence (r(x,a) in {0,1}).
    This is the setting the algorithms target; it restricts the method to verifiable domains, which the paper acknowledges as a limitation in Section 6.
  • domain assumption GRPO objective (Eq. 1) is a reasonable proxy for the KL-regularized expected reward, despite known gradient bias.
    The paper cites Tang and Munos (2025) for the bias and uses GRPO as the 'vanilla RL' baseline; the 'RL as sampling' analysis assumes the training trajectories are comparable to i.i.d. base-model samples.
  • ad hoc to paper Assumption A.3 (soft generalization): after observing one arm with outcome o, the learner can route perfectly to o and exclude a fraction rho_o of the arms in the same outcome class.
    This is the load-bearing assumption for Theorem A.4, the main theoretical justification. It is stated as 'rather unconventional' by the authors and is not measured or validated in the LLM experiments.
  • ad hoc to paper The 'transfer of diversity degradation' explanation: reduced diversity on solved questions propagates to unsolved questions.
    The observed correlation between training progress and reduced unsolved-question diversity is interpreted as a causal transfer effect without an ablation; other mechanisms (shared weights, KL drift) are not ruled out (Section 2.2).
  • standard math Standard UCB regret bound for m-armed bandits with sub-Gaussian rewards (Lemma A.4).
    Cited from Auer et al. (2002); used to bound the post-discovery phase in Theorems A.2-A.4.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Outcome-based Exploration for LLM Reasoning." pith.science (2026). https://pith.science/paper/TRBKOHBT

@misc{pith2026250906941,
  author       = {Pith},
  title        = {Pith review of: Outcome-based Exploration for LLM Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TRBKOHBT}},
  note         = {Machine review of arXiv:2509.06941}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reinforcement learning (RL) has emerged as a powerful method for improving the reasoning abilities of large language models (LLMs). Outcome-based RL, which rewards policies solely for the correctness of the final answer, yields substantial accuracy gains but also induces a systematic loss in generation diversity. This collapse undermines real-world performance, where diversity is critical for test-time scaling. We analyze this phenomenon by viewing RL post-training as a sampling process and show that, strikingly, RL can reduce effective diversity even on the training set relative to the base model. Our study highlights two central findings: (i) a transfer of diversity degradation, where reduced diversity on solved problems propagates to unsolved ones, and (ii) the tractability of the outcome space, since reasoning tasks admit only a limited set of distinct answers. Motivated by these insights, we propose outcome-based exploration, which assigns exploration bonuses according to final outcomes. We introduce two complementary algorithms: historical exploration, which encourages rarely observed answers via UCB-style bonuses, and batch exploration, which penalizes within-batch repetition to promote test-time diversity. Experiments on standard competition math with Llama and Qwen models demonstrate that both methods improve accuracy while mitigating diversity collapse. On the theoretical side, we formalize the benefit of outcome-based exploration through a new model of outcome-based bandits. Together, these contributions chart a practical path toward RL methods that enhance reasoning without sacrificing the diversity essential for scalable deployment.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 23 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Bridging Compute- and Data-Optimal Pretraining

    cs.LG 2026-07 conditional novelty 7.0

    Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.

  2. Depth-Entropy Guided Sampling for Training-Free LLM Reasoning

    cs.LG 2026-06 conditional novelty 7.0

    Layer-wise entropy collapse depth is a weak per-candidate signal that, compounded inside MCMC power sampling, yields state-of-the-art training-free LLM reasoning accuracy.

  3. Sample Where You Struggle: Sharpening Base Model Reasoning via Entropy-Guided Power Sampling

    cs.LG 2026-06 unverdicted novelty 7.0

    EGPS localizes MCMC moves to high-entropy decision points using forward-pass entropy, yielding up to 12.6× wall-clock speedup and best-or-tied accuracy on MATH500, HumanEval, and GPQA for Qwen2.5-Math-7B.

  4. OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation

    cs.LG 2026-06 unverdicted novelty 7.0

    OrderGrad supplies unbiased likelihood-ratio and reparameterization gradient estimators for finite-sample L-statistics by applying a rank-based reward transformation usable in standard policy-gradient updates.

  5. Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verification

    cs.LG 2026-06 unverdicted novelty 7.0

    TTRL-CoCoV is a confidence-conditioned test-time RL framework that selectively applies verification to address pseudo-label errors and diversity collapse, yielding +9.8% Pass@1 and +18.7% Pass@16 gains over prior TTRL...

  6. Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR

    cs.LG 2026-05 unverdicted novelty 7.0

    RLRT augments GRPO by reinforcing tokens on correct student rollouts that the teacher would not have predicted, outperforming standard self-distillation and exploration baselines on Qwen3 models.

  7. Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity

    cs.LG 2026-05 unverdicted novelty 7.0

    UCPO modifies GRPO with a uniformity penalty over correct solutions to prevent diversity collapse in RLVR, yielding up to 10% higher Pass@64 on AIME24 and 45% more equation-level diversity.

  8. Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs

    cs.AI 2025-10 conditional novelty 7.0

    MENTOR selectively mixes expert and policy token distributions at high-entropy decision points, improving RLVR accuracy and diversity on math and out-of-domain benchmarks compared with full-trajectory imitation.

  9. Instruction-Conditioned Exploration with Asymmetric Reinforcement Learning and Self-Distillation

    cs.AI 2026-08 conditional novelty 6.0

    Adding behavioral strategy hints to RL training prompts, then distilling only correct hint-conditioned answers into the unconditioned model, lifts Qwen3-1.7B math pass@1 by 5.0% over DAPO.

  10. On Advantage Estimates for Max@K Policy Gradients

    cs.LG 2026-06 unverdicted novelty 6.0

    Proposes MaxPO using a Leave-Two-Out baseline for centered unbiased advantages in max@K policy gradients, with a unified derivation of finite-batch estimators.

  11. Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learning

    cs.LG 2026-06 unverdicted novelty 6.0

    Replaces scalar reward with a distribution over reward functions and applies a non-linear objective over action sets to induce controllable diversity in contextual bandit RL, generalizing policy gradient methods.

  12. Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVR

    cs.AI 2026-05 unverdicted novelty 6.0

    REFT improves Pass@1/8/64 in RLVR by uniform first-token sampling from top-N candidates across 0.5B-7B models and multiple difficulty levels.

  13. Beyond Mode Collapse: Distribution Matching for Diverse Reasoning

    cs.AI 2026-05 unverdicted novelty 6.0

    DMPO approximates forward KL minimization in on-policy RL by aligning the policy to a group-level reward-proportional target distribution, yielding 9-12% relative gains over GRPO on NP-Bench and smaller gains on math ...

  14. Automated Reformulation of Robust Optimization via Memory-Augmented Large Language Models

    cs.AI 2026-05 unverdicted novelty 6.0

    AutoREM augments LLMs with a structured memory of failed reformulation trajectories to improve accuracy and efficiency on robust optimization tasks without parameter updates or expert knowledge.

  15. It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches

    cs.AI 2026-05 conditional novelty 6.0

    W2SPO trains a reasoning LLM by inserting short 8-token branches proposed by a weaker model into its own rollouts and updating only those branch tokens, improving Pass@1 over vanilla GRPO (64.2% vs 62.3%) with a 3.55x...

  16. The Past Is Not Past: Memory-Enhanced Dynamic Reward Shaping

    cs.LG 2026-04 unverdicted novelty 6.0

    MEDS improves LLM RL performance by up to 4.13 pass@1 and 4.37 pass@128 points by dynamically penalizing rollouts matching prevalent historical error clusters identified via memory-stored representations and density c...

  17. Beyond the Sampled Token: Preserving Candidate Support in RLVR

    cs.AI 2025-10 conditional novelty 6.0

    SimKO, an asymmetric top-K token-level regularizer for RLVR, improves pass@K on math/logic benchmarks across K=1..256 without lowering pass@1 relative to GRPO.

  18. On the optimization dynamics of RLVR: Gradient gap and step size thresholds

    cs.LG 2025-10 unverdicted novelty 6.0

    The paper defines a Gradient Gap for RLVR policy gradients and proves a sharp step-size threshold below which training converges and above which it collapses, with predictions for length and success-rate scaling valid...

  19. Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision

    cs.LG 2025-09 unverdicted novelty 6.0

    Parallel inference rollouts aggregated into pseudo-references enable reference-free RL supervision that matches expert-annotated performance on health tasks while using 9x less test-time compute.

  20. When are likely answers right? On Sequence Probability and Correctness in LLMs

    stat.ML 2026-06 unverdicted novelty 5.0

    Higher sequence probability predicts correctness across different answers in a dataset but does not reliably improve accuracy when decoding methods or hyperparameters are changed, nor does it indicate correctness for ...

  21. Reasoning or Memorization? Direction-Aware Diversity Exploration in LLM Reinforcement Learning

    cs.AI 2026-06 unverdicted novelty 5.0

    DiRL extracts a reasoning-memorization direction from model representations inside GRPO to weight gradients and shape rewards so that exploration favors reasoning trajectories over memorization ones.

  22. Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective

    cs.LG 2025-10 unverdicted novelty 5.0

    Derives a token-level entropy change approximation revealing four factors, identifies limitations in prior entropy interventions, and proposes STEER which adaptively reweights tokens to mitigate collapse and improve p...

  23. Polychromic Objectives for Reinforcement Learning

    cs.LG 2025-09 unverdicted novelty 5.0

    Introduces polychromic objectives adapted into PPO via vine sampling and modified advantages, showing higher success rates and better coverage under perturbations on BabyAI, Minigrid, and algorithmic tasks.

Reference graph

Works this paper leans on

38 extracted references · 3 canonical work pages · cited by 23 Pith papers

  1. [1]

    2:fort= 1,2,

    We will bound the regret in theseT ′ rounds by a standard UCB bound (Lemma A.4): E h TX t=τdisc (µ⋆ −µ(O t)) τdisc i ≤O p m T′ logT 18 Algorithm 2Partition-Aware UCB under Strong Generalization 1: Initialize: R (set of discovered outcomes, initially∅); rep[o](representative arm for o, undefined until discovery); no ∈N,ˆµ o ∈R for o∈ R(both0initially); U (...

    work page 1996

  2. [6]

    Dataset reset policy optimization for rlhf.arXiv preprint arXiv:2404.08495,

    Jonathan D Chang, Wenhao Zhan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D Lee, and Wen Sun. Dataset reset policy optimization for rlhf.arXiv preprint arXiv:2404.08495,

    Pith/arXiv arXiv

  3. [7]

    Pass@ k training for adaptively balancing exploration and exploitation of large reasoning models.arXiv preprint arXiv:2508.10751,

    Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@ k training for adaptively balancing exploration and exploitation of large reasoning models.arXiv preprint arXiv:2508.10751,

    Pith/arXiv arXiv

  4. [8]

    Reasoning with exploration: An entropy perspective.arXiv preprint arXiv:2506.14758,

    Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective.arXiv preprint arXiv:2506.14758,

    Pith/arXiv arXiv

  5. [9]

    Weight ensembling improves reasoning in language models.arXiv preprint arXiv:2504.10478,

    Xingyu Dang, Christina Baek, Kaiyue Wen, Zico Kolter, and Aditi Raghunathan. Weight ensembling improves reasoning in language models.arXiv preprint arXiv:2504.10478,

    arXiv

  6. [10]

    The statistical complexity of interactive decision making.arXiv:2112.13487,

    Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making.arXiv:2112.13487,

    Pith/arXiv arXiv

  7. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  8. [13]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Pith/arXiv arXiv

  9. [14]

    Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Pith/arXiv arXiv

  10. [15]

    Understanding the effects of rlhf on llm generalisation and diversity.arXiv preprint arXiv:2310.06452,

    Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on llm generalisation and diversity.arXiv preprint arXiv:2310.06452,

    Pith/arXiv arXiv

  11. [17]

    Jointly reinforcing diversity and quality in language model generations.arXiv preprint arXiv:2509.02534,

    Tianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lanchantin, and Tianlu Wang. Jointly reinforcing diversity and quality in language model generations.arXiv preprint arXiv:2509.02534,

    Pith/arXiv arXiv

  12. [20]

    Approximating kl divergence, 2020.URL http://joschu

    John Schulman. Approximating kl divergence, 2020.URL http://joschu. net/blog/kl-approx. html,

    work page 2020

  13. [22]

    Can large reasoning models self-train?arXiv preprint arXiv:2505.21444,

    Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. Can large reasoning models self-train?arXiv preprint arXiv:2505.21444,

    arXiv

  14. [23]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  15. [24]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    Pith/arXiv arXiv

  16. [25]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    Pith/arXiv arXiv

  17. [26]

    The importance of online data: Understanding preference fine-tuning via coverage.Advances in Neural Information Processing Systems, 37:12243–12270, 2024a

    Yuda Song, Gokul Swamy, Aarti Singh, J Bagnell, and Wen Sun. The importance of online data: Understanding preference fine-tuning via coverage.Advances in Neural Information Processing Systems, 37:12243–12270, 2024a. Yuda Song, Hanlin Zhang, Carson Eisenach, Sham Kakade, Dean Foster, and Udaya Ghai. Mind the gap: Examining the self-improvement capabilities...

    Pith/arXiv arXiv

  18. [27]

    On a few pitfalls in kl divergence gradient estimation for rl.arXiv preprint arXiv:2506.09477,

    Yunhao Tang and Rémi Munos. On a few pitfalls in kl divergence gradient estimation for rl.arXiv preprint arXiv:2506.09477,

    Pith/arXiv arXiv

  19. [28]

    Optimizing language models for inference time objectives using reinforcement learning.arXiv preprint arXiv:2503.19595,

    Yunhao Tang, Kunhao Zheng, Gabriel Synnaeve, and Rémi Munos. Optimizing language models for inference time objectives using reinforcement learning.arXiv preprint arXiv:2503.19595,

    Pith/arXiv arXiv

  20. [29]

    The invisible leash: Why rlvr may not escape its origin.arXiv preprint arXiv:2507.14843,

    Fang Wu, Weihao Xuan, Ximing Lu, Zaid Harchaoui, and Yejin Choi. The invisible leash: Why rlvr may not escape its origin.arXiv preprint arXiv:2507.14843,

    arXiv

  21. [30]

    Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf.arXiv preprint arXiv:2405.21046,

    Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf.arXiv preprint arXiv:2405.21046,

    Pith/arXiv arXiv

  22. [31]

    Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint.arXiv preprint arXiv:2312.11456,

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint.arXiv preprint arXiv:2312.11456,

    Pith/arXiv arXiv

  23. [33]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Pith/arXiv arXiv

  24. [34]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837,

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837,

    Pith/arXiv arXiv

  25. [35]

    The price of format: Diversity collapse in llms.arXiv preprint arXiv:2505.18949,

    Longfei Yun, Chenyang An, Zilong Wang, Letian Peng, and Jingbo Shang. The price of format: Diversity collapse in llms.arXiv preprint arXiv:2505.18949,

    Pith/arXiv arXiv

  26. [36]

    Self-exploring language models: Active preference elicitation for online alignment.arXiv preprint arXiv:2405.19332, 2024a

    Shenao Zhang, Donghan Yu, Hiteshi Sharma, Han Zhong, Zhihan Liu, Ziyi Yang, Shuohang Wang, Hany Hassan, and Zhaoran Wang. Self-exploring language models: Active preference elicitation for online alignment.arXiv preprint arXiv:2405.19332, 2024a. Zihan Zhang, Yuxin Chen, Jason D Lee, and Simon S Du. Settling the sample complexity of online reinforcement lea...

    Pith/arXiv arXiv

  27. [37]

    First return, entropy-eliciting explore.arXiv preprint arXiv:2507.07017,

    Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, et al. First return, entropy-eliciting explore.arXiv preprint arXiv:2507.07017,

    Pith/arXiv arXiv

  28. [1996]

    Optimistically optimistic exploration for provably efficient infinite- horizon reinforcement and imitation learning.arXiv preprint arXiv:2502.13900,

    Antoine Moulin, Gergely Neu, and Luca Viano. Optimistically optimistic exploration for provably efficient infinite- horizon reinforcement and imitation learning.arXiv preprint arXiv:2502.13900,

    arXiv

  29. [2002]

    Exploration by random network distillation.arXiv preprint arXiv:1810.12894,

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation.arXiv preprint arXiv:1810.12894,

    Pith/arXiv arXiv

  30. [2016]

    Diverse preference optimization.arXiv preprint arXiv:2501.18101,

    Jack Lanchantin, Angelica Chen, Shehzaad Dhuliawala, Ping Yu, Jason Weston, Sainbayar Sukhbaatar, and Ilia Kulikov. Diverse preference optimization.arXiv preprint arXiv:2501.18101,

    Pith/arXiv arXiv

  31. [2018]

    Value-incentivized preference optimization: A unified approach to online and offline rlhf.arXiv preprint arXiv:2405.19320,

    Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, and Bo Dai. Value-incentivized preference optimization: A unified approach to online and offline rlhf.arXiv preprint arXiv:2405.19320,

    Pith/arXiv arXiv

  32. [2019]

    Qwen2.5 technical report.ArXiv, abs/2412.15115,

    14 Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji L...

    Pith/arXiv arXiv

  33. [2020]

    e3: Learning to explore enables extrapolation of test-time compute for llms.arXiv preprint arXiv:2506.09026,

    Amrith Setlur, Matthew YR Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Simchowitz, and Aviral Kumar. e3: Learning to explore enables extrapolation of test-time compute for llms.arXiv preprint arXiv:2506.09026,

    Pith/arXiv arXiv

  34. [2021]

    Navigate the unknown: Enhancing llm reasoning with intrinsic motivation guided exploration.arXiv preprint arXiv:2505.17621,

    Jingtong Gao, Ling Pan, Yejing Wang, Rui Zhong, Chi Lu, Qingpeng Cai, Peng Jiang, and Xiangyu Zhao. Navigate the unknown: Enhancing llm reasoning with intrinsic motivation guided exploration.arXiv preprint arXiv:2505.17621,

    arXiv

  35. [2022]

    Attributing mode collapse in the fine-tuning of large language models

    13 Laura O’Mahony, Leo Grinsztajn, Hailey Schoelkopf, and Stella Biderman. Attributing mode collapse in the fine-tuning of large language models. InICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, volume 2,

    work page 2024

  36. [2023]

    Asymmetric rein- force for off-policy reinforcement learning: Balancing positive and negative rewards.arXiv preprint arXiv:2506.20520,

    Charles Arnal, GaĂŤtan Narozniak, Vivien Cabannes, Yunhao Tang, Julia Kempe, and Remi Munos. Asymmetric rein- force for off-policy reinforcement learning: Balancing positive and negative rewards.arXiv preprint arXiv:2506.20520,

    arXiv

  37. [2024]

    Online preference alignment for language models via count-based exploration.arXiv preprint arXiv:2501.12735,

    Chenjia Bai, Yang Zhang, Shuang Qiu, Qiaosheng Zhang, Kang Xu, and Xuelong Li. Online preference alignment for language models via count-based exploration.arXiv preprint arXiv:2501.12735,

    Pith/arXiv arXiv

  38. [2025]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.