Pith. sign in

REVIEW 5 major objections 5 minor 11 cited by

Two intrinsic curiosity rewards—answer perplexity and value-head disagreement—improve RLVR for LLM math reasoning, adding about 3 points on AIME.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 18:44 UTC pith:RJI3Y26B

load-bearing objection Useful two-part exploration recipe, but the paper sells it as one method without ever running the two parts together; the +3 claim is not yet established. the 5 major comments →

arxiv 2509.09675 v1 pith:RJI3Y26B submitted 2025-09-11 cs.CL cs.AIcs.LG

CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models

Runpeng Dai , Linfeng Song , Haolin Liu , Zhenwen Liang , Dian Yu , Haitao Mi , Zhaopeng Tu , Rui Liu
show 3 more authors
Tong Zheng Hongtu Zhu Dong Yu
This is my paper
classification cs.CL cs.AIcs.LG
keywords curiosity-driven explorationRLVRlarge language modelsexploration bonusperplexitymulti-head criticcalibration collapsecount-based exploration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that the weak exploration of RLVR for LLMs can be fixed by rewarding the model for its own curiosity, in two complementary forms. The actor's curiosity is the perplexity of the response it just generated, added to the correctness reward with an adaptive clip. The critic's curiosity is the disagreement among several bootstrapped value-prediction heads, added to the advantage. The paper proves the actor bonus re-weights updates so confident errors are penalized and novel correct answers are encouraged, and that the critic bonus is asymptotically equivalent to classical count-based exploration in linear MDPs. Empirically, on the AIME24 and AIME25 benchmarks, the bonuses yield roughly a three-point average gain over standard GRPO and PPO, and larger Pass@16 gains.

Core claim

The paper's central claim is that two inexpensive, model-derived signals—perplexity of the actor's own answer and the standard deviation of a multi-head bootstrap critic—can serve as exploration bonuses that make RLVR training more effective. The actor bonus is defined as the negative log-likelihood per token of the generated response and is blended with the sparse correctness reward using a min-clip that guarantees the original reward sign is preserved. The critic bonus is the standard deviation of K value heads that share a backbone and are trained on resampled trajectory subsets; it enters the advantage estimate directly. The paper's main theoretical results are Theorem 3.1, showing that

What carries the argument

The mechanism is a pair of intrinsic rewards that are added to the RLVR objective without any new modules beyond a shared multi-head network. The actor signal, Equation (1), is the per-token perplexity of the sampled response; Equation (2) shapes it through an adaptive weight, a cap on the bonus relative to the original reward, and a scaling factor. The critic signal, Equation (5), is the standard deviation of K bootstrap value heads; Equation (4) folds it into the GAE advantage. The two theorems supply the identities that make these heuristics principled: Theorem 3.1 decomposes the policy update by correctness and perplexity, and Theorem 3.2 identifies the head std with the count-based expl

Load-bearing premise

The formal equivalence between the critic bonus and count-based exploration assumes the LLM's reasoning process is a linear MDP with known features, independent sub-Gaussian noise, and a converging empirical covariance; the paper does not verify these conditions for transformer chain-of-thought, so if value-head variance does not track true novelty in real training, Theorem 3.2 would not support the multi-head bonus.

What would settle it

Train the same RLVR setup with the actor perplexity bonus removed and compute, on held-out OOD prompts, the correlation between the multi-head critic std and an independent trajectory-visitation count (e.g., SimHash counts). Near-zero or negative correlation would show the critic bonus does not behave like count-based exploration in the real regime; additionally, if removing the actor bonus erases the AIME gains while keeping the critic bonus, the actor signal is the effective component.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • RLVR can gain a principled exploration signal without reward models, search trees, or explicit state-action features; both bonuses are computed from the model's own outputs.
  • The perplexity bonus counteracts entropy collapse: training curves show policy entropy declines more slowly, and the gap between confidence and correctness (calibration collapse) stays closed throughout training.
  • The multi-head bonus is inexpensive: 4–16 heads already capture most of the benefit, 2 heads barely help, and the result is insensitive to the bootstrap sub-sample fraction.
  • Bonus weight decay is necessary: a staircase schedule that keeps exploration strong early and removes it abruptly works best; keeping the bonus forever prevents convergence.
  • On hard competition problems (AIME24/25), Pass@16 gains of up to about 10 points over baselines suggest the method helps the model discover multiple valid routes instead of a single lucky one.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the critic bonus is a learned substitute for counts, it should generalize to other RL settings where explicit state counters are infeasible, such as open-ended embodiment or long-horizon agentic tasks, provided the value ensemble retains diversity.
  • If calibration collapse is a general RLVR failure, the same perplexity shaping could improve inference-time trust: for example, reordering self-consistent samples by perplexity may reduce hallucinated confident answers even without retraining.
  • A direct test of Theorem 3.2 in the transformer regime would be to compute the correlation between the multi-head std and a SimHash visitation count across training; the paper's own Figure 1 suggests hash counts are too coarse, so a positive correlation is not guaranteed.
  • The two bonuses are complementary: perceptual novelty from the actor and epistemic uncertainty from the critic target different failure modes, suggesting that a single combined curiosity score may transfer to multimodal or tool-using reasoning models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces CDE, a curiosity-driven exploration framework for RLVR with LLMs. Two intrinsic bonuses are proposed: an actor-side perplexity bonus on the generated response (Eq. 2) and a critic-side multi-head bootstrap variance bonus (Eqs. 4-5). The authors provide two theoretical results: Theorem 3.1 claims the PPL bonus calibrates policy confidence, promoting high-PPL correct responses and penalizing low-PPL incorrect ones; Theorem 3.2 claims the critic standard deviation is asymptotically equivalent to count-based exploration bonuses in linear MDPs. Experiments with Qwen3-4B on DAPO-17K, evaluated on MATH, AMC23, AIME24, and AIME25, report gains of roughly +2-3 points over GRPO/PPO baselines, and the paper also describes a 'calibration collapse' phenomenon in naive GRPO training.

Significance. If the claims are borne out, CDE would be an attractive lightweight exploration method for RLVR: it requires minimal architectural changes (a multi-head critic and a reward-shaping term) and is computationally cheap. The calibration analysis is interesting and potentially practically useful, and the attempt to connect the multi-head bonus to count-based exploration is a worthwhile theoretical contribution. The detailed ablations of bonus decay schedules and head counts are valuable. However, the central empirical claim for the combined CDE framework is not directly supported by the experiments as reported, and the lack of seed variance makes the quantitative gains difficult to assess. The theoretical results are plausible under their stated idealizations, but their applicability to LLM CoT training remains unverified.

major comments (5)
  1. [§4.2, Table 1 and Fig. 6] The paper's central claim is that CDE, combining the actor PPL bonus and the multi-head critic bonus, improves RLVR by ~+3 points. However, no experiment in Table 1 runs both bonuses together. The rows labeled 'w/ PPL bonus' use only Eq. (2), and the rows labeled 'w/ 2/4/8/16 Heads' use only Eqs. (4)-(5). Figure 6 likewise compares baselines to GRPO+PPL and to PPO+16 heads separately, never to a configuration with both bonuses. Appendix A, Table 4b, provides separate hyperparameter sets for the PPL and multi-head configurations, confirming there is no combined configuration. The framework-level claim is therefore not directly tested. Please add experiments with both bonuses active (and a full 2x2 ablation: baseline, PPL only, heads only, PPL+heads).
  2. [§4.2, Tables 1-3] All reported numbers come from a single training run with no seed variance or error bars. LLM RL training is known to be high-variance; a +2-3 point margin on AIME may be within run-to-run noise. Without multiple seeds (at least 3, ideally more) and a report of mean±std, the empirical superiority of CDE over baseline GRPO/PPO is not established. This is especially important because several comparisons are close (e.g., PPO+4 Heads vs. PPO on AIME24 Pass@16: 35.5 vs. 36.0). Please provide seed statistics and, where possible, a simple significance test.
  3. [Appendix D (proof of Theorem 3.1)] There is an inconsistency between the implemented bonus in Eq. (2) and the bonus used in the proof. Eq. (2) defines the bonus as min(|r|/κ, αB), while Appendix D defines b_t = ω min{κ|r|, -α/T log π_t} and then states that the sign is preserved when κ<1. These two definitions are reciprocal in κ. With the actual hyperparameter κ=3 (Table 4b), the implemented bonus is capped at |r|/3, so the sign of a negative reward is preserved; the proof's condition κ<1 would be violated, and if the proof's form were used with κ=3, the bonus could dominate and reverse the sign. The proof needs to be aligned with Eq. (2). Also, Theorem 3.1 is derived for a single-step KL-regularized closed-form update, not for the clipped PPO/GRPO objective actually used; this gap should be stated explicitly.
  4. [Appendix E, Theorem 3.2 / Theorem E.3] Theorem E.3 states convergence to β√(ϕ^T Λ^{-1} ϕ), but the proof derives the constant as √((1-ζ)/ζ)·σ. The statement's 'β is some constant' is therefore imprecise; the limiting factor depends on ζ and σ. More substantively, the result relies on Assumption E.1 and (A1)-(A2): a linear MDP with known features, i.i.d. sub-Gaussian noise, and a convergent empirical covariance. These assumptions are not checked for Qwen3-4B CoT reasoning, and the state-action space of long reasoning traces is unlikely to satisfy them in any direct sense. The empirical support in §4.4 (average B_critic decreasing) only shows that the bonus declines over training; it does not demonstrate that the bonus tracks pseudo-counts. Please temper the claim of 'consistency with count-based exploration' or provide evidence in a regime closer to the practical one.
  5. [§5.2 and §3.2] The paper argues that entropy bonuses are sample-agnostic and suboptimal, and that count-based methods are impractical, but it does not include any baseline experiment with an entropy bonus, RND, ICM, or SimHash count-based exploration. The preliminary SimHash results are presented only as motivation (Figure 1), not as a baseline in Table 1. Without such comparisons, the observed gains could be due simply to adding any reward bonus, and the paper's claim that CDE provides 'efficient exploration' relative to existing methods is not substantiated. Please add at least an entropy-bonus baseline and, if feasible, an RND-style or count-based baseline.
minor comments (5)
  1. [Abstract and §4.2] The abstract states 'approximate +3 point improvement' on AIME benchmarks, but Table 1 does not show a +3-point gain on any single metric in a consistent way. For example, GRPO+PPL improves Avg@16 on AIME24 by +2.5 and AIME25 by +2.5; PPO+16 heads by +2.7 and +2.5. Please report exactly which metric and comparison the +3 refers to, or revise the claim.
  2. [Table 1 and Section 4.2] The table's column headings are confusing: 'Avg@1', 'Avg@16', and 'Pass@16' are not formally defined in the text, and the 'Avg' column mixes Avg@1 for MATH with Avg@16 for the other benchmarks. Please define these metrics clearly in a table caption or footnote.
  3. [§4.2, first bullet] The text says the PPL bonus gives 'about +8 points' on AIME24 Pass@16; the table shows 48.5 vs. 41.9, i.e., +6.6. The later bullet says 'around +10 points' for multi-head PPO, but only the 8-head row shows a +10.4 gain (AIME24 Pass@16); AIME25 and other configurations are closer to +5. Please correct these overstatements.
  4. [Equation (4) and surrounding text] The notation for the advantage estimate is inconsistent: the overbrace labels the first term as '≃ Ã_{i,t}' while the left-hand side is p̂A_{i,t}. Please clarify the relationship between the two and avoid using the same symbol for different quantities.
  5. [Appendix D] In the proof of Theorem 3.1, the line 'Define r̃_t(q,o)=r(q,o)+b_t(q,o) where b_t(q,o)=ω min{κ |r|, -α/T log π_t}' appears to have a typo: the coefficient before κ should be ω/κ, not ωκ, to match Eq. (2). This also contributes to the sign-preservation error discussed above.

Circularity Check

0 steps flagged

No significant circularity: the PPL and multi-head-critic bonuses are derived from stated mechanisms (Eqs. 1–2, 4–5) and Theorem 3.2 follows from explicit linear-MDP assumptions; self-citations are not load-bearing.

full rationale

The central empirical claims are grounded in direct training comparisons, not in a fitted parameter renamed as a prediction. The PPL bonus (Eq. 2) is a function of the policy being trained, but the paper does not assume the conclusion that it improves exploration; instead Theorem 3.1 is a derived algebraic consequence of the bonus definition and the KL-regularized update, and the empirical calibration results in Fig. 9 are measured, not read off from the definition. The critic bonus (Eqs. 4–5) is not calibrated to AIME scores; Theorem 3.2/E.3 derives the equivalence of the bootstrap-head std to the count-based bonus from explicit linear-MDP, sub-Gaussian noise, and covariance-convergence assumptions (Assumption E.1, (A1)–(A2)), without importing the target result. Self-citations in the related work (e.g., Dai et al. 2025a,b; Zheng et al. 2025b; Zhao et al. 2025) are contextual and do not carry the load of the CDE derivation. The absence of a single run combining both bonuses is a completeness/evidence limitation, not a circularity; likewise single-seed reporting affects statistical strength, not the derivation logic.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The reported gains depend on hand-chosen hyperparameters (kappa=3, alpha=1 or 0.5, staircase/no-decay omega schedules, zeta=1 or 0.5, K heads) and both theorems rely on idealized assumptions not verified for LLM reasoning. The perplexity bonus is computed from the policy being trained, making the training objective self-referential, but the paper analyzes the resulting update rather than assuming the conclusion.

free parameters (5)
  • kappa (clipping ratio) = 3
    Controls the maximum bonus fraction relative to |r|; chosen by hand (Table 4b) and affects reward sign preservation.
  • alpha (bonus scaling factor) = 1 for PPL, 0.5 for multi-head
    Scales raw bonuses before clipping; chosen by hand and not derived.
  • omega_t (bonus weight schedule) = Staircase for GRPO-PPL; No decay for multi-head PPO
    Annealing schedule is a design choice; ablation shows it is necessary for convergence.
  • zeta (sub-sample fraction for critic heads) = 1 for 2/4 heads, 0.5 for 8/16 heads
    Controls bootstrap head diversity; chosen per configuration and ablated.
  • K (number of critic heads) = 2, 4, 8, 16
    Chosen; paper reports performance plateaus after K>=4.
axioms (4)
  • domain assumption Linear MDP: R(s,a)=phi(s,a)^T theta and P(s'|s,a)=phi(s,a)^T psi(s') with known feature phi (Assumption E.1).
    Used to prove Theorem 3.2; unlikely to hold literally for LLM CoT generation.
  • domain assumption Regression noise epsilon_i,h are i.i.d. sigma^2-sub-Gaussian with mean zero and the empirical covariance converges to a positive definite matrix (A1-A2).
    Needed for the bootstrap variance asymptotics in Appendix E.
  • ad hoc to paper Single-step KL-regularized policy update with closed-form solution (Appendix D).
    Theoretical model of the PPL bonus effect; actual GRPO/PPO use clipping and KL to reference policy, so Theorem 3.1 is illustrative rather than a proof of full training dynamics.
  • standard math Correctness reward r in {1,-1} and kappa<1 preserve the sign of the shaped reward.
    Used in the Theorem 3.1 proof to compare rewards of responses with the same correctness label.

pith-pipeline@v1.3.0-alltime-deepseek · 17721 in / 16675 out tokens · 164072 ms · 2026-08-04T18:44:47.457121+00:00 · methodology

0 comments
read the original abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for enhancing the reasoning ability of Large Language Models (LLMs). Yet current RLVR methods often explore poorly, leading to premature convergence and entropy collapse. To address this challenge, we introduce Curiosity-Driven Exploration (CDE), a framework that leverages the model's own intrinsic sense of curiosity to guide exploration. We formalize curiosity with signals from both the actor and the critic: for the actor, we use perplexity over its generated response, and for the critic, we use the variance of value estimates from a multi-head architecture. Both signals serve as an exploration bonus within the RLVR framework to guide the model. Our theoretical analysis shows that the actor-wise bonus inherently penalizes overconfident errors and promotes diversity among correct responses; moreover, we connect the critic-wise bonus to the well-established count-based exploration bonus in RL. Empirically, our method achieves an approximate +3 point improvement over standard RLVR using GRPO/PPO on AIME benchmarks. Further analysis identifies a calibration collapse mechanism within RLVR, shedding light on common LLM failure modes.

Figures

Figures reproduced from arXiv: 2509.09675 by Dian Yu, Dong Yu, Haitao Mi, Haolin Liu, Hongtu Zhu, Linfeng Song, Rui Liu, Runpeng Dai, Tong Zheng, Zhaopeng Tu, Zhenwen Liang.

Figure 1
Figure 1. Figure 1: Distribution of num￾ber of visitations across hash￾cells. In our preliminary experiments, we investigate the direct applica￾tion of count-based exploration methods to the RLVR setting. To avoid the computational burden of matrix inversion, we adopt the SimHash technique (Tang et al., 2017), which maps the embedding of a CoT response into a discrete hash code, and then uses the vis￾itation frequency of hash… view at source ↗
Figure 2
Figure 2. Figure 2: Responses by correctness [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: An illustration of the model’s policy distribution for se￾lecting the next token. Previous analyses distinguish the PPL bonus from the entropy bonus, which is sample-agnostic at the token level. The entropy at any given step depends solely on the policy’s probability dis￾tribution and is independent of the token ultimately sampled. Because the calculation considers the entire next-token distribu￾tion πθ pv… view at source ↗
Figure 4
Figure 4. Figure 4: An illustration of two posterior distributions of the critic and their bootstrap approximations. To approximate the posterior distribution of value estimates, we adopt the classical bootstrap method (Davison & Hinkley, 1997), widely used in statistics and increasingly recognized in the RL com￾munity as an effective tool for exploration (Osband et al., 2016; Ciosek et al., 2019; Bai et al., 2021). We implem… view at source ↗
Figure 5
Figure 5. Figure 5: Illustration of the multi-head critic framework. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of Avg@16 accuracy on AIME25 over training of vanilla GRPO and PPO [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: An illustration of differ￾ent weight anneal schedules. Bonus weight decay is crucial We compare four schedules for the bonus weight ωt—No decay, Linear, Cosine, and Staircase—as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Dynamics of policy entropy over the training process. The bonus weight decay mecha￾nism follows [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Average response PPL per training step, stratified by correctness. [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: (a) The average Bcritic over training steps. (b) Distribution of the standard deviation of value heads across prompts from different datasets. In sub-figure (b) of [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: The prompt for RLVR training. C Details on Hash-based pseudo count The core idea is to map a full prompt–response trajectory to a compact hash that serves as a pseudo￾state for exploration. Given a prompt–response pair pq, oq with tokenized sequences q “ tq1, . . . , qDu and o “ to1, . . . , oTu, let the model produce last-layer hidden states “ th1, . . . , hD`Tu, hi P Rd . We form a trajectory embedding … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning

    cs.LG 2026-06 unverdicted novelty 6.0

    ExTra adds embedding-based novelty rewards and entropy-guided prefix regeneration to GRPO, improving pass@1 by ~5 and pass@16 by ~7 on math benchmarks with Qwen3-1.7B.

  2. SALT: When More Rollouts Don't Help in Group-Based Policy Optimization and How to Make Them Matter

    cs.LG 2026-06 unverdicted novelty 6.0

    SALT is a subspace-adaptive plug-in for GRPO that decomposes group-relative coefficients into shared and residual channels using mini-batch Gram geometry and amplifies residuals to mitigate signed cancellation in RLVR.

  3. Epistemic Uncertainty for Test-Time Discovery

    cs.LG 2026-05 unverdicted novelty 6.0

    UG-TTT adds epistemic uncertainty measured by adapter disagreement as an exploration bonus in RL for LLMs, raising maximum reward and diversity on scientific discovery benchmarks.

  4. DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification

    cs.CL 2026-05 unverdicted novelty 6.0

    DeltaRubric decomposes multimodal preference evaluation into self-generated planning and verification steps within a single model, producing large accuracy improvements on VL-RewardBench via multi-role reinforcement learning.

  5. Reinforcing Multimodal Reasoning Against Visual Degradation

    cs.CV 2026-05 unverdicted novelty 6.0

    ROMA improves MLLM robustness to seen and unseen visual corruptions by +2.3-2.4% over GRPO on seven reasoning benchmarks while matching clean accuracy.

  6. Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoning

    cs.LG 2026-04 unverdicted novelty 6.0

    Kernel smoothing enables accurate low-variance value and gradient estimates for policy optimization in LLM reasoning under tight sampling constraints per prompt.

  7. Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoning

    cs.LG 2026-04 unverdicted novelty 6.0

    Kernel smoothing yields accurate value and gradient estimates for low-variance policy learning in LLM reasoning under tight per-prompt sampling budgets.

  8. Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data

    cs.LG 2026-04 unverdicted novelty 6.0

    A parameter-free sampling strategy called CUTS combined with Mixed-CUTS training prevents mode collapse in RL for saturated LLM reasoning tasks and raises AIME25 Pass@1 accuracy by up to 15.1% over standard GRPO.

  9. Calibration-Aware Policy Optimization for Reasoning LLMs

    cs.LG 2026-04 unverdicted novelty 6.0

    CAPO improves LLM calibration by up to 15% while matching or exceeding GRPO accuracy through logistic AUC loss and noise masking, enabling better abstention and scaling performance.

  10. StatEval: A Comprehensive Benchmark for Large Language Models in Statistics

    cs.CL 2025-10 conditional novelty 6.0

    StatEval is a new 16,000-question statistics benchmark showing that even strong LLMs score below 60% on research-level statistical proof tasks.

  11. Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning

    cs.LG 2026-06 unverdicted novelty 5.0

    Survey mapping RL techniques onto LLM training and highlighting gaps in value-based, off-policy, and bootstrapping methods.

Reference graph

Works this paper leans on

34 extracted references · 30 linked inside Pith · cited by 10 Pith papers

  1. [1]

    A survey of exploration methods in reinforcement learning.arXiv preprint arXiv:2109.00157,

    Susan Amin, Maziar Gomrokchi, Harsh Satija, Herke Van Hoof, and Doina Precup. A survey of exploration methods in reinforcement learning.arXiv preprint arXiv:2109.00157,

    Pith/arXiv arXiv

  2. [2]

    std ´␣ ϕJpwpkq n,h ˇˇ1ďkďK (¯ . Elliptical (“count-based

    19 Technical Report For any featureϕPR d, we define the bootstrap multi-head bonus as bboot h,K pϕq“std ´␣ ϕJpwpkq n,h ˇˇ1ďkďK (¯ . Elliptical (“count-based”) bonus in (Jin et al., 2020).The ridge estimator is constructed using all data acrossntrajectories as follows pwn,h“arg minw nÿ i“1 pGi,h´ϕJ i,hwq2`ζλ}w}

    2020

  3. [4]

    Qwen3-4B-Base-GRPO

    ConfigGRPO PPO actor-lr 1e-6 1e-6 critic-lr - 1e-5 critic-warmup - 10 kl_coef 0.0 0.0 max_prompt_length 2K 2K max_response_length 3K 3K train_batch_size 256 512 ppo_mini_batch_size 256 256 clip_ratio 0.20 0.20 sample temperature 1.0 1.0 rollout.n 8 4 total_training_steps 300 300 (a) ConfigPPL 2,4 Heads 8,16 Heads κ3 3 3 α1 0.5 0.5 ωt Staircase No decay No...

    2017

  4. [5]

    The entropy mechanism of reinforcement learning for reasoning language models.arXiv preprint arXiv:2505.22617,

    13 Technical Report Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models.arXiv preprint arXiv:2505.22617,

    Pith/arXiv arXiv

  5. [6]

    Breach in the shield: Unveiling the vulnera- bilities of large language models.arXiv preprint arXiv:2504.03714, 2025a

    Runpeng Dai, Run Yang, Fan Zhou, and Hongtu Zhu. Breach in the shield: Unveiling the vulnera- bilities of large language models.arXiv preprint arXiv:2504.03714, 2025a. Runpeng Dai, Tong Zheng, Run Yang, Kaixian Yu, and Hongtu Zhu. R1-re: Cross-domain relation extraction with rlvr.arXiv preprint arXiv:2507.04642, 2025b. Chris Dann, Yishay Mansour, Mehryar ...

    Pith/arXiv arXiv

  6. [10]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence.arXiv preprint arXiv:2401.14196,

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence.arXiv preprint arXiv:2401.14196,

    Pith/arXiv arXiv

  7. [13]

    R-zero: Self-evolving reasoning llm from zero data.arXiv preprint arXiv:2508.05004,

    Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-zero: Self-evolving reasoning llm from zero data.arXiv preprint arXiv:2508.05004,

    Pith/arXiv arXiv

  8. [14]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516,

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516,

    Pith/arXiv arXiv

  9. [17]

    Continuous control with deep reinforcement learning.arXiv preprint arXiv:1509.02971,

    Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning.arXiv preprint arXiv:1509.02971,

    Pith/arXiv arXiv

  10. [18]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022,

    Pith/arXiv arXiv

  11. [19]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  12. [20]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  13. [21]

    On entropy control in llm-rl algorithms.arXiv preprint arXiv:2509.03493,

    Han Shen. On entropy control in llm-rl algorithms.arXiv preprint arXiv:2509.03493,

    arXiv

  14. [22]

    Thermometer: Towards universal calibration for large language models.arXiv preprint arXiv:2403.08819,

    Maohao Shen, Subhro Das, Kristjan Greenewald, Prasanna Sattigeri, Gregory Wornell, and Soumya Ghosh. Thermometer: Towards universal calibration for large language models.arXiv preprint arXiv:2403.08819,

    Pith/arXiv arXiv

  15. [23]

    Solving math word problems with process-and outcome-based feedback.arXiv preprint arXiv:2211.14275,

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback.arXiv preprint arXiv:2211.14275,

    Pith/arXiv arXiv

  16. [25]

    Litesearch: Efficacious tree search for llm.arXiv preprint arXiv:2407.00320, 2024a

    Ante Wang, Linfeng Song, Ye Tian, Baolin Peng, Dian Yu, Haitao Mi, Jinsong Su, and Dong Yu. Litesearch: Efficacious tree search for llm.arXiv preprint arXiv:2407.00320, 2024a. Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan, and Bo An. Q*: Im- proving multi-step reasoning for llms with deliberative planning.arXiv preprint arXiv:...

    Pith/arXiv arXiv

  17. [26]

    Towards self-improvement of llms via mcts: Leveraging stepwise knowledge with curriculum preference learning.arXiv preprint arXiv:2410.06508, 2024c

    Xiyao Wang, Linfeng Song, Ye Tian, Dian Yu, Baolin Peng, Haitao Mi, Furong Huang, and Dong Yu. Towards self-improvement of llms via mcts: Leveraging stepwise knowledge with curriculum preference learning.arXiv preprint arXiv:2410.06508, 2024c. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. S...

    Pith/arXiv arXiv

  18. [28]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  19. [29]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Pith/arXiv arXiv

  20. [30]

    Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118,

    Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118,

    Pith/arXiv arXiv

  21. [31]

    One token to fool llm-as-a-judge

    Yulai Zhao, Haolin Liu, Dian Yu, SY Kung, Haitao Mi, and Dong Yu. One token to fool llm-as-a-judge. arXiv preprint arXiv:2507.08794,

    Pith/arXiv arXiv

  22. [32]

    Learning to reason via mixture-of-thought for logical reasoning.arXiv preprint arXiv:2505.15817, 2025a

    Tong Zheng, Lichang Chen, Simeng Han, R Thomas McCoy, and Heng Huang. Learning to reason via mixture-of-thought for logical reasoning.arXiv preprint arXiv:2505.15817, 2025a. Tong Zheng, Hongming Zhang, Wenhao Yu, Xiaoyang Wang, Xinyu Yang, Runpeng Dai, Rui Liu, Huiwen Bao, Chengsong Huang, Heng Huang, and Dong Yu. Parallel-r1: Towards parallel thinking vi...

    Pith/arXiv arXiv

  23. [1997]

    Alphazero-like tree-search can guide large language model decoding and training.arXiv preprint arXiv:2309.17179,

    Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training.arXiv preprint arXiv:2309.17179,

    Pith/arXiv arXiv

  24. [2010]

    Self-rewarding vision-language model via reasoning decomposition.arXiv preprint arXiv:2508.19652,

    Zongxia Li, Wenhao Yu, Chengsong Huang, Rui Liu, Zhenwen Liang, Fuxiao Liu, Jingxi Che, Dian Yu, Jordan Boyd-Graber, Haitao Mi, et al. Self-rewarding vision-language model via reasoning decomposition.arXiv preprint arXiv:2508.19652,

    Pith/arXiv arXiv

  25. [2016]

    Navigate the unknown: Enhancing llm reasoning with intrinsic motivation guided exploration.arXiv preprint arXiv:2505.17621,

    Jingtong Gao, Ling Pan, Yejing Wang, Rui Zhong, Chi Lu, Qingpeng Cai, Peng Jiang, and Xi- angyu Zhao. Navigate the unknown: Enhancing llm reasoning with intrinsic motivation guided exploration.arXiv preprint arXiv:2505.17621,

    arXiv

  26. [2017]

    Tulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124,

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124,

    Pith/arXiv arXiv

  27. [2018]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Pith/arXiv arXiv

  28. [2019]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv

  29. [2020]

    Guangzhi Xiong, Qiao Jin, Xiao Wang, Yin Fang, Haolin Liu, Yifan Yang, Fangyuan Chen, Zhixing Song, Dengyu Wang, Minjia Zhang, et al

    URLhttps://lilianweng.github.io/posts/2020-06-07-exploration-drl/. Guangzhi Xiong, Qiao Jin, Xiao Wang, Yin Fang, Haolin Liu, Yifan Yang, Fangyuan Chen, Zhixing Song, Dengyu Wang, Minjia Zhang, et al. Rag-gym: Optimizing reasoning and search agents with process supervision.arXiv preprint arXiv:2502.13957,

    Pith/arXiv arXiv 2020

  30. [2021]

    Online prefer- ence alignment for language models via count-based exploration.arXiv preprint arXiv:2501.12735,

    Chenjia Bai, Yang Zhang, Shuang Qiu, Qiaosheng Zhang, Kang Xu, and Xuelong Li. Online prefer- ence alignment for language models via count-based exploration.arXiv preprint arXiv:2501.12735,

    Pith/arXiv arXiv

  31. [2022]

    Calibrating large language models using their generations only.arXiv preprint arXiv:2403.05973,

    Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, and Seong Joon Oh. Calibrating large language models using their generations only.arXiv preprint arXiv:2403.05973,

    Pith/arXiv arXiv

  32. [2023]

    Deep think with confidence.arXiv preprint arXiv:2508.15260,

    Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao. Deep think with confidence.arXiv preprint arXiv:2508.15260,

    Pith/arXiv arXiv

  33. [2024]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  34. [2025]

    Exploration by random network distillation.arXiv preprint arXiv:1810.12894,

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation.arXiv preprint arXiv:1810.12894,

    Pith/arXiv arXiv