{"id":"943f4837-64e0-45c8-a939-d6a0b5e2d3b7","arxiv_id":"2509.09675","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Adding actor perplexity and multi-head critic variance as intrinsic exploration bonuses improves RLVR math reasoning accuracy by roughly +2 to +3 points on AIME benchmarks.","lead":"This paper adds a curiosity bonus to reinforcement learning for math reasoning: a perplexity bonus from the model's own output and a variance bonus from a multi-head critic. These bonuses, tested on a 4B model, improve accuracy on AIME-style math benchmarks by a few points over standard GRPO and PPO.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Full CDE (PPL bonus + multi-head critic) is never tested as a combined method; all reported gains are single-seed, so the +3 central claim is not yet established.","rationale":"The reader's weakest assumption targets Theorem 3.2's applicability; I agree that is a real limitation, but it is not the most load-bearing because the empirical accuracy gain does not depend on the theorem. The most direct vulnerability is that the integrated CDE algorithm—the thing the title and abstract announce—is never run. Table 1 only reports component ablations, and the appendix lists them as separate configurations. Combined with single-seed reporting, the headline '+3 points' could stem from a single lucky run of one component, and even if the components are individually real, their interaction is unknown. A concrete multi-seed full-CDE run would settle this. This concern strengthens the reader's CONDITIONAL verdict rather than overturning it.","tokens_in":17908,"tokens_out":15013,"duration_ms":163671,"concrete_test":"Run the full CDE configuration with both bonuses simultaneously—Eq. 2 for the actor and Eqs. 4–5 with K=16 heads for the critic—under the Table 4 hyperparameters, for at least 3 independent seeds. In the same runs, also train baseline GRPO, baseline PPO, PPL-only, and 16-head-only with the same seeds and evaluation protocol. Report mean and standard deviation of AIME24/AIME25 Pass@16 (and Avg@16). If the combined CDE does not beat both single-bonus variants, or if its margin over baseline is within one standard deviation, the central +3-point framework claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that CDE—the framework with both the actor PPL bonus (Eq. 2) and the multi-head critic variance bonus (Eqs. 4–5)—improves RLVR by ~+3 points on AIME. But no experiment in Table 1 runs both bonuses together. The rows labeled 'w/ PPL bonus' (GRPO and PPO) use only Eq. 2; the rows 'w/ 2/4/8/16 Heads' use only Eqs. 4–5. Figure 6 likewise compares baselines to GRPO+PPL and to PPO+16 heads separately, never to their conjunction. Appendix A gives separate hyperparameter sets (Table 4b) for PPL and for heads, confirming there is no combined configuration. The central claim as a framework-level statement is therefore not directly supported. In addition, every number in Tables 1–3 comes from a single run with no seed variance; LLM RL training is known to be high-variance, so the +2–3 point margins may be within run-to-run noise. The theory in Theorem 3.2 also assumes a linear MDP with known features and i.i.d. sub-Gaussian noise, assumptions that are unverified for 4B-parameter CoT reasoning, but that concern is secondary: even if the theory were accepted, the empirical case for the combined CDE method has not been made.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CDE, a curiosity-driven exploration framework for RLVR with LLMs. Two intrinsic bonuses are proposed: an actor-side perplexity bonus on the generated response (Eq. 2) and a critic-side multi-head bootstrap variance bonus (Eqs. 4-5). The authors provide two theoretical results: Theorem 3.1 claims the PPL bonus calibrates policy confidence, promoting high-PPL correct responses and penalizing low-PPL incorrect ones; Theorem 3.2 claims the critic standard deviation is asymptotically equivalent to count-based exploration bonuses in linear MDPs. Experiments with Qwen3-4B on DAPO-17K, evaluated on MATH, AMC23, AIME24, and AIME25, report gains of roughly +2-3 points over GRPO/PPO baselines, and the paper also describes a 'calibration collapse' phenomenon in naive GRPO training.","tokens_in":18301,"tokens_out":8664,"duration_ms":103543,"significance":"If the claims are borne out, CDE would be an attractive lightweight exploration method for RLVR: it requires minimal architectural changes (a multi-head critic and a reward-shaping term) and is computationally cheap. The calibration analysis is interesting and potentially practically useful, and the attempt to connect the multi-head bonus to count-based exploration is a worthwhile theoretical contribution. The detailed ablations of bonus decay schedules and head counts are valuable. However, the central empirical claim for the combined CDE framework is not directly supported by the experiments as reported, and the lack of seed variance makes the quantitative gains difficult to assess. The theoretical results are plausible under their stated idealizations, but their applicability to LLM CoT training remains unverified.","major_comments":[{"comment":"The paper's central claim is that CDE, combining the actor PPL bonus and the multi-head critic bonus, improves RLVR by ~+3 points. However, no experiment in Table 1 runs both bonuses together. The rows labeled 'w/ PPL bonus' use only Eq. (2), and the rows labeled 'w/ 2/4/8/16 Heads' use only Eqs. (4)-(5). Figure 6 likewise compares baselines to GRPO+PPL and to PPO+16 heads separately, never to a configuration with both bonuses. Appendix A, Table 4b, provides separate hyperparameter sets for the PPL and multi-head configurations, confirming there is no combined configuration. The framework-level claim is therefore not directly tested. Please add experiments with both bonuses active (and a full 2x2 ablation: baseline, PPL only, heads only, PPL+heads).","section":"§4.2, Table 1 and Fig. 6"},{"comment":"All reported numbers come from a single training run with no seed variance or error bars. LLM RL training is known to be high-variance; a +2-3 point margin on AIME may be within run-to-run noise. Without multiple seeds (at least 3, ideally more) and a report of mean±std, the empirical superiority of CDE over baseline GRPO/PPO is not established. This is especially important because several comparisons are close (e.g., PPO+4 Heads vs. PPO on AIME24 Pass@16: 35.5 vs. 36.0). Please provide seed statistics and, where possible, a simple significance test.","section":"§4.2, Tables 1-3"},{"comment":"There is an inconsistency between the implemented bonus in Eq. (2) and the bonus used in the proof. Eq. (2) defines the bonus as min(|r|/κ, αB), while Appendix D defines b_t = ω min{κ|r|, -α/T log π_t} and then states that the sign is preserved when κ<1. These two definitions are reciprocal in κ. With the actual hyperparameter κ=3 (Table 4b), the implemented bonus is capped at |r|/3, so the sign of a negative reward is preserved; the proof's condition κ<1 would be violated, and if the proof's form were used with κ=3, the bonus could dominate and reverse the sign. The proof needs to be aligned with Eq. (2). Also, Theorem 3.1 is derived for a single-step KL-regularized closed-form update, not for the clipped PPO/GRPO objective actually used; this gap should be stated explicitly.","section":"Appendix D (proof of Theorem 3.1)"},{"comment":"Theorem E.3 states convergence to β√(ϕ^T Λ^{-1} ϕ), but the proof derives the constant as √((1-ζ)/ζ)·σ. The statement's 'β is some constant' is therefore imprecise; the limiting factor depends on ζ and σ. More substantively, the result relies on Assumption E.1 and (A1)-(A2): a linear MDP with known features, i.i.d. sub-Gaussian noise, and a convergent empirical covariance. These assumptions are not checked for Qwen3-4B CoT reasoning, and the state-action space of long reasoning traces is unlikely to satisfy them in any direct sense. The empirical support in §4.4 (average B_critic decreasing) only shows that the bonus declines over training; it does not demonstrate that the bonus tracks pseudo-counts. Please temper the claim of 'consistency with count-based exploration' or provide evidence in a regime closer to the practical one.","section":"Appendix E, Theorem 3.2 / Theorem E.3"},{"comment":"The paper argues that entropy bonuses are sample-agnostic and suboptimal, and that count-based methods are impractical, but it does not include any baseline experiment with an entropy bonus, RND, ICM, or SimHash count-based exploration. The preliminary SimHash results are presented only as motivation (Figure 1), not as a baseline in Table 1. Without such comparisons, the observed gains could be due simply to adding any reward bonus, and the paper's claim that CDE provides 'efficient exploration' relative to existing methods is not substantiated. Please add at least an entropy-bonus baseline and, if feasible, an RND-style or count-based baseline.","section":"§5.2 and §3.2"}],"minor_comments":[{"comment":"The abstract states 'approximate +3 point improvement' on AIME benchmarks, but Table 1 does not show a +3-point gain on any single metric in a consistent way. For example, GRPO+PPL improves Avg@16 on AIME24 by +2.5 and AIME25 by +2.5; PPO+16 heads by +2.7 and +2.5. Please report exactly which metric and comparison the +3 refers to, or revise the claim.","section":"Abstract and §4.2"},{"comment":"The table's column headings are confusing: 'Avg@1', 'Avg@16', and 'Pass@16' are not formally defined in the text, and the 'Avg' column mixes Avg@1 for MATH with Avg@16 for the other benchmarks. Please define these metrics clearly in a table caption or footnote.","section":"Table 1 and Section 4.2"},{"comment":"The text says the PPL bonus gives 'about +8 points' on AIME24 Pass@16; the table shows 48.5 vs. 41.9, i.e., +6.6. The later bullet says 'around +10 points' for multi-head PPO, but only the 8-head row shows a +10.4 gain (AIME24 Pass@16); AIME25 and other configurations are closer to +5. Please correct these overstatements.","section":"§4.2, first bullet"},{"comment":"The notation for the advantage estimate is inconsistent: the overbrace labels the first term as '≃ Ã_{i,t}' while the left-hand side is p̂A_{i,t}. Please clarify the relationship between the two and avoid using the same symbol for different quantities.","section":"Equation (4) and surrounding text"},{"comment":"In the proof of Theorem 3.1, the line 'Define r̃_t(q,o)=r(q,o)+b_t(q,o) where b_t(q,o)=ω min{κ |r|, -α/T log π_t}' appears to have a typo: the coefficient before κ should be ω/κ, not ωκ, to match Eq. (2). This also contributes to the sign-preservation error discussed above.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a promising and simple method, and the theoretical results, once the sign/notation issues are fixed, are plausible. The main blocker is the absence of an experiment running the combined CDE method and the lack of seed variance in all reported numbers. If the authors can supply those, the paper would be in much better shape. The abstract's '+3' claim should be aligned with the actual table values. I would not reject, but the current evidence is insufficient for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has two plausible exploration ingredients for RLVR — an annealed, clipped perplexity bonus on the actor and a multi-head critic variance bonus — and a nice diagnostic (calibration collapse). But the central empirical claim, that CDE as a framework gives ~+3 points on AIME, is not actually tested. Every table row uses either the PPL bonus or the multi-head critic, never both. Figure 6 compares baselines to GRPO+PPL and to PPO+16-heads separately. So the headline is a framework claim supported by two independent experiments. That is the biggest soft spot.\n\nWhat is genuinely new: the clipped/annealed PPL bonus formulation, Theorem 3.1 (higher-PPL correct responses get relatively more probability; lower-PPL incorrect responses get relatively less), and the consistency result in Theorem 3.2 connecting bootstrap critic variance to count-based bonuses in linear MDPs. Theorem 3.1 checks out on a skim of Appendix D; it's not deep, but it is a real ordering statement. The calibration collapse observation — the correct/incorrect PPL gap vanishing under naive GRPO and persisting with the bonus — is the most useful empirical insight in the paper.\n\nThe soft spots beyond the missing combined run: no error bars or multiple seeds anywhere, and LLM RL is high-variance, so the +2–3 point margins may be noise. There is also no baseline against an entropy bonus or RND-style predictor, which matters because a critic could reasonably ask whether the PPL bonus is just undirected stochasticity in disguise; the paper argues this in Figure 3 but doesn't test it. Theorem 3.2 relies on a linear MDP with known features and i.i.d. sub-Gaussian noise; for a 4B transformer doing CoT that is a motivation, not a proof, and the paper says little about the gap. None of these are fatal individually, but together they mean the empirical case is thinner than the prose suggests.\n\nI'd send this to peer review — the ideas are worth refereeing and the calibration collapse deserves scrutiny. But a reviewer should require the combined CDE experiment, seeds with standard deviations, and a direct entropy-bonus comparison before the +3 claim is taken seriously.","headline":"Useful two-part exploration recipe, but the paper sells it as one method without ever running the two parts together; the +3 claim is not yet established.","tokens_in":18749,"tokens_out":5373,"would_cite":true,"duration_ms":61329,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two intrinsic curiosity rewards—answer perplexity and value-head disagreement—improve RLVR for LLM math reasoning, adding about 3 points on AIME.","keywords":["curiosity-driven exploration","RLVR","large language models","exploration bonus","perplexity","multi-head critic","calibration collapse","count-based exploration"],"falsifier":"Train the same RLVR setup with the actor perplexity bonus removed and compute, on held-out OOD prompts, the correlation between the multi-head critic std and an independent trajectory-visitation count (e.g., SimHash counts). Near-zero or negative correlation would show the critic bonus does not behave like count-based exploration in the real regime; additionally, if removing the actor bonus erases the AIME gains while keeping the critic bonus, the actor signal is the effective component.","tokens_in":17849,"feed_emoji":"🧠","tokens_out":7439,"duration_ms":73571,"temperature":0.7,"pith_summary":"The paper claims that the weak exploration of RLVR for LLMs can be fixed by rewarding the model for its own curiosity, in two complementary forms. The actor's curiosity is the perplexity of the response it just generated, added to the correctness reward with an adaptive clip. The critic's curiosity is the disagreement among several bootstrapped value-prediction heads, added to the advantage. The paper proves the actor bonus re-weights updates so confident errors are penalized and novel correct answers are encouraged, and that the critic bonus is asymptotically equivalent to classical count-based exploration in linear MDPs. Empirically, on the AIME24 and AIME25 benchmarks, the bonuses yield roughly a three-point average gain over standard GRPO and PPO, and larger Pass@16 gains.","feed_headline":"3 points: curiosity bonuses lift LLM math scores","feed_subtitle":"Perplexity of a model's own answers and disagreement among its value heads improve exploration in RL training.","key_machinery":"The mechanism is a pair of intrinsic rewards that are added to the RLVR objective without any new modules beyond a shared multi-head network. The actor signal, Equation (1), is the per-token perplexity of the sampled response; Equation (2) shapes it through an adaptive weight, a cap on the bonus relative to the original reward, and a scaling factor. The critic signal, Equation (5), is the standard deviation of K bootstrap value heads; Equation (4) folds it into the GAE advantage. The two theorems supply the identities that make these heuristics principled: Theorem 3.1 decomposes the policy update by correctness and perplexity, and Theorem 3.2 identifies the head std with the count-based expl","core_discovery":"The paper's central claim is that two inexpensive, model-derived signals—perplexity of the actor's own answer and the standard deviation of a multi-head bootstrap critic—can serve as exploration bonuses that make RLVR training more effective. The actor bonus is defined as the negative log-likelihood per token of the generated response and is blended with the sparse correctness reward using a min-clip that guarantees the original reward sign is preserved. The critic bonus is the standard deviation of K value heads that share a backbone and are trained on resampled trajectory subsets; it enters the advantage estimate directly. The paper's main theoretical results are Theorem 3.1, showing that","pith_inferences":["Because the critic bonus is a learned substitute for counts, it should generalize to other RL settings where explicit state counters are infeasible, such as open-ended embodiment or long-horizon agentic tasks, provided the value ensemble retains diversity.","If calibration collapse is a general RLVR failure, the same perplexity shaping could improve inference-time trust: for example, reordering self-consistent samples by perplexity may reduce hallucinated confident answers even without retraining.","A direct test of Theorem 3.2 in the transformer regime would be to compute the correlation between the multi-head std and a SimHash visitation count across training; the paper's own Figure 1 suggests hash counts are too coarse, so a positive correlation is not guaranteed.","The two bonuses are complementary: perceptual novelty from the actor and epistemic uncertainty from the critic target different failure modes, suggesting that a single combined curiosity score may transfer to multimodal or tool-using reasoning models."],"forward_implications":["RLVR can gain a principled exploration signal without reward models, search trees, or explicit state-action features; both bonuses are computed from the model's own outputs.","The perplexity bonus counteracts entropy collapse: training curves show policy entropy declines more slowly, and the gap between confidence and correctness (calibration collapse) stays closed throughout training.","The multi-head bonus is inexpensive: 4–16 heads already capture most of the benefit, 2 heads barely help, and the result is insensitive to the bootstrap sub-sample fraction.","Bonus weight decay is necessary: a staircase schedule that keeps exploration strong early and removes it abruptly works best; keeping the bonus forever prevents convergence.","On hard competition problems (AIME24/25), Pass@16 gains of up to about 10 points over baselines suggest the method helps the model discover multiple valid routes instead of a single lucky one."],"fun_headline_variants":["Curiosity-driven exploration adds 3 points to LLM math scores","RLVR gets a curiosity boost: +3 AIME points","LLMs turn curious, gain 3 points on AIME benchmark","CDE: Curiosity bonuses lift LLM reasoning by 3 points"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The formal equivalence between the critic bonus and count-based exploration assumes the LLM's reasoning process is a linear MDP with known features, independent sub-Gaussian noise, and a converging empirical covariance; the paper does not verify these conditions for transformer chain-of-thought, so if value-head variance does not track true novelty in real training, Theorem 3.2 would not support the multi-head bonus.","fun_headline_variants_meta":{"raw":{"variants":["Curiosity-driven exploration adds 3 points to LLM math scores","RLVR gets a curiosity boost: +3 AIME points","LLMs turn curious, gain 3 points on AIME benchmark","CDE: Curiosity bonuses lift LLM reasoning by 3 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000532,"raw_usage":{"total_tokens":2386,"prompt_tokens":719,"completion_tokens":1667,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":1593}},"tokens_in":463,"tokens_out":1667,"duration_ms":14316,"temperature":1.0,"reasoning_tokens":1593,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T18:44:47.457121+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same RLVR setup with the actor perplexity bonus removed and compute, on held-out OOD prompts, the correlation between the multi-head critic std and an independent trajectory-visitation count (e.g., SimHash counts). Near-zero or negative correlation would show the critic bonus does not behave like count-based exploration in the real regime; additionally, if removing the actor bonus erases the AIME gains while keeping the critic bonus, the actor signal is the effective component.","supporting_citations":[],"review_version":1}