{"id":"4c984b26-b50b-4499-bab2-74b20e6d21e4","arxiv_id":"2608.03068","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"CVPO adds value-variance weighting and dynamic curriculum weighting to value-based RL for LLM math reasoning, reporting gains over VAPO on five benchmarks.","lead":"This paper combines two training tricks for reinforcement learning with large language models: weighting trajectories by the variance of learned value estimates, and weighting questions by an online estimate of their difficulty. On five math benchmarks with a 7B model, the authors report accuracy gains over the VAPO baseline, but provide no code, data, or repeated-run statistics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eqs. 15–16 appear to invert the paper's stated weighting rationale: high-variance correct trajectories get a larger advantage weight and high-variance incorrect trajectories keep the full negative weight, the opposite of 'suppress positive, explore negative.'","rationale":"The reader's CONDITIONAL verdict focuses on whether σ_i measures exploration rather than value-model noise or response length; that is a legitimate concern. However, the more decisive issue is internal consistency of §4.1.3. The paper's own weighting formulas are inverted with respect to its stated goals. This does not depend on interpreting σ_i: even if σ_i were a perfect exploration signal, Eqs. 15 and 16 move the loss in the wrong direction for the trajectory classes they are supposed to regulate. If the implementation followed the prose rather than the equations, then the method is not what is written; if it followed the equations, the method is not what the paper claims. Either way, the central mechanism is not currently supported by the text, and the Table 1 gains cannot be attributed to the proposed variance-aware correction without a sign correction and rerun. The single-run, no-code setup already made the empirical claim hard to assess; this sign issue makes the mechanism itself incoherent as written. I keep the overall verdict at CONDITIONAL rather than moving to REJECT because the flaw may be a fixable typographical sign error, but the condition set must include correcting or clarifying Eqs. 15–16, ablating the sign direction, and releasing code and per-seed numbers. If the corrected equations cannot reproduce the reported gains, the central claim should be rejected.","tokens_in":10064,"tokens_out":10805,"duration_ms":106727,"concrete_test":"Run a sign-flip ablation on the same DAPO-Math-17k setup: train CVPO with Eqs. 15–16 exactly as printed, and train it with the sign-corrected weights matching the prose (W_S = 1 − αP·sigmoid(λP(Var−μ)) for correct trajectories, W_S = 1 + αN·sigmoid(λN(μ−Var)) for incorrect trajectories), holding all other hyperparameters fixed. Compare Table 1 and the training curves. If only the as-printed version reproduces the reported gains, the paper's mechanism is the opposite of its stated design; if both versions perform similarly, the variance-weighting direction is not the active ingredient.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing problem is internal to §4.1.3. For a correct trajectory (R=1), Eq. 15 gives W_S = 1 + αP·sigmoid(λP(Var−μ)), so W_S > 1 when Var > μ. Since correct trajectories carry predominantly positive advantages, multiplying A by W_S > 1 amplifies the update for high-variance correct responses. The text says high stochasticity in correct trajectories 'needs to be suppressed' and promises 'strong constraints on positive samples'; that requires W_S < 1 when Var > μ. For an incorrect trajectory (R=0), Eq. 16 gives W_S = 1 − αN·sigmoid(λN(μ−Var)), so W_S < 1 when Var < μ and W_S ≈ 1 when Var > μ. Incorrect trajectories carry predominantly negative advantages, so low-variance ones receive a reduced penalty and high-variance ones receive the full penalty. The text says low variance should 'increase the penalty intensity' and high variance should have 'penalty reduced to encourage high-stochasticity exploration'; again the equation does the opposite. Thus, as written, CVPO does not implement the stochasticity-aware correction that the paper motivates and claims to ablate. This is independent of whether σ_i measures exploration or value-model noise; even under the paper's own interpretation, the sign of the weight is backwards. The norm bound in Eq. 13 does not determine this sign, so it cannot rescue the direction. Either the equations or the experimental implementation must be misreported, and the Table 1 gains cannot be attributed to the mechanism described.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CVPO, a reinforcement-learning post-training method for LLM mathematical reasoning. It combines a trajectory-level stochasticity-aware advantage weighting, based on the standard deviation of token-level value estimates, with a question-level dynamic curriculum weighting based on a Bayesian accuracy estimate. The method is evaluated on Qwen2.5-7B trained on DAPO-Math-17k and compared with VAPO and an asynchronous GRPO baseline on AIME24, AIME25, AMC23, AMC24, and MATH500, reporting substantial gains. The authors also provide a short theoretical argument bounding the policy gradient norm in terms of value variance and ablation studies of the two proposed components.","tokens_in":10480,"tokens_out":4675,"duration_ms":49859,"significance":"If the reported results are reproducible, the paper would make a practically useful contribution: it suggests that token-level value variance can be used to reweight advantages and that an accuracy-based curriculum mitigates difficulty drift, with gains on several externally evaluated math benchmarks. The external evaluation is a genuine strength, and the theoretical bound in Section 4.1.1, while elementary, is clearly stated. However, the central mechanism is currently internally inconsistent: the equations for the stochasticity-aware weights implement the opposite of the described weighting in the text, and the empirical evidence rests on a single configuration with no seeds, no confidence intervals, and no released code or checkpoints. The method also contains many hand-specified hyperparameters, several of which appear to be selected using the same benchmarks on which the method is evaluated. These issues prevent the paper from being accepted in its current form.","major_comments":[{"comment":"The stochasticity-aware weighting functions implement the opposite of the stated mechanism. For a correct trajectory (R=1), Eq. (15) gives W_S = 1 + α_P·sigmoid(λ_P(Var−μ_k)), so high-variance correct trajectories receive a weight larger than 1. Since correct trajectories predominantly carry positive advantages, this amplifies the update for exactly the high-variance correct responses that the text says should be suppressed, and contradicts the promised 'strong constraints on positive samples.' For an incorrect trajectory (R=0), Eq. (16) gives W_S = 1 − α_N·sigmoid(λ_N(μ_k−Var)), which makes low-variance incorrect trajectories receive a reduced penalty (W_S < 1) and high-variance incorrect trajectories keep the full penalty (W_S ≈ 1). This is again opposite to the text, which states that low variance should increase penalty intensity and high variance should reduce it. This is an internal inconsistency in the load-bearing component of the method: as written, the equations do not implement the stochasticity-aware correction that is claimed and ablated in Table 1. The authors must either change Eqs. (15)-(16) or correct the description and rerun the experiments so that the reported results correspond to the stated mechanism.","section":"Section 4.1.3, Eqs. (15)-(16)"},{"comment":"The central empirical claim rests on a single configuration: no random seeds are reported, no confidence intervals are given, and no code or checkpoints are released. Moreover, the final CVPO row uses a variance coefficient of 0.3 that is explicitly selected from the sweep shown in Figure 4 on the same benchmark family as the final evaluation. This means the reported gains over the second-best methods, e.g., 9.3% on AIME24 and 25.2% on AMC23, may in part reflect selection on the evaluation sets. I request multi-seed results with variance or confidence intervals, and either a pre-specified or separately validated choice of the 0.3 coefficient, or a clear statement that the reported numbers are the best over a small grid.","section":"Section 5.1 and Table 1"},{"comment":"Equation (13) is an upper bound on the gradient norm and demonstrates only that value variance can affect the magnitude of policy updates. It does not determine the sign of the advantage weighting, nor does it show that increasing variance in correct or incorrect trajectories is beneficial or that one trajectory type should be emphasized over another. Therefore the theory cannot rescue the sign inconsistency in Eqs. (15)-(16), and even after correcting the sign it would not justify the particular sigmoid-shaped weighting. The theoretical section should be substantially weakened, or supplemented with a directional argument or controlled experiments that isolate the effect of the variance signal.","section":"Section 4.1.1, Eq. (13)"},{"comment":"The dynamic curriculum component relies on several hand-specified thresholds and curve parameters (0.3, 0.8, 0.05, 0.03; k1, c1, k2, c2 and their primed versions) with no sensitivity analysis and no derivation. These are free parameters that appear to be chosen to match the behavior shown in Figure 2 and the benchmark outcomes. To support the claim that the curriculum is a general mechanism rather than a fitted schedule, the paper should report sensitivity to these thresholds and ideally a procedure for setting them without using the final evaluation benchmarks.","section":"Section 4.2.2, Eqs. (25)-(26)"}],"minor_comments":[{"comment":"The notation is inconsistent: the text defines σ_i = std(v_i) and uses Var(τ) in Eqs. (15)-(16), while Eq. (14) defines μ_k as the mean of σ_i within a prompt group. The unprimed and primed hyperparameters in the text also switch between μ_P/μ_N and μ_k; please unify the notation.","section":"Section 4.1.3 and Eq. (14)"},{"comment":"The trend indicator Δ_t^τ divides by the previous posterior mean \\( \\bar\\gamma^{t-1}_\\tau \\), which can be zero if no successes have been observed; a small constant or a different definition should be given for this case.","section":"Section 4.2.2, Eq. (22)"},{"comment":"There is a grammatical error in the sentence 'This shift is much more visible with CVPO, This shift in value-variance distribution...' The second sentence should be capitalized and separated.","section":"Section 6.2 and Figure 3"},{"comment":"The claim that the theoretical analysis 'proves that appropriately leveraging this variance can improve the upper bound of model performance' is stronger than what Eq. (13) actually shows; please rephrase to match the result.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The sign inconsistency in Eqs. (15)-(16) is the decisive issue for this manuscript. If the authors cannot resolve it by correcting the equations and confirming that the experiments match the corrected mechanism, the paper should not be accepted. I would also ask the editor to consider whether the benchmark-selected variance coefficient and the absence of released code make the empirical claims sufficiently verifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read of CVPO. The headline result is a solid if unremarkable combination—trajectory value-variance weighting plus an accuracy-based curriculum—and the reported margins over VAPO on AIME/AMC would matter if they held. But I don't think they can be trusted as reported, and there's a more basic problem than missing error bars.\n\nLook at Eqs. 15–16 in §4.1.3. For correct trajectories the text says high variance should be suppressed, yet Eq. 15 gives W_S > 1 when Var > μ, which amplifies a positive advantage rather than suppressing it. For incorrect trajectories the text says low variance should increase the penalty intensity and high variance should reduce it, but Eq. 16 gives W_S < 1 for low variance, which reduces the penalty, and W_S ≈ 1 for high variance, which keeps it. In both cases the direction is backwards. This isn't a minor typo: the stochasticity-aware correction is the paper's core mechanism, and the ablation attributing gains to it is unexplained. Either the equations don't match the implementation—meaning the paper misreports its own method—or the implementation doesn't do what the authors think. The norm bound in Eq. 13 doesn't fix this, since it only bounds gradient magnitude and says nothing about which sign the weights should take.\n\nBeyond that, the empirical support is thinner than the table suggests: one run, no seeds, no confidence intervals, no code or checkpoints. The final variance coefficient 0.3 is picked from the sweep in Figure 4, so the headline row is fitted on the benchmark family. The curriculum hyperparameters are also hand-set. These are fixable in principle, but they compound the sign problem. Citation pattern is standard; the relevant baselines (VAPO, GRPO, DGRO) are cited, and there's no suspicious self-citation.\n\nWhat's genuinely new: I don't know of another paper combining variance-based advantage weighting with a Beta-updated curriculum trigger, and the ablation suggests both pieces can help if the mechanism is actually implemented as intended. The topic—value-based RL for LLM reasoning—is timely, and the comparison to VAPO is the right one.\n\nBottom line: this deserves a serious referee, but not as-is. A referee should ask for the code, per-seed results, and a corrected or reinterpreted Eqs. 15–16. If the sign issue is simply a typo and the implementation matches the text, the method is plausible and worth engaging. If it isn't, the paper's central claim is unsupported. I'd send it back with that demand.","headline":"The reported gains would matter if they held, but the core weighting equations in §4.1.3 appear to invert the paper's own stated mechanism, so the method as written is not the method described.","tokens_in":10996,"tokens_out":2656,"would_cite":false,"duration_ms":26493,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a reinforcement-learning fine-tuning method, CVPO, outperforms strong value-based baselines by combining a value-variance advantage correction with a dynamic question-difficulty curriculum.","keywords":["reinforcement learning","value-based RL","advantage estimation","curriculum learning","mathematical reasoning","large language models","value variance","exploration"],"falsifier":"Measure $\\sigma_i = \\operatorname{std}(v_i)$ on generated responses of equal correctness and different response lengths, or on responses where the value model is intentionally corrupted; if the variance signal tracks length or critic noise rather than exploration, the interpretation fails. A direct test of the mechanism is to flip the weighting directions for negative trajectories while holding the curriculum fixed; if performance does not degrade, the exploration weighting is not what drives the reported gains.","tokens_in":9860,"feed_emoji":"🧮","tokens_out":5412,"duration_ms":51006,"temperature":0.7,"pith_summary":"This paper proposes CVPO, a reinforcement-learning method for large language models that makes two changes to how training advantages are computed. First, it estimates the standard deviation of token-level value predictions within each generated response and uses that variance to down-weight high-variance correct responses while preserving exploration on high-variance incorrect ones. Second, it tracks each question's recent accuracy with a Beta posterior and reweights questions so the model trains hardest on problems near its current ability, switching emphasis toward hard questions when learning plateaus. The paper reports that on Qwen2.5-7B this outperforms VAPO and GRPO variants on AIME24, AIME25, AMC23, AMC24, and MATH500, with each mechanism contributing and the variance correction contributing more. If correct, this means value-model statistics that are normally ignored, the spread of predicted values, can serve as a usable exploration signal for RL fine-tuning of reasoners.","feed_headline":"RL that weights answers by value variance tops five math benchmarks","feed_subtitle":"CVPO adds a value-variance weight and an adaptive curriculum, lifting Qwen2.5-7B AIME24 accuracy to 23.7%.","key_machinery":"The central object is the trajectory value-variance signal $\\sigma_i = \\operatorname{std}(v_i)$, computed from the critic's token-level value estimates on each response and normalized within a prompt group by $\\mu_k$. It controls an asymmetric sigmoid correction $W_S$ for positive and negative trajectories, while a separate Beta-posterior accuracy estimate $\\rho_q$ controls a difficulty curriculum weight $W_D$ that changes form when the model enters a learning bottleneck. Together they rescale every token advantage as $A' = W_S(\\operatorname{Var}(\\tau), R(\\tau)) \\cdot W_D(\\rho_q, t_c) \\cdot A$, which is the single mechanism that carries the argument: variance says how much a trajectory is exploring, accuracy says how hard a question is for the current policy, and both together decide how strongly each token's gradient should count.","core_discovery":"CVPO rests on two linked claims. At trajectory level, the spread of token-level value estimates, $\\sigma_i = \\operatorname{std}(v_i)$, is treated as a measure of exploration intensity, and the paper derives an upper bound $\\|\\nabla_\\theta J(\\theta)\\|_2 \\le C(\\sigma_Q + \\sqrt{\\operatorname{Var}(V)})$ showing that value variance bounds the policy-update magnitude. From this it constructs an asymmetric sigmoid weight $W_S$ on the advantage: correct trajectories with high variance are penalized toward stable convergence, while incorrect trajectories with high variance are lightly penalized to keep exploring. At question level, a Beta posterior over each question's accuracy $\\rho_q$ feeds a curriculum weight $W_D$ that concentrates training on roughly-50%-accuracy questions early and shifts to low-accuracy hard questions when stagnation is detected. The combined advantage $A' = W_S \\cdot W_D \\cdot A$ is what the paper trains with, and the reported gains follow from this single weighted advantage being more selective about which trajectories and which questions drive the gradient.","pith_inferences":["A consequence the paper leaves implicit: if value variance really indexes exploration, CVPO offers a way to regulate exploration without changing the reward model, which could transfer to tasks beyond math where rewards are sparse.","The Beta-progress curriculum can be extended to non-binary or partial-credit rewards by replacing the Bernoulli likelihood, but the paper only tests binary correctness.","The AMC/AIME crossover suggests the variance gain is problem-dependent; a testable extension would schedule the variance coefficient to decay as the policy matures rather than keeping it fixed.","The paper's interpretation of $\\sigma_i$ assumes critic noise is small; separating exploration variance from value-model uncertainty would require an explicit noise model, which the paper does not provide."],"forward_implications":["With the same Qwen2.5-7B backbone and DAPO-Math-17k training set, CVPO is reported to reach avg@32 accuracy of 0.237, 0.133, 0.720, 0.512, and 0.802 on AIME24, AIME25, AMC23, AMC24, and MATH500, respectively.","Each component alone beats VAPO; removing the variance correction hurts more than removing the curriculum, implying that most of the gain comes from variance-based advantage weighting.","During training past about step 1500, CVPO keeps improving while baselines plateau, because the curriculum moves weight onto harder questions once the model is stagnant.","The crossover shown in Figure 4 implies that value-variance coefficients need to be tuned per benchmark: larger coefficients help AMC problems but degrade the harder AIME questions."],"supporting_citations":[{"why":"VAPO is the value-based baseline that CVPO extends and the main performance target it must beat.","marker":"Yue et al. (2025)"},{"why":"DeepSeekMath GRPO provides the value-free baseline and group-relative advantage formulation that CVPO contrasts with.","marker":"Shao et al. (2024)"},{"why":"DAPO-Math-17k is the training set, and DAPO supplies the open-source RL system context and hyperparameter starting points.","marker":"Yu et al. (2025)"},{"why":"VC-PPO introduces value pretraining and decoupled GAE, the value-model machinery CVPO builds on for long reasoning chains.","marker":"Yuan et al. (2025)"},{"why":"The MATH dataset provides MATH-500, one of the five evaluation benchmarks used to measure CVPO's performance.","marker":"Hendrycks et al. (2021)"},{"why":"This reference motivates the difficulty-drift problem that the dynamic curriculum component is designed to address.","marker":"Qu et al. (2025)"},{"why":"Curriculum RL from easy to hard tasks is the background that the difficulty-aware weighting builds on.","marker":"Parashar et al. (2025)"},{"why":"Adaptive curriculum learning provides the comparison point for the paper's dynamic, accuracy-based question weighting.","marker":"Shi et al. (2025)"}],"fun_headline_variants":["Value-variance weighting and adaptive curriculum boost LLM math reasoning","CVPO: value variance guides exploration and curriculum, beats VAPO","Weighted by value variance, RL curriculum lifts Qwen AIME to 23.7%","Stable exploration via value-variance advantages and difficulty curriculum","CVPO: curriculum + variance-aware advantage for robust LLM reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the spread of the value model's token-level predictions within a response tells you how much the model is exploring, and that weighting trajectories by this spread the way CVPO does improves learning even though the paper only shows the spread bounds gradient size, not that the weighting direction is optimal.","fun_headline_variants_meta":{"raw":{"variants":["Value-variance weighting and adaptive curriculum boost LLM math reasoning","CVPO: value variance guides exploration and curriculum, beats VAPO","Weighted by value variance, RL curriculum lifts Qwen AIME to 23.7%","Stable exploration via value-variance advantages and difficulty curriculum","CVPO: curriculum + variance-aware advantage for robust LLM reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000882,"raw_usage":{"total_tokens":3818,"prompt_tokens":957,"completion_tokens":2861,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":2766}},"tokens_in":573,"tokens_out":2861,"duration_ms":20086,"temperature":1.0,"reasoning_tokens":2766,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T00:58:15.296894+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure $\\sigma_i = \\operatorname{std}(v_i)$ on generated responses of equal correctness and different response lengths, or on responses where the value model is intentionally corrupted; if the variance signal tracks length or critic noise rather than exploration, the interpretation fails. A direct test of the mechanism is to flip the weighting directions for negative trajectories while holding the curriculum fixed; if performance does not degrade, the exploration weighting is not what drives the reported gains.","supporting_citations":[],"review_version":1}