{"id":"427e74f6-af1d-4252-bf3d-7364c9d04f6a","arxiv_id":"2509.02333","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"DCPO combines dynamic clipping bounds and cumulative advantage standardization, achieving higher average math benchmark scores and higher response utilization than GRPO, DAPO, and GSPO.","lead":"This paper proposes DCPO, a reinforcement learning method that adaptively clips token probability ratios and smooths advantage estimates across training steps to improve LLM reasoning. On four math benchmarks across four model sizes, DCPO reports higher average accuracy and better data utilization than GRPO, DAPO, and GSPO, with the largest gains on AIME24.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AIME24/AIME25 Avg@32 gaps are within sampling noise on 30-problem sets; without seeds or clustered CIs the headline accuracy claim is unverified.","rationale":"The reader's weakest_assumption (stationarity of per-prompt reward distributions) is a plausible theoretical gap, and I partially agree that cumulative standardization in Eq. (5) could mis-calibrate under distribution drift. However, the central claim as stated is an empirical SOTA claim, and the load-bearing issue is that the supporting numbers have no quantified uncertainty. The AIME sets have only 30 problems each, and the reported margins in Table 1 correspond to a few problems. A 7.2 pp gap at p ~ 0.35 with 30 clusters is within one standard error, so a single run cannot establish 'best.' This is not an internal inconsistency with the method, so I am not claiming the method is wrong; I am claiming the evidence is insufficient. The paper does provide independent support through released code, ablation curves (Figure 3), and direct RUR/TCR measurements, which should be credited. But the accuracy headline needs clustered confidence intervals or repeated seeds. Since the reader already issued CONDITIONAL, this concern does not move the verdict; it sharpens the condition on which acceptance should depend.","tokens_in":20032,"tokens_out":11010,"duration_ms":127727,"concrete_test":"Run the exact DCPO/DAPO/GSPO/GRPO pipelines with at least 3 seeds (ideally 5) on Qwen2.5-Math-7B and Qwen2.5-14B, and report AIME24/AIME25 Avg@32 as mean +/- std. Additionally, compute cluster-bootstrap 95% confidence intervals by resampling the 30 AIME problems (keeping each problem's 32 responses as a cluster) for the current single-run checkpoints. If the DCPO interval overlaps DAPO's or GSPO's on either benchmark, the headline 'best on AIME' claim is not supported and the verdict should remain conditional pending error bars. If the intervals separate cleanly, the accuracy claim would be substantiated despite the small problem count.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1's headline comparisons are single runs on 30-problem test sets. For AIME24-7B, DCPO's 38.8 vs DAPO's 31.6 is a 7.2 percentage-point gap. Under the natural per-problem/clustered interpretation of Avg@32 (each problem contributes the fraction of 32 samples that are correct), the standard error of a proportion near p=0.35 with 30 clusters is about 8.7 pp, so this gap is less than one standard error. For AIME25-14B, 19.0 vs 10.5 (8.5 pp) has a pooled standard error of roughly 9 pp. Even if Avg@32 is instead pooled over all 32x30 responses, the paper reports no within-problem variance, no cluster-level confidence intervals, and no seeds, so correlated response noise cannot be ruled out. Since the abstract and Section 6 rest the central claim on these specific AIME gains, the reported numbers are not currently distinguishable from chance. The RUR/TCR plots are separate evidence for the efficiency mechanism, but they do not establish accuracy superiority.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DCPO (Dynamic Clipping Policy Optimization) for RLVR. It combines three modifications to GRPO: (i) token-level dynamic-adaptive clipping bounds derived from an importance-sampling variance argument (Section 2), (ii) Smooth Advantage Standardization (SAS) that mixes per-step and cumulative reward standardization (Section 3), and (iii) an 'Only Token Mean' loss that averages the policy loss over tokens within a response but not over responses. The authors report evaluations on MATH500, AMC23, AIME24, and AIME25 with four Qwen models, comparing against GRPO, DAPO, and GSPO. The paper claims state-of-the-art performance on the four benchmarks, plus a 28 pp higher response utilization ratio than GRPO, roughly doubled training efficiency over DAPO, and an order-of-magnitude lower token clipping ratio.","tokens_in":2149,"tokens_out":3934,"duration_ms":132034,"significance":"If the empirical claims held, DCPO would be a useful and broadly applicable contribution to RLVR: the underlying ideas—probability-dependent clipping and cumulative standardization—are plausible, and the paper provides a public repository, ablations, and a detailed appendix. The method is simple enough to adopt in existing GRPO-style frameworks. However, the current manuscript does not substantiate the headline claims. The reported benchmark gains on AIME24/AIME25 are within sampling noise for 30-problem sets, the 'state-of-the-art' wording is contradicted by several rows of Table 1, and the RUR improvement is largely a direct algebraic consequence of the SAS formula rather than an independent empirical finding. The theoretical derivation of the dynamic clipping bounds is also not fully sound as written. These issues are fixable in principle, but they are load-bearing for the paper's central claims.","major_comments":[{"comment":"The claim of 'state-of-the-art performance on four benchmarks based on four models' is not supported by Table 1. DCPO does not win every benchmark or every model. Examples: Qwen2.5-3B AIME24 Avg@1 is 3.3 vs 10.0 for GRPO and DAPO; Qwen2.5-Math-7B MATH500 is 82.5 vs 84.0 for GSPO; Qwen2.5-14B AMC23 Avg@32 is 79.9 vs 85.1 for DAPO and AIME24 Avg@1 is 20.0 vs 23.3 for GSPO. The strongest defensible statement is that DCPO has the best or near-best average across the four benchmarks on some models, and even that is not true for Qwen2.5-3B Avg@32, where DCPO (22.7) is below DAPO (23.1). Please qualify the claim and report a proper aggregate with error bars.","section":"§4, Table 1, Abstract"},{"comment":"The headline accuracy gains on AIME24 and AIME25 are not shown to be statistically distinguishable from noise. These sets contain 30 problems each; Avg@32 is a per-problem proportion (fraction of 32 samples answered correctly) averaged over 30 problems. Under a clustered interpretation, the standard error of the 7.2 pp gap on AIME24-7B (38.8 vs 31.6) is roughly 8-12 pp, and the 8.5 pp gap on AIME25-14B (19.0 vs 10.5) has a similar standard error. No seeds, no per-problem confidence intervals, and no within-problem variances are reported. The abstract and Section 6 rest the central claim on these specific AIME gaps; as presented, the differences are within one standard error and cannot be taken as evidence of superiority.","section":"§4-5.1, Table 1 (AIME24/AIME25)"},{"comment":"The reported 28 pp improvement in response utilization ratio (RUR) over GRPO is by construction. When all rewards for a prompt are identical in a step, the GRPO advantage is zero. Under SAS, if A_new = 0, Eq. (6)-(7) select a nonzero advantage of size (1/i) A_total (the paper states this explicitly). Therefore the high RUR does not independently show that generated responses are used more 'effectively' in the sense of improved learning; it is a direct consequence of the chosen smoothing. To support the sample-efficiency claim, the paper needs a metric that is not definitionally aligned with the method, e.g., accuracy per generated response, gradient signal quality, or a controlled comparison of final performance as a function of the number of training samples.","section":"§3, Eq. (6)-(7), Table 2"},{"comment":"The derivation of the dynamic clipping bounds is not a valid closed-form solution. The constraint |(r-1)p| ≤ ε, with p = r q, leads to the quadratic inequality -ε ≤ q(r²-r) ≤ ε. The feasible set is a union of two intervals, approximately [0, r_low] ∪ [r_high, r_high'] for q ≥ 4ε. The paper silently selects the interval containing r = 1, namely [0.5+0.5√(1-4ε/q), 0.5+0.5√(1+4ε/q)], and does so without justification for dropping the lower interval. Appendix A.5 Eq. (25) further conflates variables ('lowge ≤ 0 ≤ r ≤ highle') and Eq. (24)-(26) contain inconsistent notation. Since dynamic clipping is one of the two core contributions, the theoretical justification must either be corrected or explicitly recharacterized as an empirically motivated heuristic.","section":"§2 and Appendix A.5, Eq. (4), (24)-(27)"},{"comment":"SAS assumes that, for a fixed prompt, 'the overall reward distribution ... can be considered as drawn from the same global distribution throughout the course of training.' This assumption is not justified and is implicitly contradicted by the paper's own observations: Figure 7 shows substantial entropy dynamics over training, and the paper discusses entropy collapse and model-dependent trajectories. If the reward distribution drifts as the policy changes, using accumulated mean/σ from all past steps can systematically mis-calibrate current advantages. Provide either theoretical conditions under which the cumulative normalization remains valid, or empirical sensitivity checks (e.g., comparing against a moving-window baseline).","section":"§3 (stationarity assumption)"}],"minor_comments":[{"comment":"Typo: 'microm' should be 'microbatch'. Also, the definition of N and how microbatch size relates to the batch size of 512 and minibatch of 32 should be stated explicitly.","section":"§5.2, Eq. (9)"},{"comment":"The GSPO hyperparameters are given as clipping thresholds (3e-4, 4e-4). These seem unusually small; please clarify whether these are the values used in the original GSPO paper and how they were chosen for the comparisons.","section":"Appendix A.9"},{"comment":"The formatting of Table 2 is broken: values and model names run together. Please reformat and include the omitted DAPO row or state clearly that DAPO is excluded because its dynamic sampling makes RUR incomparable.","section":"Table 2"},{"comment":"The caption and axis labels are mislabeled: panel (b) says r(x) bounds for q∈[0,1], but the x-axis is q; the text '∅' in the y-axis is unclear. Please redraw the figure with proper labels and consistent notation.","section":"Figure 4"},{"comment":"The ablation curves are presented without error bars or repeated seeds. Given that the final performance differences are small on several benchmarks, the claim that 'each component contributes positively' and 'their combination yields substantial gains' would be more convincing with standard deviations or at least two seeds.","section":"§5.4, Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and the experimental setup is reproducible in spirit, but the evidence for the central claims is currently insufficient. The statistical weakness of the AIME results, the by-construction nature of the RUR metric, and the flawed clipping-bound derivation are all fixable, but they require either new experiments, a corrected theory section, or substantially moderated claims. I recommend major revision rather than rejection because the underlying algorithmic ideas are plausible and the code is available; however, a revision that does not address the statistical and theoretical issues would not be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the paper has two genuinely new pieces: dynamic clipping bounds that widen for low-probability tokens, and a smoothed cumulative advantage that keeps same-reward responses in the update. Second, the headline \"state-of-the-art on four benchmarks\" is not supported by the table: on several individual rows DCPO loses (e.g., MATH500-7B, AIME25-7B, AIME24-14B Avg@1), and the AIME gains that drive the SOTA claim are on 30-problem test sets with no seeds or confidence intervals. The stress-test arithmetic is right: a ~7pp gap on AIME24-7B Avg@32 is within one standard error of a 30-cluster proportion. So the accuracy superiority is unverified.\n\nWhat is actually good: the dynamic clipping derivation from an importance-sampling variance bound is a real idea, and the boundary matching with GRPO's epsilon is elegant. The OTM loss point—that batch-level averaging dilutes per-response advantage—is a legitimate observation, and the ablation suggests each component helps. The released code and the TCR/RUR plots give the method a concrete empirical footprint: a clipping ratio an order of magnitude lower is a structural effect, not a metric trick.\n\nSoft spots, in order. (1) No seeds, no error bars anywhere. On MATH500 (500 problems) the differences are still single-run and often under 2pp. (2) The RUR improvement is partly definitional: because SAS deliberately assigns 1/i times the cumulative advantage to same-reward responses, the nonzero-advantage count is designed to go up. That is not a fake result, but the 28% number should be presented as a consequence of the smoothing, not an independent discovery. (3) The stationarity assumption in Section 3—that per-prompt reward distributions do not change over training—is asserted, not tested. The paper's own entropy curves show the policy is moving; the cumulative statistics could be mis-calibrated. A simple per-prompt drift check would settle it. (4) The Appendix derivation for the clipping bounds is convoluted, and the variance-condition split is not clearly motivated.\n\nIf I were an editor, I would send this to peer review, because the core mechanism is novel and the paper is reproducible. But I would tell the authors to scale back the absolute claims, add seeds or at least clustered confidence intervals on the 30-problem sets, and address the stationarity issue. It is a solid workshop-to-conference paper needing revision, not a desk reject.","headline":"Genuinely new mechanisms and a plausible efficiency story, but the headline SOTA claim rests on 30-problem benchmarks without error bars and should not be taken at face value.","tokens_in":20769,"tokens_out":3546,"would_cite":true,"duration_ms":38253,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GRPO-style RLVR wastes most generated data through fixed clipping bounds and per-step reward standardization; DCPO replaces both, nearly doubling response utilization and lifting AIME24 to 38.8.","keywords":["reinforcement learning from verifiable rewards","GRPO","dynamic clipping","advantage standardization","sample efficiency","mathematical reasoning","large language models","response utilization ratio"],"falsifier":"Measure whether the SAS premise holds: pick a set of prompts and, at each training step, record the current-step reward mean (μ_new) and the cumulative reward mean (μ_total) for that prompt. If the standardized gap (μ_new − μ_total)/σ_total grows systematically beyond sampling noise in the second half of training, the reward distribution is drifting and the cumulative statistics mis-calibrate current advantages. The decisive experiment is to ablate SAS with a rolling window (e.g., the last 50 steps of rewards instead of all history): if rolling-window standardization beats full-history on AIME","tokens_in":19939,"feed_emoji":"🎯","tokens_out":8895,"duration_ms":89436,"temperature":0.7,"pith_summary":"Reinforcement learning from verifiable rewards (RLVR) trains reasoning models, but the standard GRPO update wastes data: fixed ratio-clipping bounds and step-level reward standardization routinely send the gradient to zero, so more than half of all generated responses never contribute to learning. DCPO claims to fix both defects. Its dynamic clipping rule widens the allowed update interval for low-probability tokens, letting the model explore rare but informative tokens, and its smooth advantage standardization mixes each step's reward statistics with statistics accumulated over all earlier responses, so responses with tied rewards still produce a learning signal. On four benchmarks across four Qwen models, the paper reports that this raises the response utilization ratio by about 28 percentage points (to roughly 72%), cuts the token clipping ratio by an order of magnitude, halves training time relative to DAPO, and improves competition-math accuracy — AIME24 Avg@32 reaches 38.8 with Qwen2.5-Math-7B versus 32.1 for GRPO. If these results hold, DCPO is the most sample-efficient of the compared RLVR methods while also improving performance on hard competition math.","feed_headline":"Smarter clipping lifts RL math score to 38.8 on AIME24","feed_subtitle":"Probability-aware clipping and cumulative reward scaling push response use from 44% to 72%.","key_machinery":"Two mechanisms carry the argument. Dynamic adaptive clipping (DAC): instead of a fixed interval around ratio 1, the bound solves |(r(x)-1)p(x)| ≤ ε, giving a closed-form per-token interval whose width grows as 1/√q(x), so rare tokens get a wider admissible update range while common tokens keep GRPO's tight interval; the ratio is hard-capped at 10 following the dual-clipping idea. Smooth advantage standardization (SAS): per-step advantage and cumulative advantage (mean and variance of all responses to the prompt so far) are blended with weights that shift as 1/i, and the smaller-magnitude blend is used, so a step where all rewards tie still contributes a signal of (1/i)·A_total. A third compo","core_discovery":"The central claim is that GRPO-family RLVR suffers two self-inflicted gradient deaths — fixed symmetric clipping of the probability ratio and standardization of rewards within a single sampling step — and that both can be removed without destabilizing training. DCPO replaces |r-1| ≤ ε with a probability-weighted constraint |(r-1)·p| ≤ ε, whose closed-form solution yields per-token clipping bounds that widen as the old probability q(x) shrinks; tokens with prior probability below about 1/(1+ε) get a much larger admissible ratio, capped at r_max = 10. It then standardizes advantages using a smoothed blend of the current-step and cumulative reward distributions, and averages loss over tokens wi","pith_inferences":["The stationarity premise behind SAS — that a prompt's reward distribution is the same at every training step — is directly testable: if reward distributions drift as the policy sharpens (which the paper's own entropy and clipping curves hint at), a rolling-window version of the cumulative statistics should outperform the full-history version; that experiment would show how far the assumption can b","The dynamic clipping rule encodes a concrete claim about the loss landscape: gradients from rare, high-entropy tokens are the valuable ones. If that is right, the same probability-dependent widening should transfer beyond math to code generation and other RLVR domains where outcome rewards are sparse and within-step ties are common.","The 28-point response-utilization gain implies DCPO's advantages estimate a different baseline object than GRPO's — a per-prompt historical baseline rather than a within-batch baseline — so a natural check is whether the smoothed advantages still behave as an unbiased variance-reduction baseline when responses come from rapidly changing policies.","The largest benchmark gains cluster on AIME (hard competition problems) rather than MATH500; if the mechanism is genuinely about rare-token exploration, the method should show smaller gains on easy items and larger gains on hard subsets of any benchmark — a prediction the paper's data supports but does not isolate."],"forward_implications":["RLVR training becomes markedly cheaper per useful update: roughly 72% of generated responses carry a nonzero learning signal instead of 44%, and DAPO's dynamic sampling filter — which discards tied-reward responses and regenerates — becomes unnecessary, cutting training time roughly in half.","Under sampling-based decoding (Avg@32), competition-math accuracy improves most: AIME24 goes from 32.1 (GRPO) to 38.8 on the 7B model, and AIME25 from 10.5 to 19.0 on the 14B model, suggesting the trained policy's sampled distribution, not just its greedy output, contains more correct solutions.","The token clipping ratio falls by an order of magnitude, meaning nearly the whole response contributes to each update; combined with the reported entropy curves, DCPO keeps policy entropy in a middle band rather than collapsing (GRPO) or fluctuating heavily (DAPO).","Ablations show each component helps alone: OTM, SAS, and DAC each beat GRPO on average Avg@32 when swapped in individually, and their combination beats all baselines, so the gains are additive rather than resting on a single trick."],"supporting_citations":[{"why":"Supplies the DAPO baseline and its Clip-Higher and Dynamic Sampling mechanisms that DCPO argues against, plus the Math-DAPO-17k training corpus used in the experiments.","marker":"Yu et al., 2025"},{"why":"Introduces GRPO with rule-based rewards, the base algorithm whose zero-gradient problem DCPO targets.","marker":"DeepSeek-AI et al., 2024"},{"why":"Provides the GRPO loss and advantage formulation (sequence-level mean, per-step standardization) that DCPO modifies with OTM and SAS.","marker":"Shao et al., 2024"},{"why":"Supplies the fixed clipping bound and importance-sampling variance argument that DCPO replaces with probability-dependent bounds.","marker":"Schulman et al., 2017"},{"why":"Defines GSPO, the sequence-level clipping baseline DCPO compares against on the 7B and 14B models.","marker":"Zheng et al., 2025"},{"why":"Supports the premise that RLVR refines an existing sampling distribution rather than reshaping it, justifying cumulative reward standardization in SAS.","marker":"Yue et al., 2025"},{"why":"Establishes that high-entropy (low-probability) tokens drive reasoning gains, motivating the widened clipping interval for rare tokens.","marker":"Wang et al., 2025"},{"why":"Contributes the dual-clipping idea that caps the adaptive ratio at r_max = 10 for both positive and negative advantages.","marker":"Ye et al., 2020"}],"fun_headline_variants":["Dynamic clipping ends zero-gradient stall in RL math reasoning","Token-aware clipping cuts RL gradient deaths and doubles efficiency","Dynamic clipping and smooth advantage standardization beat GRPO","Zero-gradient deadlock broken by token-aware clipping","Adaptive bounds boost RL math reasoning to SOTA on AIME24"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"For a fixed problem, the distribution of rewards over sampled responses is treated as unchanged across all training steps, so means and variances collected from every past response still describe the responses generated today; if the model's behavior shifts that distribution, the cumulative statistics mis-scale current advantages.","fun_headline_variants_meta":{"raw":{"variants":["Dynamic clipping ends zero-gradient stall in RL math reasoning","Token-aware clipping cuts RL gradient deaths and doubles efficiency","Dynamic clipping and smooth advantage standardization beat GRPO","Zero-gradient deadlock broken by token-aware clipping","Adaptive bounds boost RL math reasoning to SOTA on AIME24"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000951,"raw_usage":{"total_tokens":3974,"prompt_tokens":905,"completion_tokens":3069,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":2991}},"tokens_in":649,"tokens_out":3069,"duration_ms":25032,"temperature":1.0,"reasoning_tokens":2991,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:35:24.396373+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure whether the SAS premise holds: pick a set of prompts and, at each training step, record the current-step reward mean (μ_new) and the cumulative reward mean (μ_total) for that prompt. If the standardized gap (μ_new − μ_total)/σ_total grows systematically beyond sampling noise in the second half of training, the reward distribution is drifting and the cumulative statistics mis-calibrate current advantages. The decisive experiment is to ablate SAS with a rolling window (e.g., the last 50 steps of rewards instead of all history): if rolling-window standardization beats full-history on AIME","supporting_citations":[],"review_version":1}