{"id":"4588a5ec-40e3-40ac-a97f-ea22579d88f7","arxiv_id":"2607.26457","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"DHRCL, a staged RL curriculum with dense hierarchical rewards and token-level credit redistribution, improves code LLM Pass@1 by about 1 point over the strongest baseline in controlled Qwen3 experiments.","lead":"This paper proposes a new way to train code-writing AI models, layering syntax, execution, test, and structure feedback in a staged curriculum. It reports small but consistent gains in code pass rates over existing methods, with faster training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Trend-based stage scheduler uses overlapping-window OLS and repeated testing without multiplicity control; if transition times are off, DHRCL's 0.7-point gain over fixed-stage and its 1.1-point margin over VeRPO may not be robust.","rationale":"I read the paper as an empirical claim that DHRCL's combination of dense rewards, trend-based curriculum, and token weighting yields a modest but replicable Pass@1 improvement over strong baselines. The most load-bearing component for the headline margin is the trend-based scheduler: Table 7 shows it contributes 0.7 points over a fixed-stage curriculum, which is the biggest single-component delta besides the token weighting (0.9 points). The statistical test underlying it is fragile because it uses overlapping windows and repeated testing, making its p-values miscalibrated. The paper does not provide sensitivity analysis, so we cannot know whether the reported transition iterations (140/280) are stable or artifacts of the test's violations. I agree with the reader's verdict of CONDITIONAL, with the primary condition being a sensitivity analysis of the scheduler. I also note the token-credit weighting's unanalyzed bias as a secondary concern, but the scheduler is more directly tied to the central claim's magnitude. The paper deserves credit for transparent ablations, three seeds, and problem-level paired bootstrap CIs, which strengthen the empirical core.","tokens_in":17566,"tokens_out":12718,"duration_ms":140631,"concrete_test":"Re-run the Qwen3-8B DHRCL training while replacing the OLS t-test with autocorrelation-robust inference (e.g., Newey-West HAC standard errors on the slope or a block bootstrap) and also vary the window size (6, 8, 10), alpha (0.01, 0.05, 0.10), and the consecutive-failure requirement (1, 2, 3). If the median transition iterations shift by more than ~20 iterations or if final average Pass@1 changes by more than the paired-bootstrap CI half-width (about 0.45 points) or drops below VeRPO's 49.2, then the trend test's statistical fragility is load-bearing for the headline claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 'Trend-Based Automatic Stage Progression' declares stage transitions based on a one-sided linear-trend test of H0: b<=0 at alpha=0.05, applied to the most recent eight validation observations, with a transition after two consecutive failures to reject. Validation observations are collected every two training iterations, so consecutive windows overlap by seven observations. The OLS standard error of the slope assumes independent errors; overlapping windows induce autocorrelation, making p-values anti-conservative and the test more likely to declare a positive trend than warranted. Because a declaration of positive trend keeps the policy in the current stage, this bias could delay transitions. Conversely, repeated testing at every evaluation without multiplicity adjustment increases the chance of spurious 'no positive trend' results, causing premature transitions despite the two-strikes rule. The paper provides no sensitivity analysis for window length (8), significance level (0.05), or the two-consecutive-failures rule, and no robustness check with autocorrelation-consistent inference. The empirical cost is material: trend-based progression gives 50.3 vs 49.6 for fixed-stage (Table 7). If the scheduler's decisions are even partially noisy, the claimed 1.1-point margin over VeRPO (49.2) could shrink to the 0.4-point margin of fixed-stage, which may no longer be significant. This is the load-bearing component that distinguishes DHRCL from a fixed curriculum.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DHRCL, a reinforcement-learning post-training framework for code LLMs. It decomposes program-level feedback into four dense signals (syntax validity, execution success, unit-test pass rate, and AST-based structural similarity) and arranges them into a three-stage curriculum (Syntax; Execution; Pass & Structural). Stage boundaries are not fixed in advance; they are determined by a trend test on a held-out validation split. The paper also introduces stage-dependent token-level credit reweighting: confidence-based weighting in the syntax stage, uniform weighting in the execution stage, and uncertainty-based weighting in the final stage. Experiments use a matched Qwen3-8B/KodCode protocol with all baselines retrained, three seeds, problem-level paired-bootstrap CIs, and scale studies on Qwen3-4B/8B/14B. DHRCL reports the highest average Pass@1 (50.3±0.2 on Qwen3-8B) and consistent gains over the strongest baseline VeRPO across scales.","tokens_in":17915,"tokens_out":6533,"duration_ms":76919,"significance":"If the reported gains hold, DHRCL is a valuable empirical contribution: it shows that a hierarchical dense reward combined with a curriculum can improve code-RL performance without an external reward model, and it provides a carefully matched comparison protocol. The experimental design is a clear strength: all baselines are retrained under the same data, model, decoding, and budget conditions; three seeds are used; paired-bootstrap CIs support the headline comparisons; and the ablations isolate reward composition, curriculum scheduling, and token weighting. The central concern is that one of the paper's claimed contributions, the trend-based automatic stage progression, rests on a statistically fragile scheduler, and the ablation evidence for it lacks inferential tests. The empirical margin attributable to this component is small (about 0.7 Pass@1 points over a fixed schedule), so the validity of the scheduler is load-bearing for the paper's central claim.","major_comments":[{"comment":"The stage scheduler uses a one-sided OLS trend test on the most recent eight validation observations, with validation performed every two iterations. Consecutive windows overlap by seven observations, so the OLS standard errors are not valid under the independence assumption. The test is also repeated at every evaluation without multiplicity adjustment, and no sensitivity analysis is given for the window length (8), the significance level (0.05), or the two-consecutive-failure rule. This is not a purely cosmetic issue: Table 7 shows trend-based progression at 50.3 vs. 49.6 for fixed-stage, so the scheduler contributes only about 0.7 points over a fixed schedule. If that contribution is an artifact of the invalid trend test, the head-to-head margin over VeRPO (Table 1: 1.1 points) could shrink to roughly 0.4 points, which may no longer be significant. I ask for autocorrelation-robust infe","section":"Trend-Based Automatic Stage Progression"},{"comment":"The headline comparisons against baselines are supported by paired-bootstrap CIs, but the ablations used to substantiate the individual components report only means and seed standard deviations. The key contrasts are all small: Trend-based vs. Fixed-stage (50.3 vs. 49.6, Table 7), Full vs. No Weighting (50.3 vs. 49.4, Table 7), and Full Reward vs. Pass+Syntax+Exec (50.3 vs. 49.7, Table 6/Table 4). With three seeds, these differences may or may not be statistically reliable; the manuscript does not provide inferential evidence. Since these ablations are load-bearing for the claims that the curriculum and the token-weighting ordering are beneficial, I request paired-bootstrap CIs (or equivalent) for the ablation contrasts as well.","section":"Ablation Studies"}],"minor_comments":[{"comment":"In Tables 1 and 2, the column 'Ext. RM' is not defined in the text or captions. It appears to refer to 'external reward model', but this should be stated explicitly.","section":"Comparison Experiments"},{"comment":"The DGR comparison across methods is potentially confounded by reward scale: the degeneracy threshold is an absolute value (Std < 1e-2) applied to raw rewards, and different methods use different reward scales and normalization schemes. Consider normalizing rewards to a common scale before computing DGR, or softening the cross-method DGR comparison.","section":"Training-Efficiency and Reward-Signal Metrics"},{"comment":"Only median transition iterations are reported (e.g., 140/280 for Qwen3-8B in Table 7). Given the importance of the scheduler, report the full distribution across seeds (e.g., min/max or individual seed values).","section":"Trend-Based Automatic Stage Progression"},{"comment":"The phrase 'established token patterns' is informal. The mechanism uses detached token probability, and the paper later acknowledges that token probability does not identify semantic correctness. A short formal definition of what 'established' means would help reproducibility.","section":"Stage-Aware Probability-Based Token Credit Redistribution"}],"recommendation":"major_revision","confidential_remarks":"The empirical study is careful and the head-to-head comparison is well executed. The stress-test concern about the trend-based scheduler lands: it is a genuine statistical validity issue in a load-bearing component, and the paper currently offers no robustness evidence. I would be willing to accept after the authors supply (i) a sensitivity analysis or autocorrelation-consistent version of the trend test, and (ii) inferential statistics for the ablation contrasts. There is no circularity concern: the validation split is held out and final evaluation is on external benchmarks. The DGR confound is secondary and can be handled in a minor revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: DHRCL is a careful, well-executed empirical study of a modest RL recipe for code LLMs. The headline gain—about +1 Pass@1 over a strong verifiable-dense-reward baseline across three scales, with paired-bootstrap CIs that don't include zero—looks real. The weak spot is the automatic stage scheduler's statistical foundation; it's fragile enough that the paper's main novelty claim needs extra work.\n\nWhat's new: the specific combination of dense hierarchical rewards (syntax/execution/pass-rate/AST), a three-stage curriculum with trend-based automatic progression, and stage-aware token credit redistribution. Individually the ingredients appear in prior work; the combination and the ordering of the token weighting are new. The paper is honest about this: AST similarity is explicitly auxiliary, not semantic equivalence; NET/NTMU are descriptive; baselines are retrained under matched protocol. That's exactly the right way to run an empirical comparison.\n\nWhat it does well: unified protocol, three seeds, paired bootstrap, ablations isolating reward composition, curriculum scheduling, and token weighting. The ablations support the ordering claim: the full consolidation-to-uniform-to-refinement weighting beats reversed and partial variants, and trend-based progression beats fixed-stage by 0.7 points. The held-out validation split for stage transitions with external test benchmarks keeps the evaluation clean.\n\nThe soft spots, in proportion: the trend-based scheduler is the load-bearing part of the novelty, and its statistical machinery is underbuilt. The paper fits a linear trend to the most recent eight validation observations, which are collected every two iterations and therefore overlap by seven; OLS standard errors assume independence, so the p-values are anti-conservative. Repeated testing at every evaluation without multiplicity adjustment further weakens the rule. The authors give no sensitivity analysis for window length, alpha, or the two-consecutive-failures rule, and no autocorrelation-consistent inference. If the scheduler sometimes transitions at the wrong time, the 0.7-point advantage over fixed-stage could shrink; the 1.1-point margin over VeRPO would probably survive because it's not solely due to the scheduler, but it's not guaranteed. I'd like to see a robustness check with, say, a simple threshold on the smoothed validation metric or a change-point detector, plus reporting of when transitions actually occurred across seeds. Second, no code is released, which makes the exact AST similarity and syntax reward hard to reproduce; that's a significant omission for an empirical paper. Third, the token-credit weighting is a heuristic; the ablations are encouraging, but there's no analysis of how the per-token reweighting interacts with the PPO/GRPO clip or why trajectory-level mean-one normalization is the right constraint. That's a minor issue given the ablation evidence.\n\nThis is a paper for people working on RL post-training for code LLMs. It deserves a serious referee; the central empirical claim is credible, but the scheduler needs a sensitivity analysis and the code should be released before I'd fully trust the details.\n\nRecommendation: send to peer review. The authors know the limits; they state many of them. The main thing to push on is the trend-test robustness, not the overall method.","headline":"A well-run empirical paper with a modest but credible gain; the novel scheduler's statistics are fragile enough that it needs a sensitivity analysis before I'd trust the 0.7-point advantage over fixed-stage.","tokens_in":18359,"tokens_out":2177,"would_cite":true,"duration_ms":24271,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DHRCL claims that ordering code-RL rewards by prerequisite—syntax, execution, then tests and structure—with stage durations set from validation trends, raises average Pass@1 to 50.3±0.2 on Qwen3-8B, beating the strongest verifiable dense-re","keywords":["reinforcement learning","code generation","curriculum learning","dense rewards","LLM post-training","AST structural similarity","group relative policy optimization","Qwen3"],"falsifier":"Re-run DHRCL with stage durations set by an oracle or by non-overlapping validation windows with multiplicity-corrected trend tests; if the Pass@1 advantage over VeRPO falls below the reported 95% CI [0.6,1.5], the automatic progression rule, not the hierarchical rewards, is carrying the gain.","tokens_in":17496,"feed_emoji":"🤖","tokens_out":6146,"duration_ms":58655,"temperature":0.7,"pith_summary":"Reinforcement learning for code generation usually gives the model one scalar outcome—did the tests pass—or a fixed blend of signals. This paper argues that syntax validity, executability, unit-test pass rate, and program structure are prerequisite skills that should be trained in order, not weighted together from the start. Its method, DHRCL, schedules these four rewards through a three-stage curriculum, decides when to advance stages from validation trends instead of hand-set thresholds, and redistributes each trajectory's gradient across tokens according to how established the tokens are. Under a matched Qwen3-8B and KodCode protocol, DHRCL reports an average Pass@1 of 50.3±0.2, 1.1 points above VeRPO, the strongest baseline, and the advantage persists across Qwen3-4B, 8B, and 14B. If right, it suggests staged, verifiable dense rewards can substitute for learned reward models in code RL.","feed_headline":"Staged rewards lift code-RL Pass@1 past top dense baseline","feed_subtitle":"A syntax-to-tests curriculum adds 1.1 points over the strongest baseline and holds gains from 4B to 14B","key_machinery":"The load-bearing object is the stage-aware hierarchical reward coupled with the curriculum rule. Syntax validation rate is computed by an error-tolerant parser over statement- and block-level units weighted by token length; execution success is a 0/1 indicator; pass rate is the fraction of unit tests passed; AST similarity is tree-edit distance between normalized ASTs of generated and reference programs. Each stage combines pass rate with one primary signal and gates the remaining terms by a batch-level competence coefficient, set to the batch mean of that stage's primary reward. Stage transitions are decided by a one-sided test on the slope of the most recent eight validation observations a","core_discovery":"The paper's central claim is that code-generation RL should not present syntax validity, executability, unit-test pass rate, and AST-based structural similarity as one fixed reward mixture. These four signals describe progressively dependent capabilities, so DHRCL trains them in a fixed order—Syntax, then Execution, then Pass & Structural—while keeping the pass-rate reward active throughout. Stage boundaries come from a one-sided linear-trend test on the most recent eight held-out validation observations, with a transition only after two consecutive failed tests, and the best checkpoint of the stage is restored before advancing. Within each stage, trajectory-level advantages are redistribute","pith_inferences":["Editorial inference: the AST reward's contribution (+0.6 Pass@1, +0.06 AST similarity) suggests that any reference-conditioned structural prior, such as dataflow graphs or type annotations, could behave similarly, but this is not tested here.","Editorial inference: because the margin over VeRPO narrows from 1.3 to 0.7 points as the model grows from 4B to 14B, the curriculum may matter most where the base policy is weakest; larger models might need fewer or shorter stages.","Editorial inference: the probability-based credit redistribution recipe is general enough to apply to any RL task with staged verifiable objectives—for example theorem proving or data transformation—though the paper only evaluates code generation.","Editorial inference: the trend-based progression rule could be made more robust by change-point detection or multiplicity-corrected tests, and the paper's own overlapping-window design leaves that as an open improvement."],"forward_implications":["Heterogeneous code rewards can be ordered by prerequisite rather than weighted statically, with functional correctness kept active at every stage.","Stage duration can be set by validation-trend monitoring instead of manual thresholds; the rule selected median transition iterations of 140/280 on Qwen3-8B, and varied with model scale.","Trajectory-level credit can be redistributed by token confidence without an external reward model, lowering the average degenerate group ratio to 0.20.","The full DHRCL recipe reaches the GRPO reference validation Pass@1 in about half the iterations (0.49×) and about 55% of the GPU hours.","DHRCL reduces syntax errors, runtime errors, and timeouts relative to both the base model and VeRPO; executable-but-incorrect programs remain the dominant residual failure mode."],"supporting_citations":[{"why":"GRPO is the group-based policy-optimization backbone and also the binary-reward baseline all methods are compared against.","marker":"Shao et al. 2024"},{"why":"VeRPO is the strongest verifiable-dense-reward baseline and sets the protocol DHRCL is evaluated against.","marker":"Wang et al. 2026b"},{"why":"KodCode supplies the training corpus and held-out validation split used for curriculum progression, early stopping, and checkpoint selection.","marker":"Xu et al. 2025"},{"why":"Qwen3-4B, Qwen3-8B, and Qwen3-14B are the backbones on which every method is retrained.","marker":"Team 2025"},{"why":"AceCoder is the external reward-model baseline; its adapted training uses the released AceCodeRM.","marker":"Zeng et al. 2025"},{"why":"HumanEval is one of the six external benchmarks and the origin of the Pass@1 evaluation protocol.","marker":"Chen et al. 2021"},{"why":"BigCodeBench-Full and BigCodeBench-Hard are two of the six external benchmarks used for final evaluation.","marker":"Zhuo et al. 2025"},{"why":"LiveCodeBench V6 is one of the six external benchmarks used for final Pass@1 reporting.","marker":"Jain et al. 2025"},{"why":"CodeElo is one of the six external benchmarks used for final Pass@1 reporting.","marker":"Quan et al. 2025"}],"fun_headline_variants":["Hierarchical rewards and auto-curriculum lift code RL Pass@1","Code curriculum of syntax, execution, tests, structure beats sparse RL","Auto-paced code RL stages rewards from syntax to structure","Hierarchical rewards, curriculum learning lift code RL Pass@1 by 1.1","Three-stage code RL curriculum outdoes dense baselines across model sizes"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The trend-based stage switch assumes the last eight validation measurements of the stage metric follow a straight line, and that two consecutive failed positive-trend tests reliably mean the metric has plateaued, even though the measurements overlap and are tested repeatedly.","fun_headline_variants_meta":{"raw":{"variants":["Hierarchical rewards and auto-curriculum lift code RL Pass@1","Code curriculum of syntax, execution, tests, structure beats sparse RL","Auto-paced code RL stages rewards from syntax to structure","Hierarchical rewards, curriculum learning lift code RL Pass@1 by 1.1","Three-stage code RL curriculum outdoes dense baselines across model sizes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00102,"raw_usage":{"total_tokens":4163,"prompt_tokens":786,"completion_tokens":3377,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":3299}},"tokens_in":530,"tokens_out":3377,"duration_ms":28062,"temperature":1.0,"reasoning_tokens":3299,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T04:21:42.693278+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run DHRCL with stage durations set by an oracle or by non-overlapping validation windows with multiplicity-corrected trend tests; if the Pass@1 advantage over VeRPO falls below the reported 95% CI [0.6,1.5], the automatic progression rule, not the hierarchical rewards, is carrying the gain.","supporting_citations":[],"review_version":2}