{"id":"01633dde-1bf4-4398-8b69-a415aa3d92f0","arxiv_id":"2505.10594","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A three-stage training pipeline (synthetic problems, multi-agent chain-of-thought data, step-DPO self-improvement) raises LiveCodeBench pass@1 for Qwen2.5-Coder bases to 21.88 (7B) and 35.08 (32B).","lead":"CRPE is a three-stage pipeline that creates code problems, synthesizes chain-of-thought reasoning traces with a multi-agent setup, and then self-improves a model with step-level preference optimization. It reports a 32B code model at 35.08 pass@1 on LiveCodeBench, slightly above GPT-4O, and a 7B model at 21.88, which is useful evidence for practitioners building reasoning-oriented coding models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Both SFT and Step-DPO training labels come from the pipeline's own LLM-generated tests and LLM checkers; until label accuracy is audited, the claimed reasoning gains may be an artifact of weak correctness signals.","rationale":"The reader's weakest assumption is the same load-bearing concern I identify: the correctness signal used to build the Code-COT SFT data and Step-DPO preference pairs is generated by the pipeline itself, and no independent validation of those labels is reported. I agree with that assessment, and I add a concrete audit that would settle whether the concern actually lands. I do not think the concern alone warrants rejection: the pipeline is described in enough detail that the labels could be reliable, and the final LiveCodeBench scores are plausible if the labels are accurate. But the central causal claim—that the gains come from improved reasoning—is conditional on label reliability, and the paper currently provides no evidence for that reliability. The other issues noted by the reader (missing artifacts, inconsistent numbers, no error bars) are real but secondary; they affect reproducibility and trust rather than the internal logic of the training signal. Because my read does not change the reader's CONDITIONAL verdict, I mark the verdict as UNCHANGED.","tokens_in":16689,"tokens_out":5194,"duration_ms":56400,"concrete_test":"Take a random sample of 100–200 problems from the 6k synthesized problem set (or from the Code-COT dataset) and 50 Step-DPO tree nodes. For each item, have two independent programmers write hidden tests covering edge cases and constraints, with verified expected outputs. Run the paper's exact Test-Generator/Result-Checker pipeline on the retained 'accepted' solutions and on known-buggy mutated versions. Compute label error rates: false accepts (LLM tests pass but hidden tests fail) and false rejects (hidden tests pass but LLM tests fail). Also compare LLM-generated-test verdicts with official test verdicts on a subset of the 4,749 open-source problems where official tests exist.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim is that CRPE improves code generation by improving code reasoning, and that claim depends on the correctness labels used to select SFT data and Step-DPO preference pairs. Section III.C.3 says that when a problem lacks provided test cases, the Execution Agent's Test Generator creates test cases and the Result Checker—also LLM-driven—determines whether the execution output matches the expected output. Section IV.A repeats this for self-exploration, using Qwen2.5-7B-Instruct or Qwen2.5-32B-Instruct to generate test code and perform correctness checks for problems without unit tests. Since the synthesized code problem set is used heavily in training, a large fraction of the accept/reject signal comes from this self-made test infrastructure, without any reported validation against hidden or human-written tests. If the generated tests are too few, too easy, or have incorrect expected outputs, then incorrect reasoning paths can be labeled 'accepted' and genuinely correct ones can be labeled 'rejected.' Step-DPO's chosen/rejected child nodes are defined directly by these accept/reject labels, so noise propagates into the preference objective. The CoT-vs-direct ablation in Section IV.C.1 inherits the same bias because the 'correct answers generated by the LLM' in that comparison were selected by the same pipeline. The paper does not report a manual audit, a false-positive rate, a false-negative rate, or any comparison between its LLM-generated tests and independent test oracles. Without such evidence, the benchmark gains could come from the model learning the format of accepted paths, overfitting to easy self-tests, or other confounds, rather than from genuine reasoning improvement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CRPE, a three-stage pipeline for synthesizing code reasoning data and training code LLMs: (i) generation and filtering of large-scale code problem instructions, (ii) multi-agent synthesis of chain-of-thought reasoning data for supervised fine-tuning (SFT), and (iii) tree-search self-exploration with Step-DPO for iterative self-improvement. The authors report pass@1 results on LiveCodeBench (20240701-20240901), claiming that COT-Coder-7B-StepDPO reaches 21.88 and COT-Coder-32B-StepDPO reaches 35.08, outperforming GPT-4O. The central claim is that explicitly training on reasoning processes, rather than only on final answers, improves code generation capability.","tokens_in":16936,"tokens_out":5277,"duration_ms":46581,"significance":"If the reported gains are real, CRPE offers a practical, open recipe for improving code reasoning in LLMs without training a separate reward model. The controlled CoT-versus-direct comparison in Section IV.C.1 is a reasonable way to isolate the value of reasoning-process data, and the decontamination step against LiveCodeBench is a genuine safeguard. The manuscript also provides full prompt templates and training hyperparameters, which supports reproducibility. However, the absence of validation of the correctness labels used to construct SFT and Step-DPO data, combined with small unquantified performance gaps and overreaching summary claims, means the headline contributions are not yet fully established.","major_comments":[{"comment":"The abstract claims that COT-Coder-7B-StepDPO 'exceeds all models with similar or even larger sizes,' but Table I reports Llama3.3-70B-Instruct at 26.88 and Qwen2.5-Coder-32B-Instruct at 29.71, both higher than the 7B model's 21.88. Additionally, Section IV.B states that 'CRPE-32B model achieved a score of 34.22,' while Table I lists COT-Coder-32B-StepDPO at 35.09 and the abstract gives 35.08. These numerical contradictions must be resolved because the headline comparison is a central claim.","section":"Abstract and Section IV.B, Table I"},{"comment":"The correctness labels that select SFT data and Step-DPO preference pairs are produced by the pipeline's own LLM-based Test Generator and Result Checker, with no reported audit against human-written or hidden tests. Because Step-DPO's chosen/rejected child nodes are defined directly by these accept/reject labels, systematic noise in the labels could produce the observed benchmark gains without genuine improvement in reasoning ability. The authors should report a manual audit (e.g., false-positive and false-negative rates on a sample with known correct answers) or compare LLM-generated tests against independent held-out tests, and quantify how label noise affects the Step-DPO training signal.","section":"Section III.C.3 and Section IV.A"},{"comment":"The main evaluation reports a single pass@1 estimate per model with no error bars, confidence intervals, or multiple seeds. The SFT-to-StepDPO deltas (20.18 vs 21.88 for the 7B model; 33.49 vs 35.09 for the 32B model) are small enough that sampling noise could change the conclusions. Please provide variance estimates, for example via multiple evaluation seeds or bootstrap confidence intervals on pass@1.","section":"Section IV.A and Table I"},{"comment":"The comparison set lacks reasoning-oriented baselines that are most relevant to the paper's claim of expanding reasoning capability, such as OpenAI o1, QwQ, or other Step-DPO/process-supervised code reasoning models. Without these baselines, the improvement over generic instruction-tuned models does not establish the specific contribution of the proposed reasoning-enhancement method.","section":"Section II and Table I"}],"minor_comments":[{"comment":"The phrases 'Our CRPE-7B model scored 21' and 'CRPE-32B model achieved a score of 34.22' should be updated to match the exact numbers in Table I (21.88 and 35.09).","section":"Section IV.B"},{"comment":"The post-hoc correction of function names (HumanEval improvement to 74.4/68.9 and MBPP to 78.6/66.1) is not an official benchmark result and should be clearly labeled as an analysis of failure modes rather than as a score.","section":"Section IV.C.2"},{"comment":"The paper uses 'COT' and 'CoT' interchangeably, and 'step-DPO' appears alongside 'Step-DPO' in several places; please standardize the terminology.","section":"Throughout"},{"comment":"Reference [13] appears twice with inconsistent capitalization, and the bibliography contains many missing line breaks; the authors should run a standard citation formatting pass.","section":"References"},{"comment":"The paper says 'we ultimately gather several thousand code instructions' in Section III.B.1, but Section IV.A reports 4,749 open-source code problems; please harmonize these numbers.","section":"Section III.B.1 vs Section IV.A"},{"comment":"The prompt templates in Figures 7-12 are not explicitly referenced in the experimental setup; adding pointers to the relevant sections would improve reproducibility.","section":"Appendix"}],"recommendation":"major_revision","confidential_remarks":"The paper is methodologically interesting, but the abstract and main text currently contain contradictory numerical claims that need to be reconciled. More importantly, the absence of any audit of the LLM-generated correctness labels is a substantive concern that the authors must address before the results can be taken at face value. Please also ensure that the evaluation includes variance estimates and more relevant reasoning-model baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Verdict first: this is a sensible pipeline paper with eye-catching LiveCodeBench numbers, but the evidence as written is not yet sufficient to take those numbers at face value. The core issue is exactly what the stress-test flags: the training labels come from the pipeline's own LLM-generated tests and LLM checkers, with no audit of that label quality.\n\nWhat is genuinely useful: the three-stage design is clean and reproducible in principle — problem synthesis via Evol/OSS-instruct, multi-agent CoT synthesis, and tree-search Step-DPO is a reasonable integration of existing ideas. The CoT-versus-direct ablation in Table II is a nice controlled check that reasoning data helps, and the decontamination step is a real safeguard. Training details are specific enough to re-implement. The pipeline and the resulting numbers, if they hold, would be a practical contribution for practitioners without access to large proprietary reasoning models.\n\nNow the soft spots. The stress-test is right to press on label quality. For problems without provided tests, the Execution Agent's Test Generator and Result Checker are both LLMs, and the same holds during self-exploration. The paper reports no false-positive/negative rates, no manual audit, and no comparison between its generated tests and independent oracles. Since Step-DPO preference pairs are selected by these accept/reject labels, noise here propagates directly into the training signal. The CoT-versus-direct ablation partly avoids this because it uses problems with known correct answers, but the main SFT and DPO data do not share that guarantee. This is the load-bearing uncertainty.\n\nSeveral smaller issues compound it: no artifact release despite the \"open-source\" claim, no error bars or repeated seeds on a 106-problem benchmark, and inconsistent numbers (abstract 35.08, table 35.09, prose 34.22). The baseline table omits the reasoning-specialized models the paper claims to approach. None of these are fatal alone, but together they mean the central claim is not yet firm.\n\nWho is this for: practitioners building code reasoning models who want a concrete recipe, and researchers interested in self-improvement with synthetic labels. It deserves a serious referee, not a desk reject. The reviewer should ask for artifact links, a label-quality audit (or at least a manual sample), variance estimates, and a consistent baseline table. If those come back solid, this could be a useful contribution.","headline":"A plausible three-stage recipe for code reasoning that would be worth believing once the label-quality and artifact issues are addressed.","tokens_in":17570,"tokens_out":2691,"would_cite":false,"duration_ms":26106,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-stage data-and-training pipeline claims to turn base code models into benchmark-leading code reasoners, with one 7B model beating instruction-tuned models of comparable size and one 32B model beating GPT-4O on LiveCodeBench.","keywords":["code generation","chain-of-thought","Step-DPO","tree search","synthetic data","multi-agent framework","LiveCodeBench","self-improvement"],"falsifier":"Take a random sample of the code problems used in training, replace the LLM-written test cases and the LLM result checker with independent, human-written unit tests, and re-score every 'accepted' reasoning path and every Step-DPO preference pair; if a substantial fraction of accepted paths fail the independent tests, or if retraining on only independently verified pairs wipes out the reported LiveCodeBench gains, the central claim is false.","tokens_in":16461,"feed_emoji":"🧠","tokens_out":11627,"duration_ms":108766,"temperature":0.7,"pith_summary":"The paper claims that the missing ingredient in code generation is not more solve data but the reasoning process itself. It introduces CRPE, a three-stage pipeline that starts from a base code model with no instruction tuning: first it synthesizes and filters hard programming problems, then a multi-agent setup driven by a strong direct-answer model produces step-by-step chain-of-thought reasoning traces with final code, and finally the fine-tuned model explores alternate reasoning paths by tree search and is refined with step-level preference optimization. The authors report pass@1 scores of 21.88 for the 7B model and 35.08 for the 32B model on the LiveCodeBench 2024-07-01 to 2024-09-01 window, with the 32B model surpassing GPT-4O's reported score. If true, the practical point is that explicit reasoning traces and step-wise self-preference data can be synthesized and reused, so smaller open models move closer to closed reasoning models without a separate reward model.","feed_headline":"Three training stages lift a code model past GPT-4O on LiveCodeBench","feed_subtitle":"CRPE synthesizes hard problems, distills expert reasoning traces, then self-improves via tree-search step preferences.","key_machinery":"The load-bearing machinery is the three-stage loop. Stage one is a code problem pipeline that extracts code snippets from a large open corpus, prompts an LLM to write new problems inspired by them, applies Evol-Instruct-style rewriting to make them harder, filters for clarity, difficulty, and self-containedness, and decontaminates against the test benchmark. Stage two is the Code-COT Maker, a multi-agent workflow in which a strong direct-answer model plays a Thinking Agent that emits step-by-step reasoning, a Reflection Agent that judges whether the reasoning is on track and analyzes execution errors, and an Execution Agent that compiles and runs the code; when no tests exist, an LLM Test Generator writes test cases and an LLM Result Checker decides yes/no on the output. Stage three is the self-improvement loop: the SFT model samples complete reasoning paths from each node of a tree, nodes whose sampled paths all fail are marked rejected and nodes that lead to a correct answer are accepted, and accepted/rejected child-node pairs become preference data for Step-DPO, a preference objective that increases the probability of the next correct reasoning step and decreases the next wrong one. The chain-of-thought traces are delimited with <thinking>, <reflection>, and <ChainOfThought> tags, which give the model a concrete output structure to imitate.","core_discovery":"On its own terms, CRPE establishes that a code model can be upgraded by treating reasoning as a trainable intermediate product. Starting from Qwen2.5-Coder-7B/32B-Base, the pipeline produces COT-Coder-StepDPO models whose LiveCodeBench pass@1 scores (21.88 for 7B, 35.08 in the abstract and 35.09 in Table I for 32B) exceed the instruction-tuned Qwen2.5-Coder baselines (15.94 and 29.71) and, for 32B, the GPT-4O score of 33.6. The reasoning data is the carrier: supervised fine-tuning on 2,810 high-quality reasoning traces beats fine-tuning on direct answers in the paper's controlled comparison (19.15 vs 16.60 on partial data), and step-DPO on tree-search preference pairs adds further gains. The conclusion the paper draws is that synthesizing the process, not just the final code, is what improves code generation ability.","pith_inferences":["Editorial inference: the correctness signal in both the expert-data stage and the self-improvement stage is produced by LLMs that also write the tests; an important extension is to audit how often LLM-written tests accept wrong code and whether removing such cases changes the ranking.","Editorial inference: the paper does not isolate whether the gains come from genuinely better reasoning or from learning a well-structured chain-of-thought format; a control that trains on the same reasoning traces with randomly shuffled steps would separate format learning from reasoning learning.","Editorial inference: the Step-DPO preference pairs are sampled from the model being trained, so the loop may reinforce the model's own error patterns; a test using preference pairs generated by a different model would show how much of the gain is self-contained.","Editorial inference: since decontamination is done against the specific LiveCodeBench window, a natural next experiment is to run the identical pipeline on a later benchmark window and compare the score gap to the original result."],"forward_implications":["If CRPE works as reported, a base code model can gain several points of LiveCodeBench pass@1 without hand-labeling reasoning traces or training a separate reward model.","The controlled comparison of reasoning-annotated answers versus direct answers implies that the marginal value in code SFT data is the reasoning trace, so dataset builders should spend budget on process annotation rather than only on more solutions.","Tree-search sampling makes preference pairs a byproduct of exploration, which means the improvement loop can be run iteratively: the better the model becomes, the more reliable accepted nodes become for the next Step-DPO round.","Because the reported numbers are on a decontaminated benchmark window, the framework's real test is whether the same models hold their advantage on later coding benchmark windows without further training."],"supporting_citations":[{"why":"Supplies the Step-DPO objective used to train on step-wise preference pairs in the third stage.","marker":"[26]"},{"why":"Provides the tree-search expand-and-simulate procedure that the paper adapts to harvest accepted and rejected reasoning nodes.","marker":"[27]"},{"why":"Supplies the Qwen2.5-Coder base models and the n-gram decontamination method used before training.","marker":"[13]"},{"why":"Contributes the OSS-Instruct approach that the paper extends to synthesize competition-level code problems from code snippets.","marker":"[9]"},{"why":"Supplies the Evol-Instruct method used to rewrite and make the synthesized code problems harder.","marker":"[7]"},{"why":"Provides the Stack v2 seed corpus from which code snippets are extracted for problem synthesis.","marker":"[17]"},{"why":"Founds the preference-optimization formulation that Step-DPO refines to the step level.","marker":"[25]"}],"fun_headline_variants":["CRPE training beats GPT-4O on LiveCodeBench with 35.08 pass@1","Reasoning traces as product: CRPE pushes code model past GPT-4O","Process over product: CRPE's step-DPO lifts code LLM to 35.08","32B code model trained by CRPE tops GPT-4O on LiveCodeBench","CRPE: three-stage synthesis yields code model that beats GPT-4O"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire result depends on the AI-written tests and the AI yes/no checker being reliable judges of whether a solution is correct, because those labels decide which reasoning traces are kept in the SFT data and which reasoning steps are preferred during Step-DPO.","fun_headline_variants_meta":{"raw":{"variants":["CRPE training beats GPT-4O on LiveCodeBench with 35.08 pass@1","Reasoning traces as product: CRPE pushes code model past GPT-4O","Process over product: CRPE's step-DPO lifts code LLM to 35.08","32B code model trained by CRPE tops GPT-4O on LiveCodeBench","CRPE: three-stage synthesis yields code model that beats GPT-4O"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000806,"raw_usage":{"total_tokens":3566,"prompt_tokens":1001,"completion_tokens":2565,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":2452}},"tokens_in":617,"tokens_out":2565,"duration_ms":17541,"temperature":1.0,"reasoning_tokens":2452,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:17:20.924682+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the code problems used in training, replace the LLM-written test cases and the LLM result checker with independent, human-written unit tests, and re-score every 'accepted' reasoning path and every Step-DPO preference pair; if a substantial fraction of accepted paths fail the independent tests, or if retraining on only independently verified pairs wipes out the reported LiveCodeBench gains, the central claim is false.","supporting_citations":[{"cited_title":"Step-dpo: Step-wise preference optimization for long-chain reasoning of llms,","cited_arxiv_id":null,"evidence_quote":"Supplies the Step-DPO objective used to train on step-wise preference pairs in the third stage."}],"review_version":1}