{"id":"f5ff9a78-30e6-4bc0-982b-7d7482b3bd39","arxiv_id":"2507.22080","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CodeEvo uses two interacting LLM agents with keyword-guided instruction evolution and hybrid compiler-plus-LLM feedback to synthesize high-quality instruction-code pairs for fine-tuning code models.","lead":"This paper presents CodeEvo, a system that uses two AI agents, a Coder and a Reviewer, to automatically generate programming problems and verified Python solutions, creating training data for code models. The authors report that models fine-tuned on CodeEvo data outperform models trained on older synthetic data methods on standard coding benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The hybrid filter's acceptance criterion is not independently verified: the same Reviewer that proposes instructions also judges them, and compiler feedback only checks the Coder's self-written tests, so CodeEvo's data-quality and data-efficiency claims rest on unvalidated self-assessment.","rationale":"I read the paper as making a usable and potentially valuable empirical claim: CodeEvo-synthesized data improves downstream code models more than comparable baselines per training example. The reader's weakest-assumption analysis correctly identifies the quality of the Reviewer-based filter as the linchpin. The paper has genuine supporting evidence: a matched Evol-Instruct baseline with the same seed distribution and backbones, a w/o-seed ablation, a scaling curve, and planned public release of data and code. However, none of these directly verify the correctness of the retained pairs. The OSS-Instruct comparison is indeed confounded by different seed distributions and data sources, and the absence of variance/significance testing is a real weakness for the small margins on already-strong models like Qwen2.5-Coder-7B. Yet those issues weaken the quantitative precision of the claim rather than its core mechanism. The filter-quality concern is more load-bearing because the paper's own explanation of why CodeEvo is data-efficient is that hybrid feedback removes invalid, ungrounded samples; if that filter is not actually delivering high correctness, the central mechanism-level explanation collapses even if some downstream gains remain. The proposed concrete test—independent unit tests and a human check on a random sample—would settle whether the concern lands. My verdict does not change the reader's CONDITIONAL: the condition should be that the authors provide such independent validation or clearly reframe the quality claim.","tokens_in":19501,"tokens_out":8014,"duration_ms":103282,"concrete_test":"Independently validate a random sample of 200 instruction-code pairs from CodeEvo-17K and 200 from the reproduced Evol-Instruct-25K. For each pair, have a separate strong LLM (not the Reviewer, e.g., a different model family) generate 10 additional unit tests directly from the instruction alone, blind to the original code and the Coder-written tests, then execute all pairs against these independent tests. Additionally, have one human expert judge a 50-pair subset. If CodeEvo pairs do not pass the independent tests at a substantially higher rate than Evol-Instruct pairs, the hybrid-feedback quality advantage is not established and the data-efficiency claim needs reframing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central data-efficiency claim—CodeEvo outperforms baselines while using 4–5x less data—depends on the premise that the hybrid feedback mechanism retains genuinely correct, well-grounded instruction-code pairs. In Section 3.3 and Algorithm 1, retention is decided by fhybrid = Reviewer(fcomp, fNL): fcomp is compiler pass/fail on test cases g that the Coder wrote for its own solution, and fNL is the Reviewer's natural-language assessment of logical alignment. There is no independent test suite, hidden test, or external ground truth in the loop. If the Reviewer is systematically lenient, or if g has poor coverage, invalid pairs enter the 17K training set while still being labeled 'valid'. The paper's own Limitations section concedes this risk: 'it remains difficult to ensure complete correctness, especially at scale' (Test Case Quality). The survival-rate analysis in Section 5.5 is self-reported and does not validate correctness against any independent oracle. If the filter merely selects instructions that the Reviewer itself can recognize or solve, the observed gains over Evol-Instruct could come from keyword-guided difficulty and format rather than from verified functional correctness, which would undermine the mechanism-level explanation of the data-efficiency advantage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CodeEvo, a dual-agent framework for synthesizing instruction-code pairs to train code-generation LLMs. A Coder agent generates candidate solutions and self-written test cases, while a Reviewer agent produces keyword-conditioned new instructions and judges candidate solutions through a hybrid feedback mechanism that combines compiler pass/fail signals with natural-language assessment. The authors construct a dataset (CodeEvo-100K, with 17K examples used in experiments), fine-tune four base models, and report pass@1 on HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench, claiming consistent improvements over Evol-Instruct and OSS-Instruct baselines as well as superior data efficiency. The paper also presents analyses of instruction diversity, difficulty, scaling behavior, and data survival rates.","tokens_in":19761,"tokens_out":6915,"duration_ms":68910,"significance":"If the results hold, CodeEvo provides a data-efficient and largely reference-free method for synthesizing code-centric training data, reducing reliance on human curation and on large or proprietary generator models. The keyword-guided instruction evolution and the hybrid verification loop are interesting and potentially transferable ideas, and the paper ships code and data (per the abstract). However, the current evidence is weakened by the absence of variance reporting, a confounded comparison against OSS-Instruct, and the lack of any independent validation of the self-assessed filtering criterion. The significance is therefore conditional on addressing these experimental-rigor issues.","major_comments":[{"comment":"The central claim that CodeEvo-trained models \"significantly outperform\" baselines is not supported statistically. No variance, confidence intervals, or significance tests are reported for any pass@1 number, and several differences are within one point (e.g., DeepSeek-Coder-6.7B HumanEval: CodeEvo 77.4 vs OSS-Instruct 76.8; HE+ 71.3 vs 70.7; StarCoder2-7B HumanEval: 50.0 vs 50.6). I request at least three fine-tuning runs per condition with mean and standard deviation, or a paired bootstrap over benchmark problems, so that the claimed improvements can be distinguished from noise.","section":"§4.4, Table 1"},{"comment":"The comparison to OSS-Instruct is confounded. CodeEvo is trained on 17K examples derived from roughly 5K LeetCode/Codeforces-style seeds with keyword tags and includes some seed reference solutions, whereas OSS-Instruct uses the released 75K Magicoder dataset generated from open-source code snippets with a different pipeline and generator model. The \"4–5x fewer data\" efficiency claim relative to OSS-Instruct may therefore reflect seed distribution or generator differences rather than the CodeEvo mechanisms themselves. Please either re-run OSS-Instruct on the same seed set with the same Qwen-based generator models (as was done for Evol-Instruct), or explicitly delimit the claim and discuss this confound.","section":"§4.2 / Appendix C.2, Table 1"},{"comment":"The hybrid feedback acceptance criterion is self-referential and not independently verified. The same Reviewer agent that generates new instructions also judges whether the Coder's solution is correct, and the compiler signal only checks test cases that the Coder wrote for its own solution. The paper's own Limitations (Test Case Quality) concedes that \"it remains difficult to ensure complete correctness, especially at scale.\" To substantiate the characterization of retained pairs as \"well-grounded and executable\" and to explain the data-efficiency advantage, the authors should validate a random sample of the 17K retained pairs against an independent oracle (e.g., hidden tests from the seed platforms, human annotation, or a separate strong model) and report the true positive rate. Without this, the survival-rate analysis in Section 5.5 (Figure 9) only demonstrates internal consistency of the filter, not its accuracy.","section":"§3.3 and Algorithm 1"}],"minor_comments":[{"comment":"The abstract states that CodeEvo-100K is constructed, but all fine-tuning experiments and the scaling analysis (Figure 8) use at most 17K examples; clarify the relationship between CodeEvo-100K and the 17K training subset, and state whether any experiment uses the full 100K dataset.","section":"Abstract vs. §4.4"},{"comment":"There is a duplicated or broken sentence in the main-results paragraph: \"On StarCoder2-7B, included to align with the evaluation setups of WizardCoder and Magicoder, CodeEvo significantly surpasses their original synthesis designs, achieving striking gains on the more challenging benchmarks—nearly doubling the performance in some cases. CodeEvo delivers striking gains on challenging benchmarks.\" Please revise for clarity.","section":"§4.4"},{"comment":"There is a typo in the DeepSeek-Coder model description: \"we adopot this model\" should be \"we adopt this model.\"","section":"Appendix A"},{"comment":"The human evaluation of instruction difficulty uses five participants, and the solvability study does not report a sample size or inter-annotator agreement; please include these details and state whether raters were blinded to the synthesis method.","section":"§5.3 and Appendix G.2"},{"comment":"The hyperparameters [rmin, rmax], tmax, and the maximum iterations N are never given concrete values or a sensitivity analysis; please report the values used in the experiments.","section":"Algorithm 2 and Appendix E"},{"comment":"The computation of fhybrid from fcomp and fNL is underspecified: it is not clear which prompt or decision rule the Reviewer uses to produce the final \"valid\" verdict, nor whether a separate LLM call is made; providing the exact prompt or decision rule would improve reproducibility.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the controlled Evol-Instruct comparison (same seeds, same generator models) already shows consistent gains, so rejection is not warranted. The main risks to the published claims are the missing error bars, the OSS-Instruct confound, and the unvalidated self-assessment in the filtering step. I would also ask the authors to reconcile the CodeEvo-100K abstract claim with the 17K experimental scale. A major revision with added statistical rigor and an independent sample audit would make this a strong paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: CodeEvo is a genuinely useful data-synthesis method. The new bit is the combination of keyword-anchored instruction evolution with a dual-agent loop where a Reviewer filters and generates, and compiler feedback plus NL judgment decide retention. On the evidence in the paper, it works: models trained on 17K CodeEvo pairs beat baselines trained on 25K Evol-Instruct and 75K OSS-Instruct across most benchmarks, and the gains are consistent across four backbones.\n\nWhat I like: the seed-ablation result is honest and informative—dropping seed solutions doesn't hurt, which suggests the pipeline is doing the work. The release of the dataset and code is real credit. The human difficulty rating and survival-rate analysis are useful sanity checks. The paper also explicitly owns the test-case-quality limitation, which is the right place for it.\n\nSoft spots, in order of seriousness. First, the filter's correctness is not independently verified. The Reviewer that generates instructions also judges them, and the compiler only checks test cases the Coder wrote for itself. The paper's Limitations section admits this ('it remains difficult to ensure complete correctness, especially at scale'). I don't think this sinks the empirical claim—the downstream benchmarks are external and the gains are measured there—but it does mean the mechanistic story 'hybrid feedback ensures grounded data' is partly a hypothesis. Second, the OSS-Instruct comparison is confounded: different seed distributions, different data source philosophy. The Evol-Instruct comparison is cleaner because they reproduced it with the same seeds, and CodeEvo wins there too. Third, no variance or significance testing anywhere; for this kind of paper that's common but still worth noting. Fourth, minor inconsistency: main text says 8×A100, appendix says 8×H800. Fifth, no ablation isolates keyword guidance versus hybrid feedback, so we don't know which component drives the gain.\n\nOverall: the central claim—data-efficient gains from this synthetic pipeline—holds up. The mechanism-level explanation is plausible but not fully proven. The paper is honest about its main risk. A serious referee should engage it; the fixes are manageable (an independent test-set check on a sample, a cleaner OSS comparison, variance bars).\n\nRecommendation: send to peer review. I would cite it if I were working on synthetic code data.","headline":"CodeEvo is a solid, data-efficient synthesis pipeline whose gains are real, though the filter's self-validation and the OSS comparison leave the mechanism story partly unproven.","tokens_in":20273,"tokens_out":1904,"would_cite":true,"duration_ms":21420,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two AI agents, a coder and a reviewer, synthesize code-training data that beats datasets several times larger.","keywords":["code generation","synthetic data","LLM agents","instruction evolution","compiler feedback","keyword-guided synthesis","hybrid feedback","code-centric data"],"falsifier":"Audit a random sample of instruction-code pairs that CodeEvo's pipeline retains by running them against independently written hidden tests that the Coder did not produce. If a nontrivial fraction fails those tests, or solves a different task than the instruction states, the hybrid filter is not doing what the data-quality claim requires.","tokens_in":19310,"feed_emoji":"🤖","tokens_out":8530,"duration_ms":87375,"temperature":0.7,"pith_summary":"CodeEvo is a method for building synthetic instruction-code training data without human-written gold solutions. It pairs a Coder agent, which writes code and tests, with a Reviewer agent, which judges the output and writes the next, harder instruction. The paper's central claim is that this interaction, anchored by task keywords and validated by a hybrid of compiler execution and natural-language judgment, produces data that is more grounded and more executable than heuristic-evolved or open-source-derived data. Models fine-tuned on CodeEvo data outperform the baselines in most reported settings, and the gains appear with fewer training samples than competing datasets. This points toward a data-efficient, reference-free way to improve code models.","feed_headline":"Two-agent loop beats bigger code datasets","feed_subtitle":"Coder and reviewer agents, with compiler checks, produce training pairs that lift code-model scores on fewer samples.","key_machinery":"The load-bearing mechanism is the hybrid feedback $f_{hybrid}$, formed when the Reviewer fuses deterministic compiler feedback $f_{comp}$ with its own natural-language evaluation $f_{NL}$. This fused signal does two jobs: it filters which instruction-code pairs enter the dataset, and it gives the Coder a concrete reason to refine. The other pillar is keyword-guided instruction generation: new instructions are created by conditioning on a subset of the seed's keyword tags, and the same keywords are removed to simplify a problem when the Coder fails. That bidirectional keyword control keeps evolved instructions semantically grounded while pushing difficulty upward.","core_discovery":"On the paper's own terms, the discovery is that a feedback-driven agent loop can replace rigid prompting heuristics in code-data synthesis. Starting from only a few thousand seed instructions and their keyword tags, the Reviewer proposes new instructions by keeping or removing keywords, the Coder attempts them, and a hybrid signal—compiler pass/fail plus the Reviewer's semantic read of the code—decides whether the pair is kept or sent back for refinement. The resulting trajectories yield instruction-code pairs that pass both execution and logical checks. Fine-tuning four base models on this data gives higher pass@1 on HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench than training on Evol-Instruct or OSS-Instruct data, using 17K samples against 25K and 75K for those baselines.","pith_inferences":["The same keyword-anchored, bidirectional evolution could be lifted to other symbolic or structured domains—math word problems, reasoning chains, or agent trajectories—where difficulty can be parameterized by a small vocabulary of techniques.","The hybrid feedback signal, which already encodes a retain/refine verdict, could be reused as a reward signal for preference optimization or reinforcement-learning fine-tuning of code models.","A stronger test of the data-efficiency claim would hold the training budget fixed at 17K samples for every method, rather than comparing against baselines that used 25K or 75K samples.","Because the Reviewer is judging the same kind of output it helps produce, an independent human audit of a random retained subset would quantify how much of the gain comes from filtering versus from the seed and benchmark mix."],"forward_implications":["Synthetic code data can be produced from natural-language seeds alone, with no gold code and no human-authored tests, which removes a major bottleneck in dataset construction.","Data efficiency improves: comparable or better downstream pass@1 is reached with 17K CodeEvo samples than with 25K Evol-Instruct or 75K OSS-Instruct samples.","Compiler-in-the-loop validation pays off most where test coverage is stricter, as suggested by gains on HumanEval+ and MBPP+ over the base versions.","The approach transfers across general-purpose and code-specialized base models, and it works with smaller synthesis backbones, so frontier or proprietary models are not required.","Because surviving instructions get harder as rounds progress, the retained pairs form a natural difficulty ladder, which could support curriculum-style fine-tuning."],"supporting_citations":[{"why":"Defines the Evol-Instruct prompting heuristic that CodeEvo's keyword-guided generation is designed to replace.","marker":"(Xu et al., 2024a)"},{"why":"Supplies the Code Evol-Instruct baseline, reproduced with the same seed data for comparison.","marker":"(Luo et al., 2024b)"},{"why":"Supplies the OSS-Instruct 75K dataset used as a second comparison baseline.","marker":"(Wei et al., 2024b)"},{"why":"Provides HumanEval, a primary evaluation benchmark for the fine-tuned models.","marker":"(Chen et al., 2021)"},{"why":"Provides MBPP, a second primary evaluation benchmark.","marker":"(Austin et al., 2021)"},{"why":"Provides the HumanEval+ and MBPP+ augmented tests that support the compiler-in-the-loop validation claim.","marker":"(Liu et al., 2023)"},{"why":"Provides BigCodeBench, a harder benchmark used to test generalization under realistic instructions and function calls.","marker":"(Zhuo et al., 2025)"},{"why":"Provides LiveCodeBench, a contamination-aware benchmark used to test freshness and generalization.","marker":"(Jain et al., 2025)"}],"fun_headline_variants":["Agent loop with compiler checks beats larger code datasets","Coder-reviewer loop yields better code data with fewer samples","Feedback loop trumps static heuristics for code data","Dual-agent synthesis lifts code model pass@1 scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that the Reviewer's natural-language judgment, combined with compiler pass/fail on test cases the Coder wrote itself, reliably identifies correct, well-grounded solutions; the paper concedes that agent-generated test coverage can be incomplete.","fun_headline_variants_meta":{"raw":{"variants":["Agent loop with compiler checks beats larger code datasets","Coder-reviewer loop yields better code data with fewer samples","Feedback loop trumps static heuristics for code data","Dual-agent synthesis lifts code model pass@1 scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000449,"raw_usage":{"total_tokens":2235,"prompt_tokens":890,"completion_tokens":1345,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":1280}},"tokens_in":506,"tokens_out":1345,"duration_ms":11065,"temperature":1.0,"reasoning_tokens":1280,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:19:32.244876+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit a random sample of instruction-code pairs that CodeEvo's pipeline retains by running them against independently written hidden tests that the Coder did not produce. If a nontrivial fraction fails those tests, or solves a different task than the instruction states, the hybrid filter is not doing what the data-quality claim requires.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides BigCodeBench, a harder benchmark used to test generalization under realistic instructions and function calls."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides LiveCodeBench, a contamination-aware benchmark used to test freshness and generalization."}],"review_version":1}