{"id":"a98183ab-56b9-42d1-bf8e-eefbb0ada4ed","arxiv_id":"2501.07811","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CodeCoR improves LLM code generation by having four agents generate, test, prune, and repair candidate solutions, reporting a 77.8% average Pass@1 on four Python benchmarks.","lead":"CodeCoR is a multi-agent framework for code generation in which four large language model agents (prompt, coding, test, repair) each produce multiple outputs and prune weak ones. It reports an average Pass@1 of 77.8% on HumanEval, HumanEval-ET, MBPP, and MBPP-ET, a few points above prior multi-agent systems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pass@1 claim is not comparable: Algorithm 1 uses best-of-many selection and repair, while Section 4.4 defines Pass@1 as first-attempt; candidate budgets for CodeCoR and baselines are unreported.","rationale":"In good faith, the paper's contribution is a multi-agent pipeline that prunes intermediate outputs, repairs failing code, and selects a final code by the number of generated test cases passed; the headline claim is that this pipeline beats existing multi-agent frameworks. The most load-bearing condition for that claim is that the evaluation measures the same quantity for all systems. The manuscript's own definition of Pass@1 as 'first attempt without any modifications' is contradicted by Algorithm 1, which uses multiple candidates, repair loops, and final selection. Without candidate counts or compute budgets for either CodeCoR or the baselines, the Table 2 comparison cannot be attributed to the architecture. This is not an outside-consensus disagreement; it is an internal inconsistency in the evaluation protocol. The reader's weakest assumption about self-assigned pruning scores is real but secondary: even if the self-scores are noisy, ablations suggest the overall pipeline still helps, and the headline could survive; the metric mismatch, by contrast, can invalidate the headline even if pruning works perfectly. I therefore regard the reader's pruning concern as a genuine but secondary issue, not the most load-bearing one. A fixed-budget re-run with a true single-attempt Pass@1 and a best-of-k Pass@k computed from the same logs would settle whether the reported 5-point average improvement survives. The verdict should remain CONDITIONAL: the framework is plausible, and the concern is resolvable if the authors release the candidate counts, logs, and baseline settings.","tokens_in":17965,"tokens_out":7708,"duration_ms":79103,"concrete_test":"Re-run CodeCoR and MapCoder on HumanEval under a fixed LLM-call budget: same number of generated candidates per stage, same temperature, and same maximum repair rounds per failed snippet, with the final output selected by the same criterion. Additionally, compute a true first-attempt Pass@1 by taking only the first code snippet generated before any repair or selection, and separately compute Pass@k with k equal to the number of candidates actually generated. If the true Pass@1 gap between CodeCoR and MapCoder drops below the reported 6.1 points on HumanEval (86.6 vs 80.5), the headline superiority is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CodeCoR raises average Pass@1 to 77.8% over baselines. For that comparison to be valid, Pass@1 must measure the same quantity for CodeCoR and each baseline, and the reported gain must come from the framework rather than from extra sampling or additional repair attempts. Section 4.4 defines Pass@1 as 'the proportion of generated code snippets that correctly perform the intended task on the first attempt without any modifications.' However, Section 3.1 and Algorithm 1 describe a pipeline in which the Prompt Agent, Test Agent, and Coding Agent each generate 'more than one output,' failed snippets are iteratively repaired, and the final answer is the code that passes the most generated test cases. That is best-of-N selection with repair, not first-attempt pass@1. The paper does not state how many code snippets are sampled, how many test cases are generated, what temperature is used, or how many LLM calls per problem are allowed; it also does not report whether the MapCoder and CodeCoT numbers in Table 2 were produced under the same candidate budget. If CodeCoR uses more candidates or more repair rounds than the baselines, the reported average gain could reflect additional compute rather than the pruning/self-reflection architecture. The manuscript's own definition of Pass@1 is therefore internally inconsistent with its reported procedure, making the headline comparison unverifiable from the text alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CodeCoR, a multi-agent LLM framework for Python code generation. Four agents (prompt, test, coding, and repair) each generate multiple outputs and prune low-quality ones; generated code is executed against generated test cases, failed snippets are sent to the repair agent, and the snippet passing the most generated tests is returned. Experiments on HumanEval, HumanEval-ET, MBPP, and MBPP-ET with GPT-3.5-turbo report an average Pass@1 of 77.8%, together with ablations and a cost comparison.","tokens_in":18271,"tokens_out":5976,"duration_ms":55049,"significance":"If the evaluation were protocol-equivalent, CodeCoR would be a useful contribution: the idea of pruning intermediate outputs to reduce error propagation in sequential multi-agent pipelines is plausible and distinct from plain sequential frameworks, and the ablation design and the availability of source code are positive features. However, the headline result is currently unverifiable because the Pass@1 metric is defined and used inconsistently with the actual pipeline, and the benefit of the pruning mechanism is not statistically established. The central architectural idea is worth pursuing, but the evidence in this manuscript does not yet support the strong comparative claims.","major_comments":[{"comment":"The reported Pass@1 is not the metric defined in §4.4. Section 4.4 defines Pass@1 as the proportion of generated snippets that are correct 'on the first attempt without any modifications', but Algorithm 1 and Section 3.1 describe generating multiple CoT prompts, multiple code snippets, multiple test cases, iterative repair, and final selection of the code passing the most generated tests. This is best-of-N selection with repair. Table 2 compares CodeCoR against numbers taken from prior papers (§4.3) without reporting the candidate budget, temperature, number of repair rounds, or LLM calls per problem for either side. Without a protocol-equivalent measurement, the 77.8% average and the claimed gains over MapCoder and CodeCoT are not verifiable, and the gain could reflect extra compute rather than the architecture. Please report the exact sampling and selection protocol, and either re-run baselines under the same candidate budget or clearly restrict the claim to best-of-N-with-repair performance.","section":"§4.4, §3.1, Algorithm 1, Table 2"},{"comment":"Algorithm 1 as printed clears failed_code_snippets and returns C_f inside the while loop body, so the loop can perform at most one repair iteration. This contradicts the text's description of iterative repair and the repair-round experiment in §5.3 and Figure 7, where performance is shown for up to five repair rounds. The pseudocode needs a correct loop structure and an explicit repair-round limit or other termination condition; as written, the algorithm cannot implement the claimed multi-round repair process.","section":"§3.4, Algorithm 1, lines 20–32"},{"comment":"The pruning methods are the main departure from a sequential pipeline, but they rely on the same LLM assigning binary self-scores for clarity, relevance, conciseness, and context with no validation that these scores correlate with downstream quality. The ablation 'w/o Pruning Method' is the relevant test, yet the differences are mixed in magnitude (1.3 points on HumanEval-ET versus 11.5 points on MBPP) and no significance or variance is reported. Moreover, the description of this variant is self-contradictory: it says the variant 'does not follow the traditional sequential multi-agent framework' and then says it 'cannot prune the outputs' of agents. Please clarify the variant and provide evidence that self-assigned scores are reliable, for example by comparing pruned versus unpruned outputs on a held-out quality measure.","section":"§3.3, Figure 5, Table 4"},{"comment":"The paper claims 'significantly' better performance, but no statistical support is provided. The threats-to-validity section states that 10 rounds were averaged for each trial, but no standard deviations, confidence intervals, or significance tests appear anywhere in the results. Given the small per-dataset differences (for example, 79.2 versus 78.9 on MBPP for MapCoder), the word 'significantly' is not supported. Please report variances or an appropriate significance test, or soften the claim.","section":"§5.4, Table 2"}],"minor_comments":[{"comment":"The average Pass@1 is reported as 77.13% in the abstract but as 77.8% in §1 and §6; the values in Table 2 average to 77.9%. Please harmonize these numbers.","section":"Abstract, §1, §6, Table 2"},{"comment":"The text says CodeCoR's mean edit distance is 'the highest among all models,' but edit distance is a lower-is-better metric; presumably 'lowest' is intended.","section":"§4.5, Table 3"},{"comment":"The text reports MBPP and MBPP-ET scores for CodeLlama, but Table 6 contains only HumanEval and HumanEval-ET; either add the missing columns or remove the unsupported sentence.","section":"§5.1, Table 6"},{"comment":"Reference [22] cites a paper on tensor-based channel estimation in IRS-assisted MIMO systems in the context of self-revision for code generation; this appears to be a citation error and should be replaced.","section":"§2.1, Reference [22]"},{"comment":"Table 2 has a duplicated 'GPT-3.5-turbo' header row and inconsistent column alignment; please reformat the table.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case. The core idea is interesting and the paper is generally readable, but the evaluation section is not yet at the standard required for the claimed contribution. I would encourage the authors to either re-run all baselines under a unified candidate budget or explicitly rename the metric (for example, 'best-of-N with repair'), and to fix the Algorithm 1 loop, which as printed cannot implement multi-round repair. I also note that the reference list contains an unrelated citation, which suggests the manuscript may not have been carefully proofread; please check all references."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the core addition here is LLM-based pruning of intermediate outputs in a sequential multi-agent pipeline—a real, sensible extension of CodeCoT and MapCoder. Second, the headline Pass@1 numbers are likely over-claimed because the metric definition doesn't match the procedure: Section 4.4 defines Pass@1 as first-attempt success without modifications, but Algorithm 1 uses best-of-N sampling, iterative repair, and final selection by test-passing count. That inconsistency makes the central comparison to baselines unverifiable as reported.\n\nWhat the paper does well: the four-agent structure (prompt, code, test, repair) is described clearly enough to reproduce the idea, and the ablation study is genuinely informative. Removing the Test Agent drops HumanEval Pass@1 from 86.6% to 45.1%, showing the generated tests are load-bearing. The cost comparison on CPU, memory, and runtime is a nice addition, though it's only measured on 10 problems.\n\nThe Pass@1 issue is not cosmetic. The pipeline generates multiple code snippets, prunes them, repairs failures, and returns the code that passes the most generated tests. That is best-of-N with repair, not a first-attempt metric. The paper never states how many snippets or test cases are generated per task, what sampling temperature was used, or how many repair rounds are allowed. Baselines are taken from prior papers under \"the same experimental setting,\" but no evidence is given that MapCoder or CodeCoT operated under the same candidate budget. So the reported ~5-point gain over MapCoder could be extra compute, not the pruning architecture. Also, the repair-round limit is tuned on the benchmark itself (Figure 7 shows 3 is best), making it a free parameter selected on the test set. The pruning self-scores (clarity, relevance, conciseness, context) are never validated against downstream quality; the core mechanism rests on an untested self-evaluation. Minor: the abstract reports 77.13%, but the intro and conclusion say 77.8%.\n\nWho this is for: researchers working on LLM code generation and multi-agent workflows. They'll find the pruning idea worth discussing, but they should not cite the numbers as solid evidence without a rework. The paper deserves a serious referee, not a desk reject. If I were the editor, I'd send it to review with instructions to focus on metric definition and protocol equivalence—specifically, report candidate budgets for all methods, align baselines under matched compute, and replace the First-attempt label. If those are fixed, this could be a solid contribution; if not, the empirical claim should be downgraded.","headline":"A sensible pruning-based multi-agent code generation framework, but the Pass@1 claim is built on a metric that doesn't match the procedure; the evaluation needs a major rework before the numbers can be trusted.","tokens_in":18793,"tokens_out":3104,"would_cite":false,"duration_ms":30227,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CodeCoR claims that pruning each agent's outputs stops error propagation and raises average Pass@1 to 77.8 percent across four benchmarks.","keywords":["code generation","multi-agent LLM framework","self-reflection","chain-of-thought prompting","test-case generation","program repair","Pass@1","pruning"],"falsifier":"Run CodeCoR and its variant with the pruning methods removed on all four datasets under identical random seeds and API settings: if disabling pruning does not lower Pass@1 on HumanEval, HumanEval-ET, MBPP, and MBPP-ET, then the reported advantage over MapCoder is not attributable to the pruning mechanism.","tokens_in":17769,"feed_emoji":"💻","tokens_out":4253,"duration_ms":39604,"temperature":0.7,"pith_summary":"This paper proposes CodeCoR, a four-agent LLM framework for Python code generation in which a prompt agent, coding agent, test agent, and repair agent each produce multiple candidate outputs and prune low-quality ones before passing results downstream. The aim is to stop errors from propagating through a sequential generation pipeline, which the authors argue is the main weakness of earlier multi-agent frameworks such as CodeCoT and MapCoder. The paper reports that CodeCoR achieves an average Pass@1 of 77.8% on HumanEval, HumanEval-ET, MBPP, and MBPP-ET with GPT-3.5-turbo, outperforming MapCoder's 72.8%. If the result holds, it suggests that self-evaluation and pruning inside a pipeline can be more valuable than simply chaining agents.","feed_headline":"Pruned multi-agent pipeline lifts code Pass@1 to 77.8%","feed_subtitle":"Four LLM agents generate, score, test, and repair code; only the best test-passing snippet survives.","key_machinery":"The central mechanism is a prune-and-retry cycle. At each of the four phases, agents generate multiple candidates, score them (often by self-assessed binary criteria), discard poor ones, execute code locally against generated tests, and route failures to a repair agent; the returned code is the one passing the most tests. The load-bearing assumption is that the LLM's own ratings separate high-quality from low-quality intermediate outputs. The repair loop stops when repeated errors resemble previous rounds or after a set number of rounds, which the paper finds optimal at three.","core_discovery":"The central claim is that interleaving multiple candidate generations with self-pruning and repair yields more correct code than a linear chain of specialized agents. Each stage is made self-reflective: prompts are scored on clarity, relevance, conciseness, and context; tests are pruned for emptiness or invalidity; code is pruned when it fails to compile; repair advice is pruned on the same four criteria. Code that fails the generated tests goes back to the coding agent with repair advice, and the snippet passing the most tests is returned. The paper reports Pass@1 scores of 86.6, 80.5, 79.2, and 65.2 on the four datasets with GPT-3.5-turbo and states that this significantly outperforms existing baselines including MapCoder.","pith_inferences":["Editorial inference: the same self-evaluation-and-prune pattern could be lifted to other multi-draft generation tasks, such as test-suite generation or natural-language-to-SQL, because the framework is not specific to the four roles it names.","Editorial inference: because the final selection is based only on generated tests, code that passes those tests may still fail hidden tests, so an independent replication with held-out test suites would clarify how much of the reported gap is real correctness improvement.","Editorial inference: the fixed generation budget per agent suggests a testable extension where the number of candidate drafts is varied per stage and Pass@1 is measured per API call, which would separate the value of pruning from the value of simply sampling more outputs."],"forward_implications":["Sequential multi-agent code generation can be improved by pruning at every stage rather than relying on each agent's single best output.","Self-assigned quality scores by an LLM can serve as a cheap filter for prompts and repair advice, without an external validator.","Iterative local execution plus repair can lift Pass@1 beyond what single-pass or linearly chained generation achieves on standard benchmarks.","The reported benefit transfers to other LLMs such as GPT-4 and CodeLlama and persists on the harder ET variants, suggesting robustness across model strength and test difficulty."],"supporting_citations":[{"why":"Supplies the chain-of-thought prompting technique that the prompt agent uses to decompose tasks.","marker":"[6]"},{"why":"Provides the CodeCoT sequential multi-agent baseline whose error propagation motivates the pruning design.","marker":"[9]"},{"why":"Provides MapCoder, the state-of-the-art multi-agent baseline that CodeCoR is compared against and must outperform.","marker":"[10]"},{"why":"Provides the HumanEval benchmark, including tasks and hidden tests used to measure Pass@1.","marker":"[11]"},{"why":"Provides the CodeScore work that produced the enhanced HumanEval-ET and MBPP-ET datasets.","marker":"[12]"},{"why":"Supplies the idea of generating tests and ranking code by passing generated tests, which underlies the test agent and final selection.","marker":"[17]"},{"why":"Provides the MBPP benchmark used for evaluation.","marker":"[29]"}],"fun_headline_variants":["CodeCoR prunes, repairs, and lifts code generation to 77% Pass@1","Self-reflective agents fix code: 77% Pass@1 with pruning and repair","Prune-and-repair loop boosts code accuracy to 77% Pass@1","Multi-agent code pipeline hits 77% Pass@1 via pruning and repair","CodeCoR: prune bad outputs, repair failures, score 77% Pass@1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes the LLM's own one-or-zero ratings for clarity, relevance, conciseness, and context correctly identify which prompts, tests, code snippets, and repair suggestions are worth keeping, and the paper supplies no independent check that these self-scores track actual downstream quality.","fun_headline_variants_meta":{"raw":{"variants":["CodeCoR prunes, repairs, and lifts code generation to 77% Pass@1","Self-reflective agents fix code: 77% Pass@1 with pruning and repair","Prune-and-repair loop boosts code accuracy to 77% Pass@1","Multi-agent code pipeline hits 77% Pass@1 via pruning and repair","CodeCoR: prune bad outputs, repair failures, score 77% Pass@1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000873,"raw_usage":{"total_tokens":3802,"prompt_tokens":989,"completion_tokens":2813,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":2716}},"tokens_in":605,"tokens_out":2813,"duration_ms":20051,"temperature":1.0,"reasoning_tokens":2716,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:35:07.303799+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CodeCoR and its variant with the pruning methods removed on all four datasets under identical random seeds and API settings: if disabling pruning does not lower Pass@1 on HumanEval, HumanEval-ET, MBPP, and MBPP-ET, then the reported advantage over MapCoder is not attributable to the pruning mechanism.","supporting_citations":[{"cited_title":"Chain-of-thought prompting elicits reasoning in large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the chain-of-thought prompting technique that the prompt agent uses to decompose tasks."},{"cited_title":"Mapcoder: Multi-agent code generation for competitive problem solving,","cited_arxiv_id":null,"evidence_quote":"Provides MapCoder, the state-of-the-art multi-agent baseline that CodeCoR is compared against and must outperform."},{"cited_title":"Codescore: Evaluating code generation by learning code execution,","cited_arxiv_id":null,"evidence_quote":"Provides the CodeScore work that produced the enhanced HumanEval-ET and MBPP-ET datasets."}],"review_version":1}