{"id":"ae259ea9-d9d5-4fd6-9083-ade6e07f9c05","arxiv_id":"2507.18812","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A multi-agent LLM framework with a memory of past code fixes (MemoCoder) improves Pass@10 and Pass@50 on MBPP, HumanEval, and LiveCodeBench over zero-shot and self-repair baselines by 3 to 14 percentage points.","lead":"MemoCoder is a multi-agent system that uses a memory of past code fixes to help large language models repair faulty code across multiple attempts. The system outperformed standard zero-shot and self-repair methods by 3 to 14 percentage points on three coding benchmarks, suggesting that remembering past errors can materially improve AI code generation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Online adaptation lets same-benchmark fixes from earlier problems enter the fixing knowledge set, so the reported Pass@10/50 gains may reflect benchmark-specific adaptation rather than generalization to unseen tasks.","rationale":"The reader's weakest assumption identifies the same structural issue that I consider most load-bearing: the evaluation-phase knowledge set grows from the benchmarks being measured. The paper explicitly states in Section 3.3 that the Fixing Knowledge set continues to grow during evaluation, and Section 2.2.4 describes the iterative online-adaptation protocol. This is not a minor reporting omission; it changes what the headline numbers measure. If the goal is to show that MemoCoder generalizes to unseen programming tasks, the evaluation must keep the knowledge base disjoint from the evaluation benchmarks. The current protocol still supports the weaker claim that MemoCoder can improve while adapting to the benchmark distribution, but it does not support the stronger generalization claim as cleanly. The proposed frozen-knowledge test would settle this directly. Secondary issues, such as the retrieval size being manually tuned on the evaluation sets and the numerical inconsistencies between the abstract/conclusion and Table 1, are real but less fundamental; the online-adaptation concern alone justifies the conditional verdict. I credit the paper for including a replication package, prompt templates, contamination-filtered LiveCodeBench selection, and McNemar tests, but those do not resolve the benchmark-level information flow identified here.","tokens_in":20496,"tokens_out":4863,"duration_ms":52596,"concrete_test":"Run a frozen-knowledge evaluation: after the APPS knowledge-accumulation phase, freeze the Fixing Knowledge Set so that no successful repairs from MBPP, HumanEval, or LiveCodeBench are added during evaluation, and recompute Pass@10 and Pass@50 for MemoCoder, MemoCoder w/o RAG, and MemoCoder w/o Mentor under identical retrieval settings. Compare these numbers against Table 1. If the gap to Self-Repair shrinks materially (e.g., by more than 5 percentage points on any dataset), the online adaptation protocol is the source of the headline gains; if the gap is stable, the memory does transfer across tasks. To fully close the ambiguity, also log, for each retrieval, whether any retrieved example originated from the same evaluation benchmark, and report the fraction of successful problems whose repair used such same-benchmark examples.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that Pass@k on MBPP, HumanEval, and LiveCodeBench measures performance on tasks unseen by the Fixing Knowledge Set. The paper's own protocol contradicts this. Section 3.3 states: 'During this phase, the Fixing Knowledge set continues to grow by incorporating successful repairs, which are made available for retrieval in future test cases.' Section 2.2.4 describes an online-adaptation protocol in which 'only the remaining unsolved problems are passed through the system, which incorporates newly distilled fixing strategies from previously solved cases.' Thus, for a later problem, the retrieved similar-error fixing examples can include code and fixes drawn from earlier problems of the same benchmark. The protocol prevents a task from benefiting from its own fixes, but it does not prevent a task from benefiting from fixes to other tasks in the same evaluation set. Since the Self-Repair baseline has no such memory, the reported 3.1% to 12.1% Pass@10 and 1.4% to 14.5% Pass@50 improvements conflate genuine transferable repair knowledge with benchmark-specific adaptation. This is structural to the evaluation and directly affects the central claim that memory of past fixes drives the improvement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MemoCoder, a multi-agent framework for LLM-based code generation and repair that combines a Planner, a Code Writer, a Test Executor, and a Mentor agent with a persistent Fixing Knowledge Set storing successful repairs. The Mentor summarizes recurring error patterns and produces fixing suggestions, and retrieval of similar past fixes is used to guide the repair loop. The system is bootstrapped by running on the APPS dataset and is then evaluated on LiveCodeBench, MBPP, and HumanEval with two base models (LLaMA 3.1-8B-Instruct and Qwen 2.5-32B), comparing against zero-shot prompting and a Self-Repair baseline. The paper reports consistent Pass@10 and Pass@50 improvements, an ablation of the planning, retrieval, and error-pattern components, and an analysis of error-type transitions across repair iterations.","tokens_in":20694,"tokens_out":6739,"duration_ms":63909,"significance":"If the reported gains hold under a leakage-free evaluation, the work would be a useful contribution to LLM-based code repair: it demonstrates that a persistent, retrievable memory of successful fixes, combined with a supervisory agent that distills reusable repair strategies, can improve iterative refinement. Strengths include evaluation on three benchmarks with two base models, ablations of the main components, McNemar statistical testing, and a promised replication package containing the agent prompts. However, the current evaluation protocol does not support the generalization claim as stated, because fixes from earlier tasks in the same benchmark are allowed to enter the knowledge base during evaluation, and a key hyperparameter is manually tuned without a described development split. The central idea is plausible, but the evidence is not yet conclusive.","major_comments":[{"comment":"The evaluation phase is run under an online-adaptation protocol in which the Fixing Knowledge Set \"continues to grow by incorporating successful repairs, which are made available for retrieval in future test cases\" (Section 3.3), and in which \"only the remaining unsolved problems are passed through the system, which incorporates newly distilled fixing strategies from previously solved cases\" (Section 2.2.4). This means that for any later task in LCB, MBPP, or HumanEval, the retrieved fixing examples can include code and repairs from earlier tasks in the same benchmark. Since the Self-Repair baseline has no such memory, the reported Pass@10 and Pass@50 improvements conflate genuine transferable repair knowledge with benchmark-specific adaptation, so the results do not measure generalization to unseen tasks. Please re-run the evaluation with the knowledge set fixed after the APPS accumulation phase, or with a held-out split for online adaptation, and report both settings.","section":"Section 3.3 and Section 2.2.4"},{"comment":"The retrieval size is chosen by manual experimentation: \"To determine the optimal number of examples to retrieve, we manually experimented with different retrieval sizes... we set the mentor agent to retrieve up to ten examples.\" The paper does not state that this tuning was performed on a development split disjoint from LCB, MBPP, and HumanEval. If the evaluation benchmarks were used to select this hyperparameter, the reported gains are inflated by test-set tuning. Please specify the tuning protocol and, if necessary, re-run the comparison with the retrieval size fixed before seeing the evaluation sets.","section":"Section 4.1.2"},{"comment":"The Pass@k definition is ambiguous. The example in Section 3.3 treats sequential repair attempts as the k samples (\"If the model finally generates a correct solution on the fourth try... then Pass@4 becomes 1\"), which is not the standard Pass@k estimator over independent temperature samples. It must be clarified whether Pass@k is computed over the repair trajectory, over multiple independent runs, or over a combination, and the same protocol must be applied identically to MemoCoder and the baselines. This matters because the reported Pass@10 and Pass@50 values are the central evidence for the paper's claims.","section":"Section 3.3"},{"comment":"The internal validity section states that \"our ablation study (Section X) shows that performance plateaus beyond 10 retrieved examples and most successful fixes occur within 30 attempts,\" but no such ablation appears in the paper and the cross-reference \"Section X\" is unresolved. This statement is used to argue that the repair limit and retrieval size are reasonable and not overfit; without the supporting study, that argument is unsupported. Please either add the ablation or remove the claim.","section":"Section 5.2"}],"minor_comments":[{"comment":"The abstract reports Pass@10 improvements of 3.1% to 12.1%, but the largest Pass@10 improvement over either baseline in Table 1 is about 14.05 percentage points (LCB, LLaMA 3.1-8B vs Self-Repair); please reconcile the numbers.","section":"Abstract and Table 1"},{"comment":"The Code Writer description refers to the Mentor agent as \"explained in Section 4,\" but the Mentor is actually described in Section 2.2.4; the cross-reference should be corrected.","section":"Section 2.2.2"},{"comment":"The citation for AgentCoder appears as \"Huange et al.\" and should be \"Huang et al.\"","section":"Section 6.4"},{"comment":"Section 5.1 says \"over 5,000 training examples from the APPS dataset\" were used to construct the knowledge set, but Section 3.1 does not report how many APPS problems were actually solved and retained; please report the actual number of fixes in the knowledge base.","section":"Section 5.1"},{"comment":"The reference list contains duplicated entries for the same paper, e.g., Austin et al. 2021 appears as both [3] and [4]; please deduplicate the bibliography.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the main risk is the online-adaptation protocol, which currently prevents the reported numbers from being read as a generalization result. If the authors can supply a leakage-free evaluation with a fixed knowledge set and a properly held-out hyperparameter choice, the paper would be publishable. I did not see evidence of misconduct, but the unresolved \"Section X\" reference and the mismatch between the abstract's Pass@10 range and Table 1 should be fixed. The novelty relative to AgentCoder and MapCoder is incremental but appears sufficient for the venue if the evaluation is made sound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's architecture is a real step beyond the usual self-repair loop: a persistent Fixing Knowledge Set plus a Mentor Agent that distills error patterns into reusable strategies. The ablation study suggests both the memory and the mentor matter, and the use of a contamination-controlled LiveCodeBench subset is good practice. The McNemar tests are appropriate.\n\nThe soft spot is the evaluation protocol. Sections 2.2.4 and 3.3 are explicit that during the evaluation phase the Fixing Knowledge Set grows by incorporating successful repairs from previously solved test problems, available for retrieval in later test cases. That means later problems in MBPP, HumanEval, or LiveCodeBench can be solved using code and fixes drawn from earlier problems in the same benchmark. The Self-Repair baseline has no such memory. So the reported gains—3.1–12.1 points in Pass@10, 1.4–14.5 in Pass@50—conflate benchmark-specific adaptation with genuine cross-task improvement. The paper's claim that this protocol prevents data leakage is only true in the narrow sense that no task benefits from its own fixes; it does not prevent leakage between tasks in the same evaluation set. That is load-bearing for the central claim.\n\nSmaller problems: retrieval size was manually tuned, likely on the evaluation benchmarks, and no held-out validation is reported for this hyperparameter. The Pass@k definition is ambiguous because it counts iterative repair attempts, not independent samples, and it is not clear whether baselines get the same number of attempts. Finally, the abstract and conclusion numbers do not match Table 1: the abstract says up to 12.1 points Pass@10, but Table 1 shows a 14.05-point gain on LCB with LLaMA, and the conclusion claims 12/15 points Pass@10 and 16/19 points Pass@50, which do not correspond to the table. These are fixable but suggest sloppy reporting.\n\nThe core idea is credible, and the method probably does work to some degree. The paper deserves a serious referee, but not as is. A clean separation of online adaptation from transfer—or an explicit acknowledgment that the evaluation measures within-benchmark adaptation—would make the contribution much stronger.","headline":"MemoCoder's memory-augmented repair architecture is plausible and the ablations are informative, but the online-adaptation evaluation protocol lets fixes from earlier test problems feed later ones, so the headline gains do not cleanly measure generalization.","tokens_in":662,"tokens_out":2559,"would_cite":false,"duration_ms":53486,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MemoCoder claims a multi-agent repair loop with a persistent Fixing Knowledge Set beats zero-shot and self-repair on three code benchmarks.","keywords":["LLM code generation","multi-agent systems","self-repair","retrieval-augmented generation","fixing knowledge","function synthesis","Pass@k evaluation","program repair"],"falsifier":"Run MemoCoder with the Fixing Knowledge Set seeded only from APPS and then frozen during evaluation; if the Pass@50 advantage over Self-Repair mostly disappears, the gains come from in-benchmark adaptation rather than from reusable repair knowledge.","tokens_in":20240,"feed_emoji":"🤖","tokens_out":4740,"duration_ms":44825,"temperature":0.7,"pith_summary":"MemoCoder is a multi-agent framework for turning natural-language problem descriptions into working functions. The paper claims that its four specialized LLM agents, coordinated by a central Mentor Agent and backed by a Fixing Knowledge Set that stores successful repairs, consistently solve more test cases than zero-shot prompting or a Self-Repair baseline across MBPP, HumanEval, and LiveCodeBench. Reported gains range from 3.1 to 12.1 percentage points at Pass@10 and from 1.4 to 14.5 percentage points at Pass@50. A sympathetic reader would take the central insight to be that repair knowledge, once extracted from past fixes, can be reused across problems without retraining the model.","feed_headline":"Memory-guided agents lift code repair pass rates by up to 14.5 points","feed_subtitle":"It stores past fixes and distills repair strategies, beating zero-shot and self-repair on MBPP, HumanEval, and LiveCodeBench.","key_machinery":"The load-bearing object is the Fixing Knowledge Set, a persistent store of initial code, fixed code, and error messages, together with the Mentor Agent, which retrieves up to ten similar past fixes by a longest-sequential-match on error text and rewrites a small set of fixing suggestions per error type. A Planner agent supplies the starting point by generating three plans before code is written. The machinery's job is to convert single repair attempts into reusable strategies, so later problems start from accumulated experience rather than rediscovering the same fix.","core_discovery":"The paper's central claim is that a code-generation system improves when repair is organized as a collaborative loop with memory. In MemoCoder, a Planner proposes multiple solution strategies, a Code Writer implements and revises code, a Test Executor classifies failures into four error types, and a Mentor Agent retrieves similar past fixes and distills recurring error patterns into high-level fixing suggestions. Successful repairs are stored in the Fixing Knowledge Set and become retrievable for later problems. The paper reports that this design outperforms both zero-shot prompting and a Self-Repair baseline on three benchmarks, and that its ablations point to retrieval and error-pattern analysis as the components driving the higher Pass@10 and Pass@50 scores.","pith_inferences":["If MemoCoder's Fixing Knowledge Set is frozen after the APPS seeding phase, the reported Pass@50 advantage over Self-Repair may shrink; that experiment would separate within-benchmark adaptation from transfer of repair skill.","The online-adaptation protocol means the numbers should be read as describing a system that solves a stream of problems while learning from that same stream, not as describing a system that solves problems unrelated to anything it has seen.","A natural extension would test whether distilled fixing suggestions transfer across programming languages or to repository-level bugs, where error messages and test harnesses are more heterogeneous.","A cheaper baseline worth checking is Self-Repair with access to the same retrieved examples but without the Mentor Agent's distilled strategy summaries, which would isolate the value of abstraction over raw retrieval."],"forward_implications":["If the reported gains hold, a frozen LLM can keep improving at coding tasks by accumulating repair experience, without fine-tuning or retraining.","Self-repair loops that lack memory should be expected to lag on benchmarks where similar failure modes recur across problems.","Pass@1 gains are small, so the framework's value lies in iterative settings where multiple attempts are affordable rather than in first-attempt code generation.","The error-type analysis suggests that compile errors and wrong-output errors persist across iterations, making those categories the natural target for further improvement.","Retrieval and error-pattern distillation are the components that matter most for high-attempt pass rates, so omitting them would erode the reported advantage at Pass@10 and Pass@50."],"supporting_citations":[{"why":"Supplies the APPS dataset used to seed the Fixing Knowledge Set during the accumulation phase.","marker":"[11]"},{"why":"Supplies the MBPP benchmark and its assertion-style problems for evaluation.","marker":"[4]"},{"why":"Supplies the HumanEval benchmark used to measure pass rates.","marker":"[5]"},{"why":"Supplies LiveCodeBench, its contamination-free evaluation subset, and the self-repair implementation used as a baseline.","marker":"[14]"},{"why":"Provides McNemar's test, which the paper uses to judge whether pass-rate differences are statistically significant.","marker":"[21]"},{"why":"Defines the Self-Repair strategy that MemoCoder is compared against.","marker":"[27]"},{"why":"Describes LLaMA 3.1-8B-Instruct, one of the two base models powering the agents.","marker":"[9]"},{"why":"Describes Qwen 2.5-32B, the other base model powering the agents.","marker":"[32]"},{"why":"Motivates the online-adaptation protocol used during evaluation.","marker":"[34]"}],"fun_headline_variants":["Multi-agent memory boosts code repair by up to 14.5%","Mentor agent stores fixes, beats self-repair on code benchmarks","Memory-aware repair loop lifts Pass@50 by 14.5 points","Coding agents with fixing knowledge outdo self-repair","MemoCoder: collaborative agents learn from past fixes for better code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation lets fixes from problems already solved on the same benchmark feed the repair of later problems, so the reported gains measure adaptation within a benchmark rather than transfer to never-seen tasks.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent memory boosts code repair by up to 14.5%","Mentor agent stores fixes, beats self-repair on code benchmarks","Memory-aware repair loop lifts Pass@50 by 14.5 points","Coding agents with fixing knowledge outdo self-repair","MemoCoder: collaborative agents learn from past fixes for better code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000663,"raw_usage":{"total_tokens":3033,"prompt_tokens":952,"completion_tokens":2081,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":1989}},"tokens_in":568,"tokens_out":2081,"duration_ms":15033,"temperature":1.0,"reasoning_tokens":1989,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:08:25.363832+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MemoCoder with the Fixing Knowledge Set seeded only from APPS and then frozen during evaluation; if the Pass@50 advantage over Self-Repair mostly disappears, the gains come from in-benchmark adaptation rather than from reusable repair knowledge.","supporting_citations":[],"review_version":2}