{"id":"68106f19-1ac4-4336-a34b-0a2b7a5a9629","arxiv_id":"2501.06625","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A structured multi-agent framework reports 56.2% Pass@1 on HumanEval with Llama 3.1 8B int4, versus 45.4% for one-shot generation.","lead":"This paper tests a multi-agent system that breaks coding problems into small pieces, generates each piece separately, and combines them from the bottom up. On the HumanEval benchmark, a quantized Llama 3.1 8B model improved Pass@1 from 45.4% to 56.2%, a 23.79% relative gain.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported Pass@1 gain is confounded: the multi-agent pipeline retries with test feedback ('subsequent generation attempts', §II-C), while the baseline is explicitly one-shot; the improvement may be a multi-attempt artifact rather than an effect of guided decomposition.","rationale":"The reader's verdict is REJECT, and I agree that the empirical claim is unsupported. However, the reader's weakest_assumption focuses on decomposition correctness and leaf composition, which are separate risks. The more fundamental problem is that the reported Pass@1 comparison is not apples-to-apples: the framework is explicitly iterative, with test feedback fed back into 'subsequent generation attempts' (Section II-C), whereas the baseline is 'one-shot' (Table I). This is not merely missing documentation; it is an internal inconsistency between the method description and the metric label. Even if the code and prompts were released, the headline improvement could be explained entirely by the extra attempts and oracle feedback. Therefore the most load-bearing concern is the retry confound, and the concrete test is a direct-repair control with the same inference budget. Since this strengthens the reader's rejection rather than altering it, the verdict remains UNCHANGED.","tokens_in":4595,"tokens_out":6296,"duration_ms":65104,"concrete_test":"Run a control experiment on HumanEval with Llama 3.1 8B int4: give the direct one-shot baseline the same test-feedback retry loop used by the framework (same total LLM call budget, same test oracle, but no hierarchical decomposition or critic), and measure its Pass@1. If this 'direct repair' baseline reaches or exceeds 56.2%, the claimed benefit of guided decomposition is not demonstrated; also compute the framework's success rate when each node is limited to exactly one generation attempt, to verify whether the Table I number is a true one-shot Pass@1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is Table I's 45.4→56.2 Pass@1 improvement. The framework as described is not one-shot: §II-B says every leaf function is 'immediately test[ed] and validat[ed]' and solutions 'incorporate feedback from tests'; §II-C says critic and tester feedback is 'incorporated into subsequent generation attempts.' Thus the framework may produce many candidate solutions per problem, using test outcomes to retry until a solution passes, while the baseline is a single generation with no feedback. Standard HumanEval Pass@1 measures the first generated solution; a pipeline that can repair failures is effectively reporting pass@k or pass-after-repair, so the +23.79% does not establish that decomposition/guidance helps. The paper reports no retry limits, total LLM calls, or per-node attempt counts, making the comparison uninterpretable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-agent framework for code generation: a Generalist Agent recursively decomposes a coding problem into a tree of atomic functions; Code Agents generate and test leaf functions bottom-up, composing them via interface documentation; a Critic Agent and Tester Agent provide feedback that is incorporated into subsequent generation attempts. The authors report Pass@1 of 56.2% on HumanEval with Llama 3.1 8B int4 versus 45.4% for direct one-shot generation, claiming a 23.79% improvement, and they provide a qualitative example plus a \"theoretical framework\" that treats code generation as a dual problem of information retrieval and compositional reasoning.","tokens_in":4785,"tokens_out":3096,"duration_ms":25950,"significance":"The practical question—whether structured decomposition and multi-agent validation improve code generation over direct generation—is important and timely. However, the paper provides no reproducible artifacts (code, prompts, baseline details, attempt counts) and the central empirical claim is confounded by the asymmetry in the number of generation attempts allowed for the framework versus the baseline. The theoretical framework is explicitly acknowledged as hypothetical and is not connected to the results beyond a narrative. Therefore, as presented, the paper's contribution is not verifiable and the central claim is not established.","major_comments":[{"comment":"The text states that the framework's Pass@1 of 56.2% versus 45.4% represents a \"23.79 percentage point improvement.\" That arithmetic is wrong: the difference is 10.8 percentage points, while 23.79% is the relative improvement (56.2/45.4 − 1). This conflation of relative and absolute improvement should be corrected, and the claims in the Abstract and Conclusion should be reworded accordingly.","section":"III-A, Table I"},{"comment":"The central comparison is confounded by asymmetric retry budgets. The framework's leaf functions \"undergo immediate testing and validation\" (§II-B) and critic/tester feedback is \"incorporated into subsequent generation attempts\" (§II-C), so the pipeline can generate multiple candidates and repair failures. The baseline is a single one-shot generation with no feedback. Standard HumanEval Pass@1 evaluates the first generated solution; reporting the framework's result after an unrestricted number of attempts as Pass@1 does not establish that decomposition or guidance helps. The paper reports no retry limits, total LLM calls, or per-node attempt counts, making the improvement uninterpretable.","section":"II-B, II-C, III-A"},{"comment":"The method is under-specified. The paper does not define the atomization criterion for \"practically indivisible coding tasks,\" the decomposition stopping rule, the prompt templates used by the Generalist and Code agents, or how tests are obtained for leaf functions before generation. It also does not explain how upward composition is validated when the Code Agent sees only \"documentation statements\" and not child implementations. Without these details the experiment cannot be reproduced or the proposed framework applied.","section":"II-A, II-B"},{"comment":"The theoretical framework is a post-hoc rationalization rather than a derivation: it reframes the approach in terms of \"fuzzy search\" and \"interface-based integration\" but provides no formal statements, no assumptions under which the decomposition is sound, and no predictions that could be tested. The authors themselves state in the Conclusion that \"this framework remains hypothetical and requires further study.\" It should not be presented as a supporting result.","section":"II-E, IV"}],"minor_comments":[{"comment":"Reference [9] is cited for OpenHands, but the reference is to \"Openhands: Making sign language recognition accessible\" (Selvaraj et al., 2021), which is unrelated to the agentic coding system OpenHands; this should be corrected or replaced with the proper citation.","section":"References"},{"comment":"Reference [10] is cited for int4 precision quantization, but the reference is the Hermes 3 technical report; the paper does not specify the quantization method or library actually used.","section":"References"},{"comment":"The qualitative comparison with GPT-4o and Gemini 1.5 Pro is reported without a protocol, rubric, or example output, and the claim that these models \"refused\" the task is not substantiated; this section should either be expanded into a systematic evaluation or removed.","section":"III-B"},{"comment":"Section II-C introduces Chain-of-Thought prompting but conflates it with self-critique and iterative feedback; the three mechanisms claimed (\"self-critique, problem decomposition, and upward composition\") are not separately evaluated, so the attribution of the performance boost is unsupported.","section":"II-C"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be a very early-stage draft: it lacks the experimental detail and controls necessary to support the central claim, and the citation errors suggest the reference list was not carefully checked. The confound between a multi-attempt pipeline and a one-shot baseline is decisive against the empirical contribution in its current form. If the authors can supply a comparison with matched attempt budgets, explicit retry limits, and full protocol details, the core idea could merit a resubmission, but the present version does not meet the journal's evidentiary standard."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this paper reports a big practical win (23.79% relative Pass@1 improvement on HumanEval with quantized Llama 3.1 8B) for a multi-agent decompose-generate-validate framework. The method is a specific assembly of known pieces: recursive decomposition, bottom-up composition using only child interfaces, and critic/tester feedback. That assembly is new in this exact shape and is described clearly enough to implement.\n\nThe good news: the authors are honest about their limitations. They state explicitly that if the initial root decomposition is wrong, the error propagates; they call their theoretical framework \"hypothetical\" and say it needs further study. The qualitative example is anecdotal but shows what the approach can do.\n\nThe problem is the empirical comparison. The baseline is direct one-shot generation, while the framework is allowed to test each leaf, get feedback, and retry. Section II-C says critic and tester feedback is \"incorporated into subsequent generation attempts.\" So the 45.4 -> 56.2 is not a comparison of guided decomposition versus one-shot; it is one-shot versus an interactive repair loop. Standard HumanEval Pass@1 is the first sample. If the framework can emit multiple candidates and pick based on test outcomes, that is closer to pass@k or pass-after-repair, and the improvement may have nothing to do with decomposition. The paper does not report retry limits, total LLM calls, or per-node attempts, so the effect size is uninterpretable. The table also mislabels the improvement: 23.79% relative, not percentage points. That is a minor wording issue but it signals carelessness.\n\nThe lack of code, prompts, and baseline details makes the central claim unverifiable. No repeated runs, no confidence intervals, no ablations. The reference list is also sloppy (OpenHands is cited as a sign-language paper).\n\nWhere does that leave it? The idea is worth exploring. A clean comparison would hold the number of LLM calls fixed between baseline and framework, report retry counts, and include a one-shot-with-feedback baseline. Without that, the paper does not establish that guided decomposition helps. But the paper is not incoherent; it's under-evidenced. I'd send it to review only to push the authors to do the controlled experiment, and I'd expect major revision. For a serious reader today, treat the headline number as an artifact until the retry confound is resolved.\n\nRecommendation: engage with the method, ignore the number. If you are a PC for a code-generation workshop, this deserves a review with instructions to focus on the experimental protocol. Not a desk reject, but not an accept.","headline":"A plausible decomposition-based code generation framework, but the headline Pass@1 gain rests on an unfair comparison that conflates retries with guidance.","tokens_in":5279,"tokens_out":2117,"would_cite":false,"duration_ms":20513,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Structured multi-agent decomposition lifts small-model code accuracy by 23.79% relative to one-shot generation.","keywords":["Large Language Models","Code Generation","Prompting Techniques","Agents","hierarchical decomposition","bottom-up composition","chain-of-thought prompting","HumanEval"],"falsifier":"Ablate the Generalist Agent's decomposition by giving the pipeline a fixed, task-independent decomposition tree on the same HumanEval problems; if Pass@1 stays at 56.2%, the decomposition is not the driver, and if it falls to 45.4% or lower, the central claim is supported. Alternatively, feed a deliberately incorrect root plan and observe the predicted error propagation in the final code.","tokens_in":4409,"feed_emoji":"🤖","tokens_out":6566,"duration_ms":51247,"temperature":0.7,"pith_summary":"This paper tries to show that a small, quantized language model can generate complex code more reliably if the task is explicitly decomposed and reassembled instead of being written in one shot. The proposed framework breaks a coding problem into a tree of atomic functions, generates each leaf separately with test feedback, and then composes the leaves upward using only their documentation, with critic and tester agents validating each step. On the HumanEval benchmark, the framework lifts Pass@1 from 45.4% for direct one-shot generation to 56.2%, a 23.79% relative improvement, using an 8B parameter Llama 3.1 model at int4 precision. If that holds, structured prompting pipelines can compensate for model size and for LLMs' known weakness in compositional, multi-step reasoning.","feed_headline":"Split a coding task into agents, small LLM gets 23.79% better","feed_subtitle":"A three-phase guided pipeline takes an 8B model from 45.4% to 56.2% on HumanEval.","key_machinery":"The load-bearing mechanism is a three-stage agent pipeline: a Generalist Agent recursively decomposes the root problem into a tree down to atomic leaves; a Code Agent solves each leaf with chain-of-thought prompting and test feedback; then the same Code Agent composes parent solutions from child documents only, never seeing child implementations. A Critic Agent and a Tester Agent validate each solution and feed corrections back into the next attempt. The paper also supplies a conceptual frame: leaf generation is fuzzy search over familiar code patterns, while parent composition is interface-level integration, so each stage plays to the model's strength.","core_discovery":"The central claim is that code generation should be treated as two different problems: producing small atomic functions, which LLMs do well because they can retrieve and adapt familiar code patterns, and composing those functions into a complete program, which LLMs do poorly when asked all at once. The paper's framework separates the two phases, isolates each sub-problem's context, and feeds validation feedback back into generation at every node. The authors report that this guided process outperforms one-shot generation on HumanEval and, in a qualitative test, produced a full mathematical expression evaluator with lexer, parser, evaluation algorithm, and error handling where larger frontier models either refused the task or gave simpler versions. The paper attributes the gain to three mechanisms: self-critique, problem decomposition, and upward composition.","pith_inferences":["Not tested in the paper: if a root-level test fails, the pipeline could re-decompose the root rather than only validating within subtrees, which would directly address the error-propagation weakness the authors acknowledge.","The paper's framing predicts the gain shrinks when atomic pieces are novel rather than retrievable, which one could check on benchmark domains outside the model's training distribution.","Because composition uses only interface documentation, the same pipeline could be applied to code repair and refactoring, treating each change as a validated leaf before merging."],"forward_implications":["If the framework's gain is real, smaller quantized models can approach code-generation accuracy that would otherwise require much larger models, by spending more structured effort per task.","Code generation becomes a divide-and-conquer engineering process rather than a single model call, so long, multi-file tasks become tractable within fixed context windows.","The same pipeline should transfer to new tasks without retraining, because the agents only need the documentation of child functions to compose parents.","Frontier models may push the gain further, since their chain-of-thought abilities are stronger, as the paper itself suggests."],"supporting_citations":[{"why":"Provides the HumanEval benchmark that supplies the tasks and the one-shot Pass@1 baseline used in evaluation.","marker":"[11]"},{"why":"Supplies the chain-of-thought prompting technique used by the Generalist and Code Agents for planning, decomposition, and solution refinement.","marker":"[12]"},{"why":"Documents LLMs' limited compositional ability, the limitation the framework is designed to work around.","marker":"[8]"},{"why":"Cited as the source for the int4-quantized 8B model on which the experiments run.","marker":"[10]"}],"fun_headline_variants":["Agent framework guides LLMs to 23.79% better code","Break code into steps, 8B LLM gains 23.79%","Multi-agent pipeline lifts 8B model's HumanEval score 23.79%","Guided multi-agent generation beats one-shot by 23.79%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gain rests on the first decomposition being correct: the paper says an incorrect initial solution to the root problem propagates through the whole tree, and it assumes that independently validated leaf functions compose correctly from their documentation alone.","fun_headline_variants_meta":{"raw":{"variants":["Agent framework guides LLMs to 23.79% better code","Break code into steps, 8B LLM gains 23.79%","Multi-agent pipeline lifts 8B model's HumanEval score 23.79%","Guided multi-agent generation beats one-shot by 23.79%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000256,"raw_usage":{"total_tokens":1534,"prompt_tokens":866,"completion_tokens":668,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":584}},"tokens_in":482,"tokens_out":668,"duration_ms":83007,"temperature":1.0,"reasoning_tokens":584,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:56:12.194596+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ablate the Generalist Agent's decomposition by giving the pipeline a fixed, task-independent decomposition tree on the same HumanEval problems; if Pass@1 stays at 56.2%, the decomposition is not the driver, and if it falls to 45.4% or lower, the central claim is supported. Alternatively, feed a deliberately incorrect root plan and observe the predicted error propagation in the final code.","supporting_citations":[{"cited_title":"”Chain-of-thought prompting elicits reasoning in large lan- guage models,” in Advances in neural information processing systems, vol","cited_arxiv_id":null,"evidence_quote":"Supplies the chain-of-thought prompting technique used by the Generalist and Code Agents for planning, decomposition, and solution refinement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents LLMs' limited compositional ability, the limitation the framework is designed to work around."},{"cited_title":"Teknium, J","cited_arxiv_id":null,"evidence_quote":"Cited as the source for the int4-quantized 8B model on which the experiments run."}],"review_version":1}