Pith. sign in

REVIEW 4 major objections 6 minor 15 references

Role-based multi-agent code generation matches developer structure better than a single LLM on real Java repositories, yet still falls far short of complete human systems.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 20:54 UTC pith:BKSXY52V

load-bearing objection Solid empirical comparison of reflexive multi-agent vs single GPT-5 on 12 post-cutoff Java repos; relative gains look real, absolute claims are limited by SRS reconstruction and static-only assessment. the 4 major comments →

arxiv 2607.04212 v1 pith:BKSXY52V submitted 2026-07-05 cs.SE

An Evaluation of Role-Based Multi-Agent Code Generation on Repository-Scale Problems

Benedetta Donato , Noah Hagar-Dent , Aaron Worsnop , Leonardo Mariani , Valerio Terragni This is my paper
classification cs.SE
keywords multi-agent code generationrole-based agentsrepository-scaleLLM software engineeringJavarequirements-driven generationreflexive planningempirical evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether splitting large-scale code generation across specialised LLM agents that take software-team roles (planner, coder, assessor) improves on a single LLM when the target is a full Java repository rather than a toy function. The authors build a sequential pipeline and a reflexive variant that repeatedly plans, codes, and assesses against a requirements document, then compare both to a standalone model on twelve real GitHub projects stratified by size and complexity. They find that the multi-agent systems, especially the reflexive one, produce better-structured code that is syntactically and semantically closer to the original developer implementations and that covers more of the intended behavioural scenarios. At the same time the generated systems remain partial: they capture only a fraction of the original size, many classes fail to compile, and substantial human work is still required. The result matters because it shows both the practical gain of agentic workflows on realistic repository-scale problems and the clear remaining distance from fully automatic, working systems.

Core claim

On twelve repository-scale Java projects, role-based multi-agent generation—particularly the reflexive loop that re-plans after static assessment—yields higher syntactic similarity (JPlag, CrystalBLEU) and higher scenario-intent coverage than a standalone LLM, while the generated code remains far smaller than the original repositories and largely non-compiling.

What carries the argument

The sequential/reflexive multi-agent pipeline: a planning agent decomposes an SRS into ordered tasks and role-specialised agents; coding agents emit Maven Java and pass summaries; an optional assessment agent flags missing requirements and restarts planning for up to ten cycles; a final setup stage produces the build files.

Load-bearing premise

The requirements documents synthesized from each repository’s README and Javadoc, after only structural manual checks, are faithful enough proxies for the true developer intent that similarity and scenario-intent scores against the original code fairly measure generation quality.

What would settle it

Replace the synthesized SRSs with independently written, human-authored requirements for the same twelve projects and re-run the identical agentic and single-LLM pipelines; if the multi-agent similarity and scenario-intent advantages disappear or reverse, the central claim does not hold under cleaner requirements.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper evaluates role-based multi-agent code generation (sequential and reflexive pipelines with Planning/Coding/Assessment/Setup) against a standalone GPT-5 baseline on 12 real-world Java repositories stratified into Trivial/Simple/Moderate/Complex groups. Requirements are reconstructed as IEEE-style SRSs from each repository’s README and Javadoc via GPT-5 (manually checked for structure and absence of leaked signatures). Metrics include process size, generated LOCs/methods/classes and compile rate, JPlag max similarity, CrystalBleu, an LLM-as-judge scenario-intent (shared Gherkin scenarios) rate, and resource cost. The main claim, supported by Table 2 averages over five runs, is that agentic workflows—especially the reflexive variant—produce higher syntactic/data-flow similarity and higher scenario-intent coverage than a single LLM, while still generating only a small fraction of original size, with low immediate compile rates, and thus remain far from complete human implementations.

Significance. If the relative ranking holds under a cleaner requirements setup, the work is a useful empirical contribution for IEEE Software’s agentic-systems theme: it moves evaluation beyond HumanEval/MBPP-scale tasks to repository-scale Java projects, reports transparent stratified sampling and multi-metric results, and delivers practical lessons (reflexive multi-agent structure helps; full working systems are not yet automatic; human-in-the-loop remains necessary). Strengths include five-run averaging, public experimental data, honest reporting of partial size and low compile rates, and a clear process architecture (Figure 1). The significance is currently limited by load-bearing methodological choices around SRS construction and non-executable similarity metrics, which must be tightened before the ranking can be read as strong evidence of generation quality rather than reconstruction of already-visible structure.

major comments (4)
  1. Repository Selection and Design of the Experiment: The SRSs are synthesized by GPT-5 from each target repository’s README and all Javadoc, then only manually checked for section coherence and absence of explicit interfaces/signatures. These same repositories are the ground truth for JPlag, CrystalBleu, and scenario-intent. Residual leakage of domain vocabulary, layering, and scenario structure into the SRS can inflate absolute similarity for every arm while still preserving relative ranking. The paper acknowledges that SRSs “cannot be as accurate as the ones written by developers” yet treats them as fair non-leaking proxies. This assumption is load-bearing for Table 2’s central claim. Please either (a) provide a stronger validation protocol (e.g., independent human-written SRSs for a subset, or blinded leakage audits with quantitative leakage scores), or (b) reframe claims strictly as re
  2. Similarity / scenario-intent subsection and Table 2: Scenario-intent is defined via dual LLM-as-judge steps (Claude Code/Sonnet extracts Gherkin scenarios from original and generated code w.r.t. the SRS; ChatGPT 5.4 matches shared scenarios). The paper correctly states this measures “scenario-intent representation, not functional correctness,” but Table 2 and the Lessons Learned still use the 78% Agentic-refl figure as a primary quality signal. Given that the SRS itself is LLM-derived from the original artifacts, the metric risks measuring reconstruction of textually visible intent rather than generation quality. Please report inter-judge agreement or human validation rates for the Gherkin extraction/matching on a sample of projects, and separate more clearly in the abstract and conclusions what is claimed about structure/intent vs. executable behavior.
  3. Table 2 Size columns and accompanying discussion: Generated code is only ~1%–25% of original LOCs/methods/classes depending on group, while JPlag/CrystalBleu are computed by matching each generated class to its most similar original class and averaging. When the generated artifact is a small skeleton, high class-level similarity can reflect shared naming/domain abstractions rather than repository-scale completeness. The paper notes partial coverage, but the similarity claims need an explicit size-controlled or coverage-normalized analysis (e.g., similarity only over matched core domain classes; or reporting how much of the original is covered by any match). Without this, “greater similarity to developer code” is easy to over-read relative to the large absolute gap.
  4. Assessment phase and Table 2 Compile column: Reflexive assessment is static and text-based against the SRS (no compiler diagnostics, tests, or runtime feedback), with a hard cap of ten iterations. Agentic-refl’s best compile rate is still only ~40% overall. The claim that reflexive multi-agent “outperforms” is therefore mainly about structure and intent metrics, not buildability. This is acknowledged, but Lessons Learned and the abstract should more sharply condition the outperformance claim on the metrics that the loop can actually improve, and either add a compiler-in-the-loop ablation or clearly mark its absence as a primary limitation of the evaluation rather than only of the system.
minor comments (6)
  1. Table numbering inconsistency: complexity thresholds are labeled TABLE 1 in the text flow but the main results table is TABLE 2; the prose also refers to “Table 2 defines the criteria for each level” when describing thresholds—please align labels and cross-references.
  2. Typographical issues: “Y et”, “entiresoftware systemsremains”, “depeding”, “synchornization”, “apom.xmlfile”, and similar spacing/typos should be cleaned for production.
  3. Figure 1 is referenced but not fully described in text beyond phase names; a short caption-level walkthrough of data passed between Planning → Coding → Assessment → Setup would help readers who skim.
  4. Baseline fairness: the standalone LLM uses the same SRS and GPT-5 but is asked to generate code “directly.” Please state context-window / output-length handling for large SRSs and whether the single-LLM arm was allowed multi-turn self-repair comparable in token budget to Agentic-seq (even if not role-based).
  5. Resources: costs are approximate OpenAI costs on a fixed MacBook timing host; note model pricing volatility and whether token counts include planning/assessment only or full chat history.
  6. Related work: MetaGPT, AgileCoder, and RTADev are cited for small-scale evaluation; a short explicit comparison of role definitions and whether any of those frameworks could be run on the same 12 SRSs would strengthen positioning.

Circularity Check

2 steps flagged

Partial evaluation circularity: SRS reverse-engineered from target repos' README+Javadoc, so similarity/scenario metrics partially reconstruct documented structure rather than pure independent generation.

specific steps
  1. other [Repository Selection and Design of the Experiment (SRS synthesis paragraph)]
    "To generate a structured SRS for each project, since SRS are rarely available in GitHub repositories, we used the README file and all Javadoc comments as input to OpenAI GPT-5. The list of the implemented functionalities, taken from the Javadoc, paired with high-level project descriptions, taken from the readme file, provide a fairly complete overview of the implemented requirements. We manually validated each SRS to ensure proper structure: We checked that the content of each section is coherent with the section, and that no implementation details are disclosed in the SRS (e.g., interfaces, m"

    The sole input requirements document for every generation run is reverse-engineered by GPT-5 from documentation belonging to the exact repositories later used as ground truth. Domain concepts, feature inventories, and structural cues therefore enter the prompt from the target itself. Measured JPlag/CrystalBleu similarity and scenario-intent coverage against that target consequently partially reconstruct already-documented content rather than evaluate generation from independent external requirements. Manual removal of signatures does not eliminate this leakage of vocabulary and intent. Both multi-agent and LLM-only arms are affected equally, preserving relative ranking but inflating absolute similarity claims by construction of the evaluation inputs.

  2. other [Similarity metrics paragraph (LLM-as-a-judge / scenario-intent)]
    "we used Claude Code (with Sonnet 4.6) to analyze the original code and the generated implementations with respect to the input SRS. We instructed Claude Code to identify, for each requirement in the SRS, the scenarios represented in the code and to express them as Gherkin scenarios. ... We then used ChatGPT 5.4 to identify shared scenarios between the original and generated implementations. We report the rate of shared scenarios, defined as the ratio of scenarios identified in the original implementation that are also identified in the generated implementation."

    Scenarios are first extracted from the original (ground-truth) code using an SRS that itself was synthesized from that same code's README/Javadoc. Shared-scenario rate therefore measures how well generated code echoes structures already present in the original and already injected into the SRS, rather than independent functional coverage of external requirements. The metric is acknowledged as 'scenario-intent representation, not functional correctness,' yet is still reported as evidence of multi-agent superiority; the dependence on original-derived inputs makes the absolute rates partially circular.

full rationale

This is an empirical evaluation paper, not a first-principles derivation with equations or uniqueness theorems. The comparative claim (multi-agent, especially reflexive, yields higher JPlag/CrystalBleu and scenario-intent than standalone LLM on the same 12 repos) is not forced by construction: both arms receive identical SRS, absolute similarities remain modest (overall JPlag 0.43 max), sizes are 1-25% of originals, and compile rates stay low. However, a load-bearing design choice creates partial circularity: the input SRS for every project is synthesized by GPT-5 from that project's own README plus all Javadoc, then only manually checked for section coherence and absence of signatures/interfaces. Domain vocabulary, feature lists, and high-level structure therefore flow from the ground-truth repositories into the generation prompt; subsequent syntactic, data-flow, and LLM-as-judge scenario matching against those same repositories therefore partially measures reconstruction of already-visible documentation rather than generation from independent requirements. The paper itself notes the SRSs 'cannot be as accurate as the ones written by developers' yet treats them as fair non-leaking proxies. Scenario-intent extraction (Claude on original+generated vs SRS, then GPT matching) inherits the same dependence. No self-citation chain, fitted parameters renamed as predictions, or ansatz smuggling is present. Relative ranking between arms remains informative; absolute claims of 'greater similarity to developer code' are inflated by the shared leakage. Score 4 reflects this partial, non-tautological circularity confined to evaluation design.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central comparative claim rests on standard SE evaluation practice plus several paper-specific choices: auto-generated SRS as requirement proxies, chosen complexity bins, static text-based assessment loops, and non-functional similarity/intent metrics as stand-ins for system quality. No new physical entities; free parameters are design knobs (iteration cap, sampling thresholds, model choice).

free parameters (4)
  • reflexive_assessment_iteration_cap = 10
    Assessment/repair loop runs until no issues or up to ten times; this hand-chosen cap shapes Agentic-refl cost and quality.
  • complexity_group_thresholds = e.g. Complex >40K LOC, >300 classes
    LOC/class/method/CC cutoffs define Trivial/Simple/Moderate/Complex sampling strata (Table 1); chosen to approximate top percentiles of 2025 Java repos.
  • repositories_per_stratum = 3 per group
    Three randomly selected repos per complexity group (n=12 total) determine the empirical support for all claims.
  • base_LLM_choice = GPT-5 default
    GPT-5 default configuration used for both standalone and agentic runs; results may not transfer to other models.
axioms (5)
  • ad hoc to paper SRS reconstructed from README + Javadoc (then manually checked) adequately represent functional/non-functional requirements without leaking implementation details.
    Core evaluation premise in Repository Selection and Design of the Experiment; if false, similarity to original code is confounded.
  • domain assumption JPlag max similarity and CrystalBLEU, averaged over best class matches, are meaningful proxies for closeness to developer implementations when tests cannot run.
    Stated justification for similarity metrics when APIs differ; supports the 'greater similarity' claim.
  • domain assumption LLM-as-a-judge Gherkin scenario matching measures scenario-intent representation, not functional correctness.
    Authors explicitly define the metric this way; readers must not treat 78% scenario-intent as working behavior.
  • ad hoc to paper Static text-based assessment against the SRS is a valid reflexive improvement mechanism even without compiler or test feedback.
    Defines Agentic-refl; authors later note this limits compile-error repair.
  • domain assumption Selecting 2025 GitHub Java repos ranked by stars sufficiently reduces training-data leakage for GPT-5 (cutoff Sept 30, 2024).
    Leakage control strategy; residual risk from common frameworks is acknowledged.
invented entities (2)
  • Sequential and reflexive role-based multi-agent code generation pipeline (Planning/Coding/Assessment/Setup) no independent evidence
    purpose: Operationalize agentic repository-scale generation for comparison against a standalone LLM.
    Pipeline is the paper's experimental system, built similarly to prior agentic processes but evaluated at larger scale; not claimed as a new fundamental object of nature.
  • Scenario-intent shared-scenario rate via dual-LLM Gherkin extraction/matching no independent evidence
    purpose: Provide a semantic-ish coverage metric when tests cannot execute on generated APIs.
    Custom evaluation construct; authors inspect samples but provide no external validation benchmark for the metric.

pith-pipeline@v1.1.0-grok45 · 14267 in / 3541 out tokens · 44767 ms · 2026-07-11T20:54:40.905069+00:00 · methodology

0 comments
read the original abstract

Role-based multiagent code generation aims to make LLMs more effective on repository-scale problems, moving beyond small programming tasks. We evaluate this approach on 12 Java repositories, finding greater similarity to developer code than single LLMs, but a persistent gap from human implementations.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 1 linked inside Pith

  1. [1]

    X. Hou, Y . Zhao, Y . Liu, Z. Y ang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang. “Large language 6 An Evaluation of Role-Based Multi-Agent Code Generation on Repository-Scale Problems Month 2026 Special Issue on Engineering Agentic Systems models for software engineering: A systematic literature review,” ACM Trans. on Soft. Eng. and Meth. vol. ...

    2026

  2. [2]

    De- mystifying LLM-Based Software Engineering Agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang. “De- mystifying LLM-Based Software Engineering Agents,” Proc. ACM Softw. Eng. 2, FSE, 2025 (Conference proceedings)

    2025

  3. [3]

    Large Language Models for Software Engineering: Survey and Open Problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Y oo, J. M. Zhang. “Large Language Models for Software Engineering: Survey and Open Problems,” Proc. IEEE/ACM Int. Conf. on Softw. Eng.: Future of Software Engineering, 2023 (Conference proceedings)

    2023

  4. [4]

    A Survey on Code Generation with LLM- based Agents,

    Y . Dong, X. Jiang, J. Qian, T. Wang, K. Zhang, Z. Jin, G. Li, “A Survey on Code Generation with LLM- based Agents,” unpublished, arxiv.org/abs/2508.00083 (Unpublished manuscript)

    Pith/arXiv arXiv

  5. [5]

    LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead,

    J. He, C. Treude, D. Lo. “LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead,” ACM Trans. Softw. Eng. Methodol. 34, 5, 2025 (Journal)

    2025

  6. [6]

    Code digital twin: Empowering LLMs with tacit knowledge for complex software maintenance,

    X. Peng, C. Wang, M. Liu, Y . Lou, and Y . Wu, “Code digital twin: Empowering LLMs with tacit knowledge for complex software maintenance,” https://arxiv.org/abs/2503.07967 (Unpublished manuscript)

    arXiv

  7. [7]

    Systems and software engineering — Life cycle processes — Requirements engineering,

    IEEE. “Systems and software engineering — Life cycle processes — Requirements engineering,” International Standard, ISO/IEC/IEEE 29148, 2018 (standard)

    2018

  8. [8]

    Prechelt, G

    L. Prechelt, G. Malpohl, and M. Philippsen, “JPlag: Finding plagiarisms among a set of programs,“ 2000. Univ., Fak. für Informatik. (report)

    2000

  9. [9]

    Self-Planning Code Generation with Large Language Models,

    X. Jiang, Y . Dong, L. Wang, Z. Fang, Q. Shang, G. Li, Z. Jin, and W. Jiao. 2024. “Self-Planning Code Generation with Large Language Models,” ACM Trans. Softw. Eng. Methodol. 33, 7, Article 182, 2024. (Journal)

    2024

  10. [10]

    AFlow: Automating Agentic Workflow Generation,

    J. Zhang, J. Xiang, Z. Yu, F . Teng, X.-H. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang, B. Zheng, B. Liu, Y . Luo, C. Wu. “AFlow: Automating Agentic Workflow Generation,” International Conference on Learning Representations (ICLR), 2025. (Conference proceedings)

    2025

  11. [11]

    CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code,

    A. Eghbali, M. Pradel. “CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code,”. Int. Conf. Automated Software Engineering (ASE), 2022. (Con- ference proceedings)

    2022

  12. [12]

    MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework,

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Y au, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhu- ber. “MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework,” International Conference on Learning Representations (ICLR), 2024. (Conference proceedings)

    2024

  13. [13]

    AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology,

    M. H. Nguyen, T. P . Chau, P . X. Nguyen, and N. D. Q. Bui. “AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology,” Forge, 2025. (Conference proceedings)

    2025

  14. [14]

    RTADev: Intention Aligned Multi-Agent Framework for Software Development,

    J. Jie, Z. Chen, X. Wang, and others. “RTADev: Intention Aligned Multi-Agent Framework for Software Development,” Proc. Annual Meeting of the Association for Computational Linguistics (ACL), 2025. (Conference proceedings)

    2025

  15. [15]

    Hu- manEval Pro and MBPP Pro: Evaluating Large Lan- guage Models on Self-invoking Code Generation,

    Z. Yu, Y . Zhao, A. Cohan,and X. P . Zhang. “Hu- manEval Pro and MBPP Pro: Evaluating Large Lan- guage Models on Self-invoking Code Generation,” Association for Computational Linguistics (ACL), 2025. (Conference proceedings) Benedetta Donatois a Ph.D. stu- dent in Computer Science at the University of Milano-Bicocca. Her current research interests include...

    2025