{"id":"20c7cf2e-a7f3-412a-9335-2185eea5383f","arxiv_id":"2607.27877","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new multi-agent coding benchmark (MSEval) shows that collaboration topology—not just model ability—strongly shifts the speed, cost, and quality of LLM-built software.","lead":"This paper introduces MSEval, a benchmark that runs teams of LLM agents on ten real full-stack web projects under ten different coordination topologies, measuring functional quality, wall-clock time, and token cost. It reports that how agents are organized changes scores by more than 30 points, rivaling the choice of model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No manipulation check verifies the ten 'modes' produced the intended coordination behaviors; the 30-point spread may reflect prompt artifacts or agent noncompliance rather than the named policies.","rationale":"The reader's weakest assumption is exactly the mode-fidelity assumption, and I agree that it is the most load-bearing point. The paper's headline finding is causal—organizational topology determines the speed–cost–quality trade-off—but the independent variable is only defined by templates, not verified in behavior. A manipulation check is the usual scientific safeguard for treatments that are implemented through prompts and scheduling; without it, the 30-point spread could be an artifact of incidental prompt differences or agent noncompliance. Secondary concerns like single-run cells and dropped models are real but less fundamental: even perfect statistics would not rescue an unvalidated treatment. The paper does provide some supporting evidence—a detailed worked trace, TAgent's Spearman agreement with human TAs, and concrete cost accounting—but none of it directly validates mode fidelity. Therefore I would keep the reader's CONDITIONAL verdict: the benchmark could be accepted if the authors add behavioral audits of all ten modes and release the code/logs, but currently the central causal claim is not fully established.","tokens_in":74,"tokens_out":5340,"duration_ms":71732,"concrete_test":"Release full LegoGent execution logs for at least one complete run of each mode and run an automated behavior auditor that checks mode-specific invariants from timestamps, file edits, and agent messages: (1) pipeline: no stage n+1 file modifications before the stage n artifact is frozen; (2) rotation: role assignments change between rounds; (3) PM oversight: each commit is preceded by a manager approval message; (4) feature squad: file ownership stays within assigned modules; (5) QA-first: acceptance-test files exist before feature implementation code. If any mode fails its invariants in a majority of rounds, recompute the 30-point claim using only runs where the mode was behaviorally confirmed, or re-label the conditions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is causal: for identical tasks and models, varying coordination topology shifts scores by over 30 points and doubles wall-clock time. That inference depends on the assumption that each mode was actually instantiated as specified and that the only systematic difference between conditions is the coordination policy. The paper states 'The collaboration mode is the only independent variable' (Methodology), but it reports no manipulation check confirming that the ten LegoGent modes behaved as intended. Table 7 defines modes by ownership map, activation schedule, decision rule, and handoff artifact, yet the paper does not audit execution logs to verify, for example, that pipeline stages were truly serial, that rotation actually changed roles each round, that PM oversight actually gated commits, or that feature-squad agents did not cross module boundaries. The single worked trace in Figure 7 is from a feature-squad run and cannot validate the other nine modes. Without such evidence, the observed score swings and latency differences could be driven by accidental prompt wording, template bugs, or agents ignoring mode constraints—making 'topology' a label rather than the manipulated variable. This is the most load-bearing weakness because it directly undercuts the causal attribution that the abstract and conclusions assert.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MSEval, a benchmark for from-scratch multi-agent software construction, together with the LegoGent execution runtime and the TAgent automated grader. Ten real-world web projects, ten collaboration topologies, and five LLMs are combined into a 100-configuration grid (with three full-grid models run on all ten projects). The central claim is that, for identical tasks and models, the collaboration topology changes functional scores by over 30 points and doubles wall-clock time, making organizational policy a first-order determinant of the speed–cost–quality trade-off. The paper also reports per-round refinement under TAgent feedback, a failure taxonomy, and a detailed worked trace. The benchmark and infrastructure are released.","tokens_in":21107,"tokens_out":3496,"duration_ms":34512,"significance":"If the central claim is substantiated, this is a genuinely useful contribution: it moves multi-agent coding evaluation from synthetic, single-attempt settings to deployment-grounded, cost-aware, multi-round measurement. The design has real strengths: deterministic rubric-based scoring, native CI/CD deployment, cache-aware token/cost accounting, three-round refinement with item-level feedback, a clear template-based mode taxonomy, and a released codebase. The paper also ships a concrete worked example with execution logs. However, the quantitative support for the headline claim is currently fragile: most project-by-mode cells are single runs, no statistical inference is reported, and the causal attribution to 'topology' rests on an untested assumption that the ten modes were behaviorally instantiated as specified. These issues are fixable and do not undermine the value of the infrastructure, but they must be addressed before the empirical conclusions can be accepted.","major_comments":[{"comment":"The causal claim—'varying the topology shifts scores by over 30 points and doubles wall-clock time'—requires that the ten LegoGent modes actually implemented the intended coordination policies and that nothing else varied systematically. The paper states that the collaboration mode is the only independent variable, but it reports no manipulation check. Table 7 defines each mode by ownership map, activation schedule, decision rule, and handoff artifact, yet there is no evidence from execution logs that, for example, pipeline stages were truly serial, rotation actually permuted roles each round, PM oversight gated commits, or feature-squad agents respected module boundaries. Figure 7 shows a single worked trace from the feature-squad mode and cannot validate the other nine. Without an audit of the remaining 90+ configurations, the observed score and latency differences could be driven by p","section":"Methodology, 'The collaboration mode is the only independent variable'; Table 7; Figure 7"},{"comment":"The quantitative evidence for the headline 30-point spread is thin. Most project-by-mode cells in Tables 2 and 3, and all cells in Table 8, appear to derive from a single run (the limitation only says three parallel trials were run for the instant-messaging token/time evaluation). No confidence intervals, standard errors, or significance tests are provided. This matters because the cross-mode average range in Table 8 is narrow (73.5 to 83.3), and the 'over 30 points' figure comes from one model/project comparison (DeepSeek v4 Pro: pipeline 89.9 vs open-source 43.0 on P00). Without variance estimates, it is impossible to tell whether the observed mode differences are robust or partly noise. Please report per-condition repetition (at least for the headline claims), or provide a clear statistical model and effect sizes with uncertainty.","section":"Evaluation, Tables 2/3/8 and Limitations ('Randomness and scale')"},{"comment":"Qwen3.6-Flash is run on all ten modes of the reference project, scores 0 everywhere, and is then dropped from the results. This changes the claimed model comparison and the '100 runs' count. The decision to omit is reasonable only if the zero is diagnostic of the model rather than a bug in the experimental setup (e.g., a systematic failure to interact with the LegoGent runtime). Please report the Qwen runs in a table, state the exact failure mode (deployment? rubric mismatch? agent noncompliance?), and include this exclusion in the limitations. As written, the reader cannot tell whether Qwen is incapable or whether the harness was incompatible.","section":"Evaluation, 'Qwen3.6-Flash... we omit it from the results'"},{"comment":"The claim that '82.0% of adjacent transitions improve' is partly a property of the loop under study: TAgent defines the Functional Completion Score and its feedback is the refinement signal, so improvement toward the same rubric is expected. This is not a fitted-parameter circularity, but it means the refinement result is not an independent validation of TAgent or of the agents' learning ability. The reported Spearman correlation of 0.87 with TA scores is on student projects, not on agent-generated artifacts. Please add a validation of TAgent scoring on a sample of agent-generated runs (e.g., human expert re-scoring), and discuss the degree to which the refinement numbers are a self-consistency measure rather than external evidence of quality.","section":"'TAgent Feedback Drives Refinement' and 'TAgent Verification Methodology'"}],"minor_comments":[{"comment":"The abstract says 'Across 100 runs' and Figure 1 says '100 Team Configurations', but the full-grid evaluation (10 projects × 10 modes × 3 models) implies 300 runs. Please clarify whether '100' refers to configurations, project–mode pairs, or something else, and align the wording.","section":"Abstract and Figure 1"},{"comment":"The column headers 'T', 'C', 'Tok' are used without definition in the caption or text. Define them explicitly (wall-clock minutes, USD cost, million tokens) in each table caption.","section":"Table 2 and Table 3"},{"comment":"The axis titles 'USD to best round' and 'Minutes to best round' are ambiguous: are these cumulative to the best round, or the value at the best round? Clarify the exact definition of the plotted quantities.","section":"Figure 4"},{"comment":"The radar figure uses ten normalized axes, with seven from logs and three from protocols, but the mapping from axis labels to raw metrics is not given in the caption or text. Add a table or legend specifying the construction of each axis, especially 'Ownership', 'Inspect', and 'Parallel'.","section":"Figure 5"},{"comment":"The pricing table lists 'Cache-wr' for Claude and Qwen but '—' for others. This is plausible for provider billing, but the phrase 'cache writes fold into new input' is stated only in the caption. State explicitly which models bill cache-write separately and how the conversion from CNY to USD is applied (the fixed 6.6 rate is given, but the date of the rate is not).","section":"Table 9"},{"comment":"There are several typographical and formatting issues: the abstract is a single run-on paragraph, the reference 'Qwen Team 2026' is incomplete (no arXiv or URL), and some table cells in Table 2 contain ambiguous values (e.g., '1,481' vs '11481' in the Feat./Claude row seems to be a line-break artifact). Please proofread the final version.","section":"Minor editorial"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical claims are potentially interesting but currently rest on a small number of unrepeated runs and on an unverified assumption that the ten modes were instantiated faithfully. I would encourage the editor to solicit a revised version that addresses the manipulation-check and statistical-uncertainty concerns; the infrastructure release makes such a revision feasible. The paper also has a somewhat unusual number of future-dated references (2026), which is consistent with the arXiv listing but may warrant a quick check of citation validity during production."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Candid take: MSEval is the first benchmark I know that combines from-scratch full-stack projects, explicit coordination topologies, CI/CD deployment, and token/latency/cost accounting in one instrument. That combination is real and useful; the failure taxonomy (1,440 checks) and the worked refinement trace are the strongest parts. The paper is worth a serious referee. The headline causal claim—topology shifts scores by over 30 points—should not be accepted as stated.\n\nStrengths: Table 7's four-template mode definitions (ownership map, activation schedule, decision rule, handoff artifact) are concrete enough to reproduce. Cost accounting with provider price tables and per-round token breakdown is careful. TAgent's dependency-aware scheduling and item-level feedback are thoughtful. The 0.87 Spearman correlation with TA scores is useful validity evidence, even if limited.\n\nSoft spots, in order: (1) No manipulation check. The paper says mode is the only independent variable, but no audit shows pipeline stages were serial, rotation changed roles, PM gates were enforced, etc. The single feature-squad trace doesn't validate the other nine. Without that, the 30-point spread could be prompt wording, template bugs, or agents ignoring constraints. This is a load-bearing gap for the causal claim, not fatal for the benchmark's value. (2) Statistics. Most cells are single runs, no confidence intervals or tests. Triplicates only for P00 token/time. The abstract's 'structured pipelines converge fastest with highest quality' is not what Table 8 shows: QA-first and rotation tie for best average, pipeline is mid-pack, and PM oversight is near the top despite 'managerial oversight degrades'. (3) TAgent is both grader and feedback signal, so the 82% improvement is partly a property of the loop, not independent evidence about topologies. (4) Abstract says released, appendix says will release; that needs to be resolved before reproducibility claims. (5) Dropping Qwen after zero across the board is defensible, but it means 'five models' is really three for the full grid.\n\nBottom line: for benchmark designers and anyone studying multi-agent coordination, this is a valuable scaffold. If the code appears with a commit hash and the manipulation check closes, it could become a standard. I'd send it to peer review with a request for code, a manipulation check, and conservative rewording of the causal claims. Desk rejection would be a loss.","headline":"MSEval is a genuinely useful evaluation instrument for multi-agent coding, but the headline causal claims about topology outrun the evidence; send it to review, don't take the 30-point claim at face value.","tokens_in":21652,"tokens_out":3925,"would_cite":true,"duration_ms":37719,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For identical tasks and models, changing only how agents coordinate shifts functional scores by over 30 points and doubles wall-clock time, placing organizational topology on par with model capability.","keywords":["multi-agent coding","collaboration topology","coordination mode","LLM agents","software engineering benchmark","from-scratch coding","functional completion score","token cost"],"falsifier":"Run the same project and model under two modes, then swap only the mode-template text while keeping the rest of the agent prompts, sync schedule, and runtime identical; if the score difference persists, it is caused by the template content rather than the coordination policy. Alternatively, instrument agent logs to verify mode-specific behaviors—e.g., in pipeline mode confirm the backend stage never writes code before the architect stage completes, and in rotation mode confirm roles actually swap each round; a mismatch between commanded and observed behavior would undermine the causal claim.","tokens_in":20690,"feed_emoji":"⚙️","tokens_out":6029,"duration_ms":54195,"temperature":0.7,"pith_summary":"MSEval puts coordination policy on the same footing as model capability by measuring how teams of LLM agents build ten full-stack projects from scratch across ten collaboration topologies. For identical tasks and models, swapping the topology shifted functional scores by more than 30 points and roughly doubled wall-clock time, making organizational design a first-order determinant of software delivery. Structured pipeline handoffs converged fastest at the highest quality, while heavy managerial oversight and open-source-style review degraded performance. The authors note the evaluation scale is bounded by cost—three parallel trials on the reference project, spot reruns elsewhere—and scope is limited to web applications. The benchmark's ceiling appears to be integration completeness and cross-agent contracts rather than raw code generation.","feed_headline":"Swapping team layout moves coding scores by 30 points","feed_subtitle":"Same task and model — coordination choices change speed, cost, and quality as much as model choice.","key_machinery":"The mode template: each of the ten collaboration topologies is an instance of four concrete objects—ownership map (who owns which part of the repository), activation schedule (who is awake in which round), decision rule (how conflicts resolve), and required handoff artifact (what must exist before a round is accepted). LegoGent executes this template with a periodic sync loop that broadcasts a progress snapshot roughly every four minutes, an active peer mailbox for targeted questions, and an all-idle-and-artifacts completion gate; TAgent then probes the deployed artifact through UI, API, and code checks and returns weighted item-level feedback that routes the next round.","core_discovery":"The paper's central claim is that the collaboration mode—defined by an ownership map, activation schedule, decision rule, and required handoff artifact—is a first-class experimental variable in multi-agent software development, not a prompt flourish. Using MSEval, the authors show that with the same task, model family, requirement document, deployment path, and rubric, varying only the topology moves the best-round functional score by over 30 points (e.g., one model ranged from 89.9 under pipeline to 43.0 under open-source review on the instant-messaging project) and doubles wall-clock time. Mode rankings hold in the aggregate—QA-first and rotation tie at 83.3 across ten projects while open-","pith_inferences":["If the mode-fidelity assumption holds, organization design becomes a tunable hyperparameter of an agentic coding stack; engineering teams could choose a mode per project by predicting the dependency structure (e.g., pipeline for transactional CRUD, feature squad for realtime composition).","A testable extension would be an ablation that varies only one template field at a time (e.g., fix the ownership map, swap the activation schedule) to isolate which of the four fields drives most of the 30-point spread.","The mode template suggests a taxonomy for future benchmarks: any multi-agent coding system could be described by an ownership map, activation schedule, decision rule, and handoff artifact, enabling cross-benchmark comparison without adopting this paper's runtime.","Because security/transport deductions come from the deployment layer in nearly 80% of runs, re-scoring with a more permissive deployment ceiling could change relative mode rankings on integration-heavy projects."],"forward_implications":["Coordination policy should be treated as a controlled experimental variable in future multi-agent coding benchmarks, not a prompt detail.","Fit matters more than a global best: serialized-handoff modes win projects with long requirement chains or clean stage boundaries, while parallel-contention modes pay a reconciliation tax unless redundancy is cheap to score.","Feedback-driven refinement works: 82% of adjacent rounds improve and 94.7% of runs finish above their first round, so three rounds capture most of the available gain.","The score frontier is not the cost frontier: high raw token counts do not necessarily mean high budget, since cache reads are far cheaper than fresh input, and the highest-scoring model can be several times slower and more expensive than a near-tied alternative.","The benchmark's ceiling is integration completeness and cross-agent contracts, not raw code generation: one broken prerequisite can cascade-skip up to 14 downstream checks, and security/transport items recur in nearly 80% of runs due to deployment-layer integration."],"fun_headline_variants":["Team layout moves coding scores by 30+ points","Same task, same model: topology swings score 30 points","Coordination mode rivals model capability in AI coding","Structured pipelines outperform heavy oversight in coding"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper asserts that the collaboration mode is the only independent variable across runs, which requires that the ten mode templates differ only in the four defined template fields (ownership map, activation schedule, decision rule, handoff artifact) and not in prompt wording, timing noise, or hidden runtime details; no manipulation check is reported to confirm that agents actually followed the intended mode, so the 30-point spread could in principle be an artifact of accid","fun_headline_variants_meta":{"raw":{"variants":["Team layout moves coding scores by 30+ points","Same task, same model: topology swings score 30 points","Coordination mode rivals model capability in AI coding","Structured pipelines outperform heavy oversight in coding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1272,"prompt_tokens":752,"completion_tokens":520,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":470}},"tokens_in":496,"tokens_out":520,"duration_ms":4733,"temperature":1.0,"reasoning_tokens":470,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T23:45:33.433252+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same project and model under two modes, then swap only the mode-template text while keeping the rest of the agent prompts, sync schedule, and runtime identical; if the score difference persists, it is caused by the template content rather than the coordination policy. Alternatively, instrument agent logs to verify mode-specific behaviors—e.g., in pipeline mode confirm the backend stage never writes code before the architect stage completes, and in rotation mode confirm roles actually swap each round; a mismatch between commanded and observed behavior would undermine the causal claim.","supporting_citations":[],"review_version":1}