{"id":"88e725a4-2b2c-45f9-be32-fa2e1abb69a1","arxiv_id":"2505.18286","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On 15 agentic benchmarks, the accuracy advantage of multi-agent LLM systems over single-agent systems mostly disappears with stronger base models, and a hybrid single/multi-agent cascade improves accuracy and cuts cost.","lead":"This paper compares single-agent and multi-agent LLM systems across many coding, math, and planning tasks, and finds that the accuracy advantage of multi-agent systems shrinks as the underlying models get stronger. It then proposes routing and cascading designs that choose between the two system types per request, claiming better accuracy at lower cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Within-family checks in the paper's own appendix contradict the headline: on SelfCol-DS1000, upgrading Llama-3-8B to Llama-3-70B increases the MAS−SAS accuracy gap from +11.2pp to +22.7pp (Tables 6-7), so 'benefits diminish with capability' is not robust.","rationale":"The reader's CONDITIONAL verdict is the right one, but for a reason that is only partially the reader's weakest assumption. The reader flagged confidence-guided tracing and the confound of model-family and prompt changes; I agree the cross-family Table 2 comparison is weak. However, the manuscript's own appendix contains a direct, same-family, same-framework counterexample to the central trend: Llama-3-70B is strictly more capable than Llama-3-8B, yet on SelfCol-DS1000 the MAS advantage increases from +11.2pp to +22.7pp rather than diminishing. Since this is exactly the controlled comparison the headline needs, the paper must either aggregate all cells statistically or weaken the claim. The cascade accuracy gain being definitional is a separate issue, and the confidence-score method does have an oracle-flip validation in Table 5, so I would not make that the primary objection. My concern does not overturn the paper's other findings, such as the large token-cost gap or the useful defect taxonomy, but it does mean the headline generalization is not yet supported. The appropriate outcome remains CONDITIONAL: the paper can be accepted only after reporting a proper analysis of the full table of MAS−SAS differences and revising the claim accordingly.","tokens_in":66219,"tokens_out":10792,"duration_ms":94489,"concrete_test":"Recompute the MAS−SAS accuracy difference for every cell in Tables 6-12 using the raw counts (MAS pass = MAS Win + Both Pass; SAS pass = SAS Win + Both Pass). Fit a regression of this difference on model capability (e.g., SAS accuracy or model tier) with framework×dataset fixed effects, bootstrap over datasets, and report the sign and confidence interval of the slope. Then check whether removing the Llama-3-70B DS1000 cell (+22.7pp advantage) or excluding AIME flips the conclusion. As an additional check, rerun SelfCol-DS1000 with 5 seeds for Llama-3-8B and Llama-3-70B to confirm that the advantage ordering is reproducible. If the negative trend is not significant once within-family cells are pooled, the abstract's 'benefits diminish' statement should be qualified to 'on some code and math benchmarks and model families.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing empirical claim is that MAS advantage shrinks as base LLMs improve. The paper's strongest evidence should be same-family comparisons, not Table 2's old-ChatGPT versus Gemini-2.0-Flash contrast, which changes model family, prompt version, and API generation simultaneously. The appendix contains a within-family comparison that goes the wrong way. On SelfCol-DS1000, Llama-3-8B (Table 7) gives MAS 25.3% vs SAS 14.1%, a +11.2pp advantage; Llama-3-70B (Table 6) gives MAS 47.7% vs SAS 25.0%, a +22.7pp advantage. The stronger model roughly doubles the MAS advantage. Other cells are also mixed: on AIME, MAS−SAS grows from +1.7pp (GPT-3.5, Table 9) to +5.0pp (GPT-4o, Table 8), and from +5.0pp (Gemini-1.5-Flash) to +13.3pp (Gemini-2.0-Flash) (Tables 12, 11); on HumanEval, Gemini-1.5→2.0 raises the advantage from +0.6pp to +3.1pp (Tables 12, 11). The paper acknowledges AIME as an exception but not DS1000 with Llama. Thus the headline 'benefits of MAS over SAS diminish as LLM capabilities improve' is not established by the paper's own data; it depends on favorable rows or on an implicit aggregation that is never computed. At minimum, the claim needs to be qualified and accompanied by a statistical analysis across all controlled cells.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a broad empirical comparison of single-agent (SAS) and multi-agent (MAS) LLM systems across 15 datasets and 9 frameworks, concluding that MAS accuracy advantages over SAS shrink as base models become more capable while MAS cost penalties remain. It then proposes a confidence-guided method for identifying critical agents in a MAS, and two hybrid mechanisms—agent routing and agent cascade—that alternate between SAS and MAS to improve the accuracy-cost tradeoff. The manuscript includes extensive appendix tables, ablations, case studies, and a human-annotated software-engineering evaluation.","tokens_in":66549,"tokens_out":5165,"duration_ms":43223,"significance":"If the central empirical claim were established, the paper would provide timely evidence for a reassessment of MAS deployment as frontier LLMs improve. The breadth of the evaluation—multiple model families (GPT, Gemini, Llama), multiple MAS frameworks, and both discrete and continuous metrics—is a genuine strength, and the graph-based defect taxonomy (node, edge, path) is a useful organizing framework. The routing and cascade designs are practical engineering contributions, and the cost analyses with prefill/decode token accounting are valuable. However, the headline claim is not currently supported by controlled comparisons, and the cascade accuracy improvement is a construction-level guarantee rather than an empirical discovery, so the paper's contributions need to be re-scoped.","major_comments":[{"comment":"The central claim that 'the benefits of MAS over SAS diminish as LLM capabilities improve' is not established by the paper's own controlled data. Table 2 compares original ChatGPT-era published results with Gemini-2.0-Flash runs, changing model family, prompt version, and—per Appendix B.1—also changing the SelfCol architecture from three agents to two with revised prompts. Within-family comparisons in Appendix C go in the opposite direction: on SelfCol-DS1000, Llama-3-8B (Table 7) shows MAS 25.3% vs SAS 14.1% (+11.2pp), while Llama-3-70B (Table 6) shows 47.7% vs 25.0% (+22.7pp). Similar increases appear for AIME for GPT-3.5→GPT-4o (+1.7pp in Table 9 to +5.0pp in Table 8) and for Gemini-1.5-Flash→Gemini-2.0-Flash (+5.0pp in Table 12 to +13.3pp in Table 11), and for HumanEval (+0.6pp to +3.1pp). The paper acknowledges AIME as a possible exception but not the DS1000/Llama cells or the GPT/Gemini math cells. The diminishing-benefits claim should be replaced by a per-cell analysis and a statistical test across the controlled within-family comparisons, or qualified to specific datasets and families.","section":"§3.2, Tables 2, 6-9, 11-12"},{"comment":"The headline that agent cascade 'improves accuracy by 1.1-12%' is a mathematical guarantee rather than an empirical finding. Under exact verification, the cascade output is correct whenever either the SAS or MAS output is correct, so cascade accuracy is the union of the two accuracies. The paper itself states in Appendix B.4 that 'the agent cascade uniformly has higher accuracy than SAS and MAS, because as long as either SAS or MAS is correct, it would be correct in the agent cascade.' Table 4 should therefore be presented as an oracle-style upper bound, with the empirical contribution centered on the realized cost savings and the escalation rate. As reported, the accuracy comparison is internally valid but not a discovery about MAS/SAS behavior. The range also appears inconsistent: Table 4 shows a 13.3pp gain on Debate-AIME (38.3 vs 25.0), which exceeds the stated 'up to 12%' and the abstract's '1.1-12%'.","section":"§4.2, Table 4, Appendix B.4"},{"comment":"The confidence-guided critical-path method hinges on the assumption that the self-reported confidence c_i is informative about an agent's contribution, but no calibration evidence is provided. The importance score I_i = Σ_r Q_r c_i + (s−c_i)(1−Q_r) is a linear combination of the binary final outcome and the self-report; if c_i is uninformative, the critical-agent ranking in Figure 4a is not reliable. The paper should report a calibration analysis (e.g., reliability of confidence as a predictor of per-agent correctness across rounds) or an alternative robustness check, such as comparing the selected critical agent against an exhaustive leave-one-out upgrade/downgrade evaluation, which the current single math-reasoning experiment only partially covers.","section":"§4.1, Appendix B.2, Figure 4a"},{"comment":"The quantitative comparisons are presented as point estimates without error bars, confidence intervals, or significance tests, despite small sample sizes in several datasets (AIME has 60 problems; BigCodeBench-Hard has 148). Claims such as a 1.1-12% accuracy improvement or a 4-220x token ratio should be accompanied by uncertainty estimates and a statement of the number of independent runs (e.g., temperature variations are reported but used as evidence of consistency rather than as repeated trials). The paper also reports different cost-savings figures in the abstract (up to 20%), the introduction (up to 88.1%), and the conclusion (88.1%); these need to be reconciled and attributed to routing versus cascade.","section":"All tables; abstract; §4.2"}],"minor_comments":[{"comment":"The cost-savings numbers are inconsistent: the abstract says 'reducing deployment costs by up to 20%,' the introduction says 'up to 88.1%,' and the conclusion repeats 88.1%; please state which mechanism produces which saving and make the figures consistent.","section":"Abstract / Introduction / Conclusion"},{"comment":"The main text says the evaluation covers 'Gemini-2.5-Pro, Gemini-2.5-Flash,' but Appendix B.1 lists Gemini-2.0-Flash-001, Gemini-2.5-Pro, GPT-3.5-Turbo, GPT-4o, and LLaMA-3.1 models; Gemini-2.5-Flash does not appear in the experiment setup.","section":"§3.2 vs Appendix B.1"},{"comment":"The column header contains a typo ('Deocde' should be 'Decode'); also, the reasons for the very large spread in prefill token ratios (from 1.2x on HoVer to 220x on AIME) are not discussed.","section":"Table 3"},{"comment":"The two panels use different threshold axis ranges (0-5 and 0-10), and neither reports the fraction of requests actually routed to MAS or the escalation rate; reporting p or the escalation ratio would make the cost-accuracy tradeoff interpretable.","section":"Figure 4b/c"},{"comment":"The cascade cost formula omits the verification cost; Appendix B.4 acknowledges this cannot be ignored when verification uses humans or LLM judges. Please state the exact-match or test-based verification assumption in the main text.","section":"§4.2 cost formula"},{"comment":"The early-stopping ablation with intermediate message truncation is an interesting experiment, but the text should clarify whether the 25% figure refers to the fraction of messages that are necessary for the final output, or to the fraction of length that suffices; the caption and text use both phrasings.","section":"Appendix D, Figure 10"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical systems study with a very broad evaluation, and the defect taxonomy is useful. However, the two headline contributions need re-scoping: the diminishing-benefits claim is contradicted by the paper's own within-family appendix data, and the cascade accuracy gain is a construction-level property. The routing and cost analyses are more defensible and should be foregrounded. The discrepancies between abstract, introduction, and appendix figures suggest the manuscript would benefit from a careful consistency pass before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one for the scope, not for the headline. The paper runs a large empirical comparison of single-agent and multi-agent LLM systems across many frameworks, datasets, and models, and that breadth is genuinely useful. The defect taxonomy—node, edge, path—is a reasonable way to frame MAS failures, and the case studies are illustrative. The hybrid routing and cascade design is a practical contribution; the cost savings from routing easy requests to SAS and escalating hard ones to MAS are real and well-measured.\n\nBut the central claim, that MAS benefits shrink as base LLMs improve, does not hold up in the paper's own data. Table 2 compares old ChatGPT-era numbers from prior papers to new Gemini-2.0-Flash runs, changing model family, prompts, and architecture simultaneously. That's a confounded comparison. The appendix contains within-family comparisons that are supposed to be the controlled evidence, and they're mixed. On SelfCol-DS1000, Llama-3-70B gives a +22.7pp MAS advantage versus +11.2pp for Llama-3-8B—the opposite of diminishing. AIME also shows the gap growing with GPT-4o and Gemini-2.0. The paper acknowledges AIME as an exception but not DS1000.\n\nThere are other problems. The cascade accuracy improvement is guaranteed by construction when verification is exact—any request correct under SAS or MAS is correct under the cascade—so Table 4's 1.1-12% gains are not empirical discoveries. The cost savings carry the weight, and they're fine. The confidence-guided tracing assumes self-reported confidence is informative, but there's no calibration evidence. No error bars or significance tests appear anywhere, and the abstract says 'up to 20%' cost reduction while the body says 88.1%. That inconsistency should have been caught.\n\nWho's this for? People deciding whether to deploy MAS or SAS for LLM applications, and researchers working on agent orchestration. They'll find the cost data and the hybrid designs useful. But the paper needs a major rewrite of its central claim, a statistical treatment of the within-family comparisons, and a reframing of the cascade result as a property rather than a discovery. I'd send it to peer review, but I'd ask for heavy revision. The scope and the questions are worth referee time; the current analysis doesn't yet support the abstract.","headline":"The headline claim about diminishing MAS advantage is contradicted by the paper's own within-family comparisons in the appendix; the confounded Table 2 can't carry the load.","tokens_in":67108,"tokens_out":5376,"would_cite":true,"duration_ms":39883,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multi-agent LLM systems lose most of their accuracy advantage over a single agent as models improve, and a hybrid that routes requests by difficulty outperforms both while cutting cost.","keywords":["multi-agent systems","single-agent systems","large language models","agent routing","agent cascade","cost-efficient deployment","confidence-guided tracing","MAS defect analysis"],"falsifier":"Run the same MAS/SAS benchmark pair on the next frontier model generation: if the MAS-win share stops declining as capability improves, the diminishing-benefit trend fails. Separately, compare importance-score rankings against full one-agent-at-a-time ablations on a held-out set, since the tracing method fails if self-reported confidence is uncorrelated with actual contribution.","tokens_in":65994,"feed_emoji":"🤖","tokens_out":8187,"duration_ms":69464,"temperature":0.7,"pith_summary":"Large language model systems built from multiple specialized agents have been promoted as more accurate than a single agent, but this paper argues that the advantage is largely a property of weaker base models. Across 15 tasks spanning code generation, math reasoning, software engineering, travel planning, finance, retrieval-augmented QA, and scientific experimentation, the authors find that multi-agent systems (MAS) and single-agent systems (SAS) tie on roughly 80% of examples, and MAS wins shrink as the underlying LLM gets stronger. MAS still helps on the hardest problems, but typical deployments pay 4 to 220 times more input tokens and 2 to 12 times more response tokens for small or negative accuracy gains. The authors attribute the gap to three defect classes, node, edge, and path, and use a confidence-guided tracing method to locate the bottleneck agent. They then propose routing requests to SAS or MAS by difficulty and a cascade that uses SAS first and escalates to MAS only when verification fails, reporting 1.1-12% better accuracy at up to 20% lower deployment cost.","feed_headline":"Study: Multi-agent AI edge fades as models improve","feed_subtitle":"A hybrid single-agent/multi-agent design keeps the gains while cutting deployment cost up to 20 percent.","key_machinery":"The paper models agentic execution as a directed graph $G = (V, E)$ with LLM agents and tools as nodes and inter-agent messages as edges, defining SAS as $|V_{llm}| = 1$ and MAS as $|V_{llm}| > 1$, with cost as the token-weighted sum over messages. The named diagnostic mechanism is confidence-guided critical path tracing: each agent reports a confidence $c_i$ on a 1-10 scale, and after each request the importance score $$I_i = \\sum_{r \\in R} Q_r c_i + (s - c_i)(1 - Q_r)$$ is accumulated, where $s = 10$ and $Q_r$ is 1 if the final output is correct and 0 otherwise. The lowest-scoring agent is flagged as the bottleneck. The optimization mechanism is the hybrid paradigm: an LLM rater assigns difficulty scores and routes requests between SAS and MAS, plus a cascade that runs SAS first, verifies the output, and escalates to MAS only on failure.","core_discovery":"On its own terms, the paper establishes that the accuracy premium of MAS over SAS is not intrinsic: it shrinks as LLM capability improves and can invert on simple tasks where MAS overthinking hurts. Using the same frameworks that originally reported roughly 9-16% gains for ChatGPT, the authors measure gains of about 0.8-3.0% with Gemini-2.0-Flash, while MAS consumes 4-220 times more prefill tokens. Datapoint-level analysis shows most examples are ties, and MAS wins are outweighed by SAS wins on several datasets. The paper also introduces a hybrid paradigm: an LLM rater routes requests by difficulty, and a cascade verifies SAS outputs and escalates failures to MAS; on evaluated tasks this improves accuracy by 1.1-12% while cutting deployment cost by up to 20%.","pith_inferences":["If the confidence-guided tracing result generalizes, MAS frameworks could autotune themselves by replacing one agent at a time; a natural test is whether importance scores correlate with ablation gains across more frameworks and models.","The diminishing advantage suggests a moving target: each MAS design is benchmarked against the SAS of its day, so MAS research may need to focus on tasks where decomposition genuinely adds capability, such as long-horizon tool use, rather than reasoning alone.","The paper's stated scope, general-purpose models and task-solving agentic applications, leaves open whether fine-tuned or domain-specialized agents would restore a larger MAS advantage.","The cascade's gain comes from SAS being cheap and often right; as verification becomes cheaper or SAS accuracy rises, the cascade's benefit should increase, which is testable."],"forward_implications":["Deploying MAS as the default for a new agentic task is hard to justify on accuracy alone for frontier models; the workloads where MAS wins are concentrated on very hard inputs.","Upgrading only the identified critical agent captures much of the accuracy gain of upgrading all agents, so selective augmentation is a cost-effective lever.","Routing easy requests to SAS and hard requests to MAS can beat either paradigm alone on accuracy at lower cost, without fine-tuning a difficulty model.","Cascade designs are only applicable when the final output can be verified cheaply; for open-ended or subjective outputs the extra verification cost must be added.","The three defect classes give a practical checklist: check whether a single agent caps performance, whether downstream agents are overloaded by upstream messages, and whether summaries lose information that then propagates."],"supporting_citations":[{"why":"MetaGPT is one of the MAS frameworks whose originally reported gain shrinks when rerun on a stronger model; it supplies code-generation and software-engineering baselines.","marker":"[20]"},{"why":"Self-collaboration (SelfCol) code generation framework supplies the code-generation datasets and the edge-defect case studies.","marker":"[11]"},{"why":"MATH-DEBATE framework supplies the math reasoning tasks, majority-vote aggregation, and the critical-path augmentation experiment.","marker":"[14]"},{"why":"ChatDev provides the software engineering MAS baseline and the overthinking edge-defect examples.","marker":"[31]"},{"why":"Supplies the MAST failure taxonomy and the human-annotated software engineering evaluation used to categorize MAS defects.","marker":"[3]"},{"why":"Cognify defines the cost-aware workflow optimization setting and supplies the FinRobot and DocRetrieval workflows used for cost comparison.","marker":"[17]"},{"why":"FinRobot provides the financial analysis framework, datasets, and workflows used in the accuracy and cost comparisons.","marker":"[42]"},{"why":"TDAG is the multi-agent travel planning framework used for the travel planning evaluation.","marker":"[36]"}],"fun_headline_variants":["Hybrid AI agent design lifts accuracy, cuts cost","Multi-agent advantage shrinks with newer LLMs","Cascade agents: up to 12% accuracy, 20% less cost","Single-agent or multi-agent? Study says both","MAS gains over SAS fade as AI improves"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an LLM agent's self-reported confidence score is informative about how much that agent caused the final output to be correct; the paper gives no calibration evidence for this, and without it the importance-score ranking that identifies critical agents is not trustworthy.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid AI agent design lifts accuracy, cuts cost","Multi-agent advantage shrinks with newer LLMs","Cascade agents: up to 12% accuracy, 20% less cost","Single-agent or multi-agent? Study says both","MAS gains over SAS fade as AI improves"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000283,"raw_usage":{"total_tokens":1656,"prompt_tokens":916,"completion_tokens":740,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":662}},"tokens_in":532,"tokens_out":740,"duration_ms":7592,"temperature":1.0,"reasoning_tokens":662,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:33:01.120381+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same MAS/SAS benchmark pair on the next frontier model generation: if the MAS-win share stops declining as capability improves, the diminishing-benefit trend fails. Separately, compare importance-score rankings against full one-agent-at-a-time ablations on a held-out set, since the tracing method fails if self-reported confidence is uncorrelated with actual contribution.","supporting_citations":[],"review_version":1}