OrchestraBench: Evaluating Multi-Agent Orchestration Failure Modes, Recovery, and Decomposition Quality

Natan Vidra; Sharon Zheng; Spurthi Setty; Yidian Chen; Yingzi Gu

REVIEW 4 major objections 4 minor 13 references

OrchestraBench: Evaluating Multi-Agent Orchestration Failure Modes, Recovery, and Decomposition Quality

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read In controlled probes on real LLM agents, orchestration failure recovery splits into three tiers that persist across contexts and models.

desk verdict A careful, honest reliability probe worth referee time, but the headline 'never recovered' tier is partly a design artifact—and the paper's own ablation shows it. read the letter →

arxiv 2608.05263 v1 pith:HFETZMBQ submitted 2026-08-05 cs.AI

Yidian Chen , Yingzi Gu , Natan Vidra , Spurthi Setty , Sharon Zheng This is my paper · ORCID

classification cs.AI

keywords multi-agentsystemsLLMorchestrationfailurerecoverycascadepropagationinjectionroutingpolicybenchmarking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that multi-agent orchestration systems should be judged not by task accuracy alone but by how they fail, recover, and decompose work. It builds a controlled, seed-reproducible failure-injection harness and measures, per failure mode, how often a real LLM agent recovers and how far errors cascade downstream. The central finding is a three-tier structure: tool faults are fully recovered, ambiguous delegation is partially recovered, and three latent or semantic failure modes never recover, so blind retry cannot repair them. This ordering persists when the identical computation is reframed as a business workflow and across three model sizes, which the authors read as evidence that failure handling is model behavior under context, not a benchmark artifact.

What carries the argument

The central machinery is a seed-reproducible failure-injection harness that runs a real LLM agent over a verifiable arithmetic dependency chain with five injected failure modes, measuring per-failure-mode recovery rate and cascade radius (the number of downstream stages corrupted by a seeded error). The three-tier recovery split—tool faults recover fully, ambiguous delegation partially, latent or semantic modes never—is the identity that carries the argument, and its persistence under domain reframing and model changes is what turns it from a construct artifact into a claim about model behavior.

What would settle it

Run the identical five-mode injection on a genuine multi-agent pipeline with independent LLM agents per stage and check whether latent failure modes ever recover at non-zero rates or whether retry ever repairs a latent fault; either outcome would break the three-tier structure.

Watch

Extended reading notes

Core claim

Across five failure modes injected at the prompt level into a verifiable arithmetic dependency chain executed by a real Claude agent, failure handling splits into three tiers: tool-invocation errors are fully recovered (recovery 1.0), ambiguous delegation is partially recovered (0.30), and context pollution, conflicting outputs, and premature action never recover (0.0). The tier ordering survives reframing the identical computation as a loan-approval workflow and holds across Sonnet, Opus, and Haiku; only the ambiguous-delegation rate shifts with context and model. Retry reproduces latent faults and lengthens time to detection rather than repairing them, so detection and attribution are necessary containment mechanisms. Cascade radius grows with pipeline depth (mean 0.9 to 4.7 across depths 3–7) as a corroborating structural signature. The paper frames these as controlled-chain mechanism probes, not domain-workload claims.

Load-bearing premise

The claim that the three-tier recovery structure is real depends on prompt-level fault injection into a single LLM agent faithfully reproducing the failures of a genuine multi-agent system; if inter-agent interactions produce different failure dynamics, the tier rates could be a simulation artifact.

Editorial extensions

If this is right

Production orchestrators should not rely on blind retry for latent semantic failures; detection and attribution must be built into the pipeline.
Routing reliability is determined by whether the router reads surface flags or reasons over task intent; a description-reading router closes a 0% to 100% adversarial gap that flag-matching cannot.
Cascade radius provides a measurable, depth-resolved signature of error propagation that grows roughly linearly with pipeline depth for latent modes.
Decomposing tasks into delegable sub-steps exposes recoverable structure even when the final answer is correct either way.
The containment gain of an LLM routing policy is mostly the trusted-upstream-state signal rather than autonomous detection, so state verification should be a design target.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the tier structure generalizes beyond the tested Claude models and the arithmetic chain, the same three-tier split should appear in other agent frameworks whose failures are semantic rather than mechanical; this is testable with the paper's cross-model harness.
The single-agent prompt-injection design means real inter-agent interaction failures, such as one agent's output poisoning another agent's context, could behave differently; running the same five modes across genuinely independent agents would either confirm or bound the finding.
Because the trusted-state ablation shows self-correction without upstream truth collapses to baseline, the practical design implication may be to invest in state verification and context isolation rather than in stronger router prompts.
The near-linear cascade growth with depth implies the marginal value of detecting a latent fault early grows with pipeline length, which suggests a cost model for when to checkpoint could follow.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take: this is a careful, honest mechanism probe that deserves referee time, but its headline claim—three latent failure modes 'never recovered'—is overstated, and the paper's own ablation shows why.

What's actually new and good: Exp 1 cleanly isolates routing mechanism from task difficulty: a TF-IDF router reading the task description clears the adversarial set that a keyword/flag router gets 0% on. That's a reproducible result with a real takeaway. The cascade-radius metric, while partly structural on this chain (they admit it), is a useful stages-traversed quantification that MAS-FIRE lacks. And the trusted-state ablation in §5.5 is genuinely good science: they test the alternative explanation for the LLM router's containment gain and report that the gain is mostly the hint, not autonomous detection. That kind of self-audit is rare.

Soft spots, in proportion. The stress-test concern is basically right. In Exp 2/3, a latent failure is injected as a corrupted intermediate that every downstream stage consumes, and final-task success is exact-match on the arithmetic result. Unless the run is given an external trusted repair signal, final-task success is 0 by construction, regardless of what the model knows or attempts. The §5.5 ablation makes this visible: handing the router the trusted upstream value lifts latent recovery to 0.83; dropping the hint collapses it to 0.08. That reads as 'no recovery opportunity without an external trusted value,' not 'model unable to recover when given a genuine opportunity.' The paper explicitly calls cascade radius 'partly structural' but does not apply the same caveat to the 0.0 recovery cells. The abstract and conclusion say 'never recovered' without that qualifier. Table 5 also shows Opus recovering conflicting_outputs at 0.10, which contradicts the flat 'never' wording. The ordering (tool > ambiguous > latent) is more defensible, and the retry finding (retry reproduces the fault, lengthens time-to-detection) holds up as far as it goes.

Minor: the 26-case diagnostic is small and author-labeled; that's disclosed and framed as a mechanism probe, so it's a minor concern. The single-agent proxy is a real limitation, also disclosed.

Who this is for: people building or evaluating multi-agent orchestration frameworks, particularly those who want a reliability-focused measurement layer. It is not a domain-workload result. The artifact is reproducible, with committed CSVs and offline scripts.

Recommendation: send to peer review. A good referee will ask the authors to reword the 'never recovered' claim, apply the structural caveat to the 0.0 cells, and possibly add a condition where the model is given a chance to detect a corrupted intermediate without a trusted hint. The paper is close; it just overstates one cell of its own table.

Referee Report

4 major / 4 minor

Summary. OrchestraBench proposes a controlled failure-injection benchmark for multi-agent orchestration, with three main pieces: (i) a 26-case routing diagnostic comparing Fixed, Heuristic, TF-IDF, LLM, and Oracle routing policies; (ii) a failure-injection harness over a verifiable arithmetic dependency chain that measures per-mode final-task success, recovery, and cascade radius; and (iii) a decomposition-fidelity experiment plus a policy-conditioned trusted-state ablation. The headline finding is a three-tier failure-handling structure: tool-invocation faults are fully recovered (1.0), ambiguous delegation is partially recovered (0.30), and three latent/semantic modes are reported as never recovered (0.0), with the ordering persisting under a loan-approval domain reframing and across Sonnet, Opus, and Haiku. The paper also reports that blind retry does not repair latent faults, that cascade radius grows with pipeline depth, and that the apparent containment gain of an LLM router is mostly the trusted-state signal rather than autonomous detection. The authors explicitly frame the experiments as controlled mechanism probes and disclose several limitations, including the small Exp 1 diagnostic and the use of a single LLM rather than a literal multi-agent system.

Significance. The benchmark artifact and the experimental discipline are genuine strengths: exact-match ground truth on a verifiable chain, seed-reproducible failure injection, bootstrap confidence intervals, exact permutation tests, and a dedicated llm_noupstream ablation that separates the trusted-state signal from autonomous detection. If the three-tier structure were fully established, this would be a valuable contribution to multi-agent reliability evaluation. As it stands, the coarser ordering (tool faults > ambiguous delegation > latent modes; blind retry insufficient) and the reproducible harness are defensible, but the strong behavioral reading of the latent 0.0 cells is not yet established. The paper is a solid mechanism probe whose central claim needs substantial reworking in presentation and interpretation, rather than a finished benchmark result.

major comments (4)

[§5.2, §5.5, Tables 4/8] The 0.0 recovery cells for context_pollution, conflicting_outputs, and premature_action are partly fixed by the injection design, and this undermines the central 'never recovered' claim. In the verifiable arithmetic chain, the corrupted intermediate is consumed by every downstream stage and success is exact-match on the final result, so no run can succeed unless an external trusted value is supplied; the paper itself calls the deterministic 0.0 cells 'point masses by construction' (§4) and concedes that cascade radius is 'partly structural' (§7), but it does not apply the same caveat to the recovery zeros. The §5.5 ablation makes the issue concrete: handing the router the trusted upstream value lifts latent recovery from 0.08 to 0.83, while llm_noupstream collapses it back to 0.08. That is the signature of a missing external repair signal, not of a model that had a genuine opportunity to recover and failed. The paper should either rephrase the 0.0 latent cells as 'unrecoverable without trusted state in this construct' or add a condition (e.g., redundant recomputation or a consistency signal) that gives the model a non-external path to detection before interpreting the zeros as model behavior.
[Table 5 / §5.2] The claim that the latent/semantic modes 'never recover' is internally inconsistent with the cross-model results. Table 5 reports Opus 4.8 recovering conflicting_outputs in 0.10 of runs, but the abstract says 'three latent or semantic modes never recovered (0.0)' and §5.2/§6 repeat the never-recovered wording. The text in §5.2 restricts invariance to 'the two catastrophic latent modes' only after presenting the three-tier claim, so the three-tier structure is not cleanly model-invariant. The latent tier should be reported per mode, and the 'never' wording should be limited to the cells that are point masses in every model: context_pollution and premature_action in this sweep, with conflicting_outputs at 0.0 only for Sonnet and Haiku.
[§5.2, §7] The domain-reframing and cross-model robustness arguments do not provide evidence for the latent zeros. Because the 0.0 cells are construction-driven point masses, their persistence across the loan-approval framing and across Sonnet/Opus/Haiku is expected and carries no information about model behavior; only the stochastic ambiguous_delegation cell moves (0.30 to 0.40 under reframing; 0.10 to 0.33 across models), and that cell does not test the latent modes. The robustness evidence therefore supports the ordering among the non-latent modes and the context-sensitivity of ambiguous delegation, but not the claim that the latent tier is a behavioral universal.
[§7 / §5.2] The failure-recovery experiments are run as prompt-level fault injection into a single Claude agent, not as a literal multi-agent system. The paper discloses this in §7, but the disclosure does not resolve the construct-validity risk for the title-level claim about multi-agent orchestration: failures that arise from interactions between independent agents may have different recovery signatures than injected faults in one agent. A concrete next step is a two-agent version of the same chain (e.g., a router and a worker, with the failure arising from the worker's actual output) to test whether the three-tier ordering survives. Until then, the multi-agent wording in the title and abstract should be qualified more prominently, not only in Limitations.

minor comments (4)

[§4/§5.2] The retry policy implementation is not specified: the number of attempts, which stages are retried, and whether the failure is re-injected on retry are all left unclear, making the 'retry reproduces the fault' claim under-specified.
[§5.3] The cascade-radius definition needs a worked depth-3 example; the text says latent modes corrupt 'every downstream stage (cascade = depth − 2 exactly)' and reports mean 0.93 at depth 3, which is ambiguous about which stages are counted.
[§7/§5.1] The Exp 1 diagnostic is only 26 cases with author-provided labels, and the 0%-to-100% adversarial contrast rests on just 10 cases; including the gold set and annotator instructions in the repository, and reporting an inter-annotator check if one becomes available, would make the result easier to scrutinize.
[References] Reference [1] is cited only as 'Anonymous (concurrent work)', which is not verifiable; it should be replaced by an identifiable citation or removed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's results are measured, reproducible probes with explicit structural disclosures, not fitted inputs or self-citation chains.

full rationale

The paper does not derive predictions from fitted parameters or from inputs that contain the target result. Experiment 1 is a controlled diagnostic where the adversarial set is deliberately constructed to decouple surface flags from intent; the comparison across Fixed, Heuristic, TF-IDF, and LLM routers isolates mechanism, and the paper explicitly reads it as a worst-case probe rather than a general difficulty benchmark. Experiments 2-3 report empirical recovery rates and cascade radii from a real Claude agent under seed-controlled injection, with exact-match ground truth. The 0.0 recovery cells for latent modes are reported as deterministic point masses, and the paper itself discloses that a corrupted latent state always fails the final task and that cascade radius is partly structural; these disclosures prevent the structural component from masquerading as a behavioral discovery. The central ordering claim is supported by independent robustness checks: domain reframing moves absolute rates while preserving ordering, and a three-model sweep reproduces the tool-vs-latent structure. The policy-conditioned probe is explicitly labelled a trusted-state probe, and the llm_noupstream ablation directly tests the alternative explanation that the LLM router's gain came from the trusted-state signal rather than autonomous detection. No load-bearing self-citations appear; the only anonymous cross-reference (IntentBench) is explicitly described as complementary with no measurement overlap. The paper is self-contained against external benchmarks, commits code and data, and reports exact permutation tests. Construct-validity concerns about prompt-level injection in a single agent are acknowledged in Section 7 as limitations, but acknowledging a limitation is not circularity. No step in the derivation chain reduces to its own inputs by definition or by self-citation.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

The central claims rest on empirical measurements under disclosed modeling assumptions rather than on fitted constants or ad hoc entities. No numeric free parameters are fitted; design choices such as per-mode n=30 and the adversarial fraction are experimental setup. The main invented construct, cascade radius, is an operationally defined metric with an independent measurement handle, not an unexplained postulate.

assumptions (5)

domain assumption MAST taxonomy's five failure modes (ambiguous delegation, tool-invocation error, context pollution, conflicting outputs, premature action) are representative of the orchestration failure space.
The injected modes come from prior MAST taxonomy [2]; the paper does not independently validate coverage.
ad hoc to paper A downstream stage consuming the upstream value in the verifiable chain makes cascade radius partly structural (a corrupted latent state propagates to roughly depth-2 stages).
Paper states this in §7 Construct validity: cascade radius is 'partly by design'; the depth-scaling is therefore corroborating, not the paper's headline.
domain assumption Exact-match correctness on the arithmetic result is a faithful objective ground truth for recovery.
Used for Exp 2-4; avoids subjective annotation but only measures final-task success, not intermediate state quality.
ad hoc to paper Prompt-level fault injection in a single Claude agent simulates orchestration failure modes of multi-agent systems.
Acknowledged in §7 'Single LLM, not a literal multi-agent system'; the central three-tier claim depends on this proxy.
domain assumption The 26-case routing diagnostic's gold labels (including 10 adversarial cases with unambiguous intent but misleading or missing flags) are correct.
Author-labelled; a two-annotator protocol with kappa is future work per §7; Exp 1 results are bounded by label correctness.

invented entities (1)

cascade radius independent evidence
purpose: Primary metric counting the number of downstream pipeline stages corrupted by a seeded failure; a differentiator versus prior reliability work that lacks a stages-traversed metric.
Operationally defined and reproducible from committed seeds and data; where partly structural the paper says so, and the stochastic ambiguous-delegation cell shows the metric also captures model behavior.

how reviews work

0 comments

Cite this review

Pith. "Pith review of OrchestraBench: Evaluating Multi-Agent Orchestration Failure Modes, Recovery, and Decomposition Quality." pith.science (2026). https://pith.science/paper/HFETZMBQ

@misc{pith2026260805263,
  author       = {Pith},
  title        = {Pith review of: OrchestraBench: Evaluating Multi-Agent Orchestration Failure Modes, Recovery, and Decomposition Quality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HFETZMBQ}},
  note         = {Machine review of arXiv:2608.05263}
}

read the original abstract

Multi-agent orchestration frameworks are moving from demos to production, yet benchmarks typically report task accuracy without diagnosing why a pipeline failed, where a cascade began, or which routing decision caused the breakdown. OrchestraBench evaluates failure, recovery, and decomposition through a controlled, seed-reproducible failure-injection harness over templated enterprise workflows. It introduces cascade radius and per-failure-mode recovery as primary metrics and compares routing policies with bootstrap confidence intervals and paired tests. On a 26-case gold-labelled diagnostic, a keyword/flag router scored 0% on adversarial cases with misleading or missing surface flags, whereas an intent-reasoning model router scored 100%, matching the oracle. Controlled mechanism probes with a real Claude agent over a verifiable arithmetic dependency chain revealed three failure-handling tiers across five MAST modes: tool faults recovered fully (1.0), ambiguous delegation recovered partially (0.30), and three latent or semantic modes never recovered (0.0). This ordering persisted when the computation was reframed as a loan-approval workflow and across Sonnet, Opus, and Haiku, although absolute rates shifted with context. Blind retry reproduced latent faults and increased time to detection, indicating that detection and attribution are necessary for containment. Cascade radius increased with pipeline depth (mean 0.9 to 4.7 across depths 3-7). A trusted-state repair ablation showed that apparent containment gains primarily came from the trusted-state signal rather than autonomous detection. These results are controlled-chain mechanism probes, not domain-workload claims.

Figures

Figures reproduced from arXiv: 2608.05263 by the authors.

**Figure 3.** Experiment 3 — mean cascade radius vs. pipeline [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

**Figure 4.** Experiment 3 — mean latent cascade radius vs. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 2 canonical work pages

[1]

Anonymous (concurrent work). 2026. IntentBench: Intent Specification as a First- Class Evaluation Object. Concurrent spec-layer benchmark; complementary, cross-cite

work page 2026
[2]

Pan, Shuyi Yang, Lakshya A

Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ram- chandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. 2025. Why Do Multi-Agent LLM Systems Fail? arXiv:2503.13657 [cs.AI] Introduces the MAST taxonomy: 14 failure modes in 3 categories; MAST-Data 1,600...

arXiv 2025
[3]

Mengzhuo Chen, Junjie Wang, Fangwen Mu, Yawen Wang, Zhe Liu, Huanxiang Feng, and Qing Wang. 2026. Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems. arXiv:2604.22708 220 annotated failure traces; full-execution-observability attribution

arXiv 2026
[4]

Jin Jia, Zhiling Deng, Zhuangbin Chen, Yingqi Wang, and Zibin Zheng. 2026. MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems. arXiv:2602.19843 15 fault types (8 intra-/7 inter-agent); injection via prompt / response / message-routing; tested on MetaGPT, Table-Critic, CAMEL

arXiv 2026
[5]

Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. InInternational Conference on Learning Representa- tions (ICLR). arXiv:2310.06770; 2,294 real GitHub issues over 12 Python repos

arXiv 2024
[6]

Siddhant Kulkarni and Yukta Kulkarni. 2026. Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Or- chestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategies. arXiv:2603.22651 10k SEC filings; reflexive F1 0.943 @2.3x, hierarchical 0.921 @1.4x

arXiv 2026
[7]

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2024. AgentBench: Evaluating LLMs as Agents. InInternational Conference on Learning Represe...

arXiv 2024
[8]

Aaron Xuxiang Tian, Ruofan Zhang, Jiayao Tang, Young Min Cho, Xueqian Li, Qiang Yi, Ji Wang, Zhunping Zhang, Danrui Qi, Zekun Li, Xingyu Xiang, Sharath Chandra Guntuku, Lyle Ungar, Tianyu Shi, and Chi Wang. 2025. Beyond the Strongest LLM: Multi-Turn Multi-Agent Orchestration vs. Single LLMs on Benchmarks. arXiv:2509.23537 GPQA-Diamond: at least one agent ...

arXiv 2025

Show all 13 references

[9]

Weixuan Wang, Dongge Han, Daniel Madrigal Diaz, Jin Xu, Victor Rühle, and Saravan Rajmohan. 2025. OdysseyBench: Evaluating LLM Agents on Long- Horizon Complex Office Application Workflows. arXiv:2508.09124 602 long- horizon office tasks (300 real + 302 synthesized)

2025 arXiv
[10]

Yizhe Xie, Congcong Zhu, Xinyue Zhang, Tianqing Zhu, Dayong Ye, Minfeng Qi, Huajie Chen, and Wanlei Zhou. 2026. From Spark to Fire: Modeling and Mitigat- ing Error Cascades in LLM-Based Multi-Agent Collaboration. arXiv:2603.04474 Error cascades in agent pipelines. Yidian Chen,...

2026 arXiv
[11]

Geunbin Yu. 2026. AdaptOrch: Task-Adaptive Multi-Agent Orchestration in the Era of LLM Performance Convergence. arXiv:2602.16873 Task-adaptive orchestration method (+12–23%)

2026
[12]

Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and Qingyun Wu
[2025]

InProceedings of the 42nd International Conference on Machine Learning (ICML), Spotlight

Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. InProceedings of the 42nd International Conference on Machine Learning (ICML), Spotlight. Who&When dataset: failure logs from 127 multi-agent systems. arXiv:2505.00212

arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Anonymous (concurrent work). 2026. IntentBench: Intent Specification as a First- Class Evaluation Object. Concurrent spec-layer benchmark; complementary, cross-cite

work page 2026

[2] [2]

Pan, Shuyi Yang, Lakshya A

Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ram- chandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. 2025. Why Do Multi-Agent LLM Systems Fail? arXiv:2503.13657 [cs.AI] Introduces the MAST taxonomy: 14 failure modes in 3 categories; MAST-Data 1,600...

arXiv 2025

[3] [3]

Mengzhuo Chen, Junjie Wang, Fangwen Mu, Yawen Wang, Zhe Liu, Huanxiang Feng, and Qing Wang. 2026. Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems. arXiv:2604.22708 220 annotated failure traces; full-execution-observability attribution

arXiv 2026

[4] [4]

Jin Jia, Zhiling Deng, Zhuangbin Chen, Yingqi Wang, and Zibin Zheng. 2026. MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems. arXiv:2602.19843 15 fault types (8 intra-/7 inter-agent); injection via prompt / response / message-routing; tested on MetaGPT, Table-Critic, CAMEL

arXiv 2026

[5] [5]

Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. InInternational Conference on Learning Representa- tions (ICLR). arXiv:2310.06770; 2,294 real GitHub issues over 12 Python repos

arXiv 2024

[6] [6]

Siddhant Kulkarni and Yukta Kulkarni. 2026. Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Or- chestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategies. arXiv:2603.22651 10k SEC filings; reflexive F1 0.943 @2.3x, hierarchical 0.921 @1.4x

arXiv 2026

[7] [7]

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2024. AgentBench: Evaluating LLMs as Agents. InInternational Conference on Learning Represe...

arXiv 2024

[8] [8]

Aaron Xuxiang Tian, Ruofan Zhang, Jiayao Tang, Young Min Cho, Xueqian Li, Qiang Yi, Ji Wang, Zhunping Zhang, Danrui Qi, Zekun Li, Xingyu Xiang, Sharath Chandra Guntuku, Lyle Ungar, Tianyu Shi, and Chi Wang. 2025. Beyond the Strongest LLM: Multi-Turn Multi-Agent Orchestration vs. Single LLMs on Benchmarks. arXiv:2509.23537 GPQA-Diamond: at least one agent ...

arXiv 2025