Pith. sign in

REVIEW 4 major objections 4 minor 61 references

This paper claims that a multi-agent LLM system's team composition, coordination mechanism, and fusion algorithm are independent, swappable layers—and that a learning router can select the fusion per task.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:24 UTC pith:V3A25XDR

load-bearing objection A genuinely useful decoupling framework and a clean, replicable RACI effect — but the formal orthogonality claim is overstated and the router's headline result is a synthetic-oracle recovery, not real-world evidence. the 4 major comments →

arxiv 2607.25446 v1 pith:V3A25XDR submitted 2026-07-28 cs.AI cs.LG

Toward an Organizational Science of Multi-Agent LLM Systems: Decoupling Who, How, and Which Algorithm

Huan Chen , Xiang Song , Jian Jin , Pan Ren , Liang-Jie Zhang This is my paper
classification cs.AI cs.LG
keywords multi-agent LLM systemsorganization–protocol decouplingRACI accountabilitycontextual banditsadaptive routingcollaboration protocolsteam rolescoordination mechanisms
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that the three things multi-agent LLM frameworks usually hard-wire together—who is on the team, how members coordinate, and which algorithm fuses their outputs—are logically independent and can be treated as separate, swappable layers. It formalizes an organization as roles, model bindings, a coordination mechanism, and a RACI accountability assignment, and a protocol as a function over a team and a shared blackboard, then claims any organization composes with any protocol. On that basis it builds Adaptive Org Routing, a contextual bandit that picks a fusion algorithm per task under a quality-cost tradeoff, and reports that it beats all six fixed protocols in a controlled study while training online on real benchmark and judge rewards. The paper's ablations aim to expose a mechanism: accountability placement changes results exactly when the protocol routes the final deliverable through the accountable agent, and the winning placement flips across model families, so organizational design must be revalidated or learned for each model binding.

Core claim

The central claim is an orthogonality property for multi-agent LLM systems: the system's behavior is a well-defined map from the full product of organizations and protocols to results, so any organization (defined by roles, model bindings, a coordination mechanism, and RACI tags) can compose with any fusion protocol (voting, debate, mixture-of-agents, rank-and-fuse, reflexion, or plan-and-execute) without rewriting either side. This is what makes the controlled experiments possible: protocols can be held fixed while team composition, accountability tags, or coordination are varied, and protocol choice becomes a free variable that a meta-protocol can learn. The paper also asserts a mechanisti

What carries the argument

The load-bearing object is the claimed orthogonality map run: O × Π × Task → Result, which asserts that any organization O composes with any collaboration protocol π through the runtime's Team and Blackboard interfaces. That single property carries the paper: it licenses controlled ablation of the organization with the algorithm held fixed, and it makes protocol selection a well-posed learning problem. The second mechanism is the routing decision itself—a linear contextual bandit with a 7-dimensional, model-free task featurization and a cost-adjusted reward r = quality − λ·cost, updated online with rank-1 matrix updates that avoid explicit inversion.

Load-bearing premise

The orthogonality claim rests on the premise that collaboration protocols see only an anonymous pool of role-typed agents and never read roles, accountability tags, coordination, or model identities—yet the paper's own ablation indicates rank-and-fuse routes the final write to the Accountable-tagged agent, so the protocol appears to read the tag.

What would settle it

Instrument the runtime to log every read a protocol makes on the Team interface during a run. If the final-write agent changes when the Accountable tag moves (as the rank-and-fuse rows show), the protocol is reading the tag, which directly contradicts the proof premise of the orthogonality proposition; the claim would then survive only in a weaker interface-level sense, not as stated.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Teams can be reconfigured without rewriting the fusion algorithm, and fusion algorithms can be swapped without rebuilding the team; the six protocols and three organization presets are claimed to compose freely.
  • Accountability placement is only as real as the protocol's control flow: it changes outcomes when—and only when—the algorithm routes the final deliverable through the accountable agent (rank-and-fuse moves, mixture-of-agents does not).
  • Protocol choice becomes a learnable, task-dependent variable; Adaptive Org Routing beats every fixed protocol in the controlled study (82% routing accuracy, recovering 45% of the gap to the oracle) and collapses without task context.
  • The same online bandit loop trains on objective benchmark scores and LLM-judge win-rates, so a deployed system can keep learning from real rewards.
  • Organizational conclusions do not transfer across model families; the winning RACI placement flips with the model binding, so org design must be revalidated or learned per binding.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the decoupling holds, the natural extension is to treat (organization, protocol, model binding) as a joint configuration space and let the router choose over all three; this paper evaluates only protocol selection with a fixed organization, but its own orthogonality claim makes the wider space well-defined.
  • The RACI result suggests accountability is a property of the algorithm's control flow, not of the tag alone: a protocol that never reads the Accountable tag is functionally blind to who is accountable. Frameworks may need an explicit 'who writes the final answer' hook to make accountability operational.
  • Because the winning RACI placement flipped across model families in small-N runs, published 'best organization' recipes from one model family may mislead on another; an online revalidation layer may be necessary in production, not just a convenience.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents IMACS, a multi-agent LLM framework that separates organization (who), coordination (how), and collaboration protocol (which algorithm) into independently swappable layers. It formalizes an organization as O = (R, µ, κ, ρ) and a protocol as a function over (Task, Team, Blackboard), and states Proposition 1 claiming full orthogonality and a total composition map run: O × Π × Task → Result. On this basis the paper conducts controlled RACI ablations, arguing that accountability placement changes outcomes exactly when the protocol routes the deliverable through the accountable agent, and introduces Adaptive Org Routing, a LinUCB contextual bandit that selects among six protocols per task under a quality–cost tradeoff. The router is evaluated against a simulated oracle (82% final routing accuracy, beating all fixed baselines) and in small real-benchmark and judge-reward feasibility runs. The paper is transparent about small-N and oracle limitations.

Significance. If the orthogonality claim holds, the framework is a genuinely useful contribution: it provides a common interface for six published collaboration algorithms, makes organizational theory executable and declarative, enables controlled comparisons that prior frameworks do not support, and turns protocol choice into a learnable variable. The paper also ships reproducible code, seeded benchmarks, and a detailed disclosure of limitations, which are strengths. However, the central formal claim, Proposition 1, is internally inconsistent with the paper's own RACI ablations, and this inconsistency is load-bearing for the controlled-ablation interpretation and the router's well-posedness. The simulated-oracle results, while honestly labeled, only demonstrate that LinUCB can recover a hand-constructed profile; they do not by themselves substantiate real-world routing gains. The real-reward experiments are small-N feasibility runs. These issues are fixable in revision, so the work merits major revision rather than rejection, but the formal and evidential basis needs to be reworked.

major comments (4)
  1. [Formalization, Definition 2 and Proposition 1] The proof of Proposition 1 claims that π depends on O only through Team/Blackboard interfaces that 'do not expose specific roles, model identities, κ, or ρ.' This is contradicted by the RACI ablation section, which states that LLM-Blender 'hands the final fusion to the team's accountable agent' and that MoA 'consults the Accountable tag only as a fallback.' These are explicit reads of ρ through the Team interface. Definition 2 also says a protocol 'sees a pool of role-typed agents,' which exposes at least role labels. Consequently the information-hiding premise of the proof is false, and the claimed orthogonality—and the interpretation of the RACI ablation as varying O 'without touching the algorithm'—does not follow from the stated proof. The map run may well be total, but the formal argument as written is too strong. Please revise Proposition 1 or its proof to state the actual interfac
  2. [Adaptive Org Routing, 'Simulated quality oracle' paragraph] The oracle's latent means are set so that factual tasks peak at voting, reasoning at debate, code at reflexion, and so on, and then each type has 'one clearly separated optimal arm.' The headline 'outperforms every fixed protocol' and the 82% routing accuracy are therefore measurements of how well LinUCB recovers this hand-constructed profile, not of real-world protocol-selection quality. The paper discloses this, but the abstract's wording 'outperforms every fixed protocol in a controlled study' is stronger than is supported. I recommend reframing the oracle experiment as a recovery/regret test of LinUCB under a known ground truth, and moving the evidence for real routing gains to the small real-reward runs, which currently concentrate on one cheap arm and cannot demonstrate task-adaptive routing.
  3. [Organization dimension, RACI ablation (Table 3 and Table 4)] The mechanistic claim that 'accountability placement changes outcomes exactly when the protocol routes the deliverable through the accountable agent' rests on a small empirical base: N=10 per cell, three repeats, one dataset (HotpotQA) for the primary fuser grid, and Table 4 shows overlapping means on GSM8K and all-1.0 cells on HumanEval. The Blender gap (+0.08±0.02) is sign-consistent but small, and the MoA control shuffles across repeats. This is suggestive but not yet a robust mechanism demonstration. Please expand the number of datasets/repeats or explicitly weaken the claim to 'consistent with' rather than 'exactly when,' and provide a preregistered or more extensive test.
  4. [Formalization, Proposition 1 totality] Proposition 1 asserts that run is total on the full product O × Π × Task. The proof assumes any protocol can consume any organization's role pool, but real protocols in the paper impose structural requirements (e.g., MoA's proposer/aggregator layers, Reflexion's need for a critic/executor split). If an organization lacks the roles a protocol expects, the protocol may fail or behave degenerately. The manuscript does not specify role-requirement preconditions or show that all six protocols handle arbitrary role sets. Please either state the compatibility conditions under which run is total, or restrict the product to organizations that satisfy each protocol's declared role requirements.
minor comments (4)
  1. [Abstract and Introduction] The abstract contains spacing artifacts ('whois', 'howmembers', 'whichalgorithm') that should be corrected. Also, the phrase 'validated configuration' is used early; the validation evidence is small-N and should be described as 'empirically assessed' or similar.
  2. [Adaptive Org Routing, 'Regret and org-awareness'] The statement that 'our setting satisfies the assumptions by construction' for the LinUCB regret bound is asserted without proof in the main text; the supplement is referenced but the full argument should at least be summarized here, since the boundedness and linearity of the simulated-reward model is a key assumption.
  3. [Experiments, 'From synthetic oracle to real rewards'] The N=18 and N=20 real-reward runs are described as 'the router trains end-to-end with no errors' and 'the policy again concentrates on one cheap arm.' This is a useful feasibility check, but the language 'trains with no errors' is potentially misleading: it means the pipeline executed without exceptions, not that routing performance was validated. Please make this distinction explicit.
  4. [Discussion, 'Benchmark and deployment scope'] The statement that AgentsNet is 'posed as a whole-graph team deliverable rather than through its native partial-visibility harness' is honest but indicates the coordination claim is narrowed; the paper should spell out what this narrowing excludes in terms of coordination mechanisms.

Circularity Check

0 steps flagged

No significant circularity: the central formal claim is self-contained, the simulated oracle is disclosed as a controlled stand-in, and real benchmarks anchor the empirical claims.

full rationale

The paper's formal derivation (Definitions 1–2, Proposition 1) is self-contained: it does not rely on the paper's own conclusions, on a fitted parameter renamed as a prediction, or on a load-bearing self-citation. The Proposition 1 proof is stated directly from the interface definitions; whether that proof is fully accurate is a correctness question, not a circularity question. The RACI ablation is an empirical mechanism test, not a derivation that presupposes its conclusion. The adaptive router is evaluated against a simulated quality oracle whose latent profile is explicitly constructed from literature trends ('factual tasks peak at voting; reasoning peaks at debate; code peaks at reflexion...'). The paper transparently labels this as a 'controlled simulated oracle' and 'a controlled stand-in,' and separately reports real-benchmark and real-reward runs. Thus the 82% routing accuracy and superiority over fixed baselines are presented as measurements of LinUCB recovering a synthetic target, not as independent real-world evidence. That is a legitimate validation design, not circularity. No self-citations are load-bearing, and no known external result is renamed as a new organizational law. The reader's concern about the orthogonality proof contradicting protocol implementations that read RACI tags is a substantive correctness objection, but it does not make the derivation chain circular.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper's central claims rest on (1) an interface contract for orthogonality that is violated in spirit by protocols reading RACI tags; (2) a hand-set synthetic oracle that defines the router's success; (3) standard LinUCB assumptions; and (4) imported organizational-theory constructs. The real-benchmark and RACI findings carry no hidden free parameters beyond sample-size choices, but the router's headline is tied to the oracle profile and cost/reward constants.

free parameters (5)
  • Simulated oracle latent mean quality μ(task-type, protocol) = hand-set, not enumerated numerically; encodes voting→factual, debate→reasoning, reflexion→code, plan-execute→decomp, MoA
    Defines the ground truth that the router's 'controlled study' is measured against; because these values are chosen to make each type have a clearly separated optimal arm, the router's 82% accuracy is a recovery of this hand-set profile, not an independent measurement.
  • Protocol cost multiplicities c(π) = voting=1.0, MoA/blender=1.5, reflexion=2.0, plan-execute=2.5, debate=3.0
    Hand-set model-call multiplicity drives the cost-adjusted reward (Eq. 1); changing these values changes the optimal arm ordering.
  • Cost-aversion coefficient λ = 0.3 default; sweep 0.1–0.5
    Trades quality against cost; headline router-vs-baseline comparisons use λ=0.3.
  • LinUCB exploration coefficient α = 0.6
    Exploration constant in Eq. 2; chosen without a tuning study.
  • Seven-dimensional task feature mapping = intercept + six normalized counts/keyword densities in [0,1]
    The context vector is hand-designed; the intercept-only ablation shows context matters, but feature quality is not compared against alternatives.
axioms (4)
  • domain assumption LinUCB linear-payoff model with bounded rewards and d=7 context
    Router selection (Eq. 2) and the regret bound assume payoffs are approximately linear in the seven hand-designed features; the paper asserts the assumptions hold 'by construction' for the synthetic oracle, but real model rewards are not shown to be linear.
  • domain assumption Protocols interact with organizations only through Team/Blackboard interfaces that expose no roles, model identities, κ, or ρ
    This is the premise of Proposition 1's orthogonality proof, contradicted by the RACI ablation section where Blender uses the Accountable tag and MoA consults it as fallback.
  • domain assumption Belbin's nine roles, Mintzberg's five coordination mechanisms, and RACI single-owner constraint transfer to LLM agent teams
    The framework's organizations are grounded in these management theories; no independent evidence in this paper validates that the mapped roles/coordination affect LLM outcomes beyond the small RACI finding.
  • ad hoc to paper Simulated oracle quality profile (per task-type/protocol) is a faithful stand-in for real task payoffs
    The oracle is constructed by the authors to encode literature trends and a clearly separated optimal arm per type; router success is measured against this constructed ground truth.

pith-pipeline@v1.3.0-alltime-deepseek · 12458 in / 17836 out tokens · 187887 ms · 2026-08-01T02:24:37.591403+00:00 · methodology

0 comments
read the original abstract

Multi-agent frameworks built on large language models (LLMs) routinely entangle three logically distinct concerns: who is on the team (organization), how members align (coordination), and which algorithm fuses their work (collaboration protocol). IMACS (Intelligent Multi-Agent Collaboration System) separates the three into orthogonal, independently swappable layers. Classic organizational theory (Belbin roles, Mintzberg coordination, RACI accountability) becomes executable, validated configuration, and the framework places six published collaboration algorithms behind a common interface while exposing roles, coordination, and accountability as independently configurable factors. We use this separation to conduct controlled comparisons in which organizational assignments vary while the collaboration protocol is held fixed. It also turns protocol choice into a variable that can be learned: Adaptive Org Routing, a contextual-bandit meta-protocol, selects a protocol per task under an explicit quality-cost tradeoff, outperforms every fixed protocol in a controlled study, and trains online on real benchmark and LLM-judge rewards. The ablations expose a mechanism. Accountability placement changes outcomes exactly when the protocol routes the deliverable through the accountable agent, and the winning placement flips across model families, so organizational design cannot be hard-coded; it must be revalidated, or learned, for each model binding.

Figures

Figures reproduced from arXiv: 2607.25446 by Huan Chen, Jian Jin, Liang-Jie Zhang, Pan Ren, Xiang Song.

Figure 1
Figure 1. Figure 1: The seven-layer IMACS stack. Inside the dashed region, the organization (WHO/HOW) and the collaboration protocol (ALGORITHM) meet only through the runtime’s Team and Blackboard interfaces, so any organization com￾poses with any protocol (Proposition 1). The adaptive router is itself an L4 protocol that selects among the others. L2 Agent. A role-typed wrapper binding a model handle, a system prompt, and too… view at source ↗
Figure 2
Figure 2. Figure 2: Online learning (λ=0.3, T=400). Left: cumula￾tive regret vs. the cost-adjusted oracle grows sublinearly and flattens as the policy converges (consistent with the O˜( √ T) LinUCB bound). Right: last-50 routing accuracy climbs to￾ward its plateau (dashed: final 82%). Ablation: task context. The context-free bandit (constant feature) collapses to mean reward 0.628, the level of always￾voting (0.629) and below… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

61 extracted references · 5 linked inside Pith

  1. [1]

    Auer, P.; Cesa-Bianchi, N.; and Fischer, P. 2002. Finite-time Analysis of the Multiarmed Bandit Problem. Machine Learning, 47: 235--256

    2002

  2. [2]

    Belbin, R. M. 2010. Management Teams: Why They Succeed or Fail. Oxford, UK: Butterworth-Heinemann, 3rd edition. First edition 1981

    2010

  3. [3]

    Z.; Yang, S.; Agrawal, L

    Cemri, M.; Pan, M. Z.; Yang, S.; Agrawal, L. A.; Chopra, B.; Tiwari, R.; Keutzer, K.; Parameswaran, A.; Klein, D.; Ramchandran, K.; Zaharia, M.; Gonzalez, J. E.; and Stoica, I. 2025. Why Do Multi-Agent LLM Systems Fail? In Advances in Neural Information Processing Systems (NeurIPS), volume 38

    2025

  4. [4]

    Chen, L.; Zaharia, M.; and Zou, J. 2024. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. Transactions on Machine Learning Research (TMLR)

    2024

  5. [6]

    Chen, W.; Su, Y.; Zuo, J.; Yang, C.; Yuan, C.; Chan, C.-M.; Yu, H.; Lu, Y.; Hung, Y.-H.; Qian, C.; Qin, Y.; Cong, X.; Xie, R.; Liu, Z.; Sun, M.; and Zhou, J. 2024. AgentVerse : Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors. In Proceedings of ICLR

    2024

  6. [7]

    Chu, W.; Li, L.; Reyzin, L.; and Schapire, R. 2011. Contextual Bandits with Linear Payoff Functions. In Proceedings of AISTATS, 208--214

    2011

  7. [9]

    Du, H.; Su, J.; Li, J.; Ding, L.; Yang, Y.; Han, P.; Tang, X.; Zhu, K.; and You, J. 2026. Which LLM Multi-Agent Protocol to Choose? In Proceedings of ICML

    2026

  8. [10]

    B.; and Mordatch, I

    Du, Y.; Li, S.; Torralba, A.; Tenenbaum, J. B.; and Mordatch, I. 2024. Improving Factuality and Reasoning in Language Models through Multiagent Debate. In Proceedings of ICML, 11733--11763

    2024

  9. [14]

    V.; Wiest, O.; and Zhang, X

    Guo, T.; Chen, X.; Wang, Y.; Chang, R.; Pei, S.; Chawla, N. V.; Wiest, O.; and Zhang, X. 2024. Large Language Model Based Multi-Agents: A Survey of Progress and Challenges. In Proceedings of IJCAI

    2024

  10. [15]

    Hong, S.; Zhuge, M.; Chen, J.; Zheng, X.; Cheng, Y.; Zhang, C.; Wang, J.; Wang, Z.; Yau, S. K. S.; Lin, Z.; et al. 2024. MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework. In International Conference on Learning Representations (ICLR)

    2024

  11. [16]

    Jiang, D.; Ren, X.; and Lin, B. Y. 2023. LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion. In Proceedings of ACL, 14165--14178

    2023

  12. [17]

    Li, G.; Hammoud, H. A. A. K.; Itani, H.; Khizbullin, D.; and Ghanem, B. 2023. CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 51991--52008

    2023

  13. [18]

    Li, L.; Chu, W.; Langford, J.; and Schapire, R. E. 2010. A Contextual-Bandit Approach to Personalized News Article Recommendation. In Proceedings of WWW, 661--670

    2010

  14. [19]

    Liu, Z.; Zhang, Y.; Li, P.; Liu, Y.; and Yang, D. 2024. A Dynamic LLM -Powered Agent Network for Task-Oriented Agent Collaboration. In Proceedings of COLM

    2024

  15. [20]

    Mintzberg, H. 1979. The Structuring of Organizations: A Synthesis of the Research. Englewood Cliffs, NJ: Prentice-Hall

    1979

  16. [21]

    E.; Kadous, M

    Ong, I.; Almahairi, A.; Wu, V.; Chiang, W.-L.; Wu, T.; Gonzalez, J. E.; Kadous, M. W.; and Stoica, I. 2025. RouteLLM : Learning to Route LLMs from Preference Data. In The Thirteenth International Conference on Learning Representations (ICLR)

    2025

  17. [22]

    C.; and Zuo, J

    Poon, M.; Dai, X.; Liu, X.; Kong, F.; Lui, J. C.; and Zuo, J. 2026. Online Multi-LLM Selection via Contextual Bandits under Unstructured Context Evolution. In Proceedings of the 40th AAAI Conference on Artificial Intelligence, volume 40, 24855--24863

    2026

  18. [23]

    Qian, C.; Liu, W.; Liu, H.; Chen, N.; Dang, Y.; Li, J.; Yang, C.; Chen, W.; Su, Y.; Cong, X.; Xu, J.; Li, D.; Liu, Z.; and Sun, M. 2024. ChatDev: Communicative Agents for Software Development. In Proceedings of ACL, 15174--15186

    2024

  19. [24]

    Shinn, N.; Cassano, F.; Berman, E.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 8634--8652

    2023

  20. [25]

    Shnitzer, T.; Ou, A.; Silva, M.; Soule, K.; Sun, Y.; Solomon, J.; Thompson, N.; and Yurochkin, M. 2024. Large Language Model Routing with Benchmark Datasets. In Proceedings of COLM

    2024

  21. [26]

    Wang, J.; Wang, J.; Athiwaratkun, B.; Zhang, C.; and Zou, J. 2025. Mixture-of-Agents Enhances Large Language Model Capabilities. In The Thirteenth International Conference on Learning Representations (ICLR)

    2025

  22. [27]

    K.-W.; and Lim, E.-P

    Wang, L.; Xu, W.; Lan, Y.; Hu, Z.; Lan, Y.; Lee, R. K.-W.; and Lim, E.-P. 2023 a . Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. In Proceedings of ACL, 2609--2634

    2023

  23. [28]

    Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; Narang, S.; Chowdhery, A.; and Zhou, D. 2023 b . Self-Consistency Improves Chain of Thought Reasoning in Language Models. In International Conference on Learning Representations (ICLR)

    2023

  24. [29]

    H.; White, R

    Wu, Q.; Bansal, G.; Zhang, J.; Wu, Y.; Li, B.; Zhu, E.; Jiang, L.; Zhang, X.; Zhang, S.; Liu, J.; Awadallah, A. H.; White, R. W.; Burger, D.; and Wang, C. 2024. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. In Proceedings of COLM

    2024

  25. [30]

    Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations (ICLR)

    2023

  26. [31]

    Zheng, M.; Pei, J.; Logeswaran, L.; Lee, M.; and Jurgens, D. 2024. When ``A Helpful Assistant'' Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024, 15126--15154

    2024

  27. [32]

    Zhu, K.; Du, H.; Hong, Z.; Yang, X.; Guo, S.; Wang, Z.; Wang, Z.; Qian, C.; Tang, X.; Ji, H.; and You, J. 2025. MultiAgentBench : Evaluating the Collaboration and Competition of LLM Agents. In Proceedings of ACL

    2025

  28. [33]

    Zhuge, M.; Wang, W.; Kirsch, L.; Faccio, F.; Khizbullin, D.; and Schmidhuber, J. 2024. GPTSwarm : Language Agents as Optimizable Graphs. In Proceedings of ICML

    2024

  29. [34]

    2010 , edition =

    Management Teams: Why They Succeed or Fail , author =. 2010 , edition =

    2010

  30. [35]

    1979 , publisher =

    The Structuring of Organizations: A Synthesis of the Research , author =. 1979 , publisher =

    1979

  31. [36]

    The Thirteenth International Conference on Learning Representations (ICLR) , year =

    Mixture-of-Agents Enhances Large Language Model Capabilities , author =. The Thirteenth International Conference on Learning Representations (ICLR) , year =

  32. [37]

    Proceedings of ICML , pages =

    Improving Factuality and Reasoning in Language Models through Multiagent Debate , author =. Proceedings of ICML , pages =

  33. [38]

    International Conference on Learning Representations (ICLR) , year =

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author =. International Conference on Learning Representations (ICLR) , year =

  34. [39]

    Proceedings of ACL , year =

    LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion , author =. Proceedings of ACL , year =

  35. [40]

    Advances in Neural Information Processing Systems (NeurIPS) , volume =

    Reflexion: Language Agents with Verbal Reinforcement Learning , author =. Advances in Neural Information Processing Systems (NeurIPS) , volume =

  36. [41]

    International Conference on Learning Representations (ICLR) , year =

    MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework , author =. International Conference on Learning Representations (ICLR) , year =

  37. [42]

    Proceedings of ACL , year =

    Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models , author =. Proceedings of ACL , year =

  38. [43]

    Proceedings of COLM , year =

    AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation , author =. Proceedings of COLM , year =

  39. [44]

    Advances in Neural Information Processing Systems (NeurIPS) , volume =

    CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society , author =. Advances in Neural Information Processing Systems (NeurIPS) , volume =

  40. [45]

    Proceedings of ACL , pages =

    ChatDev: Communicative Agents for Software Development , author =. Proceedings of ACL , pages =

  41. [46]

    Proceedings of WWW , pages =

    A Contextual-Bandit Approach to Personalized News Article Recommendation , author =. Proceedings of WWW , pages =

  42. [47]

    Proceedings of the 40th AAAI Conference on Artificial Intelligence , volume =

    Online Multi-LLM Selection via Contextual Bandits under Unstructured Context Evolution , author =. Proceedings of the 40th AAAI Conference on Artificial Intelligence , volume =

  43. [48]

    Proceedings of ICML , year =

    Which LLM Multi-Agent Protocol to Choose? , author =. Proceedings of ICML , year =

  44. [49]

    arXiv preprint arXiv:2402.14034 , year =

    AgentScope: A Flexible yet Robust Multi-Agent Platform , author =. arXiv preprint arXiv:2402.14034 , year =

    Pith/arXiv arXiv

  45. [50]

    International Conference on Learning Representations (ICLR) , year =

    ReAct: Synergizing Reasoning and Acting in Language Models , author =. International Conference on Learning Representations (ICLR) , year =

  46. [51]

    Machine Learning , volume =

    Finite-time Analysis of the Multiarmed Bandit Problem , author =. Machine Learning , volume =. 2002 , publisher =

    2002

  47. [52]

    Proceedings of AISTATS , pages =

    Contextual Bandits with Linear Payoff Functions , author =. Proceedings of AISTATS , pages =

  48. [53]

    Transactions on Machine Learning Research (TMLR) , year =

    FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance , author =. Transactions on Machine Learning Research (TMLR) , year =

  49. [54]

    Proceedings of COLM , year =

    Large Language Model Routing with Benchmark Datasets , author =. Proceedings of COLM , year =

  50. [55]

    Proceedings of IJCAI , year =

    Large Language Model Based Multi-Agents: A Survey of Progress and Challenges , author =. Proceedings of IJCAI , year =

  51. [56]

    Advances in Neural Information Processing Systems (NeurIPS) , volume =

    Why Do Multi-Agent LLM Systems Fail? , author =. Advances in Neural Information Processing Systems (NeurIPS) , volume =

  52. [57]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , pages =

    When ``A Helpful Assistant'' Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models , author =. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages =

    2024

  53. [58]

    arXiv preprint arXiv:2110.14168 , year =

    Training Verifiers to Solve Math Word Problems , author =. arXiv preprint arXiv:2110.14168 , year =

    Pith/arXiv arXiv

  54. [59]

    arXiv preprint arXiv:2107.03374 , year =

    Evaluating Large Language Models Trained on Code , author =. arXiv preprint arXiv:2107.03374 , year =

    Pith/arXiv arXiv

  55. [60]

    Proceedings of ICML , year =

    Zhuge, Mingchen and Wang, Wenyi and Kirsch, Louis and Faccio, Francesco and Khizbullin, Dmitrii and Schmidhuber, J. Proceedings of ICML , year =

  56. [61]

    A Dynamic

    Liu, Zijun and Zhang, Yanzhe and Li, Peng and Liu, Yang and Yang, Diyi , booktitle =. A Dynamic

  57. [62]

    Chen, Weize and Su, Yusheng and Zuo, Jingwei and Yang, Cheng and Yuan, Chenfei and Chan, Chi-Min and Yu, Heyang and Lu, Yaxi and Hung, Yi-Hsin and Qian, Chen and Qin, Yujia and Cong, Xin and Xie, Ruobing and Liu, Zhiyuan and Sun, Maosong and Zhou, Jie , booktitle =

  58. [63]

    and Kadous, M Waleed and Stoica, Ion , booktitle =

    Ong, Isaac and Almahairi, Amjad and Wu, Vincent and Chiang, Wei-Lin and Wu, Tianhao and Gonzalez, Joseph E. and Kadous, M Waleed and Stoica, Ion , booktitle =

  59. [64]

    Zhu, Kunlun and Du, Hongyi and Hong, Zhaochen and Yang, Xiaocheng and Guo, Shuyi and Wang, Zhe and Wang, Zhenhailong and Qian, Cheng and Tang, Xiangru and Ji, Heng and You, Jiaxuan , booktitle =

  60. [65]

    arXiv preprint arXiv:2604.27691 , year =

    When Agents Evolve, Institutions Follow , author =. arXiv preprint arXiv:2604.27691 , year =

    Pith/arXiv arXiv

  61. [66]

    arXiv preprint arXiv:2507.08616 , year =

    Gr. arXiv preprint arXiv:2507.08616 , year =

    Pith/arXiv arXiv