Pith. sign in

REVIEW 4 major objections 5 minor 161 references

The paper argues that multi-agent debate between LLMs has converged on one narrow, untested design pattern, and offers a three-dimensional taxonomy to make comparisons reliable.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 00:27 UTC pith:K3ZCSNTM

load-bearing objection A solid, genuinely useful SLR with a real taxonomy and a reproducible protocol; the headline convergence claim in the abstract overshoots the univariate percentages, but that's a fixable framing issue, not a fatal flaw. the 4 major comments →

arxiv 2607.26212 v1 pith:K3ZCSNTM submitted 2026-07-28 cs.SE cs.AI

Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges

Quim Motger , Marc Oriol , Jordi Marco , Xavier Franch This is my paper
classification cs.SE cs.AI
keywords Multi-Agent DebateLLM agentssystematic literature reviewtaxonomyinteraction topologyagreement protocolsdesign patternsbenchmarking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that research on Multi-Agent Debate (MAD)—multiple LLM-based agents exchanging arguments to improve reasoning—has fragmented terminology but a strikingly uniform practice: nearly all systems use static, fully connected communication, exchange verbatim text, rely only on the current debate session as memory, and settle outcomes by majority vote. It argues this pattern was adopted by convention rather than because it was shown to be best, and that the few studies testing alternatives report real gains in accuracy or cost. To fix this, the paper contributes a three-dimensional taxonomy—participants, interaction, and agreement—with formal notation, derived from 141 primary studies. If the taxonomy is adopted, it would give the field a shared vocabulary, a reporting checklist, and a schema for machine-readable debate specifications, enabling controlled cost-aware benchmarking and automated tuning. The survey is descriptive, not experimental; its value is the claim that design choices are currently implicit and therefore incomparable.

Core claim

Across 141 studies (151 approaches), the paper finds Multi-Agent Debate has converged on one recipe: static fully connected topologies, verbatim natural-language exchange, short-term memory, majority-vote resolution. Alternatives—sparse or dynamic topologies, embedding messaging, persistent memory, judge-based argument evaluation—remain marginal despite early evidence of gains. The paper's synthesis is a three-dimensional taxonomy (participants, interaction, agreement) with formal notation making each design choice explicit. Since a MAD setting involves roughly a dozen interacting decisions, cross-study comparison is unreliable when they are implicit; the taxonomy is offered as a descriptive

What carries the argument

The carrying object is the three-dimensional taxonomy plus its semi-formal notation: topology becomes a directed graph G=(V,E); protocol becomes temporal edge activation E(t); message format becomes a transformation f on messages; agreement becomes an authority–resolution pair. The taxonomy's dimensions—participants (personas, roles, base model), interaction (topology, protocol, format, memory), agreement (authority, resolution)—decompose into orthogonal attribute values. This machinery works by rendering any MAD configuration as a point in a well-defined space, which makes the convergence claim precise, lets one study be expressed as a small delta over another, and turns the configuration s

Load-bearing premise

The survey's central finding rests on the assumption that the 141 studies obtained from one narrowly worded database query plus a single round of snowballing faithfully represent the whole multi-agent-debate literature; the paper itself flags this in Section 8 (external validity), noting that a different or broader search could change the observed distribution of designs.

What would settle it

A replication of the survey using broader search terms (e.g., 'multi-agent discussion', 'LLM deliberation', 'agent collaboration') across additional databases with multiple snowballing rounds, screening the same inclusion criteria; if it surfaces a substantial share of qualifying studies with dynamic topologies, embedding-based messaging, long-term memory, or non-voting resolution, the 'narrow design pattern' claim would be an artifact of search scope rather than a property of the field.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A MAD approach can be specified differentially: one study extends another by changing a few attributes, making ablations and incremental contributions precise rather than prose.
  • Reported performance can be attributed to specific design dimensions, enabling controlled benchmarking that isolates one factor while holding the rest fixed—the prerequisite for fair cross-study comparison.
  • The taxonomy identifies concrete research gaps: sparse and structured topologies, dynamic edge updating, interaction ordering, memory beyond the session, and embedding-based messaging are technically feasible but largely untested.
  • Because the notation is near-constructive, a MAD specification can become a machine-readable artifact amenable to version control and automated configuration search, including cost-aware tuning.
  • The dominant pattern (static, fully connected, verbatim, short-term, majority vote) is shown to be a convention, not an empirically justified optimum; whether alternatives win depends on measurement the field has not done.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The observed convergence may itself be a path-dependence effect: early influential debate frameworks popularized the default recipe, and later papers copied it rather than testing it; if so, the taxonomy's main practical payoff is turning 'what everyone does' into 'what has actually been compared.'
  • A testable extension: apply the taxonomy as a mandatory reporting checklist to new MAD papers; one would predict that reporting completeness correlates with the reproducibility and comparability of results.
  • The same decomposition could extend to adjacent paradigms the paper deliberately excludes—multimodal debate and human-AI deliberation—where analogous dimensions may need modality-specific extensions.
  • The taxonomy implies a research program: a cost-aware benchmark that varies one dimension at a time (e.g., topology under fixed protocol, memory type under fixed topology) would convert the field's anecdotal gains into a performance-cost map.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This manuscript reports a systematic literature review of 141 primary studies on Multi-Agent Debate (MAD) for LLM-based agentic systems. The authors define MAD, perform a Scopus-based search with backward/forward snowballing, and use independent dual screening with consensus to assemble the corpus. They inductively derive a three-dimensional taxonomy—participants, interaction, and agreement—accompanied by semi-formal notations (graphs for topology, temporal edge activation for protocols, message-transformation functions for format, authority–resolution pairs for agreement). The main empirical claim is that the field has implicitly converged on a narrow design pattern: static, fully connected topologies, verbatim natural-language exchange, short-term memory, and voting-style resolution, while alternatives such as dynamic topologies, sparse networks, embedding-based communication, and persistent memory remain marginal. The paper further positions the taxonomy as a descriptive map, a normative reporting checklist, and a potential schema for machine-readable MAD specifications and automated benchmarking.

Significance. If the conclusions hold, this is a valuable consolidation of a fast-moving and terminologically fragmented area. The study follows established SLR guidelines, provides explicit inclusion/exclusion criteria, reports dual screening with Cohen's kappa of 0.70, and—importantly—makes the full data extraction available in a replication package. The taxonomy is a useful contribution in itself, and the idea of treating MAD configurations as machine-readable artifacts is constructive. The paper's strengths are its transparency, the breadth of the surveyed corpus relative to earlier reviews, and the concrete instantiation of the taxonomy on ChatEval (Table 3). The main risk is that the central ‘narrow convergence’ claim rests on univariate percentages that may be inflated by trivial cases and multi-tagging; this needs to be addressed before the descriptive claim can be accepted as stated.

major comments (4)
  1. [§5.1.1, §5.1.2, Abstract] The central convergence claim is not yet supported by the reported margins. The paper itself notes that “all two-agent settings are trivially fixed” (§5.1.1) and that the bilateral case is the “trivial instance” of full connectivity (§5.1.2), but it never reports how many of the 151 approaches have |V|=2. Since percentages are computed over 151 approaches with multi-tagging allowed (footnote 4, §4.1), an approach can be counted as static and fully connected even when it mainly evaluates dynamic or structured alternatives. Please (i) report the number of two-agent approaches, (ii) recompute static/fully connected percentages excluding those trivial cases, and (iii) provide a joint distribution of the five headline attributes (static, fully connected, verbatim, short-term memory, voting). Without this, the phrase “implicitly converged on a narrow design pattern” conflates univariate margin
  2. [§9 vs. §5.2.1] The conclusion states that the field has converged on “static, fully connected topologies with simultaneous protocols, verbatim natural-language exchange, and short-term memory,” but the reported protocol frequencies are simultaneous 47.0%, sequential 42.4%, and hybrid 8.0% (§5.2.1). The Key Findings in §5.3.3 hedge with “simultaneous or sequential,” which is consistent with the data; the unhedged conclusion is not. Remove “simultaneous” from the summary of the convergent pattern, or substantiate it with a joint analysis showing that simultaneous protocols dominate among approaches that also exhibit the other four attributes.
  3. [§8, Internal validity; §4.1 footnote 4] Inter-rater reliability is reported only for study screening (Cohen's kappa 0.70). Data extraction and taxonomy coding—the source of every percentage in the paper—were performed by a single author per paper. Because the paper's main claim is quantitative (e.g., 88.7% static, 72.2% fully connected), the reliability of the coding is load-bearing. Please report a dual-coding reliability check on a sample of papers, or at minimum add an explicit limitation stating that the taxonomy percentages have unknown inter-coder reliability and temper the strength of the convergence claim accordingly.
  4. [§8, External validity] The paper correctly acknowledges the Scopus-only, precision-oriented search with a single snowballing iteration. However, the abstract and conclusions make claims about “the field,” not merely about the 141 included studies. To support this generalization, the authors should provide at least a small sensitivity analysis—for example, how many candidate papers were excluded solely because of terminology, or whether the dominant percentages are stable when a broader term such as “multi-agent discussion” is added to a complementary search. Without such a check, the reader cannot distinguish a property of the field from a property of the search strategy.
minor comments (5)
  1. [§6.2] Resolution percentages are computed over n=97 (approaches with a formalized resolution), while all other percentages use n=151. This denominator change is easy to miss and makes direct comparisons such as “46.4% majority vote” versus “88.7% static” misleading. Please state the denominator prominently in figures and when citing these numbers in the text.
  2. [§4.1 footnote 4] The footnote explains that totals can exceed 100% because an approach may combine or compare strategies, but it is not always clear which specific percentages in later sections are affected. For example, does the 72.2% fully-connected figure count an approach as fully connected if it includes one fully-connected comparison arm? Please make the counting rule explicit in a methodology table.
  3. [§2.2] The search string uses “MAD” as an acronym without restricting to the LLM debate context in the query itself. While precision was a stated goal, the initial search apparently returned only 29 results, suggesting the acronym did not cause major noise. A brief note on this would preempt readers' concerns.
  4. [Table 3] ChatEval's resolution is described as “Hybrid: majority vote if output is categorical; average if output is numerical.” This use of “hybrid” differs from the authority-level “Hybrid” in §6.1. Consider renaming one of the two to avoid confusion, for example “Mixed-resolution” for the Table 3 entry.
  5. [§7] The phrase “adopted by convention rather than systematic comparison” is an inference about authors' decision processes that the review cannot directly observe. It is a reasonable interpretation given the scarcity of comparative studies, but it should be framed as an interpretation, not a demonstrated fact.

Circularity Check

0 steps flagged

No significant circularity: the taxonomy is a descriptive coding of the reviewed corpus, not a derivation from fitted parameters or a self-citation chain.

full rationale

The paper's central claims are descriptive summaries of a systematically coded corpus, not predictions derived from definitions or fitted parameters. The percentages in Sections 4–6 (e.g., 88.7% static, 72.2% fully connected, 86.1% verbatim, 94.7% short-term memory, 46.4% majority vote over n=97 formalized resolutions) are computed from extracted codes and reported with explicit denominators and counting rules: footnote 4 states 'All percentages are computed over the set of distinct MAD approaches (n=151)... totals may exceed 100% when a single approach combines or compares several strategies.' The taxonomy is induced from the literature, and the paper explicitly acknowledges this in Section 8: 'the taxonomy was derived inductively from the literature rather than from a pre-established framework, meaning its categories may be shaped by the particular set of studies included.' That is an honest validity limitation of a systematic review, not a circular reduction: the paper does not use the taxonomy to generate the percentages; it reports the percentages as properties of the coded corpus. The only self-citation, [88], is used as a foundation for broadening an earlier RE-scoped mapping ('we build on that foundation to broaden the scope of the review'), but the current conclusions rest on the 141-study corpus and the described coding protocol, not on the prior paper's results, so the self-citation is not load-bearing. The skeptic's concerns about two-agent triviality, per-dimension counting, and univariate margins are interpretive/validity caveats about how strongly the percentages support the 'convergence' narrative; they do not make any equation or fitted value equal to its own input. No circular step was found; the score of 1 only acknowledges the minor, non-load-bearing self-citation.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The paper rests on domain assumptions about corpus representativeness and the adequacy of its MAD definition. No free parameters or invented entities are introduced.

axioms (3)
  • domain assumption The operationalized definition of MAD (Table 1) accurately captures the boundaries of the field.
    Inclusion/exclusion criteria require two or more AI agents exchanging opposing arguments in multiple rounds, same task, natural language only. If too restrictive, the taxonomy and prevalence percentages exclude adjacent paradigms and may not generalize.
  • domain assumption The Scopus search plus single-iteration snowballing yields a representative corpus of MAD research.
    Search string prioritizes precision over recall; snowballing is a single iteration, not to saturation. Coverage of non-Scopus venues and alternative terminology depends on citation links.
  • standard math Standard graph theory (directed graph G=(V,E), edge activation over time) is an adequate notation for MAD topologies.
    Used in Section 5 to formalize topology and protocol; assumes graph-based abstraction captures the relevant dynamics.

pith-pipeline@v1.3.0-alltime-deepseek · 33747 in / 10874 out tokens · 99083 ms · 2026-08-01T00:27:18.591000+00:00 · methodology

0 comments
read the original abstract

Multi-Agent Debate (MAD) is a promising paradigm for improving the accuracy and robustness of Large Language Model (LLM)-based agentic systems. It enables multiple agents to exchange arguments, critique each other's outputs, and iteratively converge towards a solution. However, research remains fragmented, with inconsistent terminology and no rigorous synthesis of MAD design dimensions. We present a systematic literature review characterizing 141 primary studies on MAD. We derive a three-dimensional taxonomy covering debate participants, the interaction mechanisms structuring the exchange, and the agreement protocols governing debate resolution, supported by formal notations to render MAD configurations. Our analysis reveals that the field has implicitly converged on a narrow design pattern - static, fully connected topologies, verbatim exchange, short-term memory and voting resolution strategies - adopted by convention rather than systematic comparison, while promising alternatives remain marginal. Because any MAD setting reflects roughly a dozen interacting design decisions, cross-study comparison is unreliable when these are left implicit. We position the taxonomy as a descriptive map of the research landscape, a framework for controlled benchmarking, and potentially as a schema for machine-readable MAD specifications. As future work, we propose formalizing it into an executable specification, enabling cost-aware benchmarking and automated tuning of debate configurations.

Figures

Figures reproduced from arXiv: 2607.26212 by Jordi Marco, Marc Oriol, Quim Motger, Xavier Franch.

Figure 1
Figure 1. Figure 1: Overview of the search and snowballing process [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Taxonomy of the multi-agent debate landscape: application domains and addressed tasks. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Taxonomy of participants in multi-agent debate systems. [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Taxonomy of interaction in multi-agent debate systems. [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Overview of structured network topologies for interaction in MAD systems. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Taxonomy of agreement in multi-agent debate systems. [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

161 extracted references · 13 canonical work pages

  1. [1]

    ACM SIGSOFT Empirical Standards. n.d.. Systematic Reviews Standard. https://www2.sigsoft.org/EmpiricalStandards/docs/standards?standard= SystematicReviews#. Accessed: 2026-02-27

    2026

  2. [2]

    Samee Arif, Sualeha Farid, Abdul Hameed Azeemi, Awais Athar, and Agha Ali Raza. 2025. The Fellowship of the LLMs: Multi-Model Workflows for Synthetic Preference Optimization Dataset Generation. InProcs. of the Fourth Workshop on Generation, Evaluation and Metrics. ACL, 30–45. https://aclanthology.org/2025.gem-1.4/

    2025

  3. [3]

    Ali Asad, Stephen Obadinma, Radin Shayanfar, and Xiaodan Zhu. 2025. RedDebate: Safer Responses through Multi-Agent Red Teaming Debates. arXiv(2025). arXiv:2506.11083 arXiv preprint arXiv:2506.11083

    Pith/arXiv arXiv 2025

  4. [4]

    Joshua Ashkinaze, Emily Fry, Narendra Edara, Eric Gilbert, and Ceren Budak. 2025. Plurals: A System for Guiding LLMs via Simulated Social Ensembles. InProcs. of the 2025 CHI Conference on Human Factors in Computing Systems. ACM, Article 245, 21 pages. doi:10.1145/3706598.3713675

    arXiv 2025

  5. [5]

    Yilin Bai. 2024. ConfidenceCal: Enhancing LLMs Reliability through Confidence Calibration in Multi-Agent Debate. InProcs. of the 10th International Conference on Big Data and Information Analytics

    2024

  6. [6]

    Dezheng Bao, Yueci Yang, Xin Chen, et al. 2025. PD 3: A Project Duplication Detection Framework via Adapted Multi-Agent Debate.arXiv(2025). arXiv:2505.17492 arXiv preprint arXiv:2505.17492

    Pith/arXiv arXiv 2025

  7. [7]

    Ziqun Bao, Yu Ji, Wen Wu, Xi Chen, and Liang He. 2025. Supervisor Alignment Framework: Enhancing LLM Alignment with Query-Ignoring Strategy and Multi-Agent Interaction. InProcs. of the 2025 IEEE International Conference on Acoustics, Speech and Signal Processing

    2025

  8. [8]

    Ohav Barbi, Ori Yoran, and Mor Geva. 2025. Preventing Rogue Agents Improves Multi-Agent Collaboration. InProcs. of the 1st Workshop for Research on Agent Language Models. ACL, 486–511. https://aclanthology.org/2025.realm-1.34/

    2025

  9. [9]

    Chi-Min Chan, Weize Chen, Yusheng Su, et al. 2024. ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate. InProcs. of the 12th International Conference on Learning Representations

    2024

  10. [10]

    Edward Y. Chang. 2025.EVINCE: Optimizing Adversarial LLM Dialogues via Conditional Statistics and Information Theory. ACM. https: //doi.org/10.1145/3749421.3749431

    arXiv 2025

  11. [11]

    Baiting Chen, Tong Zhu, Jiale Han, et al. 2025. Incentivizing Truthful Language Models via Peer Elicitation Games.arXiv(2025). arXiv:2505.13636 arXiv preprint arXiv:2505.13636

    arXiv 2025

  12. [12]

    Huaben Chen, Wenkang Ji, Lufeng Xu, and Shiyu Zhao. 2025. Multi-Agent Consensus Seeking via Large Language Models. arXiv:2310.20151 [cs.CL] https://arxiv.org/abs/2310.20151 arXiv preprint arXiv:2310.20151

    Pith/arXiv arXiv 2025

  13. [13]

    Justin Chen, Swarnadeep Saha, and Mohit Bansal. 2024. ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs. InProcs. of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    2024

  14. [14]

    Justin Chih-Yao Chen, Swarnadeep Saha, Elias Stengel-Eskin, and Mohit Bansal. 2024. MAGDI: structured distillation of multi-agent interaction graphs improves reasoning in smaller language models. InProcs. of the 41st International Conference on Machine Learning. Article 280, 16 pages

    2024

  15. [15]

    Xinyi Chen, Angelica Chen, Dean Foster, and Elad Hazan. 2023. Playing large games with oracles and ai debate.arXiv(2023). arXiv:2312.04792

    Pith/arXiv arXiv 2023

  16. [16]

    Xi Chen, Huahui Yi, Mingke You, et al. 2025. Enhancing diagnostic capability with multi-agents conversational large language models.NPJ digital medicine8.0, 1 (2025)

    2025

  17. [17]

    Daixuan Cheng, Shaohan Huang, and Furu Wei. 2024. Adapting Large Language Models to Domains via Reading Comprehension. arXiv:2309.09530 [cs.CL]

    Pith/arXiv arXiv 2024

  18. [18]

    Ruoxi Cheng, Haoxuan Ma, Shuirong Cao, et al. 2024. Reinforcement learning from multi-role debates as feedback for bias mitigation in llms. arXiv(2024). arXiv:2404.10160 arXiv preprint arXiv:2404.10160

    Pith/arXiv arXiv 2024

  19. [19]

    Min Choi, Keonwoo Kim, Sungwon Chae, and Sangyeop Baek. 2025. An Empirical Study of Group Conformity in Multi-Agent Systems. InFindings of the Association for Computational Linguistics: ACL 2025. ACL. doi:10.18653/v1/2025.findings-acl.265

    doi:10.18653/v1/2025.findings-acl.265 2025

  20. [20]

    Zan-Kai Chong, Hiroyuki Ohsaki, and Bryan Ng. 2025. LLM-Net: Democratizing LLMs-as-a-Service through Blockchain-based Expert Networks. InProcs. of the 14th International Conference on Software and Computer Applications. ACM, 8 pages. doi:10.1145/3731806.3731827

    arXiv 2025

  21. [21]

    Jina Chun, Qihong Chen, Jiawei Li, and Iftekhar Ahmed. 2025. Is multi-agent debate (MAD) the silver bullet? An empirical analysis of MAD in code summarization and translation.arXiv(2025). arXiv:2503.12029 arXiv preprint arXiv:2503.12029

    Pith/arXiv arXiv 2025

  22. [22]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, et al. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. InProcs. of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). ACL. doi:10.18653/v1/N19-1300

    doi:10.18653/v1/n19-1300 2019

  23. [23]

    Peter Clark, Isaac Cowhey, Oren Etzioni, et al. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge.arXiv (2018). arXiv:1803.05457 arXiv preprint arXiv:1803.05457

    Pith/arXiv arXiv 2018

  24. [24]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, et al. 2021. Training Verifiers to Solve Math Word Problems.arXiv(2021). arXiv:2110.14168 arXiv preprint arXiv:2110.14168

    Pith/arXiv arXiv 2021

  25. [25]

    Yue Cui, Liuyi Yao, Zitao Li, Yaliang Li, Bolin Ding, and Xiaofang Zhou. 2025. Efficient Leave-one-out Approximation in LLM Multi-agent Debate Based on Introspection.arXiv(2025). arXiv:2505.22192 arXiv preprint arXiv:2505.22192

    Pith/arXiv arXiv 2025

  26. [26]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLORA: efficient finetuning of quantized LLMs. InProcs. of the 37th International Conference on Neural Information Processing Systems. Article 441, 28 pages

    2023

  27. [27]

    2025.Graph Theory(6 ed.)

    Reinhard Diestel. 2025.Graph Theory(6 ed.). Graduate Texts in Mathematics, Vol. 173. Springer-Verlag. Sixth edition. Manuscript submitted to ACM 30 Motger et al

    2025

  28. [28]

    Tenenbaum, and Igor Mordatch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. Improving Factuality and Reasoning in Language Models through Multiagent Debate. InProcs. of the 41st International Conference on Machine Learning

    2024

  29. [29]

    Zhihua Duan and Jialin Wang. 2025. Enhancing multi-agent consensus through third-party LLM integration: Analyzing uncertainty and mitigating hallucinations in large language models. InProcs. of the 8th International Conference on Advanced Algorithms and Control Engineering. IEEE

    2025

  30. [30]

    Andrew Estornell and Yang Liu. 2024. Multi-LLM debate: framework, principals, and interventions. InProcs. of the 38th International Conference on Neural Information Processing Systems. Curran Associates Inc., Article 911, 27 pages

    2024

  31. [31]

    Andrew Estornell, Jean-François Ton, Yuanshun Yao, and Yang Liu. 2024. ACC-collab: An actor-critic approach to multi-agent LLM collaboration. arXiv(2024). arXiv:2411.00053 arXiv preprint arXiv:2411.00053

    Pith/arXiv arXiv 2024

  32. [32]

    Wenqi Fan, Yujuan Ding, Liangbo Ning, et al. 2024. A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models. In Procs. of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 11 pages. doi:10.1145/3637528.3671470

    arXiv 2024

  33. [33]

    Yi Fang, Moxin Li, Wenjie Wang, Lin Hui, and Fuli Feng. 2025. Counterfactual Debating with Preset Stances for Hallucination Elimination of LLMs. InProcs. of the 31st International Conference on Computational Linguistics

    2025

  34. [34]

    Zhaopeng Feng, Jiayuan Su, Jiamei Zheng, et al . 2025. M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation. InProcs. of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL, 7084–7107. doi:10.18653/v1/2025.acl-long.351

    doi:10.18653/v1/2025.acl-long.351 2025

  35. [35]

    Shengyue Guan, Jindong Wang, Jiang Bian, et al. 2026. Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey.ACM Transactions on Intelligent Systems and Technology17, 4 (2026)

    2026

  36. [36]

    Yong Guan, Hao Peng, Lei Hou, and Juanzi Li. 2025. Mmd-ere: multi-agent multi-sided debate for event relation extraction. InProcs. of the 31st International Conference on Computational Linguistics

    2025

  37. [37]

    Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, et al. 2020. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In Procs. of the 58th Annual Meeting of the Association for Computational Linguistics. ACL, 8342–8360. doi:10.18653/v1/2020.acl-main.740

    doi:10.18653/v1/2020.acl-main.740 2020

  38. [38]

    Chen Han, Wenzhen Zheng, and Xijin Tang. 2025. Debate-to-Detect: Reformulating Misinformation Detection as a Real-World Debate with Large Language Models. InProcs. of the 2025 Conference on Empirical Methods in Natural Language Processing. ACL. doi:10.18653/v1/2025.emnlp-main.764

    doi:10.18653/v1/2025.emnlp-main.764 2025

  39. [39]

    Yidong He, Yongbin Liu, Chunping Ouyang, et al. 2024. Evaluating Human-Large Language Model Alignment in Group Process. InCCF International Conference on Natural Language Processing and Chinese Computing. Springer

    2024

  40. [40]

    Dan Hendrycks, Collin Burns, Steven Basart, et al. 2021. Measuring Massive Multitask Language Understanding.arXiv(2021). arXiv:2009.03300

    Pith/arXiv arXiv 2021

  41. [41]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, et al. 2021. Measuring Mathematical Problem Solving With the MATH Dataset.arXiv(2021). arXiv:2103.03874 arXiv preprint arXiv:2103.03874

    Pith/arXiv arXiv 2021

  42. [42]

    Sirui Hong, Mingchen Zhuge, Jiaqi Chen, et al. 2024. MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework. InProcs. of the 12th International Conference on Learning Representations. https://arxiv.org/abs/2308.00352

    Pith/arXiv arXiv 2024

  43. [43]

    Wentao Hu, Wengyu Zhang, Yiyang Jiang, et al. 2025. Removal of Hallucination on Hallucination: Debate-Augmented RAG. InProcs. of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL. doi:10.18653/v1/2025.acl-long.770

    doi:10.18653/v1/2025.acl-long.770 2025

  44. [44]

    Zhe Hu, Hou Pong Chan, Jing Li, and Yu Yin. 2025. Debate-to-Write: A Persona-Driven Multi-Agent Framework for Diverse Argument Generation. InProcs. of the 31st International Conference on Computational Linguistics. ACL. https://aclanthology.org/2025.coling-main.314/

    2025

  45. [45]

    Hao Huang, Tapan Shah, John Karigiannis, and Scott Evans. 2024. Physics and Data Collaborative Root Cause Analysis: Integrating Pretrained Large Language Models and Data-Driven AI for Trustworthy Asset Health Management. InAnnual Conference of the PHM Society (2024, 1)

    2024

  46. [46]

    Ronghong Huang. 2024. Improving Spam Detection with a Multi-Agent Debate Framework. InProcs. of the IEEE 4th International Conference on Information Technology, Big Data and Artificial Intelligence, Vol. 4

    2024

  47. [47]

    Masaki Ishizaka, Akihito Taya, and Yoshito Tobe. 2024. Sparkit: A mind map-based MAS for idea generation support. InProcs. of the International Workshop on Engineering Multi-Agent Systems. Springer

    2024

  48. [48]

    Korir Nancy Jeptoo and Chengjie Sun. 2025. Enhancing Fake News Detection with Large Language Models Through Multi-agent Debates. In Natural Language Processing and Chinese Computing

    2025

  49. [49]

    Yucheng Jiang, Yijia Shao, Dekun Ma, Sina Semnani, and Monica Lam. 2024. Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations. InProcs. of the 2024 Conference on Empirical Methods in Natural Language Processing. ACL. doi:10.18653/v1/2024.emnlp-main.554

    doi:10.18653/v1/2024.emnlp-main.554 2024

  50. [50]

    Weiqiang Jin, Dafu Su, Tao Tao, et al . 2025. Courtroom-FND: a multi-role fake news detection method based on argument switching-based courtroom debate.Journal of King Saud University Computer and Information Sciences37.0, 3 (2025), 33

    2025

  51. [51]

    Priyanka Kargupta, Ishika Agarwal, Tal August, and Jiawei Han. 2025. Tree-of-Debate: Multi-Persona Debate Trees Elicit Critical Thinking for Scientific Comparative Analysis. InProcs. of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL, 29378–29403. doi:10.18653/v1/2025.acl-long.1422

    work page doi:10.18653/v1/2025.acl-long.1422 2025

  52. [52]

    Yu He Ke, Rui Yang, Sui An Lie, et al. 2024. Enhancing Diagnostic Accuracy through Multi-Agent Conversations: Using Large Language Models to Mitigate Cognitive Bias. arXiv:2401.14589

    Pith/arXiv arXiv 2024

  53. [53]

    Zachary Kenton, Noah Siegel, János Kramár, et al. 2024. On scalable oversight with weak LLMs judging strong LLMs.Advances in Neural Information Processing Systems37.0 (2024)

    2024

  54. [54]

    Dayeon Ki, Rachel Rudinger, Tianyi Zhou, and Marine Carpuat. 2025. Multiple LLM Agents Debate for Equitable Cultural Alignment. InProcs. of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL. doi:10.18653/v1/2025.acl-long.1210 Manuscript submitted to ACM Multi-Agent Debate Strategies: Survey, Taxonomy, an...

    doi:10.18653/v1/2025.acl-long.1210 2025

  55. [55]

    Alex Kim, Keonwoo Kim, and Sangwon Yoon. 2024. DEBATE: Devil’s Advocate-Based Assessment and Text Evaluation. InFindings of the Association for Computational Linguistics: ACL 2024. ACL. doi:10.18653/v1/2024.findings-acl.112

    doi:10.18653/v1/2024.findings-acl.112 2024

  56. [56]

    2007.Guidelines for performing Systematic Literature Reviews in Software Engineering

    Barbara Ann Kitchenham and Stuart Charters. 2007.Guidelines for performing Systematic Literature Reviews in Software Engineering. Technical Report EBSE 2007-001. Keele University and Durham University Joint Report

    2007

  57. [57]

    Vincent, Saab Mansour, et al

    Mahnaz Koupaee, Jake W. Vincent, Saab Mansour, et al. 2025. Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation. InProcs. of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). ACL. doi:10.18653/v1/2025...

    work page doi:10.18653/v1/2025.naacl-long.609 2025

  58. [58]

    Hao Duong Le, Xin Xia, and Zhang Chen. 2024. Multi-agent causal discovery using large language models.arXiv(2024). arXiv:2407.15073 arXiv preprint arXiv:2407.15073

    Pith/arXiv arXiv 2024

  59. [59]

    Yeonji Lee, Sangjun Park, Kyunghyun Cho, and JinYeong Bak. 2024. MentalAgora: A Gateway to Advanced Personalized Care in Mental Health through Multi-Agent Debating and Attribute Control.arXiv(2024). arXiv:2407.02736 arXiv preprint arXiv:2407.02736

    Pith/arXiv arXiv 2024

  60. [60]

    Dawei Li, Zhen Tan, Peijia Qian, et al. 2025. SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents. InProcs. of Pacific-Asia Conference on Knowledge Discovery and Data Mining. Springer, 54–65

    2025

  61. [61]

    Haitao Li, Qian Dong, Junjie Chen, et al . 2024. LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods.arXiv(2024). arXiv:2412.05579

    Pith/arXiv arXiv 2024

  62. [62]

    Ruosen Li, Teerth Patel, and Xinya Du. 2024. PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations.Transactions on Machine Learning Research(2024)

    2024

  63. [63]

    Wong, and Min Yang

    Renhao Li, Minghuan Tan, Derek F. Wong, and Min Yang. 2024. CoEvol: Constructing Better Responses for Instruction Finetuning through Multi-Agent Cooperation. InProcs. of the 2024 Conference on Empirical Methods in Natural Language Processing. 4703–4721

    2024

  64. [64]

    Tianle Li, Xueguang Ma, Alex Zhuang, Yu Gu, Yu Su, and Wenhu Chen. 2023. Few-shot In-context Learning on Knowledge Base Question Answering. InProcs. of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). doi:10.18653/v1/2023.acl-long.385

    doi:10.18653/v1/2023.acl-long.385 2023

  65. [65]

    Wenhao Li, Selvakumar Manickam, Yung wey Chong, and Shankar Karuppayah. 2025. PhishDebate: An LLM-Based Multi-Agent Framework for Phishing Website Detection.arXiv(2025). arXiv:2506.15656 arXiv preprint arXiv:2506.15656

    arXiv 2025

  66. [66]

    Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. 2024. A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth1, 1 (2024), 9. doi:10.1007/s44336-024-00009-2

    doi:10.1007/s44336-024-00009-2 2024

  67. [67]

    Yunxuan Li, Yibing Du, Jiageng Zhang, et al . 2024. Improving Multi-Agent Debate with Sparse Communication Topology. InFindings of the Association for Computational Linguistics: EMNLP 2024. 7281–7294

    2024

  68. [68]

    Yu Li, Yi Huang, Guilin Qi, et al. 2026. Harnessing Diverse Perspectives: A Multi-agent Framework for Enhanced Error Detection in Knowledge Graphs. InDatabase Systems for Advanced Applications. Springer, 446–458

    2026

  69. [69]

    Yuran Li, Jama Hussein Mohamud, Chongren Sun, Di Wu, and Benoit Boulet. 2025. Leveraging LLMs as meta-judges: A multi-agent framework for evaluating LLM judgments.arXiv(2025). arXiv:2504.17087 arXiv preprint arXiv:2504.17087

    Pith/arXiv arXiv 2025

  70. [70]

    Yu Li, Shenyu Zhang, Rui Wu, et al. 2024. MATEval: A Multi-agent Discussion Framework for Advancing Open-Ended Text Evaluation. InDatabase Systems for Advanced Applications. Springer, 415–426

    2024

  71. [71]

    Tian Liang, Zhiwei He, Wenxiang Jiao, et al. 2024. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. In Procs. of the 2024 Conference on Empirical Methods in Natural Language Processing. 17889–17904

    2024

  72. [72]

    Sehee Lim, Yejin Kim, Chi-Hyun Choi, Jy-yong Sohn, and Byung-Hoon Kim. 2024. ERD: A Framework for Improving LLM Reasoning for Cognitive Distortion Classification. InProcs. of the 6th Clinical Natural Language Processing Workshop. 292–300

    2024

  73. [73]

    Fengyuan Liu, Rui Zhao, Shuo Chen, et al. 2025. Can an Individual Manipulate the Collective Decisions of Multi-Agents?. InProcs. of the 2025 Conference on Empirical Methods in Natural Language Processing. ACL, 12158–12182. doi:10.18653/v1/2025.emnlp-main.611

    work page doi:10.18653/v1/2025.emnlp-main.611 2025

  74. [74]

    Tongxuan Liu, Xingyu Wang, Weizhe Huang, et al. 2024. GroupDebate: Enhancing the efficiency of multi-agent debate using group discussion. arXiv(2024). arXiv:2409.14051 arXiv preprint arXiv:2409.14051

    arXiv 2024

  75. [75]

    Yexiang Liu, Jie Cao, Zekun Li, Ran He, and Tieniu Tan. 2025. Breaking mental set to improve reasoning through diverse multi-agent debate. In Procs. of the 13th International Conference on Learning Representations

    2025

  76. [76]

    Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. 2024. A dynamic LLM-powered agent network for task-oriented agent collaboration. In Procs. of the 1st Conference on Language Modeling

    2024

  77. [77]

    Li-Chun Lu, Shou-Jen Chen, Tsung-Min Pai, et al . 2024. LLM discussion: Enhancing the creativity of large language models via discussion framework and role-play.arXiv(2024). arXiv:2405.06373 arXiv preprint arXiv:2405.06373

    Pith/arXiv arXiv 2024

  78. [78]

    Junyu Luo, Weizhi Zhang, Ye Yuan, et al. 2025. Large Language Model Agent: A Survey on Methodology, Applications and Challenges.arXiv (2025). arXiv:2503.21460

    Pith/arXiv arXiv 2025

  79. [79]

    Junxia Ma, Changjiang Wang, Lu Rong, Bo Wang, and Yaoli Xu. 2025. Exploring multi-agent debate for zero-shot stance detection: A novel approach.Applied Sciences15.0, 9 (2025), 4612

    2025

  80. [80]

    Yiran Rex Ma. 2025. Do Androids Question Electric Sheep? A Multi-Agent Cognitive Simulation of Philosophical Reflection on Hybrid Table Reasoning. InProcs. of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop). ACL, 143–164. doi:10.18653/v1/2025.acl-srw.9

    work page doi:10.18653/v1/2025.acl-srw.9 2025

Showing first 80 references.