Pith. sign in

REVIEW 4 major objections 4 minor 61 references

Consensus decision protocols lift multi-agent LLMs on knowledge tasks; voting and judge protocols lift them on logic tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 10:52 UTC pith:44UH3LJ5

load-bearing objection Solid open multi-agent harness and useful ablations; the headline knowledge-vs-logic protocol ranking is overstated once MuSR 8B failures, Math outlier, and forced-round confounds are counted. the 4 major comments →

arxiv 2607.05477 v1 pith:44UH3LJ5 submitted 2026-07-06 cs.MA cs.AIcs.CL

Decision Protocols in Multi-Agent Large Language Model Conversations

Lars Benedikt Kaesberg This is my paper
classification cs.MA cs.AIcs.CL
keywords multi-agent systemslarge language modelsdecision protocolsconsensusvotingtest-time computeresponse diversityconversational task solving
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This thesis builds and open-sources MALLM, a modular multi-agent framework that lets language-model agents discuss a problem and then settle on one answer via consensus, voting, or a judge. Across knowledge-heavy benchmarks (MMLU family, GPQA) and logic-heavy ones (StrategyQA, MuSR, SQuAD 2.0, math), the same agents produce higher scores when the decision rule is matched to the task type: consensus for knowledge, voting or judge for logic. Forcing every agent to draft its own first solution, rather than iterating on a single seed draft, measurably raises answer diversity and final accuracy; small tweaks to what voters see at decision time (confidence scores, extra facts, anonymity) barely move the needle. Smaller models gain more from the multi-agent setup than larger ones, at the price of substantially higher inference cost.

Core claim

When identical Llama-3 agents discuss a task, the choice of decision protocol is not neutral: consensus protocols produce higher accuracy on knowledge-intensive datasets while voting and judge protocols produce higher accuracy on logic-oriented datasets. Independently generated initial solutions increase response diversity and improve decision quality; modest changes to the information available during the final decision step do not.

What carries the argument

MALLM's three families of decision protocols—consensus (majority / supermajority / unanimity agreement thresholds), voting (simple, ranked, cumulative, approval), and a neutral judge that synthesizes a final answer—applied after a fixed multi-agent discussion under controlled response generators and discussion paradigms.

Load-bearing premise

That the knowledge-versus-logic ranking of protocols observed on a few hundred sampled items per dataset, with three agents and a fixed three-round discussion default, generalizes beyond those sampling and configuration choices.

What would settle it

Rerun the same protocol matrix on the full unsampled test sets (or substantially larger random draws) of MMLU-Pro and StrategyQA with both 8B and 70B backbones; if consensus no longer leads on the knowledge set or voting/judge no longer lead on the logic set, the central ranking claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The thesis introduces the open-source MALLM framework for multi-agent LLM discussions and systematically compares consensus (majority/supermajority/unanimity), voting (simple/ranked/cumulative/approval), and judge decision protocols on Llama-3 8B/70B across knowledge-oriented (MMLU, MMLU-Pro, GPQA) and logic-oriented (StrategyQA, MuSR, Math-lvl-5, SQuAD 2.0) benchmarks. It reports that consensus tends to outperform on knowledge tasks while voting and judge protocols do better on logic tasks (with smaller models gaining more), that independent initial drafting and collective refinement improve diversity and accuracy, that small changes to information at the decision step have negligible effect, and that post-hoc single-agent challenges rarely improve the multi-agent solution. Supporting ablations cover discussion rounds, agent count, response generators, and compute cost.

Significance. If the protocol-by-domain ranking and diversity findings hold under better-controlled conditions, the work supplies practical design guidance for multi-agent LLM systems and a reusable modular framework (MALLM) that other researchers can extend. Strengths include the open-source release, three-run standard deviations, explicit CoT and single-agent baselines, multiple ablations (diversity, vote-info variants, challenge study, compute), and the honest null result on decision-phase information alterations. These are genuine contributions even if the strongest domain-split claim requires qualification. The compute-time analysis and persona-driven agent scaling observations are also useful for the test-time compute literature.

major comments (4)
  1. [§4.1, Table 2] §4.1 / Table 2 (8B MuSR column) and the accompanying text: the large voting/judge advantage on MuSR (~55–59 % vs. consensus ~28 %) is driven by the acknowledged failure of the 8B model (and consensus extraction) to follow the long-context prompt, while the 70B table (Table 3) largely removes the gap. This cell therefore confounds the claimed “logic-task preference for voting/judge” and should be either excluded from the domain-average claim, re-run with a prompt that both families can follow, or explicitly flagged as an artifact rather than evidence for the ranking.
  2. [§3.1.1, Table 4, Fig. 6a] §3.1.1 and Table 4 / Appendix A.2: voting and judge protocols are forced to three discussion rounds before any decision (citing Du et al.), whereas consensus protocols routinely terminate after 1–2 rounds. The paper’s own ablation (Fig. 6a) shows that additional forced rounds degrade accuracy. The protocol comparison therefore confounds decision rule with discussion length; a matched-round or early-voting control is needed before the knowledge-vs-logic ranking can be attributed to the decision mechanism itself.
  3. [§4.1.1] §4.1.1 Takeaways item 1 states the opposite of the Abstract, §4.1 body, Tables 2–3 and §5.1: “Voting protocols excel in knowledge-based tasks, while consensus protocols perform better in logic-based…”. This inversion of the central empirical claim must be corrected; as written it misleads readers who rely on the takeaway boxes.
  4. [§4.1, Fig. 5, §3.2] Math-lvl-5 is an explicit outlier (unanimity consensus best) and SQuAD unanswerable vs. answerable splits reverse the overall voting advantage (Fig. 5). Combined with the finite-population subsamples of ~250–375 items × 3 seeds (§3.2), the power to support a clean knowledge-vs-logic protocol ranking is limited. The claim should be restated with these caveats or supported by a formal interaction test / larger sample.
minor comments (4)
  1. [throughout] Several typographical and formatting issues remain (e.g., “paragdims”, “G¨ ottingen”, inconsistent spacing around percentages, “Anonymus Voting”). A careful proof-read is needed.
  2. [Tables 2–4, Figs. 7–9] Approval voting is correctly excluded from averages because of systematic ties, yet it still appears in several figures and tables without a clear visual cue that it is non-comparable; a consistent “failed-to-decide” marker would help.
  3. [§4.5] Compute-time measurements (§4.5) include queue wait and are acknowledged as approximate; reporting token counts or wall-clock under a fixed concurrency budget would make the 5–15 imes cost claims more reproducible.
  4. [Appendix D] Appendix D examples are valuable but long; a short caption summarizing success vs. failure modes would improve navigability.

Circularity Check

0 steps flagged

No circularity: pure empirical protocol comparison on external benchmarks with no fitted predictions or self-definitional reductions.

full rationale

This is an empirical systems paper that implements decision protocols (voting variants, consensus thresholds, judge) inside the MALLM framework and reports measured accuracy/F1 on fixed external datasets (MMLU, MMLU-Pro, GPQA, StrategyQA, MuSR, Math-lvl-5, SQuAD 2.0) against single-agent and CoT baselines. All claims (e.g., consensus stronger on knowledge tasks, voting/judge stronger on logic tasks; diversity helps; information alterations at decision time do not) are direct numerical outcomes of those runs (Tables 2–3, Figs. 5–13, Appendix A). There are no equations that redefine a target quantity in terms of itself, no parameters fitted to a subset then re-presented as predictions of a related quantity, no uniqueness theorems, no ansatz imported via self-citation, and no renaming of a known empirical pattern as a derived result. Self-citations (e.g., to Du et al. for the three-round default, or related multi-agent frameworks) supply only experimental defaults or context; the measured scores remain independent of those citations. The derivation chain is therefore self-contained against external labels and contains zero circular steps.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 1 invented entities

Load-bearing structure is experimental design choices and domain assumptions about LLM multi-agent chat, not free physical constants or new ontological entities. The central claim rests on sampling design, fixed agent/round defaults, persona generation, and the operational definitions of consensus/voting/judge inside MALLM.

free parameters (5)
  • agents_per_discussion
    Default num_agents=3 (Appendix C); agent-count sweep on StrategyQA is secondary. Rankings depend on this choice.
  • forced_discussion_rounds_before_vote
    Default three rounds before voting/judge, justified by Du et al.; Figure 6a shows sensitivity to this hyperparameter.
  • consensus_thresholds
    Majority 50%, supermajority 66%, unanimity 100% are design choices that define when consensus fires.
  • evaluation_sample_sizes
    Per-dataset n from finite-population 95%/5% formula (§3.2); performance estimates and protocol rankings are conditioned on these subsets × 3 seeds.
  • max_turns_and_tie_fallback
    Max turns (often 5) and fallback to first agent on unresolved ties affect approval voting and long discussions.
axioms (4)
  • domain assumption Persona-conditioned Llama-3 agents plus iterative discussion approximate useful multi-expert collaboration for the chosen tasks.
    Invoked throughout §3.1 and related work on personas; without it, multi-agent overhead is not expected to beat CoT.
  • domain assumption External benchmark labels (MMLU-family, GPQA, StrategyQA, MuSR, MATH-lvl-5, SQuAD 2.0) are valid ground truth for comparing decision protocols.
    All reported accuracy/F1 comparisons rest on these labels (§3.2, §4).
  • ad hoc to paper Memory discussion paradigm and Simple response generator are representative enough defaults for protocol comparison.
    Main tables use memory + simple; other paradigms appear mainly in diversity ablations (§4.1–4.2).
  • domain assumption Standard multi-choice / F1 scoring and regex/prompt extraction of agreement and final answers faithfully recover agent decisions.
    Consensus extraction and final-answer prompts in Appendix B; failures noted on MuSR 8B.
invented entities (1)
  • MALLM modular multi-agent discussion framework independent evidence
    purpose: Unify persona generation, response generators, discussion paradigms, and decision protocols for conversational task solving.
    Software system introduced in §3.1; not a physical entity. Independent handle is the claimed public repository and batch configs, not a new scientific object.

pith-pipeline@v1.1.0-grok45 · 48855 in / 3464 out tokens · 36956 ms · 2026-07-11T10:52:09.393392+00:00 · methodology

0 comments
read the original abstract

Improving the task performance of Large Language Models (LLMs) is essential, yet scaling these models faces significant challenges such as diminishing returns and high costs. Multi-Agent Systems (MAS) offer a promising solution by distributing tasks among specialized agents to improve the overall task performance. This can reduce training costs at the expense of increased test time due to the discussion and decision-making process. The decision protocol is a critical component of MAS because it specifies how multiple agents collaborate to create a final solution. This thesis introduces the Multi-Agent LLM (MALLM) framework, which implements and evaluates various decision protocols, namely voting, consensus, and judge decision mechanisms, to simulate multi-agent discussions for conversational task solving. Unlike previous work that used a single decision protocol or tested them on limited datasets, this study systematically examines their impact on a diverse set of tasks, ranging from knowledge-based datasets (MMLU, MMLU-Pro, GPQA) and logic-based datasets (StrategyQA, MuSR, Math-lvl-5, SQuAD 2.0). The results indicate that consensus protocols excel in knowledge-intensive domains while voting and judge protocols are more effective for logic-based tasks. Increasing response diversity through independent solution generation improves decision quality, while changes in information access during the decision process have minimal impact.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

61 extracted references · 3 canonical work pages

  1. [1]

    The elephant in the room: Analyzing the presence of big tech in natural language processing research,

    Abdalla, M., Wahle, J. P., Ruas, T., et al., “The elephant in the room: Analyzing the presence of big tech in natural language processing research,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2023. [Online]. Available: https://aclanthology.org/2023.acl-long.734

    2023

  2. [2]

    The computational difficulty of manipulating an election,

    Bartholdi, J. J., Tovey, C. A., and Trick, M. A., “The computational difficulty of manipulating an election,” Social Choice and Welfare , no. 3, 1989, issn: 1432-217X. [Online]. Available: https://doi.org/10.1007/BF00295861

    doi:10.1007/bf00295861 1989

  3. [3]

    [Online]

    Becker, J., Multi-Agent Large Language Models for Conversational Task-Solving, 2024. [Online]. Available: https://arxiv.org/abs/2410.22932

    Pith/arXiv arXiv 2024

  4. [4]

    Language models are few-shot learners,

    Brown, T. B., Mann, B., Ryder, N., et al. , “Language models are few-shot learners,” in Advances in Neural Information Processing Systems 33: Annual Conference on Neu- ral Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual ,

    2020

  5. [5]

    Available: https : / / proceedings

    [Online]. Available: https : / / proceedings . neurips . cc / paper / 2020 / hash / 1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html

    2020

  6. [6]

    Cecilia Navarra, Janˇ cov´ a, L., and Ioannides, I.,Qualified majority voting in common foreign and security policy: a cost of non Europe report. 2023. [Online]. Available: https://data. europa.eu/doi/10.2861/509043

    work page doi:10.2861/509043 2023

  7. [7]

    Chen, J. C. -Y., Saha, S., and Bansal, M., ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs , 2023. [Online]. Available: https://arxiv. org/abs/2309.13007

    Pith/arXiv arXiv 2023

  8. [8]

    G., Sampling techniques (Sampling techniques)

    Cochran, W. G., Sampling techniques (Sampling techniques). 1953

    1953

  9. [9]

    Social Pressure Influences Decisions of Individuals: Evidence from the Behavior of Football Referees,

    Dohmen, T., “Social Pressure Influences Decisions of Individuals: Evidence from the Behavior of Football Referees,” SSRN Electronic Journal, 2005, issn: 1556-5068. [Online]. Available: https://www.ssrn.com/abstract=725541

    2005

  10. [10]

    B., and Mordatch, I., Improving Factuality and Reasoning in Language Models through Multiagent Debate , 2023

    Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mordatch, I., Improving Factuality and Reasoning in Language Models through Multiagent Debate , 2023. [Online]. Available: https://arxiv.org/abs/2305.14325

    Pith/arXiv arXiv 2023

  11. [11]

    , The Llama 3 Herd of Models , 2024

    Dubey, A., Jauhri, A., Pandey, A., et al. , The Llama 3 Herd of Models , 2024. [Online]. Available: https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  12. [12]

    Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies,

    Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., and Berant, J., “Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies,” Transac- tions of the Association for Computational Linguistics , 2021. [Online]. Available: https : //aclanthology.org/2021.tacl-1.21

    2021

  13. [13]

    WORKING DOCUMENT on overcoming the deadlock of unanimity voting,

    Gozi, S., “WORKING DOCUMENT on overcoming the deadlock of unanimity voting,” European Parliament, Committee on Constitutional Affairs , 2021. 38 Decision Protocols in Multi-Agent Large Language Model Conversations 6. References

    2021

  14. [14]

    Measuring massive multitask language under- standing,

    Hendrycks, D., Burns, C., Basart, S., et al., “Measuring massive multitask language under- standing,” in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 , 2021. [Online]. Available: https://openreview.net/forum? id=d7KBjmI3GmQ

    2021

  15. [15]

    [Online]

    Hendrycks, D., Burns, C., Kadavath, S., et al., Measuring Mathematical Problem Solving With the MATH Dataset , 2021. [Online]. Available: https://arxiv.org/abs/2103.03874

    Pith/arXiv arXiv 2021

  16. [16]

    , MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework, 2023

    Hong, S., Zhuge, M., Chen, J., et al. , MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework, 2023. [Online]. Available: https://arxiv.org/abs/2308.00352

    Pith/arXiv arXiv 2023

  17. [17]

    [Online]

    Huang, J., Chen, X., Mishra, S., et al., Large Language Models Cannot Self-Correct Reasoning Yet, 2023. [Online]. Available: https://arxiv.org/abs/2310.01798

    Pith/arXiv arXiv 2023

  18. [18]

    PersonaLLM: Investigating the ability of large language models to express personality traits,

    Jiang, H., Zhang, X., Cao, X., Breazeal, C., Roy, D., and Kabbara, J., “PersonaLLM: Investigating the ability of large language models to express personality traits,” in Findings of the Association for Computational Linguistics: NAACL 2024 , 2024. [Online]. Available: https://aclanthology.org/2024.findings-naacl.229

    2024

  19. [19]

    A comparison of consensus and voting in public decision making,

    Jones, B., “A comparison of consensus and voting in public decision making,” Negotiation Journal, no. 2, 1994, issn: 1571-9979. [Online]. Available: https : / / doi . org / 10 . 1007 / BF02184175

    1994

  20. [20]

    [Online]

    Kim, J., Yang, N., and Jung, K., Persona is a Double-edged Sword: Mitigating the Negative Impact of Role-playing Prompts in Zero-shot Reasoning Tasks , 2024. [Online]. Available: https://arxiv.org/abs/2408.08631

    Pith/arXiv arXiv 2024

  21. [21]

    Conformity in Groups: The Effects of Others’ Views on Expressed Attitudes and Attitude Change,

    Levitan, L. C. and Verhulst, B., “Conformity in Groups: The Effects of Others’ Views on Expressed Attitudes and Attitude Change,” Political Behavior, no. 2, 2016, issn: 1573-6687. [Online]. Available: https://doi.org/10.1007/s11109-015-9312-x

    work page doi:10.1007/s11109-015-9312-x 2016

  22. [22]

    [Online]

    Li, J., Chen, J., Ren, R., et al., The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models , 2024. [Online]. Available: https://arxiv.org/ abs/2401.03205

    Pith/arXiv arXiv 2024

  23. [23]

    Social Choice Theory,

    List, C., “Social Choice Theory,” in The Stanford Encyclopedia of Philosophy , Winter 2022,

    2022

  24. [24]

    Available: https://plato.stanford.edu/archives/win2022/entries/ social-choice/

    [Online]. Available: https://plato.stanford.edu/archives/win2022/entries/ social-choice/

  25. [25]

    and Yin, Y., Green AI: Exploring Carbon Footprints, Mitigation Strategies, and Trade Offs in Large Language Model Training , 2024

    Liu, V. and Yin, Y., Green AI: Exploring Carbon Footprints, Mitigation Strategies, and Trade Offs in Large Language Model Training , 2024. [Online]. Available: https://arxiv.org/abs/ 2404.01157

    Pith/arXiv arXiv 2024

  26. [26]

    Self-refine: Iterative refinement with self-feedback,

    Madaan, A., Tandon, N., Gupta, P.,et al., “Self-refine: Iterative refinement with self-feedback,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023. [Online]. Available: http://papers.nips.cc/paper%5C_files/paper/ 2023/...

    2023

  27. [27]

    The effects of information and social conformity on opinion change,

    Mallinson, D. J. and Hatemi, P. K., “The effects of information and social conformity on opinion change,” PLOS ONE, no. 5, 2018, issn: 1932-6203. [Online]. Available: https: //journals.plos.org/plosone/article?id=10.1371/journal.pone.0196600

    work page doi:10.1371/journal.pone.0196600 2018

  28. [28]

    The Society of Mind,

    Minsky, M., “The Society of Mind,” The Personalist Forum, no. 1, 1987, issn: 0889-065X. [Online]. Available: https://www.jstor.org/stable/20708493

    arXiv 1987

  29. [29]

    U., Qiu, S., et al

    Naveed, H., Khan, A. U., Qiu, S., et al. , A Comprehensive Overview of Large Language Models, 2023. [Online]. Available: https://arxiv.org/abs/2307.06435

    Pith/arXiv arXiv 2023

  30. [30]

    , GPT-4 Technical Report, 2023

    OpenAI, Achiam, J., Adler, S., et al. , GPT-4 Technical Report, 2023. [Online]. Available: https://arxiv.org/abs/2303.08774

    Pith/arXiv arXiv 2023

  31. [31]

    Know what you don’t know: Unanswerable questions for SQuAD,

    Rajpurkar, P., Jia, R., and Liang, P., “Know what you don’t know: Unanswerable questions for SQuAD,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , 2018. [Online]. Available: https://aclanthology. org/P18-2124

    2018

  32. [32]

    Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging,

    Reimers, N. and Gurevych, I., “Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , 2017. [Online]. Available: https : //aclanthology.org/D17-1035

    2017

  33. [33]

    Sentence-BERT: Sentence embeddings using Siamese BERT- networks,

    Reimers, N. and Gurevych, I., “Sentence-BERT: Sentence embeddings using Siamese BERT- networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019. [Online]. Available: https://aclanthology.org/D19-1410

    2019

  34. [34]

    L., Stickland, A

    Rein, D., Hou, B. L., Stickland, A. C., et al., GPQA: A Graduate-Level Google-Proof Q&A Benchmark, 2023. [Online]. Available: https://arxiv.org/abs/2311.12022

    Pith/arXiv arXiv 2023

  35. [35]

    P., Zhou, Y., et al., PersonaGym: Evaluating Persona Agents and LLMs ,

    Samuel, V., Zou, H. P., Zhou, Y., et al., PersonaGym: Evaluating Persona Agents and LLMs ,

  36. [36]

    Available: https://arxiv.org/abs/2407.18416

    [Online]. Available: https://arxiv.org/abs/2407.18416

    Pith/arXiv arXiv

  37. [37]

    [Online]

    Sharma, M., Tong, M., Korbak, T., et al., Towards Understanding Sycophancy in Language Models, 2023. [Online]. Available: https://arxiv.org/abs/2310.13548

    Pith/arXiv arXiv 2023

  38. [38]

    Large language models can be easily distracted by irrelevant context,

    Shi, F., Chen, X., Misra, K., et al. , “Large language models can be easily distracted by irrelevant context,” in International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , ser. Proceedings of Machine Learning Research, 2023. [Online]. Available: https://proceedings.mlr.press/v202/shi23a.html

    2023

  39. [39]

    [Online]

    Snell, C., Lee, J., Xu, K., and Kumar, A., Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters , 2024. [Online]. Available: https : //arxiv.org/abs/2408.03314

    Pith/arXiv arXiv 2024

  40. [40]

    [Online]

    Sprague, Z., Ye, X., Bostrom, K., Chaudhuri, S., and Durrett, G., MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning , 2023. [Online]. Available: https: //arxiv.org/abs/2310.16049. 40 Decision Protocols in Multi-Agent Large Language Model Conversations 6. References

    Pith/arXiv arXiv 2023

  41. [41]

    The future of ai development

    Sutskever, I. “The future of ai development.” (2024), [Online]. Available: https : / / www . theverge.com/2024/12/13/24320811/what-ilya-sutskever-sees-openai-model-data- training

    2024

  42. [42]

    [Online]

    Team, Q., QwQ: Reflect Deeply on the Boundaries of the Unknown , 2024. [Online]. Available: http://qwenlm.github.io/blog/qwq-32b-preview/

    2024

  43. [43]

    K., Sampling (Wiley series in probability and statistics), 3rd ed

    Thompson, S. K., Sampling (Wiley series in probability and statistics), 3rd ed. 2012, isbn: 978-0-470-40231-3

    2012

  44. [44]

    Are Neural Language Models Good Plagiarists? A Benchmark for Neural Paraphrase Detection,

    Wahle, J. P., Ruas, T., Meuschke, N., and Gipp, B., “Are Neural Language Models Good Plagiarists? A Benchmark for Neural Paraphrase Detection,” in 2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL) , 2021, isbn: 978-1-66541-770-9. [Online]. Available: https://ieeexplore.ieee.org/document/9651895/

    arXiv 2021

  45. [45]

    Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?

    Wang, Q., Wang, Z., Su, Y., Tong, H., and Song, Y., “Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?” In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2024. [Online]. Available: https://aclanthology.org/2024.acl-long.331

    2024

  46. [46]

    Self-consistency improves chain of thought reasoning in language models,

    Wang, X., Wei, J., Schuurmans, D., et al. , “Self-consistency improves chain of thought reasoning in language models,” in The Eleventh International Conference on Learning Rep- resentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 , 2023. [Online]. Available: https: //openreview.net/pdf?id=1PL1NIMMrw

    2023

  47. [47]

    [Online]

    Wang, Y., Ma, X., Zhang, G., et al., MMLU-Pro: A More Robust and Challenging Multi- Task Language Understanding Benchmark (Published at NeurIPS 2024 Track Datasets and Benchmarks), 2024. [Online]. Available: https://arxiv.org/abs/2406.01574

    Pith/arXiv arXiv 2024

  48. [48]

    Unleashing the emergent cognitive syn- ergy in large language models: A task-solving agent through multi-persona self-collaboration,

    Wang, Z., Mao, S., Wu, W., Ge, T., Wei, F., and Ji, H., “Unleashing the emergent cognitive syn- ergy in large language models: A task-solving agent through multi-persona self-collaboration,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pape...

    2024

  49. [49]

    Chain-of-thought prompting elicits reasoning in large language models,

    Wei, J., Wang, X., Schuurmans, D., et al., “Chain-of-thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 , 2022. [Online]. Available: http://papers.nips. cc/...

    2022

  50. [50]

    System 2 thinking in OpenAI’s o1-preview model: Near-perfect performance on a mathematics exam,

    Winter, J. d., Dodou, D., and Eisma, Y. B., “System 2 thinking in OpenAI’s o1-preview model: Near-perfect performance on a mathematics exam,” ArXiv preprint, 2024. [Online]. Available: https://arxiv.org/abs/2410.07114. 41 Decision Protocols in Multi-Agent Large Language Model Conversations 6. References

    Pith/arXiv arXiv 2024

  51. [51]

    [Online]

    Wu, Q., Bansal, G., Zhang, J., et al., AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation, 2023. [Online]. Available: https://arxiv.org/abs/2308.08155

    Pith/arXiv arXiv 2023

  52. [52]

    , Qwen2 Technical Report , 2024

    Yang, A., Yang, B., Hui, B., et al. , Qwen2 Technical Report , 2024. [Online]. Available: https://arxiv.org/abs/2407.10671

    Pith/arXiv arXiv 2024

  53. [53]

    [Online]

    Yang, H., Wang, Y., Xu, X., Zhang, H., and Bian, Y., Can We Trust LLMs? Mitigate Overconfidence Bias in LLMs through Knowledge Transfer , 2024. [Online]. Available: https: //arxiv.org/abs/2405.16856

    Pith/arXiv arXiv 2024

  54. [54]

    C., Korecki, M., Dailisan, D., Hausladen, C

    Yang, J. C., Korecki, M., Dailisan, D., Hausladen, C. I., and Helbing, D., LLM Voting: Human Choices and AI Collective Decision Making , 2024. [Online]. Available: https://arxiv.org/ abs/2402.01766

    Pith/arXiv arXiv 2024

  55. [55]

    Exchange-of-thought: Enhancing large language model capabilities through cross-model communication,

    Yin, Z., Sun, Q., Chang, C., et al., “Exchange-of-thought: Enhancing large language model capabilities through cross-model communication,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , 2023. [Online]. Available: https : //aclanthology.org/2023.emnlp-main.936

    2023

  56. [56]

    Judging llm-as-a-judge with mt-bench and chat- bot arena,

    Zheng, L., Chiang, W., Sheng, Y., et al., “Judging llm-as-a-judge with mt-bench and chat- bot arena,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023. [Online]. Available: http://papers.nips.cc/paper%5C_ files/paper/2...

    2023

  57. [57]

    [Online]

    Zhuge, M., Liu, H., Faccio, F., et al., Mindstorms in Natural Language-Based Societies of Mind, 2023. [Online]. Available: https://arxiv.org/abs/2305.17066

    arXiv 2023

  58. [58]

    https://api.openai.com/v1

    Zhuge, M., Wang, W., Kirsch, L., Faccio, F., Khizbullin, D., and Schmidhuber, J., Language Agents as Optimizable Graphs , 2024. [Online]. Available: https://arxiv.org/abs/2402. 16823. 42 Decision Protocols in Multi-Agent Large Language Model Conversations A. Results A. Results Additional results for all experiments to provide further information. A.1. Alt...

    2024

  59. [59]

    Concert Promoter Local Music Expert ranked answers:

  60. [60]

    Concert Promoter Concert Promoter ranked answers:

  61. [61]

    Concert Promoter xxviii Decision Protocols in Multi-Agent Large Language Model Conversations D. Example Discussions Result Judge Local Music Enthusiast final answer: Southampton Choral Society Music Researcher final answer: Southampton Philharmonic Choir Promoter final answer: Southampton Philharmonic Choir Judge created new solution: Southampton Philharm...