AgentCDM: Enhancing Multi-Agent Collaborative Decision-Making via ACH-Inspired Structured Reasoning

Hualong Yu; Liting Zhang; Qicheng Li; Shiwan Zhao; Xuyang Zhao

REVIEW 5 major objections 5 minor 41 references

AgentCDM: Enhancing Multi-Agent Collaborative Decision-Making via ACH-Inspired Structured Reasoning

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AgentCDM claims that a small decision agent trained to reason through competing hypotheses, evidence matrices, and falsification can beat voting and untrained dictatorial arbiters in LLM multi-agent systems.

desk verdict The training recipe is genuinely new, but the reported ARC-Challenge numbers disagree across tables—the empirical core needs fixing before the claims can be trusted. read the letter →

arxiv 2508.11995 v1 pith:X7ITUCFD submitted 2025-08-16 cs.AI cs.MA

Xuyang Zhao , Shiwan Zhao , Hualong Yu , Liting Zhang , Qicheng Li This is my paper · ORCID

classification cs.AIcs.MA

keywords multi-agentsystemscollaborativedecision-makingAnalysisofCompetingHypothesesreinforcementlearningGRPOLLMreasoningMMLU-Prostructured

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the final step of an LLM multi-agent system—the step that turns several candidate answers into one output—is itself trainable, and that a 7B model trained for that role can outperform voting rules, untrained arbiters, and even stronger single models. The training recipe borrows the Analysis of Competing Hypotheses (ACH) protocol from cognitive science: propose hypotheses, log evidence, build a hypothesis-evidence matrix, then deliberately try to falsify the leading answer before reporting. AgentCDM first rewards strict adherence to that protocol, then rewards semantic similarity to it while fading the scaffold, so the model internalizes the reasoning instead of parroting it. On MMLU, MMLU-Pro, and ARC-Challenge the reported accuracy gains over single agents reach +29.3 points on MMLU-Pro, and a model trained on MMLU-Pro transfers to other datasets. If the result holds, multi-agent system design should focus as much on how decisions are made as on how agents talk to each other.

What carries the argument

The load-bearing mechanism is the ACH decision protocol plus the two-stage reinforcement-learning schedule that internalizes it. Stage 1 trains with a seven-step scaffold—enumerate hypotheses from each agent's answer, extract tagged evidence, fill a hypothesis-evidence matrix with consistent/inconsistent/irrelevant marks, refine the matrix, pick the hypothesis with the fewest inconsistencies, challenge it with the strongest counter-argument, and produce a final analytic report—while a pattern-matching reward verifies the presence of those steps inside <think> tags. Stage 2 replaces strict pattern matching with a semantic-similarity reward computed from embedding cosine similarity using the BGE-M3 model, and uses a cosine annealing schedule to sample from 'full ACH' versus 'simplified ACH' prompts, gradually withdrawing the scaffold so the model learns to reason this way unprompted. The optimizer is GRPO with a composite reward of format, answer accuracy, and ACH adherence in both stages.

What would settle it

Run the identical two-stage training with the evaluation split explicitly quarantined and documented; if MMLU-Pro accuracy drops from 65.2 toward the ACH-protocol-only level (58.0) or the +29.3 gain over single-agent shrinks to single digits, then the headline gains are training-set leakage rather than learned reasoning.

Watch

Extended reading notes

Core claim

The central claim, on the paper's own terms, is that replacing an unstructured 'decider' prompt with a learned ACH-structured decision policy causes a large, consistent jump in collaborative decision accuracy. The trained decision agent receives the query plus the execution agents' answers, runs through hypothesis generation and evidence evaluation internally, and returns a final answer; the composite rule-based reward (format plus answer correctness plus ACH adherence) supplies all learning signal, so no human-annotated reasoning traces are used in training. The most striking reported results are on MMLU-Pro, where AgentCDM improves average accuracy from 47.9 single-agent to 65.2, and cross-dataset transfer where training on MMLU-Pro yields 80.8 on MMLU and 94.0 on ARC-Challenge, higher than training on those datasets directly. The ablation attributes this to the two stages working together: scaffolding-only collapses, exploration-only plateaus, full model wins.

Load-bearing premise

The training prompts for the main experiments are assumed to come from held-out training splits of MMLU, MMLU-Pro, and ARC-Challenge, but the paper never states the split and only says "previously unseen test sets" in the cross-dataset section.

Editorial extensions

If this is right

Collaborative decision-making in MAS can be improved by training the decision agent itself, not just by choosing a better aggregation rule or prompting strategy.
A single decision agent trained on one hard benchmark can transfer to other benchmarks, which would let system builders train decision policies once on a deliberately difficult, disagreement-rich dataset and deploy them elsewhere.
The two-stage schedule is load-bearing: skipping either stage measurably hurts, suggesting that scaffold-then-anneal is a general recipe for internalizing structured reasoning protocols.
The scaling result implies that adding more execution agents is not automatically helpful; system designers may need to tune agent count to agent capability, or let the decision agent handle heterogeneous noisy inputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension the paper leaves open is to make the decision agent also choose how many execution agents to consult; the scalability results suggest this could improve weak-executor settings. This is an editorial inference, not a paper claim.
The transfer result suggests that the difficulty of training data, not just its domain coverage, drives the learned decision skill; a direct test would be to train on a deliberately disagreement-rich subset of MMLU-Pro rather than the full set. This is an extrapolation, not reported.
The Stage-1 pattern-matching ACH reward may be doing double duty as a format regularizer that reduces output variance; comparing against a format-only reward would isolate how much of the gain is structural reasoning versus output stability. This is an inference the paper does not test.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the one thing to know: AgentCDM is a real idea, but the paper currently cannot support its own headline numbers. The same claimed configuration gets 88.4 on ARC-Challenge in Table 1, 89.5 in Table 2, 93.0 in Table 3, and 96.0 in Table 4, with no disclosed difference in setup. Since Table 1 anchors the SOTA claim, Table 2 the generalization claim, and Table 4 the two-stage synergy claim, at least one of the three central empirical pillars is not represented by a reproducible number. That is the first thing I'd fix.

What's actually new: applying ACH to the decision agent in an LLM MAS, and especially the two-stage RL curriculum—explicit ACH scaffolding with pattern-matching rewards, then a soft ACH reward based on embedding similarity plus curriculum annealing that removes the scaffold. That is a sensible adaptation of DeepSeek-R1-style rule-based RL to the multi-agent setting, and I don't know of prior work doing exactly this. The control comparison that matters—AgentCDM vs Qwen-7B-R1 under the same ACH prompt—shows consistent gains, which is evidence the training, not just the prompt, is doing something. The cross-dataset transfer experiment is also a good design, and the conclusion that harder training data transfers better is interesting.

Soft spots in proportion: the inconsistent ARC numbers are load-bearing. The train/test split for the RL training is never stated; only the cross-dataset section mentions 'previously unseen test sets.' If accuracy rewards were computed on prompts later reported as test, the gains are fitting artifacts. Single runs, no variance or significance tests, no code or data release. These are fixable by disclosure and a few repeated seeds. The ACH-reward circularity worry is less serious than it looks, because the accuracy reward is external; the soft ACH reward mainly shapes the text to resemble the protocol, which would matter more if there were no accuracy signal.

Who this is for: anyone working on aggregation in LLM-based MAS, and people interested in RL curricula for reasoning. The method section is worth reading. But the experimental claims as written are not verifiable, so I wouldn't cite the numbers yet.

Recommendation: send it to peer review, but with a clear request to reconcile the ARC scores, state the exact train/test split, and add variance estimates. The idea is worth referee time; the current reporting isn't.

Referee Report

5 major / 5 minor

Summary. The paper proposes AgentCDM, a framework for collaborative decision-making in LLM-based multi-agent systems. A decision agent is trained in two reinforcement-learning stages: Stage 1 uses an explicit ACH-inspired reasoning protocol as a scaffold, rewarded by format, accuracy, and pattern-matched ACH adherence; Stage 2 replaces the explicit protocol with a curriculum over full and simplified ACH prompts and adds a soft ACH reward based on BGE-M3 cosine similarity. The trained Qwen-7B decision agent is evaluated on MMLU, MMLU-Pro, and ARC-Challenge against dictatorial, voting, and ACH-prompted baselines, plus cross-dataset transfer, scalability, and ablation experiments. The paper reports large gains, e.g., +29.3 points over a Mistral-7B single-agent baseline on MMLU-Pro, and claims state-of-the-art performance and strong generalization.

Significance. The core idea is timely and the comparisons are mostly well chosen; in particular, comparing AgentCDM against Qwen-7B-R1 under the same ACH protocol is the right control and shows a consistent advantage on the reported averages. If the numbers are reproducible, the two-stage scaffold-then-soften RL curriculum is a useful contribution to multi-agent decision-making, and the cross-dataset transfer results are notable. However, several load-bearing empirical details are missing or internally inconsistent, so the current manuscript does not yet establish the claimed state-of-the-art performance or generalization.

major comments (5)

[Tables 1, 2, and 4] The ARC-Challenge accuracy reported for the same AgentCDM system is internally inconsistent: Table 1's ARC average row gives 88.4, Table 2's diagonal entry (training on ARC-Challenge, evaluating on ARC-Challenge) gives 89.5, and Table 4's 'Full' row, defined as the complete two-stage AgentCDM framework, gives 96.0. No experimental configuration difference (execution-agent pool, number of agents, prompt variant, or train/eval split) is disclosed that would explain a gap of 6.5–7.6 points. Because Table 1 grounds the state-of-the-art claim, Table 2 grounds the cross-dataset generalization claim, and Table 4 grounds the two-stage synergy claim, the reader cannot tell which number is the true performance; please reconcile the three tables or explicitly label the different configurations.
[Models and Implementation Details / Main Results] The paper never states the train/test split used for the RL training in the main experiments. The implementation paragraph says only that batches sample P=256 prompts, and the phrase 'previously unseen test sets' appears only in the cross-dataset section. If the accuracy reward in Eqs. (3) and (6) is computed on the same MMLU/MMLU-Pro/ARC-Challenge items later reported as evaluation, the reported gains are fitting artifacts. Please state explicitly which split was used for training, validation (if any), and testing, and confirm that no test-set item contributed to reward computation.
[Main Results (Table 1)] The blanket claim that AgentCDM 'consistently and significantly outperforms all baselines' is contradicted by the table's own GPT-4 rows: on MMLU, Informed Dictatorial scores 85.7 versus AgentCDM's 85.6, and on MMLU-Pro, Informed Dictatorial scores 71.2 versus AgentCDM's 71.0. The claim should be restricted to aggregate/average performance or to non-GPT-4 execution pools, or the rows should be recomputed and the text corrected.
[Stage Two (Eq. 6) and Ablation Study (Table 4)] Stage 2 is described as 'Scaffolding Removal and Autonomous Exploration,' but the Stage-2 reward still contains R_softACH, which explicitly rewards similarity to the authors' own ACH protocol. Consequently, the ablation label 'Exploration-Only' is not unguided exploration, and the conclusion that unguided exploration is inefficient is not supported. Please clarify what is actually removed in Stage 2, and consider an additional ablation without any ACH-shaped reward to substantiate the autonomy claim.
[Main Results and Ablation Study (Tables 1 and 4)] No variance estimates, number of seeds, or significance tests are reported; all tables give single-run accuracy values. The text repeatedly says AgentCDM 'significantly outperforms' baselines, but several comparisons are within one point (e.g., Table 1, GPT-4 MMLU row), and the claim of statistical significance is unsubstantiated. Please report standard deviations across runs/seeds and, where appropriate, significance tests, or soften the wording to describe the observed run-level improvements.

minor comments (5)

[Robustness and Scalability Analysis (Figure 4)] The scalability plot has no error bars or per-condition sample sizes, so the claimed 'dual phenomenon' may be partly noise; please add variability information or state the number of queries per point.
[Appendix references] The text refers to Appendices A, B, and C for implementation and benchmark details, but no appendix content appears in the manuscript; please include these sections or remove the references.
[Robustness and Scalability Analysis (Table 3)] Table 3 reports only two conditions and does not describe how the three random outputs were sampled from the heterogeneous pool or how many queries were used; please add these details.
[References] The BGE-M3 citation is given as 'Multi-Granularity 2024'; please use the proper bibliographic entry for the M3-Embedding paper.
[Abstract and Introduction] The phrase 'state-of-the-art' is stronger than the comparison set supports, since the baselines are limited to the GEDI configurations and no comparison to other recently proposed MAS decision or aggregation methods is included; consider qualifying the claim.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: ACH training rewards are not predictions, and the GEDI baseline citation is not load-bearing.

full rationale

The paper's central empirical claim is that AgentCDM improves accuracy on MMLU, MMLU-Pro, and ARC-Challenge. This is tested by external accuracy rewards against ground-truth labels, so the reported gains are not equivalent to the ACH reward by construction. The ACH and soft-ACH rewards are training signals that condition the model's output process, but the dependent variable reported in Tables 1, 2, 3, and 4 is accuracy, which is independently measurable. The concern that the ACH reward is self-referential does not meet the required standard: the model is trained to follow a protocol and then its decisions are scored by accuracy; the protocol is the treatment, not the evaluation metric. The only apparent self-citation is to GEDI (Zhao, Wang, and Peng 2024) for baseline configurations; this is an experimental-setup citation, not a load-bearing theorem, uniqueness claim, or fitted parameter. The cross-dataset generalization results are empirical transfer numbers, not definitions. The three different ARC-Challenge values in Tables 1, 2, and 4 are a reproducibility/consistency concern for the experimental section, but inconsistency is not circularity and does not show that any prediction reduces to its input. No equation in the paper defines an output in terms of the quantity it claims to predict, and no 'prediction' is statistically forced by fitting a parameter to the same data used for evaluation. Under the hard-rule standard of quoting a specific reduction, no circular step can be exhibited; the derivation chain is empirical and self-contained apart from minor experimental-citation overlap.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of using ACH-style structured reasoning as a reward signal, the embedding-similarity proxy in Stage 2, the quality and diversity of execution agents' outputs, and an implicit clean train/test split. No numerical free parameters are reported, and no new entities are postulated. The improvement is measured against external accuracy labels, so the core comparison is not definitional, but the protocol's benefits are partly baked into the reward design.

assumptions (4)

domain assumption ACH-style structured reasoning reduces cognitive bias in LLM decision-makers.
Stated in the Introduction (Section 1) and used to justify the method; no prior LLM-specific evidence is cited.
domain assumption Cosine similarity between BGE-M3 embeddings of the agent's thought and the ACH protocol is a valid soft reward for reasoning quality.
Stage 2 (Equation 6) uses this reward without validating the proxy.
domain assumption The RL training data for each benchmark come from the training split, so in-domain test results are held-out.
Not explicitly stated; the cross-dataset section implies 'previously unseen test sets' are used only there.
domain assumption Execution agents produce sufficiently accurate and diverse candidate answers for hypothesis evaluation to be meaningful.
The authors themselves list this as a limitation in the Conclusion.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AgentCDM: Enhancing Multi-Agent Collaborative Decision-Making via ACH-Inspired Structured Reasoning." pith.science (2026). https://pith.science/paper/X7ITUCFD

@misc{pith2026250811995,
  author       = {Pith},
  title        = {Pith review of: AgentCDM: Enhancing Multi-Agent Collaborative Decision-Making via ACH-Inspired Structured Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X7ITUCFD}},
  note         = {Machine review of arXiv:2508.11995}
}

read the original abstract

Multi-agent systems (MAS) powered by large language models (LLMs) hold significant promise for solving complex decision-making tasks. However, the core process of collaborative decision-making (CDM) within these systems remains underexplored. Existing approaches often rely on either ``dictatorial" strategies that are vulnerable to the cognitive biases of a single agent, or ``voting-based" methods that fail to fully harness collective intelligence. To address these limitations, we propose \textbf{AgentCDM}, a structured framework for enhancing collaborative decision-making in LLM-based multi-agent systems. Drawing inspiration from the Analysis of Competing Hypotheses (ACH) in cognitive science, AgentCDM introduces a structured reasoning paradigm that systematically mitigates cognitive biases and shifts decision-making from passive answer selection to active hypothesis evaluation and construction. To internalize this reasoning process, we develop a two-stage training paradigm: the first stage uses explicit ACH-inspired scaffolding to guide the model through structured reasoning, while the second stage progressively removes this scaffolding to encourage autonomous generalization. Experiments on multiple benchmark datasets demonstrate that AgentCDM achieves state-of-the-art performance and exhibits strong generalization, validating its effectiveness in improving the quality and robustness of collaborative decisions in MAS.

Figures

Figures reproduced from arXiv: 2508.11995 by the authors.

**Figure 1.** Comparison of collaborative decision-making strategies: (a) Voting-based methods fails due to a lack of consensus [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

**Figure 2.** The AgentCDM framework. In the first stage, the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Decision-Making Process Based on Analysis of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Effect of agent quantity on MMLU-PRO perfor [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 13 canonical work pages

[1]

L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023
[2]

Bai, J.; Bai, S.; Chu, Y.; Cui, Z.; Dang, K.; Deng, X.; Fan, Y.; Ge, W.; Han, Y.; Huang, F.; et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

arXiv 2023
[3]

Bose, T.; Reina, A.; and Marshall, J. A. 2017. Collective decision-making. Current opinion in behavioral sciences, 16: 30--34

work page 2017
[4]

J.; Kilgour, D

Brams, S. J.; Kilgour, D. M.; and Sanver, M. R. 2007. A minimax procedure for electing committees. Public Choice, 132(3): 401--420

work page 2007
[5]

Chen, Z.; Liu, K.; Wang, Q.; Liu, J.; Zhang, W.; Chen, K.; and Zhao, F. 2024. Mindsearch: Mimicking human minds elicits deep ai searcher. arXiv preprint arXiv:2407.20183

arXiv 2024
[6]

Chu, X.; Huang, H.; Zhang, X.; Wei, F.; and Wang, Y. 2025. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02546

arXiv 2025
[7]

Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

arXiv 2018
[8]

Davies, J.; Katsirelos, G.; Narodytska, N.; Walsh, T.; and Xia, L. 2014. Complexity of and algorithms for the manipulation of Borda, Nanson's and Baldwin's voting rules. Artificial Intelligence, 217: 20--42

work page 2014

Show all 41 references

[9]

B.; and Mordatch, I

Du, Y.; Li, S.; Torralba, A.; Tenenbaum, J. B.; and Mordatch, I. 2023. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning

2023
[10]

Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The llama 3 herd of models. arXiv e-prints, arXiv--2407

2024
[11]

Emerson, P. 2013. The original Borda count and partial voting. Social Choice and Welfare, 40(2): 353--358

2013
[12]

R.; Rothe, J.; and Schend, L

Erd \'e lyi, G.; Fellows, M. R.; Rothe, J.; and Schend, L. 2015. Control complexity in Bucklin and fallback voting: A theoretical analysis. Journal of Computer and System Sciences, 81(4): 632--660

2015
[13]

Freeman, R.; Brill, M.; and Conitzer, V. 2014. On the axiomatic characterization of runoff voting rules. In Proceedings of the AAAI conference on artificial intelligence, volume 28

2014
[14]

GLM, T.; Zeng, A.; Xu, B.; Wang, B.; Zhang, C.; Yin, D.; Zhang, D.; Rojas, D.; Feng, G.; Zhao, H.; et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

2024 arXiv
[15]

Grattafiori, A.; Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Vaughan, A.; et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

2024 arXiv
[16]

Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; Zhu, Q.; Ma, S.; Wang, P.; Bi, X.; et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

2025 arXiv
[17]

U.; Qureshi, R.; Shah, A.; Irfan, M.; Zafar, A.; Shaikh, M

Hadi, M. U.; Qureshi, R.; Shah, A.; Irfan, M.; Zafar, A.; Shaikh, M. B.; Akhtar, N.; Wu, J.; Mirjalili, S.; et al. 2023. Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects. Authorea Preprints, 1: 1--26

2023
[18]

Hao, R.; Hu, L.; Qi, W.; Wu, Q.; Zhang, Y.; and Nie, L. 2025. Chatllm network: More brains, more intelligence. AI Open, 6: 45--52

2025
[19]

Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

2020 arXiv
[20]

Heuer, R. J. 1999. Psychology of intelligence analysis. Center for the Study of Intelligence

1999
[21]

Hong, S.; Zheng, X.; Chen, J.; Cheng, Y.; Wang, J.; Zhang, C.; Wang, Z.; Yau, S. K. S.; Lin, Z.; Zhou, L.; et al. 2023. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 3(4): 6

2023 arXiv
[22]

M.; Luck, M.; Bu, Q.; Qing, Y.; and Cui, H

Huang, D.; Zhang, J. M.; Luck, M.; Bu, Q.; Qing, Y.; and Cui, H. 2023. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010

2023 arXiv
[23]

Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D

Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; de las Casas, D.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; Lavaud, L. R.; Lachaux, M.-A.; Stock, P.; Scao, T. L.; Lavril, T.; Wang, T.; Lacroix, T.; and Sayed, W. E. 2023. Mistral 7B. arXiv:2310.06825

2023 arXiv
[24]

J.; and Cowlishaw, G

King, A. J.; and Cowlishaw, G. 2007. When to use social information: the advantage of large group size in individual decision making. Biology letters, 3(2): 137--139

2007
[25]

Li, G.; Hammoud, H.; Itani, H.; Khizbullin, D.; and Ghanem, B. 2023. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36: 51991--52008

2023
[26]

Li, J.; Zhang, Q.; Yu, Y.; Fu, Q.; and Ye, D. 2024. More agents is all you need. arXiv preprint arXiv:2402.05120

2024 arXiv
[27]

Liang, T.; He, Z.; Jiao, W.; Wang, X.; Wang, Y.; Wang, R.; Yang, Y.; Shi, S.; and Tu, Z. 2023. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118

2023 arXiv
[28]

Y.; Fu, Y.; Yang, K.; Brahman, F.; Huang, S.; Bhagavatula, C.; Ammanabrolu, P.; Choi, Y.; and Ren, X

Lin, B. Y.; Fu, Y.; Yang, K.; Brahman, F.; Huang, S.; Bhagavatula, C.; Ammanabrolu, P.; Choi, Y.; and Ren, X. 2023. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. Advances in Neural Information Processing Systems, 36: 23813--23825

2023
[29]

S.; and Lin, M

Liu, Z.; Chen, C.; Li, W.; Qi, P.; Pang, T.; Du, C.; Lee, W. S.; and Lin, M. 2025. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783

2025 arXiv
[30]

T.; Foerster, J.; Clune, J.; and Ha, D

Lu, C.; Lu, C.; Lange, R. T.; Foerster, J.; Clune, J.; and Ha, D. 2024. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292

2024 arXiv
[31]

W.; Iyyer, M.; Zettlemoyer, L.; and Hajishirzi, H

Min, S.; Krishna, K.; Lyu, X.; Lewis, M.; Yih, W.-t.; Koh, P. W.; Iyyer, M.; Zettlemoyer, L.; and Hajishirzi, H. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251

2023 arXiv
[32]

Multi-Granularity, M.-L. M.-F. 2024. M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation

2024
[33]

Shen, W.; Li, C.; Chen, H.; Yan, M.; Quan, X.; Chen, H.; Zhang, J.; and Huang, F. 2024. Small llms are weak tool learners: A multi-llm agent. arXiv preprint arXiv:2401.07324

2024 arXiv
[34]

Team, Q. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

2024 arXiv
[35]

Wang, Y.; Ma, X.; Zhang, G.; Ni, Y.; Chandra, A.; Guo, S.; Ren, W.; Arulraj, A.; He, X.; Jiang, Z.; et al. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37: 95266--95290

2024
[36]

Wolf, Y.; Wies, N.; Avnery, O.; Levine, Y.; and Shashua, A. 2023. Fundamental limitations of alignment in large language models. arXiv preprint arXiv:2304.11082

2023 arXiv
[37]

Wu, Q.; Bansal, G.; Zhang, J.; Wu, Y.; Li, B.; Zhu, E.; Jiang, L.; Zhang, X.; Zhang, S.; Liu, J.; et al. 2024. Autogen: Enabling next-gen LLM applications via multi-agent conversations. In First Conference on Language Modeling

2024
[38]

Xiong, W.; Yao, J.; Xu, Y.; Pang, B.; Wang, L.; Sahoo, D.; Li, J.; Jiang, N.; Zhang, T.; Xiong, C.; et al. 2025. A minimalist approach to llm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504.11343

2025 arXiv
[39]

Xu, Z.; Shi, S.; Hu, B.; Yu, J.; Li, D.; Zhang, M.; and Wu, Y. 2023. Towards reasoning in large language models via multi-agent peer review collaboration. arXiv preprint arXiv:2311.08152

2023 arXiv
[40]

Yu, Q.; Zhang, Z.; Zhu, R.; Yuan, Y.; Zuo, X.; Yue, Y.; Dai, W.; Fan, T.; Liu, G.; Liu, L.; et al. 2025. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476

2025 arXiv
[41]

Zhao, X.; Wang, K.; and Peng, W. 2024. An electoral approach to diversify llm-based multi-agent collective decision-making. arXiv preprint arXiv:2410.15168

2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023

[2] [2]

Bai, J.; Bai, S.; Chu, Y.; Cui, Z.; Dang, K.; Deng, X.; Fan, Y.; Ge, W.; Han, Y.; Huang, F.; et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

arXiv 2023

[3] [3]

Bose, T.; Reina, A.; and Marshall, J. A. 2017. Collective decision-making. Current opinion in behavioral sciences, 16: 30--34

work page 2017

[4] [4]

J.; Kilgour, D

Brams, S. J.; Kilgour, D. M.; and Sanver, M. R. 2007. A minimax procedure for electing committees. Public Choice, 132(3): 401--420

work page 2007

[5] [5]

Chen, Z.; Liu, K.; Wang, Q.; Liu, J.; Zhang, W.; Chen, K.; and Zhao, F. 2024. Mindsearch: Mimicking human minds elicits deep ai searcher. arXiv preprint arXiv:2407.20183

arXiv 2024

[6] [6]

Chu, X.; Huang, H.; Zhang, X.; Wei, F.; and Wang, Y. 2025. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02546

arXiv 2025

[7] [7]

Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

arXiv 2018

[8] [8]

Davies, J.; Katsirelos, G.; Narodytska, N.; Walsh, T.; and Xia, L. 2014. Complexity of and algorithms for the manipulation of Borda, Nanson's and Baldwin's voting rules. Artificial Intelligence, 217: 20--42

work page 2014