Pith. sign in

REVIEW 3 major objections 5 minor 8 cited by

Exploring Advanced LLM Multi-Agent Systems Based on Blackboard Architecture

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that a blackboard-based LLM multi-agent system, whose agents share a workspace and are selected from its current contents, matches or beats fixed and workflow-searched baselines in accuracy while spending fewer tokens.

desk verdict First LLM-native blackboard architecture with dynamic per-round agent selection is a real contribution; the reported accuracy edge is not yet supported due to single unseeded runs and test-set tuning. read the letter →

arxiv 2507.01701 v1 pith:SPQA6K6T submitted 2025-07-02 cs.MA cs.AI

Bochen Han , Songmao Zhang This is my paper
classification cs.MAcs.AI
keywords LLMmulti-agentsystemsblackboardarchitecturedynamicagentselectiontokenefficiencyconsensusreasoningbenchmarksmathematicalgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes reviving the blackboard architecture—the 1980s expert-system design in which independent specialists write to a shared workspace and a control unit decides who acts next—for LLM multi-agent systems, and it implements that proposal as LbMAS, which it calls the first blackboard-based LLM multi-agent system. The central claim is that a dynamic cycle of blackboard reading, writing, and agent selection is competitive with both fixed and autonomously searched multi-agent systems while consuming fewer tokens. Across six knowledge, reasoning, and mathematical benchmarks, LbMAS reports the best average accuracy, 81.68, against 80.56 for the strongest static baseline, Chateval, and on MATH it spends 4,721,489 total tokens versus 5,449,425 for the cheapest autonomous baseline. The authors argue the advantage comes from the blackboard replacing per-agent memory, from selecting only currently useful agents, and from avoiding a separate supervised workflow-search step. If the claim holds, a decades-old coordination scheme can make LLM agent teams cheaper and more adaptive on tasks where no fixed workflow exists.

What carries the argument

The load-bearing mechanism is the blackboard cycle, organized around three components: the agent group, the blackboard, and an LLM-based control unit. The blackboard is split into a public space, which every agent reads and writes, and private spaces for debate and self-reflection; because all historical context lives there, agents do not need their own memory modules. The control unit is an LLM that takes the query, the current blackboard contents, and the agent role descriptions and outputs which agents run next. Around this cycle sit constant agents—planner, decider, critic, cleaner, and conflict-resolver—plus generated expert agents, with the cleaner deleting redundant messages to save tokens and the conflict-resolver sending disagreeing agents into the private space to reconcile. This mechanism is what makes collaboration structure dynamic in the moment rather than fixed in advance.

What would settle it

Rerun LbMAS and its main baselines multiple times with different random seeds on MMLU and MATH, recording accuracy and token counts per run; if the spread across seeds for LbMAS is comparable to or larger than the reported 1.12-point gap over Chateval, the competitiveness claim is not established. The paper's own ablations and consensus proportions would also be checkable in the same runs.

Watch

Extended reading notes

Core claim

The paper's central discovery, stated on its own terms, is that the blackboard architecture transfers intact to LLM multi-agent systems and buys both accuracy and token efficiency. LbMAS generates query-related expert agents on the fly, adds them to constant role agents (planner, decider, critic, cleaner, conflict-resolver), and assigns each agent a base LLM chosen randomly from Llama-3.1-70b-Instruct and Qwen-2.5-72b-Instruct. All agents read the full current blackboard and append their outputs; an LLM control unit then selects the next agents from the current blackboard contents, and the loop runs until the decider judges the messages sufficient or a round limit is reached, with majority vote as a fallback. The system obtains the best average accuracy among all compared methods on six benchmarks, 81.68, and on MATH uses the second-lowest total tokens among static and autonomous baselines, while avoiding the training-data-dependent search step that autonomous baselines require. The authors conclude that shared blackboard context plus dynamic on-the-fly agent selection is a viable alternative to fixed and pre-searched collaboration mechanisms.

Load-bearing premise

The load-bearing premise is that a single unseeded run of a stochastic system measures its true performance: agents are randomly generated, each picks one of two base LLMs at random, and the LLM control unit makes probabilistic choices, so the reported 1.12-point average lead over Chateval and the token savings could be within run-to-run noise.

Editorial extensions

If this is right

  • If the measured results hold, a MAS can match or beat fixed collaboration graphs without ever specifying who talks to whom in advance; the blackboard state itself drives agent selection.
  • Token budgets improve because only selected agents execute, redundant messages are removed, and the expensive supervised workflow-search step of autonomous MASs is eliminated.
  • Decider-based stopping and majority-vote stopping perform almost identically, so a consensus signal on the blackboard is a reliable stopping condition in most cases.
  • Because the agent group is open-ended, new roles such as tool-using or code-writing agents can be added without redesigning the blackboard or control unit, which is the paper's stated path for scaling the framework.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One stress test the paper leaves implicit is repeated seeding: agent identities, per-agent base LLM choices, and control-unit selections are all stochastic, so the single reported run is one draw from a distribution whose width is not yet characterized.
  • If the shared-blackboard context—not just the dynamic selection—is the main driver, then simpler scheduling rules, even a fixed round-robin over the same agent group, might capture much of the accuracy at lower engineering cost; the paper's ablation shows removing the control unit costs a large token increase but only a small accuracy change.
  • The public/private space split is a reusable conflict-resolution primitive: it could be transplanted into non-blackboard MAS architectures as a standard way to quarantine disagreements before final voting.
  • Extending LbMAS to domains with tool calls or long horizons would test whether removing per-agent memory changes the scaling behavior under tight token budgets, a consequence the paper names only as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes bMAS, a general blackboard architecture for LLM multi-agent systems, and LbMAS, a concrete implementation. LbMAS generates query-specific expert agents, maintains a public and private blackboard, selects agents with an LLM control unit, and repeats the cycle until a decider stops it or a maximum round K is reached; the final answer is chosen either by the decider or by cumulative similarity voting. The authors evaluate on MMLU, ARC-Challenge, GPQA-Diamond, BBH, MATH, and GSM8K, reporting best average accuracy (81.68) and lower token use than several static and autonomous baselines, and they provide ablations of K, the control unit, and the cleaner. The paper is an architecture proposal with empirical claims about SOTA competitiveness and token efficiency.

Significance. The proposed direction is timely and plausible: replacing fixed or search-optimized workflows with a blackboard where an LLM control unit selects agents on the fly could be a useful design point, and the paper's token accounting (Table 3) and ablations (control unit and cleaner) are valuable and informative. The paper also makes a falsifiable claim (best average performance at lower token cost) and provides detailed prompts and dataset statistics in the appendix. However, the current empirical support is insufficient: the reported superiority rests on single unseeded runs of a deliberately stochastic system, and the maximum-round hyperparameter is selected on the same test sets. These are fixable with additional experiments, and the framework itself remains a reasonable contribution.

major comments (3)
  1. [4.1, Table 1] The headline claim of 'best average performance' is not supported by the reported experiments. The system is stochastic by construction: agent generation randomly draws 1–3 expert roles (Eq. 1), each new agent randomly selects one of two base LLMs (Section 4.1), and all LLM calls use temperature 0.7. Yet Table 1 reports a single run with no standard deviations, confidence intervals, or significance tests. The average lead over Chateval is only 1.12 percentage points (81.68 vs 80.56), and for GPQA-Diamond (198 examples) a handful of questions accounts for the gap. Please run multiple seeds (at least 5, ideally more on GPQA-Diamond/MATH) and report mean ± std, or provide a paired significance test; if variance is large, the claim should be downgraded to 'competitive' rather than 'best.' The Limitations section also does not mention this missing repeated-trial analysis, which is the most load-bearing gap for the SOTA claim.
  2. [4.5, Table 4] The maximum blackboard cycle K=4 is selected using the same test benchmarks on which the main results are reported. Section 4.5 tests K=2, 4, 6, 8 on MMLU, GPQA, and MATH and then reports Table 1 with K=4, so the reported numbers are partially tuned on the test sets. The justification is also internally inconsistent: the text gives average rounds of 2.88, 3.05, 3.29 for MMLU, GPQA, MATH, while the K=4 row of Table 4 shows 2.88, 3.29, 3.04; and K=8 gives a higher MATH accuracy (74.80) than K=4 (72.60). Use a validation split or pre-specify K before seeing test results, and report the chosen setting's test performance separately from the hyperparameter sweep.
  3. [3.2, final answer selection] The cumulative-similarity voting rule V(ai)=Σ_{j≠i} sim(ai,aj) is not fully specified: the similarity function sim is never defined, and the handling of ties or of cases where all answers are dissimilar is not described. Because this rule determines the final answer whenever the decider does not stop the cycle, the method is not reproducible without these details. Please specify sim (e.g., exact-match, token F1, embedding cosine), the tie-breaking rule, and the threshold used to decide whether a consensus exists.
minor comments (5)
  1. [4.5, Table 4] The average-round values in the text (2.88, 3.05, 3.29) do not match the K=4 row of Table 4 (2.88, 3.29, 3.04); please correct the numbers or the table.
  2. [Abstract, Section 4.3] The abstract and conclusion claim that the system 'spend[s] less tokens,' but token costs are reported only for the MATH benchmark (Table 3). Please qualify the claim as applying to the measured benchmark, or provide token-cost tables for the other five datasets.
  3. [Footnote 1] The 'Code will be released soon here' footnote contains no repository URL. Provide a link or a clear statement of availability to support reproducibility.
  4. [Tables 1 and 4] The column header 'MA TH' appears to contain a typo; it should read 'MATH'.
  5. [Section 4.1] The statement that all baseline MASs 'adopted this LLM selection way' needs more detail: provide the exact modification of each baseline (e.g., how agent-to-LLM assignment was randomized) in the appendix, since it affects the fairness of the comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LbMAS's claims rest on benchmark measurements and ablations, not on derivations that reduce to their own inputs.

full rationale

LbMAS is an empirical systems paper: its central claims, competitive with SOTA static/dynamic MASs and lower token cost, are supported by direct benchmark measurements (Tables 1 and 3) and ablations (Tables 4 and 5), not by a mathematical derivation that could be equivalent to its inputs. The blackboard cycle, control-unit selection (Eq. 2), and cumulative-similarity voting are defined independently of the reported outcomes; no fitted parameter is renamed as a prediction. The one self-referential design choice is K=4, justified in Sec. 4.5 by observed average rounds on MMLU/GPQA/MATH; this is post-hoc test-set tuning and a validity weakness, but it is not circular under the stated criteria because the reported accuracies are not defined as a function of K and Table 1 is presented as measurement, not as a prediction derived from K. The stochastic agent-generation and base-LLM choice (Sec. 3.1, Sec. 4.1) with no repeated seeds is a reproducibility/correctness gap, not a circularity. No load-bearing argument rests on self-citations; external citations (e.g., Zhang et al. 2025c for model diversity, classic blackboard references) are independent support. The Limitations section acknowledges benchmark breadth and agent-generation simplicity but not the absence of repeated trials; that is weighed as a correctness risk rather than a circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claims are empirical rather than derived. The main free choices are the manually selected LLM pair, the stochastic temperature, and K tuned on test data. The key domain axioms are that role-prompted LLMs cooperate through a shared blackboard and that random model assignment is a fair and beneficial design. No new physical entities are introduced; the private blackboard space is the only novel system component without isolated evidence.

free parameters (3)
  • Maximum blackboard cycle K = 4
    Chosen after testing K=2,4,6,8 on the same MMLU, GPQA, and MATH test sets (Table 4). The authors state 'This is why we set the maximum round number to 4', but MATH accuracy is actually highest at K=8, indicating post hoc selection.
  • Base LLM set = Llama-3.1-70b-Instruct; Qwen-2.5-72b-Instruct
    Manually selected pair of models used for all agents, with random assignment per agent. Table 2 shows results differ substantially by model, so this choice affects the reported outcomes.
  • Inference temperature = 0.7
    Set in Section 4.1 for all models; a standard stochastic sampling choice that interacts with the absence of seeded runs.
assumptions (4)
  • domain assumption LLM agents can reliably follow role prompts and use the full blackboard context to produce useful contributions.
    Every experimental result depends on this; the prompts in Appendix A.3 are the only mechanism enforcing role behavior.
  • domain assumption Randomly assigning one of two base LLMs to each new agent produces fair comparisons and beneficial diversity.
    Section 3.1 cites Zhang et al. 2025c for the diversity claim, but the paper does not control for random assignment variance or report repeated runs.
  • ad hoc to paper The answer with maximum cumulative similarity to other agents' answers is the best final answer.
    Section 3.2 defines V(ai) and uses argmax as the final selection rule, but no experimental evidence validates this consensus metric over simpler alternatives.
  • ad hoc to paper Hyperparameters tuned on the same test benchmarks generalize to unseen problems.
    The maximum round K and the decision to use both decider and majority vote are selected using the evaluation datasets, so the configuration is not independent of the test set.
invented entities (1)
  • Private blackboard space for debate and self-reflection
    purpose: Allows conflicting agents to discuss or revise their views without writing directly to the public blackboard; used by conflict resolver workflows.
    The private space appears in the case study and architecture diagram, but no ablation isolates its contribution, so its effectiveness is not independently measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Advanced LLM Multi-Agent Systems Based on Blackboard Architecture." pith.science (2026). https://pith.science/paper/SPQA6K6T

@misc{pith2026250701701,
  author       = {Pith},
  title        = {Pith review of: Exploring Advanced LLM Multi-Agent Systems Based on Blackboard Architecture},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SPQA6K6T}},
  note         = {Machine review of arXiv:2507.01701}
}
read the original abstract

In this paper, we propose to incorporate the blackboard architecture into LLM multi-agent systems (MASs) so that (1) agents with various roles can share all the information and others' messages during the whole problem-solving process, (2) agents that will take actions are selected based on the current content of the blackboard, and (3) the selection and execution round is repeated until a consensus is reached on the blackboard. We develop the first implementation of this proposal and conduct experiments on commonsense knowledge, reasoning and mathematical datasets. The results show that our system can be competitive with the SOTA static and dynamic MASs by achieving the best average performance, and at the same time manage to spend less tokens. Our proposal has the potential to enable complex and dynamic problem-solving where well-defined structures or workflows are unavailable.

Figures

Figures reproduced from arXiv: 2507.01701 by the authors.

Figure 1
Figure 1. The general framework of bMAS is shown in the red dashed box. The illustration of LbMAS is in the [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. The number of queries answered correctly by [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Case study of LbMAS. they are needed. Meanwhile, as a medium for agent interaction, the blackboard can meet various kinds of interaction needs including public vs. private space, flat vs. hierarchical structure, and so on. Why does the blackboard-based LLM multi￾agent system work? Compared with the SOTA static and autonomous MAS methods, we analyze that the performance improvement and cost sav￾ing of LbMAS can come … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Agent prompt 13 [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Agent-generation prompt Control Unit Prompt Your task is to schedule other agents to cooperate and solve the given problem. The agent names and descriptions are listed below:\n {role_list}. The given problem is:{question}. Agents are sharing information on the blackboa…
Figure 6
Figure 6. Figure 6: Control unit prompt [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Max number of agents having a consensus in three datasets. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Decentralized Multi-Agent Systems with Shared Context

    cs.MA 2026-06 unverdicted novelty 7.0 of 10

    DeLM decentralizes LLM multi-agent coordination with shared verified context, delivering up to 10.5pp gains on SWE-bench Verified and 5.7pp on LongBench-v2 while cutting cost per task by ~50%.

  2. ARIADNE: Agentic Reward-Informed Adaptive Decision Exploration via Blackboard-Driven MCTS for Competitive Program Generation

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    ARIADNE combines blackboard architecture with MCTS to coordinate strategy, code, test, evaluation, and repair stages, yielding higher Pass@1 scores than prior LLM baselines on APPS, CodeContests, and related benchmarks.

  3. RMA: an Agentic System for Research-Level Mathematical Problems

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    RMA, a multi-agent system with structured memory and iterative feedback loops, solves 8 out of 10 research-level math problems on the new First Proof benchmark and outperforms GPT-5.2R and Aletheia according to expert...

  4. STAR: Failure-Aware Markovian Routing for Multi-Agent Spatiotemporal Reasoning

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    STAR presents a failure-aware routing framework using a state-conditioned transition policy and an agent routing matrix combining expert routes with learned recoveries from execution traces to improve multi-agent spat...

  5. STAR: Failure-Aware Markovian Routing for Multi-Agent Spatiotemporal Reasoning

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    STAR combines expert nominal routes with trace-learned recovery transitions in a failure-typed routing matrix, improving multi-agent spatiotemporal reasoning over baselines especially on error-deviating queries.

  6. STAR: Failure-Aware Markovian Routing for Multi-Agent Spatiotemporal Reasoning

    cs.AI 2026-05 unverdicted novelty 5.0 of 10

    STAR is a failure-aware Markovian router that learns recovery transitions from both successful and unsuccessful execution traces to improve multi-agent performance on spatiotemporal benchmarks.

  7. DySRec: Dynamic Context-Aware Psychometric Scale Recommendation via Multi-Agent Collaboration

    cs.HC 2026-05 unverdicted novelty 5.0 of 10

    DySRec is a multi-agent conversational system that dynamically recommends psychometric scales by integrating user context, behaviors, and risk signals through interactive dialogue and closed-loop refinement.

  8. Towards Cybersecurity SuperIntelligence (CSI): What's the best harness for cybersecurity?

    cs.CR 2026-05 unverdicted novelty 4.0 of 10

    CSI meta-scaffold unifies five LLM agent harnesses; a blackboard multi-agent system solves 19/33 cybench challenges (57.6%) versus 15/33 for the best single scaffold.

Reference graph

Works this paper leans on

66 extracted references · 23 canonical work pages · cited by 6 Pith papers

  1. [1]

    Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. 2025. https://doi.org/10.18653/v1/2025.naacl-long.342 R esearch A gent: Iterative research idea generation over scientific literature with large language models . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistic...

    doi:10.18653/v1/2025.naacl-long.342 2025

  2. [2]

    Marc Cavazza, Steven J Mead, Alexander I Strachan, and Alex Whittaker. 2001. A blackboard system for interpreting agent messages. In Proceedings GameOn 2000: International Conference on Intelligent Games &Simulation

    work page 2001

  3. [3]

    Chateval: Towards better llm-based evaluators through multi-agent debate, 2023

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate, 2023. URL https://arxiv. org/abs/2308.07201, 3

    arXiv 2023

  4. [4]

    Huaben Chen, Wenkang Ji, Lufeng Xu, and Shiyu Zhao. 2025. https://doi.org/10.48550/arXiv.2310.20151 Multi- Agent Consensus Seeking via Large Language Models . arXiv preprint. ArXiv:2310.20151 [cs]

    arXiv doi:10.48550/arxiv.2310.20151 2025

  5. [5]

    Junzhe Chen, Xuming Hu, Shuodi Liu, Shiyu Huang, Wei-Wei Tu, Zhaofeng He, and Lijie Wen. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.705 LLMArena : Assessing Capabilities of Large Language Models in Dynamic Multi - Agent Environments . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers...

    doi:10.18653/v1/2024.acl-long.705 2024

  6. [6]

    Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2024 b . https://openreview.net/forum?id=EHg5GDnyq1 Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors . In The Twelfth Internat...

    work page 2024

  7. [7]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://arxiv.org/abs/1803.05457 Think you have solved question answering? try arc, the ai2 reasoning challenge . Preprint, arXiv:1803.05457

    arXiv 2018

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    arXiv 2021

Show all 66 references
  1. [9]

    Daniel D Corkill. 1991. Blackboard systems

    1991

  2. [10]

    Ayushman Das, Shu-Ching Chen, Mei-Ling Shyu, and Saad Sadiq. 2023. https://doi.org/10.1109/CIC58953.2023.00021 Enabling synergistic knowledge sharing and reasoning in large language models with collaborative multi-agents . In 2023 IEEE 9th International Conference on Collabora...

    2023

  3. [11]

    J. Dong, S. Chen, and J.-J. Jeng. 2005. https://doi.org/10.1109/ITCC.2005.149 Event-based blackboard architecture for multi-agent systems . In International Conference on Information Technology: Coding and Computing (ITCC'05) - Volume II, volume 2, pages 379--384 Vol. 2

    2005 doi

  4. [12]

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning

    2023

  5. [13]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 doi arXiv

  6. [14]

    Ettabaa, I.R

    K.S. Ettabaa, I.R. Farah, B. Solaiman, and M.B. Ahmed. 2006. https://doi.org/10.1109/ICTTA.2006.1684906 Distributed blackboard architecture for multi-spectral image interpretation based on multi-agent system . In 2006 2nd International Conference on Information & Communication...

    2006 arXiv

  7. [15]

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992

    2023 arXiv

  8. [16]

    Barbara Hayes-Roth. 1985. https://doi.org/10.1016/0004-3702(85)90063-3 A blackboard architecture for control . Artificial Intelligence, 26(3):251--321

    1985 doi

  9. [17]

    Zhitao He, Pengfei Cao, Yubo Chen, Kang Liu, Ruopeng Li, Mengshu Sun, and Jun Zhao. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.613 LEGO : A multi-agent collaborative framework with role-playing and iterative feedback for causality explanation generation . In Finding...

    2023 doi

  10. [18]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021 a . https://openreview.net/forum?id=d7KBjmI3GmQ Measuring massive multitask language understanding . In International Conference on Learning Representations

    2021

  11. [19]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 b . https://openreview.net/forum?id=7Bywt2mQsCe Measuring mathematical problem solving with the MATH dataset . In Thirty-fifth Conference on Neural Informa...

    2021

  12. [20]

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and J \"u rgen Schmidhuber. 2024. https://openreview.net/forum?id=VtmBAGCN7o Meta GP...

    2024

  13. [21]

    Shengran Hu, Cong Lu, and Jeff Clune. 2025. https://openreview.net/forum?id=t9U3LW7JVX Automated design of agentic systems . In The Thirteenth International Conference on Learning Representations

    2025

  14. [22]

    Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez

    Md. Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2024. https://doi.org/10.18653/v1/2024.acl-long.269 M ap C oder: Multi-agent code generation for competitive problem solving . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisti...

    2024 doi

  15. [23]

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. https://doi.org/10.18653/v1/2023.acl-long.792 LLM -blender: Ensembling large language models with pairwise ranking and generative fusion . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguis...

    2023 doi

  16. [24]

    junyou li, Qin Zhang, Yangbin Yu, QIANG FU, and Deheng Ye. 2024. https://openreview.net/forum?id=bgzUSZ8aeg More agents is all you need . Transactions on Machine Learning Research

    2024

  17. [25]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan A, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. https://openreview.net/forum?id=sY5N0zY5Od DSP y: Compil...

    2024

  18. [26]

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023 a . Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991--52008

    2023

  19. [27]

    Huao Li, Yu Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Charles Lewis, and Katia Sycara. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-main.13 Theory of mind for multi-agent collaboration via large language models . In Proceedings of the 2023 Conference on Empiric...

    2023 doi

  20. [28]

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.992 Encouraging divergent thinking in large language models through multi-agent debate . In Proceedings of the 2024 ...

    2024 doi

  21. [29]

    Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. 2024 a . A dynamic llm-powered agent network for task-oriented agent collaboration. In First Conference on Language Modeling

    2024

  22. [30]

    Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. 2024 b . https://openreview.net/forum?id=XII0Wp1XA9 A dynamic LLM -powered agent network for task-oriented agent collaboration . In First Conference on Language Modeling

    2024

  23. [31]

    Do Xuan Long, Duong Ngoc Yen, Anh Tuan Luu, Kenji Kawaguchi, Min-Yen Kan, and Nancy F. Chen. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1135 Multi-expert prompting improves reliability, safety and usefulness of large language models . In Proceedings of the 2024 Conferen...

    2024 doi

  24. [32]

    Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, and 1 others. 2025. Towards accurate differential diagnosis with large language models. Nature, pages 1--7

    2025

  25. [33]

    Hana Munira Muhd Mukhtar, Husna Sarirah Husin, Suriana Ismail, Azizah Rahmat, and Roslan Ismail. 2025. https://doi.org/10.1109/IMCOM64595.2025.10857578 Advancing Blackboard Framework : Responsive Trigger Mechanisms and Dynamic Feasibility Conditions for Optimal Knowledge Sourc...

    2025

  26. [34]

    H Penny Nii. 1986. Blackboard systems. Knowledge Systems Laboratory, Depts. of Medical and Computer Science …

    1986

  27. [35]

    Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. 2024. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.07199

    2024 arXiv

  28. [36]

    Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Jiang, Chengfei Lv, and Huajun Chen. 2024. https://doi.org/10.18653/v1/2024.acl-long.165 A uto A ct: Automatic agent learning from scratch for QA via self-planning . In Proceedings of the 62nd Annual ...

    2024 doi

  29. [37]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. https://arxiv.org/abs/2412.15115 Qwe...

    2025 arXiv

  30. [38]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. https://openreview.net/forum?id=Ti67584b98 GPQA : A graduate-level google-proof q&a benchmark . In First Conference on Language Modeling

    2024

  31. [39]

    Darya Rudenko and Arkady Borisov. 2008. Blackboard architecture for product life cycle stage definition. In Proceedings of the 14th International Conference on Soft Computing (Mendel 2008), pages 252--257

    2008

  32. [40]

    Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. 2025 a . https://arxiv.org/abs/2501.04227 Agent laboratory: Using llm agents as research assistants . Preprint, arXiv:2501.04227

    2025 arXiv

  33. [41]

    Samuel Schmidgall, Rojin Ziaei, Carl William Harris, Ji Woong Kim, Eduardo Pontes Reis, Jeffrey K Jopling, and Michael Moor. 2025 b . https://openreview.net/forum?id=ak7r4He1qH Agentclinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments

    2025

  34. [42]

    Yu Shang, Yu Li, Keyu Zhao, Likai Ma, Jiahe Liu, Fengli Xu, and Yong Li. 2025. https://openreview.net/forum?id=mPdmDYIQ7f Agentsquare: Automatic LLM agent search in modular design space . In The Thirteenth International Conference on Learning Representations

    2025

  35. [43]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023. https://openreview.net/forum?id=vAElhFcKW6 Reflexion: language agents with verbal reinforcement learning . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  36. [44]

    Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel Ni, Heung-Yeung Shum, and Jian Guo. 2024. https://openreview.net/forum?id=nnVO1PvbTv Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph . In The Twelfth...

    2024

  37. [45]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. https://doi.org/10.18653/v1/2023.findings-acl.824 Challenging BIG -bench tasks and whether chain-of-thought c...

    2023 doi

  38. [46]

    Lukasz Szymanski, Bartłomiej Śnieżyński, and Bipin Indurkhya. 2018. https://doi.org/10.7494/csci.2018.19.4.3007 Multi-agent blackboard architecture for supporting legal decision making . Computer Science, 19:459

    2018 doi

  39. [47]

    Yashar Talebirad and Amirhossein Nadiri. 2023. Multi-agent collaboration: Harnessing the power of intelligent llm agents. arXiv preprint arXiv:2306.03314

    2023 arXiv

  40. [48]

    Bissyand \'e

    Xunzhu Tang, Kisub Kim, Yewei Song, Cedric Lothritz, Bei Li, Saad Ezzini, Haoye Tian, Jacques Klein, and Tegawend \'e F. Bissyand \'e . 2024. https://doi.org/10.18653/v1/2024.emnlp-main.632 C ode A gent: Autonomous communicative agents for code review . In Proceedings of the 2...

    2024 doi

  41. [49]

    Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O'Sullivan, and Hoang D Nguyen. 2025. Multi-agent collaboration mechanisms: A survey of llms. arXiv preprint arXiv:2501.06322

    2025 arXiv

  42. [50]

    Tao Tu, Mike Schaekermann, Anil Palepu, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Yong Cheng, and 1 others. 2025. Towards conversational diagnostic artificial intelligence. Nature, pages 1--9

    2025

  43. [51]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. https://openreview.net/forum?id=_VjQlMeSB_J Chain of thought prompting elicits reasoning in large language models . In Advances in Neural Information Proc...

    2022

  44. [52]

    Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. 2025. https://arxiv.org/abs/2305.14688 Expertprompting: Instructing large language models to be distinguished experts . Preprint, arXiv:2305.14688

    2025 arXiv

  45. [53]

    Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See-Kiong Ng, and Jiashi Feng. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.416 MA g IC : Investigation of large language model powered multi-agent in cognition, adaptability, rationality and collaborat...

    2024 doi

  46. [54]

    John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528--50652

    2024

  47. [55]

    Zhangyue Yin, Qiushi Sun, Cheng Chang, Qipeng Guo, Junqi Dai, Xuanjing Huang, and Xipeng Qiu. 2023. https://openreview.net/forum?id=30kbnyD9hF Exchange-of-thought: Enhancing large language model capabilities through cross-model communication . In The 2023 Conference on Empiric...

    2023

  48. [56]

    Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi. 2025. https://arxiv.org/abs/2502.11133 Masrouter: Learning to route llms for multi-agent systems . Preprint, arXiv:2502.11133

    2025 arXiv

  49. [57]

    Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, Lei Bai, and Xiang Wang. 2025 a . Multi-agent architecture search via agentic supernet. arXiv preprint arXiv:2502.04180

    2025 arXiv

  50. [58]

    Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. 2025 b . https://openreview.net/forum?id=LkzuPorQ5L Cut the crap: An economical communication pipeline for LLM -based multi-agent systems . In The Thirteent...

    2025

  51. [59]

    Hangfan Zhang, Zhiyao Cui, Xinrun Wang, Qiaosheng Zhang, Zhen Wang, Dinghao Wu, and Shuyue Hu. 2025 c . https://doi.org/10.48550/arXiv.2502.08788 If Multi - Agent Debate is the Answer , What is the Question ? arXiv preprint. ArXiv:2502.08788 [cs]

    2025 doi arXiv

  52. [60]

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xiong-Hui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. 2025 d . https://openreview.net/forum?id=z5uVAKwmjf AF low: Automating agentic workflow gene...

    2025

  53. [61]

    Jintian Zhang, Xin Xu, Ningyu Zhang, Ruibo Liu, Bryan Hooi, and Shumin Deng. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.782 Exploring collaboration mechanisms for LLM agents: A social psychology view . In Proceedings of the 62nd Annual Meeting of the Association for Co...

    2024 doi

  54. [62]

    Yifan Zhang, Jingqin Yang, Yang Yuan, and Andrew Chi-Chih Yao. 2023. Cumulative reasoning with large language models. arXiv preprint arXiv:2308.04371

    2023 arXiv

  55. [63]

    Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. 2024 b . https://arxiv.org/abs/2404.13501 A survey on the memory mechanism of large language model based agents . Preprint, arXiv:2404.13501

    2024 arXiv

  56. [64]

    Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and J \"u rgen Schmidhuber. 2024. https://openreview.net/forum?id=uTC9AFXIhg GPTS warm: Language agents as optimizable graphs . In Forty-first International Conference on Machine Learning

    2024

  57. [65]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  58. [66]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.