REVIEW 2 major objections 1 minor 1 cited by
Reviewed by Pith at T0; open to challenge.
T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →
T0 review · grok-4.3
QueenBee Planner lets an LLM learn to generate communication topologies that improve multi-agent performance and efficiency over fixed designs.
2026-06-29 00:49 UTC pith:7FTLTLIL
load-bearing objection QueenBee introduces an outer LLM planner that generates and distills temporal DAG topologies with explicit safeguards, reporting RMSE drops on Count-Frequency and directional gains on Silo-Bench, but the abstract leaves the safeguards' effectiveness untested. the 2 major comments →
QueenBee Planner: Skill-Evolving Communication Topologies for Token-Efficient LLM Multi-Agent Systems
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
With fixed workers, self-evolved graph generation produces communication structures that improve over fixed topologies and cold generation. In the CF fulltest setting, the best generated graph reduces RMSE from 12.53 for the strongest fixed topology to 7.87 while also reducing messages, model calls, and token cost. Similar improvements appear in Silo-style tasks. These results suggest that multi-agent systems can learn reusable architectural design knowledge.
What carries the argument
The QueenBee Planner, an outer LLM that generates temporal communication DAGs and distills execution traces into evidence-backed design rules with Preserve, Modify, and Avoid actions, protected by held-out acceptance gates and other safeguards.
Load-bearing premise
The combination of held-out acceptance gates, variance-aware credit, motif-level attribution, transfer trust, insight falsification, and structural deduplication suffices to extract generalizable design rules rather than task-specific memorization from the execution traces.
What would settle it
Running the planner on a new, unrelated task and finding that the generated graphs perform no better than the best fixed topology or random generation would falsify the claim of learning reusable rules.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces QueenBee Planner, a framework in which an outer LLM planner learns to generate temporal communication DAGs for fixed worker agents in multi-agent LLM systems. Execution traces are distilled into Preserve/Modify/Avoid design rules, protected by held-out acceptance gates, variance-aware credit, motif-level attribution, transfer trust, insight falsification, and structural deduplication. On Count-Frequency aggregation and Silo-Bench tasks, self-evolved graphs outperform fixed topologies and cold generation; the best generated graph reduces RMSE from 12.53 to 7.87 in the CF fulltest setting while also lowering message, model-call, and token costs. The central claim is that these mechanisms enable extraction of reusable architectural design knowledge rather than task-specific memorization.
Significance. If the safeguards demonstrably prevent overfitting and the reported gains prove robust and transferable, the work would be significant for multi-agent LLM research by establishing a concrete method for self-improving communication topologies that jointly improve accuracy and efficiency. Treating topology generation as a retrievable skill with explicit anti-spurious-correlation machinery is a clear contribution; the direction of results (better RMSE plus lower cost) aligns with practical needs in distributed LLM coordination.
major comments (2)
- [Abstract] Abstract: The central claim that the six listed safeguards collectively extract generalizable design rules is load-bearing, yet the manuscript supplies no ablation results (e.g., performance when any single safeguard is removed), no transfer experiments on out-of-distribution task variants, and no quantitative metric showing that distilled rules beat random or heuristic graphs on held-out task families. Without such evidence the RMSE drop (12.53 → 7.87) could be explained by overfitting to the narrow evaluation distributions.
- [Abstract] Abstract / Evaluation section: The reported RMSE improvement and efficiency gains are presented without any description of the number of runs, statistical significance tests, exact task definitions for CF fulltest and Silo-Bench, or how the safeguards were validated on the execution traces. This absence prevents assessment of whether the gains are reproducible or merely directional.
minor comments (1)
- [Abstract] Abstract: The abbreviation 'CF' is used without expansion on first use; a brief parenthetical definition would improve readability.
Simulated Author's Rebuttal
Thank you for the constructive feedback on our manuscript. We appreciate the emphasis on strengthening evidence for the generalizability of the distilled design rules and on improving the description of experimental details for reproducibility. We address each major comment below and will revise the manuscript to incorporate additional analyses and clarifications where needed.
read point-by-point responses
-
Referee: [Abstract] Abstract: The central claim that the six listed safeguards collectively extract generalizable design rules is load-bearing, yet the manuscript supplies no ablation results (e.g., performance when any single safeguard is removed), no transfer experiments on out-of-distribution task variants, and no quantitative metric showing that distilled rules beat random or heuristic graphs on held-out task families. Without such evidence the RMSE drop (12.53 → 7.87) could be explained by overfitting to the narrow evaluation distributions.
Authors: We agree that the manuscript does not include explicit ablation studies on individual safeguards, transfer experiments on out-of-distribution task variants, or direct quantitative comparisons of distilled rules against random or heuristic graphs on held-out families. The reported results demonstrate that self-evolved graphs outperform fixed topologies and cold generation on the evaluated tasks, providing directional support for the value of the evolution process. However, this does not fully rule out overfitting explanations without the requested controls. In the revised manuscript we will add ablation experiments (removing key safeguards one at a time), evaluate on additional task variants, and include comparisons against random and heuristic baselines on held-out task families to better substantiate the claim of reusable architectural knowledge. revision: yes
-
Referee: [Abstract] Abstract / Evaluation section: The reported RMSE improvement and efficiency gains are presented without any description of the number of runs, statistical significance tests, exact task definitions for CF fulltest and Silo-Bench, or how the safeguards were validated on the execution traces. This absence prevents assessment of whether the gains are reproducible or merely directional.
Authors: We acknowledge that the current manuscript omits these experimental details. The abstract and evaluation section report the RMSE and efficiency numbers but do not specify the number of runs, statistical tests, precise task definitions, or the validation procedure for the safeguards on traces. In the revision we will expand the evaluation section to include the number of independent runs performed, results of statistical significance tests, exact definitions of the Count-Frequency fulltest and Silo-Bench tasks, and a description of how each safeguard was applied and validated during trace distillation. revision: yes
Circularity Check
No significant circularity; empirical results are benchmark comparisons, not reductions to inputs
full rationale
The paper describes an empirical framework for self-evolving communication DAGs via an outer LLM planner, with execution traces distilled into Preserve/Modify/Avoid rules under multiple safeguards (held-out gates, variance-aware credit, motif attribution, transfer trust, falsification, deduplication). Reported gains, such as RMSE drop from 12.53 to 7.87 on CF fulltest, are direct measurements against fixed-topology and cold-generation baselines on held-out task executions. No equations, fitted parameters renamed as predictions, or self-citation chains appear in the provided text; the central claim rests on observable performance deltas rather than any derivation that loops back to its own inputs by construction. The setup is self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
read the original abstract
Large language model (LLM) multi-agent systems increasingly depend not only on how individual agents reason, but also on how agents are connected. This paper introduces QueenBee Planner, a framework that treats inter-agent communication topology as a retrievable and self-improving design skill. A pool of worker agents, the task adapter, and the scoring function are frozen; only an outer LLM planner learns to generate temporal communication DAGs specifying who sends information to whom, in which round, who merges messages, and who emits the final answer. Execution traces are distilled into evidence-backed design rules with three actions: \emph{Preserve}, \emph{Modify}, and \emph{Avoid}. To prevent self-evolution from turning lucky runs or plausible but false explanations into policy, QueenBee uses held-out acceptance gates, variance-aware credit, motif-level attribution, transfer trust, insight falsification, and structural deduplication. We evaluate the method on Count-Frequency aggregation and Silo-Bench-style distributed coordination tasks. With fixed workers, self-evolved graph generation produces communication structures that improve over fixed topologies and cold generation. In the CF fulltest setting, the best generated graph reduces RMSE from 12.53 for the strongest fixed topology to 7.87 while also reducing messages, model calls, and token cost; Silo-style results show the same direction of improvement over cold and fixed-topology baselines. These results suggest that multi-agent systems can learn reusable architectural design knowledge rather than merely memorizing task answers.
Figures
Forward citations
Cited by 1 Pith paper
-
Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops
A survey of 1,250 papers organizes AI self-improvement along two axes—what is improved and loop closure—finding that demonstrated self-improvement strength tracks a verification hierarchy from formal verifiers down to...
Reference graph
Works this paper leans on
-
[1]
Graph of thoughts: Solving elaborate problems with large language models
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17682–17690. AAA...
2024
-
[2]
MapReduce: Simplified data processing on large clusters.Communi- cations of the ACM, 51(1):107–113, 2008
Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified data processing on large clusters.Communi- cations of the ACM, 51(1):107–113, 2008
2008
-
[3]
Tenenbaum, and Igor Mordatch
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. InProceedings of the 41st International Conference on Machine Learning (ICML 2024), volume 235 ofProceedings of Machine Learning Research, pages 11733–11763. PMLR, 2024
2024
-
[4]
MetaGPT: Meta programming for a multi-agent collaborative framework
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations (IC...
2024
-
[5]
Automated design of agentic systems
Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. InThe Thirteenth International Conference on Learning Representations (ICLR 2025), 2025
2025
-
[6]
Large language models cannot self-correct reasoning yet
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. InThe Twelfth International Conference on Learning Representations (ICLR 2024), 2024
2024
-
[7]
Tree search for language model agents, 2024
Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents, 2024
2024
-
[8]
CAMEL: Communicative agents for “mind” exploration of large language model society
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: Communicative agents for “mind” exploration of large language model society. InAdvances in Neural Information Processing Systems 36 (NeurIPS 2023), pages 51991–52008, 2023
2023
-
[9]
Encouraging divergent thinking in large language models through multi-agent debate
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), pages 17889–17904, Miami, Florida, USA, 2024. Association ...
2024
-
[10]
A dynamic LLM-powered agent network for task-oriented agent collaboration, 2023
Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. A dynamic LLM-powered agent network for task-oriented agent collaboration, 2023
2023
-
[11]
Bowman, and Shi Feng
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. InAdvances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024
2024
-
[12]
Scaling large-language-model-based multi-agent collaboration
Chen Qian, Zihao Xie, Yifei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Scaling large-language-model-based multi-agent collaboration. InThe Thirteenth International Conference on Learning Representations (ICLR 2025), 2025
2025
-
[13]
Reflexion: Language agents with verbal reinforcement learning
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023
2023
-
[14]
V oyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Research, 2024
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Research, 2024
2024
-
[15]
Le, Ed H
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V . Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. InThe Eleventh International Conference on Learning Representations (ICLR 2023), 2023
2023
-
[16]
Agent workflow memory
Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. In Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), volume 267 of Proceedings of Machine Learning Research. PMLR, 2025
2025
-
[17]
Chi, Quoc V
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V . Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems 35 (NeurIPS 2022), 2022. 15
2022
-
[18]
White, Doug Burger, and Chi Wang
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen LLM applications via multi-agent conversation. InConference on Language Modeling (COLM), 2024
2024
-
[19]
Griffiths, Yuan Cao, and Karthik Narasimhan
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. InAdvances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023
2023
-
[20]
ReAct: Synergizing reasoning and acting in language models
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InThe Eleventh International Conference on Learning Representations (ICLR 2023), 2023
2023
-
[21]
Self-rewarding language models
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. InProceedings of the 41st International Conference on Machine Learning (ICML 2024), volume 235 ofProceedings of Machine Learning Research, pages 57905–57923. PMLR, 2024
2024
-
[22]
G-Designer: Architecting multi-agent communication topologies via graph neural networks
Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, Tianlong Chen, and Dawei Cheng. G-Designer: Architecting multi-agent communication topologies via graph neural networks. InProceedings of the 42nd International Conference on Machine Learning (ICML 2025), volume 267 ofProceedings of Machine Learning Research, pages 766...
2025
-
[23]
AFlow: Automating agentic workflow generation
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating agentic workflow generation. InThe Thirteenth International Conference on Learning Representations (ICLR 2025), 2025
2025
-
[24]
Silo-Bench: A scalable environment for evaluating distributed coordination in multi-agent LLM systems, 2026
Yuzhe Zhang, Feiran Liu, Yi Shan, Xinyi Huang, Xin Yang, Yueqi Zhu, Xuxin Cheng, Cao Liu, Ke Zeng, Terry Jingchen Zhang, and Wenyuan Jiang. Silo-Bench: A scalable environment for evaluating distributed coordination in multi-agent LLM systems, 2026
2026
-
[25]
ExpeL: Llm agents are experiential learners
Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: Llm agents are experiential learners. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632–19642. AAAI Press, 2024
2024
-
[26]
Xing, Hao Zhang, Joseph E
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM- as-a-judge with MT-Bench and chatbot arena. InAdvances in Neural Information Processing Systems 36 (NeurIPS 2023) Datasets and Benchmarks Track, 2023
2023
-
[27]
Han Zhou, Xingchen Wan, Ruoxi Sun, Hamid Palangi, Shariq Iqbal, Ivan Vuli ´c, Anna Korhonen, and Sercan Ö. Arık. Multi-agent design: Optimizing agents with better prompts and topologies, 2025
2025
-
[28]
GPTSwarm: Language agents as optimizable graphs
Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmid- huber. GPTSwarm: Language agents as optimizable graphs. InProceedings of the 41st International Conference on Machine Learning (ICML 2024), volume 235 ofProceedings of Machine Learning Research, pages 62743–62767. PMLR, 2024. 16 A Representative Silo Skill C...
2024
-
[29]
one_peer_exponential_dag: distance 1 propagation
-
[30]
one_peer_exponential_dag: distance 2 propagation
-
[31]
one_peer_exponential_dag: distance 4 propagation
-
[32]
one_peer_exponential_dag aggregation: star gather to agent 4 skill_id: silo__cf_avoid_chain rule_action: avoid topology_name: chain strength: negative routing evidence weakness: dominated by stronger CF topology choices protocol steps:
-
[33]
chain: agent 0 sends to agent 1
-
[34]
chain: agent 1 sends to agent 2
-
[35]
chain: agent 2 sends to agent 3
-
[36]
It preserves a high-coverage peer- propagation scaffold, while also recording that a simple chain is a negative design under the observed Silo condition
chain: agent 3 sends to agent 4 The first accepted batch therefore does two things at once. It preserves a high-coverage peer- propagation scaffold, while also recording that a simple chain is a negative design under the observed Silo condition. This is why the bank grows quickly in round 1: it stores both reusable positive structure and reusable countere...
-
[37]
Initial vote counting and distribution
-
[38]
protocol steps:
Final aggregation step to determine the winner skill_id: silo__cf_topology_generated:staged_pair_reduce_to_sink__a5__arr0 rule_action: preserve topology_name: generated:staged_pair_reduce_to_sink lesson: Observed CF evidence for topology generated:staged_pair_reduce_to_sink. protocol steps:
-
[39]
Initial local palindrome computations and boundary exchanges
-
[40]
Aggregating results from adjacent agents
-
[41]
Round 2 adds more specific reduction patterns
Final aggregation at the sink agent. Round 2 adds more specific reduction patterns. Unlike the round-1 named topology scaffold, these are generated temporal DAGs. Their raw card text emphasizes staged reduction, bounded fan-in, and reachability to the selected primary. 17 A.3 Round 3: Star-Sink Transfer and Mesh Avoidance skill_id: silo__cf_topology_gener...
-
[42]
Initial local distinct counting and first stage of aggregation
-
[43]
Final aggregation and submission of global distinct count
-
[44]
sink-coverage repair skill_id: silo__cf_avoid_mesh_star rule_action: avoid topology_name: mesh_star strength: negative routing evidence weakness: dominated by stronger CF topology choices protocol steps:
-
[45]
mesh: all agents broadcast to all other agents
-
[46]
mesh aggregation: star gather to agent 4 Round 3 illustrates why the memory is not a fixed-topology selector. A generated star-sink variant is preserved because it carries positive evidence with an explicit repair step, while the dense mesh-star topology becomes an avoid rule under the same condition. A.4 Round 4: Star Aggregation Variants skill_id: silo_...
-
[47]
Initial local voting count distribution
-
[48]
Intermediate aggregation of vote counts
-
[49]
protocol steps:
Final decision making step skill_id: silo__cf_topology_generated:star_aggregate_to_sink__a5__arr0 rule_action: preserve topology_name: generated:star_aggregate_to_sink lesson: Observed CF evidence for topology generated:star_aggregate_to_sink. protocol steps:
-
[50]
Round 4 expands the bank with star-style aggregation variants
Initial XOR computation and distribution. Round 4 expands the bank with star-style aggregation variants. These cards are useful not because the topology name is universal, but because the planner can retrieve their structural motifs when the task feature slot calls for a single sink or low-depth aggregation. A.5 Round 5: Negative Memory for a Formerly Pos...
-
[51]
Initial local maximum computation and sharing 18 Round 5 shows that the bank can revise a family with negative evidence. The same broad star-sink motif can be preserved in one evidence context and avoided in another; the planner receives both signals as task-conditioned design memory rather than as a single hard-coded topology choice. 19
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.