Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

HALO: Hierarchical Autonomous Logic-Oriented Orchestration for Multi-Agent LLM Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Treating LLM multi-agent cooperation as a search over dynamic roles yields 14.4% average gains.

desk verdict A multi-agent framework with huge headline gains, but the published algorithm doesn't actually implement the MCTS search that the whole story rests on. read the letter →

arxiv 2505.13516 v1 pith:7HCKRNU2 submitted 2025-05-17 cs.MA cs.AI

Zhipeng Hou , Junyi Tang , Yipeng Wang This is my paper · ORCID
classification cs.MAcs.AI
keywords multi-agentsystemslargelanguagemodelsMonteCarlotreesearchhierarchicalreasoningadaptivepromptrefinementworkflowcodegenerationmathematical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HALO is a three-stage multi-agent architecture that replaces fixed agent roles and static communication graphs with dynamically generated roles and search over reasoning workflows. The paper claims that a high-level planner, mid-level role designers, and low-level inference agents, coordinated by Monte Carlo Tree Search, beat six established baselines by 14.4% on average across HumanEval, MMLU, and MATH when every method runs on GPT-4o. A four-agent prompt-refinement stage first converts raw user queries into structured prompts, and ablations show that removing either prompt refinement or task decomposition costs 5.3 and 11.3 percentage points of average accuracy respectively. If the results hold, multi-agent LLM systems can be organized on the fly rather than handcrafted for each domain.

What carries the argument

The central mechanism is the Workflow Search Engine layered on a Hierarchical Reasoning Stack. Each node in the search tree is a role-specific agent's intermediate output for a subtask, and edges are transitions between reasoning states; selection uses UCT with a quality score, expansion instantiates a new agent, simulation runs hypothetical future agents, and backpropagation updates node values using a reward signal adjusted by the terminal status label. The hierarchical stack supplies the structure that makes the search space meaningful: a high-level planner decomposes the task, mid-level role-design agents instantiate specialized roles, and low-level inference agents execute each subtask. The Adaptive Prompt Refinement module is the entry point that turns raw user queries into structured prompts for both planning and inference.

What would settle it

Run HALO with the Judge/Score-guided MCTS replaced by a fixed random or greedy expansion of the same number of role-specific agents, keeping the hierarchical stack and prompt refinement unchanged; if accuracy on HumanEval, MMLU, and MATH does not drop, the reported gains come from added compute or prompt engineering, not from search. A complementary check is to feed the Judge known-wrong but fluent answers and see whether its scores still rise with apparent confidence.

Watch

Extended reading notes

Core claim

The paper's central claim is that subtask execution in a multi-agent LLM system should be treated as a search problem, not a fixed pipeline. HALO's high-level planning agent emits one subtask at a time based on the execution history; mid-level role-design agents generate specialist roles and role-specific system prompts for each subtask; low-level inference agents carry out the subtask. A Monte Carlo Tree Search over the agent action space scores every intermediate output with a Judge agent (success/fail/continue) and a Score agent (0-1), then backpropagates a status-adjusted value to select the highest-scoring reasoning trajectory. In the paper's experiments, HALO reports 95.2% pass@1 on HumanEval, 81.6% accuracy on MMLU, and 58.9% accuracy on MATH, with the largest improvements on abstract MMLU subjects and computationally intensive MATH subareas.

Load-bearing premise

The load-bearing premise is that the internal Judge and Score agents give reward signals that track true correctness, so the search selects genuinely better reasoning trajectories rather than merely more fluent-looking ones.

Editorial extensions

If this is right

  • If HALO's results are right, multi-agent systems can be assembled without hand-designing agent roles or communication workflows for each new domain.
  • The largest gains on expert-level subjects imply that hierarchical decomposition plus search helps most exactly where fixed-role teams struggle.
  • The ablations indicate that both query refinement and iterative task decomposition are needed, not optional extras, for the reported accuracy.
  • Because all baselines run on the same LLM, the reported improvements are attributable to the orchestration architecture rather than to a stronger model.
  • The early-stopping rule could make the framework cheaper than exhaustive search by halting when 66% of completed subtasks agree on an answer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Judge and Score agents are never validated against ground truth, so a natural ablation would replace MCTS with a fixed-depth parallel rollout of the same agents to separate gains from search and gains from extra compute.
  • A testable extension is to take a workflow optimized on one subject and run it on a held-out subject, which would show whether the search discovers reusable orchestration patterns or merely overfits an internal reward signal.
  • The 66% consistency threshold is borrowed from Byzantine fault tolerance as a heuristic; sweeping that threshold would reveal whether the early stop is well calibrated or too aggressive.
  • The same planner-role-inference stack with search could extend to tool-use or retrieval action spaces, where node values can be grounded in task feedback instead of a language-model judge.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HALO, a three-stage hierarchical multi-agent framework for LLM-based reasoning. The first stage refines raw user queries through four prompt-engineering agents; the second stage decomposes the task with a planning agent and dynamically instantiates role-specific agents for each subtask; the third stage is described as an MCTS-driven workflow search over agent reasoning trajectories. Experiments on HumanEval, MMLU, and MATH report large gains over six baselines, with a 78.6% average versus 64.0% for the strongest baseline ADAS, and ablations show performance drops when the prompt-refinement module or the planning layer is removed.

Significance. If the central claim held, HALO would be a useful contribution to adaptive multi-agent orchestration: it directly addresses the limitation of static role designs and fixed communication workflows, and it combines hierarchical decomposition with search-based workflow construction. The final evaluation is grounded in external correctness signals (HumanEval unit tests, MMLU and MATH ground-truth answers), which is a strength. However, the paper's significance depends on two load-bearing points that are not currently established: that the implemented algorithm is actually the MCTS search described in Section 3.4, and that the reported gains are not simply a consequence of added inference compute and self-referential internal scoring. Both issues are addressable but require substantive revision.

major comments (4)
  1. [Algorithm 1 vs. §3.4] Algorithm 1 does not implement the MCTS workflow search described in Section 3.4. Lines 12–16 loop once over the generated role agents, evaluate each output, and call an undefined MCTS_Backpropagate; there is no selection stage using Eq. (9), no expansion, no simulation using Eq. (10), no visit-count updates, and no explicit tree structure. The only executable-level specification therefore corresponds to generate-and-score with aggregation, not to UCT-based tree search. The reported gains cannot be attributed to the claimed MCTS mechanism unless the missing search loop is specified and shown in the algorithm, or the mechanism description is revised to match the actual implementation.
  2. [§3.4, Appendix B] The reward signal for the workflow search is produced by GPT-4o-based Judge and Score agents, and the paper provides no evidence that these internal scores correlate with true correctness on MMLU or MATH. Since the same model family generates the candidate answers and evaluates them, the search may be optimizing a self-referential proxy. The authors should validate the Judge/Score outputs against ground truth on a held-out subset and report agreement rates, or otherwise justify that the search reward is meaningful.
  3. [§4.1, Tables 2 and 3] The empirical comparison is not sufficiently controlled. Tables 2 and 3 report results "averaged over three runs" without error bars or standard deviations. The MMLU and MATH subsampling (13% and 500 items) is described as random but no seeds are given, and there is no statement that all baselines were evaluated on the identical question subsets. In addition, HALO uses multiple role-specific agents per subtask plus an iterative planning loop, while the baselines do not appear to be matched for inference budget; the reported improvements may therefore partly reflect additional compute rather than the proposed mechanism. Please provide seed-controlled, budget-matched comparisons or explicitly discuss computational cost.
  4. [§4.3, Figure 4] The ablation study does not isolate the contribution of the MCTS workflow search. It reports only the full system, the system without prompt refinement, and the system without the planning layer; there is no ablation that removes or replaces the MCTS search while keeping the hierarchical prompt-refinement and role-instantiation modules. Without this, the paper does not establish that the search-based workflow exploration, rather than the hierarchical prompting structure and increased inference budget, drives the reported gains.
minor comments (5)
  1. [Figure 1] The figure caption contains a typo: "Adapative Prompt Refinement" should be "Adaptive Prompt Refinement."
  2. [Abstract and §4.2] The abstract reports a 14.4% average improvement over state-of-the-art baselines, while §4.2 states a 14.6% average improvement (78.6 vs. 64.0 in Table 2) and Table 3 reports a 14.4% gain on the abstract MMLU subjects. Please clarify which comparison the abstract refers to.
  3. [Algorithm 1, lines 17 and 22] The algorithm is imprecise about how the final answer is formed: line 17 updates H_k with a value \hat{Y}_k that is never computed inside the loop, and line 22 aggregates {y^{(i)}_k} after the loop has terminated, leaving the scope of the variable k unclear. Please make the aggregation and history-update steps explicit.
  4. [References] References [25] and [44] are duplicates of the same work, which makes it harder for readers to identify the intended citations for the baseline comparison.
  5. [§2.3] There is a grammatical error in the sentence "some researchs apply reinforcement learning to refine cooperation strategies"; "researchs" should be "researchers" or "research works."

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation is anchored by external benchmarks and independently defined components, with no fitted predictions or load-bearing self-citations.

full rationale

The paper's central empirical claim is that HALO outperforms baselines on HumanEval, MMLU, and MATH. These benchmarks are external to the method: HumanEval uses unit tests, and MMLU/MATH use ground-truth answers, so the reported accuracies are not constructed from the method's own outputs or reward signals. No parameter is fitted to a subset of the test data and then renamed as a prediction; the 14.4% average improvement is an observed result, not a fitted quantity. The architecture components (Adaptive Prompt Refinement, Hierarchical Reasoning Stack, Workflow Search Engine) are defined operationally through their system prompts and equations, and none of these definitions presuppose the benchmark outcomes. The internal Judge and Score agents do provide the MCTS reward signal using the same GPT-4o model that generates the answers, which is a self-referential bias risk and a legitimate experimental concern, but it is not a definitional circularity: the final evaluation is external, and the search objective is not claimed to be identical to the benchmark metric by construction. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via prior work by the same authors. The apparent mismatch between Algorithm 1 and the four-stage MCTS description in Section 3.4 is a reproducibility or correctness issue, not an instance of a prediction reducing to its inputs, so it does not raise the circularity score under the specified rubric.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The framework introduces no new physical entities. Its free parameters are the MCTS exploration coefficient, the status-label reward factors, the subtask budget, the early-stop threshold, and the number of agent roles per subtask, none of which are reported with concrete values. The load-bearing assumptions are the reliability of LLM-based judges and the applicability of Byzantine consensus to answer aggregation.

free parameters (5)
  • alpha (UCT exploration coefficient) = not reported
    Appears in Eq. 9 as the exploration weight in UCT; no value is given anywhere in the paper, so the search behavior cannot be replicated.
  • lambda (status-label impact factor) = not reported
    Appears in Eq. 11 as lambda(success/fail/continue) for reward adjustment; values are not specified, so backpropagation cannot be reproduced.
  • K (subtask budget) = not reported
    The maximum number of subtasks is an input to Algorithm 1 but no default or range is given.
  • Early-stop consensus threshold = 0.66 (66%)
    The stopping rule terminates when at least 66% of completed subtasks agree; this constant is motivated by Byzantine consensus but not derived and not chosen via experiments.
  • Number of role-specific agents per subtask = not reported
    The mid-level role-design agent generates a list of agent roles, but the number generated and the branching factor in MCTS expansion are unspecified.
assumptions (3)
  • domain assumption LLM-based Judge and Score agents provide reliable feedback correlated with correctness.
    The MCTS reward (Eq. 11) depends on Judge and Score agents implemented as GPT-4o with prompts in Appendix B; if they misjudge intermediate steps, the search does not necessarily find better workflows.
  • ad hoc to paper Byzantine consensus principles apply to the aggregation of LLM subtask answers.
    Section 3.3 invokes 'Byzantine Consensus theory' to justify the 66% consistency threshold for early stopping, but the formal theorem concerns fault tolerance in message-passing systems, not answer agreement among LLM agents.
  • domain assumption The search space of agent roles and actions is sufficiently covered by LLM-generated expansions.
    Expansion in MCTS instantiates new roles generated by an LLM (Appendix C); whether these roles capture the actions needed for the task is assumed rather than verified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HALO: Hierarchical Autonomous Logic-Oriented Orchestration for Multi-Agent LLM Systems." pith.science (2026). https://pith.science/paper/7HCKRNU2

@misc{pith2026250513516,
  author       = {Pith},
  title        = {Pith review of: HALO: Hierarchical Autonomous Logic-Oriented Orchestration for Multi-Agent LLM Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7HCKRNU2}},
  note         = {Machine review of arXiv:2505.13516}
}
read the original abstract

Recent advancements in Multi-Agent Systems (MAS) powered by Large Language Models (LLMs) have demonstrated tremendous potential in diverse task scenarios. Nonetheless, existing agentic systems typically rely on predefined agent-role design spaces and static communication structures, limiting their adaptability as well as flexibility in complex interaction environments and leading to subpar performance on highly specialized and expert-level tasks. To address these issues, we introduce HALO, a multi-agent collaboration framework based on a hierarchical reasoning architecture. Specifically, we incorporate a high-level planning agent for task decomposition, mid-level role-design agents for subtask-specific agent instantiation, and low-level inference agents for subtask execution. Particularly, subtask execution is reformulated as a structured workflow search problem, where Monte Carlo Tree Search (MCTS) systematically explores the agentic action space to construct optimal reasoning trajectories. Additionally, as the majority of users lack expertise in prompt engineering, we leverage an Adaptive Prompt Refinement module to transform raw queries into task-specific prompts. Empirical evaluations on Code Generation (HumanEval), General Reasoning (MMLU), and Arithmetic Reasoning (MATH) benchmark datasets highlight the effectiveness of HALO, yielding a 14.4% average improvement over state-of-the-art baselines. Notably, HALO achieves up to 13.3% performance gain on the Moral Scenarios subject in the MMLU benchmark and up to 19.6% performance gain on the Algebra subarea in the MATH benchmark, indicating its advanced proficiency in tackling highly specialized and expert-level tasks. The code repository is available at https://github.com/23japhone/HALO.

Figures

Figures reproduced from arXiv: 2505.13516 by the authors.

Figure 1
Figure 1. The overview of HALO framework. HALO consists of three modules: (1) Adaptive Prompt [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The illustration of how Monte Carlo Tree Search (MCTS) guides multi-agent reasoning [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison on three com￾putationally intensive subareas selected from the MATH dataset. Metrics are reported as accuracy (%) averaged over three runs. HumanEval MMLU MATH Avg. 40 50 60 70 80 90 100 Performance (%) 95.2 81.6 58.9 78.6 90.3 75.4 54.1 73.3 83.8 73.5 44.7 67.3 HALO (full) w/o Adaptive Prompt Refinement module w/o task decomposition layer [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: System prompts used in the Adaptive Prompt Refinement module. The refinement process [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: System prompts for the high-Level planning agent and Workflow Search Engine module, [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: System prompts for mid-level role-design agents, including role generation and prompt [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Prompts used for HumanEval, MMLU, and MATH experiments, including query prefixes [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-Agent LLMs Fail to Explore Each Other

    cs.MA 2026-07 conditional novelty 6.5 of 10

    Modern multi-agent LLM systems fail to explore peers effectively; explicit LinUCB-style peer selection (MACE) cuts regret and lifts task performance, with gains scaling in agent diversity.

  2. InSight-doc: Agentic Visual Perception for Long-Document Understanding

    cs.CV 2026-08 conditional novelty 6.0 of 10

    InSight-doc trains an 8B vision-language model to zoom into document sub-regions on demand, improving long-document VQA accuracy by up to 16.4 points while cutting latency by 41-68%.

  3. The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Reasoning messages between heterogeneous VLMs can be routed through the image-token span: a distilled universal codec plus affine alignment transmits latent traces across model families, cutting wall-clock time in sma...

  4. Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives

    cs.AI 2025-06 conditional novelty 4.0 of 10

    A survey proposes a macro-meso-micro value framework for agentic AI alignment and maps applications, methods, and benchmarks onto it.

Reference graph

Works this paper leans on

56 extracted references · 26 canonical work pages · cited by 4 Pith papers

  1. [1]

    Competitive programming with large reasoning models,

    A. El-Kishky, A. Wei, A. Saraiva, B. Minaiev, D. Selsam, D. Dohan, F. Song, H. Lightman, I. Clavera, J. Pa- chocki et al., “Competitive programming with large reasoning models,” arXiv preprint arXiv:2502.06807, 2025

    arXiv 2025

  2. [2]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y . Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. W...

    arXiv 2025

  3. [3]

    Debug like a human: A large language model debugger via verifying runtime execution step-by-step,

    L. Zhong, Z. Wang, and J. Shang, “Debug like a human: A large language model debugger via verifying runtime execution step-by-step,” arXiv preprint arXiv:2402.16906, 2024

    arXiv 2024

  4. [4]

    Reflexion: language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. R. Narasimhan, and S. Yao, “Reflexion: language agents with verbal reinforcement learning,” in Thirty-seventh Conference on Neural Information Processing Systems,

    work page

  5. [5]

    Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration,

    J. Wang, H. Xu, H. Jia, X. Zhang, M. Yan, W. Shen, J. Zhang, F. Huang, and J. Sang, “Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration,”arXiv preprint arXiv:2406.01014, 2024

    arXiv 2024

  6. [6]

    Appagent v2: Advanced agent for flexible mobile interactions,

    Y . Li, C. Zhang, W. Yang, B. Fu, P. Cheng, X. Chen, L. Chen, and Y . Wei, “Appagent v2: Advanced agent for flexible mobile interactions,” arXiv preprint arXiv:2408.11824, 2024

    arXiv 2024

  7. [7]

    V oyager: An open-ended embodied agent with large language models,

    G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar, “V oyager: An open-ended embodied agent with large language models,” arXiv preprint arXiv:2305.16291, 2023

    arXiv 2023

  8. [8]

    Ai-vqa: visual question answering based on agent interaction with interpretability,

    R. Li, C. Xu, Z. Guo, B. Fan, R. Zhang, W. Liu, Y . Zhao, W. Gong, and E. Wang, “Ai-vqa: visual question answering based on agent interaction with interpretability,” inProceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 5274–5282

    work page 2022

Show all 56 references
  1. [9]

    Chain-of-table: Evolving tables in the reasoning chain for table understanding,

    Z. Wang, H. Zhang, C.-L. Li, J. M. Eisenschlos, V . Perot, Z. Wang, L. Miculicich, Y . Fujii, J. Shang, C.-Y . Lee et al., “Chain-of-table: Evolving tables in the reasoning chain for table understanding,” arXiv preprint arXiv:2401.04398, 2024

    2024 arXiv

  2. [10]

    Perceive, reflect, and plan: Designing llm agent for goal-directed city navigation without instructions,

    Q. Zeng, Q. Yang, S. Dong, H. Du, L. Zheng, F. Xu, and Y . Li, “Perceive, reflect, and plan: Designing llm agent for goal-directed city navigation without instructions,” arXiv preprint arXiv:2408.04168, 2024

    2024 arXiv

  3. [11]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations (ICLR), 2023

    2023

  4. [12]

    Camel: Communicative agents for

    G. Li, H. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem, “Camel: Communicative agents for" mind" exploration of large language model society,”Advances in Neural Information Processing Systems, vol. 36, pp. 51 991–52 008, 2023

    2023

  5. [13]

    A survey of monte carlo tree search methods,

    C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, P. I. Cowling, P. Rohlfshagen, S. Tavener, D. Perez, S. Samothrakis, and S. Colton, “A survey of monte carlo tree search methods,” IEEE Transactions on Computational Intelligence and AI in games, vol. 4, no. 1, pp. 1–43, 2012

    2012

  6. [14]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021. 10

    2021 arXiv

  7. [15]

    Measuring massive multitask language understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  8. [16]

    Measuring mathematical problem solving with the math dataset,

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, “Measuring mathematical problem solving with the math dataset,” arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  9. [17]

    Promptor: A conversational and autonomous prompt generation agent for intelligent text entry techniques,

    J. Shen, J. J. Dudley, J. Zheng, B. Byrne, and P. O. Kristensson, “Promptor: A conversational and autonomous prompt generation agent for intelligent text entry techniques,” 2023. [Online]. Available: https://arxiv.org/abs/2310.08101

    2023 arXiv

  10. [18]

    Comfybench: Benchmarking llm-based agents in comfyui for autonomously designing collaborative ai systems,

    X. Xue, Z. Lu, D. Huang, Z. Wang, W. Ouyang, and L. Bai, “Comfybench: Benchmarking llm-based agents in comfyui for autonomously designing collaborative ai systems,” 2024. [Online]. Available: https://arxiv.org/abs/2409.01392

    2024

  11. [19]

    Metagpt: Meta programming for multi-agent collaborative framework,

    S. Hong, X. Zheng, J. Chen, Y . Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhouet al., “Metagpt: Meta programming for multi-agent collaborative framework,” arXiv preprint arXiv:2308.00352, vol. 3, no. 4, p. 6, 2023

    2023 arXiv

  12. [20]

    Tptu: Task planning and tool usage of large language model-based ai agents,

    J. Ruan, Y . Chen, B. Zhang, Z. Xu, T. Bao, H. Mao, Z. Li, X. Zeng, R. Zhaoet al., “Tptu: Task planning and tool usage of large language model-based ai agents,” inNeurIPS 2023 Foundation Models for Decision Making Workshop, 2023

    2023

  13. [21]

    A dynamic llm-powered agent network for task-oriented agent collaboration,

    Z. Liu, Y . Zhang, P. Li, Y . Liu, and D. Yang, “A dynamic llm-powered agent network for task-oriented agent collaboration,” in First Conference on Language Modeling, 2024

    2024

  14. [22]

    Agent laboratory: Using llm agents as research assistants,

    S. Schmidgall, Y . Su, Z. Wang, X. Sun, J. Wu, X. Yu, J. Liu, Z. Liu, and E. Barsoum, “Agent laboratory: Using llm agents as research assistants,” arXiv preprint arXiv:2501.04227, 2025

    2025 arXiv

  15. [23]

    Scoreflow: Mastering llm agent workflows via score-based preference optimization,

    Y . Wang, L. Yang, G. Li, M. Wang, and B. Aragam, “Scoreflow: Mastering llm agent workflows via score-based preference optimization,” arXiv preprint arXiv:2502.04306, 2025

    2025 arXiv

  16. [24]

    Workflowllm: Enhancing workflow orchestration capability of large language models,

    S. Fan, X. Cong, Y . Fu, Z. Zhang, S. Zhang, Y . Liu, Y . Wu, Y . Lin, Z. Liu, and M. Sun, “Workflowllm: Enhancing workflow orchestration capability of large language models,” arXiv preprint arXiv:2411.05451, 2024

    2024 arXiv

  17. [26]

    Game-theoretic llm: Agent workflow for negotiation games,

    W. Hua, O. Liu, L. Li, A. Amayuelas, J. Chen, L. Jiang, M. Jin, L. Fan, F. Sun, W. Wang, X. Wang, and Y . Zhang, “Game-theoretic llm: Agent workflow for negotiation games,” 2024. [Online]. Available: https://arxiv.org/abs/2411.05990

    2024 arXiv

  18. [27]

    Aflow: Automating agentic workflow generation,

    J. Zhang, J. Xiang, Z. Yu, F. Teng, X. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang, B. Zheng, B. Liu, Y . Luo, and C. Wu, “Aflow: Automating agentic workflow generation,” 2025. [Online]. Available: https://arxiv.org/abs/2410.10762

    2025 arXiv

  19. [28]

    Fincon: A synthesized llm multi-agent system with conceptual verbal reinforcement for enhanced financial decision making,

    Y . Yu, Z. Yao, H. Li, Z. Deng, Y . Jiang, Y . Cao, Z. Chen, J. Suchow, Z. Cui, R. Liuet al., “Fincon: A synthesized llm multi-agent system with conceptual verbal reinforcement for enhanced financial decision making,” Advances in Neural Information Processing Systems, vol. 37,...

    2024

  20. [29]

    Multi-agent reinforcement learning: An overview,

    L. Bu¸ soniu, R. Babuška, and B. De Schutter, “Multi-agent reinforcement learning: An overview,”Innova- tions in multi-agent systems and applications-1, pp. 183–221, 2010

    2010

  21. [30]

    Reinforcement learning-based multi-agent system for network traffic signal control,

    I. Arel, C. Liu, T. Urbanik, and A. G. Kohls, “Reinforcement learning-based multi-agent system for network traffic signal control,” IET Intelligent Transport Systems, vol. 4, no. 2, pp. 128–135, 2010

    2010

  22. [31]

    Multi-agent reinforcement learning: A selective overview of theories and algorithms,

    K. Zhang, Z. Yang, and T. Ba¸ sar, “Multi-agent reinforcement learning: A selective overview of theories and algorithms,” Handbook of reinforcement learning and control, pp. 321–384, 2021

    2021

  23. [32]

    Reinforcement learning of coordination in heterogeneous cooperative multi-agent systems,

    S. Kapetanakis and D. Kudenko, “Reinforcement learning of coordination in heterogeneous cooperative multi-agent systems,” in Symposium on Adaptive Agents and Multi-agent Systems. Springer, 2003, pp. 119–131

    2003

  24. [33]

    Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors,

    W. Chen, Y . Su, J. Zuo, C. Yang, C. Yuan, C.-M. Chan, H. Yu, Y . Lu, Y .-H. Hung, C. Qian, Y . Qin, X. Cong, R. Xie, Z. Liu, M. Sun, and J. Zhou, “Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors,” 2023. [Online]. Available: https://arxiv.or...

    2023 arXiv

  25. [34]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  26. [35]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  27. [36]

    Least-to-most prompting enables complex reasoning in large language models,

    D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. Le et al., “Least-to-most prompting enables complex reasoning in large language models,” arXiv preprint arXiv:2205.10625, 2022

    2022 arXiv

  28. [37]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y . Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” Advances in neural information processing systems, vol. 36, pp. 11 809–11 822, 2023

    2023

  29. [38]

    Large language models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large language models are zero-shot reasoners,” Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  30. [39]

    Model context protocol (mcp): Landscape, security threats, and future research directions,

    X. Hou, Y . Zhao, S. Wang, and H. Wang, “Model context protocol (mcp): Landscape, security threats, and future research directions,” 2025. [Online]. Available: https://arxiv.org/abs/2503.23278

    2025 arXiv

  31. [40]

    Practical byzantine fault tolerance,

    M. Castro, B. Liskov et al., “Practical byzantine fault tolerance,” in OsDI, vol. 99, no. 1999, 1999, pp. 173–186

    1999

  32. [41]

    Bandit based monte-carlo planning,

    L. Kocsis and C. Szepesvári, “Bandit based monte-carlo planning,” inEuropean conference on machine learning. Springer, 2006, pp. 282–293

    2006

  33. [42]

    Coat: Chain-of-associated-thoughts framework for enhancing large language models reasoning,

    J. Pan, S. Deng, and S. Huang, “Coat: Chain-of-associated-thoughts framework for enhancing large language models reasoning,” arXiv preprint arXiv:2502.02390, 2025

    2025

  34. [43]

    Improving factuality and reasoning in language models through multiagent debate,

    Y . Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch, “Improving factuality and reasoning in language models through multiagent debate,” in Forty-first International Conference on Machine Learning, 2023

    2023

  35. [44]

    Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems,

    B. Liu, X. Li, J. Zhang, J. Wang, T. He, S. Hong, H. Liu, S. Zhang, K. Song, K. Zhu, Y . Cheng, S. Wang, X. Wang, Y . Luo, H. Jin, P. Zhang, O. Liu, J. Chen, H. Zhang, Z. Yu, H. Shi, B. Li, D. Wu, F. Teng, X. Jia, J. Xu, J. Xiang, Y . Lin, T. Liu, T. Liu, Y . Su, H. Sun, G. Be...

    2025 arXiv

  36. [45]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023. 12 A The system prompt of Adaptive Prompt Refinement You are a Task Parser Agent. You...

    2023 arXiv

  37. [47]

    analyzetrends,

    **Core Intent**: The main goal or purpose of the task (e.g., "analyzetrends," "generate summary," "predict outcomes").3. **Key Details**: Any specific instructions or constraints that are important for task execution (e.g., "for the last 3 months," "in Python,""based on given ...

  38. [48]

    analyze trends,

    **Task Type**: The general category of the task as identified by Task ParserAgent (e.g., data analysis, text generation, image processing).2. **Core Intent**: The main objective or purpose of the task as defined by Task Parser Agent (e.g., "analyze trends," "generate summary,"...

  39. [49]

    **Core Intent**: The main objective or purpose of the task as defined by the Task Parser Agent.3

    **Task Type**: The general category of the task as identified by theTask Parser Agent.2. **Core Intent**: The main objective or purpose of the task as defined by the Task Parser Agent.3. **Key Details**: Any important constraints or instructions forexecuting the task, as ident...

  40. [50]

    Task Type

    **Optimized Prompt Template**: The optimized version of the prompttemplate provided by the Prompt Template Generator Agent. This should include any necessary improvements, such as simplifyinglanguage, clarifying instructions, and removing unnecessary complexity. Your output sh...

  41. [51]

    Task Type

    **Task Type**: The general category of the task as identified by TaskParser Agent.2. **Core Intent**: The main objective or purpose of the task as defined by Task Parser Agent.3. **Key Details**: Any important constraints or instructions for executing the task, as identified b...

  42. [52]

    Make the subtask as specific and actionable as possible

    Generate exactly one new subtask that advances toward fulfilling the Core Intent.3. Make the subtask as specific and actionable as possible

  43. [53]

    next subtask

    If the overall task is already complete (no further decomposition needed), reply with exactly: stop. ### Output Format Return either:- A JSON object containing a single key `"next subtask"` whose value is your new subtask description, for example: ```json{ "next subtask": "<de...

  44. [54]

    Assess whether the **Output of the Agent Role** fully satisfies the**Current Subtask** in light of the **Task Type**, **Core Intent**, and**Key Details**

  45. [55]

    success"**. 3. If the subtask cannot be completed based on this output (e.g., wrong method or errors), return **

    If the subtask is correctly and completely done, return **"success"**. 3. If the subtask cannot be completed based on this output (e.g., wrong method or errors), return **"fail"**. 4. If the subtask is partially completed or needs further refinement beforemoving on, return **"...

  46. [56]

    Remove duplicate entriesfrom the sales dataset

    **Subtask**: the specific subtask instructions (e.g., “Remove duplicate entriesfrom the sales dataset.”).3. **Prompt Template**: a detailed template string that will instruct the agent how to perform the subtask, incorporating placeholders for any inputs or parameters (forexam...

  47. [57]

    Removeduplicate entries from the sales dataset

    **Subtask**: the specific subtask instructions (e.g., “Removeduplicate entries from the sales dataset.”).3. **Optimized Prompt Template**: your improved version of the prompt template, with clearer language and precise instructions. Example format:{ "Agent Role": "<agent_role>...

  48. [2023]

    Available: https://openreview.net/forum?id=vAElhFcKW6

    [Online]. Available: https://openreview.net/forum?id=vAElhFcKW6

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.