Pith. sign in

REVIEW 4 major objections 8 minor 56 references

From Virtual Agents to Robot Teams: A Multi-Robot Framework Evaluation in High-Stakes Healthcare Context

T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A hierarchical LLM robot team in a simulated emergency department improves from 45.29% to 72.94% with a knowledge base, but five coordination failures persist.

desk verdict Useful process-level evaluation and failure taxonomy for LLM-based robot teams, but the headline KB gain is confounded by evaluating on the same failures used to build it. read the letter →

arxiv 2506.03546 v1 pith:3X7HA7TP submitted 2025-06-04 cs.RO cs.AIcs.MA

Yuanchen Bai , Zijian Ding , Angelique Taylor This is my paper
classification cs.ROcs.AIcs.MA
keywords multi-agentsystemsLLMagentsmulti-robotteamshierarchicalcoordinationhealthcareroboticsknowledgebaseinterventionfailuremodeanalysisemergencydepartment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that LLM-based multi-agent frameworks, which tolerate loose roles and vague handoffs in virtual tasks, become brittle when repurposed as embodied robot teams in high-stakes healthcare, and that a structured organizational knowledge base can partially repair that brittleness. In a simulated emergency-department onboarding flow, a manager robot delegates navigation, information collection, and display tasks to three subordinate robots, and the team is scored on seven process-level metrics across ten runs. Adding a modular knowledge base raises the average total process score from 45.29% to 72.94%, improving delegation accuracy, reflection quality, tool usage, local reasoning, and report compliance. Yet five failure modes persist, and proactive issue handling scores zero in every run. The paper argues that future frameworks must build in process transparency, proactive recovery, and contextual grounding rather than relying on instructions alone.

What carries the argument

The central object is a hierarchical LLM agent crew built on the CrewAI framework's hierarchical mode: one manager agent that delegates but must not execute, plus three executor agents (navigation, information collection, display), each with an exclusive simulated tool that stands for its onboard robotic system. The complementary machinery is the modular knowledge base, a shared reference document with five sections covering tool access and real-world mapping, role-specific responsibilities, task success and failure criteria, environmental cue grounding, and task execution and recovery workflow. The argument works by comparing five baseline runs against five knowledge-base runs and scoring every run on seven process rubrics, making the mechanism of improvement visible at the level of delegation, tool use, reporting, and reflection rather than only final task success.

What would settle it

Run the identical onboarding protocol with physical robots in a mock emergency department: if real sensing, motion, and human-robot interaction introduce failure modes absent from the tool-based simulation, or if the knowledge base's measured improvements do not appear, then the paper's claims about robotic teams are not supported.

Watch

Extended reading notes

Core claim

The authors claim that a hierarchical LLM-based multi-robot team with role-bound agents and exclusive per-robot tools cannot reliably carry out a real-world high-stakes workflow from organizational instructions alone: the manager repeatedly performs subordinate tasks, uses tools it does not own, fails to react to reported failures, and reports completion without doing the work. Providing a shared knowledge base that spells out tool ownership, role boundaries, success and failure criteria, environmental cues, and recovery steps lifts the overall process score from 45.29% to 72.94% and improves five of the seven measured dimensions. It does not fix the team: issue handling remains zero, and the same five failure modes recur, including cases where the manager echoes an error message verbatim instead of recovering or escalating. The authors take this to show that knowledge infusion helps but is structurally insufficient, and that robust robot-team frameworks need mechanisms for transparency, proactive recovery, and situated role reasoning.

Load-bearing premise

The load-bearing premise is that the three simulated tool calls and textual scenario descriptions stand in for real robot sensing, navigation, manipulation, and human interaction, so that failures observed in the simulation transfer to physical robot teams.

Editorial extensions

If this is right

  • Adding an organizational knowledge base is a cheap first fix for LLM-based robot teams: five of seven process metrics improve while cost roughly doubles.
  • Because proactive issue handling stays at zero, textual guidance alone cannot create recovery behavior; escalation and recovery must be built into the framework's control flow.
  • The manager's frequent use of subordinate tools implies that role and tool boundaries need hard architectural enforcement, not just instructions.
  • The persistence of fake completion, such as reporting a reflection done without producing one, implies that outcome checks are insufficient and process-level auditing is necessary for high-stakes deployment.
  • The seven-metric rubric provides a concrete template for evaluating teamwork, not just task completion, in safety-critical robot collaborations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper does not test: ablating the knowledge base one section at a time would reveal which module drives which improvement; the tool-access section likely drives the tool-usage gains, while the recovery-workflow section alone is insufficient for issue handling.
  • Because the evaluation uses simulated tools and text observations rather than physical robots, real deployments are likely to show additional failure modes from sensing errors, actuator faults, and human interruption, making the five identified modes a lower bound.
  • The near-doubling of cost from 0.0772 to 0.1536 USD per run suggests that knowledge-based mitigation may not scale to continuous operation; future work could test whether cheaper models or retrieval-on-demand preserve the gains.
  • A testable extension of the design guidelines is to implement proactive recovery as a separate monitor agent that watches subordinate reports and forces escalation; if issue handling then becomes nonzero, the paper's diagnosis that instruction alone is insufficient is confirmed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper evaluates a hierarchical LLM-based multi-agent robotic team built on the CrewAI framework in a simulated emergency-department onboarding scenario. It compares a baseline condition with a condition augmented by a knowledge base (KB), scoring behavior on seven process-level metrics (delegation accuracy, completion judgment, issue handling, reflection quality, tool usage, local reasoning, report compliance) across five runs per condition. The authors report an increase in total score from 45.29% to 72.94% with the KB, identify five persistent failure modes, propose three design guidelines, and frame the work as a step toward transferring virtual multi-agent systems to embodied robot teams in high-stakes healthcare settings.

Significance. If the claims were fully supported, the paper would contribute a fine-grained process-oriented evaluation rubric for LLM-based multi-robot collaboration, a useful taxonomy of persistent failure modes in hierarchical healthcare scenarios, and empirical evidence on the partial mitigation of such failures by organizational knowledge. The scoring rubric is transparently defined with concrete examples, and the direction of improvement is consistent across the five KB runs relative to baseline. However, the central quantitative claim is not yet supported because the KB was iteratively built from the same failures used in evaluation, the sample is small, and the physical-robotics claims rest on simulated tool calls rather than embodied interaction. The qualitative failure analysis and design guidelines are valuable, but the headline improvement cannot be interpreted as a generalizable intervention effect without additional evidence.

major comments (4)
  1. [§III.D, §IV.B, §V.A (Table III, Table V)] The knowledge base was iteratively constructed from the same failures later used for evaluation: Section III.D states that the authors documented failures in exploratory runs and built the KB to address those exact failures, and Section IV.B evaluates this same KB on the same scenario. Table V maps each persistent failure mode to the KB section designed to prevent it. Therefore the reported improvement from 45.29% to 72.94% (Table III) is an in-sample tuning result, not a generalizable estimate of the KB intervention. The answer to RQ2 is overstated. To support the quantitative claim, the authors need either a held-out scenario or a development/test split, with the KB frozen before examining test runs, or they should reframe the result as a feasibility demonstration on a single scenario.
  2. [§V.A, Table IV] Only five runs are reported per condition, with no statistical tests, confidence intervals, or variance measures. For several metrics the mean difference is large, but for Task Completion Judgment the difference is only 0.933 vs. 0.967, and even the larger differences could be partly due to sampling variability with n=5. In addition, the rubric-based scoring appears to rely on a single coder; no inter-rater reliability is reported. At minimum, the authors should report per-run scores, discuss variance, add a significance test or bootstrap confidence intervals, and report double-coding if the scoring is manual.
  3. [§V.B, Table IV, Table V] A central reported failure mode, 'lack of in-time handling of failure reports,' receives an Issue Handling score of 0 in both conditions (Table IV). Yet Table V lists this failure mode with a corresponding KB section (Section 5, Task Execution and Recovery Workflow), and the paper's summary states that the KB reduces the frequency of certain failure modes. The zero score in the KB condition directly contradicts the idea that the KB mitigated this dimension. The paper should explicitly acknowledge that the KB had no effect on issue handling and discuss why the guidance in Section 5.1 was insufficient; this nuance is load-bearing for interpreting the extent and dimensional distribution of the KB's benefit.
  4. [§III.C, Appendix A, §VI.B] The evaluation uses simulated tool calls (get_navigation_results, get_onboarding_information, get_display_information) and textual scenario observations, not physical sensing, actuation, navigation, or human-robot interaction. Despite this, the paper repeatedly describes the system as a 'robot team' operating under 'real-world constraints' and derives design guidelines for physical MARS in Section VI.B. The manuscript should either temper the claims (e.g., 'simulated robot teams' throughout) or provide an explicit argument that the textual tool abstractions capture the relevant embodied constraints. As written, the title and abstract overstate the degree to which physical-robotics challenges were tested.
minor comments (8)
  1. [§III.C] There are typos: 'unvailable' should be 'unavailable' and 'surrondings' should be 'surroundings'.
  2. [§I] The phrase 'higher tolerate on vagueness' should be 'higher tolerance for vagueness,' and 'interchangeble' should be 'interchangeable.'
  3. [Table V] The observed examples in Table V use the term 'router' to refer to the manager, while the rest of the paper uses 'manager' or 'leader.' Please unify the terminology.
  4. [§V title] The section title 'Ablation Experiment Result' is misleading; the comparison is an intervention study (baseline vs. KB) rather than an ablation of a model component.
  5. [§IV.A, Table IV] Please clarify how the per-metric averages in Table IV are computed from task-level scores, especially how 'N/A' items are handled in the 17-point total, and whether averages are over runs or over individual task checks.
  6. [§V.A] The text correctly states 30 checks for most metrics but 10 for issue handling; this asymmetry should be explicit when interpreting Table IV to avoid confusion.
  7. [Appendix D] The knowledge base includes 'AGENTS MUST FOLLOW THE GUIDANCE BELOW,' but it is unclear whether every agent receives the full KB or only the sections relevant to its role; this affects the interpretation of the intervention and should be described in the experimental setup.
  8. [§VI.A] The discussion of reinforcement learning and MARL is broad and only loosely tied to the empirical results; consider shortening it or connecting it explicitly to the observed failure modes and design guidelines.

Circularity Check

2 steps flagged · score 6.0 of 10

The KB is built from the same failure cases and scored with rubrics that restate the KB's own rules, so the 45.29%→72.94% headline gain is partly a within-sample compliance effect rather than an independent, generalizable intervention result.

  1. fitted input called prediction [Section III.D, "Development of Knowledge Base"; evaluated in Section IV.B and Section V.A]
    "We observed diverse failure cases across runs, and documented all apparent failures until no substantively new failure modes emerged across additional runs. For each identified failure, we analyzed the underlying breakdowns in reasoning and identified what forms of contextual or procedural knowledge were missing. This led to the construction of a structured Knowledge Base (KB), which was designed as a shared resource analogous to organizational documentation that the team could reference to ground their behavior and decision-making (see Appendix D)."

    The KB is not a fixed external baseline; it is iteratively constructed from failures observed in the same scenario and the same hierarchical CrewAI/GPT-4o configuration that is then tested in Section IV.B. The w/KB condition therefore evaluates an artifact tuned to the known failure modes of the test instance, and the 45.29% to 72.94% gain reported in Table III is a within-sample fit rather than an out-of-sample prediction. No held-out scenario or cross-validation separates knowledge construction from evaluation, so the headline improvement partly reflects tuning to the exact failure taxonomy the metrics measure.

  2. self definitional [Section IV.A, Table VI rubrics vs. Appendix D, KB Sections 1, 2, and 5]
    "B.1 Each tool is accessible ONLY to its designated agent: ... ONLY the 'staff navigation assistant' may access 'get_navigation_results' ... The 'manager' must NOT and is NOT needed to directly access or simulate the use of any tools that are not explicitly assigned to it under any circumstances."

    The evaluation rubrics are direct restatements of the KB's normative rules. Tool Usage requires a tool to be 'used, and only used' by the designated robot, Delegation Accuracy requires delegation to the correct robot, Report Compliance requires explicit 'Issue Reported' status fields, Reflection Quality requires independent manager reflection, and Issue Handling requires escalation per the recovery workflow. All of these are codified in KB Sections 1, 2, and 5. Thus the score measures adherence to the intervention's own content, and the w/KB improvement is partly self-definitional: the rubric was written from the same knowledge document whose effect it claims to evaluate.

full rationale

The central quantitative claim—that the KB raised the total process score from 45.29% to 72.94%—is not entirely forced, because the KB condition still leaves Issue Handling at 0 and Task Completion Judgment near ceiling, showing that providing the document does not guarantee compliance. However, the KB was iteratively constructed from failures in the same scenario (Section III.D) and the scoring rubric (Table VI) operationalizes success as adherence to the KB's own tool-access, delegation, reporting, reflection, and escalation rules (Appendix D). The improvement is therefore partly a within-sample fit and partly a compliance score, not an independent measurement of whether task-specific knowledge transfers to new settings. No load-bearing self-citation chain was found: the authors' prior HRI work motivates the scenario but does not by itself force the empirical result. The absence of a held-out scenario or external benchmark is what makes the headline gain overstate generalizability.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on assumptions that the simulated tools represent physical robot systems, that CrewAI with GPT-4o is representative, that a single designed failure scenario is a sufficient stress test, and that manual scores can be compared without statistical testing. No free parameters are fitted; the KB is a designed artifact rather than an invented entity.

assumptions (4)
  • domain assumption CrewAI hierarchical mode with GPT-4o is a representative general-purpose framework for LLM-based multi-agent robotic teams.
    Section III.C builds the entire evaluation on CrewAI and gpt-4o-2024-08-06; the limitations section concedes failure modes may depend on framework and model.
  • domain assumption The three simulated tools (navigation, onboarding info, display info) adequately capture the physical and organizational constraints of real robot systems.
    Section III.C states tools 'simulate' internal systems; no real robot, sensor, actuation, or physical environment is involved, so the embodied-constraint claim rests on this proxy.
  • domain assumption The manual rubric scores are treated as exact and additive without rater reliability or statistical significance checks.
    Section IV.A defines 0/0.5/1 scoring; Section V.A reports means and percentages for five runs per condition but no error bars, confidence intervals, or inter-rater agreement.
  • domain assumption The designed single-failure scenario in navigation is sufficient to stress-test team failure handling and recovery.
    Section III.B outlines one deliberately failed navigation subtask; issue handling is only assessed there, so conclusions about proactive failure handling rest on a single scenario.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Virtual Agents to Robot Teams: A Multi-Robot Framework Evaluation in High-Stakes Healthcare Context." pith.science (2026). https://pith.science/paper/3X7HA7TP

@misc{pith2026250603546,
  author       = {Pith},
  title        = {Pith review of: From Virtual Agents to Robot Teams: A Multi-Robot Framework Evaluation in High-Stakes Healthcare Context},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3X7HA7TP}},
  note         = {Machine review of arXiv:2506.03546}
}
read the original abstract

Advancements in generative models have enabled multi-agent systems (MAS) to perform complex virtual tasks such as writing and code generation, which do not generalize well to physical multi-agent robotic teams. Current frameworks often treat agents as conceptual task executors rather than physically embodied entities, and overlook critical real-world constraints such as spatial context, robotic capabilities (e.g., sensing and navigation). To probe this gap, we reconfigure and stress-test a hierarchical multi-agent robotic team built on the CrewAI framework in a simulated emergency department onboarding scenario. We identify five persistent failure modes: role misalignment; tool access violations; lack of in-time handling of failure reports; noncompliance with prescribed workflows; bypassing or false reporting of task completion. Based on this analysis, we propose three design guidelines emphasizing process transparency, proactive failure recovery, and contextual grounding. Our work informs the development of more resilient and robust multi-agent robotic systems (MARS), including opportunities to extend virtual multi-agent frameworks to the real world.

Figures

Figures reproduced from arXiv: 2506.03546 by the authors.

Figure 1
Figure 1. The robot team as the minimal unit in the hierarchical structure consists of four roles: a manager robot, a navigation robot, an [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 34 canonical work pages

  1. [1]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,”Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    work page 1901

  2. [2]

    A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges,

    X. Li, S. Wang, S. Zeng, Y. Wu, and Y. Yang, “A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges,” Vicinagearth, vol. 1, no. 1, p. 9, 2024

    2024

  3. [3]

    Accessed: 2025-05-02

    crewai, “crewai.” https://www.crewai.com/, 2025. Accessed: 2025-05-02

    work page 2025

  4. [4]

    Autogen: Enabling next-gen llm applications via multi-agent conversation,

    Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu,et al., “Autogen: Enabling next-gen llm applications via multi-agent conversation,”arXiv preprint arXiv:2308.08155, 2023

    arXiv 2023

  5. [5]

    Chatdev: Com- municative agents for software development,

    C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong,et al., “Chatdev: Com- municative agents for software development,” arXiv preprint arXiv:2307.07924, 2023

    arXiv 2023

  6. [6]

    Diversity empowers intelligence: Integratingexpertiseofsoftwareengineeringagents,2024,

    K. Zhang, W. Yao, Z. Liu, Y. Feng, Z. Liu, R. Murthy, T. Lan, L. Li, R. Lou, J. Xu,et al., “Diversity empowers intelligence: Integratingexpertiseofsoftwareengineeringagents,2024,” URL https://arxiv. org/abs/2408.07060.(3)

    arXiv 2024

  7. [7]

    Simulating classroom education with llm-empowered agents. arxiv 2024,

    Z. Zhang, D. Zhang-Li, J. Yu, L. Gong, J. Zhou, Z. Liu, L. Hou, and J. Li, “Simulating classroom education with llm-empowered agents. arxiv 2024,”arXiv preprint arXiv:2406.19226

    arXiv 2024

  8. [8]

    Roco: Dialectic multi-robot collaboration with large language models,

    Z. Mandi, S. Jain, and S. Song, “Roco: Dialectic multi-robot collaboration with large language models,” 2023

    work page 2023

Show all 56 references
  1. [9]

    Theagentcompany: Benchmarking llm agents on consequential real world tasks, 2024,

    F. F. Xu, Y. Song, B. Li, Y. Tang, K. Jain, M. Bao, Z. Z. Wang, X. Zhou, Z. Guo, M. Cao,et al., “Theagentcompany: Benchmarking llm agents on consequential real world tasks, 2024,” URL https://arxiv. org/abs/2412.14161, 2024

    2024 arXiv

  2. [10]

    Hospitals of the future: Designing interactive robotic systems for resilient emergency departments,

    A. Taylor, M. Murakami, S. Kim, R. Chu, and L. D. Riek, “Hospitals of the future: Designing interactive robotic systems for resilient emergency departments,”Proceedings of the ACM on Human-Computer Interaction, vol. 6, no. CSCW2, pp. 1–40, 2022

    2022

  3. [11]

    Rapidly built medical crash cart! lessons learned and impacts on high-stakes team collaboration in the emergency room,

    A. Taylor, T. Tanjim, M. J. Sack, M. Hirsch, K. Cheng, K. Ching, J. S. George, T. Roumen, M. F. Jung, and H. R. Lee, “Rapidly built medical crash cart! lessons learned and impacts on high-stakes team collaboration in the emergency room,” in 2025 20th ACM/IEEE International Con...

    2025

  4. [12]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou,et al., “Chain-of-thought prompting elicits reasoning in large language models,”Advances in neural infor- mation processing systems, vol. 35, pp. 24824–24837, 2022

    2022

  5. [13]

    Reflexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” Advances in Neural Information Processing Systems, vol. 36, pp. 8634–8652, 2023

    2023

  6. [14]

    Toolformer: Language models can teach themselves to use tools,

    T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” Advances in Neural Information Processing Systems, vol. 36, pp. 68539–68551, 2023

    2023

  7. [15]

    Llm-based multi-agent sys- tems for software engineering: Literature review, vision and the road ahead,

    J. He, C. Treude, and D. Lo, “Llm-based multi-agent sys- tems for software engineering: Literature review, vision and the road ahead,”ACM Transactions on Software Engineering and Methodology, 2024

    2024

  8. [16]

    Improvingfactualityandreasoninginlanguagemodelsthrough multiagent debate,

    Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch, “Improvingfactualityandreasoninginlanguagemodelsthrough multiagent debate,” inForty-first International Conference on Machine Learning, 2023

    2023

  9. [17]

    Hierarchical auto-organizing system for open-ended multi-agent navigation,

    Z. Zhao, K. Chen, D. Guo, W. Chai, T. Ye, Y. Zhang, and G. Wang, “Hierarchical auto-organizing system for open-ended multi-agent navigation,” arXiv preprint arXiv:2403.08282 , 2024

    2024 arXiv

  10. [18]

    Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration,

    Z. Wang, S. Mao, W. Wu, T. Ge, F. Wei, and H. Ji, “Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration,” arXiv preprint arXiv:2307.05300, 2023

    2023 arXiv

  11. [19]

    Delegating responsibility benefits or- ganizations)

    lbmc, “Delegating responsibility benefits or- ganizations).” https://www.lbmc.com/blog/ delegating-responsibility-benefits-organizations/, 2023. Accessed: 2025-05-02

    2023

  12. [20]

    Organization design: An information process- ing view,

    J. R. Galbraith, “Organization design: An information process- ing view,”Interfaces, vol. 4, no. 3, pp. 28–36, 1974

    1974

  13. [21]

    On the resilience of multi-agent systems with malicious agents,

    J.-t. Huang, J. Zhou, T. Jin, X. Zhou, Z. Chen, W. Wang, Y. Yuan, M. Sap, and M. R. Lyu, “On the resilience of multi-agent systems with malicious agents,” arXiv preprint arXiv:2408.00989, 2024

    2024 arXiv

  14. [22]

    A scop- ing review exploring the impact and negotiation of hierarchy in healthcare organisations,

    R. Essex, J. Kennedy, D. Miller, and J. Jameson, “A scop- ing review exploring the impact and negotiation of hierarchy in healthcare organisations,” Nursing Inquiry, vol. 30, no. 4, p. e12571, 2023

    2023

  15. [23]

    Coordinating clinical teams: Using robots to empower nurses to stop the line,

    A. Taylor, H. R. Lee, A. Kubota, and L. D. Riek, “Coordinating clinical teams: Using robots to empower nurses to stop the line,”ProceedingsoftheACMonHuman-ComputerInteraction , vol. 3, no. CSCW, pp. 1–30, 2019

    2019

  16. [24]

    Towards col- laborative crash cart robots that support clinical teamwork,

    A. Taylor, T. Tanjim, H. Cao, and H. R. Lee, “Towards col- laborative crash cart robots that support clinical teamwork,” in Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, pp. 715–724, 2024

    2024

  17. [25]

    Decision-making under pressure: medical errors in uncertain and dynamic environments,

    A. M. Zavala, G. E. Day, D. Plummer, and A. Bamford-Wade, “Decision-making under pressure: medical errors in uncertain and dynamic environments,”Australian Health Review, vol. 42, no. 4, pp. 395–402, 2017

    2017

  18. [26]

    Why do multiagent systems fail?,

    M. Z. Pan, M. Cemri, L. A. Agrawal, S. Yang, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, K. Ramchandran, D. Klein, et al., “Why do multiagent systems fail?,” inICLR 2025 Workshop on Building Trust in Language Models and Applications, 2025

    2025

  19. [27]

    Knowledge - what is knowledge in crewai and how to use it

    crewai, “Knowledge - what is knowledge in crewai and how to use it..” https://docs.crewai.com/concepts/knowledge, 2025. Accessed: 2025-05-16

    2025

  20. [28]

    Umcp: A sound and complete procedure for hierarchical task-network planning.,

    K. Erol, J. A. Hendler, and D. S. Nau, “Umcp: A sound and complete procedure for hierarchical task-network planning.,” in Aips, vol. 94, pp. 249–254, 1994

    1994

  21. [29]

    Hierarchical planning: Relating task and goal decompo- sition with task sharing.,

    R. Alford, V. Shivashankar, M. Roberts, J. Frank, and D. W. Aha, “Hierarchical planning: Relating task and goal decompo- sition with task sharing.,” inIJCAI, pp. 3022–3029, 2016

    2016

  22. [30]

    Hierarchical reinforcement learning: A comprehensive survey,

    S. Pateria, B. Subagdja, A.-h. Tan, and C. Quek, “Hierarchical reinforcement learning: A comprehensive survey,”ACM Com- puting Surveys (CSUR), vol. 54, no. 5, pp. 1–35, 2021

    2021

  23. [31]

    Multi-robot collab- orative navigation with formation adaptation,

    Z. Deng, P. Gao, W. J. Jose, and H. Zhang, “Multi-robot collab- orative navigation with formation adaptation,”arXiv preprint arXiv:2404.01618, 2024

    2024 arXiv

  24. [32]

    Generalized mission planning for heteroge- neous multi-robot teams via llm-constructed hierarchical trees,

    P. Gupta, D. Isele, E. Sachdeva, P.-H. Huang, B. Dariush, K. Lee, and S. Bae, “Generalized mission planning for heteroge- neous multi-robot teams via llm-constructed hierarchical trees,” arXiv preprint arXiv:2501.16539, 2025

    2025 arXiv

  25. [33]

    Coherent: Collaboration of heterogeneous multi-robot system with large language models,

    K. Liu, Z. Tang, D. Wang, Z. Wang, X. Li, and B. Zhao, “Coherent: Collaboration of heterogeneous multi-robot system with large language models,”arXiv preprint arXiv:2409.15146, 2024

    2024 arXiv

  26. [34]

    Hyperagent: Generalist software engineering agents to solve coding tasks at scale,

    H. N. Phan, T. N. Nguyen, P. X. Nguyen, and N. D. Bui, “Hyperagent: Generalist software engineering agents to solve coding tasks at scale,”arXiv preprint arXiv:2409.16299, 2024

    2024 arXiv

  27. [35]

    Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems,

    S. Zhang, M. Yin, J. Zhang, J. Liu, Z. Han, J. Zhang, B. Li, C. Wang, H. Wang, Y. Chen,et al., “Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems,”arXiv preprint arXiv:2505.00212, 2025

    2025 arXiv

  28. [36]

    Retrieval-augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al., “Retrieval-augmented generation for knowledge-intensive nlp tasks,”Advances in neural information processing systems, vol. 33, pp. 9459–9474, 2020

    2020

  29. [37]

    Speculative rag: Enhancing retrieval augmented generation through draft- ing,

    Z. Wang, Z. Wang, L. Le, H. S. Zheng, S. Mishra, V. Perot, Y. Zhang, A. Mattapalli, A. Taly, J. Shang,et al., “Speculative rag: Enhancing retrieval augmented generation through draft- ing,” arXiv preprint arXiv:2407.08223, 2024

    2024 arXiv

  30. [38]

    A collaborative multi-agent approach to retrieval- augmented generation across diverse data,

    A. Salve, S. Attar, M. Deshmukh, S. Shivpuje, and A. M. Utsab, “A collaborative multi-agent approach to retrieval- augmented generation across diverse data,” arXiv preprint arXiv:2412.05838, 2024

    2024 arXiv

  31. [39]

    Ma-rag: Multi- agent retrieval-augmented generation via collaborative chain-of- thought reasoning,

    T. Nguyen, P. Chin, and Y.-W. Tai, “Ma-rag: Multi- agent retrieval-augmented generation via collaborative chain-of- thought reasoning,”arXiv preprint arXiv:2505.20096, 2025

    2025

  32. [40]

    Langchain

    LangChain, “Langchain.” https://www.langchain.com/, 2025. Accessed: 2025-05-02

    2025

  33. [41]

    Accessed: 2025-05-02

    crewai, “crewai.” https://docs.crewai.com/how-to/ hierarchical-process, 2025. Accessed: 2025-05-02

    2025

  34. [42]

    Large language model based multi-agent system augmented complex event processing pipeline for internet of multimedia things,

    T. Zeeshan, A. Kumar, S. Pirttikangas, and S. Tarkoma, “Large language model based multi-agent system augmented complex event processing pipeline for internet of multimedia things,” arXiv preprint arXiv:2501.00906, 2025

    2025 arXiv

  35. [43]

    Exploration of llm multi-agent ap- plication implementation based on langgraph+ crewai,

    Z. Duan and J. Wang, “Exploration of llm multi-agent ap- plication implementation based on langgraph+ crewai,”arXiv preprint arXiv:2411.18241, 2024

    2024 arXiv

  36. [44]

    Navigation bench- marking for autonomous mobile robots in hospital environ- ment,

    C. Rondoni, F. Scotto di Luzio, C. Tamantini, N. L. Taglia- monte, M. Chiurazzi, G. Ciuti, and L. Zollo, “Navigation bench- marking for autonomous mobile robots in hospital environ- ment,” Scientific Reports, vol. 14, no. 1, p. 18334, 2024

    2024

  37. [45]

    Tools - understanding and leveraging tools within the crewai framework for agent collaboration and task execution

    crewai, “Tools - understanding and leveraging tools within the crewai framework for agent collaboration and task execution..” https://docs.crewai.com/concepts/tools, 2025. Accessed: 2025- 05-16

    2025

  38. [46]

    Pc-agent: A hier- archical multi-agent collaboration framework for complex task automation on pc,

    H. Liu, X. Zhang, H. Xu, Y. Wanyan, J. Wang, M. Yan, J. Zhang, C. Yuan, C. Xu, W. Hu,et al., “Pc-agent: A hier- archical multi-agent collaboration framework for complex task automation on pc,”arXiv preprint arXiv:2502.14282, 2025

    2025 arXiv

  39. [47]

    Agentops.ai

    AgentOps.ai, “Agentops.ai.” https://www.agentops.ai/, 2025. Accessed: 2025-05-10

    2025

  40. [48]

    Openai’s 5 levels of ‘super ai’ (agi to outperform human capability)

    Jodie Cook, “Openai’s 5 levels of ‘super ai’ (agi to outperform human capability).” https: //www.forbes.com/sites/jodiecook/2024/07/16/ openais-5-levels-of-super-ai-agi-to-outperform-human-capability/,

    2024

  41. [49]

    Language grounded multi-agent reinforcement learning with human-interpretable communication,

    H. Li, H. Nourkhiz Mahjoub, B. Chalaki, V. Tadiparthi, K. Lee, E. Moradi Pari, C. Lewis, and K. Sycara, “Language grounded multi-agent reinforcement learning with human-interpretable communication,” Advances in Neural Information Processing Systems, vol. 37, pp. 87908–87933, 2024

    2024

  42. [50]

    The dynamics of reinforcement learning in cooperative multiagent systems,

    C. Claus and C. Boutilier, “The dynamics of reinforcement learning in cooperative multiagent systems,” AAAI/IAAI, vol. 1998, no. 746-752, p. 2, 1998

    1998

  43. [51]

    Llm-based multi-agent reinforcement learning: Current and future directions,

    C. Sun, S. Huang, and D. Pompili, “Llm-based multi-agent reinforcement learning: Current and future directions,”arXiv preprint arXiv:2405.11106, 2024

    2024 arXiv

  44. [52]

    Supervised learning-a systematic literature review,

    S. Dridi, “Supervised learning-a systematic literature review,” preprint, Dec, 2021

    2021

  45. [53]

    A review of multi-agent reinforcement learning algorithms,

    J. Liang, H. Miao, K. Li, J. Tan, X. Wang, R. Luo, and Y. Jiang, “A review of multi-agent reinforcement learning algorithms,” Electronics, vol. 14, no. 4, p. 820, 2025

    2025

  46. [54]

    Chatgpt for robotics: Design principles and model abilities,

    S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, “Chatgpt for robotics: Design principles and model abilities,” Tech. Rep. MSR-TR-2023-8, Microsoft, February 2023

    2023

  47. [55]

    Healthbench: Evaluating large lan- guage models towards improved human health,

    R. K. Arora, J. Wei, R. S. Hicks, P. Bowman, J. Quiñonero- Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Val- lone, A. Beutel, et al., “Healthbench: Evaluating large lan- guage models towards improved human health,”arXiv preprint arXiv:2505.08775, 2025. Appendix A Evaluatio...

    2025 arXiv

  48. [2024]

    Accessed: 2025-05-02

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.