Two Heads Are Better Than One: Collaborative LLM Embodied Agents for Human-Robot Interaction

Marc. G Carmichael; Mitchell Rosser

REVIEW 3 major objections 5 minor 12 references

Two Heads Are Better Than One: Collaborative LLM Embodied Agents for Human-Robot Interaction

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A coder-plus-reviewer pair of LLM agents beats both a single agent and a three-agent team at controlling a robot.

desk verdict A useful but underpowered pilot: two-agent LLM architecture beats one and three on a physical HRI task, but the rerun-and-rescore protocol makes the main positive claim shaky. read the letter →

arxiv 2411.16723 v1 pith:MSYHO64R submitted 2024-11-23 cs.MA cs.AIcs.RO

Mitchell Rosser , Marc. G Carmichael This is my paper · ORCID

classification cs.MAcs.AIcs.RO

keywords largelanguagemodelsmulti-agentsystemshuman-robotinteractioncodegenerationrobottaskplanningerrorrateabstractreasoningcollaborativeAI

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether teams of large-language-model agents controlling a robot outperform a single agent. Across seven natural-language tasks with a quadrupedal robot, it finds no general trend: more agents do not mean better performance. The two-agent configuration, a coder paired with a reviewer, had the fewest initial code-execution errors, 9 and 14 percentage points lower than the single agent and the three-agent planner-coder-reviewer configuration. The same configuration scored markedly higher on vague and abstract tasks, while the three-agent configuration never outperformed the others. The takeaway is that a reviewer loop can add reliability, but an extra planning stage can hurt.

What carries the argument

The central mechanism is a multi-agent group chat with role-separated prompts: a coder agent writes Python code, a reviewer agent critiques it for correctness, task completion, and safety, and a chat manager decides when the code is ready to execute. Configuration C inserts a planner agent that first converts the prompt into natural-language instructions for the coder. The reviewer loop is what carries the claimed benefit, while the planner stage is associated with degraded performance, possibly through longer context and constrained code generation.

What would settle it

Rerun the seven trials with a protocol that records observer scores for the first code attempt, including failures, with a larger panel of observers; if configuration B no longer outperforms A and C on the abstract tasks, the central claim fails. Alternatively, measure configuration C's error rate after trimming its system prompt; if errors persist, context saturation is not the primary cause.

Watch

Extended reading notes

Core claim

The paper's central claim is that in human-robot interaction, a collaborative two-agent system consisting of a coder and a reviewer is more reliable and better at abstract tasks than either a single agent or a three-agent planner-coder-reviewer system, and that there is no monotonic benefit from adding agents. The evidence is empirical: on the first code-execution attempt, configuration B failed 9 percentage points less often than configuration A and 14 percentage points less often than configuration C, and on the two most abstract prompts observers rated B markedly higher on success, safety, and sociability. The authors argue that the three-agent system's poor showing may stem from context-window saturation due to bloated token use, or from the planner's natural-language scaffold constraining the coder's flexibility.

Load-bearing premise

The comparison assumes that discarding trials whose first code attempt errors does not bias observer scores; if observers infer the configuration from the rerun, or if reruns inflate ratings, the claimed advantage of the two-agent configuration on abstract tasks is unreliable.

Editorial extensions

If this is right

A two-agent coder-reviewer architecture can serve as a reliability layer for LLM-driven robot control without requiring an explicit planning stage.
Agent count is not a proxy for capability: adding a planner before the coder can make performance worse, not better.
Token usage and context length should be treated as first-order design variables when composing agent teams, since they may explain why more agents fail.
For vague or abstract commands, a reviewer agent may provide the largest gains, while for simple concrete commands a single agent may be enough.
The error-rate advantage of configuration B suggests that the reviewer loop catches coding mistakes that a lone agent lets through.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the context-saturation explanation is right, shortening system prompts or using retrieval-augmented generation should reduce configuration C's error rate; this is a directly testable extension.
The value of a reviewer agent may generalize beyond human-robot interaction to any embodied code-generating LLM setting, because the reviewer catches mismatches between generated code and the environment.
The observer protocol that discards failed first attempts before scoring may have compressed the measured performance gaps; scoring first attempts as-is would likely show a larger advantage for the two-agent configuration.
The results suggest an inverted-U relationship between the number of agents and task performance in embodied settings, which could be probed with four or more agents.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a small, honest empirical study of LLM agent architectures for a physical HRI task, and its most useful result is a caution: more agents is not monotonically better. The two-agent configuration (coder plus reviewer) had the lowest first-attempt error rate and scored higher on vague and abstract prompts, but the evidence for that strong claim is thinner than the prose suggests.

What is actually new: prior multi-agent work targeted code generation, mechanics problems, and supply-chain optimization. Testing AutoGen-style collaboration on a real Spot robot with blind observers is a reasonable next step, and the non-monotonic result—the three-agent configuration with a planner was worse, not better—is a useful empirical counterweight to the enthusiasm for stacking agents. The paper is clearly written, the three configurations are described well enough to reproduce the setup, and the authors explicitly acknowledge the limited data and the possibility that perception or locomotion failures influenced scores.

Where it is soft: the headline error-rate difference is built on 21 initial executions per configuration, and a 9–14 percentage point gap is two or three trials. There are no confidence intervals or significance tests, so 'clearly least error prone' outruns the data. More load-bearing, Section 3.3 says that when code errors, the observer records nothing and the code is regenerated until a score can be recorded. That makes every observer rating conditional on a successful rerun, and the number of regeneration attempts is neither logged nor controlled. The claimed B advantage on the abstract tasks (Trials 3 and 5) could therefore be an artifact of differential regeneration counts—C, with the highest initial error rate, may have been scored on a later, more corrected output distribution than B. Blindness is also compromised if observers watch repeated failures. This is fixable by logging retries and scoring first-attempt behavior, but as it stands the central positive claim is not established. The token-saturation explanation in the discussion is post hoc speculation, though the authors flag it as such.

Bottom line: this is a reasonable descriptive pilot, not a confirmed result. It deserves peer review because the question is genuine and the field needs more head-to-head empirical comparisons of agent architectures, but it needs heavy revision: raw data, exact prompts, regeneration counts, and proper uncertainty quantification. I would bring it to a reading group as a methodological cautionary tale.

Referee Report

3 major / 5 minor

Summary. The paper empirically compares three LLM-based agent architectures for controlling a quadrupedal robot through natural-language commands: configuration A (a single coder agent), configuration B (coder plus reviewer), and configuration C (planner, coder, and reviewer). Each configuration was tested on seven human-robot interaction prompts, with three repetitions per configuration, during which observer ratings of task success, safety, sociability, and expectation matching were collected, along with code-execution error rates and usage metrics (time and tokens). The reported findings are that there is no overall trend with the number of agents, but configuration B is the least error-prone and shows markedly higher observer-rated performance on the more abstract/vague prompts (Trials 3 and 5), while configuration C performs worst on initial error rate and never best on observer scores. A small case study with a single technical reviewer on anonymized code samples is also presented.

Significance. If the empirical claims were fully supported, the paper would make a useful contribution to the emerging literature on multi-agent LLM systems for embodied human-robot interaction, suggesting that adding a reviewer agent improves reliability and abstract problem-solving while adding a planner agent does not. The work has tangible strengths: the experimental setup is concrete and reproducible through the open 'spottyai' library, the prompts and trial categories are clearly tabulated, the code example in Figure 4 is informative, and the authors acknowledge several limitations in the Discussion. The core comparison is, however, small in scale (21 initial executions per configuration for error rates, three observer ratings per bar) and is affected by a protocol issue in Section 3.3 that conditions the observer scores on successful code regeneration. These concerns are load-bearing for the headline claims, so the contribution cannot yet be regarded as established.

major comments (3)

[§3.3, Figures 6 and 9] The rerun-and-rescore protocol is a load-bearing threat to the central claim. Section 3.3 states that if the generated code errors, the participant records nothing and the code is regenerated until a score can be recorded. Since configuration C has the highest initial error rate (Figure 6), its observer-rated trials in Figures 8–10 are drawn from a later, more heavily corrected part of its output distribution than configuration B's trials. The conclusion that configuration B shows 'markedly higher performance' on the abstract Tasks 3 and 5 (Figure 9) therefore conflates architecture quality with the number of free regeneration attempts each configuration received before scoring. The number of regenerations is not logged. I recommend reporting the regeneration counts per configuration and either scoring first-attempt outputs for all configurations or including the regeneration count as a covariate in the analysis.
[§4, Figure 6] The error-rate comparison is based on only 21 initial executions per configuration (three repetitions across seven trials). A 9–14 percentage point difference corresponds to just 2–3 error trials per configuration. The paper provides no confidence intervals, exact counts, or statistical test, yet states that configuration B is 'clearly least error prone' (Section 4, first paragraph). Please report the underlying contingency table and, at minimum, a two-sided Fisher exact test p-value or an exact binomial confidence interval for each proportion so that the reader can judge the strength of this conclusion.
[§3.2, Figures 8–10] The observer-rated comparisons rest on three participants per configuration/trial, all self-reported as having low familiarity with AI and robotics. The claims of 'markedly higher performance' on Trials 3 and 5 and 'best performance' on Trial 2 are descriptive statements over n=3 per bar, with no reported inter-rater reliability, no error bars, and no inferential statistic. The paper should either show the individual participant scores, provide a measure of rating dispersion, or explicitly label these as illustrative observations rather than as evidence of a systematic performance advantage.

minor comments (5)

[§3.3] The statement that 'all code was pre-generated for all of the configurations and trials' is difficult to reconcile with the subsequent sentence explaining that errored code is regenerated; please clarify whether pre-generation involved multiple attempts per trial and whether observers were present during regeneration.
[§3.1] The paper does not specify the exact ChatGPT-4 model version, sampling temperature, or other generation parameters; adding these would aid reproducibility.
[§3.3] The protocol describes the observer feedback as blind, but it does not state whether observers knew that multiple AI configurations were being compared or whether they could infer errors from the instruction to record nothing; this should be made explicit.
[§3.4] The case study uses two attempts per configuration across two prompts, reviewed by one technically proficient person; the quoted qualitative conclusions should be presented as illustrative, since they are based on a very small sample.
[§5] The statement that multi-agent chat architecture 'seems to almost exponentially increase input token usage' is not supported by the data in Figure 7(c), which shows only three architecture points; consider softening this claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: this is an empirical comparison with no fitted parameters or self-derived predictions.

full rationale

The paper contains no derivation chain whose output is equivalent to its inputs. All central claims are observational comparisons of three LLM agent configurations on error rates, observer ratings, and usage characteristics. There are no fitted parameters, no equations that reduce to one another, and no quantity is defined in terms of the outcome it is used to predict. The only self-citation is [Rosser, 2024], which provides the spottyai robot-control library and the detailed system prompts used to instantiate the agent roles; this is implementation tooling rather than evidence for the experimental conclusion, and no result from that repository is invoked to justify the paper's findings. The abstract's claim about error-rate differences is a direct reading of Figure 6, and the claim about configuration B's performance on abstract tasks is a direct reading of Figure 9. The rerun-and-rescore protocol in Section 3.3, in which errored code is regenerated before an observer score is recorded, is a genuine methodological threat to validity because observer ratings are conditional on successful regeneration and regeneration counts are not logged. That limitation affects whether the compared scores are unbiased estimates of first-attempt capability, but it does not make any result circular: the outcome is not defined in terms of the protocol, and the protocol does not force a particular configuration to win. The small sample size and lack of statistical testing likewise weaken the strength of the conclusions but do not constitute circular reasoning. No step in the paper reduces, by definition or by self-citation, to its own inputs, so the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical and uses no fitted constants or invented entities. It rests on assumptions about the validity of observer scoring, the retry protocol, uniform distribution of platform failures, and consistency of LLM behavior across repetitions.

assumptions (4)

domain assumption Observer ratings of task success, safety, and sociability are reliable and comparable across configurations.
Each repetition uses a different untrained observer and ratings are scored on 1-5 scales; no inter-rater reliability or validation of these scales is reported (Section 3.2).
ad hoc to paper Discarding trials with initial code errors and regenerating code before observer scoring does not distort the relative performance ratings.
The protocol in Section 3.3 is specific to this study and is assumed to preserve comparability, but it removes failed attempts from exactly the measurements used to compare success, safety, and sociability.
domain assumption Failures in the robot's perception and locomotion systems are spread evenly across configurations and do not confound the comparison.
The discussion acknowledges such failures but asserts without evidence that they 'should not have an impact on the overall results' (Section 5).
domain assumption ChatGPT-4 as configured through the system prompts produces sufficiently consistent behavior across the three repetitions for per-cell comparisons.
No temperature, random seed, or sampling parameters are reported, yet three repetitions are used as the sampling unit (Section 3.3).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Two Heads Are Better Than One: Collaborative LLM Embodied Agents for Human-Robot Interaction." pith.science (2026). https://pith.science/paper/MSYHO64R

@misc{pith2026241116723,
  author       = {Pith},
  title        = {Pith review of: Two Heads Are Better Than One: Collaborative LLM Embodied Agents for Human-Robot Interaction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MSYHO64R}},
  note         = {Machine review of arXiv:2411.16723}
}

read the original abstract

With the recent development of natural language generation models - termed as large language models (LLMs) - a potential use case has opened up to improve the way that humans interact with robot assistants. These LLMs should be able to leverage their large breadth of understanding to interpret natural language commands into effective, task appropriate and safe robot task executions. However, in reality, these models suffer from hallucinations, which may cause safety issues or deviations from the task. In other domains, these issues have been improved through the use of collaborative AI systems where multiple LLM agents can work together to collectively plan, code and self-check outputs. In this research, multiple collaborative AI systems were tested against a single independent AI agent to determine whether the success in other domains would translate into improved human-robot interaction performance. The results show that there is no defined trend between the number of agents and the success of the model. However, it is clear that some collaborative AI agent architectures can exhibit a greatly improved capacity to produce error-free code and to solve abstract problems.

Figures

Figures reproduced from arXiv: 2411.16723 by the authors.

**Figure 2.** Architecture of Multi-Agent Cooperating AI sys [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Testing Flowchart. Trial prompts are provided to [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Python code output from configuration B on Trial [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]

**Figure 5.** Figure 5: AI system architecture diagram for configs A, B and [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

**Figure 6.** Figure 6: Error rate of configs A B and C. The graph shows [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

**Figure 8.** Figure 8: Observer provided scores for configs A, B and C [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]

**Figure 10.** Figure 10: Observer provided scores for configs A, B and C on [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 8 canonical work pages

[1]

[Ahn et al. , 2022] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrish- nan, Karol Hausman, Alex Herzog, Daniel Ho, Jas- mine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kala...

work page 2022
[4]

[Liu et al

arXiv:2209.07753 [cs]. [Liu et al. , 2024] Nelson F. Liu, Kevin Lin, John He- witt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the Middle: How Language Models Use Long Contexts. Trans- actions of the Association for Computational Linguis- tics, 12:157–173, February

arXiv 2024
[6]

sheep- skins/spottyai, September

[Rosser, 2024] Mitchell Rosser. sheep- skins/spottyai, September

work page 2024
[7]

[Sorin and Klang, 2023 ] Vera Sorin and Eyal Klang

https://github.com/sheepskins/spottyai. [Sorin and Klang, 2023 ] Vera Sorin and Eyal Klang. Large language models and the emergence phenom- ena. European Journal of Radiology Open, 10, January

work page 2023
[8]

[Staniaszek et al

Publisher: Elsevier. [Staniaszek et al. , 2024] Michal Staniaszek, Esther, Dave Niewinski, Koki Shinjo, maubrunn, Chris Iverach-Brereton, Yoshiki Obinata, Kei Okada, Michel Heinemann, Naoya Yamaguchi, Harel Biggie, jeremysee2, Naoto Tsukamoto, Juan Miguel Jimeno, Kenji Brameld (TRACLabs), Lucas Walter, Luke, Mario Gini, matthew rt, Victor Mittermair, Tobi...

work page 2024
[9]

‘Fighting fire with fire’ — using LLMs to combat LLM hallucinations

[Verspoor, 2024] Karin Verspoor. ‘Fighting fire with fire’ — using LLMs to combat LLM hallucinations. Na- ture, 630(8017):569–570, June

work page 2024
[10]

[Wu et al

Bandiera abtest: a Cg type: News And Views Publisher: Nature Pub- lishing Group Subject term: Machine learning, Com- puter science. [Wu et al. , 2023] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xi- aoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Has- san Awadallah, Ryen W. White, Doug Burger, and Chi Wang. AutoGen: Enab...

work page 2023
[11]

[Zan et al

arXiv:2308.08155 [cs]. [Zan et al. , 2022] Daoguang Zan, Bei Chen, Zeqi Lin, Bei Guan, Yongji Wang, and Jian-Guang Lou. When Language Model Meets Private Library, October

arXiv 2022

Show all 12 references

[12]

arXiv:2210.17236 [cs]

arXiv
[2022]

[Firoozi et al. , 2023] Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, Brian Ichter, Danny Driess, Jiajun Wu, Cewu Lu, and Mac Schwager. Foundation Mod- els in Robotics: Applications...

2023
[2023]

[Liang et al

arXiv:2312.07843 [cs]. [Liang et al. , 2023] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Flo- rence, and Andy Zeng. Code as Policies: Language Model Programs for Embodied Control, May

2023 arXiv
[2024]

[Ni and Buehler, 2024 ] Bo Ni and Markus J. Buehler. MechAgents: Large language model multi-agent col- laborations can solve mechanics problems, generate new data, and integrate knowledge. Extreme Mechan- ics Letters, 67:102131, March

2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

[Ahn et al. , 2022] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrish- nan, Karol Hausman, Alex Herzog, Daniel Ho, Jas- mine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kala...

work page 2022

[2] [4]

[Liu et al

arXiv:2209.07753 [cs]. [Liu et al. , 2024] Nelson F. Liu, Kevin Lin, John He- witt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the Middle: How Language Models Use Long Contexts. Trans- actions of the Association for Computational Linguis- tics, 12:157–173, February

arXiv 2024

[3] [6]

sheep- skins/spottyai, September

[Rosser, 2024] Mitchell Rosser. sheep- skins/spottyai, September

work page 2024

[4] [7]

[Sorin and Klang, 2023 ] Vera Sorin and Eyal Klang

https://github.com/sheepskins/spottyai. [Sorin and Klang, 2023 ] Vera Sorin and Eyal Klang. Large language models and the emergence phenom- ena. European Journal of Radiology Open, 10, January

work page 2023

[5] [8]

[Staniaszek et al

Publisher: Elsevier. [Staniaszek et al. , 2024] Michal Staniaszek, Esther, Dave Niewinski, Koki Shinjo, maubrunn, Chris Iverach-Brereton, Yoshiki Obinata, Kei Okada, Michel Heinemann, Naoya Yamaguchi, Harel Biggie, jeremysee2, Naoto Tsukamoto, Juan Miguel Jimeno, Kenji Brameld (TRACLabs), Lucas Walter, Luke, Mario Gini, matthew rt, Victor Mittermair, Tobi...

work page 2024

[6] [9]

‘Fighting fire with fire’ — using LLMs to combat LLM hallucinations

[Verspoor, 2024] Karin Verspoor. ‘Fighting fire with fire’ — using LLMs to combat LLM hallucinations. Na- ture, 630(8017):569–570, June

work page 2024

[7] [10]

[Wu et al

Bandiera abtest: a Cg type: News And Views Publisher: Nature Pub- lishing Group Subject term: Machine learning, Com- puter science. [Wu et al. , 2023] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xi- aoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Has- san Awadallah, Ryen W. White, Doug Burger, and Chi Wang. AutoGen: Enab...

work page 2023

[8] [11]

[Zan et al

arXiv:2308.08155 [cs]. [Zan et al. , 2022] Daoguang Zan, Bei Chen, Zeqi Lin, Bei Guan, Yongji Wang, and Jian-Guang Lou. When Language Model Meets Private Library, October

arXiv 2022