{"id":"e93fd37d-807f-47ef-b88a-669913efa2c5","arxiv_id":"2504.17950","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new Minecraft benchmark shows current LLM agents lose over 15% performance when they must communicate step-by-step plans in collaborative tasks.","lead":"This paper introduces a Minecraft-based platform and benchmark for testing how well LLM agents collaborate in embodied tasks. It reports that current models struggle most when they must communicate detailed plans, with performance dropping about 15%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 15% 'communication bottleneck' drop is not isolated: the plan-visibility ablations (Table 4, Fig 3c/d) change information access, not communication cost, and with n=11-14 the reported differences are within ~1-2 trials.","rationale":"The central claim is a causal statement about the primary bottleneck in multi-agent embodied collaboration. Its strongest evidence is the plan-visibility ablation, which compares full-information agents with agents that lack part of the task plan. The reader's weakest-assumption analysis correctly identifies that this comparison changes information access, not communication cost. I agree, and the quantitative detail strengthens the point: per-condition n is 11-14 in Table 4, so a 15% drop is about one or two trials; several model-condition pairs do not drop at all, and the cooking table shows non-monotonic patterns. A full-information-with-bottleneck control would isolate the communication variable. The concern is load-bearing because the abstract's headline conclusion, that existing LLM agents are ill-optimized for collaboration because they cannot efficiently share information, would be false if the drop is caused by information asymmetry or by general incompetence at the task. Since the reader already assigned CONDITIONAL based on this and related weaknesses, my stress-test does not move the verdict; it reinforces the need for the requested revisions before the claim is accepted.","tokens_in":22718,"tokens_out":9166,"duration_ms":93723,"concrete_test":"Run the crafting plan ablation (Table 4) with two new cells, matched for n>=50 and identical task distributions: (A) Full Plan with a hard communication bottleneck, e.g., each agent may send at most 3 messages per episode or each message is truncated to 80 tokens; (B) Full Plan with no extra constraint. If cell A reproduces the ~15% drop relative to cell B, the communication-burden explanation is supported. If not, the original Partial-Plan drop is attributable to the information asymmetry itself, and the headline claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim that 'the primary bottleneck in collaborating effectively ... is efficient natural language communication' rests on the plan-visibility ablations in Section 5 (Figure 3c/3d, Tables 4-5). The manipulated variable is not communication cost but information access: in the crafting Partial-Plan condition (Table 4) one agent is denied the crafting recipe, and in Hell's Kitchen one or both agents lack the recipe for the item they must produce. An agent lacking the recipe cannot proceed until it obtains the missing content, so the observed drop conflates information asymmetry with the burden of communicating. The pattern is also not monotonic in the direction predicted by a communication-burden account: in cooking, GPT-4o succeeds more when both agents are blocked (83.3%) than when one is blocked (58.3%), and in crafting LLaMA-3.3-70B and LLaMA-8B-SFT improve slightly under Partial Plan. Compounding this, per-cell sample sizes in Table 4 are n=14 and n=11; the headline 'over 15%' drops (Claude: 85.7% to 72.7%; GPT-4o: 35.7% to 27.3%) correspond to 1-2 trial flips and are reported without confidence intervals or significance tests. Without a condition that holds information constant and varies only the communication channel's capacity, the data show that agents perform worse when they lack information that must be requested and transmitted; they do not establish communication efficiency as the primary bottleneck.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MINDcraft, a plugin-based Minecraft platform for LLM agents, and MineCollab, a benchmark of cooking, crafting, and construction tasks. It evaluates several closed and open LLMs (GPT-4o, Claude 3.5 Sonnet, LLaMA-3.3-70B-Instruct, LLaMA-3-8B, and an SFT variant) on these tasks, ablates agent count, plan visibility, and blueprint complexity, and trains an 8B model on oracle-generated SFT data. The authors' central claim is that natural-language communication is the primary bottleneck in multi-agent embodied collaboration, with performance dropping by up to 15% when agents must communicate detailed plans.","tokens_in":22998,"tokens_out":7619,"duration_ms":65448,"significance":"The platform and benchmark are timely and potentially widely useful: the procedural train/test split, the open-weight SFT pipeline, and the qualitative failure analyses are concrete contributions. If the communication-bottleneck claim were cleanly established, it would redirect research toward inter-agent communication efficiency rather than individual task competence. However, the claim currently rests on an information-access confound and on very small samples, so the paper's headline conclusion is not yet established.","major_comments":[{"comment":"The plan-visibility manipulation changes information access, not communication efficiency. In the crafting Partial Plan condition one agent is denied the crafting plan, and in the Hell's Kitchen variant agents lack recipes for required dishes; an agent without the plan cannot proceed until the missing content is transmitted. The observed performance drops therefore conflate the absence of information with the cost of communicating it. The pattern is also not monotonic in the way a communication-burden account predicts: Table 5 shows GPT-4o succeeding more when both agents are blocked (83.3%) than when one is blocked (58.3%), and Table 4 shows LLaMA-3.3-70B and LLaMA-8B-SFT improving slightly under Partial Plan (35.71%→36.36% and 42.86%→45.45%, respectively). Please add a condition that holds the information content constant and varies only the communication channel's capacity (e.g., message-length limits, added latency, or shared vs. private state) to isolate communication efficiency, or substantially soften the headline claim.","section":"Section 5 (Figure 3c/3d; Tables 4 and 5)"},{"comment":"The reported differences are within sampling noise. Per-cell sample sizes are n=14 and n=11 in Table 4, so the headline drops (Claude: 85.71%→72.73%; GPT-4o: 35.71%→27.27%) correspond to one or two trial flips. No confidence intervals, bootstrap estimates, or significance tests are reported for any ablation in Section 5, including Table 7. Please report per-cell trial counts and uncertainty measures, and interpret effect sizes accordingly.","section":"Tables 4-5 and Figure 3"},{"comment":"The construction evaluation metric is ambiguous. Section 4 defines construction scoring as an edit-distance metric and Table 3's caption calls the construction row 'the average of those edit distance scores,' yet Figure 3e/3f and Table 6 label the same construction evaluations as 'Success Rate,' and the Introduction interprets the construction numbers as 'percentage of blocks placed.' These are different quantities with opposite direction of goodness. Please state the precise metric, use consistent labels across all tables and figures, and report both the raw distance and a threshold-based success rate if both are intended.","section":"Section 4, Table 3, Table 6, Figure 3e/3f"}],"minor_comments":[{"comment":"The phrase 'gold truth crafting plan' should be 'ground truth crafting plan.'","section":"Section 5"},{"comment":"Please report per-cell sample sizes for the No Agents Blocked / 1 Agent Blocked / Both Agents Blocked conditions.","section":"Table 5"},{"comment":"The caption says 'Using llama 70B' but the table does not state the number of trials per ablation; please add n and clarify whether this is the base LLaMA-3.3-70B-Instruct model.","section":"Table 7"},{"comment":"The 'Success' column is ambiguous; clarify that these counts are the number of successful runs retained for SFT construction, not success rates.","section":"Table 2"},{"comment":"The text states the tool library contains 47 parameterized tools, but Table 8 in Appendix 12 appears to list fewer; reconcile the count or note that some tools are omitted.","section":"Section 3.1"},{"comment":"Reference [13] is also titled 'MindCraft'; the name collision with this paper's MINDcraft should be explicitly acknowledged.","section":"References"},{"comment":"The 'Partial Obs' column is checked for all platforms, but the observation mechanisms differ widely (tool queries vs. full-state access); add a short definition in the caption or table notes.","section":"Table 1"},{"comment":"The caption says 'models drop by over 15% when forced to communicate these plans'; this should be rephrased to 'when denied access to these plans' to match the actual manipulation.","section":"Figure 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The platform and benchmark are valuable community contributions, and the SFT results are interesting. My main concern is that the abstract's headline claim is currently stronger than the evidence. I would encourage the editor to ask for either the proposed information-held-constant control condition or a careful rewriting of the central claim before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is worth a read, but do not quote the abstract's 15% claim without reading the ablation tables. MineCollab is a real asset for the community, and the paper is honest about its own scaffolding. The communication-bottleneck conclusion, however, is not supported by the data as presented.\n\nWhat is genuinely new: combining cooking, crafting, and construction in one Minecraft-based suite that requires multi-turn chat, partial observability, long horizons, and quantitative scoring. I don't know of another platform that hits all of those simultaneously. The SFT pipeline and released dataset (2,000 trials generated with LLaMA-70B, filtered to roughly 200 successful runs, 16k examples) is a concrete reproducibility contribution. The appendix failure transcripts are useful qualitative evidence. The authors also deserve credit for openly acknowledging that MINDcraft predates this paper as hobbyist open-source software, and for pinning their evaluations to a frozen version.\n\nThe main soft spot is the headline claim that the primary bottleneck in collaboration is efficient natural language communication. The hidden-plan manipulations do not isolate communication cost. In crafting Partial Plan, one agent simply lacks the recipe; in Hell's Kitchen, one or both agents lack the recipe. An agent without a recipe cannot proceed until it acquires missing information, so the observed drop conflates information asymmetry with the burden of communicating. The pattern is also not monotonic in the direction a communication-burden account predicts: GPT-4o is better when both agents are blocked (83%) than when one is blocked (58%), and LLaMA-3.3-70B slightly improves under Partial Plan. With per-cell n of 11-14, the headline drops are one- or two-trial flips, reported without confidence intervals or significance tests. A cleaner experiment would hold information constant and vary only the communication channel's capacity. Table 7's 0% success with no communication is a different ablation and does not rescue the inference.\n\nA secondary concern: the SFT models are trained only on successful oracle runs. That is a defensible distillation choice, but it makes the \"improves performance by 17%\" numbers hard to compare fairly against base models, and the paper does not dwell on this selection effect.\n\nThe citation pattern looks fair; they position against PARTNR, MineLand, Overcooked, CerealBar, and Voyager, and they acknowledge their own platform's pre-existing status. Who is this for? Researchers building embodied multi-agent LLM benchmarks, and practitioners who want a drop-in Minecraft platform with a large generated dataset. The benchmark and qualitative failure modes are the durable value; the quantitative headline should be treated with caution until the confound is addressed. This deserves serious peer review: send it out, but expect a revise-and-resubmit that either reframes the claims or adds an information-constant communication-bottleneck condition.","headline":"MineCollab is a genuinely useful benchmark and the paper is clearly written, but the headline 'communication bottleneck' claim rests on ablations that change information access rather than communication cost.","tokens_in":23571,"tokens_out":2290,"would_cite":true,"duration_ms":26564,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that for current LLM agents, the main barrier to embodied teamwork is inefficient natural-language communication, not missing task skill.","keywords":["multi-agent collaboration","embodied reasoning","large language models","Minecraft","natural language communication","benchmark","supervised fine-tuning"],"falsifier":"Give every agent the full recipe in its prompt but restrict messages to a fixed small budget (or force strictly alternating one-at-a-time turns); if success stays equal to free-chat performance, communication cost is not the cause of the 15% drop. Alternatively, give one agent the recipe, allow unrestricted chat, then delete the recipe from that agent's memory mid-episode; if success collapses, the bottleneck is retained information, not transmission.","tokens_in":22496,"feed_emoji":"🗣️","tokens_out":7200,"duration_ms":69762,"temperature":0.7,"pith_summary":"Current large language models can perform individual embodied tasks, but this paper argues that when they must work together, the scarce resource is not skill—it is efficient natural-language communication. To study this, the authors build MINDcraft, a Minecraft platform that gives agents 47 high-level tools and pairwise chat, and MineCollab, a benchmark spanning cooking, crafting, and construction with teams of two to five agents. Across several leading models, success drops by more than 15% when teams are forced to transfer a detailed step-by-step plan, and scaling from two to five agents can lower success from about 90% to below 30%. The paper also shows that fine-tuning a small model on successful collaborative trajectories brings it close to a much larger model, which it reads as evidence that current prompting and imitation methods are not enough for collaboration.","feed_headline":"LLM agents fail at teamwork because talking is the bottleneck","feed_subtitle":"Minecraft trials show success drops up to 15 percent when agents must explain step-by-step plans to each other.","key_machinery":"The load-bearing mechanism is MineCollab's plan-visibility ablation, implemented inside MINDcraft's agent architecture. MINDcraft gives each agent 47 parameterized tools, such as !givePlayer, !getCraftingPlan, and !checkBlueprintLevel, plus a pairwise conversation manager, so task failures cannot be blamed on low-level movement or syntax. The decisive conditions are Full Plan, where every agent sees the recipe; Partial Plan, where one agent must obtain it from a partner; and Hell's Kitchen, where each agent holds only part of the recipe; these are combined with team-size sweeps and blueprint-complexity sweeps to separate having information from being forced to communicate it.","core_discovery":"The paper claims that existing LLM agents are ill-optimized for multi-agent collaboration, especially in embodied settings, and that the main bottleneck is efficient natural-language communication rather than lack of underlying task competence. The evidence is a set of controlled ablations: hiding the recipe or crafting plan from one agent and forcing the team to ask for and explain it lowers success across all tested models by over 15%; increasing team size from two to five agents drops success rates from up to 90% to under 30%; and in blueprint construction the best model still places less than 40% of required blocks. A related result is that an 8B model trained by behavior cloning on filtered successful runs of a 70B model roughly matches or exceeds the larger models on crafting and construction, suggesting the platform's data is usable for improving smaller agents.","pith_inferences":["A sharper test of the communication-bottleneck claim would keep task information identical across agents and throttle only the channel—for example, capping message length or forcing strictly alternating turns; if success does not drop, the 15% effect is about missing knowledge, not communication cost.","The pairwise chat design implies coordination cost likely grows superlinearly with team size, so a broadcast or shared-blackboard protocol is a natural next experiment the paper does not run.","The hidden-plan design transfers to human-AI collaboration: one would predict that splitting instructions between a person and an agent and forcing verbal transfer degrades joint performance more than giving both parties the full instructions.","Because construction trajectories last over 100 steps on average and failures compound—agents undo each other's work—the communication penalty may be larger than 15% on long-horizon tasks even though the paper reports the main drop on shorter cooking and crafting tasks."],"forward_implications":["If communication is the bottleneck, then improving the message layer—what agents say, when, and how briefly—should help current systems more than adding task-specific planning skills.","Naively adding agents hurts: the two-to-five-agent curves drop from up to 90% to under 30% success, so parallelization without coordination-load management is counterproductive.","Filtered successful trajectories are a usable training signal: a small model fine-tuned on them matches or beats much larger models on crafting and construction.","Plan visibility should be reported as a standard axis in collaboration benchmarks, because in several conditions it changes success more than the choice of model."],"supporting_citations":[{"why":"supplies the cooking-task paradigm and the human-AI coordination baseline this benchmark extends.","marker":"[3]"},{"why":"provides the situated instruction-following collaboration setting that MineCollab's dialogue tasks build on.","marker":"[4]"},{"why":"defines the embodied multi-agent planning benchmark that MineCollab compares against for long-horizon tasks.","marker":"[8]"},{"why":"contributes the high-level skill and tool abstraction in Minecraft that MINDcraft's 47-command library adapts.","marker":"[11]"},{"why":"supplies the chat-driven embodied agent design that motivates the communication evaluation in Section 5.","marker":"[21]"},{"why":"provides the efficiency-optimization baseline for multi-agent LLM systems that the paper contrasts with standard prompting.","marker":"[22]"},{"why":"defines the low-level MineRL action space that MINDcraft's high-level commands abstract away, framing the tool design.","marker":"[27]"},{"why":"underlies the JavaScript bot-control layer on which all MINDcraft command execution runs.","marker":"[29]"},{"why":"motivates the retrieval-augmented few-shot example selection used to prompt agents within the platform.","marker":"[30]"}],"fun_headline_variants":["LLM agents flop at teamwork because they can't explain plans","Minecraft shows LLM agents fail at teamwork when they must talk","Explaining tasks costs LLM agents 15% success in Minecraft","The talk bottleneck: LLM agents flounder in embodied teamwork","For LLM agents, collaboration fails at the language step"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's interpretation of the plan-visibility results depends on treating the hidden-recipe conditions as a measure of communication burden, but those conditions also withhold the recipe from one agent, so the performance drop could be caused by missing information itself rather than by the cost of transmitting it.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents flop at teamwork because they can't explain plans","Minecraft shows LLM agents fail at teamwork when they must talk","Explaining tasks costs LLM agents 15% success in Minecraft","The talk bottleneck: LLM agents flounder in embodied teamwork","For LLM agents, collaboration fails at the language step"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000469,"raw_usage":{"total_tokens":2302,"prompt_tokens":875,"completion_tokens":1427,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":1339}},"tokens_in":491,"tokens_out":1427,"duration_ms":11233,"temperature":1.0,"reasoning_tokens":1339,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:28:06.383343+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give every agent the full recipe in its prompt but restrict messages to a fixed small budget (or force strictly alternating one-at-a-time turns); if success stays equal to free-chat performance, communication cost is not the cause of the 15% drop. Alternatively, give one agent the recipe, allow unrestricted chat, then delete the recipe from that agent's memory mid-episode; if success collapses, the bottleneck is retained information, not transmission.","supporting_citations":[{"cited_title":"Executing instructions in situated collaborative interactions","cited_arxiv_id":null,"evidence_quote":"provides the situated instruction-following collaboration setting that MineCollab's dialogue tasks build on."},{"cited_title":"V oyager: An open-ended embodied agent with large language models","cited_arxiv_id":null,"evidence_quote":"contributes the high-level skill and tool abstraction in Minecraft that MINDcraft's 47-command library adapts."},{"cited_title":"Teach: Task-driven embodied agents that chat","cited_arxiv_id":null,"evidence_quote":"supplies the chat-driven embodied agent design that motivates the communication evaluation in Section 5."},{"cited_title":"Minerl diamond 2021 competition: Overview, results, and lessons learned","cited_arxiv_id":null,"evidence_quote":"defines the low-level MineRL action space that MINDcraft's high-level commands abstract away, framing the tool design."},{"cited_title":"Mineflayer","cited_arxiv_id":null,"evidence_quote":"underlies the JavaScript bot-control layer on which all MINDcraft command execution runs."}],"review_version":1}