{"id":"d35b7fe9-b3a8-4b90-bb81-4d75f47c53bd","arxiv_id":"2509.06235","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new open Minecraft benchmark for 2v2 LLM-agent competition, and a system, TactiCrafter, that beats its baselines on points and win rate.","lead":"This paper builds a Minecraft-based testbed where two teams of LLM-controlled agents fight over resources, plus a system called TactiCrafter that plans tactics and learns from past games. The benchmark is open source and meant to give AI teams a fair, repeatable way to measure competitive multi-agent behavior.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fixed red-side protocol plus self-admitted blue-side map advantage in Dash & Dine may make Table IV's TactiCrafter-vs-CoT ranking side-dependent.","rationale":"The manuscript is a solid benchmark-engineering contribution: open-sourced code, Docker/Hydra reproducibility, detailed scenario descriptions, and rich case studies. The central comparative claim is plausible, but the fixed red-side protocol plus the paper's own admission of a blue-side sweet-berry advantage in Dash & Dine is the clearest concrete threat to it. The reader's weakest assumption points at the same issue, and I agree with that emphasis. I would not reject the paper because the concern is directly testable and the authors provide code; the appropriate state is the reader's CONDITIONAL verdict, with the side-swap experiment as the specific condition that could lift or confirm the objection. Other issues such as missing confidence intervals and the arbitrary 10x scaling are real but secondary; this concern can change the ranking rather than only the error bars.","tokens_in":19398,"tokens_out":11791,"duration_ms":137859,"concrete_test":"Run a side-swapped replication of the main protocol (Sec. V-A) for Dash & Dine: let TactiCrafter and CoT play as the blue team while the built-in opponent plays red, using the same 5-episode blocks × 3 repeats, and compute the TactiCrafter−CoT differences in P, D, and W per side. If the differences flip sign or shrink to noise when the contestant is blue, Table IV's ranking is side-dependent and the benchmark's fairness claim fails; if the differences are preserved, the asymmetry concern is quantitatively bounded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption behind 'TactiCrafter outperforms baselines' is that Table IV measures agent skill rather than side assignment. The paper itself dissolves this assumption in Sec. V-B: 'the arena's asymmetry places sweet berry bushes closer to the blue team, giving them an edge in self-play.' Every headline result is computed with the contestant on red and the built-in opponent on blue (Eq. 10 uses S_red and S_blue; the protocol assigns the tested system to red). Because Dash & Dine is the scenario where TactiCrafter's advantage over CoT is concentrated (Fig. 11), the asymmetry is not a constant offset: the value of close berries depends on the agent's tactics, and TactiCrafter explicitly learns to destroy/ignore berry bushes while CoT does not adapt. A side swap could therefore shrink, eliminate, or reverse the reported margins (ΔP=1.72, ΔD=1.72, ΔW=0.04). The 'fair, reproducible comparisons' claim in the abstract requires either a mirrored map or side-balanced reporting; as written, the benchmark's headline ranking is conditional on an untested side assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PillagerBench, a Minecraft-based benchmark with two two-versus-two competitive scenarios (Mushroom War and Dash & Dine), rule-based built-in opponents, and an API for LLM-driven agents. It then presents TactiCrafter, an LLM multi-agent system combining a tactics module, a causal model, an opponent model, and Voyager-style base agents. Evaluations report that TactiCrafter outperforms random and CoT baselines on points, sabotage, point difference, and win rate, and that it adapts through self-play and opponent specialization. The manuscript also includes ablations on LLM backbones and modules, and qualitative case studies of emergent tactics and learned causal graphs.","tokens_in":19632,"tokens_out":9583,"duration_ms":114057,"significance":"If the main empirical claim survives corrective experiments, PillagerBench would be a useful community resource for evaluating competitive LLM multi-agent systems, and the open-source, modular API plus rule-based opponents are genuine strengths. The authors are also candid about limitations, including self-play overfitting and the inventory-only scope of the causal model. However, the current evidence does not yet establish the headline ranking: the fixed-side protocol interacts with the acknowledged arena asymmetry, and the reported point estimates have no attached uncertainty. The contribution is potentially solid but needs methodological tightening.","major_comments":[{"comment":"The main comparison is confounded by side assignment. All reported scores are defined from the red team's perspective (Eq. 10 uses S_red and S_blue), and the evaluation appears to place the tested system on red against built-in opponents on blue. In §V-B the authors state that 'the arena's asymmetry places sweet berry bushes closer to the blue team, giving them an edge in self-play.' This edge is not a constant offset: TactiCrafter's adaptation explicitly targets berry-related strategies (e.g., destroying or ignoring sweet berry bushes in the Dash & Dine case study), so a side swap could shrink or even reverse the headline margins (P 13.05 vs 11.33, D -1.16 vs -2.88, W 0.46 vs 0.42). Please report side-balanced averages or mirrored-map results before claiming fair comparisons.","section":"§V-B, Eq. (10)"},{"comment":"Point estimates are reported without any measure of uncertainty. A typical full run consists of 5 consecutive episodes per built-in opponent, repeated 3 times, yet Tables IV-VII give only aggregate values. In particular, the win-rate margin of 0.04 (0.46 vs 0.42) is within the standard error for samples of this size, and the P/S/D margins may also be driven by a few episodes. Provide confidence intervals, significance tests, or at least per-scenario/per-opponent breakdowns; without these, 'TactiCrafter achieves the highest points, sabotage, points difference, and win rate' is not established.","section":"§V-A, Tables IV-VII"},{"comment":"The aggregate metrics depend on an arbitrary scaling of Dash & Dine scores: 'we divide the points in the Dash & Dine scenario by 10 to bring it in line with the average point gain in Mushroom War.' Because Fig. 11 indicates TactiCrafter's advantage over CoT is concentrated in Dash & Dine, the choice of divisor directly affects the magnitude, and possibly the sign, of the aggregate margins. Please report per-scenario results and justify the scaling, or show sensitivity to it.","section":"§V-A, Table IV"},{"comment":"The causal-graph results are not evaluated in a way that supports the 'learns causal dependencies' claim. The graph is initialized from the scenario description (Eq. 3) and updated from inventory-based observations; the reported '100% causes / 94.5% effects' accuracy is a manual audit against the same scenario knowledge, with no independent ground truth, no baseline, and no evaluation protocol. Moreover, Table VI shows that removing the Causal Model leaves D essentially unchanged (-0.92 vs -1.16) and improves S (2.08 vs 1.55), so the contribution of this module to the headline result needs to be stated more carefully.","section":"§V-E, Appendix A"}],"minor_comments":[{"comment":"The shaded area is labeled '5% confidence interval'; this is presumably a 95% confidence interval, and the method used to compute it should be described.","section":"Figures 8 and 10"},{"comment":"The abstract says TactiCrafter 'showcases adaptive learning through self-play,' but Fig. 13 shows Dash & Dine performance decreasing after 20 self-play episodes. Please qualify the claim or clarify that self-play improvement is scenario-dependent.","section":"Abstract, Fig. 13"},{"comment":"The averages include episodes against the 'do nothing' opponent. Since those episodes are trivially easier, include a version that excludes them as a robustness check.","section":"Tables IV-VII"},{"comment":"Typos: 'the the' in §IV-C and 'about about' in §IV-D should be corrected.","section":"§IV-C, §IV-D"},{"comment":"The caption states the conclusion ('TactiCrafter achieves the highest...') rather than describing the content; consider moving this interpretation to the main text.","section":"Table IV caption"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the side-asymmetry confound. I would ask for a side-swap or mirrored-map experiment and basic uncertainty quantification before considering acceptance. The paper is otherwise within scope, and the authors are transparent about limitations; I do not see a broader circularity problem except in the causal-graph evaluation, which is secondary to the headline empirical claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: PillagerBench is a real contribution—the first open benchmark I've seen for LLM-based multi-agent teams in competitive Minecraft, with two scenarios and a set of rule-based built-in opponents—and the qualitative ranking TactiCrafter > CoT > Random is directionally credible from the reported numbers. But the headline comparison is confounded by a side asymmetry the paper itself admits, and the statistical support is thinner than the abstract's language suggests. Worth refereeing, not worth accepting as-is.\n\nWhat's new: Mushroom War (task allocation under time pressure) and Dash & Dine (planning, causal dependencies, and opponent adaptation) are genuinely new scenarios. The built-in opponents are varied and clearly specified. TactiCrafter integrates existing pieces—Voyager-style iterative prompting, ADAM's causal model without interventions, contract-style tactics, opponent modeling—but the integration is non-trivial, and the ablation studies are informative. The paper is also candid about limits: the causal model captures only inventory-based dependencies, self-play overfits and degrades in Dash & Dine, and the learned graphs have persistent errors. That honesty counts.\n\nThe soft spots are real. Every headline result places the tested system on the red team and the built-in opponent on blue. In Dash & Dine, the arena is asymmetric: sweet berry bushes are closer to the blue team, and the paper says this gives blue an edge. Since TactiCrafter's advantage over CoT is concentrated in Dash & Dine, the ranking could change on a side swap. The abstract calls the comparisons 'fair'—that needs a mirrored map or side-balanced reporting.\n\nSecond, Tables IV–VII are point estimates only. Sample sizes are small (roughly 15 games per condition), there are no confidence intervals or significance tests, and the aggregate P/S/D metrics mix scenarios using an arbitrary 10x scaling for Dash & Dine. Third, the causal-learning claim is narrower than the abstract implies: the graph is initialized from the scenario description and then assessed against that same description, so the 'learning' is partly re-estimating a prior; the paper's own error analysis shows many errors persist.\n\nNone of this falsifies the central ranking, but it means PillagerBench is not yet a calibrated measurement instrument. For someone building or evaluating LLM agents in competitive environments, the benchmark and code are worth engaging with; for someone citing the 'TactiCrafter outperforms' claim, hold off until the side-asymmetry is addressed.\n\nRecommendation: send to peer review, with the clear expectation of side-balanced evaluation, confidence intervals, and a softer abstract.","headline":"A genuinely new benchmark with a plausible but statistically under-supported ranking, confounded by an admitted side asymmetry that should be fixed before the headline claim is taken at face value.","tokens_in":20142,"tokens_out":4723,"would_cite":true,"duration_ms":50136,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces PillagerBench, a two-vs-two Minecraft benchmark for LLM-based agents, and claims its proposed system TactiCrafter outperforms chain-of-thought and random baselines by coordinating through readable tactics, learned caus","keywords":["competitive multi-agent systems","LLM-based agents","Minecraft benchmark","team-vs-team","self-play adaptation","opponent modeling","causal graph learning","reproducible benchmarking"],"falsifier":"Swap the teams' starting areas—let TactiCrafter play the blue side against red built-in opponents—and rerun the same episodes; if the side swap reverses or erases the reported points difference and win-rate gaps, the benchmark's headline comparisons are confounded by map imbalance.","tokens_in":19288,"feed_emoji":"🎮","tokens_out":5799,"duration_ms":63921,"temperature":0.7,"pith_summary":"This paper claims that large language models can drive two-agent teams in real-time competitive Minecraft matches, and proposes both a benchmark to measure this and a system to do it. The benchmark, PillagerBench, offers two team-vs-team scenarios—resource harvesting in Mushroom War and multi-step food crafting in Dash & Dine—with rule-based built-in opponents that sabotage in scripted ways. The system, TactiCrafter, coordinates agents through human-readable tactics, learns a causal graph of item dependencies from chat and inventory observations, and maintains a model of the opponent's tactics. In the paper's experiments, TactiCrafter scores more points, denies more points, and wins more often than random and chain-of-thought baselines, though its win rate against the built-in opponents stays below half. The paper also reports that self-play improves performance against a familiar opponent but can overfit, hurting performance against different opponents in the strategy-heavy Dash & Dine scenario.","feed_headline":"New Minecraft benchmark pits LLM teams against built-in rivals","feed_subtitle":"PillagerBench scores points, sabotage, and win rate; the best system adapts tactics and causal knowledge across episodes.","key_machinery":"The load-bearing machinery is the split of responsibilities across four modules. The Tactics Module defines the unit of coordination: no more than six numbered conditional instructions in natural language that name each agent, which any human or model can read. The Causal Model encodes the world as action-cause-effect triples, updated from chat logs and inventory states—deliberately dropping the intervention-based verification of prior work for speed. The Opponent Model turns the enemy team's broadcast chat into a hypothesis about their current tactics, which feeds the next episode's tactics update. The Base Agents close the loop by writing executable code for each action, running it, and cr","core_discovery":"The central claim is that competitive team-vs-team play in an open-ended environment like Minecraft is a tractable and informative testbed for LLM-based multi-agent systems, and that a modular design with explicit tactical, causal, and opponent components outperforms a monolithic chain-of-thought solver. TactiCrafter separates concerns: a Tactics Module generates a short natural-language game plan each episode; a Causal Model builds and updates a graph of which items cause which outcomes, inferred from the agents' chat messages and inventories; an Opponent Model attributes tactics to the opposing team from their visible chat; and Base Agents turn the plan into executable code with iterative","pith_inferences":["The paper's aggregate table divides Dash & Dine points by ten to combine scenarios; the reported 'points' and 'sabotage' magnitudes are therefore scaling-dependent, and per-scenario comparisons are more informative.","The adaptive-learning claim could be tested more sharply by ablating the Opponent Model and history windows: if performance persists, the gains may come from prompt context rather than genuine strategy learning.","The arena asymmetry the paper flags suggests a concrete design fix—symmetrize maps or switch team sides between episodes—which would make self-play results fairer and strengthen the benchmark.","The observed overfitting in self-play hints that opponent modeling should be separated from world knowledge in any continual-learning agent, a principle that transfers beyond Minecraft."],"forward_implications":["If correct, an LLM multi-agent team can coordinate by generating and revising a short readable game plan each episode, without finetuning.","The benchmark gives the community a reproducible, modular arena where any team size or policy can be plugged into the same API and compared against scripted opponents.","The causal graph, learned from chat and inventory alone, captures item-flow recipes but omits spatial, block-state, and temporal dependencies, so the system's world model is partial by design.","Self-play improves action efficiency in the simpler scenario but causes overspecialization in the strategy-sensitive one; keeping world knowledge while resetting opponent-specific knowledge is a suggested remedy.","Headline win rates remain below 0.5, so the contribution is a measurement method and a relative improvement, not yet an absolute winning system."],"supporting_citations":[{"why":"Supplies the executable-code action space and iterative self-improvement loop that Base Agents inherit.","marker":"[19]"},{"why":"Provides the LLM-based causal-graph approach that the Causal Model adapts by dropping interventions for speed.","marker":"[25]"},{"why":"Defines chain-of-thought prompting, used in all module prompts and as the main LLM baseline.","marker":"[21]"},{"why":"Inspires the contract-style natural-language coordination format used for tactics in TactiCrafter.","marker":"[24]"},{"why":"Frames the competitive team-vs-team benchmarking context that PillagerBench extends from StarCraft to Minecraft.","marker":"[16]"},{"why":"Another competitive benchmark used to position PillagerBench's real-time, open-ended environment contrast.","marker":"[18]"}],"fun_headline_variants":["Minecraft arena tests LLM agent teamwork","LLM teams battle in new Minecraft benchmark","TactiCrafter wins in competitive Minecraft test","PillagerBench: LLM agents adapt in team fights","Minecraft team fights reveal adaptive LLM agents"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The comparisons assume the map is neutral enough that the red team's results against blue built-in opponents measure agent skill; the paper itself notes sweet berry bushes sit closer to the blue side, which could bias points and win rates independently of the system being tested.","fun_headline_variants_meta":{"raw":{"variants":["Minecraft arena tests LLM agent teamwork","LLM teams battle in new Minecraft benchmark","TactiCrafter wins in competitive Minecraft test","PillagerBench: LLM agents adapt in team fights","Minecraft team fights reveal adaptive LLM agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000606,"raw_usage":{"total_tokens":2625,"prompt_tokens":671,"completion_tokens":1954,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":415,"completion_tokens_details":{"reasoning_tokens":1895}},"tokens_in":415,"tokens_out":1954,"duration_ms":17318,"temperature":1.0,"reasoning_tokens":1895,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T23:51:07.780177+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Swap the teams' starting areas—let TactiCrafter play the blue side against red built-in opponents—and rerun the same episodes; if the side swap reverses or erases the reported points difference and win-rate gaps, the benchmark's headline comparisons are confounded by map imbalance.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the executable-code action space and iterative self-improvement loop that Base Agents inherit."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LLM-based causal-graph approach that the Causal Model adapts by dropping interventions for speed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines chain-of-thought prompting, used in all module prompts and as the main LLM baseline."},{"cited_title":"Mitigating Generative Agent Social Dilem- mas","cited_arxiv_id":null,"evidence_quote":"Inspires the contract-style natural-language coordination format used for tactics in TactiCrafter."},{"cited_title":"2023.URL: https: //kaggle.com/competitions/lux-ai-season-2","cited_arxiv_id":null,"evidence_quote":"Another competitive benchmark used to position PillagerBench's real-time, open-ended environment contrast."}],"review_version":1}