REVIEW 4 major objections 5 minor 29 references
This paper introduces PillagerBench, a two-vs-two Minecraft benchmark for LLM-based agents, and claims its proposed system TactiCrafter outperforms chain-of-thought and random baselines by coordinating through readable tactics, learned caus
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A new open Minecraft benchmark for 2v2 LLM-agent competition, and a system, TactiCrafter, that beats its baselines on points and win rate.
T0 review reviewed 2026-08-04 challenge →
load-bearing objection A genuinely new benchmark with a plausible but statistically under-supported ranking, confounded by an admitted side asymmetry that should be fixed before the headline claim is taken at face value. the 4 major comments →
PillagerBench: Benchmarking LLM-Based Agents in Competitive Minecraft Team Environments
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central claim is that competitive team-vs-team play in an open-ended environment like Minecraft is a tractable and informative testbed for LLM-based multi-agent systems, and that a modular design with explicit tactical, causal, and opponent components outperforms a monolithic chain-of-thought solver. TactiCrafter separates concerns: a Tactics Module generates a short natural-language game plan each episode; a Causal Model builds and updates a graph of which items cause which outcomes, inferred from the agents' chat messages and inventories; an Opponent Model attributes tactics to the opposing team from their visible chat; and Base Agents turn the plan into executable code with iterative
What carries the argument
The load-bearing machinery is the split of responsibilities across four modules. The Tactics Module defines the unit of coordination: no more than six numbered conditional instructions in natural language that name each agent, which any human or model can read. The Causal Model encodes the world as action-cause-effect triples, updated from chat logs and inventory states—deliberately dropping the intervention-based verification of prior work for speed. The Opponent Model turns the enemy team's broadcast chat into a hypothesis about their current tactics, which feeds the next episode's tactics update. The Base Agents close the loop by writing executable code for each action, running it, and cr
Load-bearing premise
The comparisons assume the map is neutral enough that the red team's results against blue built-in opponents measure agent skill; the paper itself notes sweet berry bushes sit closer to the blue side, which could bias points and win rates independently of the system being tested.
What would settle it
Swap the teams' starting areas—let TactiCrafter play the blue side against red built-in opponents—and rerun the same episodes; if the side swap reverses or erases the reported points difference and win-rate gaps, the benchmark's headline comparisons are confounded by map imbalance.
If this is right
- If correct, an LLM multi-agent team can coordinate by generating and revising a short readable game plan each episode, without finetuning.
- The benchmark gives the community a reproducible, modular arena where any team size or policy can be plugged into the same API and compared against scripted opponents.
- The causal graph, learned from chat and inventory alone, captures item-flow recipes but omits spatial, block-state, and temporal dependencies, so the system's world model is partial by design.
- Self-play improves action efficiency in the simpler scenario but causes overspecialization in the strategy-sensitive one; keeping world knowledge while resetting opponent-specific knowledge is a suggested remedy.
- Headline win rates remain below 0.5, so the contribution is a measurement method and a relative improvement, not yet an absolute winning system.
Where Pith is reading between the lines
- The paper's aggregate table divides Dash & Dine points by ten to combine scenarios; the reported 'points' and 'sabotage' magnitudes are therefore scaling-dependent, and per-scenario comparisons are more informative.
- The adaptive-learning claim could be tested more sharply by ablating the Opponent Model and history windows: if performance persists, the gains may come from prompt context rather than genuine strategy learning.
- The arena asymmetry the paper flags suggests a concrete design fix—symmetrize maps or switch team sides between episodes—which would make self-play results fairer and strengthen the benchmark.
- The observed overfitting in self-play hints that opponent modeling should be separated from world knowledge in any continual-learning agent, a principle that transfers beyond Minecraft.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces PillagerBench, a Minecraft-based benchmark with two two-versus-two competitive scenarios (Mushroom War and Dash & Dine), rule-based built-in opponents, and an API for LLM-driven agents. It then presents TactiCrafter, an LLM multi-agent system combining a tactics module, a causal model, an opponent model, and Voyager-style base agents. Evaluations report that TactiCrafter outperforms random and CoT baselines on points, sabotage, point difference, and win rate, and that it adapts through self-play and opponent specialization. The manuscript also includes ablations on LLM backbones and modules, and qualitative case studies of emergent tactics and learned causal graphs.
Significance. If the main empirical claim survives corrective experiments, PillagerBench would be a useful community resource for evaluating competitive LLM multi-agent systems, and the open-source, modular API plus rule-based opponents are genuine strengths. The authors are also candid about limitations, including self-play overfitting and the inventory-only scope of the causal model. However, the current evidence does not yet establish the headline ranking: the fixed-side protocol interacts with the acknowledged arena asymmetry, and the reported point estimates have no attached uncertainty. The contribution is potentially solid but needs methodological tightening.
major comments (4)
- [§V-B, Eq. (10)] The main comparison is confounded by side assignment. All reported scores are defined from the red team's perspective (Eq. 10 uses S_red and S_blue), and the evaluation appears to place the tested system on red against built-in opponents on blue. In §V-B the authors state that 'the arena's asymmetry places sweet berry bushes closer to the blue team, giving them an edge in self-play.' This edge is not a constant offset: TactiCrafter's adaptation explicitly targets berry-related strategies (e.g., destroying or ignoring sweet berry bushes in the Dash & Dine case study), so a side swap could shrink or even reverse the headline margins (P 13.05 vs 11.33, D -1.16 vs -2.88, W 0.46 vs 0.42). Please report side-balanced averages or mirrored-map results before claiming fair comparisons.
- [§V-A, Tables IV-VII] Point estimates are reported without any measure of uncertainty. A typical full run consists of 5 consecutive episodes per built-in opponent, repeated 3 times, yet Tables IV-VII give only aggregate values. In particular, the win-rate margin of 0.04 (0.46 vs 0.42) is within the standard error for samples of this size, and the P/S/D margins may also be driven by a few episodes. Provide confidence intervals, significance tests, or at least per-scenario/per-opponent breakdowns; without these, 'TactiCrafter achieves the highest points, sabotage, points difference, and win rate' is not established.
- [§V-A, Table IV] The aggregate metrics depend on an arbitrary scaling of Dash & Dine scores: 'we divide the points in the Dash & Dine scenario by 10 to bring it in line with the average point gain in Mushroom War.' Because Fig. 11 indicates TactiCrafter's advantage over CoT is concentrated in Dash & Dine, the choice of divisor directly affects the magnitude, and possibly the sign, of the aggregate margins. Please report per-scenario results and justify the scaling, or show sensitivity to it.
- [§V-E, Appendix A] The causal-graph results are not evaluated in a way that supports the 'learns causal dependencies' claim. The graph is initialized from the scenario description (Eq. 3) and updated from inventory-based observations; the reported '100% causes / 94.5% effects' accuracy is a manual audit against the same scenario knowledge, with no independent ground truth, no baseline, and no evaluation protocol. Moreover, Table VI shows that removing the Causal Model leaves D essentially unchanged (-0.92 vs -1.16) and improves S (2.08 vs 1.55), so the contribution of this module to the headline result needs to be stated more carefully.
minor comments (5)
- [Figures 8 and 10] The shaded area is labeled '5% confidence interval'; this is presumably a 95% confidence interval, and the method used to compute it should be described.
- [Abstract, Fig. 13] The abstract says TactiCrafter 'showcases adaptive learning through self-play,' but Fig. 13 shows Dash & Dine performance decreasing after 20 self-play episodes. Please qualify the claim or clarify that self-play improvement is scenario-dependent.
- [Tables IV-VII] The averages include episodes against the 'do nothing' opponent. Since those episodes are trivially easier, include a version that excludes them as a robustness check.
- [§IV-C, §IV-D] Typos: 'the the' in §IV-C and 'about about' in §IV-D should be corrected.
- [Table IV caption] The caption states the conclusion ('TactiCrafter achieves the highest...') rather than describing the content; consider moving this interpretation to the main text.
Circularity Check
No significant circularity: the central evaluation is empirical, all citations are external, and no metric is fitted to the systems being compared.
full rationale
The paper's headline claim—that TactiCrafter outperforms Random and CoT baselines on PillagerBench—rests on direct gameplay measurements (Eq. 10) against fixed rule-based opponents, with no parameter fitted to the tested systems and no prediction derived from an input that already contains the answer. The 'sabotage' metric normalizes by σ_blue from do-nothing baseline runs, but this is an external calibration, not a fitted constant used to produce the ranking. Method components are adapted from external prior work (Voyager, ADAM, CoT) with no author-overlap self-citations carrying load; the causal model explicitly starts from the scenario description (Eq. 3) and is updated from observations, so any later accuracy discussion concerns LLM extraction quality, not a circular derivation. The paper itself acknowledges two limitations that are correctness risks but not circularity: the Dash & Dine arena asymmetry ('the arena's asymmetry places sweet berry bushes closer to the blue team, giving them an edge in self-play') and the causal model's inventory-only scope. These could confound or bound the results, but they do not make the measured comparisons equivalent to their inputs. No circular step can be exhibited from the text, so the appropriate finding is no significant circularity.
Axiom & Free-Parameter Ledger
free parameters (3)
- Dash & Dine score scaling factor =
1/10 (divide by 10)
- LLM sampling temperature =
0.3
- Evaluation schedule =
5 episodes per opponent, repeated 3 times
axioms (5)
- domain assumption Minecraft environment constants (random tick rate 200, 80 wait ticks, 2-minute episodes) are a faithful proxy for the intended real-time team-competition setting.
- domain assumption LLM-written JavaScript executes reliably enough that more roll-out iterations indicate better agents rather than lower latency.
- domain assumption Built-in rule-based opponents are adequate, non-adaptive yardsticks for comparing multi-agent systems.
- standard math The game is zero-sum with utility U = P_A - P_B, justifying the minimax and self-play interpretation.
- domain assumption LLM prior knowledge of Minecraft is part of the system rather than uncontrolled leakage.
Cite this review
Pith. "Pith review of PillagerBench: Benchmarking LLM-Based Agents in Competitive Minecraft Team Environments." pith.science (2026). https://pith.science/paper/KUIRSTLL
@misc{pith2026250906235,
author = {Pith},
title = {Pith review of: PillagerBench: Benchmarking LLM-Based Agents in Competitive Minecraft Team Environments},
year = {2026},
howpublished = {\url{https://pith.science/paper/KUIRSTLL}},
note = {Machine review of arXiv:2509.06235}
}
read the original abstract
LLM-based agents have shown promise in various cooperative and strategic reasoning tasks, but their effectiveness in competitive multi-agent environments remains underexplored. To address this gap, we introduce PillagerBench, a novel framework for evaluating multi-agent systems in real-time competitive team-vs-team scenarios in Minecraft. It provides an extensible API, multi-round testing, and rule-based built-in opponents for fair, reproducible comparisons. We also propose TactiCrafter, an LLM-based multi-agent system that facilitates teamwork through human-readable tactics, learns causal dependencies, and adapts to opponent strategies. Our evaluation demonstrates that TactiCrafter outperforms baseline approaches and showcases adaptive learning through self-play. Additionally, we analyze its learning process and strategic evolution over multiple game episodes. To encourage further research, we have open-sourced PillagerBench, fostering advancements in multi-agent AI for competitive environments.
Figures
Reference graph
Works this paper leans on
-
[1]
Saaket Agashe, Yue Fan, Anthony Reyna, and Xin Eric Wang. “Llm-coordination: evaluating and analyz- ing multi-agent coordination abilities in large language models”. In:arXiv preprint arXiv:2310.03903(2023)
Pith/arXiv arXiv 2023
-
[2]
Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune.Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos. 2022. arXiv: 2206.11795[cs.LG].URL: https: //arxiv.org/abs/2206.11795
Pith/arXiv arXiv 2022
-
[3]
AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi- Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruob- ing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors. 2023. arXiv: 2308 . 10848[cs.CL].URL: https : / / arxiv. org / abs / 2308 . 10848
work page 2023
-
[4]
Yubo Dong, Xukun Zhu, Zhengzhe Pan, Linchao Zhu, and Yi Yang. “VillagerAgent: A Graph-Based Multi- Agent Framework for Coordinating Complex Task De- pendencies in Minecraft”. In:Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL). 2024.URL: https : / / arxiv. org / abs / 2406.05720
Pith/arXiv arXiv 2024
-
[5]
Interven- tions and causal inference
Frederick Eberhardt and Richard Scheines. “Interven- tions and causal inference”. In:Philosophy of science 74.5 (2007), pp. 981–995
work page 2007
-
[6]
William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov.MineRL: A Large-Scale Dataset of Minecraft Demonstrations. 2019. arXiv: 1907.13440 [cs.LG].URL: https://arxiv.org/abs/1907.13440
Pith/arXiv arXiv 2019
-
[7]
Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and J¨urgen Schmidhuber.MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. 2024. arXiv: 2308.00352[cs.AI].URL: https://arxiv.org/abs/2308. 00352
Pith/arXiv arXiv 2024
-
[8]
FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learning
Wenzhe Li, Zihan Ding, Seth Karten, and Chi Jin. FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learning. 2024. arXiv: 2406 . 02081 [cs.MA].URL: https://arxiv.org/abs/2406.02081
Pith/arXiv arXiv 2024
-
[9]
Docker: lightweight linux containers for consistent development and deployment
Dirk Merkel. “Docker: lightweight linux containers for consistent development and deployment”. In:Linux journal2014.239 (2014), p. 2
work page 2014
-
[10]
Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannaneh Hajishirzi, Sameer Singh, and Roy Fox.Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling. 2023. arXiv: 2301 . 12050 [cs.LG].URL: https://arxiv.org/abs/2301.12050
Pith/arXiv arXiv 2023
-
[11]
2024.URL: https://openai.com/ index/hello-gpt-4o/
OpenAI.Hello GPT-4o. 2024.URL: https://openai.com/ index/hello-gpt-4o/
work page 2024
-
[12]
2025.URL: https : / / openrouter.ai/
OpenRouter.OpenRouter. 2025.URL: https : / / openrouter.ai/
work page 2025
-
[13]
Emerald Group Pub- lishing, 2013
Guillermo Owen.Game theory. Emerald Group Pub- lishing, 2013
work page 2013
-
[14]
Jonas Peters, Dominik Janzing, and Bernhard Sch¨olkopf.Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017
work page 2017
-
[15]
PrismarineJS.Prismarinejs/mineflayer: Create minecraft bots with a powerful, stable, and high level javascript api.2013.URL: https : / / github. com / PrismarineJS/mineflayer
work page 2013
-
[16]
The StarCraft Multi-Agent Challenge
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. “The StarCraft Multi-Agent Challenge”. In:CoRRabs/1902.04043 (2019)
Pith/arXiv arXiv 1902
-
[17]
Causation, prediction, and search
Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, prediction, and search. MIT press, 2001
2001
-
[18]
2023.URL: https: //kaggle.com/competitions/lux-ai-season-2
Stone Tao, Isabelle Pan, Bovard Doerschuk-Tiberi, and Addison Howard.Lux AI Season 2. 2023.URL: https: //kaggle.com/competitions/lux-ai-season-2
work page 2023
-
[19]
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Man- dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and An- ima Anandkumar.Voyager: An Open-Ended Embodied Agent with Large Language Models. 2023. arXiv: 2305. 16291[cs.AI].URL: https : / / arxiv. org / abs / 2305 . 16291
work page 2023
-
[20]
A survey on large language model based autonomous agents
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. “A survey on large language model based autonomous agents”. In:Frontiers of Computer Science18.6 (Mar. 2024).ISSN: 2095-2236.DOI: 10. 1007/s11704-024-40231-1.URL: http://dx.doi.org/10. 1007/s1170...
work page 2024
-
[21]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.Chain-of-Thought Prompting Elicits Rea- soning in Large Language Models. 2023. arXiv: 2201. 11903[cs.CL].URL: https : / / arxiv. org / abs / 2201 . 11903
work page 2023
-
[22]
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, a...
Pith/arXiv arXiv 2023
-
[23]
Omry Yadan.Hydra - A framework for elegantly config- uring complex applications. Github. 2019.URL: https: //github.com/facebookresearch/hydra
work page 2019
-
[24]
Mitigating Generative Agent Social Dilem- mas
Julian Yocum, Phillip Christoffersen, Mehul Damani, Justin Svegliato, Dylan Hadfield-Menell, and Stuart Russell. “Mitigating Generative Agent Social Dilem- mas”. In:NeurIPS 2023 Foundation Models for Deci- sion Making Workshop. 2023.URL: https://openreview. net/forum?id=5TIdOk7XQ6
work page 2023
-
[25]
Shu Yu and Chaochao Lu.ADAM: An Embodied Causal Agent in Open-World Environments. 2024. arXiv: 2410. 22194[cs.AI].URL: https : / / arxiv. org / abs / 2410 . 22194
work page 2024
-
[26]
Xianhao Yu, Jiaqi Fu, Renjia Deng, and Wenjuan Han. MineLand: Simulating Large-Scale Multi-Agent Inter- actions with Limited Multimodal Senses and Physical Needs. 2024. arXiv: 2403.19267[cs.CL].URL: https: //arxiv.org/abs/2403.19267
Pith/arXiv arXiv 2024
-
[27]
A sur- vey of large language models
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. “A sur- vey of large language models”. In:arXiv preprint arXiv:2303.182231.2 (2023)
Pith/arXiv arXiv 2023
-
[28]
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, and Jifeng Dai.Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Lan- guage Models with Text-based Knowledge and Memory
-
[2023]
arXiv: 2305.17144[cs.AI].URL: https://arxiv. org/abs/2305.17144. APPENDIX This section gives a few examples of the learned causal graphs by TactiCrafter playing against itself for 20 consecutive episodes in both scenarios in PillagerBench. Listing 1 shows the graph for Mushroom war, and listing 2 shows the graph for Dash and Dine. Listing 1. Learned causa...
This paper was first reviewed by deepseek-v4-flash on August 4, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.