Pith. sign in

REVIEW 3 major objections 7 minor 55 references

Frontier AI models achieve only a small fraction of human performance when asked to run a stochastic, long-horizon amusement-park business simulator.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 21:16 UTC pith:YXZEMZ6Q

load-bearing objection A genuinely useful new benchmark with a likely-robust headline gap, but the reported human-AI multiples are not internally consistent and the human denominator is essentially unmeasured. the 3 major comments →

arxiv 2511.15830 v3 pith:YXZEMZ6Q submitted 2025-11-19 cs.AI

Mini Amusement Parks (MAPs): A Testbed for Modelling Business Decisions

St\'ephane Aroca-Ouellette , Ian Berlot-Attwell , Panagiotis Lymperopoulos , Abhiramon Rajasekharan , Tongqi Zhu , Herin Kang , Kaheer Suleman , Sam Pasupalak This is my paper
classification cs.AI
keywords business decision-making benchmarkamusement park simulatorLLM agentslong-horizon planningactive world-model learningspatial reasoningstochastic environmentshuman-AI performance gap
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces Mini Amusement Parks (MAPs), a simulator in which a manager must maximize park value by placing rides, shops, staff, and research while stochastic guests react to the park's design. The central claim is that state-of-the-art language-model agents fail badly at this integrated task: the strongest evaluated agent reaches about 13.9% of human park value in easy mode and 7.16% in medium mode. The authors argue that the deficit is not one skill but a cluster—long-horizon planning, active world-model learning, spatial reasoning, and handling stochastic outcomes—and that MAPs can isolate and measure each within a single grounded environment. A sympathetic reader would care because it converts the abstract challenge of business-style decision-making into a quantified, repeatable testbed with a human upper bound.

Core claim

The paper's discovery is a diagnostic benchmark and an empirical result around it. MAPs is a 20x20 amusement-park grid with terrain, rides, shops, staff, research tiers, and guests with hidden needs; the objective is total park value after 50 (easy) or 100 (medium) in-game days. Benchmarking five frontier LLM agents under a ReAct protocol with full game documentation, the strongest model earns 13.89% of the human average on easy and 7.16% on medium. Giving agents 100 in-game days of sandbox exploration with tools such as undo, reset, grants, and layout switching does not consistently help: the strongest model improves somewhat, but most models regress. A simple placement heuristic that favor

What carries the argument

The central object is MAPs itself: a stochastic, 20x20 park simulator with an open-ended objective (maximize park value) and a structured action space. Its load-bearing components are (1) the ReAct agent loop, which receives JSON observations and a 5-state history; (2) a sandbox mode with counterfactual actions (undo, max money, reset, switch layouts) used to test active world-model learning; (3) a spatial-placement heuristic that overrides LLM-chosen coordinates; and (4) random-shooting MPC paired with either an oracle or a learned world model to isolate stochastic planning. Each component isolates one of the five challenges while the environment remains fixed, making the benchmark diagnost

Load-bearing premise

The entire reported gap rests on the online leaderboard human scores being a valid upper bound; if those scores are inflated by retries and shared strategies, or if the AI evaluation protocol (short history, fixed prompt budget) artificially suppresses model scores, the headline gap shrinks.

What would settle it

Run MAPs medium, one fixed layout, with (a) a fresh human cohort playing each layout exactly once with no communication and (b) the same five agents given longer histories and more lookahead rollouts; if the fresh human average falls well below the leaderboard average while agent scores rise, the paper's central claim about the size of the gap would fail.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the gap is real, MAPs offers a single scoreboard for tracking whether new agent designs actually improve integrated decision-making, avoiding fragmented subskill benchmarks.
  • The failure of sandbox learning implies that giving agents exploration tools is not enough; sample-efficient, active world-model learning must become a first-class research target.
  • The oracle-versus-learned world model contrast implies that stochastic world models, not search procedures, are the current bottleneck; better learned dynamics could unlock larger gains than better policies.
  • The spatial heuristic result implies that hybrid systems that offload low-level spatial placement to rule-based components can outperform end-to-end LLM control today.
  • The benchmark provides a stable human upper bound that future agents can be measured against, making progress or stagnation in business-style decision-making visible.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: Because the human baseline is an online leaderboard that permits retries and strategy sharing, and the agents run with a short 5-state history, the exact multiples (roughly 7x on easy, 14x on medium) should be read as an upper bound on the human–AI gap rather than a calibrated ratio.
  • Editorial inference: The success of an oracle world model with only a 4-step lookahead suggests a concrete next experiment: combine a learned stochastic world model with a more powerful search procedure than random shooting, to test whether the bottleneck shifts from modelling to planning.
  • Editorial inference: The paper leaves untested whether models that are fine-tuned rather than prompted on sandbox experience, or that maintain an evolving belief state, could use the 100-day budget more effectively; this is a testable extension within the same MAPs harness.
  • Editorial inference: Because MAPs provides counterfactual sandbox actions, it could also serve as an evaluation suite for causal discovery and active learning algorithms, a use the authors only gesture at.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces Mini Amusement Parks (MAPs), a simulator-based benchmark for business-style decision making that requires agents to optimize an open-ended objective over long horizons, learn environment dynamics actively, reason spatially, and handle stochasticity. The authors provide human baselines from an online leaderboard and evaluate five state-of-the-art LLM-based agents (GPT-5, GPT-5 Nano, Grok 4, Sonnet 4.5, Gemini 2.5 Pro) under several protocols (ReAct, sandbox learning, spatial heuristic, and MPC with learned/oracle world models). The central claim is that the best model, GPT-5, achieves only 7.16% of human performance on medium difficulty, and that MAPs exposes persistent weaknesses in long-horizon planning, active world-model learning, spatial reasoning, and stochasticity handling (CH1–CH5).

Significance. If the empirical picture holds, MAPs is a valuable addition to the agent-benchmark landscape: it combines an open-ended objective, stochastic transitions, spatial reasoning, and an online human leaderboard in a single grounded simulator, with code released. The qualitative error analyses (Sections F–H) are rich and actionable, and the use of a hand-crafted spatial heuristic and an oracle world model as interpretable reference points, rather than fitted baselines, is a methodological strength. The paper is also commendable for explicitly acknowledging that the human leaderboard is an upper bound. However, the headline quantitative claim currently rests on an insufficiently documented human denominator and is stated with inconsistent numbers in different parts of the manuscript; these issues need to be resolved before the central CH1 claim can be taken as a precise measurement.

major comments (3)
  1. [Abstract and Section 4.1 / Table 2] The central quantitative claim is internally inconsistent. The full-text abstract reports humans outperform AI by 6.5x on easy and 9.8x on medium, while the arXiv metadata abstract reports 11.4x and 15.3x. Table 2 implies for GPT-5 a gap of 100/13.89 ≈ 7.2x on easy and 100/7.16 ≈ 14.0x on medium. None of the three sets of numbers agree. Since the magnitude of the human–AI gap is the paper's primary contribution, please reconcile these values and specify exactly how the ratios are computed from the per-layout normalization.
  2. [Section 3.3 and Table 2] The human baseline is taken from an online leaderboard where participants can retry layouts or share strategies, and the paper itself labels it 'an empirical upper bound rather than a direct comparison.' Yet every AI score in Tables 2–6 is normalized by this leaderboard value, and the abstract/CH1 state the resulting 7.16% figure as a direct gap. The paper does not report the number of participants, the aggregation rule (mean vs. best), or any controlled single-attempt human baseline. The 7.16% is therefore a lower bound on the AI/human ratio relative to a typical human, not a controlled measurement. Please either provide a controlled human baseline with participant count and protocol, or consistently present the headline as an upper-bound comparison.
  3. [Section 4.2 / Table 3] The claim that relative performance deteriorates as long-horizon planning demands increase (CH2) confounds two factors: easy mode has a 50-day horizon with no research, while medium has a 100-day horizon plus the research mechanic. The observed drop in relative scores could be driven by the added research complexity rather than by horizon length itself. To cleanly support the CH2 conclusion, the comparison should include a condition that varies only the horizon (e.g., a 100-day easy-like setting without research), or the authors should explicitly argue that the research mechanic is a minor contributor. As presented, the CH2 result is not uniquely identified.
minor comments (7)
  1. [Abstract vs. metadata] The discrepancy between the full-text abstract and the arXiv metadata abstract (6.5x/9.8x vs. 11.4x/15.3x) should also be corrected in the arXiv record itself, not only in the PDF.
  2. [Table 2 caption] The caption states 'Score (%) indicates performance relative to humans per layout.' Please clarify whether the reported percentage is the mean of the per-layout ratios or the ratio of the mean park values, since this affects how the numbers in the abstract are derived.
  3. [Sections 4.4 and 4.5] The text says 'results, shown in Section 4.4' and 'Section 4.5' when referring to Tables 5 and 6. Please fix the cross-references.
  4. [Throughout] The benchmark name is inconsistently styled as 'MAPS' and 'MAPs' (e.g., abstract vs. Section 1). Please standardize.
  5. [Table 2] Several standard deviations are very large (e.g., GPT-5 easy 13.89 ± 11.16; Grok 4 easy 6.62 ± 14.50). When discussing model rankings, please add significance tests or at least note the overlap in confidence intervals.
  6. [Section 3.1 and Table 4 caption] The sandbox budget is described as '100 in-game days' in Section 3.1 and '100 in-game steps' in the Table 4 caption. Please make the units consistent.
  7. [Throughout] Spellings of 'W ALL-E' / 'Walle' / 'WALL-E' should be normalized.

Circularity Check

0 steps flagged

No significant circularity: the benchmark results are direct simulator measurements, not derivations from fitted parameters or self-citing constraints.

full rationale

This is an empirical benchmark paper. The central claims are direct measurements taken from the MAPs simulator: the GPT-5 score of 7.16% of human performance (Table 2) is the observed ratio of final park value to a human leaderboard score, not a quantity derived from a fitted parameter or from an equation that embeds the conclusion. The spatial-placement heuristic in Section 3.1/4.4 is a hand-coded baseline (density, water adjacency, path adjacency) compared against the same LLM agents; it is not fit to the data it is later used to explain. The sandbox 'active world-model learning' experiments use a fixed 100-day protocol with learnings appended to the prompt; the finding that these learnings do not help is again a measured outcome, not an artifact of construction. The oracle world model is the ground-truth simulator used as an upper bound, which is a legitimate comparison target rather than a circularity. The paper explicitly acknowledges that the human leaderboard reflects an unlimited setting and 'serve[s] as an empirical upper bound rather than as a direct comparison' (Section 3.3); this is a caveat about the strength of the human baseline, not a circular step in the derivation. The only self-citation, Aroca-Ouellette et al. (2025), appears in the introduction as background support for the claim that LLMs struggle with long-horizon planning; it is not load-bearing for MAPs' own results, and no uniqueness theorem or ansatz is imported from prior work by the same authors. The inconsistency between the abstract's 6.5x/9.8x and the introduction's 11.4x/15.3x performance ratios is a reporting/consistency issue, not a circular reduction. Overall, the paper's evaluations are self-contained measurements against an external agent and human baseline; no prediction reduces to its input by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper's central claim rests on the design of the MAPs simulator and the evaluation protocol. The free parameters listed are hand-chosen design choices in the benchmark and agent baselines; they are not fitted to produce the headline results, but they influence the sub-analyses. No new physical or theoretical entities are introduced.

free parameters (5)
  • ReAct history length = 5
    The baseline ReAct agent is limited to the past 5 states/actions in its context (Section 3.1, CH1). This hand-chosen length may disadvantage LLMs and affect the measured gap.
  • MPC rollout count = 5
    Random-shooting MPC uses k=5 rollouts (Appendix B.2). This is a design choice that affects the magnitude of the oracle and WALL-E MPC results.
  • MPC lookahead steps = 4
    MPC simulates four world-model steps before selecting an action (Section 4.5). Longer or shorter lookahead could change the reported improvements.
  • Sandbox learning budget = 100 in-game days
    Agents get 100 normal actions in sandbox mode (Section 3.1, CH3). This budget is arbitrary and influences conclusions about active world-model learning.
  • Spatial heuristic weights = not explicitly quantified
    The spatial heuristic uses three rules: density sub-selection, water adjacency upweighting for rides, and path adjacency for specialty shops (Section 3.1, CH4). No numeric weights are given; the design is hand-crafted.
axioms (4)
  • domain assumption MAPs simulator captures the interconnected challenges of real-world business decision-making.
    The paper assumes that success in MAPs transfers to real-world business competence (Section 1). This is a domain assumption that underpins the benchmark's significance.
  • domain assumption Human leaderboard scores represent expert performance.
    Section 3.3 treats leaderboard results as a high-water mark, even though players can retry layouts and share strategies. If these scores are not truly expert-level, the reported gap is inflated.
  • domain assumption ReAct with 5-state history is a representative evaluation protocol for SOTA LLM agents.
    The paper evaluates all LLMs with a standard ReAct loop and limited context (Section 3.1). More powerful scaffolding could reduce the gap.
  • domain assumption Park value is a valid measure of business decision quality.
    The primary metric is the final park value, defined as a sum of cash, sellable assets, and research IP (Section 3.3). This is a reasonable but unvalidated proxy for business success.

pith-pipeline@v1.3.0-alltime-deepseek · 35297 in / 10460 out tokens · 103868 ms · 2026-08-03T21:16:15.976635+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Mini Amusement Parks (MAPs): A Testbed for Modelling Business Decisions." pith.science (2026). https://pith.science/paper/YXZEMZ6Q

@misc{pith2026251115830,
  author       = {Pith},
  title        = {Pith review of: Mini Amusement Parks (MAPs): A Testbed for Modelling Business Decisions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YXZEMZ6Q}},
  note         = {Machine review of arXiv:2511.15830}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Despite rapid progress in artificial intelligence, current systems struggle with the interconnected challenges that define real-world decision making. Practical domains such as business management require open-ended optimization, actively learning environment dynamics from sparse experience, planning over long horizons in stochastic settings, and reasoning over spatial information. Yet no existing human--AI benchmarks assess how well agents integrate these challenges in a grounded decision-making context. To this end, we introduce Mini Amusement Parks (MAPs), an amusement-park simulator designed to evaluate an agent's ability to model its environment, anticipate long-term consequences under uncertainty, and strategically operate a complex business. We provide expert human performance and a comprehensive evaluation of state-of-the-art agents, finding experts outperform these systems by 11.4x on easy mode and 15.3x on medium mode. Our analysis reveals persistent weaknesses in long-horizon planning, sample-efficient learning, spatial reasoning, and modelling uncertainty. By unifying these challenges within a single environment, MAPs offers a new foundation for benchmarking agents capable of adaptable decision making. Code: https://github.com/Skyfall-Research/MAPs

Figures

Figures reproduced from arXiv: 2511.15830 by Abhiramon Rajasekharan, Herin Kang, Ian Berlot-Attwell, Kaheer Suleman, Panagiotis Lymperopoulos, Sam Pasupalak, St\'ephane Aroca-Ouellette, Tongqi Zhu.

Figure 1
Figure 1. Figure 1: The GUI view of MAPS. 2023b) baseline in which we condition agents on the his￾tory of past actions and observations before thinking and generating the next action. We limit the history to the past 5 states and actions to avoid issues with context given the large and growing state space of the game. We include the objective of maximizing park value, the game manual, and the current settings in the system pr… view at source ↗
Figure 2
Figure 2. Figure 2: The per-day and full trajectory coefficients of variation for revenue, money, and park value across several full games [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of the park design with and without the added spatial heuristic overriding the LLMs placement choices. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 3 canonical work pages · 1 internal anchor

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    R., Smith, K

    Allen, K. R., Smith, K. A., and Tenenbaum, J. B. Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning. Proceedings of the National Academy of Sciences of the United States of America, 117 0 (47): 0 29302—29310, November 2020. ISSN 0027-8424. doi:10.1073/pnas.1912341117. URL https://europepmc.org/articles/PMC7703630

    doi:10.1073/pnas.1912341117 2020

  3. [3]

    Herobench: A benchmark for long-horizon planning and structured reasoning in virtual worlds, 2025

    Anokhin, P., Khalikov, R., Rebrikov, S., Volkov, V., Sorokin, A., and Bissonnette, V. Herobench: A benchmark for long-horizon planning and structured reasoning in virtual worlds, 2025. URL https://arxiv.org/abs/2508.12782

    Pith/arXiv arXiv 2025

  4. [4]

    Introducing claude sonnet 4.5

    Anthropic . Introducing claude sonnet 4.5. https://www.anthropic.com/news/claude-sonnet-4-5, September 2025

    2025

  5. [5]

    R e S eeding latent states for sequential language understanding

    Aroca-Ouellette, S., von der Wense, K., and Roncone, A. R e S eeding latent states for sequential language understanding. In Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V. (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp.\ 25233--25247, Suzhou, China, November 2025. Association for Computa...

    work page doi:10.18653/v1/2025.emnlp-main.1281 2025

  6. [6]

    and Petersson, L

    Backlund, A. and Petersson, L. Vending-bench: A benchmark for long-term coherence of autonomous agents, 2025. URL https://arxiv.org/abs/2502.15840

    Pith/arXiv arXiv 2025

  7. [7]

    K., Koss, E., and Lichtenbaum, C

    Bai, M., Cohen, A. K., Koss, E., and Lichtenbaum, C. Stuck in the matrix: Probing spatial reasoning in large language models, 2025. URL https://arxiv.org/abs/2510.20198

    arXiv 2025

  8. [8]

    G., Naddaf , Y., Veness , J., and Bowling , M

    Bellemare , M. G., Naddaf , Y., Veness , J., and Bowling , M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 0 253--279, jun 2013

    2013

  9. [9]

    When is tree search useful for LLM planning? it depends on the discriminator

    Chen, Z., White, M., Mooney, R., Payani, A., Su, Y., and Sun, H. When is tree search useful for LLM planning? it depends on the discriminator. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 13659--13678, Bangkok, Thailand, August 2024...

    doi:10.18653/v1/2024.acl-long.738 2024

  10. [10]

    H., and Bengio, Y

    Chevalier-Boisvert, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., and Bengio, Y. Baby AI : First steps towards grounded language learning with a human in the loop. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rJeXCo0cYX

    2019

  11. [11]

    On the measure of intelligence, 2019

    Chollet, F. On the measure of intelligence, 2019. URL https://arxiv.org/abs/1911.01547

    Pith/arXiv arXiv 2019

  12. [12]

    Arc-agi-2: A new challenge for frontier ai reasoning systems, 2025

    Chollet, F., Knoop, M., Kamradt, G., Landers, B., and Pinkard, H. Arc-agi-2: A new challenge for frontier ai reasoning systems, 2025. URL https://arxiv.org/abs/2505.11831

    Pith/arXiv arXiv 2025

  13. [13]

    Self-directed learning in new and changing environments: understanding human algorithms for exploration

    Collignon, N. Self-directed learning in new and changing environments: understanding human algorithms for exploration. PhD thesis, University of Edinburgh, Jul 25 2020. URL https://hdl.handle.net/1842/37293. PhD Thesis, Informatics, School of Informatics

    2020

  14. [14]

    LLM-Guided Probabilistic Program Induction for POMDP Model Estimation

    Curtis, A., Tang, H., Veloso, T., Ellis, K., Tenenbaum, J. B., Lozano - P \' e rez, T., and Kaelbling, L. P. Llm-guided probabilistic program induction for POMDP model estimation. CoRR, abs/2505.02216, 2025. doi:10.48550/ARXIV.2505.02216. URL https://doi.org/10.48550/arXiv.2505.02216

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2505.02216 2025

  15. [15]

    Generating code world models with large language models guided by monte carlo tree search

    Dainese, N., Merler, M., Alakuijala, M., and Marttinen, P. Generating code world models with large language models guided by monte carlo tree search. In Globersons, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processin...

    2024

  16. [16]

    Minedojo: Building open-ended embodied agents with internet-scale knowledge

    Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https://openreview.net/forum?id=rc8o_j8I8PX

    2022

  17. [17]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Gemini Team, Google DeepMind . Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Technical Report arXiv:2507.06261, Google DeepMind, July 2025

    Pith/arXiv arXiv 2025

  18. [18]

    and Sobel, D

    Gopnik, A. and Sobel, D. M. Detecting blickets: How young children use information about novel causal powers in categorization and induction. Child Development, 71 0 (5): 0 1205--1222, 2000. doi:10.1111/1467-8624.00224

    arXiv 2000

  19. [19]

    The Scientist in the Crib: Minds, Brains, and How Children Learn

    Gopnik, A., Meltzoff, A., and Kuhl, P. The Scientist in the Crib: Minds, Brains, and How Children Learn. Harper Collins, 1999

    1999

  20. [20]

    Benchmarking the spectrum of agent capabilities

    Hafner, D. Benchmarking the spectrum of agent capabilities. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=1W0z96MFEoH

    2022

  21. [21]

    Reasoning with language model is planning with world model

    Hao, S., Gu, Y., Ma, H., Hong, J., Wang, Z., Wang, D., and Hu, Z. Reasoning with language model is planning with world model. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 8154--8173, Singapore, December 2023. Association for Computational Linguistics. doi:10.186...

    doi:10.18653/v1/2023.emnlp-main.507 2023

  22. [22]

    W., Li, L., Yang, Z., Wang, L., and Cheng, Y

    Hao, Y., Gu, J., Wang, H. W., Li, L., Yang, Z., Wang, L., and Cheng, Y. Can MLLM s reason in multimodality? EMMA : An enhanced multimodal reasoning benchmark. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=v26vwjxOEz

    2025

  23. [23]

    D., Majumder, B

    Jansen, P., C \^o t \'e , M.-A., Khot, T., Bransom, E., Mishra, B. D., Majumder, B. P., Tafjord, O., and Clark, P. Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net...

    2024

  24. [24]

    M., Bommarito, M

    Katz, D. M., Bommarito, M. J., Gao, S., and Arredondo, P. D. Gpt-4 passes the bar exam. SSRN, 2023

    2023

  25. [25]

    One life to learn: Inferring symbolic world models for stochastic environments from unguided exploration, 2025

    Khan, Z., Prasad, A., Stengel-Eskin, E., Cho, J., and Bansal, M. One life to learn: Inferring symbolic world models for stochastic environments from unguided exploration, 2025. URL https://arxiv.org/abs/2510.12088

    Pith/arXiv arXiv 2025

  26. [26]

    u ttler, H., Nardelli, N., Miller, A., Raileanu, R., Selvatici, M., Grefenstette, E., and Rockt\

    K\" u ttler, H., Nardelli, N., Miller, A., Raileanu, R., Selvatici, M., Grefenstette, E., and Rockt\" a schel, T. The nethack learning environment. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 7671--7684. Curran Associates, Inc., 2020. URL https://proceeding...

    2020

  27. [27]

    Competition-level code generation with alphacode

    Li, Y., Choi, D., Chung, J., et al. Competition-level code generation with alphacode. Science, 378 0 (6624): 0 1092--1097, 2022

    2022

  28. [28]

    Performance of chatgpt across different versions in medical licensing examinations worldwide: systematic review and meta-analysis

    Liu, M., Okuhara, T., Chang, X., Shirabe, R., Nishiie, Y., Okada, H., and Kiuchi, T. Performance of chatgpt across different versions in medical licensing examinations worldwide: systematic review and meta-analysis. Journal of medical Internet research, 26: 0 e60807, 2024 a

    2024

  29. [29]

    Are LLM s capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data

    Liu, X., Wu, Z., Wu, X., Lu, P., Chang, K.-W., and Feng, Y. Are LLM s capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp.\ 9215--9235, Bangkok, Thailand, August 2024 b . As...

    work page doi:10.18653/v1/2024.findings-acl.548 2024

  30. [30]

    Non-myopic generation of language models for reasoning and planning

    Ma, C., Zhao, H., Zhang, J., He, J., and Kong, L. Non-myopic generation of language models for reasoning and planning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=OoNazl6T7D

    2025

  31. [31]

    Introducing gpt-5

    OpenAI . Introducing gpt-5. https://openai.com/index/introducing-gpt-5/, August 2025

    2025

  32. [32]

    Textquests: How good are llms at text-based video games?, 2025

    Phan, L., Mazeika, M., Zou, A., and Hendrycks, D. Textquests: How good are llms at text-based video games?, 2025. URL https://arxiv.org/abs/2507.23701

    Pith/arXiv arXiv 2025

  33. [33]

    T., Liang, Y., Tang, H., Weller, A., Kryven, M., and Ellis, K

    Piriyakulkij, W. T., Liang, Y., Tang, H., Weller, A., Kryven, M., and Ellis, K. Poe-world: Compositional world modeling with products of programmatic experts. CoRR, abs/2505.10819, 2025. doi:10.48550/ARXIV.2505.10819. URL https://doi.org/10.48550/arXiv.2505.10819

    doi:10.48550/arxiv.2505.10819 2025

  34. [34]

    Reasoning under uncertainty: Exploring probabilistic reasoning capabilities of llms, 2025

    Pournemat, M., Rezaei, K., Sriramanan, G., Zarei, A., Fu, J., Wang, Y., Eghbalzadeh, H., and Feizi, S. Reasoning under uncertainty: Exploring probabilistic reasoning capabilities of llms, 2025. URL https://arxiv.org/abs/2509.10739

    arXiv 2025

  35. [35]

    Synthesis of model predictive control and reinforcement learning: Survey and classification

    Reiter, R., Hoffmann, J., Reinhardt, D., Messerer, F., Baumg \" a rtner, K., Sawant, S., Boedecker, J., Diehl, M., and Gros, S. Synthesis of model predictive control and reinforcement learning: Survey and classification. CoRR, abs/2502.02133, 2025. doi:10.48550/ARXIV.2502.02133. URL https://doi.org/10.48550/arXiv.2502.02133

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.02133 2025

  36. [36]

    \ ALFW \ orld: Aligning text and embodied environments for interactive learning

    Shridhar, M., Yuan, X., Cote, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. \ ALFW \ orld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=0IOX0YcCdTn

    2021

  37. [37]

    Mastering the game of go without human knowledge

    Silver, D., Schrittwieser, J., Simonyan, K., et al. Mastering the game of go without human knowledge. Nature, 550 0 (7676): 0 354--359, 2017

    2017

  38. [38]

    Y., and Ellis, K

    Tang, H., Key, D. Y., and Ellis, K. Worldcoder, a model-based LLM agent: Building world models by writing code and interacting with the environment. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=QGJSXMhVaL

    2024

  39. [39]

    B., Kemp, C., Griffiths, T

    Tenenbaum, J. B., Kemp, C., Griffiths, T. L., and Goodman, N. D. How to grow a mind: Statistics, structure, and abstraction. Science, 331 0 (6022): 0 1279--1285, 2011. doi:10.1126/science.1192788. URL https://www.science.org/doi/abs/10.1126/science.1192788

    doi:10.1126/science.1192788 2011

  40. [40]

    On the planning abilities of large language models - a critical investigation

    Valmeekam, K., Marquez, M., Sreedharan, S., and Kambhampati, S. On the planning abilities of large language models - a critical investigation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=X6dEqXIsEW

    2023

  41. [41]

    X., King, M., Porcel, N

    Wang, J. X., King, M., Porcel, N. P. M., Kurth-Nelson, Z., Zhu, T., Deck, C., Choy, P., Cassin, M., Reynolds, M., Song, H. F., Buttimore, G., Reichert, D. P., Rabinowitz, N. C., Matthey, L., Hassabis, D., Lerchner, A., and Botvinick, M. Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents. In Thirty-fifth Conference on Neural I...

    2021

  42. [42]

    S cience W orld: Is your agent smarter than a 5th grader? In Goldberg, Y., Kozareva, Z., and Zhang, Y

    Wang, R., Jansen, P., C \^o t \'e , M.-A., and Ammanabrolu, P. S cience W orld: Is your agent smarter than a 5th grader? In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.\ 11279--11298, Abu Dhabi, United Arab Emirates, December 2022. Association for Computationa...

    doi:10.18653/v1/2022.emnlp-main.775 2022

  43. [43]

    B., Vollmer, S., Ellis, K., and Tavares, Z

    Warrier, A., Nguyen, D., Naim, M., Jain, M., Liang, Y., Schroeder, K., Yang, C., Tenenbaum, J. B., Vollmer, S., Ellis, K., and Tavares, Z. Benchmarking world-model learning, 2025. URL https://arxiv.org/abs/2510.19788

    Pith/arXiv arXiv 2025

  44. [44]

    Grok 4 model card

    xAI . Grok 4 model card. Technical report, xAI, August 2025

    2025

  45. [45]

    Revealing the barriers of language agents in planning

    Xie, J., Zhang, K., Chen, J., Yuan, S., Zhang, K., Zhang, Y., Li, L., and Xiao, Y. Revealing the barriers of language agents in planning. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long P...

    doi:10.18653/v 2025

  46. [46]

    J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., and Yu, T

    Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., and Yu, T. OSW orld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchm...

    2024

  47. [47]

    Interactive visual reasoning under uncertainty

    Xu, M., Jiang, G., Liang, W., Zhang, C., and Zhu, Y. Interactive visual reasoning under uncertainty. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\ 42409--42432. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/f...

    2023

  48. [48]

    Yao, S., Chen, H., Yang, J., and Narasimhan, K. R. Webshop: Towards scalable real-world web interaction with grounded language agents. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=R9KnuFlvnU

    2022

  49. [49]

    L., Cao, Y., and Narasimhan, K

    Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. R. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023 a . URL https://openreview.net/forum?id=5Xc1ecxO1h

    2023

  50. [50]

    R., and Cao, Y

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023 b . URL https://openreview.net/forum?id=WE_vluYUL-X

    2023

  51. [51]

    M., Sharma, P., Colas, C., Zhao, K

    Ying, L., Collins, K. M., Sharma, P., Colas, C., Zhao, K. I., Weller, A., Tavares, Z., Isola, P., Gershman, S. J., Andreas, J. D., Griffiths, T. L., Chollet, F., Allen, K. R., and Tenenbaum, J. B. Assessing adaptive world models in machines with novel games, 2025. URL https://arxiv.org/abs/2507.12821

    Pith/arXiv arXiv 2025

  52. [52]

    S., and Hsu, D

    Zhao, Z., Lee, W. S., and Hsu, D. Large language models as commonsense knowledge for large-scale task planning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, Decem...

    2023

  53. [53]

    F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y., Fried, D., Alon, U., et al

    Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y., Fried, D., Alon, U., et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023

    Pith/arXiv arXiv 2023

  54. [54]

    Wall-e: World alignment by rule learning improves world model-based llm agents

    Zhou, S., Zhou, T., Yang, Y., Long, G., Ye, D., Jiang, J., and Zhang, C. Wall-e: World alignment by rule learning improves world model-based llm agents. arXiv preprint arXiv:2410.07484, 2024

    Pith/arXiv arXiv 2024

  55. [55]

    WALL-E 2.0: World alignment by neurosymbolic learning improves world model-based LLM agents

    Zhou, S., Zhou, T., Yang, Y., Long, G., Ye, D., Jiang, J., and Zhang, C. WALL-E 2.0: World alignment by neurosymbolic learning improves world model-based LLM agents. CoRR, abs/2504.15785, 2025. doi:10.48550/ARXIV.2504.15785. URL https://doi.org/10.48550/arXiv.2504.15785

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.15785 2025