Pith. sign in

REVIEW 2 major objections 4 minor 36 references

DiG-bench: Discovery in Games

T0 review · 2 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DiG-bench is a benchmark of 70 text games with hidden rules and hidden win conditions, and its control experiment shows that supplying the ground-truth rules lifts a frontier model from 18/70 to 69/70 wins.

desk verdict A well-built text-based benchmark for active discovery; the rules ablation is the strongest evidence, and the main soft spot is the unverifiable novelty claim for the private games. read the letter →

arxiv 2608.12593 v1 pith:HTZADGU2 submitted 2026-08-12 cs.AI cs.LG

Ruairidh M. Battleday , Kai Sandbrink , Jimi Cullen-Drohan , Zihan Yan , Timothy Muller , Clare Maguire , Ales Kubicek , Fraser Greenlee-Scott
show 8 more authors
Sukrit Sumant Tri Dao Jürgen Schmidhuber Michal Valko Joshua Tenenbaum Thomas L. Griffiths Zeb Kurth-Nelson James C.R. Whittington
This is my paper · ORCID
classification cs.AIcs.LG
keywords discoverybenchmarkruleinductionactiveexperimentationlargelanguagemodelsagenticharnesstextgamesscientifichiddenrules
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DiG-bench is a benchmark of 70 handcrafted, text-only games in which both the transformation rules and the win conditions are hidden, forcing a player to discover them through active experimentation. The paper's central claim is that the benchmark isolates discovery from perception, planning, and memorized instructions, because the games live in the text domain of large language models and contain no visual confounds. The key supporting evidence is a rules-given control: when one frontier model received the ground-truth rules in natural language, its win rate rose from 18 of 70 games to 69 of 70, and humans solved all 70 games on first attempt while the strongest model solved 50. If the claim holds, DiG-bench offers a calibrated, human-verified probe of a capability that most current benchmarks either confound with perception or bypass by stating the rules upfront.

What carries the argument

The central object is the hidden-rule game itself: a miniature world encoded as a short Unicode string that acts as a partially observable Markov decision process (POMDP), where both the transition rules and the win conditions must be inferred from the observation-action history. Levels, lives, and per-level step limits structure progression, and many games include a creative mode—a sandbox where steps are free—so the player can run informative experiments before spending scarce survival steps. The attribution mechanism is the rules-given control condition, in which a concise natural-language description of the true dynamics and win condition is added to the prompt; the jump from 18/70 to 69/70 wins under that condition isolates rule inference as the difficult component.

What would settle it

After the 49 private games are released, measure the same models' win rates on them before and after the games enter public training corpora; a sharp rise after release would show that contamination, not in-context discovery, explains the original scores.

Watch

Extended reading notes

Core claim

The paper's central discovery claim is that rule discovery—finding the hidden laws of a novel interactive system—is the operative bottleneck for current language-model agents, and that DiG-bench measures this bottleneck in isolation. Seventy short-string games, each with unique mechanics and unknown objectives, are beaten by at least one human on first exposure, yet the best tested model wins only 50 of 70, and models in agentic harnesses did not outperform their base models on the hardest tiers. The decisive comparison is the rules-given condition: with the true rules supplied as concise natural language, one frontier model won 69 of 70 games, against 18 of 70 without them, which the paper reads as direct evidence that finding out the rules is the primary challenge.

Load-bearing premise

The benchmark is a valid measure of discovery, rather than memorized behavior, only if the games and their near variants are genuinely absent from every tested model's training data, a premise the authors assert for the 49 private games but cannot verify.

Editorial extensions

If this is right

  • DiG-bench provides a perception-free, text-only setting in which discovery and experimentation can be scored directly, so model performance on it is not explained by visual priors or spatial reasoning.
  • Because supplying the rules lifts a frontier model from 18/70 to 69/70 wins, the benchmark's hard part is rule inference, which points future work toward hypothesis generation and experiment selection rather than planning or tool access.
  • The tested agentic harnesses did not beat their base models on the hardest tiers, indicating that current tool-scaffolding approaches do not yet add discovery ability.
  • All 70 games are human-solvable on a first attempt, establishing a human-calibrated ceiling and showing the benchmark is neither trivially easy nor impossible.
  • Even with the ground-truth rules, one game remained unsolved by the frontier model, so a small part of DiG-bench's difficulty lies outside rule discovery itself.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same hidden-rule format could be extended to scientific-discovery domains by encoding physical or causal systems as short strings, with the private-game split serving as a built-in contamination control.
  • Editorial inference: the rules-given control could be turned into a graded intervention—partial rules, hints, or candidate hypotheses—to map how much of discovery difficulty is hypothesis generation versus experiment planning, which the paper does not do.
  • Editorial inference: the one game that resisted even with the rules supplied suggests that complete rule knowledge does not always suffice, so a follow-up could check whether humans solve that game when the rules are stated, which would isolate execution failures from discovery failures.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces DiG-bench, a benchmark of 70 text-based games designed to isolate the capacity for active discovery. Each game has hidden transformation rules and unknown win conditions; agents observe short strings and act through single-character actions, and many games include a creative mode for sandboxed experimentation. The benchmark includes 21 public games and 49 private games arranged into seven difficulty tiers. The authors evaluate eight LLMs in a basic harness and four agentic harnesses, report that agentic harnesses do not improve over the basic harness, and show that giving Gemini 3.1 Pro the ground-truth rules raises its win rate from 18/70 to 69/70 while nearly eliminating creative-mode use. They also report that all 70 games were solved by at least one human on first attempt. The paper argues that the rules ablation supports the conclusion that the primary challenge of the benchmark is discovering the rules.

Significance. If the central claim holds, DiG-bench fills a genuine gap in the benchmark landscape: a controlled, perception-free environment for measuring active discovery, separable from planning and from rule induction from fixed demonstrations. The design has notable strengths: the private-game split supports secure evaluation, the human gold-standard calibration is a valuable sanity check, and the rules-given condition is a strong internal control for the discovery bottleneck. The release of a public API for the 21 public games supports reproducibility. The authors' candid discussion in Section 6 of possible interpolation from training data is also a strength, although it directly highlights the main risk to construct validity.

major comments (2)
  1. [Section 6 and Figure 5D] The central claim that no-rules wins reflect in-context discovery rather than memorization depends on the assertion that the games 'do not exist anywhere on the internet' (Section 6). This premise is verifiable only for the 21 public games; the 49 private games are not released, so contamination cannot be checked from the preprint. The public/private win-rate comparison in Figure 5D does not address this concern, because public games are intentionally released and private games could still leak into training corpora or appear as near variants. The paper's own Section 6 speculation that many short-horizon games 'lie at points that can be interpolated from training data' concedes the risk. I recommend a concrete contamination-check protocol, such as probing models for game-specific knowledge, testing on mutated or obfuscated variants, or evaluating on a fresh held-out set with a pre-registered protocol.
  2. [Section 3, Figure 5B] The rules ablation (18/70 to 69/70 for Gemini 3.1 Pro) demonstrates that knowing the rules is sufficient for near-perfect performance, but it does not establish that every no-rules win was achieved by active discovery. A win with minimal exploration or creative-mode use would be consistent with retrieval or interpolation, and the paper does not provide per-game exploration statistics. The average creative-mode reduction in Figure 5C is informative, but the central interpretation would be strengthened by reporting, for each no-rules win, the number of creative-mode entries, the number of distinct actions taken, and the number of novel states visited, to show that wins were preceded by informative experimentation.
minor comments (4)
  1. [Figure 5D] The rendered Figure 5D appears to contain stray axis labels ('0.5' and '15') that are not explained in the caption; please clean up the panel.
  2. [Section 2] The tier assignment is described only as 'according to machine difficulty'; please specify whether tiers were set using pilot runs, how many pilot runs per game, and whether the final reported evaluations were independent of the tier-assignment runs.
  3. [Appendix A] The first-turn prompt states that the TASK DESCRIPTION gives 'the objective and any special actions (not the rules)'. Since the objective may itself reveal part of the win condition, please clarify exactly what information was contained in each game's task description, so readers can assess how much rule structure is hidden.
  4. [Section 4.1] The paper notes that most model×game pairs had a single run and that cap-stopped runs are counted as losses; it would help to report how many games were lost solely because of the cost or wall-clock cap, since such losses are evidence of persistence rather than inability to discover.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark's claims are empirical measurements against handcrafted games, and the rules-ablation is a deliberate control rather than a fitted prediction; the private-game novelty concern is a validity risk, not a circular step.

full rationale

DiG-bench makes no fitted-parameter or derivation claim whose output equals its input. The central assertion that 'a primary challenge of the benchmark is finding out the rules' is supported by an intervention: rerunning Gemini 3.1 Pro with the ground-truth rules supplied and observing a win-rate increase from 18/70 to 69/70. This is an experimental control, not a circular inference, because the rules-given condition is an extra input rather than a quantity derived from the model's own play, and the improvement is measured against the same external games. Human beatability of all 70 games is likewise an external measurement. The paper's definition of discovery via compression cites Schmidhuber's prior work [21], but this definition does not determine any benchmark outcome; it only frames the construct being measured. The Section 6 statement that the games 'do not exist anywhere on the internet' and the authors' speculation that frontier models may interpolate from 'a vast array of RL environments in training' point to a genuine threat to construct validity—namely, that some no-rules wins may reflect memorization or interpolation rather than active discovery. That is an unverified premise about the evaluation environment, not a reduction of the benchmark's claimed result to its inputs, and it is not resolved by any equation or fitted parameter in the paper. Consequently, there are no circular steps; the appropriate score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim is a benchmark characterization, not a derivation, so there are no fitted constants in the mathematical sense. The ledger records the empirical calibration choices and unverifiable premises that the benchmark's validity rests on.

free parameters (1)
  • Difficulty tier assignment = 7 tiers, assigned from model win rates
    Games are sorted into seven tiers 'according to machine difficulty' (Section 2), based on the empirical win rates of the evaluated models. This is a hand/empirically chosen calibration that shapes Figure 1 and the claim that the benchmark is frontier-calibrated; it is not a derived quantity.
assumptions (3)
  • domain assumption The 70 games are novel and do not exist in models' training data
    Section 2 and Discussion assert the games are handcrafted and 'do not exist anywhere on the internet'. If any private or public game appears in training corpora, model wins could reflect retrieval rather than discovery, which would undermine the benchmark's construct validity.
  • domain assumption The rules-given vs no-rules comparison isolates rule discovery without introducing other helpful cues
    Section 3 and Figure 5B compare Gemini with and without a concise natural-language rules field. The interpretation that 'a primary challenge of the benchmark is finding out the rules' assumes the rules text adds only rule knowledge and not task-format cues, action semantics hints, or extra planning support.
  • domain assumption Single-seed runs with heterogeneous caps support the model rankings
    Section 4.1 states 'most model×game pairs had only a single run' and that capped runs count as losses; models also differed in context window and cost cap. The headline comparisons (Opus 5 50/70, Qwen 1/70, harness results) assume these protocol differences do not dominate the true capability signal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DiG-bench: Discovery in Games." pith.science (2026). https://pith.science/paper/HTZADGU2

@misc{pith2026260812593,
  author       = {Pith},
  title        = {Pith review of: DiG-bench: Discovery in Games},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HTZADGU2}},
  note         = {Machine review of arXiv:2608.12593}
}
read the original abstract

Discovery---formulating novel generalizations---is a central part of the scientific process. Despite its importance, there is a gap in the current AI benchmark landscape, with few benchmarks directly probing the capacity for discovering new knowledge with experimentation in controlled environments where the objective is unknown. To address this gap, we release a new benchmark: DiG-bench (Discovery in Games). DiG-bench consists of a set of 70 independent games. Each game is encoded as a short string and has unique transformation rules that must be discovered through interaction and experimentation. The levels of the game present a series of challenges to test whether the rules have been discovered, where the win conditions for each level are also unknown. We provide games at seven tiers of difficulty for AI agents. The lowest tier is routinely solvable by multiple models, while the highest tier challenges the best models in agentic harnesses. All 70 games were solved by at least one human on first attempt. A subset of 21 games is released publicly, and the remainder is held private for secure evaluation.

Figures

Figures reproduced from arXiv: 2608.12593 by the authors.

Figure 1
Figure 1. Model performance on DiG-bench. Proportion of games won by each model in each tier. Models in the basic harness were evaluated on all seven tiers (left). Models in agentic harnesses were evaluated only on tiers 6 and 7 (right). The horizontal dashed line indicates that humans beat every game. 1 arXiv:2608.12593v1 [cs.AI] 12 Aug 2026 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The DiG-bench platform. The platform and equivalent human/model interfaces. 1 the available actions (action ‘2’ is highlighted because the player is selecting it); 2 creative mode, an option in some games that allows a player to enter a sandbox for experimentation; 3 the current game observation; 4 indicators of current level, remaining lives and remaining steps; and 5 action/state history. 1 Introduction Discoverin… view at source ↗
Figure 3
Figure 3. Gameplay in game P-21. An abridged example of a gameplay timeline. Having beaten Levels 1 and 2, the player applies the rule learned on Level 2 (not shown): activating a held n with ‘.’ builds a bridge. But Level 3 contains a triple barrier vvv, and the bridge is too short. The player enters creative mode, a sandbox with a less restrictive step limit. Experimenting, the player discovers that standing on ~ has a trip… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Descriptive statistics of games. Distribution over all 70 games of the number of levels per game (left), the number of available actions (center), and the mean observation size (right). 0 50 100 150 200 250 Steps to beat a level 0.0 0.1 0.2 0.3 0.4 Fraction of levels A…
Figure 5
Figure 5. Figure 5: Gameplay data. (A) Steps to beat a level by humans versus Gemini 3.1 Pro, for all the levels beaten by both; the final bin includes 250 steps or more. (B) Giving Gemini access to the ground-truth rules of the game, specified in concise natural language, raises the win …
Figure 6
Figure 6. Figure 6: Progress on the one game Gemini 3.1 Pro did not beat even when given the rules. Levels beaten against steps taken. Dashed line is the run with rules supplied, solid line is without rules. Both conditions are Gemini 3.1 Pro. The dotted horizontal line marks game complet…
Figure 7
Figure 7. Figure 7: Performance on the five ARC-like games. (A) Games beaten by Gemini 3.1 Pro without rules. (B) Mean steps taken to beat a level by humans versus Gemini 3.1 Pro. Every level beaten by the best human play or the best Gemini 3.1 Pro run on each of the five games contribute…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 16 canonical work pages

  1. [1]

    ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence, March 2026

    ARC Prize Foundation. ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence, March 2026. URL http://arxiv.org/abs/2603.24621. arXiv:2603.24621 [cs.AI]

    arXiv 2026

  2. [2]

    FALSIFYBENCH: Evaluating Inductive Reasoning in LLMs with Rule Discovery Games

    Leonardo Bertolazzi, Katya Tentori, and Raffaella Bernardi. FALSIFYBENCH: Evaluating Inductive Reasoning in LLMs with Rule Discovery Games, June 2026. URLhttp://arxiv.org/abs/2606.04751 . arXiv:2606.04751 [cs.CL]

    work page Pith review arXiv 2026

  3. [3]

    How enabling two settings tripled our scores on the ARC-AGI-3 benchmark, July 2026

    Ilan Bigio and Ted Sanders. How enabling two settings tripled our scores on the ARC-AGI-3 benchmark, July 2026. URLhttps://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/

    work page 2026

  4. [4]

    PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors, July 2025

    Yimeng Chen, Piotr Piękos, Mateusz Ostaszewski, Firas Laakom, and Jürgen Schmidhuber. PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors, July 2025. URL http: //arxiv.org/abs/2507.15550. arXiv:2507.15550 [cs.AI]. 10

    arXiv 2025

  5. [5]

    CausalGame: Benchmarking Causal Thinking of LLM Agents in Games, July 2026

    Zhenhao Chen, Yongqiang Chen, Chenxi Liu, Junchi Yu, Xiangchen Song, Zijian Li, Jialin Li, Philip Torr, Bo Han, and Kun Zhang. CausalGame: Benchmarking Causal Thinking of LLM Agents in Games, July 2026. URLhttp://arxiv.org/abs/2607.04293. arXiv:2607.04293 [cs.CL]

    arXiv 2026

  6. [6]

    ARC Prize 2024: Technical Report, January 2025

    Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. ARC Prize 2024: Technical Report, January 2025. URLhttp://arxiv.org/abs/2412.04604. arXiv:2412.04604 [cs.AI]

    arXiv 2024

  7. [7]

    On the Measure of Intelligence, November 2019

    François Chollet. On the Measure of Intelligence, November 2019. URLhttp://arxiv.org/abs/1911.01547. arXiv:1911.01547 [cs.AI]

    arXiv 2019

  8. [8]

    ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems, May 2025

    François Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems, May 2025. URLhttp://arxiv.org/abs/2505.11831. arXiv:2505.11831 [cs.AI]

    arXiv 2025

Show all 36 references
  1. [9]

    Maddison

    Haonan Duan, Stephen Zhewen Lu, Caitlin Fiona Harrigan, Nishkrit Desai, Jiarui Lu, Michał Koziarski, Leonardo Cotta, and Chris J. Maddison. Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab, July 2025. URLhttp://arxiv.org/abs/2507.02083. arXiv...

    2025 arXiv

  2. [10]

    PRO-LONG: Programmatic memory enables long-horizon reasoning, July 2026

    Alexis Fox, Junlin Wang, Paul Rosu, and Bhuwan Dhingra. PRO-LONG: Programmatic memory enables long-horizon reasoning, July 2026. URLhttps://arxiv.org/abs/2607.20064. arXiv:2607.20064 [cs.AI]

    2026 arXiv

  3. [11]

    Li, Lyle Goodyear, Agam Bhatia, Louise Li, Aditi Bhaskar, Mohammed Zaman, and Noah D

    Kanishk Gandhi, Michael Y. Li, Lyle Goodyear, Agam Bhatia, Louise Li, Aditi Bhaskar, Mohammed Zaman, and Noah D. Goodman. BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery, January 2025. URLhttp://arxiv.org/abs/2501.01540. arXiv:2501.01540 [cs.AI]

    2025

  4. [12]

    Griffiths

    Jiayi Geng, Howard Chen, Dilip Arumugam, and Thomas L. Griffiths. Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systems, May 2025. URLhttp://arxiv.org/ab s/2505.17968. arXiv:2505.17968 [cs.CL]

    2025 arXiv

  5. [13]

    Szostkiewicz, Dmytro Shved, Gavin J

    Ali Essam Ghareeb, Benjamin Chang, Ludovico Mitchener, Angela Yiu, Caralyn J. Szostkiewicz, Dmytro Shved, Gavin J. Gyimesi, Jon M. Laurent, Samantha M. Wright, Muhammed T. Razzak, Andrew D. White, Silvia C. Finnemann, Michaela M. Hinks, and Samuel G. Rodriques. A multi-agent s...

    2026 doi

  6. [14]

    IOLBENCH: Benchmarking LLMs on Linguistic Reasoning, September 2025

    Satyam Goyal and Soham Dan. IOLBENCH: Benchmarking LLMs on Linguistic Reasoning, September 2025. URLhttp://arxiv.org/abs/2501.04249. arXiv:2501.04249 [cs.CL]

    2025

  7. [15]

    DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents, October 2024

    Peter Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord, and Peter Clark. DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents, October 2024. URLhttp://...

    2024

  8. [16]

    karpathy/autoresearch, June 2026

    Andrej Karpathy. karpathy/autoresearch, June 2026. URLhttps://github.com/karpathy/autoresearch. original-date: 2026-03-06T22:00:43Z

    2026

  9. [17]

    Zhang, Kevin Thomas, Sebastian Müller, and Prime Intellect Team

    Seth Karten, Alex L. Zhang, Kevin Thomas, Sebastian Müller, and Prime Intellect Team. Prime agent: A self-improving rlm harness.Prime Intellect Blog, August 2026. https://www.primeintellect.ai/blog/prime- agent

    2026

  10. [18]

    The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain, May 2023

    Arseny Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain, May 2023. URLhttp://arxiv.org/abs/2305.07141. arXiv:2305.07141 [cs.LG]

    2023 arXiv

  11. [19]

    Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning

    Weili Nie, Zhiding Yu, Lei Mao, Ankit B Patel, Yuke Zhu, and Anima Anandkumar. Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning. InAdvances in Neural Information Processing Systems, volume 33, pages 16468–16480. Curran Associates, Inc., 2020. URLhtt...

    2020

  12. [20]

    BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games, April 2025

    Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rocktäschel. BALROG: Benchmarking Agentic LLM and VLM Reasoning ...

    2025 arXiv

  13. [21]

    Jürgen Schmidhuber. Driven by compression progress: A simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes. InWorkshop on anticipatory behavior in adaptive learning sy...

    2008

  14. [22]

    ScienceWorld: Is your Agent Smarter than a 5th Grader?, November 2022

    Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. ScienceWorld: Is your Agent Smarter than a 5th Grader?, November 2022. URLhttp://arxiv.org/abs/2203.07540. arXiv:2203.07540 [cs.CL]

    2022 arXiv

  15. [23]

    Xinhe Wang, Jin Huang, Xingjian Zhang, Tianhao Wang, and Jiaqi W. Ma. Your Reasoning Benchmark May Not Test Reasoning: Revealing Perception Bottleneck in Abstract Reasoning Benchmarks, January 2026. URLhttp://arxiv.org/abs/2512.21329. arXiv:2512.21329 [cs.CL] version: 2

    2026

  16. [24]

    Holyoak, and Hongjing Lu

    Taylor Webb, Keith J. Holyoak, and Hongjing Lu. Emergent Analogical Reasoning in Large Language Models, August 2023. URLhttp://arxiv.org/abs/2212.09196. arXiv:2212.09196 [cs.AI]

    2023 arXiv

  17. [25]

    Springer

    Georgios N Yannakakis and Julian Togelius.Artificial intelligence and games, volume 2. Springer

  18. [26]

    Allen, Thomas L

    Lance Ying, Ryan Truong, Prafull Sharma, Kaiya Ivy Zhao, Nathan Cloos, Kelsey R. Allen, Thomas L. Griffiths, Katherine M. Collins, José Hernández-Orallo, Phillip Isola, Samuel J. Gershman, and Joshua B. Tenenbaum. AI Gamestore: Scalable, Open-Ended Evaluation of Machine Genera...

    2026

  19. [27]

    A Benchmark for Compositional Visual Reasoning, June 2022

    Aimen Zerroug, Mohit Vaishnav, Julien Colin, Sebastian Musslick, and Thomas Serre. A Benchmark for Compositional Visual Reasoning, June 2022. URLhttp://arxiv.org/abs/2206.05379. arXiv:2206.05379 [cs.CV]

    2022 arXiv

  20. [28]

    RAVEN: A Dataset for Relational and Analogical Visual rEasoNing, March 2019

    Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. RAVEN: A Dataset for Relational and Analogical Visual rEasoNing, March 2019. URLhttp://arxiv.org/abs/1903.02741 . arXiv:1903.02741 [cs.CV]

    2019 arXiv

  21. [29]

    creative mode

    Tianshi Zheng, Kelvin Kiu-Wai Tam, Newt Hue-Nam K. Nguyen, Baixuan Xu, Zhaowei Wang, Jiayang Cheng, Hong Ting Tsang, Weiqi Wang, Jiaxin Bai, Tianqing Fang, Yangqiu Song, Ginny Y. Wong, and Simon See. NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agent...

    2025

  22. [31]

    {session_id}

    Your game session is ALREADY started for you: session_id="{session_id}", game="{game}". You do NOT start or choose a game—you only have the{tools} tools, scoped to this one session, and you must pass thissession_id to every call. Your starting state (step_index={step_index}) i...

  23. [33]

    step” tool: session_id=

    Make a move with the “step” tool: session_id="{session_id}", step_index=<the server’s last returned step_index + 1> , action=<EXACTLY ONE string from the current state’s actions list> . Your first move uses step_index={first_step}. A step_index mismatch is a 409—always step of...

  24. [34]

    {session_id}

    Your game session is ALREADY started for you: session_id="{session_id}", game="{game}". You do NOT start or choose a game— you only have the game_tool.step and game_tool.get_session functions, scoped to this one session, and you must pass this session_id to every call. Your st...

  25. [35]

    Each turn, read the current state and reason from these fields:observation (the rendered screen), level, max_level, lives_left, steps_remaining, status, done, the actions list (your legal moves), andmode/transitionwhen present

  26. [36]

    {session_id}

    Make a move with game_tool.step(session_id="{session_id}", step_index=<the server’s last returned step_index + 1>, action=<EXACTLY ONE string from the current state’s actions list>) . Your first move uses step_index={first_step}. A step_index mismatch is a 409—always step off ...

  27. [37]

    Keep playing, one move per turn, until the state’sdone is true (statusgame_over or completed)

    Infer what each action does from how the state changes, and build on what you learn across turns. Keep playing, one move per turn, until the state’sdone is true (statusgame_over or completed). When the game is done, STOP making moves and write your debrief: the mechanics you d...

    2026

  28. [6769]

    arXiv:2406.06769 [cs.AI]

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.