{"id":"44d38f28-e255-41a0-a96c-143211797481","arxiv_id":"2608.12593","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A 70-game interactive benchmark where agents must discover hidden rules and objectives, with human beatability on every game and frontier models failing on the hardest tiers.","lead":"DiG-bench is a benchmark of 70 original text-based games in which AI agents must uncover hidden rules and win conditions through trial and error. It aims to measure whether language models can perform genuine discovery rather than retrieve memorized answers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's construct validity rests on the games being unseen; novelty for the 49 private games is asserted but unverifiable, and Section 6's interpolation speculation concedes the risk. A contamination check plus fresh held-out games is needed before no-rules wins can be read as discovery.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the games' novelty is asserted but not verifiable for the private set, and contamination would inflate no-rules win rates. I agree with this assessment and do not see a reason to move the verdict. The rules ablation is genuinely strong evidence for the benchmark's construct validity for Gemini, and the paper is honest about single-seed runs, cost caps, and context truncation, which supports a conditional rather than unconditional reading. The main reason the concern is load-bearing rather than cosmetic is that the paper's central comparison—win rate without rules versus with rules—interprets no-rules wins as discovery. If a meaningful fraction of those wins come from training-data interpolation, the headline 18/70 (and the collective 57/70) overstates the discovery capability being measured. Section 6's own speculation about RL-environment training data makes this more than a hypothetical. The proposed contamination probe would settle the question for the public games and, via a fresh held-out set, provide evidence about the private games. Until then, the conditional verdict is appropriate.","tokens_in":14002,"tokens_out":7012,"duration_ms":75614,"concrete_test":"Run a two-part contamination probe before further claims. (1) For the 21 public games, compute exact and near-duplicate similarity between each game's observation strings and rule summaries and a current training-corpus snapshot, and run a zero-shot probe: give the model only the initial state with the instruction to write the rules and a winning action sequence, with no environment feedback. If any public game is solved zero-shot or matches the corpus, exclude it from the headline rates and re-report. (2) To cover the private set, generate 10 fresh games using the same design process, keep them secret, and compare model win rates against the existing private set; a large drop would indicate that the reported private-game wins are partly memorization rather than discovery.","verdict_should_be":"UNCHANGED","load_bearing_attack":"DiG-bench's central claim—that no-rules wins reflect in-context discovery rather than memorized or interpolated behavior—depends on the assumption that the 70 games are novel to the models. Section 6 states that the games 'do not exist anywhere on the internet,' but this is checkable only for the 21 public games; the 49 private games are not released, so the premise cannot be verified from the preprint. The paper itself flags the residual risk in Section 6: the authors 'speculate this is due to the trend in frontier models toward including a vast array of RL environments in training; therefore many short-horizon games effectively lie at points that can be interpolated from training data.' Exact duplication is not required: if a model has trained on many similar short-horizon mechanics, a no-rules win can come from interpolation rather than from the active experimentation the benchmark is designed to measure. The rules ablation (18/70 to 69/70 for Gemini) shows that knowing the rules is sufficient for near-perfect performance, but it does not show that every no-rules win was achieved by discovery; a win with little exploration or creative-mode use is consistent with retrieval. Figure 5D's public/private win-rate match does not address contamination, since public games are released and private games can leak through training data. This is a load-bearing unverified premise, not an internal inconsistency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DiG-bench, a benchmark of 70 text-based games designed to isolate the capacity for active discovery. Each game has hidden transformation rules and unknown win conditions; agents observe short strings and act through single-character actions, and many games include a creative mode for sandboxed experimentation. The benchmark includes 21 public games and 49 private games arranged into seven difficulty tiers. The authors evaluate eight LLMs in a basic harness and four agentic harnesses, report that agentic harnesses do not improve over the basic harness, and show that giving Gemini 3.1 Pro the ground-truth rules raises its win rate from 18/70 to 69/70 while nearly eliminating creative-mode use. They also report that all 70 games were solved by at least one human on first attempt. The paper argues that the rules ablation supports the conclusion that the primary challenge of the benchmark is discovering the rules.","tokens_in":14258,"tokens_out":6106,"duration_ms":61644,"significance":"If the central claim holds, DiG-bench fills a genuine gap in the benchmark landscape: a controlled, perception-free environment for measuring active discovery, separable from planning and from rule induction from fixed demonstrations. The design has notable strengths: the private-game split supports secure evaluation, the human gold-standard calibration is a valuable sanity check, and the rules-given condition is a strong internal control for the discovery bottleneck. The release of a public API for the 21 public games supports reproducibility. The authors' candid discussion in Section 6 of possible interpolation from training data is also a strength, although it directly highlights the main risk to construct validity.","major_comments":[{"comment":"The central claim that no-rules wins reflect in-context discovery rather than memorization depends on the assertion that the games 'do not exist anywhere on the internet' (Section 6). This premise is verifiable only for the 21 public games; the 49 private games are not released, so contamination cannot be checked from the preprint. The public/private win-rate comparison in Figure 5D does not address this concern, because public games are intentionally released and private games could still leak into training corpora or appear as near variants. The paper's own Section 6 speculation that many short-horizon games 'lie at points that can be interpolated from training data' concedes the risk. I recommend a concrete contamination-check protocol, such as probing models for game-specific knowledge, testing on mutated or obfuscated variants, or evaluating on a fresh held-out set with a pre-registered protocol.","section":"Section 6 and Figure 5D"},{"comment":"The rules ablation (18/70 to 69/70 for Gemini 3.1 Pro) demonstrates that knowing the rules is sufficient for near-perfect performance, but it does not establish that every no-rules win was achieved by active discovery. A win with minimal exploration or creative-mode use would be consistent with retrieval or interpolation, and the paper does not provide per-game exploration statistics. The average creative-mode reduction in Figure 5C is informative, but the central interpretation would be strengthened by reporting, for each no-rules win, the number of creative-mode entries, the number of distinct actions taken, and the number of novel states visited, to show that wins were preceded by informative experimentation.","section":"Section 3, Figure 5B"}],"minor_comments":[{"comment":"The rendered Figure 5D appears to contain stray axis labels ('0.5' and '15') that are not explained in the caption; please clean up the panel.","section":"Figure 5D"},{"comment":"The tier assignment is described only as 'according to machine difficulty'; please specify whether tiers were set using pilot runs, how many pilot runs per game, and whether the final reported evaluations were independent of the tier-assignment runs.","section":"Section 2"},{"comment":"The first-turn prompt states that the TASK DESCRIPTION gives 'the objective and any special actions (not the rules)'. Since the objective may itself reveal part of the win condition, please clarify exactly what information was contained in each game's task description, so readers can assess how much rule structure is hidden.","section":"Appendix A"},{"comment":"The paper notes that most model×game pairs had a single run and that cap-stopped runs are counted as losses; it would help to report how many games were lost solely because of the cost or wall-clock cap, since such losses are evidence of persistence rather than inability to discover.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and the core experiments are internally consistent, but the construct validity of the benchmark rests on the unverified novelty of the 49 private games. The authors' own Section 6 interpolation speculation makes this a load-bearing issue rather than a stylistic concern. I would encourage the editor to ask for a concrete contamination-check protocol and, ideally, a small fresh held-out set before publication. The single-seed evaluations are a secondary concern but are not fatal for the central discovery claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DiG-bench is a genuinely new artifact: 70 handcrafted, text-only games with hidden rules and hidden win conditions, a creative-mode sandbox for experimentation, difficulty tiers calibrated to model performance, and a 21/49 public/private split. That combination is not in prior benchmarks, which tend to confound discovery with perception, prior knowledge, or planning. The paper's core result is the rules ablation: giving Gemini 3.1 Pro the ground-truth rules lifts its win rate from 18/70 to 69/70 and nearly eliminates creative-mode use. That is a strong control and the best evidence that the bottleneck is discovering the rules, not executing them. The human first-attempt solves of all 70 games are also solid evidence that the games are beatable without external knowledge.\n\nThe soft spots, in rough order of importance. First, the novelty claim for the 49 private games is asserted but unverifiable. The paper says the games 'do not exist anywhere on the internet,' but only the 21 public games can be checked. Section 6 even concedes that frontier models may interpolate from training on similar short-horizon environments, which means a no-rules win could reflect memorized or interpolated behavior rather than active discovery. The rules ablation does not close this: it shows rules are sufficient for near-perfect performance, not that every no-rules win required discovery. This is a real construct-validity concern, but it is the standard risk of any private benchmark and the authors flag it themselves. Contamination checks on the public games and eventual release of the private set would address it.\n\nSecond, the evaluations are mostly single-seed and budgets vary by model, so the tier curves and model rankings have unknown variance. The agentic-harness comparison is only partially matched, with different models and, for Prime Agent and PRO-LONG, a different interface. This is a moderate weakness, not load-bearing.\n\nThird, context windows differ (GPT-5.5 and Kimi K3 were capped well below 1M tokens), which could affect long games. The authors disclose this but do not control for it. Minor.\n\nCitation pattern is clean; the self-citations (Schmidhuber on compression, Griffiths/Tenenbaum on simulation) are on topic, and the related-work section covers the relevant adjacent benchmarks.\n\nThe paper deserves a serious referee. The benchmark is useful now, and the central claim holds up as far as the evidence goes. My recommendation: send it to review, and ask the authors for contamination analysis on the public games and at least a few seeds on the borderline tiers.","headline":"A well-built text-based benchmark for active discovery; the rules ablation is the strongest evidence, and the main soft spot is the unverifiable novelty claim for the private games.","tokens_in":14866,"tokens_out":3983,"would_cite":true,"duration_ms":36790,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DiG-bench is a benchmark of 70 text games with hidden rules and hidden win conditions, and its control experiment shows that supplying the ground-truth rules lifts a frontier model from 18/70 to 69/70 wins.","keywords":["discovery benchmark","rule induction","active experimentation","large language models","agentic harness","text games","scientific discovery","hidden rules"],"falsifier":"After the 49 private games are released, measure the same models' win rates on them before and after the games enter public training corpora; a sharp rise after release would show that contamination, not in-context discovery, explains the original scores.","tokens_in":13802,"feed_emoji":"🎮","tokens_out":9048,"duration_ms":78847,"temperature":0.7,"pith_summary":"DiG-bench is a benchmark of 70 handcrafted, text-only games in which both the transformation rules and the win conditions are hidden, forcing a player to discover them through active experimentation. The paper's central claim is that the benchmark isolates discovery from perception, planning, and memorized instructions, because the games live in the text domain of large language models and contain no visual confounds. The key supporting evidence is a rules-given control: when one frontier model received the ground-truth rules in natural language, its win rate rose from 18 of 70 games to 69 of 70, and humans solved all 70 games on first attempt while the strongest model solved 50. If the claim holds, DiG-bench offers a calibrated, human-verified probe of a capability that most current benchmarks either confound with perception or bypass by stating the rules upfront.","feed_headline":"Giving AI the rules lifts wins from 18 to 69 of 70","feed_subtitle":"DiG-bench hides the rules of 70 text games; humans solve all, top models stall until rules are given.","key_machinery":"The central object is the hidden-rule game itself: a miniature world encoded as a short Unicode string that acts as a partially observable Markov decision process (POMDP), where both the transition rules and the win conditions must be inferred from the observation-action history. Levels, lives, and per-level step limits structure progression, and many games include a creative mode—a sandbox where steps are free—so the player can run informative experiments before spending scarce survival steps. The attribution mechanism is the rules-given control condition, in which a concise natural-language description of the true dynamics and win condition is added to the prompt; the jump from 18/70 to 69/70 wins under that condition isolates rule inference as the difficult component.","core_discovery":"The paper's central discovery claim is that rule discovery—finding the hidden laws of a novel interactive system—is the operative bottleneck for current language-model agents, and that DiG-bench measures this bottleneck in isolation. Seventy short-string games, each with unique mechanics and unknown objectives, are beaten by at least one human on first exposure, yet the best tested model wins only 50 of 70, and models in agentic harnesses did not outperform their base models on the hardest tiers. The decisive comparison is the rules-given condition: with the true rules supplied as concise natural language, one frontier model won 69 of 70 games, against 18 of 70 without them, which the paper reads as direct evidence that finding out the rules is the primary challenge.","pith_inferences":["Editorial inference: the same hidden-rule format could be extended to scientific-discovery domains by encoding physical or causal systems as short strings, with the private-game split serving as a built-in contamination control.","Editorial inference: the rules-given control could be turned into a graded intervention—partial rules, hints, or candidate hypotheses—to map how much of discovery difficulty is hypothesis generation versus experiment planning, which the paper does not do.","Editorial inference: the one game that resisted even with the rules supplied suggests that complete rule knowledge does not always suffice, so a follow-up could check whether humans solve that game when the rules are stated, which would isolate execution failures from discovery failures."],"forward_implications":["DiG-bench provides a perception-free, text-only setting in which discovery and experimentation can be scored directly, so model performance on it is not explained by visual priors or spatial reasoning.","Because supplying the rules lifts a frontier model from 18/70 to 69/70 wins, the benchmark's hard part is rule inference, which points future work toward hypothesis generation and experiment selection rather than planning or tool access.","The tested agentic harnesses did not beat their base models on the hardest tiers, indicating that current tool-scaffolding approaches do not yet add discovery ability.","All 70 games are human-solvable on a first attempt, establishing a human-calibrated ceiling and showing the benchmark is neither trivially easy nor impossible.","Even with the ground-truth rules, one game remained unsolved by the frontier model, so a small part of DiG-bench's difficulty lies outside rule discovery itself."],"supporting_citations":[{"why":"Supplies the closest interactive-discovery baseline, whose visual-grid format this benchmark removes to isolate discovery.","marker":"[1]"},{"why":"Represents a prior experimentation benchmark that entangles discovery with prior scientific knowledge, motivating the controlled text-game design.","marker":"[15]"},{"why":"An agentic game-reasoning suite with partly known mechanics; establishes the class of environments whose confounds DiG-bench strips away.","marker":"[20]"},{"why":"An agentic harness reported to score highly on an interactive reasoning benchmark; its lack of improvement here supports the agentic-scaffolding result.","marker":"[17]"},{"why":"An agentic harness used with a frontier model; it did not outperform the basic harness, supporting the claim about limited gains from scaffolding.","marker":"[10]"},{"why":"Prior study of reverse-engineering black-box programs showing that active intervention helps only patchily, motivating the focus on experimentation.","marker":"[12]"},{"why":"Supplies the compression-progress definition of discovery that frames the benchmark's goal.","marker":"[21]"},{"why":"Justifies preserving native reasoning-continuity mechanisms in the basic harness so models receive their best chance of performing well.","marker":"[3]"}],"fun_headline_variants":["Human 70/70, AI 18/70 until rules revealed → 69","Rule discovery bottleneck: AI wins 18/70, 69 with hints","DiG-bench: Hidden rules are the AI bottleneck","DiG-bench: 70 games, hidden rules trip AI","AI needs rule book: 18 vs 69 wins on DiG-bench"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark is a valid measure of discovery, rather than memorized behavior, only if the games and their near variants are genuinely absent from every tested model's training data, a premise the authors assert for the 49 private games but cannot verify.","fun_headline_variants_meta":{"raw":{"variants":["Human 70/70, AI 18/70 until rules revealed → 69","Rule discovery bottleneck: AI wins 18/70, 69 with hints","DiG-bench: Hidden rules are the AI bottleneck","DiG-bench: 70 games, hidden rules trip AI","AI needs rule book: 18 vs 69 wins on DiG-bench"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00096,"raw_usage":{"total_tokens":4055,"prompt_tokens":879,"completion_tokens":3176,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":3078}},"tokens_in":495,"tokens_out":3176,"duration_ms":21056,"temperature":1.0,"reasoning_tokens":3078,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:03:42.152276+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"After the 49 private games are released, measure the same models' win rates on them before and after the games enter public training corpora; a sharp rise after release would show that contamination, not in-context discovery, explains the original scores.","supporting_citations":[{"cited_title":"DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents, October 2024","cited_arxiv_id":null,"evidence_quote":"Represents a prior experimentation benchmark that entangles discovery with prior scientific knowledge, motivating the controlled text-game design."},{"cited_title":"Zhang, Kevin Thomas, Sebastian Müller, and Prime Intellect Team","cited_arxiv_id":null,"evidence_quote":"An agentic harness reported to score highly on an interactive reasoning benchmark; its lack of improvement here supports the agentic-scaffolding result."},{"cited_title":"PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning","cited_arxiv_id":"2607.20064","evidence_quote":"An agentic harness used with a frontier model; it did not outperform the basic harness, supporting the claim about limited gains from scaffolding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the compression-progress definition of discovery that frames the benchmark's goal."},{"cited_title":"How enabling two settings tripled our scores on the ARC-AGI-3 benchmark, July 2026","cited_arxiv_id":null,"evidence_quote":"Justifies preserving native reasoning-continuity mechanisms in the basic harness so models receive their best chance of performing well."}],"review_version":1}