{"id":"638e682d-af79-434a-9360-2669f64b9736","arxiv_id":"2412.13549","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new escape-room benchmark measures creative reasoning in language-model agents, and the EscapeAgent framework (Foresight plus Reflection) reduces hints and steps by up to 40%.","lead":"EscapeBench puts language-model agents inside 36 text-based escape rooms that require creative tool use and implicit goal discovery. Current models make only about 15% of total progress without hints, and a new agent framework called EscapeAgent cuts hints and steps by up to 40%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 15% progress metric conflates creative reasoning with environment-following and long-horizon execution; without a recipe-following control, the benchmark's creativity attribution is unsupported.","rationale":"The most load-bearing assumption is that EscapeBench's progress metric isolates creative reasoning. The paper presents no control condition that separates 'thinking of the unconventional action' from 'executing a long sequence of actions in a text environment.' Since the abstract's headline number (15% progress without hints) is the primary evidence for LMs' lack of creativity, a confounding variable would invalidate the core diagnostic claim and, by extension, the interpretation of EscapeAgent's improvements. The proposed recipe condition is a minimal, feasible check: it uses the authors' own annotated key steps as instructions, removing the need for creative discovery while keeping every other aspect (state space, feedback, hint threshold) identical. If the recipe condition also fails, the benchmark may still be a useful stress test of agent robustness, but it cannot be interpreted as a creativity benchmark. This concern does not require rejecting the paper; it requires additional evidence, so the reader's CONDITIONAL verdict is appropriate.","tokens_in":23222,"tokens_out":10965,"duration_ms":101611,"concrete_test":"Use GPT-4o BaseAgent on all 36 settings under two conditions: (a) the original EscapeBench, and (b) a 'recipe' variant where, at each step, the system prompt includes the next annotated key step and the agent is instructed to execute it (identical action space, environment, and hint policy). Compare early-exit progress and hints used. If progress in (b) is near-ceiling while (a) remains ~25%, the deficit is specifically about discovering the creative step. If (b) shows similar stalls and low progress, the benchmark's difficulty is dominated by environment-following or long-horizon execution, and the 'creativity' attribution is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that LMs achieve only ~15% progress without hints because of limited creative reasoning—rests on the operational definition in §5.1 that 'making progress' means achieving a human-annotated key step or collecting a tool. This assumes the gap between models and this ground truth is attributable to creativity. But the paper's own error analysis (Table 4) identifies environment-following as a separate failure mode, and the hint mechanism (Appendix D.1, 50-action stall threshold) means the metric is entangled with the annotation granularity and the linearized solution chain. Critically, no condition is run that controls for the non-creative demands of the task: navigating a large state space, parsing feedback, and executing long action sequences. Without such a control, low early-exit progress could be explained by memory limitations (working memory length is fixed at 10), action-format errors, or the sheer difficulty of 100+ step credit assignment, rather than by a specific deficit in 'creative reasoning.' The human baseline in Table 3 is also suspect: Appendix E indicates it is measured from the annotators who authored the solution chains, which can only inflate the perceived human–LM gap. A recipe-following condition—where the agent is told the next key step at each point—would isolate whether the bottleneck is the discovery of the creative step or the ability to carry out a long, structured plan.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"EscapeBench constructs 36 text-based room-escape game settings with three difficulty variants and a five-action interface (Move, Click, Apply, Input, Craft), annotated with key steps and linearized solution chains by eight annotators. The paper evaluates ten LMs inside a BaseAgent that has chain-of-thought reasoning and working memory, reports that models make only about 15% average early-exit progress without hints, and then introduces EscapeAgent with Foresight and Reflection modules that reduce hint usage and total steps. The central claim is that these results measure creative intelligence and that current LMs are markedly deficient in it.","tokens_in":23505,"tokens_out":6513,"duration_ms":60991,"significance":"The resource itself is potentially valuable: the environments target genuine open-ended tool use and implicit goal discovery, and the release of code and data plus the detailed appendix documentation are strengths. The multi-model comparison and ablations give a first map of agent performance in this setting. However, the paper's headline quantity, creative reasoning, is not yet cleanly isolated: the absence of a control condition that removes the creative-discovery component, together with the small author-annotator human baseline, means the strong interpretation of the results is not yet established. If the authors add a recipe-following control and a more credible human baseline, the benchmark could become a useful community resource.","major_comments":[{"comment":"The load-bearing interpretation that roughly 15% early-exit progress reflects a deficit in creative reasoning is not supported as stated. Progress is defined as achieving a human-annotated key step or collecting a tool, and the hint mechanism in Appendix D.1 gives the next uncompleted action from a linearized solution chain whenever the agent stalls for 50 actions. These design choices mean the metric is entangled with annotator granularity, the linearization of the solution chain, long-horizon credit assignment, and the fixed memory length of 10, in addition to any creative-discovery difficulty. The paper's own error analysis in Table 4 lists environment-following as a separate failure mode. To support the creativity attribution, the authors should add a control condition in which the agent is told the next key step, or given the next target action, at each point, so that discovering the creative step is not required; if such a recipe-following agent still shows low progress or high hint dependence, the bottleneck is execution, not creativity.","section":"§5.1, 'Early Exit Progress' definition; Table 3"},{"comment":"The 'Average Human' row is not a fair reference for the human-model gap claimed in §5.3 and Figure 7. Appendix E states that the human baseline is drawn from the eight annotators who authored the game logic and solution chains; they therefore know the solution to each game before playing, and the group is small, homogeneous (U.S.-based CS students), and self-selected for prior room-escape experience. The comparison between this expert/author baseline and a zero-shot LM baseline is not an average-human comparison. The authors should either run a separate user study with naive, independent players, reporting sample size, variance, and completion rates, or explicitly relabel the baseline as an expert/author upper bound and soften claims such as 'humans rarely make mistakes.'","section":"Appendix E; Table 3"},{"comment":"All model numbers are reported as point estimates from a single run at T=0 with no confidence intervals or repeated seeds. Closed-source APIs are not deterministic in practice, and many of the differences that support the headline reductions, such as hint reductions ranging from 21% to 51% across models and early-exit progress improvements of 0.08 to 22.28 points, are presented without any noise model. The quantitative claims in the abstract and §5.3, including 'achieve only 15% average progress' and 'up to 40% fewer steps and hints,' need at least three repeated runs per model and per condition, or another explicit account of why single-run T=0 evaluation is stable for this environment. This is especially important because the human comparison in Table 3 itself has no variance.","section":"Tables 3 and 5; §5.3"},{"comment":"The paper does not state whether EscapeAgent's prompts, module triggers, and thresholds were tuned or selected using the same 36 game settings and key-step annotations that are later used for evaluation. The Foresight prompts in Appendix C contain substantial task-specific guidance, such as evaluating each tool against the task list and deciding among click, apply, and input, and the hint mechanism uses the same annotation chain. If the framework was iteratively developed on these games, the reported improvement over BaseAgent is an optimistic in-sample estimate. Reporting a development/validation split, or at minimum describing any tuning process and how much the design was adjusted on the test set, is necessary to assess the generality of EscapeAgent.","section":"§4-§5; Appendix C"}],"minor_comments":[{"comment":"The claim of logical coherence over 1,000-step action chains appears in the abstract, but no explicit coherence metric is defined; clarify whether this refers to completion curves, action success rates, or task-list consistency.","section":"Abstract; §5.3"},{"comment":"Typographical errors should be corrected, including 'EscapAgent' in the Table 5 caption, 'LLama-3.1-70B' in §3.4, 'out main table' in §6.1, 'whcih' in Appendix C, 'acton' in Appendix C.3, and 'shuold' in Appendix C.3.","section":"Throughout"},{"comment":"The notation with an arrow plus two numbers, such as '5.03↓5.27,' is ambiguous; state clearly which value is BaseAgent, which is EscapeAgent, and what the arrow denotes, and use consistent formatting with Table 3.","section":"Tables 3 and 5"},{"comment":"The normalized columns 'Tool Hints Used (percentage)' and 'Key Step Hints Used (percentage)' should specify the normalization denominator in the table caption or text; as printed, the raw count followed by a percentage is easy to misread.","section":"Table 3; Figure 5"},{"comment":"The help instruction says the next target location and next target action will be provided, but it is unclear whether the hint is repeated verbatim at every stalled step or only once; specify the exact help schedule.","section":"Appendix D.1"},{"comment":"The claim of being 'the first benchmark for LM creativity' is too strong given that the paper itself discusses earlier AUT and TTCT-style evaluations; suggest softening to 'the first agentic room-escape benchmark for LM creativity.'","section":"§8"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The benchmark resource and release are real contributions, but the current framing overclaims creative intelligence. The strongest fixes are the recipe-following control condition and a clearly labeled, independent human baseline; without them, the central attribution of low progress to a creativity deficit remains unsupported. The concerns are about experimental design and statistical reporting, not about misconduct."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on agent evaluation. The benchmark is a real artifact: 36 hand-annotated room escape games with three difficulty variants, human-annotated key steps, and released code and data. The early-exit progress metric (key steps plus tool collections before the first hint) is a reasonable measure of autonomous advancement in a long-horizon task. And EscapeAgent shows consistent, reproducible gains across a dozen models, with ablations supporting both the Reflection and Foresight modules. That is the core contribution, and it is solid.\n\nBut the paper overinterprets what the metric measures. The headline claim that LMs achieve only ~15% progress because they lack 'creative reasoning' is not supported by the experiments as designed. The paper's own error analysis lists environment-following as a separate failure mode, and there is no control condition that gives the agent the next key step to isolate the creative-discovery component from memory, action parsing, and credit assignment over 100+ steps. The 50-action hint threshold also ties the metric to annotation granularity. Without a recipe-following baseline, the creativity attribution is an assertion, not a finding.\n\nThe human baseline is a larger soft spot. Appendix E essentially measures the eight annotators who authored the solution chains, which surely inflates the human-LM gap, and with no variance or significance testing on any of the headline numbers, the up-to-40% improvements are plausible but not nailed down. These are fixable: add the recipe-following control, report run-to-run variance (T=0 notwithstanding), and either recruit a fresh human sample or stop calling it 'average human.'\n\nNone of this sinks the benchmark. If you read it as a testbed for long-horizon tool-use exploration and agent efficiency, the work is valuable and the EscapeAgent gains are real. It just is not a clean measure of 'creative intelligence' yet.\n\nFor peer review: yes, send it out. A serious referee should ask for the control experiment and a recalibrated discussion of creativity. But the environment and the released code are worth the field engaging with.","headline":"A solid new benchmark for long-horizon agent exploration, but the 'creativity' label outruns what the metrics actually isolate.","tokens_in":24025,"tokens_out":1463,"would_cite":true,"duration_ms":16361,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Language-model agents, even with working memory and chain-of-thought reasoning, make only about 15 percent of the progress needed to finish room-escape puzzles without hints, and a new framework with Foresight and Reflection modules cuts…","keywords":["creative intelligence","language model agents","room escape games","benchmark","tool use","implicit goal discovery","Foresight and Reflection","long-horizon reasoning"],"falsifier":"Run EscapeBench under an ablation that replaces Foresight and Reflection with a simple no-repeats memory that records every tried action and forbids retrying it; if that rule-based agent reaches EscapeAgent-level hint-free progress, then the measured gains come from memory management, not creative reasoning.","tokens_in":23055,"feed_emoji":"🧩","tokens_out":7379,"duration_ms":59548,"temperature":0.7,"pith_summary":"This paper introduces EscapeBench, a suite of 36 room-escape game settings designed to test whether language-model agents can reason creatively in unfamiliar environments, not just follow explicit goals. The central finding is that current models, even when equipped with working memory and Chain-of-Thought reasoning, make only about 15% progress on their own before needing a hint, while average humans reach about 60% before their first hint. To close part of that gap, the paper proposes EscapeAgent, which adds a Foresight module for hypothesizing tool uses before acting and a Reflection module that keeps an explicit task list of unsolved goals and failed attempts. Across ten models, EscapeAgent reduces hint use and total steps by up to 40%, sustains logically coherent action chains of more than 1,000 steps, and improves robustness across difficulty levels. The paper's contribution is a reproducible way to measure creative intelligence in agents and a framework that raises that measured performance.","feed_headline":"Language agents make only ~15% progress in escape-room games unaided","feed_subtitle":"EscapeAgent's Foresight and Reflection modules cut hint use by up to 40 percent in escape-room tasks.","key_machinery":"The machinery is a graph-structured game engine of scenes, items, and tools with a five-action space (Move, Click, Apply, Input, Craft) and human-annotated 'key steps' that define progress; an agent is scored on Hints Used, Total Steps, and Early Exit Progress. On top of the BaseAgent (working memory plus Chain-of-Thought), EscapeAgent adds two modules: Foresight, which hypothesizes and evaluates tool applications and crafting combinations before acting, and Reflection, which maintains a task list with new/update/delete operations so that failed attempts are recorded and completed goals are removed. The key steps and the hint-after-50-stalls rule are what convert free-form puzzle-solving into a measurable creativity score.","core_discovery":"The paper's core claim is that creative reasoning can be operationalized and measured in a text-based agent benchmark, and that current language models fail at it. The evidence is that the best BaseAgent configurations—GPT-4o and Claude-3.5-Sonnet—achieve only about 24–29% early-exit progress (progress made before the first hint), that most hints are consumed on 'key steps' that require creative reasoning, and that 'Input' and 'Craft' actions, the most open-ended, demand the highest relative hint usage. EscapeAgent then shows that these failures are partly addressable: combining Foresight (proposing and evaluating tool-use hypotheses before executing) and Reflection (maintaining an unsolved-task list) improves early-exit progress, reduces hint dependence and steps, and lets agents carry out coherent action chains exceeding 1,000 steps.","pith_inferences":["If progress without hints is accepted as a proxy for creativity, the benchmark invites a direct construct-validity test: vary feedback granularity and memory width while holding game logic fixed and see how much of the human–model gap is attributable to creativity versus memory or instruction-following.","The Foresight–Reflection pattern could transfer to other open-ended agent settings, such as web navigation or embodied exploration, where goals are implicit and tool affordances are not memorized.","The authors' own data suggest an additional testable extension: giving agents a 'creativity hint' that points to an unusual tool affordance (e.g., 'the stick's hardness') might selectively improve performance, which would verify that the bottleneck is hypothesis generation rather than action execution.","A step-reward reinforcement learning variant, suggested in the appendix, would convert EscapeBench from a measurement instrument into a training signal; if step-level rewards improve hint-free progress more than outcome-only rewards, that would support the benchmark's decomposition of creativity into identifiable progress steps."],"forward_implications":["The 15-percent hint-free progress figure gives a concrete, reproducible baseline: an agent that improves this number without extra hints is demonstrably better at creative problem-solving in unfamiliar environments.","EscapeAgent's gains are consistent across closed and open models, with larger models benefiting most, which implies the core model's reasoning capacity remains a binding constraint on creativity.","The ablation shows Foresight and Reflection are complementary, so either module alone is a weaker intervention; a full creative-reasoning framework needs both goal tracking and hypothesis generation.","Models below about 7B parameters behave close to randomly on this benchmark, establishing a practical scale floor for future agent-creativity evaluations.","Domain-specialized models (e.g., math-tuned) can lose creative flexibility, suggesting that over-specialization is measurable in this benchmark."],"supporting_citations":[{"why":"Supplies the Chain-of-Thought reasoning that the BaseAgent baseline uses at every step.","marker":"Wei et al., 2022"},{"why":"Provides the triarchic theory of intelligence that motivates treating creative intelligence as a distinct axis to benchmark.","marker":"Sternberg, 1984"},{"why":"Defines the three mechanisms of AI creativity that frame the benchmark's creative-tool-use tasks.","marker":"Boden, 1998"},{"why":"TextWorld is the canonical text-based game environment that EscapeBench extends toward creativity-oriented puzzles.","marker":"Côté et al., 2019"},{"why":"GPT-4o is the strongest closed model benchmarked and the backbone in the main case studies.","marker":"Hurst et al., 2024"}],"fun_headline_variants":["Language agents only 15% progress in creative escape-room tasks","EscapeBench: Language models fail at creative reasoning, 15% progress","Creative escape rooms: Language agents need hints, only 15% independent progress","EscapeAgent cuts hints by 40% in escape-room creative reasoning test","Language agents: 15% creative progress, EscapeAgent shows 40% improvement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's definition of creativity is that making progress (hitting a human-annotated key step or collecting a tool) without hints reflects creative reasoning; if that progress is actually gated by memory width, action parsing, or environment-following rather than creativity, the central conclusion about creative intelligence is unsupported.","fun_headline_variants_meta":{"raw":{"variants":["Language agents only 15% progress in creative escape-room tasks","EscapeBench: Language models fail at creative reasoning, 15% progress","Creative escape rooms: Language agents need hints, only 15% independent progress","EscapeAgent cuts hints by 40% in escape-room creative reasoning test","Language agents: 15% creative progress, EscapeAgent shows 40% improvement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000938,"raw_usage":{"total_tokens":3985,"prompt_tokens":895,"completion_tokens":3090,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":2991}},"tokens_in":511,"tokens_out":3090,"duration_ms":20681,"temperature":1.0,"reasoning_tokens":2991,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:00:46.359539+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EscapeBench under an ablation that replaces Foresight and Reflection with a simple no-repeats memory that records every tried action and forbids retrying it; if that rule-based agent reaches EscapeAgent-level hint-free progress, then the measured gains come from memory management, not creative reasoning.","supporting_citations":[],"review_version":1}