{"id":"4c3d0853-75ba-4e2c-8900-2c7655e5ef66","arxiv_id":"2507.23701","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Frontier LLMs complete few of 25 Infocom text adventures even when given the official hint booklets, revealing a weakness in sustained long-context reasoning.","lead":"The paper introduces TextQuests, a benchmark of 25 classic Infocom text adventure games, and runs frontier LLMs on them with and without official clue booklets. It finds that even the best models complete few games, placing most of the burden on long-context exploratory reasoning rather than tool use or memorized knowledge.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Checkpoint labels are the load-bearing pillar: Section 3.2's Game Progress metric is unvalidated, with no annotation protocol or inter-rater reliability, so the headline 37.8%/70.0% numbers may encode labeler choices as much as model ability.","rationale":"The central quantitative claim is the average Game Progress scores in Tables 1 and 4; without these numbers the paper is mostly a collection of qualitative failure anecdotes. Those averages are computed from P_g, the maximum percentage of manually labeled checkpoints completed, so the labels are the conversion function between raw trajectories and every headline number. If checkpoint sets are not comparable in size, difficulty, or ordering across the 25 games, the benchmark averages are not a clean measure of model ability. The reader's weakest-assumption analysis identifies exactly this point, and I agree. I considered whether the single-run/no-error-bars issue is more load-bearing; it is real but secondary, because the 0/25 completion counts and the large NO CLUES/WITH CLUES gap would almost certainly survive re-seeding. I also considered the absence of a human baseline; that affects interpretation of 'far from human-level' but not the internal validity of the progress metric. The right remedy is not rejection: the completion counts give some independent support, and the checkpoint files are promised at textquests.ai. If the released repository contains the checkpoint definitions and reproduction scripts, and an independent annotation check confirms stability, the paper should move to ACCEPT. Until then, CONDITIONAL is the correct verdict, so my read leaves the reader's verdict unchanged.","tokens_in":18008,"tokens_out":7186,"duration_ms":79322,"concrete_test":"Release the full checkpoint definitions and rubric for all 25 games, then have two independent annotators, blind to model trajectories, re-derive checkpoint sets for a stratified sample of five games from official walkthroughs and solution transcripts. Compute agreement on checkpoint membership and ordering (e.g., Cohen's kappa), then recompute Tables 1 and 4 with each annotator's sets. If headline averages move by more than roughly 5 percentage points, or if any top-model ranking flips, the Game Progress metric is not robust enough to support the quantitative claims. Also report per-game checkpoint counts and the distribution of P_g so readers can assess density effects.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 3.1 claim that TextQuests measures 'self-contained problem-solving' and that frontier LLMs make minimal progress, with GPT-5 at 37.8% NO CLUES and 70.0% WITH CLUES. Every quantitative score in Tables 1 and 4 is the average of P_g, defined in Section 3.2 as 'the maximum percentage of checkpoints the agent completes.' The validity of this proxy is therefore load-bearing. The paper states that the checkpoints are labeled milestones for 'essential puzzles and game milestones' and that the benchmark builds on Hendrycks et al. (2021c), but it provides no annotation protocol, no inter-rater reliability, no checklist of which game events count as checkpoints, and no independent check that checkpoint density or ordering is comparable across the 25 games. If one game has a single early checkpoint while another has twenty late checkpoints, the same behavior yields very different P_g values, so the averaged rankings in Tables 1 and 4 are partly artifacts of the labelers' choices. The 0/25 completion counts are objective and robust, but the headline percentage and the 'minimal progress' magnitude are not. The missing audit is fixable but currently unresolved, and without it the central quantitative claim is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TextQuests, a benchmark of 25 classic Infocom interactive fiction games, and evaluates a range of LLMs as agents in a no-tools, no-truncation, single-session setting. The agent receives the full growing history of observations, reasoning, and actions, and can use an autosave/restore mechanism. The central claim is that frontier LLMs make minimal progress without assistance, with the headline numbers being GPT-5 at 37.8% Game Progress in NO CLUES mode and 70.0% WITH CLUES, and zero games completed by any model in NO CLUES mode. The paper also reports harm scores and qualitative failure modes such as hallucinating prior actions and failing to reverse spatial paths, and it provides token-efficiency analyses. The empirical conclusion that current models are far from solving these exploratory games is plausible and partly supported by objective completion counts, but the quantitative Game Progress scores rest on unvalidated manual checkpoints and single runs per model.","tokens_in":18227,"tokens_out":6325,"duration_ms":68464,"significance":"If the quantitative metric is validated, TextQuests would fill a genuine gap: most agent benchmarks test tool use or structured tasks, while TextQuests targets sustained, self-directed reasoning over a long, growing context. The benchmark release, the autosave/restore design, the 500-to-800-step saturation analysis, and the detailed token-cost accounting are concrete contributions. The objective completion counts (0/25 in NO CLUES for every model, at most 5/25 WITH CLUES) are robust and independently support the qualitative finding that current models struggle. The qualitative trajectory analyses, especially the Wishbringer and Zork I examples, are informative and give the paper face validity. The main weaknesses are the unvalidated checkpoint-based progress metric and the lack of repeated runs, both of which affect the precision of the headline percentages.","major_comments":[{"comment":"The Game Progress metric is the load-bearing quantitative instrument: Tables 1 and 4 average P_g, the maximum fraction of manually labeled checkpoints reached. Yet the paper reports no annotation protocol, no inter-rater reliability, no checkpoint inventory, and no independent check that checkpoint density or ordering is comparable across the 25 games. If one game has a single early checkpoint while another has twenty late checkpoints, the same agent behavior would yield very different P_g values, so the averaged rankings and the magnitudes such as 37.8% and 70.0% are partly determined by labeler choices. Appendix F provides illustrative comparisons but not a systematic audit. This issue is fixable: the authors should release the checkpoint lists, describe the annotation guidelines, report inter-rater agreement, and provide a sensitivity analysis (for example, leave-one-game-out or an alternative checkpoint construction).","section":"Section 3.2"},{"comment":"The evaluation uses a single run per model per mode, with no repeated seeds, confidence intervals, or per-game variance. P_g is the maximum over a stochastic trajectory, and the qualitative examples show that one wrong restore or one hallucinated action can halt progress, so the point estimates may be highly variable. Claims such as the importance of model scale in Figure 3 and the ordering of close scores (for example, Gemini 2.5 Pro at 23.2% versus Claude Sonnet 4 at 24.7% in NO CLUES mode) are not statistically supported. The authors should report per-game scores, provide robust standard errors across the 25 games, and repeat at least the frontier-model evaluations with multiple seeds.","section":"Section 3.1 / Table 1"},{"comment":"The paper interprets the trajectories as evidence of 'better long-horizon reasoning' in more capable models, but no controlled comparison isolates context length from general planning ability. A model with better short-horizon planning would show the same pattern of sustained progress, so the advertised specificity to 'intrinsic long-context reasoning' is not yet demonstrated. In addition, the 'Human Optimal' line in Figure 2 is not described with a protocol or a human baseline; it appears to be an optimal walkthrough. The claims should either be softened or supported by an ablation such as the same model with truncated or summarized history, or with an external memory.","section":"Section 4.2 / Figure 2"}],"minor_comments":[{"comment":"Figure 8 is corrupted in the manuscript text, with many '/gid...' token-like strings replacing the intended diagram and JSON example; it should be regenerated as a clean figure.","section":"Appendix D / Figure 8"},{"comment":"The reference list contains formatting artifacts, including a split name entry ('Huanxu, Liu') and a duplicate 'Jay Paek' entry from the HLE author list; these should be cleaned.","section":"References"},{"comment":"The abstract emphasizes 'precluding the use of external tools' and 'self-contained problem-solving,' but Appendix A.1.1 states that feelie text is provided in both NO CLUES and WITH CLUES modes. The paper should clarify that 'self-contained' means no external tools or lookups, not that no external information is ever placed in context.","section":"Abstract / Appendix A.1.1"},{"comment":"Because the Harm metric is cumulative, the raw 500-step versus 800-step comparisons in Table 3 partly reflect the longer trajectory; the paper acknowledges this and recommends normalization, but the table itself would be more informative if a per-step harm rate were also reported.","section":"Section 3.2 / Table 3"}],"recommendation":"major_revision","confidential_remarks":"The primary blocking issue is the checkpoint audit for the Game Progress metric, followed by the single-run evaluation. Both are fixable without changing the benchmark design. I would be inclined to accept after the requested validation and repeated runs are supplied. The completion counts are robust enough that I do not view the central qualitative finding as in doubt, but the headline percentages should not be published without the checkpoint validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TextQuests is worth your attention. It's a cleanly packaged benchmark for long-horizon, tool-free exploratory agent evaluation, and the qualitative story is plausible: frontier models struggle to make sustained progress in Infocom games even when given the official hint booklets. The new pieces—clue mode, autosave/restore, and a checkpoint-based progress metric—are real additions over the prior Hendrycks et al. (2021c) game suite and Jericho baselines. The 22-model comparison is useful, and the completion counts are stark: zero of 25 games solved without clues, at most 5 of 25 with clues. That is an objective result that doesn't depend on how progress is measured.\n\nThe soft spots are real but fixable. The Game Progress metric is the load-bearing measure for the headline percentages (37.8% for GPT-5 no clues, 70% with clues), and the paper gives no annotation protocol, no inter-rater reliability, and no independent validation of the checkpoint sets. If checkpoint density varies across the 25 games, the averaged percentages reflect labeler choices as much as model ability. The stress-test note is right on this. Also, each model was run once per mode, so there are no error bars; that's a minor issue for a benchmark paper but worth fixing. And there's no measured human or heuristic baseline, which makes \"minimal progress\" hard to calibrate.\n\nI don't think these flaws sink the paper. The qualitative failure modes—hallucinated item locations, failure to reverse a path, loops in navigation—are concrete and consistent with known long-context reasoning problems. The checkpoints may well be sensible; the paper just needs to show the annotations and confirm they're consistent. The token analysis and the saturation check at 800 steps are good hygiene.\n\nVerdict: send it to peer review with a request for checkpoint transparency, repeated runs, and ideally a human or random-baseline comparison. The benchmark fills a real gap, and the core finding is believable even before those fixes.\n\nRecommendation: engage with it. Cite it if you work on agent evaluation. Bring it to reading group—there's a good discussion to be had about what \"progress\" means in these environments.","headline":"TextQuests is a useful benchmark for long-horizon agent evaluation, but the headline progress numbers rest on unvalidated checkpoint labels and single runs.","tokens_in":18798,"tokens_out":2334,"would_cite":true,"duration_ms":21779,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TextQuests shows that current LLMs, with no tools and only their own transcript, complete none of 25 Infocom games and reach at most 37.8% progress; adding the official hint booklets raises the best model to 70% but most games remain…","keywords":["TextQuests","interactive fiction games","LLM agents","long-context reasoning","exploratory environments","Game Progress metric","autosave","InvisiClues"],"falsifier":"Re-label the 25 games' checkpoints with independent annotators and re-run each model several times per condition: if Game Progress scores move by more than the gaps between the listed models, or if reruns of the same model vary by tens of percent, the central comparison is measurement noise rather than capability.","tokens_in":17778,"feed_emoji":"🎮","tokens_out":7510,"duration_ms":73650,"temperature":0.7,"pith_summary":"The paper argues that existing agent benchmarks over-rely on tools and scaffolding, so they miss a fundamental capability: sustained, self-directed reasoning in an exploratory environment. To test that capability directly, TextQuests drops an LLM into 25 classic Infocom text adventures with no external tools, no summarization, no notepads, and asks it to solve the game from its own growing transcript. The empirical claim is stark: across every evaluated frontier model, not one game was completed without clues, and the best average progress (GPT-5) was 37.8 percent. Providing the official InvisiClues hint booklets raises the best model to 70 percent average progress and five completions, but most games remain unfinished. If this benchmark is right, frontier models are still far short of the kind of long-context, trial-and-error problem solving that these games demand from humans.","feed_headline":"No frontier LLM completes a single 1980s text adventure unaided","feed_subtitle":"Across 25 Infocom games the best model, GPT-5, reaches 37.8% progress without clues and 70% with the hint booklets.","key_machinery":"The load-bearing object is the Game Progress metric: for each game, a set of labeled checkpoints marking essential puzzles and milestones, with an agent's score defined as the maximum percentage of checkpoints it ever completes in a trajectory; the benchmark average is the mean over 25 games. The evaluation environment assembles this metric with a full-history interaction protocol (no truncation, no external tools), an autosave/restore mechanism that lets the agent backtrack to any earlier state, and an optional WITH CLUES condition that supplies the official InvisiClues booklets as tiered hints. The checkpoint metric does the argument's work: unlike built-in game scores, which reward exploration and can be decoupled from completion, the checkpoints are intended to track progress along the path to finishing each game. The trajectory data, cut off at 500 steps because longer runs yield little additional gain, is what supports the claim that capable models sustain progress longer but still plateau far below completion.","core_discovery":"In its own terms, the paper's core discovery is that LLM agents make minimal progress on TextQuests without external assistance, despite frontier-level performance on static knowledge benchmarks. The benchmark consists of 25 Infocom interactive fiction games run through the original game runtime, with each model given the full ever-growing history of observations, its own reasoning, and its actions; at each turn the model must output one command. Game Progress is the maximum percentage of hand-labeled checkpoints marking essential puzzles and milestones that the agent reaches during a 500-step run. Across all models tested in NO CLUES mode, the completion count is zero out of 25; with the complete official clue booklets in context, at most five games are completed, and progress on the rest remains partial. The authors interpret this as evidence that current LLMs lack the intrinsic long-context reasoning and sustained planning needed for exploratory, stateful problem-solving, and that their trajectories show characteristic failures such as hallucinating past actions and looping on repeated behaviors.","pith_inferences":["An implication left implicit is that the single unvalidated checkpoint set per game carries the whole ranking; independent re-labeling and repeated runs would show how much of the order is metric noise rather than capability.","The failure to reverse an ascent path in Wishbringer suggests the bottleneck may be constructing and updating a mental map from a transcript, which would predict correlations with other spatial or causal backtracking tasks that the paper does not test.","Because autosave makes all exploration reversible, TextQuests measures problem-solving without permanent consequences; introducing irreversible costs, limited resources, or time pressure could change both the absolute scores and the model ordering.","Since the puzzles come from 1980s games with parser conventions and cultural references, part of the gap could reflect unfamiliarity with the genre rather than pure reasoning; adding newly authored interactive fiction would help separate those factors."],"forward_implications":["If TextQuests measures what it claims, then frontier LLMs still lack a core component of autonomous agency: the ability to make steady, self-directed progress in a novel stateful environment using only internal reasoning.","Because the environment forbids tools, gains on TextQuests are attributable to the base model itself; this gives a cleaner signal of intrinsic long-context reasoning than tool-augmented agent benchmarks.","The 500-step saturation point means the benchmark can differentiate models without unbounded runs, but it also suggests that raw context length is not the bottleneck; the bottleneck is using the history to plan.","The gap between WITH CLUES and NO CLUES shows that these models can partially use structured external hints, yet still fail to integrate them into a complete solution.","Scaling appears to matter: the larger standard models outperform their mini variants by large margins on exploratory tasks, consistent with test-time-compute effects that plateau after a budget."],"supporting_citations":[{"why":"Supplies the game files, annotations, and harm-scoring framework that TextQuests extends with clues, autosave, and the new progress metric.","marker":"Hendrycks et al. (2021c)"},{"why":"Provides the interpreter used to run the original game files and the built-in scoring baseline that the checkpoint metric is designed to replace.","marker":"Hausknecht et al., 2020"},{"why":"Source for the claim that these games take human players over 30 hours and require hundreds of precise actions.","marker":"Smetale, 1983"},{"why":"Documents long-context repetition in game-playing agents, cited as a comparable failure mode and as motivation for removing scaffolding.","marker":"Gemini Team, 2025"},{"why":"Reports the test-time-compute scaling trend that the paper re-observes in its dynamic-thinking analysis.","marker":"OpenAI, 2024"},{"why":"Represents the prior text-game agent evaluation that relied on built-in scoring, the baseline TextQuests argues is a weak proxy.","marker":"Yao et al., 2020"}],"fun_headline_variants":["TextQuests: no LLM finishes a single Infocom game unaided","Zero completions for frontier LLMs on 25 Infocom text adventures","LLM agents score 37.8% max on 1980s text games without clues","Long-context reasoning fails in exploratory text games","Stateful puzzles without tools stump GPT-5 and others in TextQuests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that the hand-labeled checkpoints, and the single 500-step run used for each model, give an unbiased picture of how much progress an agent actually makes on each game.","fun_headline_variants_meta":{"raw":{"variants":["TextQuests: no LLM finishes a single Infocom game unaided","Zero completions for frontier LLMs on 25 Infocom text adventures","LLM agents score 37.8% max on 1980s text games without clues","Long-context reasoning fails in exploratory text games","Stateful puzzles without tools stump GPT-5 and others in TextQuests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000603,"raw_usage":{"total_tokens":2814,"prompt_tokens":943,"completion_tokens":1871,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":1783}},"tokens_in":559,"tokens_out":1871,"duration_ms":13017,"temperature":1.0,"reasoning_tokens":1783,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:27:44.108603+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-label the 25 games' checkpoints with independent annotators and re-run each model several times per condition: if Game Progress scores move by more than the gaps between the listed models, or if reruns of the same model vary by tens of percent, the central comparison is measurement noise rather than capability.","supporting_citations":[],"review_version":1}