{"id":"b1f6625b-3c54-4bf5-8a22-88f82762abe6","arxiv_id":"2505.12439","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A framework that combines dynamic knowledge-graph mapping, action-space learning, and experience reflection improves LLM agents' scores on interactive fiction games, yet the reported gains are inflated by an unfair baseline comparison and possible training-set leakage.","lead":"This paper proposes a modular system, LPLH, that helps large language models play text-based adventure games by building maps, learning valid commands, and reflecting on past successes and failures. The authors report score improvements over plain LLM agents on several classic games, but the evaluation contains methodological flaws that weaken the central claim.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's raw-score comparison is asymmetric: LPLH averages its last three epochs while baselines average all ten, and Dragon appears in both fm training and evaluation; the headline gains may not survive a matched evaluation.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the asymmetric scoring window for raw scores in Table 1, plus the potential Dragon train/test overlap. Both are visible in the manuscript itself (Section 4.3 and Table 1 note; Appendix A vs Table 1), so they are not artifacts of the review pipeline. The framework is plausible and the Zork1 ablation in Table 2 is more carefully controlled, which is why the issue is not the idea but the evidence. Still, the headline cross-game comparisons are the primary support for the central claim that LPLH enhances LLM performance, and they are currently not interpretable. Because the reader already recommended REJECT and this stress-test finds the same flaw, the verdict is unchanged. Secondary issues, such as the novelty overclaim about being the first LLM system for IF games (contradicted by Tsai et al. 2023, which the paper itself cites) and the 'training-free' label despite the fine-tuned fm, do not drive the verdict but should be corrected.","tokens_in":19692,"tokens_out":9263,"duration_ms":89184,"concrete_test":"Recompute Table 1's raw scores with the same scoring window for both sides: for each LLM baseline, compute the average over epochs 8-10 (matching LPLH's reported 'learning outcomes') and also compute LPLH's average over all ten epochs. If the LPLH-vs-baseline raw gap narrows or reverses under either matched comparison, the headline improvements are an artifact of the asymmetric window.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim in Section 5.2 is that LPLH 'markedly improves the performance of LLMs-based agents.' The evidence is Table 1, but the raw scores are computed under different policies: the table note states that 'the raw on base computes the average score in all runs, while the raw on LPLH computes the last three runs as learning outcomes,' and Section 4.3 confirms this design. Averaging LPLH over its final three epochs, after seven epochs of accumulated maps, action spaces, and experiences, while averaging the baseline over all ten epochs, confounds the framework's effect with the choice of scoring window. The baseline has no cross-epoch memory and may be roughly stationary, but LPLH's reported number is explicitly a late-phase estimate. The correct comparison for post-learning performance is baseline's final-epoch or last-three-epoch score against LPLH's last-three-epoch score, or both averaged over all ten epochs. Without this matched comparison, the 6.8x Detective improvement and similar gains are not interpretable. A second, independent inconsistency: Appendix A says the fine-tuned fm is trained on 'Dragon, Karn, and Night' and calls them 'not in our test game,' yet Table 1 reports results for Dragon. If Dragon was used to train fm, those Dragon numbers are contaminated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes LPLH, a modular framework for adapting LLM agents to interactive fiction games. It combines (1) a dynamically updated knowledge-graph map, (2) an action-space learner that stores validated verb–object pairs, and (3) an experience library that summarizes scoring events and retrieves them via RAG; the main agent acts in a zero-shot manner using these structures. The authors evaluate on nine Jericho games with four LLM backbones against RL and LLM baselines, report large raw-score improvements (e.g., Qwen-7B on Detective 68 vs 10), an ablation on Zork1, and case studies of learning from failure and success. They conclude that LPLH delivers human-like, interpretable learning and is the first system to leverage LLMs for IF game play.","tokens_in":19894,"tokens_out":8293,"duration_ms":82897,"significance":"The framework is clearly described, modular, and easy to reproduce from the appendix; the authors provide prompt templates, hyperparameters, and an ablation study with standard deviations, which is a strength. If the headline comparisons were valid, LPLH would be a useful step toward memory-augmented LLM agents in text-based games, and the cognitive-science framing, while qualitative, is a reasonable design inspiration. However, the current empirical support is compromised by an asymmetric scoring protocol and a train/evaluation overlap for the auxiliary model, so the claimed improvements are not yet interpretable as framework gains.","major_comments":[{"comment":"The raw score for the base LLM averages all ten epochs, while the raw score for LPLH averages only the final three epochs ('learning outcomes'). Since LPLH accumulates KG-map, action-space, and experience-library content across epochs, its late-epoch scores reflect a learned policy; the base agent has no cross-epoch memory, so its all-epoch average is not the appropriate baseline for post-learning performance. The Section 5.2 headline examples (e.g., Qwen-7B on Detective: 68/100 vs 10/100, called a 6.8x improvement) therefore confound the framework's effect with the scoring-window choice. Please provide matched comparisons: for both conditions, report the final-three-epoch average, the final-epoch score, or the all-ten-epoch average, together with per-epoch curves or deviations.","section":"Section 4.3, Table 1"},{"comment":"Appendix A states that the fine-tuned fm is trained on 'Dragon,' 'Karn,' and 'Night' and describes these as 'not in our test game,' yet Table 1 includes Dragon among the evaluation games. Because fm is used inside LPLH for action validation, relation extraction, and verb/object decomposition, the LPLH results on Dragon are in-domain for that auxiliary model and are not comparable to the base results without fm. Remove Dragon from the evaluation or retrain fm without Dragon and rerun the affected table entries.","section":"Appendix A, Table 1"},{"comment":"The paper interprets score, vocabulary, and room-exploration curves as evidence that LPLH 'exhibits... learning dynamics that parallel human cognitive strategies' (Section 5.1). The only human reference is a single expert score line; no human vocabulary-growth, room-exploration, or score trajectories are collected. The qualitative alignment with the cited cognitive-science literature is plausible but untested. Either collect human play data or revise the abstract and conclusion claims from 'human-like' to 'cognitively inspired' or similar.","section":"Section 5.1, Figure 3"},{"comment":"The framework is called 'training-free' and said to 'eliminate the need for fine-tuning' (Sections 1 and 3.5), but Section 4.3 fine-tunes Qwen2.5-1.5B with LoRA for three auxiliary tasks. This is an internal inconsistency in a central framing claim; please clarify that the main action-generation LLM is zero-shot while the auxiliary fm is fine-tuned.","section":"Sections 1, 3.5, 4.3"}],"minor_comments":[{"comment":"The acronym 'LHLP' should be 'LPLH'.","section":"Section 4.3"},{"comment":"The heading 'LPHP' should be 'LPLH'.","section":"Section 3.3"},{"comment":"The heading 'Zore-shot' should be 'Zero-shot'.","section":"Section 3.5"},{"comment":"The claim that LPLH is the first LLM-based system for IF games is contradicted by the paper's own review of Tsai et al. (2023), who evaluated ChatGPT and GPT-4 on Zork1; please temper the novelty statement.","section":"Introduction and Conclusion"},{"comment":"Reporting standard deviations or confidence intervals, as in Table 2, would help assess whether the observed differences are meaningful.","section":"Table 1"},{"comment":"The phrase 'In both games' follows a list of three games and should read 'In these games'.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper's modular design is worth pursuing, but the current evaluation must be redone before the claims can be accepted. I would ask the authors to rerun with matched scoring windows, remove Dragon from evaluation or retrain fm without it, and either add human data or soften the human-like claim. If the improvements persist under a matched protocol, the paper could be a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline results don't hold as reported. Table 1 averages LPLH raw scores over the final three epochs but baselines over all ten; the table note says so explicitly. That is not a subtle artifact, it is a structurally unfair comparison, and the 6.8x Detective gain is uninterpretable until both sides are scored on the same window. The second load-bearing issue is the Dragon leak: Appendix A says the fine-tuned fm was trained on 'Dragon, Karn, and Night' and calls them 'not in our test game,' yet Dragon appears in Table 1. That is a direct train/test contamination, even if fm is a small auxiliary model. Third, the novelty claim is simply wrong: the paper says it is the first system to leverage LLMs for IF games while citing Tsai et al. 2023, which played Zork1 with ChatGPT/GPT-4. The authors even describe that as the only prior LLM work a few lines later. These are not minor blemishes; they undercut the central empirical and novelty claims.\n\nWhat the paper does well: the three-module design (dynamic KG map, action-space learning, experience memory) is coherent, and the Zork1 ablation in Table 2 is more careful. It compares component combinations against each other and against a plain baseline, and the pattern—each component helps, the full system helps most—is plausible. The error analysis of puzzle bottlenecks is honest and useful. The learning-curve interpretation is suggestive but overreaches: calling the curves 'human-like' without any human data is interpretive, not evidence.\n\nOther soft spots: 'training-free' is misleading because fm is fine-tuned on Qwen2.5-1.5B. The paper also has typos that suggest haste, including 'LHLP' for 'LPLH' in Section 4.3.\n\nThe framework is worth a serious look, but the paper as submitted does not support its claims. The fix is clear: report matched raw scores (same epoch window for both), rerun Dragon without the contaminated fm or remove Dragon from evaluation, and correct the novelty claim. If those are addressed, the ablation and framework could make a useful contribution to LLM-agent research for text-based games.\n\nThe right move is to send it to peer review with a clear request for a matched evaluation and a reworked novelty statement. A serious referee can push on those points; desk rejection would lose a plausible engineering idea over fixable flaws.","headline":"The framework is plausible and the Zork1 ablation is careful, but the headline results rest on an unfair scoring window and a Dragon train/test leak, and the 'first LLM system' claim is contradicted by the authors' own citation.","tokens_in":20458,"tokens_out":2933,"would_cite":false,"duration_ms":30054,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Building a live map, learning valid commands, and reflecting on scored events lets an LLM play interactive fiction far better than plain prompting, the paper reports.","keywords":["interactive fiction","large language models","text-based games","knowledge graph map","action-space learning","experience reflection","zero-shot decision-making"],"falsifier":"Re-run the base LLM under the same scoring protocol used for LPLH—averaging only the final three of ten epochs—and compare; if the base model's last-three average approaches LPLH's raw score, the reported gains are an artifact of how scores were averaged. Separately, retrain the three fine-tuned helper modules on games that exclude Dragon, re-run Dragon, and check whether the score drops, which would indicate memorization rather than generalizable map, action, and experience learning.","tokens_in":19435,"feed_emoji":"🎮","tokens_out":11799,"duration_ms":101862,"temperature":0.7,"pith_summary":"This paper asks whether LLMs can learn to play interactive fiction the way human players do, and argues that the answer is yes: it introduces LPLH, a framework that adds three memory structures—a dynamic knowledge-graph map, a growing action space of valid verb-object commands, and an experience library of summarized successes and failures—to a zero-shot LLM's next-command generation. On nine text-adventure games, LPLH raises raw and maximum scores across four LLM backbones, with the largest reported jump going from 10/100 to 68/100 for a 7B model on Detective. Ablations on Zork1 show each module helps and the full combination works best, which the authors take as evidence that human-like comprehension, rather than score-centric reinforcement learning, is the productive route for text-world agents. If the framework is right, interactive fiction becomes a learning problem for LLMs, not just a benchmark for RL.","feed_headline":"LLM agent improves text-game scores by building maps and memories","feed_subtitle":"LPLH gives LLMs a dynamic map, learned commands, and an experience library to play interactive fiction like humans.","key_machinery":"The load-bearing mechanism is the LPLH decision loop, which couples three memory structures: a dynamic knowledge-graph map updated after every observation and action; an action space grown from validated verb-object pairs; and an experience library written by an LLM summarizer whenever the score changes. At each step, the current map, viable action-object pairings for the current location, and retrieved experiences are composed into a prompt, and a zero-shot LLM emits the next command. This turns gameplay into structured learning with external memory rather than policy optimization.","core_discovery":"The paper's central claim is that an LLM playing an interactive fiction game becomes substantially better when three human-like behaviors are simulated: maintaining an up-to-date map of locations and objects, remembering which verb-object commands actually work, and reflecting on scoring events to extract reusable lessons. These structures are fed to a zero-shot LLM that produces the next command, with retrieved experiences supplied through retrieval-augmented generation. Empirically, LPLH improves raw and maximum scores over every base LLM tested across the nine games; for example, a 7B open-weight model on Detective rises from 10 to 68 raw points, and a 14B variant on Spellbrkr reaches 41.7/60, beating the DBERT-DRRN RL baseline. Ablations on Zork1 (LPLH14B 39.7 raw, base 9.0) indicate the gains come from combining the map, action space, and experience modules, with a chain-of-thought variant adding a further small improvement. The authors conclude that modeling human-like comprehension and decision-making enhances LLM performance and yields more interpretable gameplay.","pith_inferences":["Editorial inference: the map-action-experience recipe should transfer to other partially observable language tasks, such as interactive instruction following or web navigation, where humans likewise build a mental model, learn a command vocabulary, and reuse lessons from success and failure.","Editorial inference: the contribution of the experience module likely depends on retrieval quality; changing the number of retrieved experiences or adding recency weighting could alter raw scores more than further prompt engineering of the summarizer.","Editorial inference: because the fine-tuned helper modules were trained on Dragon, Karn, and Night, and Dragon is itself one of the evaluation games, a held-out evaluation is needed to separate generalizable module learning from memorization of specific games."],"forward_implications":["Open-weight LLMs using LPLH can match or exceed the maximum scores of RL baselines on some games without reward shaping, as seen on Omniquest and Balances.","The module ablations imply that the value of the framework is additive: the full Zork1 configuration (39.7 raw) outperforms KG-map only (11.0), experience only (25.6), and action-space only (26.6).","Because the action LLM is prompted zero-shot, the framework is model-agnostic: changing only the backbone changes scores continuously, as shown across open-weight and proprietary models.","LPLH remains bounded by the same puzzle bottlenecks as RL agents, so its gains are specific to exploration, memory, and command selection rather than to solving obscure domain-specific puzzles."],"supporting_citations":[{"why":"Supplies the benchmark environment and the nine-game test set on which all scores are reported.","marker":"(Guo et al., 2020)"},{"why":"Defines the DRNN reinforcement-learning baseline that LPLH is compared against.","marker":"(He et al., 2016)"},{"why":"Supplies the KG-A2C baseline and the graph-constrained action-space idea that the dynamic KG-map extends.","marker":"(Ammanabrolu and Hausknecht, 2020)"},{"why":"Supplies the DBERT-DRRNL baseline, a transformer-enhanced RL agent LPLH is compared with.","marker":"(Singh et al., 2022)"},{"why":"Established the limitation of plain LLMs in constructing a coherent world model, which LPLH's map and memory modules target.","marker":"(Tsai et al., 2023)"},{"why":"Supplies retrieval-augmented generation, the mechanism LPLH uses to retrieve relevant experiences.","marker":"(Lewis et al., 2020)"},{"why":"Supplies the zero-shot reasoning prompting method used for LPLH's next-command generation.","marker":"(Kojima et al., 2022)"},{"why":"Provides a strategic-exploration baseline and documents the puzzle-bottleneck problems that LPLH still encounters.","marker":"(Tuyls et al., 2022)"}],"fun_headline_variants":["LLM maps, acts, reflects, then excels at text games","LPLH: Make LLMs play interactive fiction like humans do","Map, remember, act: LLM gains human-like edge in text games","Human-like play from maps and memories boosts LLM scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on a fair comparison: the baseline's raw score is averaged over all ten epochs, while LPLH's raw score is averaged over only the final three, and the fine-tuned helper modules were trained on games that include Dragon, which also appears among the evaluation games.","fun_headline_variants_meta":{"raw":{"variants":["LLM maps, acts, reflects, then excels at text games","LPLH: Make LLMs play interactive fiction like humans do","Map, remember, act: LLM gains human-like edge in text games","Human-like play from maps and memories boosts LLM scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001293,"raw_usage":{"total_tokens":5304,"prompt_tokens":999,"completion_tokens":4305,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":4230}},"tokens_in":615,"tokens_out":4305,"duration_ms":30495,"temperature":1.0,"reasoning_tokens":4230,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:33:11.321806+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the base LLM under the same scoring protocol used for LPLH—averaging only the final three of ten epochs—and compare; if the base model's last-three average approaches LPLH's raw score, the reported gains are an artifact of how scores were averaged. Separately, retrain the three fine-tuned helper modules on games that exclude Dragon, re-run Dragon, and check whether the score drops, which would indicate memorization rather than generalizable map, action, and experience learning.","supporting_citations":[],"review_version":1}