{"id":"d88f8d15-aae9-4ae3-857b-1d6687e7e71b","arxiv_id":"2505.15146","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"lmgame-Bench turns six classic games into a scaffolded LLM evaluation suite, ranks 13 models, detects contamination, and reports RL transfer from Sokoban or Tetris to unseen games and planning tasks.","lead":"This paper builds lmgame-Bench, a suite that wraps six classic video games in a standard interface and adds perception and memory helpers so LLM agents can be measured while playing. It reports that the benchmark separates 13 leading models, links game skills to standard AI benchmarks, and shows that training on one game improves other games and planning tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The harness is not a neutral scaffold: o3-generated memory reflections contain explicit next-move suggestions, so leaderboard scores may reflect o3's policy rather than the target model's own ability.","rationale":"The reader's weakest assumption (o3 scaffold neutrality) is precisely the load-bearing point, and the paper's own Appendix F makes the concern concrete: o3 reflections contain explicit action recommendations, not just state descriptions. This is a genuine confound for the leaderboard and for the 'reliable evaluation' claim, so the CONDITIONAL verdict is appropriate. I partial-agree with the reader because my concern focuses specifically on the memory/perception scaffold leakage, whereas the reader also emphasizes the o3-as-judge role; both are real, but the scaffold leakage is the more load-bearing mechanism. I do not recommend rejection because the underlying resource, the statistical harness comparisons, and the transfer study are useful and the leakage can be tested and, if confirmed, mitigated with a neutral scaffold. The proposed check would settle the matter directly. Since the reader already reached CONDITIONAL and my analysis reinforces that rather than shifting it, the verdict should remain UNCHANGED.","tokens_in":34118,"tokens_out":2843,"duration_ms":27909,"concrete_test":"Rerun the harnessed leaderboard on Super Mario Bros, Ace Attorney, and 2048 with o3 replaced by a neutral deterministic perception parser and a non-reasoning reflection generator (e.g., GPT-4o with instructions to only restate board state, never to suggest moves). If the Spearman rank correlation between the original and replaced-scaffold leaderboards drops below roughly 0.9, or if the o3-favored models lose their top rankings, the o3 scaffold is not neutral and the ability attribution fails. A complementary check: strip all sentences containing future-move suggestions ('Next:', 'Prefer', 'avoid Up') from the o3 reflections and measure the per-model score drop; if scores for weaker models fall substantially, the leakage channel is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that lmgame-Bench yields reliable model evaluations requires the perception and memory scaffolds to be neutral mediators. This condition fails on the paper's own evidence. Section 2.2.1 states that o3 is used to generate reflections in the memory module, and Appendix F's memory-module prompt explicitly asks for 'strategic insights' and 'actionable' advice. The qualitative examples show o3 reflections that are not neutral state summaries but policy guidance: e.g., 'Next: keep sliding Right/Down to compress toward the bottom-right; avoid Up unless desperate' and 'Prefer more up/left sequences to clear column 2'. Similarly, the perception module uses o3 to convert Super Mario Bros frames and Ace Attorney dialogue into text, so o3's understanding of the game enters the observation stream for every evaluated model. When o3 supplies such solution-relevant hints, a weak model's harnessed score measures how well it can follow o3's coaching rather than its own game-playing ability. The leaderboard separation between reasoning and non-reasoning models, and the claim that 86.7% of runs beat random play, are therefore confounded by a leakage channel that the paper does not control for. The o3-as-judge step in the Ace Attorney contamination mitigation (Appendix B.2) compounds this: post-mitigation rankings are validated by the same model that generates the scaffold, so the two uses cannot independently confirm each other. This is not an internal inconsistency, but it is a correctness risk to the benchmark's reliability claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces lmgame-Bench, a benchmark that wraps six off-the-shelf video games in a Gym-style API with optional perception, memory, and reasoning scaffolds. Across 13 models, the authors report that the harness raises most runs above random play and separates reasoning models from non-reasoning models. They also analyze correlations between game performance and 20 established benchmarks via Spearman correlation, low-rank factorization, and linear modeling, and report that RL fine-tuning on simplified Sokoban/Tetris transfers to planning and agentic tasks. Contamination tests for Super Mario Bros. and Ace Attorney, a DSPy-based prompt standardization procedure, and detailed appendices support the main text.","tokens_in":34404,"tokens_out":8201,"duration_ms":69522,"significance":"If the harness were a neutral mediator, lmgame-Bench would be a valuable contribution: video games are a scalable source of long-horizon, partially observable interactive tasks, and the RL transfer results to Blocksworld and WebShop are interesting. The paper's strengths include a released codebase, a clean MDP formalism, toggleable harness modules, explicit contamination checks, and a quantitative attempt to connect game performance to capabilities measured by established benchmarks. However, the core reliability claim is compromised by the finding that the memory reflection module is generated by o3 and contains explicit action-level policy advice. Because o3 is itself the top-ranked model, harnessed scores may partly reflect how well target models follow o3's coaching rather than their own game-playing ability. The single-run results reported with zero variance and the selective exclusion of Super Mario Bros. from capability analyses further weaken the quantitative claims. The benchmark artifact is plausible, but the measurement validity needs to be established with additional ablations and reporting changes.","major_comments":[{"comment":"The memory module is not a neutral scaffold. Section 3.1 states that \"we employ o3 to generate reflections\" and the Appendix F prompt asks for \"strategic insights\" and \"actionable\" advice; the example reflections contain explicit policy directives such as \"Next: keep sliding Right/Down to compress toward the bottom-right; avoid Up unless desperate\" and \"Prefer more up/left sequences to clear column 2.\" For every evaluated model that uses the harness's memory module, these o3-generated reflections enter the context before the model chooses an action. The paper provides no ablation showing that rankings are stable when this o3 policy content is removed or replaced by neutral state summaries. Since o3 is itself the top-ranked model, the harnessed leaderboard scores may measure how well weaker models follow o3's coaching rather than their own game-playing ability. The same concern applies to the o3-based perception module used for Super Mario Bros. and Ace Attorney. This is a load-bearing confound for the central claim that lmgame-Bench gives reliable model evaluations.","section":"§2.2.1, Appendix F"},{"comment":"Several reported zero standard deviations come from single runs. Table 1 marks o1-2024-12-17 and o3-2025-04-16 as single-run models, yet reports values such as 3445.0±0.0, and the Ace Attorney column is marked with a star throughout, implying a single run for every model in that game while still printing ±0.0. A single observation has no sample variance; reporting ±0.0 is misleading and makes the \"harness reduces variance\" and \"models separate well\" claims appear stronger than the data support. Since Ace Attorney and the o3 row are used directly in the leaderboard discussion (o3 and o1 both score 16), the affected results need to be either re-run or reported without error bars and with appropriate caveats.","section":"Table 1"},{"comment":"The capability attribution claims are not supported for two of the six games. Super Mario Bros. is excluded from the main correlation and factorization analyses after observing its high variance, and the supplementary inclusion (Appendix D.2) shows weak, diffuse loadings. Table 3 shows the linear model cannot predict SMB (r=0.295) or 2048 (r=0.248), yet the text states that \"long-horizon games like Sokoban, Tetris and 2048's rankings are driven primarily by math and coding performance.\" This is contradicted by the 2048 row, whose offset weight (3.130) dominates and whose coding weight is 0. The claim that \"every game probes a unique blend of capabilities\" therefore holds only for the subset of games on which the model fits; the paper should either restrict the claim or present a model that fits 2048.","section":"§3.2, Table 3"},{"comment":"It is unclear whether the main benchmark scores for Ace Attorney are collected with the contamination-mitigation prompts. Appendix B.2 detects a strong pre-mitigation correlation between script similarity and performance, then shows this correlation disappears after name-masking, paraphrasing, and enforced reasoning. But Table 1 presents Ace Attorney scores without stating whether these are raw or mitigated. If the leaderboard uses unmitigated prompts, the contamination claim in the abstract (\"designed to ... remove contamination\") is not satisfied for the reported rankings. If it uses mitigated prompts, the paper should say so explicitly and state which prompts were used for Table 1. The o3-as-judge validation in Figure 10b does not resolve this, because the judge is generated by the same model family that supplies the scaffold.","section":"§2.2.2, Appendix B.2"}],"minor_comments":[{"comment":"The sign of the reported correlations is inconsistent: the text says similarity strongly correlates with leaderboard rank (r=-0.773, p=0.005), while Figure 9a reports r=0.773 for similarity vs. score and r=-0.911 for similarity vs. rank; align the signs and captions with the actual variables.","section":"Appendix B.2, Figure 9"},{"comment":"The claim of being \"the first benchmark that uses video games to evaluate state-of-the-art LLMs with or without scaffolds\" is too strong given the BALROG, GameBench, and SmartPlay systems discussed in Related Work; soften to a comparative claim.","section":"§1"},{"comment":"The text says the first version (Figure 13b) excludes Super Mario Bros. and the second version (Figure 13a) includes it, but the figure captions state the opposite; the labels and text should be reconciled.","section":"Appendix D.3, Figure 13"},{"comment":"There are typos in the caption: \"empirically deisgn prompt pairs\" and \"DSPy optimzied prompt pairs\" should be corrected, and the variables Δe and Δp should be defined in the caption.","section":"Table 9 caption"},{"comment":"The word \"narative\" should be \"narrative,\" and the sentence on long-horizon rewards could clarify why Sokoban is classified as long-horizon rather than progression-based.","section":"§2.1.3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time mainly for the artifact and the transfer results. The six-game Gym-style suite with togglable perception/memory/reasoning modules, the DSPy prompt standardization, and the contamination checks are concrete, and the Sokoban/Tetris RL transfer to Blocksworld and WebShop is a genuinely interesting empirical result on Qwen2.5-7B. The appendices are unusually detailed; the code is public. That is real work.\n\nThe soft spot is not a minor quibble. The memory module uses o3 to write reflections, and Appendix F shows the prompt asks for 'strategic insights' and 'actionable' advice. The qualitative examples are literally next-move suggestions: 'keep sliding Right/Down to compress toward the bottom-right; avoid Up unless desperate' and 'Prefer more up/left sequences to clear column 2.' That is policy guidance, not a neutral state summary. Since harnessed runs are the basis for the leaderboard and the 86.7% claim, the scores partly measure how well a model follows o3's coaching. o3 also serves as the judge that validates the Ace Attorney contamination mitigation, so the mitigation check is not independent. The stress-test note lands on this, and the paper's own text supports it.\n\nOther issues are smaller. o1 and o3 are single runs reported with zero variance, which overstates precision. Super Mario Bros. is dropped from the capability analysis after showing high variance; the paper is transparent about this, but it makes the 'games probe unique capabilities' claim weaker. The contamination mitigation is validated on six models, and the linear model fits a lot of weights on eight models, so those attributions should be read as suggestive. The 'first benchmark' claim is wrong in spirit, and the related-work section itself lists BALROG, GameBench, and SmartPlay.\n\nNone of this kills the artifact. The public code, the clear metrics, and the transfer experiments are worth engaging with seriously. The central reliability claim, though, needs an independent scaffold—preferably reflections from a weaker open model or rule-based summaries—and a head-to-head comparison with existing game benchmarks before the leaderboard should be taken at face value.\n\nSend it to review, yes. I would ask the authors to add those controls and to report variance for single-run models. The paper is a conditional accept, not a desk reject.","headline":"A useful game-harness benchmark and an interesting RL transfer study, but the o3-generated memory reflections leak policy advice into the harness, so the reliable-ranking claim needs an independent-scaffold check.","tokens_in":34954,"tokens_out":2316,"would_cite":true,"duration_ms":21598,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"lmgame-Bench claims that off-the-shelf video games, wrapped in perception and memory scaffolds, can reliably rank LLMs, and that game-based reinforcement learning transfers to planning tasks.","keywords":["LLM evaluation","video game benchmarks","reinforcement learning transfer","planning","data contamination","prompt optimization","agentic AI","benchmark scaffolding"],"falsifier":"Run the harness on the same six games with the o3 scaffold replaced by an open-weight descriptor that never sees solutions, and compare leaderboards. If model rankings shift substantially, or if the o3 reflections for failed runs already contain the correct next move, the central claim that lmgame-Bench measures target-model gaming ability is falsified. A second check: after Ace Attorney mitigation, verify whether the o3 judge's quality scores still predict rankings when the judge is given only entity-masked testimony and no access to the original script.","tokens_in":33895,"feed_emoji":"🎮","tokens_out":5748,"duration_ms":49220,"temperature":0.7,"pith_summary":"lmgame-Bench asks whether off-the-shelf video games can be turned into a dependable way to evaluate large language models, and answers yes, provided the games come with carefully built scaffolding. Directly dropping models into games leaves most runs near random play, so the benchmark adds perception modules that convert screens into text and memory modules that write reflections, plus contamination checks and prompt standardization. With those supports, 86.7% of model runs beat a random baseline and the 13-model leaderboard separates reasoning from non-reasoning models. The paper also reports that reinforcement learning on a single game transfers to unseen games and to planning and agentic tasks such as Blocksworld and WebShop. A reader would care because games stress perception, memory, and planning together, and the results suggest games can serve both as a testing ground and as a training ground for those abilities.","feed_headline":"Benchmark turns video games into a reliable LLM test","feed_subtitle":"Harness lifts 86.7% of runs past random; Sokoban RL training boosts Blocksworld and WebShop scores.","key_machinery":"The load-bearing mechanism is the gaming harness, a set of modular scaffolds wrapped around a Gym-style API. Perception modules read game backends and convert grids or UI into textual state descriptions, with o3 used for complex frames; memory modules store recent state-action-reward tuples and generate reflections that compress lessons; reasoning modules allow long chain-of-thought. Prompt standardization is carried out by a two-stage procedure that starts from empirical agent prompts and then uses a bootstrapping optimizer to refine prompts against game rewards across all target models. Contamination checks use frame-reordering for Super Mario Bros. and Sentence-BERT similarity against public transcripts for Ace Attorney. These components together lift scores above random, reduce variance, and make model rankings interpretable.","core_discovery":"The paper claims that classic games—Super Mario Bros., Tetris, Sokoban, Candy Crush, 2048, and Ace Attorney—can be made into an effective LLM benchmark if the raw game is wrapped in a harness that fixes three failure modes: brittle vision, prompt sensitivity, and contamination. The harness converts game states into symbolic text or o3-generated descriptions, keeps a short memory of past states and reflections, and lets reasoning models emit long chain-of-thought traces. Contamination is measured directly: Ace Attorney performance initially tracks similarity to public transcripts, and the correlation disappears after entity masking, paraphrasing, and enforced reasoning. After these fixes, harnessed runs separate models far from random play, and rank correlation plus low-rank decomposition show each game draws on a distinct blend of math, code, language, physics, and puzzle abilities. The paper's strongest empirical claim is that fine-tuning a 7B model with reinforcement learning on simplified Sokoban or Tetris improves performance on larger boards, on the other game, on Blocksworld, and on WebShop, while leaving math and coding largely unchanged.","pith_inferences":["If scaffold neutrality holds, the leaderboard measures planning over textual state representations, not raw visual perception; the benchmark is thus a proxy for agentic reasoning with a text interface rather than for human-like game vision.","The transfer results suggest games could serve as a cheap, verifiable-reward training source for planning skills; a natural extension is to test whether Sokoban-style training transfers to other long-horizon agent benchmarks beyond Blocksworld and WebShop.","Because the scaffolds are partly generated by o3, the benchmark is coupled to one model's perception; an open-weight scaffold or a purely backend-derived state representation would make the benchmark self-contained and reusable as models evolve.","The contamination findings imply other narrative or visual benchmarks built from public assets should run the same similarity-against-performance check before trusting leaderboards."],"forward_implications":["With the harness, 86.7% of model runs beat random play, so the benchmark can separate models that raw games cannot.","Rankings on lmgame-Bench separate reasoning models from non-reasoning ones, giving a scalable signal of agentic competence.","RL on simplified Sokoban lifts Blocksworld 2D from 9.0 to 29.5 and WebShop from 7.0 to 19.1, suggesting game-based training transfers to planning and agentic tasks.","Ace Attorney scores no longer track transcript similarity after mitigation, so post-mitigation rankings plausibly reflect reasoning rather than memorization.","Prompt standardization cuts variance between candidate prompts by 33.8% to 63.5% on 2048, making comparisons across models more stable."],"supporting_citations":[{"why":"Supplies the multi-turn RL training loop used for the Sokoban and Tetris transfer experiments.","marker":"[3]"},{"why":"Provides the optimizer and compilation procedure used to standardize game prompts and cut variance.","marker":"[36]"},{"why":"Motivates the structured prompt interventions used to suppress memorized transcripts in Ace Attorney.","marker":"[31]"},{"why":"Supplies the memorization-testing technique behind the text-level contamination check.","marker":"[32]"},{"why":"Provides the Sokoban environment used for training and same-game evaluation.","marker":"[62]"},{"why":"Provides the Tetris environment used for training and cross-game evaluation.","marker":"[63]"},{"why":"Is the base model that is fine-tuned in the transfer study.","marker":"[64]"},{"why":"Supplies the Blocksworld planning tasks used to measure cross-task transfer.","marker":"[84]"},{"why":"Supplies the WebShop agentic task used to measure cross-task transfer.","marker":"[87]"}],"fun_headline_variants":["Games become reliable LLM benchmarks with harness","RL on one game improves LLM performance on others","Contamination-free game benchmark separates LLMs","Symbolic wrappers turn games into valid LLM evals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's discriminative power rests on the assumption that the o3-generated text scaffolds are neutral: if o3's perception descriptions or memory reflections leak solution information, rankings measure scaffold leakage rather than the target model's ability.","fun_headline_variants_meta":{"raw":{"variants":["Games become reliable LLM benchmarks with harness","RL on one game improves LLM performance on others","Contamination-free game benchmark separates LLMs","Symbolic wrappers turn games into valid LLM evals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000231,"raw_usage":{"total_tokens":1496,"prompt_tokens":964,"completion_tokens":532,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":471}},"tokens_in":580,"tokens_out":532,"duration_ms":5139,"temperature":1.0,"reasoning_tokens":471,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:22:58.305860+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the harness on the same six games with the o3 scaffold replaced by an open-weight descriptor that never sees solutions, and compare leaderboards. If model rankings shift substantially, or if the o3 reflections for failed runs already contain the correct next move, the central claim that lmgame-Bench measures target-model gaming ability is falsified. A second check: after Ace Attorney mitigation, verify whether the o3 judge's quality scores still predict rankings when the judge is given only entity-masked testimony and no access to the original script.","supporting_citations":[{"cited_title":"Advances in Neural Information Processing Systems 35(2022) 20744–20757","cited_arxiv_id":null,"evidence_quote":"Supplies the WebShop agentic task used to measure cross-task transfer."},{"cited_title":"Advances in Neural Information Processing Systems36(2023) 38975–38987","cited_arxiv_id":null,"evidence_quote":"Supplies the Blocksworld planning tasks used to measure cross-task transfer."},{"cited_title":"https://github.com/jaybutera/ tetrisRL(2023) GitHub repository","cited_arxiv_id":null,"evidence_quote":"Provides the Tetris environment used for training and cross-game evaluation."}],"review_version":1}