{"id":"1f137d29-c518-4f4d-b9b8-d4b3a744ac1b","arxiv_id":"2505.14552","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"KORGym introduces a 51-game, text and visual, multi-turn benchmark with a normalized scoring scheme, and uses it to compare 19 LLMs and 8 VLMs on six reasoning dimensions.","lead":"KORGym is a new benchmark platform that puts large language models through more than fifty interactive games, from Sudoku to Plants vs. Zombies, to test reasoning. It ranks 19 text models and 8 vision models, and the authors use the results to argue that game-based evaluation reveals stable reasoning strengths and weaknesses within model families.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing premise is knowledge orthogonality; Appendix B defines it but never verifies it, and Table 1 includes canonical games (Sudoku, Wordle, Hanoi, 2048) whose strategies are abundant in pretraining data, so scores may reflect memorized knowledge rather than rule-following.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern I identify: knowledge orthogonality is asserted but not verified. I agree with that identification and find it more central than the normalization and error-bar issues, because the benchmark's stated purpose is to measure intrinsic reasoning independent of memorization. The strongest independent support for the paper is the platform itself and the breadth of the empirical sweep, but neither addresses leakage from pretraining knowledge. The manuscript contains no per-game leakage test, no comparison against a rule-only solver, and no surface-randomized control. The internal evidence in Section 5.2 actually cuts against the authors: documenting that models use code templates and named algorithms shows that scores depend on K, not just R. A CONDITIONAL verdict remains appropriate: the concern is concrete and testable, but not yet demonstrated to be fatal. The proposed surface-randomization experiment would settle whether classic games in KORGym are knowledge-orthogonal or whether the leaderboard reflects memorized game-specific strategies.","tokens_in":22793,"tokens_out":4174,"duration_ms":39109,"concrete_test":"Release an official KORGym-LeakControl variant for the ten highest-exposure classic games (Sudoku, Tower of Hanoi, Wordle, 2048, Minesweeper, Sokoban, Tetris, Spider Solitaire, Trust Evolution, Plants vs. Zombies) in which every surface cue is systematically remapped while preserving the formal game state graph: digits 1-9 become arbitrary alien glyphs; Wordle's dictionary is replaced by random strings and feedback colors are renamed; Hanoi pegs and disks get novel labels; Minesweeper numbers become colored symbols; 2048 tile values become letters; and rule prompts use fresh vocabulary. Rerun the same 19 LLMs and 8 VLMs under identical seeds and scoring.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that KORGym measures 'intrinsic reasoning abilities ... independent of pretraining knowledge' (Section 1), but this rests entirely on the knowledge-orthogonality premise formalized in Appendix B. That appendix defines conditions (R ⊥ K, β ≈ 0, rule centrality) yet provides no operational evidence that any of the 51 games satisfy them. This is not a minor gap: Table 1 includes Sudoku, Tower of Hanoi, Wordle, 2048, Minesweeper, Sokoban, Tetris, Spider Solitaire, and Trust Evolution, all canonical tasks with well-known strategies, heuristics, and code templates that are abundant in pretraining corpora. For such games, a model can score highly by retrieving a memorized policy (the K channel) rather than by applying in-context rules (the R channel). The paper's own Section 5.2 documents that models invoke established algorithms such as Dijkstra and Eulerian path and use code generation, and that ablating these paradigms changes scores, which is direct evidence that pretraining knowledge is entangled with measured performance. Because the leaderboard aggregates over these games, reported rankings such as O3-mini first and Gemini-2.5-pro second may reflect recall of game-specific strategies rather than general reasoning ability. The premise is asserted, not verified; no per-game leakage control or surface-randomization experiment is reported anywhere in the manuscript.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces KORGym, a game-based benchmark built on Gymnasium that comprises 51 text and visual games across six reasoning dimensions. It describes a modular architecture (inference, game interaction, evaluation, communication), a scoring scheme with per-game normalization and dimension-wise aggregation, and an evaluation campaign covering 19 LLMs and 8 VLMs. The main empirical claims are that KORGym provides stable model rankings, that model families exhibit consistent reasoning profiles, that modality and reasoning-paradigm choices matter, and that response length correlates with performance. The central premise is that the games are 'knowledge-orthogonal,' meaning they measure rule-following reasoning rather than pretraining knowledge.","tokens_in":23099,"tokens_out":5320,"duration_ms":48291,"significance":"If the central claims held, KORGym would be a useful and reusable benchmark: it offers multi-turn interaction, RL support, configurable difficulty, both textual and visual modalities, and a relatively large set of games. The paper reports raw per-game scores in the appendix, describes the evaluation protocol in enough detail to be reimplemented, and compares against several existing game benchmarks. The main limitation is that the load-bearing knowledge-orthogonality premise is asserted rather than verified, and the scoring and statistical analyses contain issues that directly affect the reported rankings. With additional leakage controls, a corrected normalization rule, and proper uncertainty reporting, the platform could become a valuable evaluation resource, but in its current form the empirical conclusions are not fully supported.","major_comments":[{"comment":"The paper's central claim that KORGym measures 'intrinsic reasoning abilities ... independent of pretraining knowledge' rests entirely on the knowledge-orthogonality premise formalized in Appendix B, but that appendix only defines conditions (R ⊥ K, β ≈ 0, rule centrality) and never operationalizes or tests them. Several games in Table 1 (Sudoku, Tower of Hanoi, Wordle, 2048, Minesweeper, Sokoban, Tetris) are canonical tasks with abundant strategies and code templates in pretraining corpora, so high scores can reflect retrieved knowledge rather than in-context rule application. Section 5.2's own evidence—models invoking Dijkstra's algorithm and Eulerian paths, generating code, and changing behavior when paradigms are ablated—shows that the K channel is entangled with measured performance. I am not asking for a philosophical guarantee; I am asking for a concrete leakage control, such as surface-randomized instances, per-game memorization probes, or a demonstration that each game requires rules that are novel at instance level. Without this, Table 4's leaderboard cannot support the 'knowledge-orthogonal' interpretation.","section":"Appendix B; Section 1"},{"comment":"The normalization rule assigns every model a normalized score of 0.5 whenever all models tie on a game (bg = ag). This is exactly what happens for Spider Solitaire, where all 19 entries in Table 11 are 0.000. The resulting 0.5 credit artificially inflates the Strategic Reasoning dimension for every model and changes the aggregated means in Table 4. Degenerate games should be excluded or treated as missing data, with per-game results reported separately, rather than being mapped to a constant score.","section":"Section 3.3, Eq. (3); Table 11"},{"comment":"Section 5.3 states that Doubao-1.5-thinking-pro was RL-trained on 'classic games (e.g., 24-point, mazes, and Sudoku)'. KORGym contains directly analogous games (N-point, Maze, Sudoku). This is overlapping training data between the RL-tuned model and the benchmark, which is exactly the memorization channel the benchmark is designed to eliminate. At minimum, the paper must disclose this overlap and quantify its effect, for example by evaluating Doubao-1.5-thinking-pro on held-out game variants; as written, the fourth-place ranking of this model in Table 4 is confounded.","section":"Section 5.3"},{"comment":"There is a direct numerical contradiction in the PCA analysis. Figure 6b reports PC1 = 71.8% and PC2 = 5.4% (77.2% total), while Appendix D states PC1 = 91.9% and PC2 = 4.3% (96.2% total). Since Section 5.2 uses PCA clusters to support claims about thinking versus non-thinking models and model-series consistency, the variance attribution and cluster labels must be corrected and made consistent among the figure, caption, and appendix.","section":"Section 5.2; Figure 6b; Appendix D"},{"comment":"All reported scores are point estimates without error bars, confidence intervals, or significance tests. With 50 runs per single-epoch game and 20 per multi-epoch game, Table 4 separates models such as Doubao-1.5-thinking-pro (72) and DeepSeek-R1 (71) by a single point; without variance information this difference is not interpretable. The stability analysis in Figure 6a reports standard deviations across dimensions but not across seeds. Please report seed-level variance and either pairwise significance tests or bootstrap confidence intervals for the headline rankings.","section":"Section 4.2, Table 4"}],"minor_comments":[{"comment":"The multiple-epoch protocol says 'we initialize 20 game environments' but also says 'we vary the seed parameter in the generate API from 1 to 50'; clarify how the 20 runs are sampled from the 50 seeds.","section":"Section 4.1"},{"comment":"The phrase 'As shown in Figure 5 compares the performance...' is ungrammatical and should be reworded.","section":"Figure 5 caption"},{"comment":"The model name 'Gemini-2.5-Pro' in Table 5 differs from 'Gemini-2.5-pro-03-25' used elsewhere; use consistent naming across tables and figures.","section":"Table 5"},{"comment":"The game 'Crossword Puzzle' appears twice with different task descriptions (date arithmetic versus linguistic clues); the duplicate entry and intended content should be fixed.","section":"Table 1"},{"comment":"The power-law fit is invoked to support a 'diminishing returns' claim, but no coefficients, exponent, or R² are reported; provide the fitted parameters and goodness-of-fit.","section":"Section 5.4, Figure 8"},{"comment":"The cluster 0 description says 'Claude-3.7' where Figure 6b lists 'Claude-3.7-thinking'; correct the cluster members to match the figure.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The manuscript evaluates a model developed by the same organization (Doubao) and Section 5.3 describes its proprietary RL training; the overlap between that training set and KORGym games is a substantive validity concern that should be addressed head-on. The paper also relies heavily on companion works from the same group for the knowledge-orthogonality framework; independent validation of the leakage controls would strengthen the submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: KORGym is a genuinely reusable game benchmark and the 27-model sweep is substantial, but the headline claim — that it measures reasoning independent of pretraining knowledge — is asserted, not verified. The game list is full of canonical puzzles whose strategies are all over pretraining data, and the paper's own analysis shows models pulling up Dijkstra, Eulerian paths, and code templates when solving them.\n\nWhat's actually new: the combination — 51 games, six labeled reasoning dimensions, textual and visual versions of nine games, multi-turn interaction, controllable difficulty, Gymnasium-style RL interface, and a leaderboard. That is a solid subfield contribution. CDAM is a reasonable aggregation scheme, and the empirical sweep across 19 LLMs and 8 VLMs is real work. The reasoning-paradigm ablation (Section 5.2) is the most interesting result: disabling a paradigm sometimes improves scores, which fits the authors' claim that models over-rely on pretraining patterns.\n\nSoft spots, in order of size:\n\n1. Knowledge orthogonality. Appendix B defines conditions (R ⊥ K, β ≈ 0) but never checks them for a single game. No leakage control, no surface-randomization experiment. Since several games (Sudoku, Wordle, Tower of Hanoi, 2048, Minesweeper) have well-known solved strategies, the 'intrinsic reasoning' framing outruns the evidence. The fix is either per-game leakage checks or re-scoping the claim to 'reasoning in rule-provided interactive environments.' The platform survives either way; the framing does not.\n\n2. Normalization artifact. When every model scores zero on a game (Spider Solitaire, Table 11), the min-max rule gives everyone 0.5. It applies equally so it mostly inflates dimension averages rather than reordering models, but a game where everyone failed should contribute 0, not 0.5.\n\n3. PCA numbers conflict. Figure 6b says PC1 71.8%, PC2 5.4%; Appendix D says 91.9% and 4.3%. Direct contradiction.\n\n4. No error bars or significance tests on any ranking, despite 50 seeds per game being available.\n\n5. The RL conclusion (Section 5.3) compares one externally trained model against the field; there is no controlled RL run inside the benchmark, so the claim that 'appropriate RL enhances reasoning' is suggestive, not tested.\n\nMinor: the power-law fit in Figure 8 has a free exponent and no uncertainty bounds.\n\nWho this is for: anyone building or using LLM reasoning benchmarks or game-based RL environments. The GitHub resource will get reused.\n\nMy call: yes, send it to peer review. The resource is valuable, the flaws are addressable, and the PCA and normalization fixes are mechanical. The main ask for revision is to either verify knowledge orthogonality on a subset of games or soften the claim.","headline":"A genuinely useful game benchmark with a serious model sweep, but its 'knowledge-orthogonal' framing is asserted rather than verified — accept for review, ask for leakage checks or a softer claim.","tokens_in":23700,"tokens_out":5869,"would_cite":true,"duration_ms":48946,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"KORGym is a game platform with more than fifty text and visual games that tests whether LLMs reason from rules rather than memorized knowledge, ranking 19 LLMs and 8 VLMs across six reasoning dimensions.","keywords":["knowledge-orthogonal reasoning","LLM reasoning evaluation","game-based benchmark","multi-turn interaction","reinforcement learning","multimodal reasoning","reasoning dimensions","interactive game platform"],"falsifier":"Measure the knowledge impact factor $\\beta$ defined in Appendix B per game by comparing a model's score when given the full rule text $R$ against its score when the rule text is withheld; if many games yield large positive $\\beta$, or if fine-tuning on KORGym playthrough transcripts substantially raises scores, the platform measures memorization rather than rule application.","tokens_in":22616,"feed_emoji":"🎮","tokens_out":9857,"duration_ms":81202,"temperature":0.7,"pith_summary":"The paper introduces KORGym, a platform of more than fifty text- and vision-based games designed to measure whether large language models can reason from rules they are given, rather than from facts memorized during pretraining. The authors argue that because the games demand multi-turn interaction, state tracking, and adaptive planning, they capture general reasoning better than static question-answer benchmarks. Using the platform, they evaluate 19 LLMs and 8 VLMs, find consistent strength–weakness profiles within model families, and report that closed-source and explicitly 'thinking' models lead the leaderboard. The practical payoff is a reusable benchmark, with scores normalized across six reasoning dimensions, that can also serve as a reinforcement-learning environment for training reasoning skills.","feed_headline":"Fifty-one games score LLMs on reasoning, not memory","feed_subtitle":"A multi-turn game platform ranks 19 language models and 8 vision models across six reasoning dimensions.","key_machinery":"The load-bearing mechanisms are the formal definition of knowledge orthogonality from the paper's Appendix B and the standardized game interaction loop. Knowledge orthogonality decomposes a task into rule information $R$, background knowledge $K$, question $Q$, and answer $A$, then requires $R \\perp K$ and a knowledge impact factor $\\beta$ near zero, meaning the model's belief in $A$ comes from applying the supplied rules, not from retrieved knowledge. The evaluation loop runs each game as a multi-turn environment through generate (initialize), print_board (render state), and verify (apply action, return reward), which is the same interface family used for reinforcement-learning environments. Scores are combined by the Capability Dimension Aggregated Mean: raw scores are log-compressed when unbounded, normalized per game to the $[0,1]$ interval, and averaged within six capability dimensions to produce the final leaderboard.","core_discovery":"KORGym's central claim is that a set of over fifty games, each exposing only its own rules to the model, can serve as a knowledge-orthogonal evaluation of intrinsic reasoning. Following the knowledge-orthogonality framework, each task is modeled as a question $Q$ whose answer $A$ should be derivable almost entirely from the rule information $R$ supplied in the prompt, with background knowledge $K$ contributing a negligible knowledge impact factor $\\beta$. The platform standardizes each game behind three APIs, generate, print_board, and verify, so model decisions are scored turn by turn and aggregated by a per-dimension normalized mean that log-compresses unbounded scores and min-max normalizes per game. On this measure the top ranks are held by closed-source and explicitly reasoning-oriented models, thinking models outperform non-thinking counterparts at matched scale, and models within a family show correlated strengths and weaknesses, which the authors take as evidence that KORGym exposes stable reasoning traits rather than task-specific memorization.","pith_inferences":["If knowledge orthogonality is taken at face value, one could use KORGym to build a curriculum that trains models on rule-following without domain knowledge; the paper's paradigm ablations suggest such training should explicitly discourage reliance on memorized code, math, or algorithmic templates.","The per-game min-max normalization makes scores relative to the model pool, so a reader should treat reported percentages as comparative rankings rather than absolute ability levels, and expect shifts as new models join.","Because KORGym deliberately excludes opponent dynamics that allow hacked strategies, its strategic-reasoning dimension likely measures planning against fixed policies rather than full adversarial interaction; extending it with robust multi-agent games would test social reasoning.","A direct test of the orthogonality assumption would be to measure the Appendix-B knowledge impact factor per game by comparing accuracy with and without the supplied rule text, which would tell the field which games are truly knowledge-orthogonal."],"forward_implications":["A model's KORGym score becomes evidence about its ability to follow novel rules under interactive feedback, distinct from its ability to recall facts or solve familiar problem templates.","The platform can double as a reinforcement-learning environment: states and rewards from the same APIs can be used to train models, and the paper reports that an RL-trained thinking model reaches the top tier.","The reported rankings imply that within a model family, relative strengths and weaknesses persist across diverse tasks, so a single family's profile may predict its behavior on new interactive problems.","Textual versions of games generally outscore visual versions, especially for open-source vision-language models, identifying multimodal grounding as a distinct bottleneck.","Response length correlates positively with score, with diminishing returns, so verbosity is a real but saturating contributor to reasoning performance."],"supporting_citations":[{"why":"Supplies the knowledge-orthogonal reasoning formalism ($R$, $K$, $\\beta$) that KORGym adopts as its core design goal.","marker":"[21]"},{"why":"Provides the reinforcement-learning environment interface standard on which KORGym's game interaction module is built.","marker":"[36]"},{"why":"A single-turn game benchmark used as the main contrast motivating KORGym's multi-turn evaluation design.","marker":"[12]"},{"why":"A multi-turn benchmark whose opponent-driven variability KORGym identifies as a confound it deliberately avoids.","marker":"[11]"},{"why":"The strategic-planning benchmark compared in Table 2 that KORGym positions itself against.","marker":"[41]"},{"why":"A 100+ game reasoning benchmark used in Table 2 to show KORGym's combination of RL support, controllable difficulty, and multimodality.","marker":"[32]"}],"fun_headline_variants":["Fifty games, zero memorization: LLM reasoning scored","KORGym: 51 games rank LLMs by reasoning, not recall","Interactive games reveal LLM reasoning strengths, not memorization","Reasoning gym: 51 games score LLMs on thinking, not recall","51 games evaluate LLM reasoning dynamically, not memorized facts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The games are truly knowledge-orthogonal: success comes from applying the supplied rules, not from retrieving strategies or patterns the model memorized during pretraining.","fun_headline_variants_meta":{"raw":{"variants":["Fifty games, zero memorization: LLM reasoning scored","KORGym: 51 games rank LLMs by reasoning, not recall","Interactive games reveal LLM reasoning strengths, not memorization","Reasoning gym: 51 games score LLMs on thinking, not recall","51 games evaluate LLM reasoning dynamically, not memorized facts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00059,"raw_usage":{"total_tokens":2747,"prompt_tokens":901,"completion_tokens":1846,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":1765}},"tokens_in":517,"tokens_out":1846,"duration_ms":12868,"temperature":1.0,"reasoning_tokens":1765,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:31:47.281029+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the knowledge impact factor $\\beta$ defined in Appendix B per game by comparing a model's score when given the full rule text $R$ against its score when the rule text is withheld; if many games yield large positive $\\beta$, or if fine-tuning on KORGym playthrough transcripts substantially raises scores, the platform measures memorization rather than rule application.","supporting_citations":[{"cited_title":"Reasonggym","cited_arxiv_id":null,"evidence_quote":"A 100+ game reasoning benchmark used in Table 2 to show KORGym's combination of RL support, controllable difficulty, and multimodality."}],"review_version":1}