Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read KORGym is a game platform with more than fifty text and visual games that tests whether LLMs reason from rules rather than memorized knowledge, ranking 19 LLMs and 8 VLMs across six reasoning dimensions.

desk verdict A genuinely useful game benchmark with a serious model sweep, but its 'knowledge-orthogonal' framing is asserted rather than verified — accept for review, ask for leakage checks or a softer claim. read the letter →

arxiv 2505.14552 v2 pith:Q5EJQCVF submitted 2025-05-20 cs.CL cs.AIcs.LG

Jiajun Shi , Jian Yang , Jiaheng Liu , Xingyuan Bu , Jiangjie Chen , Junting Zhou , Kaijing Ma , Zhoufutu Wen
show 21 more authors
Bingli Wang Yancheng He Liang Song Hualei Zhu Shilong Li Xingjian Wang Wei Zhang Ruibin Yuan Yifan Yao Wenjun Yang Yunli Wang Siyuan Fang Siyu Yuan Qianyu He Xiangru Tang Yingshui Tan Wangchunshu Zhou Zhaoxiang Zhang Zhoujun Li Wenhao Huang Ge Zhang
This is my paper
classification cs.CLcs.AIcs.LG
keywords knowledge-orthogonalreasoningLLMevaluationgame-basedbenchmarkmulti-turninteractionreinforcementlearningmultimodaldimensionsinteractivegameplatform
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces KORGym, a platform of more than fifty text- and vision-based games designed to measure whether large language models can reason from rules they are given, rather than from facts memorized during pretraining. The authors argue that because the games demand multi-turn interaction, state tracking, and adaptive planning, they capture general reasoning better than static question-answer benchmarks. Using the platform, they evaluate 19 LLMs and 8 VLMs, find consistent strength–weakness profiles within model families, and report that closed-source and explicitly 'thinking' models lead the leaderboard. The practical payoff is a reusable benchmark, with scores normalized across six reasoning dimensions, that can also serve as a reinforcement-learning environment for training reasoning skills.

What carries the argument

The load-bearing mechanisms are the formal definition of knowledge orthogonality from the paper's Appendix B and the standardized game interaction loop. Knowledge orthogonality decomposes a task into rule information $R$, background knowledge $K$, question $Q$, and answer $A$, then requires $R \perp K$ and a knowledge impact factor $\beta$ near zero, meaning the model's belief in $A$ comes from applying the supplied rules, not from retrieved knowledge. The evaluation loop runs each game as a multi-turn environment through generate (initialize), print_board (render state), and verify (apply action, return reward), which is the same interface family used for reinforcement-learning environments. Scores are combined by the Capability Dimension Aggregated Mean: raw scores are log-compressed when unbounded, normalized per game to the $[0,1]$ interval, and averaged within six capability dimensions to produce the final leaderboard.

What would settle it

Measure the knowledge impact factor $\beta$ defined in Appendix B per game by comparing a model's score when given the full rule text $R$ against its score when the rule text is withheld; if many games yield large positive $\beta$, or if fine-tuning on KORGym playthrough transcripts substantially raises scores, the platform measures memorization rather than rule application.

Watch

Extended reading notes

Core claim

KORGym's central claim is that a set of over fifty games, each exposing only its own rules to the model, can serve as a knowledge-orthogonal evaluation of intrinsic reasoning. Following the knowledge-orthogonality framework, each task is modeled as a question $Q$ whose answer $A$ should be derivable almost entirely from the rule information $R$ supplied in the prompt, with background knowledge $K$ contributing a negligible knowledge impact factor $\beta$. The platform standardizes each game behind three APIs, generate, print_board, and verify, so model decisions are scored turn by turn and aggregated by a per-dimension normalized mean that log-compresses unbounded scores and min-max normalizes per game. On this measure the top ranks are held by closed-source and explicitly reasoning-oriented models, thinking models outperform non-thinking counterparts at matched scale, and models within a family show correlated strengths and weaknesses, which the authors take as evidence that KORGym exposes stable reasoning traits rather than task-specific memorization.

Load-bearing premise

The games are truly knowledge-orthogonal: success comes from applying the supplied rules, not from retrieving strategies or patterns the model memorized during pretraining.

Editorial extensions

If this is right

  • A model's KORGym score becomes evidence about its ability to follow novel rules under interactive feedback, distinct from its ability to recall facts or solve familiar problem templates.
  • The platform can double as a reinforcement-learning environment: states and rewards from the same APIs can be used to train models, and the paper reports that an RL-trained thinking model reaches the top tier.
  • The reported rankings imply that within a model family, relative strengths and weaknesses persist across diverse tasks, so a single family's profile may predict its behavior on new interactive problems.
  • Textual versions of games generally outscore visual versions, especially for open-source vision-language models, identifying multimodal grounding as a distinct bottleneck.
  • Response length correlates positively with score, with diminishing returns, so verbosity is a real but saturating contributor to reasoning performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If knowledge orthogonality is taken at face value, one could use KORGym to build a curriculum that trains models on rule-following without domain knowledge; the paper's paradigm ablations suggest such training should explicitly discourage reliance on memorized code, math, or algorithmic templates.
  • The per-game min-max normalization makes scores relative to the model pool, so a reader should treat reported percentages as comparative rankings rather than absolute ability levels, and expect shifts as new models join.
  • Because KORGym deliberately excludes opponent dynamics that allow hacked strategies, its strategic-reasoning dimension likely measures planning against fixed policies rather than full adversarial interaction; extending it with robust multi-agent games would test social reasoning.
  • A direct test of the orthogonality assumption would be to measure the Appendix-B knowledge impact factor per game by comparing accuracy with and without the supplied rule text, which would tell the field which games are truly knowledge-orthogonal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces KORGym, a game-based benchmark built on Gymnasium that comprises 51 text and visual games across six reasoning dimensions. It describes a modular architecture (inference, game interaction, evaluation, communication), a scoring scheme with per-game normalization and dimension-wise aggregation, and an evaluation campaign covering 19 LLMs and 8 VLMs. The main empirical claims are that KORGym provides stable model rankings, that model families exhibit consistent reasoning profiles, that modality and reasoning-paradigm choices matter, and that response length correlates with performance. The central premise is that the games are 'knowledge-orthogonal,' meaning they measure rule-following reasoning rather than pretraining knowledge.

Significance. If the central claims held, KORGym would be a useful and reusable benchmark: it offers multi-turn interaction, RL support, configurable difficulty, both textual and visual modalities, and a relatively large set of games. The paper reports raw per-game scores in the appendix, describes the evaluation protocol in enough detail to be reimplemented, and compares against several existing game benchmarks. The main limitation is that the load-bearing knowledge-orthogonality premise is asserted rather than verified, and the scoring and statistical analyses contain issues that directly affect the reported rankings. With additional leakage controls, a corrected normalization rule, and proper uncertainty reporting, the platform could become a valuable evaluation resource, but in its current form the empirical conclusions are not fully supported.

major comments (5)
  1. [Appendix B; Section 1] The paper's central claim that KORGym measures 'intrinsic reasoning abilities ... independent of pretraining knowledge' rests entirely on the knowledge-orthogonality premise formalized in Appendix B, but that appendix only defines conditions (R ⊥ K, β ≈ 0, rule centrality) and never operationalizes or tests them. Several games in Table 1 (Sudoku, Tower of Hanoi, Wordle, 2048, Minesweeper, Sokoban, Tetris) are canonical tasks with abundant strategies and code templates in pretraining corpora, so high scores can reflect retrieved knowledge rather than in-context rule application. Section 5.2's own evidence—models invoking Dijkstra's algorithm and Eulerian paths, generating code, and changing behavior when paradigms are ablated—shows that the K channel is entangled with measured performance. I am not asking for a philosophical guarantee; I am asking for a concrete leakage control, such as surface-randomized instances, per-game memorization probes, or a demonstration that each game requires rules that are novel at instance level. Without this, Table 4's leaderboard cannot support the 'knowledge-orthogonal' interpretation.
  2. [Section 3.3, Eq. (3); Table 11] The normalization rule assigns every model a normalized score of 0.5 whenever all models tie on a game (bg = ag). This is exactly what happens for Spider Solitaire, where all 19 entries in Table 11 are 0.000. The resulting 0.5 credit artificially inflates the Strategic Reasoning dimension for every model and changes the aggregated means in Table 4. Degenerate games should be excluded or treated as missing data, with per-game results reported separately, rather than being mapped to a constant score.
  3. [Section 5.3] Section 5.3 states that Doubao-1.5-thinking-pro was RL-trained on 'classic games (e.g., 24-point, mazes, and Sudoku)'. KORGym contains directly analogous games (N-point, Maze, Sudoku). This is overlapping training data between the RL-tuned model and the benchmark, which is exactly the memorization channel the benchmark is designed to eliminate. At minimum, the paper must disclose this overlap and quantify its effect, for example by evaluating Doubao-1.5-thinking-pro on held-out game variants; as written, the fourth-place ranking of this model in Table 4 is confounded.
  4. [Section 5.2; Figure 6b; Appendix D] There is a direct numerical contradiction in the PCA analysis. Figure 6b reports PC1 = 71.8% and PC2 = 5.4% (77.2% total), while Appendix D states PC1 = 91.9% and PC2 = 4.3% (96.2% total). Since Section 5.2 uses PCA clusters to support claims about thinking versus non-thinking models and model-series consistency, the variance attribution and cluster labels must be corrected and made consistent among the figure, caption, and appendix.
  5. [Section 4.2, Table 4] All reported scores are point estimates without error bars, confidence intervals, or significance tests. With 50 runs per single-epoch game and 20 per multi-epoch game, Table 4 separates models such as Doubao-1.5-thinking-pro (72) and DeepSeek-R1 (71) by a single point; without variance information this difference is not interpretable. The stability analysis in Figure 6a reports standard deviations across dimensions but not across seeds. Please report seed-level variance and either pairwise significance tests or bootstrap confidence intervals for the headline rankings.
minor comments (6)
  1. [Section 4.1] The multiple-epoch protocol says 'we initialize 20 game environments' but also says 'we vary the seed parameter in the generate API from 1 to 50'; clarify how the 20 runs are sampled from the 50 seeds.
  2. [Figure 5 caption] The phrase 'As shown in Figure 5 compares the performance...' is ungrammatical and should be reworded.
  3. [Table 5] The model name 'Gemini-2.5-Pro' in Table 5 differs from 'Gemini-2.5-pro-03-25' used elsewhere; use consistent naming across tables and figures.
  4. [Table 1] The game 'Crossword Puzzle' appears twice with different task descriptions (date arithmetic versus linguistic clues); the duplicate entry and intended content should be fixed.
  5. [Section 5.4, Figure 8] The power-law fit is invoked to support a 'diminishing returns' claim, but no coefficients, exponent, or R² are reported; provide the fitted parameters and goodness-of-fit.
  6. [Appendix D] The cluster 0 description says 'Claude-3.7' where Figure 6b lists 'Claude-3.7-thinking'; correct the cluster members to match the figure.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular reduction in the scoring or ranking; the only self-citation (KOR-Bench) is not load-bearing, though the knowledge-orthogonality premise is unverified.

full rationale

The paper's central measurements are direct game scores: Section 3.3 defines raw scores via binary, proportional, and cumulative rules, then applies a log transform and min-max normalization across models (Eqs. 1-4). No parameter is fitted to the models being ranked and no prediction is derived from a fitted input; the leaderboard in Table 4 is an aggregation of measured game outcomes. The only self-citation is KOR-Bench [21] for the knowledge-orthogonality framework, and several authors overlap between KORGym and KOR-Bench; however, the formal conditions are restated in Appendix B rather than imported as an external proof, and the benchmark scores themselves do not depend on that definition. The load-bearing assumption that the games satisfy knowledge orthogonality (R ⊥ K, β ≈ 0) is asserted without per-game verification, and Table 1 includes canonical games such as Sudoku, Wordle, Tower of Hanoi, 2048, Minesweeper, Sokoban, and Tetris whose strategies are common in pretraining data. Section 5.2 further documents models invoking Dijkstra, Eulerian paths, and code generation, which suggests pretraining knowledge can be entangled with measured performance. This is a validity and contamination concern, not a circular reduction: no equation in the paper reduces to its own input, and the scores are not constructed from the claims they are said to support. Score 2 reflects the minor self-citation only; the central benchmark construction is self-contained.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central evaluation rests on the knowledge-orthogonality assumption inherited from KOR-Bench and on the game-score proxy for reasoning. The only fitted numerical parameters reported are the unreported power-law coefficients in RQ4. No new physical or conceptual entities are introduced.

free parameters (1)
  • Power-law exponent and coefficient for response length vs score = not reported
    Section 5.4 and Figure 8 fit a power law to support the diminishing-returns claim; parameters are not given, so the correlation cannot be independently evaluated.
assumptions (4)
  • domain assumption KORGym games satisfy knowledge orthogonality (R ⊥ K and rule centrality of Appendix B)
    The paper asserts without per-game verification that performance reflects rule application rather than pretraining knowledge; this underpins the claim of measuring intrinsic reasoning.
  • domain assumption Raw game scores are valid proxies for reasoning ability
    Section 3.3 defines scoring and aggregation but provides no validation that game scores correlate with other reasoning measures or human judgment.
  • domain assumption GPT-4o annotations of reasoning paradigms are accurate
    Section 5.2 uses GPT-4o to classify model responses into four reasoning paradigms; incorrect labels would change the computed per-paradigm scores.
  • domain assumption PCA clusters on normalized per-game scores reflect behavioral traits
    Figure 6b and Appendix D interpret PCA projections as behavioral patterns; the normalization makes scores relative to the evaluated model set, so clusters depend on which models are included.

how reviews work

0 comments
Cite this review

Pith. "Pith review of KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation." pith.science (2026). https://pith.science/paper/Q5EJQCVF

@misc{pith2026250514552,
  author       = {Pith},
  title        = {Pith review of: KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q5EJQCVF}},
  note         = {Machine review of arXiv:2505.14552}
}
read the original abstract

Recent advancements in large language models (LLMs) underscore the need for more comprehensive evaluation methods to accurately assess their reasoning capabilities. Existing benchmarks are often domain-specific and thus cannot fully capture an LLM's general reasoning potential. To address this limitation, we introduce the Knowledge Orthogonal Reasoning Gymnasium (KORGym), a dynamic evaluation platform inspired by KOR-Bench and Gymnasium. KORGym offers over fifty games in either textual or visual formats and supports interactive, multi-turn assessments with reinforcement learning scenarios. Using KORGym, we conduct extensive experiments on 19 LLMs and 8 VLMs, revealing consistent reasoning patterns within model families and demonstrating the superior performance of closed-source models. Further analysis examines the effects of modality, reasoning strategies, reinforcement learning techniques, and response length on model performance. We expect KORGym to become a valuable resource for advancing LLM reasoning research and developing evaluation methodologies suited to complex, interactive environments.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SciDA: Scientific Dynamic Assessor of LLMs

    cs.CL 2025-06 conditional novelty 5.0 of 10

    SciDA is a dynamically initialized, multi-discipline olympiad benchmark that shows LLMs perform substantially worse when problem variables are randomized, which the authors attribute to memorization of fixed numerical...

Reference graph

Works this paper leans on

50 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    Claude 3.7 sonnet and claude code, 2025

    Anthropic. Claude 3.7 sonnet and claude code, 2025. URLhttps://www.anthropic.com/claude/sonnet

    work page 2025

  2. [2]

    Qwen2.5-VL Technical Report.arXiv e-prints, art

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL Technical Report.ar...

    arXiv doi:10.48550/arxiv.2502.13923 2025

  3. [3]

    Zebralogic: Benchmarking the logical reasoning ability of language models, 2024

    Yejin Choi Bill Yuchen Lin, Ronan Le Bras. Zebralogic: Benchmarking the logical reasoning ability of language models, 2024. URL https://hf.co/spaces/allenai/ZebraLogicBench-Leaderboard

    work page 2024

  4. [4]

    Doubao-vision-pro, 2025

    ByteDance. Doubao-vision-pro, 2025

    work page 2025

  5. [5]

    Doubao-1.5-pro, 2025

    ByteDance. Doubao-1.5-pro, 2025. URL https://team.doubao.com/en/special/doubao_1_5_pro

    work page 2025

  6. [6]

    Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks.CoRR, abs/2306.13831, 2023

    Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks.CoRR, abs/2306.13831, 2023

    arXiv 2023

  7. [7]

    Training Verifiers to Solve Math Word Problems.arXiv e-prints, art

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems.arXiv e-prints, art. arXiv:2110.14168, October 2021. doi: 10.48550/arXiv.2110.14168

    arXiv doi:10.48550/arxiv.2110.14168

  8. [8]

    Gemini 2.0 flash thinking, 2024

    Google DeepMind. Gemini 2.0 flash thinking, 2024. URLhttps://deepmind.google/technologies/gemini/ flash-thinking/

    2024

Show all 50 references
  1. [9]

    Gemini 2.5: Our most intelligent ai model, 2025

    Google DeepMind. Gemini 2.5: Our most intelligent ai model, 2025. URLhttps://blog.google/technology/ google-deepmind/gemini-model-thinking-updates-march-2025/

    2025

  2. [10]

    Mindagent: Emergent gaming interaction

    Ran Gong, Qiuyuan Huang, Xiaojian Ma, Yusuke Noda, Zane Durante, Zilong Zheng, Demetri Terzopoulos, Li Fei-Fei, Jianfeng Gao, and Hoi Vo. Mindagent: Emergent gaming interaction. In Kevin Duh, Helena Gómez- Adorno, and Steven Bethard, editors,Findings of the Association for Com...

    2024 doi

  3. [11]

    TextArena.arXiv e-prints, art

    Leon Guertler, Bobby Cheng, Simon Yu, Bo Liu, Leshem Choshen, and Cheston Tan. TextArena.arXiv e-prints, art. arXiv:2504.11442, April 2025. doi: 10.48550/arXiv.2504.11442

    2025 doi arXiv

  4. [12]

    LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models.arXiv e-prints, art

    Jiayi Gui, Yiming Liu, Jiale Cheng, Xiaotao Gu, Xiao Liu, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models.arXiv e-prints, art. arXiv:2408.15778, August 2024. doi: 10.48550/arXiv.2408.15778

    2024 doi arXiv

  5. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  6. [14]

    Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Victoria Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, and Dragomir Radev

    Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alex Wardle-Solano, Hannah Szabo, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni,...

    2022 doi arXiv

  7. [15]

    Measuring Massive Multitask Language Understanding.arXiv e-prints, art

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding.arXiv e-prints, art. arXiv:2009.03300, September 2020. doi: 10.48550/arXiv.2009.03300

    2009 doi arXiv

  8. [16]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  9. [17]

    GameArena: Evaluating LLM Reasoning through Live Computer Games.arXiv e-prints, art

    Lanxiang Hu, Qiyu Li, Anze Xie, Nan Jiang, Ion Stoica, Haojian Jin, and Hao Zhang. GameArena: Evaluating LLM Reasoning through Live Computer Games.arXiv e-prints, art. arXiv:2412.06394, December 2024. doi: 10.48550/arXiv.2412.06394. 14

    2024 doi arXiv

  10. [18]

    Avalonbench: Evaluating llms playing the game of avalon

    Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. Avalonbench: Evaluating llms playing the game of avalon. arXiv preprint arXiv:2310.05036, 2023

    2023 arXiv

  11. [19]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  12. [20]

    AgentBench: Evaluating LLMs as Agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench:...

    2023 doi arXiv

  13. [21]

    KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Tasks.arXiv e-prints, art

    Kaijing Ma, Xinrun Du, Yunran Wang, Haoran Zhang, Zhoufutu Wen, Xingwei Qu, Jian Yang, Jiaheng Liu, Minghao Liu, Xiang Yue, Wenhao Huang, and Ge Zhang. KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Tasks.arXiv e-prints, art. arXiv:2410.06526, Octobe...

    doi arXiv

  14. [22]

    Learning to reason with llms, 2024

    OpenAI. Learning to reason with llms, 2024. URL https://openai.com/index/ learning-to-reason-with-llms/

    2024

  15. [23]

    Gpt-4o system card

    OpenAI. Gpt-4o system card. Technical report, OpenAI, 2024.https://www.openai.com/research/gpt-4o

    2024

  16. [24]

    Learning to reason with llms, 2025

    OpenAI. Learning to reason with llms, 2025. URLhttps://openai.com/index/openai-o3-mini/

    2025

  17. [25]

    SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines.arXiv e-prints, art

    P Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixin Deng, Shawn Gavin, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, Sirun Li, Yizhi Li, Yunwen Li, David Ma, Yuanshe...

    2025 doi arXiv

  18. [26]

    Suttle, Mengdi Wang, Amrit Singh Bedi, and Dinesh Manocha

    Bhrij Patel, Souradip Chakraborty, Wesley A. Suttle, Mengdi Wang, Amrit Singh Bedi, and Dinesh Manocha. AIME: AI System Optimization via Multiple LLM Evaluators.arXiv e-prints, art. arXiv:2410.03131, October

  19. [27]

    Wang, John-Clark Levin, Mstyslav Kazakov, Fiona Feng, Steven Y

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Dmitry Dodonov, T...

    2025 doi arXiv

  20. [28]

    PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models.arXiv e-prints, art

    Shi Qiu, Shaoyang Guo, Zhuo-Yang Song, Yunbo Sun, Zeyu Cai, Jiashen Wei, Tianyu Luo, Yixuan Yin, Haoxu Zhang, Yi Hu, Chenyang Wang, Chencheng Tang, Haoling Chang, Qi Liu, Ziheng Zhou, Tianyu Zhang, Jingtian Zhang, Zhangyi Liu, Minghao Li, Yuku Zhang, Boxuan Jing, Xianqi Yin, Y...

    2025 doi arXiv

  21. [29]

    Seed-Thinking-v1.5: Advancing Superb Reasoning Models with Reinforcement Learning.arXiv e-prints, art

    ByteDance Seed, :, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, Yufeng Yuan, Yu Yue, Lin Yan, Qiying Yu, Xiaochen Zuo, Chi Zhang, Ruofei Zhu, Zhecheng An, Zhihao Bai, Yu Bao, Xingyan Bin, Jiangjie Chen, Feng...

  22. [30]

    Cryptox : Compositional reasoning evaluation of large language models,

    Jiajun Shi, Chaoren Wei, Liqun Yang, Zekun Moore Wang, Chenghao Yang, Ge Zhang, Stephen Huang, Tao Peng, Jian Yang, and Zhoufutu Wen. Cryptox : Compositional reasoning evaluation of large language models,

  23. [31]

    CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. arXiv e-prints, art. arXiv:1811.00937, November 2018. doi: 10.48550/arXiv.1811.00937

    2018 doi arXiv

  24. [32]

    Reasonggym

    Open-Thought Team. Reasonggym. URLhttps://github.com/open-thought/reasoning-gym

  25. [33]

    URL https://arxiv.org/abs/2502.07813

    arXiv

  26. [34]

    Qwen3, 2025

    Qwen Team. Qwen3, 2025. URLhttps://qwen3.org/. 16

    2025

  27. [35]

    Qwen-qwq, 2025

    Qwen Team. Qwen-qwq, 2025. URLhttps://qwenlm.github.io/zh/blog/qwq-32b/

    2025

  28. [36]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024. URLhttps://qwenlm.github.io/blog/ qwen2.5/

    2024

  29. [37]

    Measuring General Intelligence with Generated Games

    Vivek Verma, David Huang, William Chen, Dan Klein, and Nicholas Tomlin. Measuring General Intelligence with Generated Games. arXiv e-prints, art. arXiv:2505.07215, May 2025

    2025 arXiv

  30. [38]

    Evaluating reading comprehension exercises generated by LLMs: A showcase of ChatGPT in education applications

    Changrong Xiao, Sean Xin Xu, Kunpeng Zhang, Yufang Wang, and Lei Xia. Evaluating reading comprehension exercises generated by LLMs: A showcase of ChatGPT in education applications. In Ekaterina Kochmar, Jill Burstein, Andrea Horbach, Ronja Laarmann-Quante, Nitin Madnani, Anaïs...

    2023

  31. [39]

    Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Hannah Tan, and Omar G. Younis. Gymnasium: A Standar...

    2024 doi arXiv

  32. [40]

    Exploring large language models for communication games: An empirical study on werewolf.arXiv preprint arXiv:2309.04658, 2023

    Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf.arXiv preprint arXiv:2309.04658, 2023

    2023 arXiv

  33. [41]

    SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially? arXiv e-prints, art

    Jianzhu Yao, Kevin Wang, Ryan Hsieh, Haisu Zhou, Tianqing Zou, Zerui Cheng, Zhangyang Wang, and Pramod Viswanath. SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially? arXiv e-prints, art. arXiv:2503.12349, March 2025. doi: 10.48550/arXiv.2503.12349

    2025 doi

  34. [42]

    Travelplanner: A benchmark for real-world planning with language agents.arXiv preprint arXiv:2402.01622, 2024

    Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents.arXiv preprint arXiv:2402.01622, 2024

    2024 arXiv

  35. [43]

    InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models.arXiv e-prints, art

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li,...

    2025 doi arXiv

  36. [45]

    Spin-bench: How well do llms plan strategically and reason socially?arXiv preprint arXiv:2503.12349, 2025

    Jianzhu Yao, Kevin Wang, Ryan Hsieh, Haisu Zhou, Tianqing Zou, Zerui Cheng, Zhangyang Wang, and Pramod Viswanath. Spin-bench: How well do llms plan strategically and reason socially?arXiv preprint arXiv:2503.12349, 2025

    2025

  37. [47]

    Knowledge-Rule Decoupling: Rule R is logically self-contained and independent ofK. R ⊥ K

  38. [48]

    Knowledge Assistiveness: Background knowledgeK may support or interfere with the derivation ofA from Q, but does not play a central role in reasoning. The extent of this influence is quantified by the Knowledge Impact Factor (β), defined as: β = P (Q → A | R, K) − P (Q → A | R...

  39. [49]

    P (Q → A | R, K) ≈ P (Q → A | R) ≫ P (Q → A | K)

    Rule Centrality: Correctness relies on understanding and applyingR, with R having significantly greater influence than K. P (Q → A | R, K) ≈ P (Q → A | R) ≫ P (Q → A | K)

  40. [50]

    P (Q → A | R, K) =P (Q → A | R) · (1 +β) 19 C Detailed Scores on KORGym Table 7 Mathematical and logical reasoning abilities of different models on KORGym

    Derivation Adjustment: This formula adjusts the reasoning process based onR, incorporating the influence of K with β reflecting its effect. P (Q → A | R, K) =P (Q → A | R) · (1 +β) 19 C Detailed Scores on KORGym Table 7 Mathematical and logical reasoning abilities of different...

    2024

  41. [2024]

    doi: 10.48550/arXiv.2410.03131

    doi

  42. [2025]

    doi: 10.48550/arXiv.2504.13914

    doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.