{"id":"67d274e1-33b8-4654-a6df-b48849ae309b","arxiv_id":"2506.10209","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Models that ace hard math benchmarks score much lower on a new benchmark of simple strategic board games, exposing a gap in basic reasoning.","lead":"This paper introduces a new benchmark of simple Tic-Tac-Toe style games and finds that many top math-solving AI models perform poorly on them. The finding matters because it suggests current reasoning models may lack the basic strategic and spatial reasoning that humans find trivial.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Answer keys rely on an unverified heuristic that can misclassify forks and whose printed pseudo-code is internally inconsistent, so the reported performance gap may be an artifact of incorrect labels.","rationale":"The reader correctly identified the unverified heuristic as the weakest assumption, and this stress-test agrees that answer-key correctness is the most load-bearing condition for the central claim. Our critique sharpens that concern in two ways: the get_forks definition of a fork is game-theoretically incomplete because it ignores common-block squares, and the printed Algorithm 1 is internally inconsistent in a way that would prevent Fork verdicts from being generated at all. Both issues are concrete and testable. We do not see a reason to change the reader's CONDITIONAL verdict: the benchmark is valuable and the evaluation is otherwise consistent, but the central comparative claim cannot be fully trusted until the keys are verified by an exact solver. The proposed minimax check directly settles whether the concern lands. If the keys are mostly correct, the performance gap likely reflects a real limitation; if not, the headline overclaims. This is a conditional, not a rejection, because all identified issues are addressable with code-level verification and re-scoring.","tokens_in":18595,"tokens_out":7810,"duration_ms":101024,"concrete_test":"Implement an exact minimax solver for all four games (oTTT, dTTT, cTTT, sTTT) that assigns each board state a game-theoretic value (win/loss/draw) and enumerates all optimal moves for the player to move. Run it on all 412 generated questions and compare the solver's optimal-move set to the benchmark's answer keys for every question. Then re-score all 26 models against the corrected keys and recompute the average ΔPass@1 versus MATH 500 and AIME 2024. If the recomputed gaps remain comparable (e.g., >30 points versus MATH 500 and still negative versus AIME 2024), the central claim is robust; if the gaps shrink materially or change sign, the headline finding is an artifact of heuristic labeling errors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that LRMs excel at math but fail at simple strategic games—stands or falls on the correctness of the benchmark's answer keys, which are produced solely by Algorithm 1 (Appendix C). That algorithm has two concrete problems. First, the 'Fork' check in get_forks counts any two winning lines that contain n-1 current-player stones and no opponent stones, but it never tests whether a single empty square lies at the intersection of both threat lines. If such a common square exists, the opponent can play there and block both threats, so the move is not actually a fork. A model that correctly identifies that common-square block would be marked wrong. Second, the printed get_solution function returns 'None' whenever the opponent has zero or multiple immediate winning moves (the 'else return None' branch), which makes the subsequent get_forks call unreachable and would prevent any 'Fork' verdict from being generated. Figure 2 reports Fork verdicts, so either the appendix contains a transcription error or the real implementation differs from the published algorithm. Either way, the answer-key generation is not reproducible from the paper alone. No minimax verification or human baseline is provided to confirm that the heuristic's move set equals the game-theoretically optimal move set. If the keys contain even a modest fraction of false labels, the measured Pass@1 scores and the headline ΔPass@1 gaps relative to MATH 500 and AIME 2024 could shift substantially, undermining the claimed 'fundamental gap' in basic strategic reasoning.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TTT-Bench, a benchmark of four two-player Tic-Tac-Toe-style games (oTTT, dTTT, cTTT, sTTT) with programmatically generated questions that ask for the next best move. Answers are labeled by a heuristic (Algorithm 1) into Win, Blocked, or Fork verdicts. The authors evaluate a wide range of open LRMs plus o3-mini-medium and DeepSeek-R1 on TTT-Bench and compare with MATH 500 and AIME 2024, reporting that models score on average 41% lower than MATH 500 and 5% lower than AIME 2024, with the largest drops on Fork-type questions. They also report that models use longer chains of thought on TTT-Bench than on MATH 500, and conclude that LRMs that excel at hard math frequently fail at simple strategic and spatial reasoning tasks.","tokens_in":18886,"tokens_out":11124,"duration_ms":138589,"significance":"If the answer keys are correct, TTT-Bench is a valuable and contamination-resistant addition to reasoning evaluation: it is simple, scalable, and covers a domain (strategic and spatial reasoning) that is underrepresented in math-centric benchmarks. The paper's strengths include releasing the benchmark, evaluating a broad set of open models, and reporting per-verdict results that yield a falsifiable difficulty ordering. The central claim, however, rests on the correctness of the heuristic-generated labels and on the assertion that the games are trivial for humans; neither is currently established. The reported performance gaps could be artifacts of mislabeled optimal moves or of protocol differences, so benchmark validation is needed before the main conclusion can be accepted.","major_comments":[{"comment":"As printed, the `else return None` branch after the Blocked check means that `get_solution` returns whenever the opponent has zero or more than one immediate winning move. Consequently, `get_forks` is unreachable and no Fork verdict can ever be generated, which directly contradicts Figure 2 and Table 4, both of which report substantial numbers of Fork questions. This is a load-bearing inconsistency: the answer-key generation is not reproducible from the paper alone. Please correct the pseudocode or provide the exact implementation used, and confirm that the released benchmark actually contains Fork-labeled items.","section":"Appendix C, Algorithm 1, get_solution"},{"comment":"The paper asserts that the generated solutions are 'optimal next-best moves', but Algorithm 1 is a heuristic and no minimax or exhaustive game-tree verification is provided. The fork test counts any two winning lines with n-1 current-player stones and no opponent stones; the printed pseudocode does not explicitly require the two threats to share the empty square played. In the reachable state space, after `get_wins` has returned no winning moves, this condition may be automatic, but the paper does not demonstrate this, and the control-flow error above makes the procedure unreproducible. Without an independent check that the Win/Blocked/Fork labels equal the game-theoretically optimal move sets, a model that selects a different winning or drawing move can be scored as wrong, so the reported Pass@1 gaps and the headline 41%/5% numbers could be artifacts. Please provide executable generator code, a minimax-based validation of all labels, and a clear statement of any disagreements.","section":"Appendix C, Algorithm 1, get_forks; Section 3.1"},{"comment":"The paper repeatedly claims that these games are trivial for humans ('humans can effortlessly solve from a young age', 'easy for humans to play even from a young age'), but no human evaluation is reported. This is load-bearing because the main conclusion contrasts model failures with human ease. Please include a small human baseline (even a few participants on a sample of questions) or explicitly reframe the triviality claim as an assumption rather than an established fact.","section":"Abstract; Section 1; Section 3"},{"comment":"Pass@1 is computed with k=16 responses for open models, while the frontier models (o3-mini-medium and DeepSeek-R1) are evaluated with a single response per TTT-Bench test sample and their math scores are taken from published results with unknown sampling protocols. This makes the head-to-head comparison in Table 2 and the statement that frontier models find TTT-Bench 'as difficult as the standard math benchmarks' not apples-to-apples. Please either rerun the frontier models under the same protocol or report the comparison with explicit caveats; confidence intervals for all reported Pass@1 values would also help assess whether the smaller AIME-2024 gaps (e.g., 5%) are meaningful.","section":"Section 4.1; Table 2"}],"minor_comments":[{"comment":"The quantity computed is the sample mean of correctness over k=16 responses, not the usual Pass@1 estimator; the name is misleading and no confidence intervals are reported.","section":"Section 4.1, Eq. (1)"},{"comment":"There are typos in the pseudocode ('Wining States' should be 'Winning States', 'wining moves' should be 'winning moves'), and the variables `W_current` and `W_opponent` are defined with notation that is hard to parse; a cleaner, runnable version would improve reproducibility.","section":"Appendix C, Algorithm 1"},{"comment":"The phrase 'Winning Fork' is defined informally and is used both to filter generated states and to define the Fork label; please align the informal definition with the exact condition implemented in Algorithm 1.","section":"Section 3.1"},{"comment":"The difficulty ordering 'oTTT<dTTT<sTTT<cTTT' is introduced with the < symbol but not defined; the claim that this ordering is 'consistent across' both MATH 500 and AIME 2024 comparisons would benefit from statistical support rather than visual inspection of Figure 3.","section":"Section 4.2"},{"comment":"The published math results for frontier models are cited only in footnotes; please include full references with access dates in the bibliography.","section":"Table 2 and footnotes"},{"comment":"The text refers to 'Appendix 1' when describing the solution-generation algorithm; the algorithm is actually in Appendix C.","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and the broad model evaluation are useful, but the internal inconsistency in Algorithm 1 and the absence of label verification are serious enough that the headline claim cannot be trusted as published. The authors should be asked to release the exact generator code, validate the labels with game-tree search, and add a human baseline. I do not see evidence of citation or novelty problems beyond minor incompleteness, and the topic fits the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful benchmark paper and the main phenomenon—strong math reasoners doing poorly on simple board-game moves—is likely real. But the quantitative gap is only as trustworthy as the answer-key generator, and right now that generator is neither verified nor even reproducible from the printed pseudo-code. I would send it to review, not reject it.\n\nWhat's new: the three game variants (dTTT, cTTT, sTTT) are novel, the generation pipeline is simple and scalable, and the benchmark is (as far as I can tell) contamination-resistant. The evaluation covers a wide range of open reasoning models plus two frontier models, and the relative ordering of game difficulty is consistent across models. The overthinking observation—longer traces with more circular reasoning on easy game states—is worth following up.\n\nWhere it's soft: the answer keys. Algorithm 1 in Appendix C defines 'Fork' as any move with two winning lines containing n-1 of the player's stones and no opponent stones, but it never checks whether both threats share a single empty square. If they don't share, the opponent can't block both at once, so the move is a real fork; if they do share, the move is not a fork and a model that plays the shared square should be counted correct but isn't. The pseudo-code also has a control-flow bug: get_solution returns None in the else branch of the blocked check, so get_forks is unreachable in the printed algorithm, even though Figure 2 and Table 4 report Fork verdicts. That's either a transcription error or a mismatch between the paper and the actual code—either way the benchmark cannot be regenerated from the paper alone. No minimax audit or human baseline is provided, so the 'trivial for humans' claim is asserted, not measured. The protocol mismatch for frontier models (k=1 for TTT-Bench, published numbers for math) and the absence of confidence intervals make the headline 41% drop looser than it looks.\n\nI think the direction of the effect is robust—the same pattern shows up across many models and four game types—but the magnitude and the 'fundamental gap' language are not earned yet. The heuristic could plausibly be fixed and verified with a small minimax solver, and a human baseline would take an afternoon. If the labels hold up, this is a solid benchmark; if they don't, the whole comparison needs redoing.\n\nThis paper deserves a serious referee. I'd recommend conditional acceptance with a mandatory audit of the answer-key generator and uniform evaluation protocol. I wouldn't cite it as evidence of a fundamental reasoning gap until that audit exists.","headline":"Useful benchmark, but the headline gap rests on an unverified answer-key heuristic that is also not reproducible from the printed pseudo-code.","tokens_in":19401,"tokens_out":3014,"would_cite":false,"duration_ms":36041,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large reasoning models that ace hard math benchmarks often struggle on simple Tic-Tac-Toe-style games, and this paper backs that claim with a new verifiable benchmark whose answer keys come from a Win/Blocked/Fork heuristic.","keywords":["Tic-Tac-Toe","large reasoning models","reasoning benchmark","strategic reasoning","spatial reasoning","game tree search","benchmark generation","chain-of-thought"],"falsifier":"Run an exhaustive minimax search over every board state in TTT-Bench and compare the set of moves it classifies as optimal — wins, or draws when no win exists — against the 'Win', 'Blocked', and 'Fork' labels produced by Algorithm 1. If any labeled move is not actually optimal under perfect play, or any unlabeled move is equally good, the answer keys and the measured accuracy gaps are not a faithful measure of reasoning ability.","tokens_in":18410,"feed_emoji":"🎮","tokens_out":7776,"duration_ms":80799,"temperature":0.7,"pith_summary":"TTT-Bench is a benchmark of four simple two-player games based on Tic-Tac-Toe, built to test whether large reasoning models that solve hard math problems can also handle the basic strategic, spatial, and logical reasoning that humans use effortlessly. The paper's central claim is that most of them cannot: across the evaluated models, Pass@1 on TTT-Bench is on average 41 points lower than on MATH 500 and about 5 points lower than on AIME 2024, with the biggest failures on novel game variants and on moves that require blocking an opponent or creating a fork. If this holds, current reasoning models have a broad gap in intuitive, long-horizon strategic thinking rather than a narrow math deficit. The paper also reports that models 'overthink' these trivial puzzles, generating long, repetitive reasoning traces that do not improve accuracy.","feed_headline":"AI that aces math flunks simple Tic-Tac-Toe","feed_subtitle":"Even top reasoning models miss obvious moves in four games humans find trivial.","key_machinery":"The load-bearing mechanism is the programmatic question generator and its scoring heuristic (Algorithm 1). Starting from all reachable game states after N moves, the pipeline keeps only states in which neither player has already won and the current player has not already established a 'winning fork' — two or more immediate winning lines one move short of completion. For each kept state, the heuristic labels the next best moves with one of three verdicts: 'Win' (a move that immediately wins), 'Blocked' (a move that is the sole way to stop the opponent's immediate win), or 'Fork' (a move creating two or more such threats, guaranteeing a win whatever the opponent does). This three-way taxonomy makes the 412 questions verifiable without full game-tree search, and the benchmark's validity rests on the assumption that these labels cover every optimal move.","core_discovery":"The central discovery is a disconnect between mathematical competence and game-strategic competence in large reasoning models. On the four TTT-Bench games — ordinary TTT, double TTT, cube TTT, and square TTT — models that excel on MATH 500 and AIME 2024 frequently fail to find the next best move, scoring on average 41% and 5% lower than on those math benchmarks. The gap is largest on the three novel games and on questions whose correct move is labeled 'Blocked' or 'Fork', which require anticipating the opponent or planning two moves ahead; immediate 'Win' moves are found far more reliably. On top of the accuracy drop, the models use as many or more thinking tokens on these simple puzzles as on olympiad-level math, often producing circular and repetitive reasoning instead of the short, direct solution a human would give.","pith_inferences":["One plausible explanation the paper leaves implicit is that reinforcement-learning training on formal, step-by-step math may be teaching a narrow prover-like skill rather than general competence; TTT-Bench could test this by fine-tuning a small model on Fork/Blocked cases and checking whether gains transfer to other spatial-strategy tasks.","Because the tasks are text-only, the gap could partly reflect mental-visualization load rather than strategy; giving models rendered images of the boards (a testable variant the paper itself notes) would separate these two causes.","The Fork category specifically requires holding two simultaneous threats in mind; a synthetic probe varying the number of simultaneous threats could test whether failures track working-memory-style load, which would make the deficit more mechanistic than 'strategic reasoning' implies."],"forward_implications":["If the claim is correct, current reasoning models cannot be trusted for tasks that combine spatial layout, opponent modeling, and multi-step planning, even when every individual step is elementary.","Math performance is not a reliable proxy for general reasoning: benchmark suites should include simple, novel, verifiable game states alongside STEM problems to expose this gap.","TTT-Bench's heuristic-based pipeline can generate unlimited fresh questions for any finite, deterministic two-player game, enabling contamination-free evaluation as models improve.","Long chain-of-thought is not helping on these tasks: models that produce shorter traces on TTT-Bench tend to score higher, suggesting overthinking is actively harmful for simple problems."],"supporting_citations":[{"why":"Defines MATH 500, the high-school math benchmark whose scores anchor the paper's comparison of LRM math accuracy against TTT-Bench.","marker":"Lightman et al., 2024"},{"why":"Defines AIME 2024, the olympiad-level math benchmark used as the second comparison point.","marker":"MAA, 2024"},{"why":"Supplies the DeepSeek-R1 model family (and the RLVR training paradigm) that makes up much of the evaluated model set.","marker":"DeepSeek-AI, 2025"},{"why":"A prior Tic-Tac-Toe LLM benchmark the authors cite as contaminated or known, motivating their need for novel uncontaminated games.","marker":"Topsakal and Harper, 2024"},{"why":"GameBench, a prior game-based strategic reasoning benchmark, used to position TTT-Bench's novelty.","marker":"Costarelli et al., 2024"}],"fun_headline_variants":["Math whiz AI tripped up by tic-tac-toe","Top reasoning models can't win at tic-tac-toe","When tic-tac-toe beats AI math champions","Simple games expose AI reasoning gaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The answer keys assume that the heuristic in Algorithm 1 labels every optimal next move, so any move outside its sets is scored wrong even if full game-tree search would show it wins or draws; if the heuristic misses optimal moves, the reported performance gap could be an artifact of the scoring rather than a genuine reasoning failure.","fun_headline_variants_meta":{"raw":{"variants":["Math whiz AI tripped up by tic-tac-toe","Top reasoning models can't win at tic-tac-toe","When tic-tac-toe beats AI math champions","Simple games expose AI reasoning gaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1413,"prompt_tokens":984,"completion_tokens":429,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":362}},"tokens_in":600,"tokens_out":429,"duration_ms":5747,"temperature":1.0,"reasoning_tokens":362,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:31:54.566344+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an exhaustive minimax search over every board state in TTT-Bench and compare the set of moves it classifies as optimal — wins, or draws when no win exists — against the 'Win', 'Blocked', and 'Fork' labels produced by Algorithm 1. If any labeled move is not actually optimal under perfect play, or any unlabeled move is equally good, the answer keys and the measured accuracy gaps are not a faithful measure of reasoning ability.","supporting_citations":[],"review_version":1}