{"id":"84d2b891-4f55-4cb7-942a-b2b99266231d","arxiv_id":"2507.07313","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Frontier thinking models degrade sharply on procedurally scaled 'easy' reasoning tasks and fail on curated trivialized puzzles, often by regurgitating the original puzzle's solution.","lead":"This paper tests frontier AI models, including reasoning models, on simple but tedious tasks like counting, logic, and travel planning, and finds they often fail as the tasks get longer. It also introduces a new benchmark of deliberately trivialized versions of famous puzzles, where models sometimes repeat the answer to the original hard puzzle instead of the easy new answer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unpuzzles are curated to elicit the failures they then report; the headline performance gaps are partly built into the selection procedure.","rationale":"The paper's central contribution beyond previous work is the Unpuzzles benchmark and the claim that making tasks easier can degrade performance. The procedural tasks in Section 3 are valuable and, despite small sample sizes, the appendix error bars support a qualitative scaling failure for thinking and non-thinking models. The Unpuzzles claim, however, rests on a dataset that was constructed by selecting exactly the cases where large models fail on trivialized puzzles (Appendix C.1) and where the generating model solves the context-shifted controls (Appendix C.2). The reader's weakest_assumption identifies precisely this selection bias. I agree with that assessment. The concern is load-bearing because the abstract's 'making tasks easier does not necessarily imply improved performance' is the headline novelty, and the 'delirium' interpretation used to attribute failures to memorization depends on the filtered CS comparison. If the filters were removed and the gaps disappeared, the paper's broadest conclusion would reduce to a narrower existence result. I do not recommend rejection: the procedural results independently demonstrate that frontier thinking models degrade on long, self-contained easy tasks, and the Unpuzzles remain valid as a curated stress-test set, provided the paper clearly frames the gaps as properties of the benchmark's selection procedure rather than unbiased estimates of model behavior. The reader's CONDITIONAL verdict already requires disclosing the selection rules in the main text, so my assessment does not change the outcome; I would only emphasize that the disclosure should be accompanied by an explicit statement that the dataset was intentionally selected to elicit the reported failures.","tokens_in":36888,"tokens_out":5560,"duration_ms":56604,"concrete_test":"Generate a new batch of unpuzzles following the same minimal-edit instructions but without the filter in Appendix C.1 (do not check whether any model reproduces the original solution; keep every reasonable minimal edit). Similarly, generate context-shifted versions without the solvability filter in Appendix C.2 (keep all rewrites that preserve logical structure, regardless of whether the generating model answers correctly). Evaluate the same frontier models on both unfiltered sets and compare the puzzle/unpuzzle and unpuzzle/CS gaps to Tables 1 and 3. If the gaps shrink substantially (e.g., by more than half), the Unpuzzles results are largely selection artifacts; if the gaps persist, the concern is mitigated. The released repository already provides prompts and code, so this is a feasible reproducibility check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the Unpuzzles benchmark's headline results are partly guaranteed by its construction. Appendix C.1 instructs curators to 'Check that large models still use the original solution to erroneously solve the modified puzzle or give the original (incorrect) answer. If not, repeat from step 3' — so an unpuzzle is retained only if a large model exhibits the failure the paper then reports. The puzzle-vs-unpuzzle gap in Table 1 and the 'delirium' rates in Table 2 therefore reflect the selection rule as much as model behavior. Appendix C.2 applies a second filter: context-shifted unpuzzles are kept only when 'We query the same model with the new unpuzzle; if the correct answer is not returned, return to step 1' — i.e., the generating model had to solve the shifted version. That biases the CS-vs-unpuzzle comparison in Table 3 toward the paper's conclusion that failures come from memorization rather than lack of ability. The appendix discloses these rules, but the abstract and main text present the phenomenon as a general property of frontier models and use it to support 'making tasks easier does not necessarily imply improved performance.' Because the dataset is explicitly curated to maximize the failure and the control is filtered for solvability, the quantitative gaps are not unbiased evidence about how models handle trivialized versions of familiar puzzles; they are evidence that such failures can be induced for a hand-picked set. This does not destroy the existence claim, but it is the weakest load-bearing point in the strongest interpretation of the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies frontier LLMs on simple reasoning tasks. The first half introduces procedurally generated tasks (character/word counting, first-order logic evaluation and negation, proof-tree math word problems, and travel planning) with tunable parameters that increase the amount of computation but not the conceptual difficulty. Across these tasks, the authors report that both traditional and thinking models degrade as the parameters grow, and they attribute the failures to a taxonomy of causes including accumulation of errors, long-context difficulty, statistical shortcuts, poor state tracking, out-of-distribution generalization, and tokenization. The second half introduces the Unpuzzles dataset: 97 well-known puzzles with manually trivialized versions, plus a subset of 64 context-shifted unpuzzles. The paper reports that models perform much better on the original puzzles than on the unpuzzles, that many unpuzzle solutions contain context corruption, and that all tested models perform better on the context-shifted unpuzzles than on the corresponding unpuzzles. These observations are interpreted as evidence that frontier models rely on memorization of the original puzzle text rather than on the logical structure of the problem, and the paper concludes that making tasks easier does not necessarily improve performance.","tokens_in":37055,"tokens_out":9647,"duration_ms":108938,"significance":"If the claims are taken at face value, the paper is a useful contribution: it provides a broad, reproducible set of procedural tasks that can be re-generated at arbitrary scale, a publicly released dataset of puzzles and unpuzzles, careful manual annotation of failure modes, and a concrete failure taxonomy for thinking models. The juxtaposition of puzzle, unpuzzle, and context-shifted unpuzzle is a clever experimental design for probing memorization. The paper also deserves credit for discussing and partially automating evaluation, for reporting confidence intervals in the appendix, and for acknowledging the closed-source limitation. However, the significance of the Unpuzzles results is substantially weakened by the dataset construction rules in Appendix C, which select items precisely because they induce the reported failures; the headline numbers in Tables 1 and 2 are therefore partly a property of the curation process rather than an unbiased measurement of model behavior on arbitrary trivializations.","major_comments":[{"comment":"Please verify the comment text is complete.","section":"Appendix C.1 and Section 4.2"},{"comment":"Please verify the comment text is complete.","section":"Appendix C.2 and Table 3"},{"comment":"Please verify the comment text is complete.","section":"Section 3.2 and Appendix B"},{"comment":"Please verify the comment text is complete.","section":"Section 3.2, Figure 5 vs. Appendix B, Figure 13"}],"minor_comments":[{"comment":"Please verify the comment text is complete.","section":"Abstract"},{"comment":"Please verify the comment text is complete.","section":"Section 5"},{"comment":"Please verify the comment text is complete.","section":"Section 4.2"},{"comment":"Please verify the comment text is complete.","section":"Section 1"},{"comment":"Please verify the comment text is complete.","section":"Appendix C.2"},{"comment":"Please verify the comment text is complete.","section":"Figure 2 and Appendix C.3"}],"recommendation":"major_revision","confidential_remarks":"The central risk for the editor is the gap between the main-text generalization and the curated nature of the Unpuzzles dataset. The selection rules are disclosed in the appendix, which is good practice, but the abstract and Section 4.2 do not carry the caveat, and the headline numbers in Tables 1-3 are likely to be overinterpreted. The travel-planning inconsistency between Figure 5 and Appendix B should be resolved before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The procedural part of this paper is genuinely useful: it tests several frontier models, including thinking variants, on four families of easy but tedious tasks and shows that accuracy drops as parameters grow. That is not new in kind—Opedal et al. and Shojaee et al. did similar things—but the breadth is real, and the failure-mode taxonomy (accumulated errors, statistical shortcuts, long-context problems, poor state tracking) is well organized and backed by concrete examples. The released datasets and code are a plus.\n\nThe Unpuzzles idea is clever, and the context-shifted control is a nice way to test memorization. The problem is that the headline gap is partly guaranteed by construction. Appendix C.1 tells curators to keep a trivialized puzzle only if large models still reproduce the original solution or answer; Appendix C.2 filters context-shifted unpuzzles so that the generating model must solve them. So Tables 1-3 measure a dataset selected to produce the phenomenon. That does not destroy the existence claim—it does show such failures can be induced—but the abstract's claim that 'making tasks easier does not necessarily imply improved performance' is too strong for this evidence. The gap is a property of the curated sample, not of the distribution of all trivialized puzzles.\n\nTwo smaller issues. First, the procedural results rest on 20 random problems per cell with wide confidence intervals; the appendix shows them, but the main text omits error bars for several tables, which overstates precision. Second, the novelty claim about thinking models is inaccurate: Shojaee et al. already evaluated thinking models on complexity-controlled puzzles, and the paper's dismissal of that work as flawed does not make the claim 'no earlier papers demonstrated' true.\n\nWho is this for? Anyone building or using LLM evaluation suites. The Unpuzzles dataset, despite its selection bias, is a useful resource if the selection rules are respected in interpretation. The paper deserves a serious referee; I would send it out with a request for major revision: move the selection rules to the main text, show error bars throughout, and soften the claims so they track what the data actually support.","headline":"Useful broad evaluation of thinking models on easy procedural tasks, but the Unpuzzles headline result is partly built into the benchmark's own selection rules.","tokens_in":37685,"tokens_out":2501,"would_cite":true,"duration_ms":29187,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Frontier language models, including thinking models, consistently fail on easy reasoning tasks as the tasks are made longer or more tedious, and they perform worse on trivialized versions of well-known puzzles than on the puzzles…","keywords":["LLM reasoning","thinking models","procedurally generated tasks","Unpuzzles","reasoning delirium","out-of-distribution generalization","memorization","benchmark saturation"],"falsifier":"Take a fresh set of 50 unpuzzles created from the same puzzle list using the same trivialization instructions but without the curation rule that requires a large model to reproduce the original solution; if frontier models solve these at near-perfect accuracy, the unpuzzle gap would be an artifact of the selection process rather than a general property of easier problems.","tokens_in":36609,"feed_emoji":"🧩","tokens_out":7518,"duration_ms":68150,"temperature":0.7,"pith_summary":"This paper sets out to show that the most capable language models, including the newer 'thinking' models that devote extra computation to reasoning, still break down on problems that are easy for humans. The authors build four procedurally generated task families—word and character counting, first-order logic evaluation and negation, proof-tree math word problems, and travel planning—with parameters that inflate the amount of work needed without raising the underlying difficulty. Across them, all tested models lose accuracy as the parameters grow, failing through statistical shortcuts, arithmetic slips in intermediate steps, and poor handling of long contexts. On a new human-curated dataset called Unpuzzles, models answer trivialized versions of famous puzzles far worse than the originals, often reproducing the original puzzle's solution verbatim, a failure the paper names 'reasoning delirium.' The paper's conclusion is that out-of-distribution generalization remains unreliable even for frontier thinking models, and that making a task easier does not guarantee better performance.","feed_headline":"Thinking models fail easy reasoning as tasks get longer","feed_subtitle":"New benchmarks show even top reasoning models stumble on counting, logic, and trivialized puzzles.","key_machinery":"Two constructions carry the argument. The first is a suite of procedurally generated tasks whose parameters (paragraph length, formula depth, number of cities, number of irrelevant agents and sentences) increase the required computation and working memory without increasing the conceptual difficulty; this lets the authors scale tasks arbitrarily and observe where accuracy collapses. The second is the Unpuzzles dataset, built by making minimal textual edits to 97 well-known puzzles so the answer becomes obvious, together with a 64-item context-shifted subset that preserves the logic but changes the setting; comparing performance on puzzles, unpuzzles, and context-shifted unpuzzles isolates the role of memorized wording, and the observed reproduction of original solutions is termed 'reasoning delirium.'","core_discovery":"The paper's central claim is that frontier LLMs, including reasoning-optimized thinking models, consistently fail on simple reasoning tasks once the tasks are scaled in 'tediousness'—more words to count, deeper logic trees, more cities to plan through, irrelevant sentences to ignore—while the conceptual difficulty stays constant. The authors further claim that on the Unpuzzles benchmark, all tested models do substantially worse on trivialized versions of well-known puzzles than on the original puzzles, and that the failures are often not random: the models import whole reasoning steps from the original puzzle's solution (context corruption), sometimes to the point of answering the original puzzle's question instead of the modified one. Because every model performs better on context-shifted versions of the same trivialized problems, the paper argues the poor performance is driven by memorized associations with the specific wording of famous puzzles, not by an inability to reason about the simple logic itself.","pith_inferences":["One testable extension is an automated unpuzzle generator that starts from any internet-famous puzzle and checks whether a model reproduces the original solution; the paper's 75% minimal-edit success rate in building context-shifted unpuzzles suggests such a pipeline is feasible.","If the failures are driven by training-data memorization, then models trained with stricter deduplication of commonly repeated puzzle texts, or prompted to explicitly flag that the task has been altered, could shrink the unpuzzle gap.","The parameter-scaling results suggest that 'thinking' compute budgets improve within-distribution accuracy but do not confer robustness to longer unfamiliar inputs; architectures with external memory or verifiable intermediate states may be needed.","The four-task procedural suite and the Unpuzzles measure different failure sources—computational scaling versus memorization—so combining them gives a more complete picture of a model's reasoning robustness than either alone."],"forward_implications":["Evaluation of reasoning models should include the simplest problems a model struggles with, not only the hardest it can solve.","The performance degradation with scale applies to thinking models in the same qualitative way as to non-thinking models, so extra inference compute does not remove these failure modes.","The Unpuzzles gap implies that high scores on well-known puzzles overstate reasoning ability, since the same models fail when the puzzle is trivially altered.","The procedurally generated task suite offers a refreshable benchmark that can be re-sampled to resist saturation as future models improve.","Making a task easier can degrade performance when the easier version is textually close to a memorized harder version, so 'easier' is not a reliable predictor of better model behavior."],"supporting_citations":[{"why":"Supplies MathGAP, the proof-tree generation framework that the paper extends with new logical forms, non-commutative statements, and irrelevant information.","marker":"Opedal et al. (2025)"},{"why":"Shows that perturbing GSM8K numeric values and adding irrelevant information drops accuracy; the paper contrasts its own approach of decreasing difficulty rather than perturbing at constant difficulty.","marker":"Mirzadeh et al. (2024)"},{"why":"Provides evidence that LLM accuracy tracks the likelihood of task formulations in training data, which underlies the paper's memorization explanation for unpuzzle failures.","marker":"McCoy et al. (2024)"},{"why":"Introduces a benchmark of 30 easy problems LLMs fail on; the paper's puzzle evaluation is positioned as more comprehensive.","marker":"Williams and Huckle (2024)"},{"why":"Concurrent work showing thinking LLMs collapse on puzzles with programmable complexity; the paper compares its own qualitative accuracy decline with this result.","marker":"Shojaee et al. (2025)"},{"why":"Supplies the travel planning benchmark on which the paper's travel planning task is based.","marker":"Xie et al. (2024)"},{"why":"Establishes that LLMs are far from saturating planning benchmarks, motivating the travel planning evaluation.","marker":"Valmeekam et al. (2024a,b)"},{"why":"Documents limits of transformers on compositionality and multi-step reasoning, one of the failure categories the paper measures.","marker":"Dziri et al. (2024)"}],"fun_headline_variants":["Easy reasoning trips up frontier models as tasks lengthen","Trivial puzzle versions fool LLMs that ace the originals","Scaling task size makes easy logic hard for thinking models","Memorization, not reasoning, explains LLM failures on easy tasks","Frontier LLMs fail easy logic when trivial puzzles are modified"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Unpuzzles dataset was built by hand-selecting only trivialized puzzles on which a large model demonstrably reproduces the original solution or answer, and the context-shifted versions were kept only when the generating model solved them, so the reported performance gaps are partly a property of how the benchmark was curated rather than a random sample of easy problems.","fun_headline_variants_meta":{"raw":{"variants":["Easy reasoning trips up frontier models as tasks lengthen","Trivial puzzle versions fool LLMs that ace the originals","Scaling task size makes easy logic hard for thinking models","Memorization, not reasoning, explains LLM failures on easy tasks","Frontier LLMs fail easy logic when trivial puzzles are modified"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000436,"raw_usage":{"total_tokens":2254,"prompt_tokens":1016,"completion_tokens":1238,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":1153}},"tokens_in":632,"tokens_out":1238,"duration_ms":13300,"temperature":1.0,"reasoning_tokens":1153,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:43:28.898170+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh set of 50 unpuzzles created from the same puzzle list using the same trivialization instructions but without the curation rule that requires a large model to reproduce the original solution; if frontier models solve these at near-perfect accuracy, the unpuzzle gap would be an artifact of the selection process rather than a general property of easier problems.","supporting_citations":[],"review_version":1}