{"id":"14506670-882b-4c07-9d6c-bbfdab1962e7","arxiv_id":"2505.04914","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper introduces enigme, a procedurally generated text-puzzle library for benchmarking reasoning in transformer-decoder language models.","lead":"This paper presents enigme, an open-source library that generates text-based reasoning puzzles in three classes, numeric, sequence, and physics, designed to test whether language models truly reason or simply match templates. A generalist might read it because it offers a fresh tool for the contested question of whether LLMs genuinely reason or are just memorizing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's load-bearing premise is unvalidated: template-generated ASCII puzzles may be solvable by a simple frame-diff heuristic, so the claim that they require 2D/3D world-model building is not yet supported.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the paper assumes the generated puzzles require the intended abstract reasoning without evidence. I agree. The package is real and the CLI produces puzzles with controlled ground truth, which supports the existence claim, but the paper's more substantive claim — that these tasks probe the boundary of transformer-decoder reasoning — requires construct validation. The proposed diff-based baseline test is a direct way to check whether the puzzles are actually hard for the reasons claimed. Since the reader already issued CONDITIONAL, this stress-test does not move the verdict; it reinforces the condition by making the required validation concrete.","tokens_in":6878,"tokens_out":2878,"duration_ms":32002,"concrete_test":"Implement a non-neural baseline for the sequence and physics classes: parse each puzzle's frames into character grids; for every adjacent pair, compute the changed cells and the displacement (dx,dy) that best aligns the changed region; classify the transition as translation, reflection, expansion/contraction, or boundary collision; then predict the next frame from the inferred rule. Run this baseline on 1000 generated instances per class and complexity level (1, 2, 3) using the enigme CLI, and compare predicted frames to the provided ground truth. If exact-match accuracy is materially above zero on the 2D and 3D variants, the assertion that these puzzles require non-sequential world-model building is not supported, and the benchmark's construct validity fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that enigme puzzles probe the boundary of transformer-decoder reasoning because they require 2D/3D visualization and world-model building — rests on an unvalidated construct assumption. The paper asserts in Section I-A that 'pure sequential analysis makes the task very difficult' and that large variation spaces make memorization difficult, but it reports no experiments with humans or models. Because the generation process is template-based (Section II-D), shortcut solving is plausible. In the sequence and physics classes, each puzzle appears as a series of ASCII frames whose hidden transformation belongs to a small family of simple operations: translation, reflection, scaling, or boundary interaction. A solver that never builds a world model can parse consecutive frames as grids, compute coordinate-wise character diffs, and extrapolate the observed local motion. Such a diff-based heuristic would exploit exactly the kind of surface pattern continuity the paper claims to rule out. The enormous variation estimates in Table I do not block this objection, because the variation is in surface tokens while the underlying rule family remains small. Without a baseline that controls for these template/surface-similarity shortcuts, the load-bearing premise that the puzzles require the intended abstract reasoning is unsecured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces enigme, an open-source Python library for procedurally generating text-based reasoning puzzles in three classes (numeric, sequence, and physics). The motivation is that transformer-decoder language models encode latent structure through token interactions and positional embeddings, so puzzles requiring 2D/3D visualization, world-model building, and intuitive physics should probe the boundary of their reasoning abilities. The paper describes the generation processes, provides three example puzzles, reports estimated variation counts in Table I, and argues that the large variation space makes memorization difficult, but it contains no empirical measurements with models or humans.","tokens_in":7097,"tokens_out":3312,"duration_ms":34607,"significance":"If its construct-validity claims were established, enigme could be a useful lightweight benchmark generator: it is open-source, parameterized, and designed around plausible hypotheses about transformer-decoder limitations. The paper also usefully draws attention to the distinction between token-sequence matching and spatial/world-model reasoning. However, the current contribution is a design hypothesis rather than a validated evaluation tool, because the load-bearing claims about task difficulty and the impossibility of shortcut solving are asserted without empirical support.","major_comments":[{"comment":"The central claim that \"pure sequential analysis makes the task very difficult\" and that the large variation space makes memorization difficult is asserted without evidence. The manuscript reports no experiments with humans or models and no comparison to surface-level baselines. Since the generation is template-based (Section II-D), a solver that computes coordinate-wise character diffs across frames could plausibly solve many sequence and physics puzzles without building a world model, so the load-bearing premise is unsecured. The authors should add baseline experiments with simple diff-based or sequence-extrapolation heuristics and at least one LLM evaluation to support or qualify the claim.","section":"Section I-A"},{"comment":"The variation counts are presented as estimates, but no derivation, formula, or reproducible calculation is provided. The text says the estimates use the \"largest possible configuration\" and form a \"good lower bound approximation,\" but without the parameter ranges and counting method the reader cannot verify the memorization-difficulty argument. Please include the exact counting procedure or a script that computes these numbers.","section":"Table I"},{"comment":"The paper states that the generation process allows \"programmatic control over the accuracy of the solutions,\" but it never demonstrates that every generated puzzle has a unique correct answer or that the answer can be automatically verified. A benchmark for evaluating reasoning must guarantee answer determinism. Please describe the verifier, report its behavior on a sample of generated instances, and specify the exact answer format expected from a model.","section":"Section II-D"},{"comment":"The sequence and physics puzzle classes are said to require abductive reasoning and naive physics, but the hidden transformations appear to belong to a small family of simple operations (translation, reflection, scaling, boundary interaction). The token-level variation estimates in Table I do not establish that the underlying rule family is hard to infer from surface patterns. Please provide a formal characterization of the transformation family and empirical evidence that the intended rules are recoverable only through the intended abstract reasoning, rather than through template or frame-diff shortcuts.","section":"Sections II-B and II-C"}],"minor_comments":[{"comment":"There are several typos: \"visuualisation\" and \"probelm\" in Section I-A, \"dimenions\" in Section II-B, and \"bare some similarity\" should be \"bear some similarity.\"","section":"Throughout"},{"comment":"The example figures are not accompanied by the expected answer or a brief explanation of the rule, which makes it difficult for a reader to understand what the puzzle is testing. Adding the gold answer and rule description for each example would improve clarity.","section":"Figures 2-4"},{"comment":"Reference [7] points to a social media post; this is not an appropriate citable source for a methodological claim about data contamination, and it should be replaced with a published study or preprint with verifiable methodology.","section":"References"},{"comment":"The availability statement gives a GitHub link, but no version number or installation/test instructions; please include a versioned release and a minimal code snippet that reproduces the examples in Figures 2-4.","section":"Section IV"}],"recommendation":"major_revision","confidential_remarks":"The paper is best understood as a tool/dataset proposal rather than a completed empirical study. The main risk is the absence of any validation of the puzzles' construct validity, which I consider load-bearing for the stated purpose of evaluating reasoning. The authors can address this by adding baseline experiments, verifier checks, and a derivation of the variation counts; these are concrete and within the scope of a revision. If the journal has a policy that benchmark/tool papers must include empirical evaluation, then the paper is closer to rejection, but as written I would invite a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"John, quick take on 2505.04914. It's a tool paper: an open-source library that generates three families of ASCII text puzzles (numeric, sequence, physics) intended to test reasoning in transformer-decoder LLMs. The code is real and on PyPI/GitHub, and the examples in Figures 2–4 look like what you'd expect. The new bit is not procedural generation of abstract reasoning tasks per se; ARC and others do that. It's the text-only, 2D/3D ASCII format with physics-flavored sequences, aimed at the contamination problem. That's a legitimate niche.\n\nWhat the paper does well: it names a real problem (memorization/contamination in reasoning benchmarks), connects it to the architecture (positional embeddings making sequential processing natural), and proposes a concrete, parameterized generator. The three classes are distinct, the complexity/dimension parameter is simple, and the variation counts in Table I, while asserted without derivation, are plausible for template-based generation. Shipping code counts for something.\n\nSoft spots, in order of size. First, there are zero experiments. No human solvability checks, no model evaluations, no uniqueness verification. The load-bearing premise—that these puzzles 'should probe' the boundary of transformer reasoning because they require 2D/3D visualization and world-model building—is a hypothesis. The stress-test note about a frame-diff heuristic is fair: a solver could parse consecutive ASCII frames, compute coordinate-wise diffs, and extrapolate motion for translation/reflection/scaling without building any world model. That wouldn't solve every physics puzzle, but it's a plausible shortcut for a large chunk, and the paper gives no baseline to argue otherwise. Second, the variation counts don't block shortcut solving; variation is in surface tokens, but the rule family is small. Third, the 'latent variable structure' argument in Section I-A is intuition, not analysis—fine as motivation, but it's presented as design rationale for why tasks should be hard, and that framing overreaches.\n\nI think the reader's conditional verdict is right. The paper deserves a serious referee, but any referee should demand at least: (1) human performance on a sample, (2) a few LLMs with and without simple baselines (frame-diff, template-match), and (3) uniqueness checks on generated puzzles. If those come back, this could be a useful benchmark. As is, it's a promising scaffold with an unvalidated headline claim.\n\nFor you: worth a glance if you care about LLM reasoning eval or contamination; I wouldn't cite it yet. Send it to review? Yes, as a tool/resource paper—with the expectation of major revision. Serious thinker: yes; the writing is clear and the limitations are not hidden, even if the claims outrun the evidence.","headline":"A real, open-source puzzle generator with an unvalidated central claim: no experiments, so the 'probes reasoning' premise is a hypothesis.","tokens_in":7581,"tokens_out":2294,"would_cite":false,"duration_ms":23404,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that transformer-decoder language models can be tested at the edge of their reasoning ability by puzzles whose solutions require interpreting ASCII text as two- and three-dimensional structure rather than a linear token…","keywords":["language models","reasoning evaluation","text puzzles","procedural generation","abstract reasoning","world models","benchmark data","transformer-decoder"],"falsifier":"Take a fixed batch of Enigme puzzles and compare a transformer-decoder model's accuracy on the original ASCII layouts against versions with line breaks removed and all characters flattened into one long sequence; if accuracy does not drop, the 2D/3D structure is not load-bearing. Also check whether a simple positional baseline that counts symbol positions can solve the numeric puzzles without any world-model reasoning.","tokens_in":6679,"feed_emoji":"🧩","tokens_out":7645,"duration_ms":64665,"temperature":0.7,"pith_summary":"The paper argues that transformer-decoder language models, because they process tokens sequentially with positional embeddings, are structurally biased toward template matching rather than genuine reasoning. To test that boundary, it introduces Enigme, an open-source library that procedurally generates text-based puzzles in three classes—numeric, sequence, and physics—whose solutions require the model to treat ASCII text as 2D layouts or 3D slices rather than as a linear token stream. The puzzles are designed so that pure sequential analysis is very difficult and the parameter space of variations is so large that memorization becomes impractical. A sympathetic reader would care because the library offers a concrete way to probe whether LLMs can build world models from impoverished symbolic data, an ability the paper argues is central to general-purpose reasoning.","feed_headline":"Text puzzles aim to probe the reasoning boundary of language models","feed_subtitle":"Enigme generates numeric, sequence, and physics puzzles whose solutions need 2D and 3D visualization.","key_machinery":"The central object is the puzzle-generation engine, a template-based substitution system with random parameters that produces ASCII puzzles in three classes: numeric, sequence, and physics. The numeric class embeds substitution patterns in an instruction block that require tracking word position, character position, and alphabet position; the sequence class builds multidimensional grids of background and foreground characters and then modifies them across frames; the physics class simulates simple movement, momentum, and collision. The engine provides programmatic control over a dimension/complexity parameter and yields estimated variation counts, which the paper uses to argue that memorization of specific instances is impractical.","core_discovery":"The central claim is that reasoning tasks expressed as linear text are too easily solved by template matching in the latent variable space of transformers, so the paper constructs puzzles that force spatial and physical interpretation of ASCII characters. The puzzles require the solver to infer patterns across 2D grids, track object motion across text frames, or perform self-referential numeric substitution; the author hypothesizes these demands sit at the boundary of what transformer-decoders can do. The paper offers no experimental results; its discovery is a generative methodology plus the hypothesis that these puzzles constitute a harder, more diagnostic test of reasoning than conventional logical-reasoning benchmarks.","pith_inferences":["A concrete testable extension is to fine-tune a model on Enigme puzzles and measure transfer to other abstract reasoning tasks; the paper does not claim such transfer, but if it occurs, Enigme would double as a training curriculum.","Flattening the ASCII layout into a pure token sequence, as suggested in the falsifier, would also reveal whether the 2D failure is caused by tokenization of line breaks rather than by a true spatial reasoning deficit, which could motivate 2D positional encodings.","The same puzzle structures could be rendered as images to create an ARC-style visual benchmark, although the paper deliberately keeps everything text-only to reduce resource demands.","If strong models solve these puzzles easily, that would push the hypothesized reasoning boundary elsewhere, for instance toward causal or temporal reasoning rather than spatial inference."],"forward_implications":["If the puzzles indeed require 2D and 3D visualization, transformer-decoder models should show measurably lower accuracy on Enigme than on standard logical reasoning benchmarks, directly supporting the template-matching critique.","The very large variation counts (up to $10^{33}$ for sequence puzzles in Table I) mean a model that succeeds on these tasks cannot easily be credited with memorizing training instances.","The dimension/complexity parameter gives researchers a controlled dial for puzzle difficulty, allowing systematic measurement of where reasoning breaks down.","Because the three puzzle classes target numeric self-reference, abductive sequence inference, and naive physics, the library can separate which reasoning modes a model lacks.","The open-source generator lets researchers produce fresh puzzle instances on demand, reducing the risk of benchmark contamination."],"supporting_citations":[{"why":"Frames benchmark design as measuring methodology rather than mere performance, motivating Enigme's existence.","marker":"[10]"},{"why":"Provides the token-bias evidence that LLMs are template matchers, the hypothesis Enigme is built to test.","marker":"[24]"},{"why":"Shows that superficial variation in symbolic math degrades performance, supporting the template-matching view.","marker":"[25]"},{"why":"Demonstrates premise-order sensitivity in logical reasoning, cited as evidence of template matching rather than reasoning.","marker":"[26]"},{"why":"Establishes that language models perform poorly on text-based abstract reasoning, the gap Enigme targets.","marker":"[27]"},{"why":"Introduces evaluation of world knowledge in LMs, which Enigme extends to world-model-building puzzles.","marker":"[8]"},{"why":"Supplies the inductive-bias concept used to justify reasoning from few examples.","marker":"[28]"}],"fun_headline_variants":["Enigme: text puzzles push LMs to visualize 2D and 3D space","New library generates puzzles that force spatial reasoning in AI","Probing LLM reasoning with puzzles that require 3D mental models","Enigme: puzzles designed to break template matching in transformers","Generate spatial puzzles to test language models beyond text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the generated puzzles genuinely require spatial and physical world-model building and cannot be solved by simpler statistical regularities or template matching; the paper asserts this but does not provide evidence that the puzzles are not crackable by heuristics.","fun_headline_variants_meta":{"raw":{"variants":["Enigme: text puzzles push LMs to visualize 2D and 3D space","New library generates puzzles that force spatial reasoning in AI","Probing LLM reasoning with puzzles that require 3D mental models","Enigme: puzzles designed to break template matching in transformers","Generate spatial puzzles to test language models beyond text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000295,"raw_usage":{"total_tokens":1635,"prompt_tokens":790,"completion_tokens":845,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":406,"completion_tokens_details":{"reasoning_tokens":755}},"tokens_in":406,"tokens_out":845,"duration_ms":7498,"temperature":1.0,"reasoning_tokens":755,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:17:13.591347+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed batch of Enigme puzzles and compare a transformer-decoder model's accuracy on the original ASCII layouts against versions with line breaks removed and all characters flattened into one long sequence; if accuracy does not drop, the 2D/3D structure is not load-bearing. Also check whether a simple positional baseline that counts symbol positions can solve the numeric puzzles without any world-model reasoning.","supporting_citations":[],"review_version":1}