Enigme: Generative Text Puzzles for Evaluating Reasoning in Language Models

John Hawkins

REVIEW 4 major objections 4 minor 29 references

Enigme: Generative Text Puzzles for Evaluating Reasoning in Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that transformer-decoder language models can be tested at the edge of their reasoning ability by puzzles whose solutions require interpreting ASCII text as two- and three-dimensional structure rather than a linear token…

desk verdict A real, open-source puzzle generator with an unvalidated central claim: no experiments, so the 'probes reasoning' premise is a hypothesis. read the letter →

arxiv 2505.04914 v1 pith:O3UNT6VJ submitted 2025-05-08 cs.AI cs.CL

John Hawkins This is my paper · ORCID

classification cs.AIcs.CL

keywords languagemodelsreasoningevaluationtextpuzzlesproceduralgenerationabstractworldbenchmarkdatatransformer-decoder

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that transformer-decoder language models, because they process tokens sequentially with positional embeddings, are structurally biased toward template matching rather than genuine reasoning. To test that boundary, it introduces Enigme, an open-source library that procedurally generates text-based puzzles in three classes—numeric, sequence, and physics—whose solutions require the model to treat ASCII text as 2D layouts or 3D slices rather than as a linear token stream. The puzzles are designed so that pure sequential analysis is very difficult and the parameter space of variations is so large that memorization becomes impractical. A sympathetic reader would care because the library offers a concrete way to probe whether LLMs can build world models from impoverished symbolic data, an ability the paper argues is central to general-purpose reasoning.

What carries the argument

The central object is the puzzle-generation engine, a template-based substitution system with random parameters that produces ASCII puzzles in three classes: numeric, sequence, and physics. The numeric class embeds substitution patterns in an instruction block that require tracking word position, character position, and alphabet position; the sequence class builds multidimensional grids of background and foreground characters and then modifies them across frames; the physics class simulates simple movement, momentum, and collision. The engine provides programmatic control over a dimension/complexity parameter and yields estimated variation counts, which the paper uses to argue that memorization of specific instances is impractical.

What would settle it

Take a fixed batch of Enigme puzzles and compare a transformer-decoder model's accuracy on the original ASCII layouts against versions with line breaks removed and all characters flattened into one long sequence; if accuracy does not drop, the 2D/3D structure is not load-bearing. Also check whether a simple positional baseline that counts symbol positions can solve the numeric puzzles without any world-model reasoning.

Watch

Extended reading notes

Core claim

The central claim is that reasoning tasks expressed as linear text are too easily solved by template matching in the latent variable space of transformers, so the paper constructs puzzles that force spatial and physical interpretation of ASCII characters. The puzzles require the solver to infer patterns across 2D grids, track object motion across text frames, or perform self-referential numeric substitution; the author hypothesizes these demands sit at the boundary of what transformer-decoders can do. The paper offers no experimental results; its discovery is a generative methodology plus the hypothesis that these puzzles constitute a harder, more diagnostic test of reasoning than conventional logical-reasoning benchmarks.

Load-bearing premise

The load-bearing premise is that the generated puzzles genuinely require spatial and physical world-model building and cannot be solved by simpler statistical regularities or template matching; the paper asserts this but does not provide evidence that the puzzles are not crackable by heuristics.

Editorial extensions

If this is right

If the puzzles indeed require 2D and 3D visualization, transformer-decoder models should show measurably lower accuracy on Enigme than on standard logical reasoning benchmarks, directly supporting the template-matching critique.
The very large variation counts (up to $10^{33}$ for sequence puzzles in Table I) mean a model that succeeds on these tasks cannot easily be credited with memorizing training instances.
The dimension/complexity parameter gives researchers a controlled dial for puzzle difficulty, allowing systematic measurement of where reasoning breaks down.
Because the three puzzle classes target numeric self-reference, abductive sequence inference, and naive physics, the library can separate which reasoning modes a model lacks.
The open-source generator lets researchers produce fresh puzzle instances on demand, reducing the risk of benchmark contamination.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A concrete testable extension is to fine-tune a model on Enigme puzzles and measure transfer to other abstract reasoning tasks; the paper does not claim such transfer, but if it occurs, Enigme would double as a training curriculum.
Flattening the ASCII layout into a pure token sequence, as suggested in the falsifier, would also reveal whether the 2D failure is caused by tokenization of line breaks rather than by a true spatial reasoning deficit, which could motivate 2D positional encodings.
The same puzzle structures could be rendered as images to create an ARC-style visual benchmark, although the paper deliberately keeps everything text-only to reduce resource demands.
If strong models solve these puzzles easily, that would push the hypothesized reasoning boundary elsewhere, for instance toward causal or temporal reasoning rather than spatial inference.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

John, quick take on 2505.04914. It's a tool paper: an open-source library that generates three families of ASCII text puzzles (numeric, sequence, physics) intended to test reasoning in transformer-decoder LLMs. The code is real and on PyPI/GitHub, and the examples in Figures 2–4 look like what you'd expect. The new bit is not procedural generation of abstract reasoning tasks per se; ARC and others do that. It's the text-only, 2D/3D ASCII format with physics-flavored sequences, aimed at the contamination problem. That's a legitimate niche.

What the paper does well: it names a real problem (memorization/contamination in reasoning benchmarks), connects it to the architecture (positional embeddings making sequential processing natural), and proposes a concrete, parameterized generator. The three classes are distinct, the complexity/dimension parameter is simple, and the variation counts in Table I, while asserted without derivation, are plausible for template-based generation. Shipping code counts for something.

Soft spots, in order of size. First, there are zero experiments. No human solvability checks, no model evaluations, no uniqueness verification. The load-bearing premise—that these puzzles 'should probe' the boundary of transformer reasoning because they require 2D/3D visualization and world-model building—is a hypothesis. The stress-test note about a frame-diff heuristic is fair: a solver could parse consecutive ASCII frames, compute coordinate-wise diffs, and extrapolate motion for translation/reflection/scaling without building any world model. That wouldn't solve every physics puzzle, but it's a plausible shortcut for a large chunk, and the paper gives no baseline to argue otherwise. Second, the variation counts don't block shortcut solving; variation is in surface tokens, but the rule family is small. Third, the 'latent variable structure' argument in Section I-A is intuition, not analysis—fine as motivation, but it's presented as design rationale for why tasks should be hard, and that framing overreaches.

I think the reader's conditional verdict is right. The paper deserves a serious referee, but any referee should demand at least: (1) human performance on a sample, (2) a few LLMs with and without simple baselines (frame-diff, template-match), and (3) uniqueness checks on generated puzzles. If those come back, this could be a useful benchmark. As is, it's a promising scaffold with an unvalidated headline claim.

For you: worth a glance if you care about LLM reasoning eval or contamination; I wouldn't cite it yet. Send it to review? Yes, as a tool/resource paper—with the expectation of major revision. Serious thinker: yes; the writing is clear and the limitations are not hidden, even if the claims outrun the evidence.

Referee Report

4 major / 4 minor

Summary. The paper introduces enigme, an open-source Python library for procedurally generating text-based reasoning puzzles in three classes (numeric, sequence, and physics). The motivation is that transformer-decoder language models encode latent structure through token interactions and positional embeddings, so puzzles requiring 2D/3D visualization, world-model building, and intuitive physics should probe the boundary of their reasoning abilities. The paper describes the generation processes, provides three example puzzles, reports estimated variation counts in Table I, and argues that the large variation space makes memorization difficult, but it contains no empirical measurements with models or humans.

Significance. If its construct-validity claims were established, enigme could be a useful lightweight benchmark generator: it is open-source, parameterized, and designed around plausible hypotheses about transformer-decoder limitations. The paper also usefully draws attention to the distinction between token-sequence matching and spatial/world-model reasoning. However, the current contribution is a design hypothesis rather than a validated evaluation tool, because the load-bearing claims about task difficulty and the impossibility of shortcut solving are asserted without empirical support.

major comments (4)

[Section I-A] The central claim that "pure sequential analysis makes the task very difficult" and that the large variation space makes memorization difficult is asserted without evidence. The manuscript reports no experiments with humans or models and no comparison to surface-level baselines. Since the generation is template-based (Section II-D), a solver that computes coordinate-wise character diffs across frames could plausibly solve many sequence and physics puzzles without building a world model, so the load-bearing premise is unsecured. The authors should add baseline experiments with simple diff-based or sequence-extrapolation heuristics and at least one LLM evaluation to support or qualify the claim.
[Table I] The variation counts are presented as estimates, but no derivation, formula, or reproducible calculation is provided. The text says the estimates use the "largest possible configuration" and form a "good lower bound approximation," but without the parameter ranges and counting method the reader cannot verify the memorization-difficulty argument. Please include the exact counting procedure or a script that computes these numbers.
[Section II-D] The paper states that the generation process allows "programmatic control over the accuracy of the solutions," but it never demonstrates that every generated puzzle has a unique correct answer or that the answer can be automatically verified. A benchmark for evaluating reasoning must guarantee answer determinism. Please describe the verifier, report its behavior on a sample of generated instances, and specify the exact answer format expected from a model.
[Sections II-B and II-C] The sequence and physics puzzle classes are said to require abductive reasoning and naive physics, but the hidden transformations appear to belong to a small family of simple operations (translation, reflection, scaling, boundary interaction). The token-level variation estimates in Table I do not establish that the underlying rule family is hard to infer from surface patterns. Please provide a formal characterization of the transformation family and empirical evidence that the intended rules are recoverable only through the intended abstract reasoning, rather than through template or frame-diff shortcuts.

minor comments (4)

[Throughout] There are several typos: "visuualisation" and "probelm" in Section I-A, "dimenions" in Section II-B, and "bare some similarity" should be "bear some similarity."
[Figures 2-4] The example figures are not accompanied by the expected answer or a brief explanation of the rule, which makes it difficult for a reader to understand what the puzzle is testing. Adding the gold answer and rule description for each example would improve clarity.
[References] Reference [7] points to a social media post; this is not an appropriate citable source for a methodological claim about data contamination, and it should be replaced with a published study or preprint with verifiable methodology.
[Section IV] The availability statement gives a GitHub link, but no version number or installation/test instructions; please include a versioned release and a minimal code snippet that reproduces the examples in Figures 2-4.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper presents a benchmark generator; its central claims are design intentions and estimates, not derived predictions fitted to their own inputs.

full rationale

The paper describes an open-source library that procedurally generates text puzzles in three classes (numeric, sequence, physics). There is no derivation chain in which an output is computed from an input that already contains the output. The closest thing to a load-bearing claim is the design statement in Section I-A that the puzzles are arranged so that 'pure sequential analysis makes the task very difficult' and that the variation space is large enough that 'memorisation will be difficult.' These are hypotheses about how the generated puzzles will behave when given to transformer-decoder models; they are not derived from the templates by construction, nor are they fitted parameters renamed as predictions. The variation estimates in Table I are counts of generated configurations from the template parameters, and the paper presents them as estimates, not as measured model performance. No load-bearing step is justified solely by a self-citation: the references are to external work on LLM reasoning and evaluation, and the author does not cite a prior uniqueness theorem or ansatz from the same authors. The skeptical concern that template-generated puzzles might be solvable by a frame-diff heuristic is a construct-validity or benchmark-quality objection, not a circularity objection; it challenges whether the puzzles test what they intend, not whether the paper's claims reduce to their inputs. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The free parameters are design choices of the generator, not fitted quantities. The axioms are untested assumptions about transformer limitations, memorization resistance, and puzzle solvability, all of which the central claim depends on but none of which are verified.

free parameters (4)

complexity/dimension parameter
Controls the number of dimensions (1-3) of the puzzles, set by the user, not fitted to data. Central to difficulty and variation counts.
background character set = ._,'‘
Set of background characters for sequence and physics puzzles, chosen by hand.
foreground character set size = 38
Number of characters that can replace background cells, chosen by the generator design.
random seed for generation
The generator uses randomness; without a fixed seed, exact examples are not reproducible.

assumptions (3)

domain assumption Transformer-decoder models encode text via token interaction and positional embeddings, which limits their ability to process 2D/3D spatial patterns.
Invoked in Section I-A to justify the puzzle design. Not proven or tested in the paper.
domain assumption Large variation counts prevent memorization by language models.
Stated in Section I-A. No evidence is given that the variation space is too large for memorization or cannot be solved by template matching.
domain assumption The generated puzzles have unique, well-defined answers.
Implied by the claim of 'programmatic control over the accuracy of the solutions' in Section II-D, but not demonstrated with examples or formal checks.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enigme: Generative Text Puzzles for Evaluating Reasoning in Language Models." pith.science (2026). https://pith.science/paper/O3UNT6VJ

@misc{pith2026250504914,
  author       = {Pith},
  title        = {Pith review of: Enigme: Generative Text Puzzles for Evaluating Reasoning in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O3UNT6VJ}},
  note         = {Machine review of arXiv:2505.04914}
}

read the original abstract

Transformer-decoder language models are a core innovation in text based generative artificial intelligence. These models are being deployed as general-purpose intelligence systems in many applications. Central to their utility is the capacity to understand natural language commands and exploit the reasoning embedded in human text corpora to apply some form of reasoning process to a wide variety of novel tasks. To understand the limitations of this approach to generating reasoning we argue that we need to consider the architectural constraints of these systems. Consideration of the latent variable structure of transformer-decoder models allows us to design reasoning tasks that should probe the boundary of their capacity to reason. We present enigme, an open-source library for generating text-based puzzles to be used in training and evaluating reasoning skills within transformer-decoder models and future AI architectures.

Figures

Figures reproduced from arXiv: 2505.04914 by the authors.

**Figure 2.** Example 1: Enigme Numeric Puzzle - Self Referential Numeric Puzzle [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Example 2: Enigme Sequence Puzzle - Patterns Represented with ASCII Text [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Example 3: Enigme Physics Puzzle - Physical Object Movement Represented with ASCII Text [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 7 canonical work pages

[1]

Language models are few-shot learners,

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amode...

2020
[2]

Large language models are zero-shot reasoners,

T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large language models are zero-shot reasoners,” in Advances in Neural In- formation Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 22 199–22 213

2022
[3]

Emergent abilities of large language models,

J. Wei, Y . Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus, “Emergent abilities of large language models,” Transactions on Machine Learning Research , 2022, survey Certification. [Online]. Available: https://openreview.net/forum?id=yzkSU5zdwD

work page 2022
[4]

Sparks of artificial general intelligence: Early experiments with gpt-4,

S. Bubeck, V . Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y . T. Lee, Y . Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y . Zhang, “Sparks of artificial general intelligence: Early experiments with gpt-4,” 2023. [Online]. Available: https://arxiv.org/abs/2303.12712

arXiv 2023
[5]

Memorization without overfitting: Analyzing the training dynamics of large language models,

K. Tirumala, A. H. Markosyan, L. Zettlemoyer, and A. Aghajanyan, “Memorization without overfitting: Analyzing the training dynamics of large language models,” 2022. [Online]. Available: https://arxiv.org/abs/ 2205.10770

arXiv 2022
[6]

Sok: Memorization in general-purpose large language models,

V . Hartmann, A. Suri, V . Bindschaedler, D. Evans, S. Tople, and R. West, “Sok: Memorization in general-purpose large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2310.18362

arXiv 2023
[7]

H. He. (2023) Gpt-4’s performance is influenced by data contamination. [Online]. Available: https://x.com/cHHillee/status/ 1635790330854526981

work page 2023
[8]

Elements of world knowledge (ewok): A cognition-inspired framework for evaluating basic world knowledge in language models,

A. A. Ivanova, A. Sathe, B. Lipkin, U. Kumar, S. Radkani, T. H. Clark, C. Kauf, J. Hu, R. T. Pramod, G. Grand, V . Paulun, M. Ryskina, E. Aky ¨urek, E. Wilcox, N. Rashid, L. Choshen, R. Levy, E. Fedorenko, J. Tenenbaum, and J. Andreas, “Elements of world knowledge (ewok): A cognition-inspired framework for evaluating basic world knowledge in language mode...

arXiv 2024

Show all 29 references

[9]

McCorduck, Machines Who Think: A Personal Inquiry into the History and Prospects of Artificial Intelligence

P. McCorduck, Machines Who Think: A Personal Inquiry into the History and Prospects of Artificial Intelligence . AK Peters Ltd, 2004

2004
[10]

On the measure of intelligence,

F. Chollet, “On the measure of intelligence,” 2019. [Online]. Available: https://arxiv.org/abs/1911.01547

2019 arXiv
[11]

Evaluating large language models: A comprehensive survey,

Z. Guo, R. Jin, C. Liu, Y . Huang, D. Shi, Supryadi, L. Yu, Y . Liu, J. Li, B. Xiong, and D. Xiong, “Evaluating large language models: A comprehensive survey,” 2023. [Online]. Available: https://arxiv.org/abs/2310.19736

2023 arXiv
[12]

A survey on evaluation of large language models,

Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wang, W. Ye, Y . Zhang, Y . Chang, P. S. Yu, 5 Fig. 4. Example 3: Enigme Physics Puzzle - Physical Object Movement Represented with ASCII Text Q. Yang, and X. Xie, “A survey on evaluation of lar...

2024 doi
[13]

Challenges and applications of large language models,

J. Kaddour, J. Harris, M. Mozes, H. Bradley, R. Raileanu, and R. McHardy, “Challenges and applications of large language models,”
[14]

Llmrg: Improving recommendations through large language model reasoning graphs,

Y . Wang, Z. Chu, X. Ouyang, S. Wang, H. Hao, Y . Shen, J. Gu, S. Xue, J. Zhang, Q. Cui, L. Li, J. Zhou, and S. Li, “Llmrg: Improving recommendations through large language model reasoning graphs,” Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 17...

2024
[15]

Language models as agent models,

J. Andreas, “Language models as agent models,” in Findings of the Association for Computational Linguistics: EMNLP 2022 , Y . Goldberg, Z. Kozareva, and Y . Zhang, Eds. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 5769–5779. [Onlin...

2022
[16]

ReAct: Synergizing reasoning and acting in language models,

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “ReAct: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations (ICLR) , 2023

2023
[17]

Chain-of-thought prompting elicits reasoning in large language models,

J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Proceedings of the 36th International Conference on Neural Information Processing Systems , ser. NIPS ’22...

2024
[18]

Few shot chain-of- thought driven reasoning to prompt llms for open ended medical question answering,

O. Gramopadhye, S. S. Nachane, P. Chanda, G. Ramakrishnan, K. S. Jadhav, Y . Nandwani, D. Raghu, and S. Joshi, “Few shot chain-of- thought driven reasoning to prompt llms for open ended medical question answering,” 2024. [Online]. Available: https://arxiv.org/abs/2403.04890

2024 arXiv
[19]

Automatic chain of thought prompting in large language models,

Z. Zhang, A. Zhang, M. Li, and A. Smola, “Automatic chain of thought prompting in large language models,” in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available: https://openreview.net/forum?id=5NTt8GFjUHkr

2023
[20]

Rethinking the bounds of llm reasoning: Are multi-agent discussions the key?

Q. Wang, Z. Wang, Y . Su, H. Tong, and Y . Song, “Rethinking the bounds of llm reasoning: Are multi-agent discussions the key?” 2024. [Online]. Available: https://arxiv.org/abs/2402.18272

2024 arXiv
[21]

Can neural networks understand monotonicity reasoning?

H. Yanaka, K. Mineshima, D. Bekki, K. Inui, S. Sekine, L. Abzianidze, and J. Bos, “Can neural networks understand monotonicity reasoning?” in Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , T. Linzen, G. Chrupała, Y . Beli...

2019
[22]

Idea: Enhancing the rule learning ability of large language model agent through induction, deduction, and abduction,

K. He, M. Zhang, S. Yan, P. Wu, and Z. Z. Chen, “Idea: Enhancing the rule learning ability of large language model agent through induction, deduction, and abduction,” 2024. [Online]. Available: https://arxiv.org/abs/2408.10455

2024
[23]

Are large language models really good logical reasoners? a comprehensive evaluation and beyond,

F. Xu, Q. Lin, J. Han, T. Zhao, J. Liu, and E. Cambria, “Are large language models really good logical reasoners? a comprehensive evaluation and beyond,” 2023. [Online]. Available: https://arxiv.org/abs/2306.09841

2023 arXiv
[24]

A peek into token bias: Large language models are not yet genuine reasoners,

B. Jiang, Y . Xie, Z. Hao, X. Wang, T. Mallick, W. J. Su, C. J. Taylor, and D. Roth, “A peek into token bias: Large language models are not yet genuine reasoners,” 2024. [Online]. Available: https://arxiv.org/abs/2406.11050

2024 arXiv
[25]

Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models,

I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar, “Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2410.05229

2024 arXiv
[26]

Premise order matters in reasoning with large language models,

X. Chen, R. A. Chi, X. Wang, and D. Zhou, “Premise order matters in reasoning with large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2402.08939

2024 arXiv
[27]

Large language models are not strong abstract reasoners,

G. Gendron, Q. Bao, M. Witbrock, and G. Dobbie, “Large language models are not strong abstract reasoners,” 2024. [Online]. Available: https://arxiv.org/abs/2305.19555

2024 arXiv
[28]

Inductive biases for deep learning of higher-level cognition,

A. Goyal and Y . Bengio, “Inductive biases for deep learning of higher-level cognition,” 2022. [Online]. Available: https://arxiv.org/abs/ 2011.15091

2022 arXiv
[2023]

Available: https://arxiv.org/abs/2307.10169

[Online]. Available: https://arxiv.org/abs/2307.10169

arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Language models are few-shot learners,

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amode...

2020

[2] [2]

Large language models are zero-shot reasoners,

T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large language models are zero-shot reasoners,” in Advances in Neural In- formation Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 22 199–22 213

2022

[3] [3]

Emergent abilities of large language models,

J. Wei, Y . Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus, “Emergent abilities of large language models,” Transactions on Machine Learning Research , 2022, survey Certification. [Online]. Available: https://openreview.net/forum?id=yzkSU5zdwD

work page 2022

[4] [4]

Sparks of artificial general intelligence: Early experiments with gpt-4,

S. Bubeck, V . Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y . T. Lee, Y . Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y . Zhang, “Sparks of artificial general intelligence: Early experiments with gpt-4,” 2023. [Online]. Available: https://arxiv.org/abs/2303.12712

arXiv 2023

[5] [5]

Memorization without overfitting: Analyzing the training dynamics of large language models,

K. Tirumala, A. H. Markosyan, L. Zettlemoyer, and A. Aghajanyan, “Memorization without overfitting: Analyzing the training dynamics of large language models,” 2022. [Online]. Available: https://arxiv.org/abs/ 2205.10770

arXiv 2022

[6] [6]

Sok: Memorization in general-purpose large language models,

V . Hartmann, A. Suri, V . Bindschaedler, D. Evans, S. Tople, and R. West, “Sok: Memorization in general-purpose large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2310.18362

arXiv 2023

[7] [7]

H. He. (2023) Gpt-4’s performance is influenced by data contamination. [Online]. Available: https://x.com/cHHillee/status/ 1635790330854526981

work page 2023

[8] [8]

Elements of world knowledge (ewok): A cognition-inspired framework for evaluating basic world knowledge in language models,

A. A. Ivanova, A. Sathe, B. Lipkin, U. Kumar, S. Radkani, T. H. Clark, C. Kauf, J. Hu, R. T. Pramod, G. Grand, V . Paulun, M. Ryskina, E. Aky ¨urek, E. Wilcox, N. Rashid, L. Choshen, R. Levy, E. Fedorenko, J. Tenenbaum, and J. Andreas, “Elements of world knowledge (ewok): A cognition-inspired framework for evaluating basic world knowledge in language mode...

arXiv 2024