Pith. sign in

REVIEW 3 major objections 5 minor 40 references

What is an "Abstract Reasoner"? Revisiting Experiments and Arguments about Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that poor zero-shot performance does not show LLMs lack abstract reasoning, because tuning only the input encoding lifts frozen LLMs to near-perfect scores on reasoning benchmarks.

desk verdict Useful empirical revisit that complicates the zero-shot negative result, with an honest discussion about what it does and doesn't show. read the letter →

arxiv 2507.22457 v1 pith:WXODBQJ5 submitted 2025-07-30 cs.CL cs.AI

Tian Yun , Chen Sun , Ellie Pavlick This is my paper · ORCID
classification cs.CLcs.AI
keywords abstractreasoninglargelanguagemodelszero-shottransferembeddingfinetuningvisualmultimodalLLMrepresentationalignmentfrozentransformerblocks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper challenges the recent conclusion that LLMs are not abstract reasoners because they score low on reasoning benchmarks in a zero-shot setting. It reproduces the low zero-shot numbers but shows that tuning only the input encoding—the token embedding layer, or a visual encoder feeding a frozen language backbone—lifts performance to near-perfect on several tasks. The authors argue this means the transformer blocks already contain transferable inferential capability; what was missing was input-side adaptation, not reasoning itself. They also show the adaptation does not necessarily transfer across datasets, and use the results to reopen the question of what should count as abstract reasoning and why we care.

What carries the argument

The mechanism is input-side adaptation: finetuning the token embedding layer (for text) or finetuning the visual encoder and projection layer (for images) while leaving all transformer blocks frozen. This is analogous to converting an input into the format the system expects. The paper also uses a control in which the embedding is finetuned on the same task with randomly shuffled labels, showing that most positive transfer comes from low-level visual features shared by task variants rather than from the reasoning logic itself.

What would settle it

Finetune the embedding layer on a task with the training labels randomly shuffled while keeping token statistics identical; if the model fits training data but scores at chance on held-out items that require the true rule, the near-perfect results come from surface associations rather than abstract transfer.

Watch

Extended reading notes

Core claim

The central claim is that poor zero-shot performance on abstract reasoning tasks does not establish that LLMs lack abstract reasoning. Keeping the transformer blocks of a pretrained LLM completely frozen, the authors finetune only the input layers: the token embedding layer for text tasks, and a from-scratch visual encoder plus projection layer for multimodal tasks. Across ACRE, RAVEN, PVR, and related benchmarks, embedding-only finetuning reaches 100% on some tasks and 97.6% on PVR after near-random zero-shot scores, and a frozen LLaMA2-7b with a trained visual encoder reaches 95.5/97.5/86.5 on ACRE splits. They interpret this as evidence that the model's representations and inferential machinery are highly transferable, but that nontrivial input-side effort is required to harness them per task. The paper does not claim the adapted model is a cognitive abstract reasoner; it argues that zero-shot transfer is not an appropriate operationalization.

Load-bearing premise

The interpretation depends on the tuned embedding layer functioning mainly as an input re-encoder rather than learning the task's surface answer mapping; the paper acknowledges this distinction is not yet pinned down.

Editorial extensions

If this is right

  • Zero-shot accuracy should not be treated as a direct measure of abstract reasoning capacity, since a small input-side adaptation can transform near-random scores into near-perfect ones.
  • The transformer blocks of a pretrained LLM appear to carry task-relevant inferential machinery that is reusable across diverse textual and visual tasks, as long as the input encoder is tuned.
  • Adaptation does not transfer automatically across datasets, so practical deployment still needs per-task input engineering.
  • If the goal is human-like flexibility, transfer after adaptation matters more than zero-shot transfer; if the goal is cheap zero-shot scaling, the input interface is the bottleneck.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence not explored in the paper: the amount of input-side adaptation needed, whether measured in parameters tuned or examples seen, could itself serve as a measure of a model's abstraction level, complementing raw accuracy.
  • The data-efficiency results imply that for multimodal reasoning, progress may come more from better visual encoders than from larger language backbones.
  • If the embedding layer merely aligns inputs, then adversarial rephrasings or novel symbol sets that preserve task structure should not hurt performance; this is a direct way to test the alignment interpretation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper revisits the argument that large language models (LLMs) are not 'abstract reasoners' because of their poor zero-shot performance on reasoning benchmarks. The authors replicate the zero-shot results of Gendron et al. (2024) on LLaMA2-7b, then show that finetuning only the token embedding layer (with transformer blocks frozen) yields near-perfect accuracy on ACRE-T, RAVEN-T-Symb, and PVR, comparable to LoRA finetuning of the full model. They further show that a frozen LLaMA2-7b with a train-from-scratch visual encoder achieves high accuracy on ACRE and MEWL, outperforming several prior approaches. A data-efficiency analysis on RAVEN indicates that adaptation transfers only limited reasoning content, and the authors discuss the philosophical implications of their findings, arguing against zero-shot transfer as a sole criterion for abstract reasoning. The paper includes code and acknowledges parser-sensitivity.

Significance. The empirical results are valuable and transparently reported; if the interpretation is accepted, they challenge the common conclusion that poor zero-shot scores indicate an absence of reasoning in LLMs. The paper's strengths include reproducible zero-shot evaluations (with parser-sensitivity acknowledged), a shuffled-label control in the transfer analysis, and public code. However, the central interpretive claim — that the frozen transformer blocks contribute the reasoning — is not established by the experimental design, particularly in the visual-encoder experiments where the input encoder is trained end-to-end with access to task labels.

major comments (3)
  1. [6 (Tables 2 and 3)] The claim that a frozen pretrained LLM with a train-from-scratch visual encoder can perform well on abstract visual reasoning tasks does not demonstrate that the transformer blocks of the LLM perform the reasoning. Because the visual encoder and projection layer are trained end-to-end with gradients flowing through the frozen LLM, the encoder could learn task-specific surface-to-answer mappings, and a linear classifier on the same features might achieve similar accuracy. I recommend adding an ablation that replaces the frozen LLM with a randomly initialized transformer of the same architecture (or with a directly trained classifier) to verify that the pretrained transformer blocks are necessary. The paper's footnote 6 concedes this gap, but it directly affects the interpretation in Section 1.
  2. [5 (Figure 4)] The shuffled-label condition in Figure 4 is a useful control for the transfer experiment, but it does not resolve whether the tuned embedding layer implements task logic on the task it was tuned for. The result that positive transfer arises primarily from low-level visual features actually supports the view that the embedding tuning is mostly re-encoding, yet the absolute accuracy on the tuned task could still reflect task-specific computations in the embedding. An additional control, such as comparing against an embedding trained with a non-task objective or against a bag-of-tokens baseline, would strengthen the claim that the transformer blocks are the source of the underlying reasoning.
  3. [Abstract and Section 7] The abstract states that 'much of their representations and inferential capabilities appear to be transferable across very diverse tasks,' but the experiments only show that input-layer tuning is sufficient to achieve high task accuracy with frozen transformer blocks. Whether the transformer blocks themselves contribute the inference, or merely act as a read-out layer over the tuned input, remains unresolved. Since Section 7 explicitly acknowledges that no framework distinguishes representational alignment from true abstraction, I recommend either softening this interpretive claim or adding the controls suggested above to justify it.
minor comments (5)
  1. [5] In the paragraph after Figure 3, 'aganist' should be 'against'.
  2. [3.2] In the first sentence, 'transfer fo' should be 'transfer for'.
  3. [Table 1] The MCQA column headers are duplicated and misaligned (RA VENT appears twice); please re-check the table formatting.
  4. [Figure 3] The legend label 'Pretrained' for the zero-shot bars is ambiguous; consider using 'Frozen, zero-shot' for clarity.
  5. [7 (footnote 6)] The footnote marker placement is unclear; the superscript appears after 'task' but the footnote text is placed later in the paragraph. Please ensure the marker appears at the end of the relevant sentence.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; central results are held-out evaluations against external benchmarks, with the paper's own control and explicit caveat.

full rationale

The paper's two main empirical claims—(i) embedding-only finetuning produces near-perfect scores on ACRET and RAVENT, and (ii) a frozen LLaMA2-7b with a trained-from-scratch visual encoder reaches high ACRE and MEWL scores—are tested on held-out splits of external benchmarks, not derived from the fitted parameters by construction. The shuffled-label condition (center-single-shuffled) is a genuine control for low-level visual transfer, and the paper explicitly reports that positive transfer is 'primarily explained by the lower-level visual features rather than the reasoning logic of the tasks.' The principal weakness is interpretive, not circular: Section 7 and footnote 6 concede that tuning the input embedding 'might do more than simply "rerepresent"' and that 'a more robust theoretical framework' is needed to distinguish representational alignment from true abstraction. That caveat limits the strength of the 'input misalignment' conclusion, but it does not make any equation or fitted parameter identical to a reported prediction. Related-work citations overlap with the authors (Merullo et al. 2023; Sun et al. 2024) but are not load-bearing; the support for the claims comes from this paper's own experiments. No self-definitional, fitted-input-as-prediction, or imported-uniqueness step is present.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

This is an empirical paper, not a derivation, so no fitted constants enter any equation. The free parameters are hand-chosen training and input-encoding choices whose ablation is not reported. The axiom list shows that the central interpretive claim (input tuning reveals latent reasoning) rests on domain assumptions about task validity, model representativeness, and the philosophical premise that adaptation is compatible with abstraction, several of which the authors themselves flag (footnotes 3 and 6, Appendix A). No invented entities are introduced.

free parameters (3)
  • Finetuning hyperparameters (learning rate, epochs, batch size) = lr 3e-5; 50 text epochs; 20 to 40 visual epochs; batch 64
    Hand-chosen standard settings in Sections 5 and 6.3. They do not enter any derivation, but no ablations are reported, so robustness of the qualitative claim to these choices is untested.
  • Visual encoder architecture and input variants = 2-layer ViT, 4 heads, 768 hidden; Object variant uses ground-truth crops
    Hand-designed encoder in Section 6.3. The Object variant assumes oracle object detection, which is a strong input-side assumption that drives the best results.
  • Object location feature vector = [x1, y1, x2, y2, w, h, w*h]
    Hand-designed input feature in Section 6.3 including a multiplicative w*h term; only affects the Symbol variant.
assumptions (6)
  • domain assumption Gendron et al.'s seven-task benchmark operationalizes abstract reasoning
    The tasks are adopted wholesale from Gendron et al. (2024) in Section 3; the paper critiques the operationalization philosophically but uses the same tasks for all measurements.
  • domain assumption LLaMA2-7b-chat is representative of LLMs
    All experiments use one model; Appendix A acknowledges this and proposes future multi-model runs.
  • ad hoc to paper LoRA finetuning is an upper bound on attainable task performance
    Stated in Section 5 as the permissive-condition reference against which embedding tuning is compared; not separately justified.
  • domain assumption Text versions ACRET and RAVENT are valid instantiations of the visual tasks
    The Figure 3 caption concedes text versions 'may make the tasks easier to solve'; the near-perfect headline numbers come largely from these versions.
  • domain assumption Input adaptation is compatible with being an abstract reasoner (GOFAI and Dennett analogies)
    The interpretive core of Section 7; a philosophical stance argued by analogy, not established empirically, and explicitly qualified in footnote 6.
  • domain assumption Published SOTA numbers in Tables 2 and 3 are comparable to the authors' setup
    IV-CL, LRR, and Flamingo results are taken from their papers; the authors do not show splits and backbones are matched, and the comparison is only favorable for Symbol/Object input variants.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What is an "Abstract Reasoner"? Revisiting Experiments and Arguments about Large Language Models." pith.science (2026). https://pith.science/paper/WXODBQJ5

@misc{pith2026250722457,
  author       = {Pith},
  title        = {Pith review of: What is an "Abstract Reasoner"? Revisiting Experiments and Arguments about Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WXODBQJ5}},
  note         = {Machine review of arXiv:2507.22457}
}
read the original abstract

Recent work has argued that large language models (LLMs) are not "abstract reasoners", citing their poor zero-shot performance on a variety of challenging tasks as evidence. We revisit these experiments in order to add nuance to the claim. First, we show that while LLMs indeed perform poorly in a zero-shot setting, even tuning a small subset of parameters for input encoding can enable near-perfect performance. However, we also show that this finetuning does not necessarily transfer across datasets. We take this collection of empirical results as an invitation to (re-)open the discussion of what it means to be an "abstract reasoner", and why it matters whether LLMs fit the bill.

Figures

Figures reproduced from arXiv: 2507.22457 by the authors.

Figure 1
Figure 1. Illustration of our experimental settings. In Setting (a), we freeze the whole LLM and run evaluations. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the use of language models for text-based and image-based versions of ACRE. Each data [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance of finetuned LLMs on OPQA (ARC and PVR) and MCQA (ACRE [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Data efficiency analyses on LLaMA2-7b with token embedding layer finetuned on [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 22 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Yaniv Benny, Niv Pekar, and Lior Wolf. 2021. Scale-localized abstract reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12557--12565

    work page 2021

  5. [5]

    Apratim Bhattacharyya, Sunny Panchal, Reza Pourreza, Mingu Lee, Pulkit Madan, and Roland Memisevic. 2023. Look, remember and reason: Grounded reasoning in videos with language models. In The Twelfth International Conference on Learning Representations

    work page 2023

  6. [6]

    Fran c ois Chollet. 2019. On the measure of intelligence. arXiv preprint arXiv:1911.01547

    arXiv 2019

  7. [7]

    Daniel C. Dennett. 1997. https://doi.org/10.7551/mitpress/4626.003.0003 True believers:the intentional strategy and why it works . In Mind Design II: Philosophy, Psychology, and Artificial Intelligence. The MIT Press

    work page doi:10.7551/mitpress/4626.003.0003 1997

  8. [8]

    David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matt Botvinick. 2021. Attention over learned object embeddings enables complex visual reasoning. Advances in neural information processing systems, 34:9112--9124

    work page 2021

Show all 40 references
  1. [9]

    Alexey Dosovitskiy. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    2020 arXiv

  2. [10]

    Ga \"e l Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbie. 2024. Large language models are not strong abstract reasoners. In IJCAR

    2024

  3. [11]

    Dedre Gentner, Keith J Holyoak, and Boicho N Kokinov. 2001. Introduction: The place of analogy in cognition. The analogical mind: Perspectives from cognitive science, pages 1--19

    2001

  4. [12]

    Simon Jerome Han, Keith J Ransom, Andrew Perfors, and Charles Kemp. 2024. Inductive reasoning in humans and large language models. Cognitive Systems Research, 83:101155

    2024

  5. [13]

    Douglas R Hofstadter, Melanie Mitchell, et al. 1995. The copycat project: A model of mental fluidity and analogy-making. Advances in connectionist and neural computation theory, 2:205--267

    1995

  6. [14]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021 a . Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  7. [15]

    Sheng Hu, Yuqing Ma, Xianglong Liu, Yanlu Wei, and Shihao Bai. 2021 b . Stratified rule-aware network for abstract visual reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1567--1574

    2021

  8. [16]

    Xiaoyang Hu, Shane Storks, Richard L Lewis, and Joyce Chai. 2023. In-context analogical reasoning with pre-trained language models. In The 61st Annual Meeting Of The Association For Computational Linguistics

    2023

  9. [17]

    Drew Hudson and Christopher D Manning. 2019. Learning by abstraction: The neural state machine. Advances in Neural Information Processing Systems, 32

    2019

  10. [18]

    Guangyuan Jiang, Manjie Xu, Shiji Xin, Wei Liang, Yujia Peng, Chi Zhang, and Yixin Zhu. 2023. Mewl: Few-shot multimodal word learning with referential uncertainty. In International Conference on Machine Learning, pages 15144--15169. PMLR

    2023

  11. [19]

    Martha Lewis and Melanie Mitchell. 2024. Using counterfactual tasks to evaluate the generality of analogical reasoning in large language models. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 46

    2024

  12. [20]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730--19742. PMLR

    2023

  13. [21]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296--26306

    2024

  14. [22]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892--34916

    2023

  15. [23]

    Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations

    2019

  16. [24]

    Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. 2022. Frozen pretrained transformers as universal computation engines. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 7628--7636

    2022

  17. [25]

    Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. 2019. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv:1904.12584

    2019 arXiv

  18. [26]

    Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. 2023. Linearly mapping from image to text space. In The Eleventh International Conference on Learning Representations

    2023

  19. [27]

    Melanie Mitchell. 2021. Abstraction and analogy-making in artificial intelligence. Annals of the New York Academy of Sciences, 1505(1):79--101

    2021

  20. [28]

    Melanie Mitchell, Alessandro B Palmarini, and Arseny Moskvichev. 2023. Comparing humans, gpt-4, and gpt-4v on abstraction and reasoning tasks. arXiv preprint arXiv:2311.09247

    2023 arXiv

  21. [29]

    Arsenii Kirillovich Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. 2023. The conceptarc benchmark: Evaluating understanding and generalization in the arc domain. Transactions on machine learning research

    2023

  22. [30]

    Sam Musker, Alex Duchnowski, Rapha \"e l Milli \`e re, and Ellie Pavlick. 2024. Semantic structure-mapping in llm and human analogical reasoning. arXiv preprint arXiv:2406.13803

    2024

  23. [31]

    Joshua Stewart Rule. 2020. The child as hacker: building more human-like models of learning. Ph.D. thesis, Massachusetts Institute of Technology

    2020

  24. [32]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv p...

    2022 arXiv

  25. [33]

    Claire E Stevenson, Alexandra Pafford, Han LJ van der Maas, and Melanie Mitchell. 2024. Can large language models generalize analogy solving like people can? arXiv preprint arXiv:2411.02348

    2024

  26. [34]

    Chen Sun, Calvin Luo, Xingyi Zhou, Anurag Arnab, and Cordelia Schmid. 2024. Does visual pretraining help end-to-end reasoning? Advances in Neural Information Processing Systems, 36

    2024

  27. [35]

    Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. 2025. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Pr...

    2025

  28. [36]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  29. [37]

    Taylor Webb, Keith J Holyoak, and Hongjing Lu. 2023. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526--1541

    2023

  30. [38]

    Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. 2019. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5317--5327

    2019

  31. [39]

    Chi Zhang, Baoxiong Jia, Mark Edmonds, Song-Chun Zhu, and Yixin Zhu. 2021 a . Acre: Abstract causal reasoning beyond covariation. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 10643--10653

    2021

  32. [40]

    Chiyuan Zhang, Maithra Raghu, Jon Kleinberg, and Samy Bengio. 2021 b . Pointer value retrieval: A new benchmark for understanding the limits of neural network generalization. arXiv preprint arXiv:2107.12580

    2021 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.