Pith. sign in

REVIEW 5 cited by

Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.09247 v3 pith:4WGSU6KF submitted 2023-11-14 cs.AI cs.LG

Melanie Mitchell , Alessandro B. Palmarini , Arseny Moskvichev This is my paper · ORCID
classification cs.AIcs.LG
keywords gpt-4reasoningtasksversionsabilitiesabstractionconceptarcevaluating
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We explore the abstract reasoning abilities of text-only and multimodal versions of GPT-4, using the ConceptARC benchmark [10], which is designed to evaluate robust understanding and reasoning with core-knowledge concepts. We extend the work of Moskvichev et al. [10] by evaluating GPT-4 on more detailed, one-shot prompting (rather than simple, zero-shot prompts) with text versions of ConceptARC tasks, and by evaluating GPT-4V, the multimodal version of GPT-4, on zero- and one-shot prompts using image versions of the simplest tasks. Our experimental results support the conclusion that neither version of GPT-4 has developed robust abstraction abilities at humanlike levels.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Think Through a Bottleneck: Hourglass Reasoning for Rigorous Induction

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Strict stage isolation that passes only a compressed symbolic schema and rule between LLM calls improves few-shot inductive reasoning more than self-refinement or explicit verbalization alone.

  2. Large Language Models Show Signs of Alignment with Human Neurocognition During Abstract Reasoning

    q-bio.NC 2025-08 unverdicted novelty 6.0 of 10

    Only the largest tested LLMs (about 70 billion parameters) match human accuracy on an abstract reasoning task, and the internal geometry of their best layers correlates moderately with human frontal EEG activity.

  3. Dynamic Reinforcement Learning for Actors

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A reinforcement learning update that adjusts each neuron's input-output sensitivity using TD error can replace external exploration noise and backpropagation through time in small actor-critic tasks.

  4. What is an "Abstract Reasoner"? Revisiting Experiments and Arguments about Large Language Models

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Frozen LLMs reach near-perfect accuracy on several abstract reasoning benchmarks after tuning only the token embedding layer, implying poor zero-shot scores reflect input mismatch rather than absent reasoning.

  5. Position: We Need An Algorithmic Understanding of Generative AI

    cs.AI 2025-07 conditional novelty 5.0 of 10

    The paper argues for a systematic algorithmic understanding of LLMs and presents a case study suggesting that Llama models do not implement BFS or DFS on graph navigation tasks.

Pith tools