Pith. sign in

REVIEW 2 cited by

Do large language models solve verbal analogies like children do?

Not yet reviewed by Pith; the record is open.

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.20384 v1 pith:M4IZD5MI submitted 2023-10-31 cs.CL cs.AI

classification cs.CLcs.AI
keywords analogieschildrenlevelsolvetextitverballikellms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Analogy-making lies at the heart of human cognition. Adults solve analogies such as \textit{Horse belongs to stable like chicken belongs to ...?} by mapping relations (\textit{kept in}) and answering \textit{chicken coop}. In contrast, children often use association, e.g., answering \textit{egg}. This paper investigates whether large language models (LLMs) solve verbal analogies in A:B::C:? form using associations, similar to what children do. We use verbal analogies extracted from an online adaptive learning environment, where 14,002 7-12 year-olds from the Netherlands solved 622 analogies in Dutch. The six tested Dutch monolingual and multilingual LLMs performed around the same level as children, with MGPT performing worst, around the 7-year-old level, and XLM-V and GPT-3 the best, slightly above the 11-year-old level. However, when we control for associative processes this picture changes and each model's performance level drops 1-2 years. Further experiments demonstrate that associative processes often underlie correctly solved analogies. We conclude that the LLMs we tested indeed tend to solve verbal analogies by association with C like children do.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Automatic Extraction of Metaphoric Analogies from Literary Texts: Task Formulation, Dataset Construction, and Evaluation

    cs.CL 2024-12 conditional novelty 5.0 of 10

    GPT-4 can extract the four concepts of a proportional metaphoric analogy from short literary texts with 77% frame-wise head-noun accuracy, but full quadruple accuracy is 61%.

  2. Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoning

    cs.AI 2024-12 conditional novelty 5.0 of 10

    With oracle text attributes, GPT-4 and Llama-3 solve Raven matrices but their arithmetic-rule accuracy drops below 10% on larger grids and value ranges, while the neuro-symbolic ARLC model stays accurate.

Pith tools