LLM performance on naturalistic reasoning tasks tracks performance on equivalent Python code simulation, but the effect is partly driven by pattern matching and memorization rather than faithful execution.
Evaluating large language models with runtime behavior of program execution, 2024 a
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Code Simulation as a Proxy for High-order Tasks in Large Language Models
LLM performance on naturalistic reasoning tasks tracks performance on equivalent Python code simulation, but the effect is partly driven by pattern matching and memorization rather than faithful execution.