LLM world models are mental: Output layer evidence of brittle world model use in LLM mechanical reasoning

Cole Robertson; Philip Wolff

REVIEW 4 major objections 4 minor 14 references

LLM world models are mental: Output layer evidence of brittle world model use in LLM mechanical reasoning

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs solve pulley problems with a counting heuristic and coarse spatial cues, not by simulating force flow, and fall to chance when connectivity alone distinguishes the systems.

desk verdict A solid, honest behavioral probe of LLM mechanical reasoning whose central Study 3 claim—no force-flow world model—needs the TikZ-parsing control before it can be called more than plausible. read the letter →

arxiv 2507.15521 v1 pith:3LA5NQTW submitted 2025-07-21 cs.AI

Cole Robertson , Philip Wolff This is my paper · ORCID

classification cs.AI

keywords largelanguagemodelsworldmechanicalreasoningpulleysystemsmentalout-of-distributiongeneralizationheuristicsadvantage

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models reason about physical systems by maintaining internal world models or by exploiting surface statistics. To find out, the authors test four LLMs on pulley-system diagrams rendered as TikZ source code, a format they argue is outside the models' training distribution. The models estimate mechanical advantage above chance, but their answers track the number of pulleys rather than the system's actual force-transmitting structure. When tested on connected-but-non-functional systems, performance collapses to chance, suggesting the models lack robust reasoning over structural connectivity.

What carries the argument

The load-bearing device is the pulley system as a testbed for mechanical advantage (MA), the ratio of input distance to load displacement. MA is determined by how ropes and pulleys are connected, not by the number of components, which makes it a sharp probe for force-flow reasoning. The paper's critical manipulation is the contrast between jumbled non-functional systems, which models can reject using coarse spatial cues, and connected-but-non-functional systems, which require tracking whether force reaches the weight; the collapse on the latter does the argumentative work.

What would settle it

Providing the same Study 3 comparisons in a synthetic diagram language never seen in training, with pulley count fully deconfounded from MA, would settle the point: above-chance accuracy in that format would undercut the claim that LLMs lack structural-connectivity reasoning, while chance performance (and negative mirrored-trial correlations) would confirm it.

Watch

Extended reading notes

Core claim

On the authors' own account, the results show that LLMs develop partial, brittle world models rather than full simulations of mechanical systems. Tested models estimate mechanical advantage above chance, but their estimates are driven mainly by number of pulleys, and true MA contributes little when pulley count is controlled. Models distinguish functional systems from randomly jumbled ones (F1 = 0.80), yet when the comparison is between a working system and one that is connected up but transmits no force, they guess (F1 = 0.46) and their choices do not generalize across mirrored trials. The authors conclude that LLMs can represent statistical associations and approximate spatial layout but lack the representational fidelity to reason over structural connectivity.

Load-bearing premise

The argument rests on the assumption that TikZ code is out-of-distribution for the models, so failure on Study 3 shows a world-model limit rather than a format-parsing problem.

Editorial extensions

If this is right

LLM performance on physical-reasoning benchmarks can be inflated by heuristics tied to surface features, such as counting pulleys, that correlate with the target quantity.
Failing on connected-but-non-functional systems implies current LLMs cannot reliably detect when a mechanism actually transmits force, limiting their use in planning or robotics tasks that require such judgments.
The mixed results reconcile two views: LLMs form partial world models, enough for coarse spatial judgments, but lack the representational fidelity required for robust, generalizable physical reasoning.
Cognitive-science methods for probing human mental models can be adapted to expose the brittleness of LLM reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: If the TikZ-format explanation for the Study 3 collapse is correct, then presenting the same pulley systems in a different novel diagram language, or as natural-language schematic descriptions, should reproduce the chance-level result; a format-independent failure would strengthen the claim that LLMs lack structural-connectivity reasoning.
Inference: The pulley-counting heuristic the authors identify suggests a testable prediction for other mechanical domains: LLM accuracy on levers, gears, or circuits should track salient countable parts whenever those parts correlate with the target output, and collapse when they do not.
Inference: The negative mirrored-trial correlation in Study 3 implies that even the marginally above-chance performance of gpt-4o was unstable; a direct replication with more trials per diagram pair could determine whether the effect is genuinely zero or merely weak.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know before you read it. First, the empirical pattern is real and worth taking seriously: models estimate mechanical advantage above chance, identify jumbled vs functional systems at F1=0.8, and fall to chance when the nonfunctional system is connected but transfers no force. Second, the authors' interpretation of Study 3 as evidence against force-flow world models is plausible but not forced—the missing control is a human baseline on the same TikZ source, and the withheld stimuli make that hard to check.

What's new: the three-level functional/jumbled/connected-nonfunctional contrast as a probe, the TikZ stimulus form, and the mirrored-trial artifact check. The paper does genuine work. The design controls pulley count, adds distractors, uses frozen checkpoints, and validates the GPT-4 data-cleaning step against human annotations (κ=0.99). The negative correlation across mirrored trials (r=-0.28 for both models) is a good test of whether above-chance performance is stable; it supports the artifact reading, though with n=30 pairs it's underpowered.

The soft spots are in the central inference. Study 3's chance performance can be read as a world-model limit, but it's equally consistent with the models failing to parse the spatial configuration from TikZ markup. Study 2's high accuracy might rest on coarse code-level cues—jumbled coordinates are trivial to flag—rather than on representation of connectivity. The authors' OOD claim is an untested assumption; they concede exposure during training can't be ruled out, and the confound is broader than memorization: the format itself may be the obstacle. There is no human baseline on the same task, and no rendered-image condition to separate parsing from reasoning. The paper's own Limitations section acknowledges the training-exposure point but not the parsing-deficit alternative.

None of this sinks the paper. The authors are appropriately modest—they frame Study 1 as heuristic use, not simulation—and the three-study progression is a useful template. The paper is for LLM reasoning researchers and cognitive scientists interested in mental-model probes; they'll find a reusable paradigm and a clean negative result that needs the right control before it becomes a strong claim.

Recommendation: yes, send it to serious peer review. The empirical core is solid, the interpretation is contestable, and the fixes (release stimuli, add human and rendered-image baselines) are achievable in revision.

One more thing: the reader's skeptic about OOD is right, but I'd push back on the harshest version—the paper doesn't pretend to have solved that, and the pulley-counting heuristic evidence in Study 1 doesn't depend on the OOD assumption. That part holds.

Referee Report

4 major / 4 minor

Summary. This paper adapts methods from the cognitive science of human mental models to test whether LLMs can reason about pulley systems presented as TikZ source code. In Study 1, four LLMs estimate mechanical advantage (MA) from TikZ diagrams; all are above chance on exact match and their estimates correlate with ground truth, but regression analyses indicate a dominant pulley-counting heuristic. In Study 2, models distinguish functional pulley systems from jumbled, randomly placed systems with high accuracy (F1=0.80). In Study 3, models compare functional systems with connected-but-non-functional systems and perform at chance overall (F1=0.46). The authors conclude that LLMs use coarse heuristics and spatial/gist representations rather than robust force-flow simulation, and advocate cognitive-science methods for evaluating LLM world models.

Significance. If the central interpretation withstands scrutiny, the paper contributes a useful test bed for probing LLM world models and a careful demonstration of a performance ceiling. The study has several strengths: frozen model checkpoints for reproducibility, a ground-truth MA formula with no fitted parameters, validated GPT-4-based data cleaning (human kappa = 0.99), OSF scripts, and appropriately modest conclusions. However, the load-bearing inference from Study 3 -- that chance performance reveals a lack of force-flow world models rather than a parsing failure -- depends on an untested assumption that TikZ stimuli are transparent and out-of-distribution. The absence of human baselines, non-TikZ controls, and public stimuli leaves the central claim underdetermined.

major comments (4)

[§4.2, §5] The central inference of Study 3 -- that chance-level discrimination of functional versus connected-but-non-functional systems shows LLMs lack force-flow world models -- assumes that TikZ source code is a transparent vehicle for recovering the mechanical structure. The authors report no human baseline on the raw TikZ stimuli and no condition in which the same systems are shown as rendered images or structured connection descriptions. A parsing deficit, rather than a world-model deficit, is equally consistent with all Study 3 results. The manuscript's own Limitations section (§5) concedes that training exposure cannot be ruled out, but the confound is broader than memorization: the functional/non-functional distinction requires tracing force-transmitting paths from coordinates, which is not explicit in TikZ. This control is necessary before the claim about world-model fidelity can be made.
[§4.2, Table A3] The interpretation that gpt-4o's marginally above-chance Study 3 accuracy (0.553, p=.037) is 'a statistical artifact' is not supported by the mirrored-trial correlation test. With n=30 diagram pairs, the non-significant negative correlations r(28)=-0.28 have wide confidence intervals and low power, and failure to reject the null hypothesis does not establish that performance is unreliable. A more appropriate analysis would use an agreement measure across mirrored presentations or a test designed to detect true above-chance consistency; the current analysis does not license the conclusion that the significant binomial result is spurious.
[§4.1 vs. §3.1] Study 3 changes the prompt from 'which system has greater mechanical advantage' (Study 2) to 'which system will move the block with LESS force', and also changes the number of repetitions per diagram pair. These changes could affect model behavior independently of the structural distinction being tested, and trials where models assert that MA is equal are coded as failures, which may bias results against models that correctly detect functional equivalence. The authors should either use the same judgment frame across studies or demonstrate that prompt wording does not drive the observed difference.
[§2.1, §5] The decision to withhold stimuli ('Materials are available upon request... not public') directly conflicts with the need to verify the out-of-distribution assumption. Without public stimuli or a release mechanism, independent researchers cannot test whether the TikZ format is genuinely novel to the models, nor can they assess the solvability of the task in this format. The authors should provide a delayed public release, a hashed stimulus set, or a set of held-out analogous stimuli to make the central OOD claim falsifiable.

minor comments (4)

[Throughout] There are several typos and spacing inconsistencies, including 'Ti kZ' (e.g., §2.1, §4.1) and 'gpt-4o' without a hyphen in places; a careful proofreading pass is needed.
[§1, first paragraph] The word 'open' appears as a stray fragment in the sentence about implications for AGI, interrupting the text.
[§2.2, Figure 2] Figure 2 is referenced as showing mean model estimates over true MA, but the figure lacks axis labels and a legend identifying which curve corresponds to which model; this should be fixed for readability.
[§2.1, Data cleaning] The sentence 'we instructedgpt-4 to return' is missing a space between 'instructed' and 'gpt-4'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ground-truth MA is physically defined, model outputs are raw API responses, and no fitted parameter or self-citation defines the conclusions.

full rationale

The paper's central outcome is ground-truth mechanical advantage, which is fixed by the physical definition of MA (ratio of input distance to load displacement) and built into the stimuli; model estimates are raw API responses that are then summarized with OLS regression, so the reported correlations are descriptive of model behavior rather than predictions generated by a fitted model. Study 2 and Study 3 correctness labels are determined by stimulus construction (functional vs. jumbled vs. connected-but-force-free), not by any parameter fitted to model outputs, and no equation in the paper defines the target in terms of the predictors. The only auxiliary model use, GPT-4 data cleaning, is validated against human annotations with near-perfect agreement (Cohen's kappa = 0.99) and does not enter the substantive conclusions. The reference list contains no load-bearing self-citations, and no uniqueness theorem or prior result by the authors is invoked to force the interpretation. The paper's own Limitations section explicitly concedes that training exposure cannot be ruled out and that deconfounding pulley count from MA remains for future work; this is an external-validity or underdetermination concern about whether TikZ parsing difficulty explains Study 3 chance performance, not a circular derivation. Because the claimed results reduce to independently defined stimulus properties and unmodified model outputs, no circular step is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper's central inference rests on domain assumptions about physics ground truth, the OOD status of TikZ stimuli, and the validity of output-layer responses as evidence about internal world models. No free parameters are fitted to data and no new entities are introduced.

assumptions (3)

domain assumption Mechanical advantage of an idealized pulley system is determined by the number of supporting rope segments and by whether the system transfers force to the weight
Used to define ground-truth MA in Studies 1-3 and the functional versus non-functional distinction in Studies 2 and 3.
domain assumption TikZ source code presented to the models is sufficiently out-of-distribution that correct answers cannot be retrieved from memorized training examples
Stated in Section 1.2 ('we deem it unlikely that they have been systematically trained on pulley system reasoning in the precise form presented') and acknowledged as untestable in the Limitations.
domain assumption The text responses of the LLMs are valid observations of their latent world-model representations, despite the acknowledged separation between output-layer tokens and internal weights
The paper's entire inference from output behavior to world-model content relies on this, and the Limitations explicitly note that deconfounding output tokens from internal representations is difficult.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLM world models are mental: Output layer evidence of brittle world model use in LLM mechanical reasoning." pith.science (2026). https://pith.science/paper/3LA5NQTW

@misc{pith2026250715521,
  author       = {Pith},
  title        = {Pith review of: LLM world models are mental: Output layer evidence of brittle world model use in LLM mechanical reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3LA5NQTW}},
  note         = {Machine review of arXiv:2507.15521}
}

read the original abstract

Do large language models (LLMs) construct and manipulate internal world models, or do they rely solely on statistical associations represented as output layer token probabilities? We adapt cognitive science methodologies from human mental models research to test LLMs on pulley system problems using TikZ-rendered stimuli. Study 1 examines whether LLMs can estimate mechanical advantage (MA). State-of-the-art models performed marginally but significantly above chance, and their estimates correlated significantly with ground-truth MA. Significant correlations between number of pulleys and model estimates suggest that models employed a pulley counting heuristic, without necessarily simulating pulley systems to derive precise values. Study 2 tested this by probing whether LLMs represent global features crucial to MA estimation. Models evaluated a functionally connected pulley system against a fake system with randomly placed components. Without explicit cues, models identified the functional system as having greater MA with F1=0.8, suggesting LLMs could represent systems well enough to differentiate jumbled from functional systems. Study 3 built on this by asking LLMs to compare functional systems with matched systems which were connected up but which transferred no force to the weight; LLMs identified the functional system with F1=0.46, suggesting random guessing. Insofar as they may generalize, these findings are compatible with the notion that LLMs manipulate internal world models, sufficient to exploit statistical associations between pulley count and MA (Study 1), and to approximately represent system components' spatial relations (Study 2). However, they may lack the facility to reason over nuanced structural connectivity (Study 3). We conclude by advocating the utility of cognitive scientific methods to evaluate the world-modeling capacities of artificial intelligence systems.

Figures

Figures reproduced from arXiv: 2507.15521 by the authors.

**Figure 1.** Example of a system with MA = 4, as TikZ source code excerpt (left) and rendered as a PDF (right). In all studies, LLMs were only shown TikZ source code as in left. Marcus [2020], and argue AI researchers should look to explicitly incorporate elements of human cognitive models to develop more flexible, trustworthy, and robust world modeling capabilities in AI systems Marcus [2020], Jabarian [2024]. If AI systems are… view at source ↗

**Figure 2.** Mean model estimates of MA over true MA. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Model estimates of MA regressed over objective pulley system features. Features where all [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: a) Study 2 stimuli: non-functional (left) and functional (right) paired pulley systems; b) In Study 3, non-functional diagrams were “connected up” but transferred no force to the weight. a non-functional diagram that was precisely matched for the number of pulleys, wei…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 3 canonical work pages

[9]

Daniel L

doi: 10.48550/arXiv.2303.08774. Daniel L. Schwartz and John B. Black. Analog imagery in mental model reasoning: Depictive models. Cognitive Psychology, 30(2):154–219,

arXiv doi:10.48550/arxiv.2303.08774
[11]

doi: 10.48550/arXiv.2303.12712. Roger N. Shepard and Jacqueline Metzler. Mental rotation of three-dimensional objects. Science, 171(3972):701–703,

arXiv doi:10.48550/arxiv.2303.12712
[13]

Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu

doi: 10.48550/arXiv.2211.00241. Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu. Language models meet world models: Embodied experiences enhance language models. Advances in Neural Information Processing Systems, 36:1–21,

arXiv doi:10.48550/arxiv.2211.00241
[14]

doi: 10.48550/arXiv.2305.10626. 10 6 Technical Appendices and Supplementary Material Table A1: Regression results by model, Study 1 (MA-only) gpt-4-0314 gpt-4-vision-preview claude-3-opus-20240229 gpt-4o-2024-11-20 Intercept 0.000 0.000 0.000 0.000 (0.032) (0.034) (0.034) (0.027) MA 0.424*** 0.200*** 0.283*** 0.651*** (0.032) (0.034) (0.034) (0.027) R2 0....

arXiv doi:10.48550/arxiv.2305.10626 2024
[1971]

Till Tantau

doi: 10.1126/science.171.3972.701. Till Tantau. The TikZ and PGF Packages: Manual for version 3.1.10,

doi:10.1126/science.171.3972.701
[1977]

doi: 10.2307/2529310

ISSN 0006341X. doi: 10.2307/2529310. Yann LeCun. A path towards autonomous machine intelligence. Open Review, (0.9.2):1–62,

doi:10.2307/2529310
[1988]

Zhiting Hu and Tianmin Shu

doi: 10.1016/0010-0285(88)90019-9. Zhiting Hu and Tianmin Shu. Language models, agent models, and world models: The LAW for machine reasoning and planning. arXiv,

work page doi:10.1016/0010-0285(88)90019-9
[1994]

doi: 10.3758/BF03200867

ISSN 0090502X. doi: 10.3758/BF03200867. Mary Hegarty and Kathryn Steinhoff. Individual differences in use of diagrams as external memory in mechanical reasoning. Learning and Individual Differences, 9(1):19–42,

work page doi:10.3758/bf03200867

Show all 14 references

[1996]

Sébastien Bubeck et al

doi: 10.1006/cogp.1996.0006. Sébastien Bubeck et al. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv,

1996
[1998]

Mary Hegarty

doi: 10.1006/cogp.1998.0694. Mary Hegarty. Mechanical reasoning by mental simulation. Trends in Cognitive Sciences, 8(6): 280–285,

1998
[2004]

doi: 10.1016/j.tics.2004.04.001

ISSN 13646613. doi: 10.1016/j.tics.2004.04.001. Mary Hegarty and Valerie K. Sims. Individual differences in mental animation during mechanical rea- soning. Memory & Cognition, 22(4):411–430,

2004 doi
[2020]

doi: 10.48550/arXiv.2002.06177. OpenAI. GPT-4 technical report. arXiv, 2023a. doi: 10.48550/arXiv.2303.08774. OpenAI. GPT-4V(ision) system card. OpenAI, 2023b. OpenAI. GPT-4o System Card. arXiv,

2002 doi arXiv
[2023]

Brian Jabarian

doi: 10.48550/arXiv.2312.05230. Brian Jabarian. Black boxes: Mental models and AI models. In Jonathan H Hamilton and Anindya Banerjee, editors, Oxford research encyclopedia of economics and finance . Oxford University Press,

doi arXiv
[2024]

Gary Marcus

doi: 10.48550/arXiv.2402.08955. Gary Marcus. The next decade in AI: Four steps towards robust artificial intelligence. arXiv,

doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [9]

Daniel L

doi: 10.48550/arXiv.2303.08774. Daniel L. Schwartz and John B. Black. Analog imagery in mental model reasoning: Depictive models. Cognitive Psychology, 30(2):154–219,

arXiv doi:10.48550/arxiv.2303.08774

[2] [11]

doi: 10.48550/arXiv.2303.12712. Roger N. Shepard and Jacqueline Metzler. Mental rotation of three-dimensional objects. Science, 171(3972):701–703,

arXiv doi:10.48550/arxiv.2303.12712

[3] [13]

Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu

doi: 10.48550/arXiv.2211.00241. Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu. Language models meet world models: Embodied experiences enhance language models. Advances in Neural Information Processing Systems, 36:1–21,

arXiv doi:10.48550/arxiv.2211.00241

[4] [14]

doi: 10.48550/arXiv.2305.10626. 10 6 Technical Appendices and Supplementary Material Table A1: Regression results by model, Study 1 (MA-only) gpt-4-0314 gpt-4-vision-preview claude-3-opus-20240229 gpt-4o-2024-11-20 Intercept 0.000 0.000 0.000 0.000 (0.032) (0.034) (0.034) (0.027) MA 0.424*** 0.200*** 0.283*** 0.651*** (0.032) (0.034) (0.034) (0.027) R2 0....

arXiv doi:10.48550/arxiv.2305.10626 2024

[5] [1971]

Till Tantau

doi: 10.1126/science.171.3972.701. Till Tantau. The TikZ and PGF Packages: Manual for version 3.1.10,

doi:10.1126/science.171.3972.701

[6] [1977]

doi: 10.2307/2529310

ISSN 0006341X. doi: 10.2307/2529310. Yann LeCun. A path towards autonomous machine intelligence. Open Review, (0.9.2):1–62,

doi:10.2307/2529310

[7] [1988]

Zhiting Hu and Tianmin Shu

doi: 10.1016/0010-0285(88)90019-9. Zhiting Hu and Tianmin Shu. Language models, agent models, and world models: The LAW for machine reasoning and planning. arXiv,

work page doi:10.1016/0010-0285(88)90019-9

[8] [1994]

doi: 10.3758/BF03200867

ISSN 0090502X. doi: 10.3758/BF03200867. Mary Hegarty and Kathryn Steinhoff. Individual differences in use of diagrams as external memory in mechanical reasoning. Learning and Individual Differences, 9(1):19–42,

work page doi:10.3758/bf03200867