Why LLMs Hallucinate on Structured Knowledge: A Mechanistic Analysis of Reasoning over Linearized Representations

Jinda Han; Kenan Kamel A Alghythee; Langzhou He; Philip S. Yu; Shanghao Li; Yibo Wang; Yuanjie Zhu; Zihe Song

Why LLMs Hallucinate on Structured Knowledge: A Mechanistic Analysis of Reasoning over Linearized Representations

T0 review · 2 major / 0 minor · reviewed 2026-06-29 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs hallucinate on linearized graphs and tables because attention locks onto structural shortcuts while feed-forward layers fail to ground the supplied knowledge.

desk verdict The paper documents consistent correlations between attention concentration on structural shortcuts, feed-forward grounding failures, and hallucinations on linearized graphs/tables, but provides no causal tests. read the letter →

arxiv 2605.26362 v1 pith:BDCWMAH7 submitted 2026-05-25 cs.CL cs.AI

Shanghao Li , Jinda Han , Yibo Wang , Yuanjie Zhu , Zihe Song , Langzhou He , Kenan Kamel A Alghythee , Philip S. Yu This is my paper

classification cs.CLcs.AI

keywords LLMhallucinationsstructuredknowledgemechanisticinterpretabilityattentionmechanismsfeed-forwardlayerslinearizedrepresentationsgraphreasoningtabular

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper examines why large language models still generate incorrect answers even when given complete structured knowledge in sequential form. It traces the errors to repeatable internal processing patterns instead of unpredictable noise. Attention layers focus heavily on simple cue-like elements of the input structure rather than spreading across the full sequence. At the same time, feed-forward layers do not successfully bind the external facts into the model's representations, so the output falls back on patterns learned during pretraining. These same patterns appear in both single-step and multi-step graph tasks as well as in table-based reasoning.

What carries the argument

The paired internal dynamics of attention concentration on shortcut structural cues and semantic-grounding failure inside feed-forward layers when processing linearized structured inputs.

What would settle it

A controlled intervention that forces attention to distribute evenly across the full context or forces feed-forward layers to bind the external knowledge, followed by a drop in hallucination rate on the same inputs, would support the claim; no drop would undermine it.

Watch

Extended reading notes

Core claim

Hallucinations arise from systematic internal dynamics rather than random noise. Attention disproportionately concentrates toward shortcut-like structural cues rather than distributing across the full context. Feed-forward representations fail to ground the provided knowledge, causing the model to revert to parametric memory. Hallucination is consistently associated with failures in semantic grounding within feed-forward layers, while attention allocation exhibits greater task-dependent variability. These mechanistic patterns generalize beyond single-hop graphs to multi-hop and tabular settings.

Load-bearing premise

The measured associations between attention shortcuts, feed-forward grounding failures, and hallucinated outputs reflect causal mechanisms rather than mere correlations.

Editorial extensions

If this is right

Hallucination is tied more reliably to feed-forward grounding failures than to variability in attention patterns.
The identified dynamics appear in single-hop graphs, multi-hop graphs, and tabular data alike.
The patterns support detection of likely hallucinations across different structured-knowledge formats.
Models default to parametric memory once the supplied knowledge is not grounded in feed-forward representations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Targeted changes to how feed-forward layers integrate external tokens could reduce reliance on parametric memory.
Detection methods based on monitoring feed-forward grounding signals may transfer to new structured-input formats.
The task-dependent nature of attention suggests that mitigation strategies focused only on attention may need per-task tuning.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The core observation is that hallucinations on structured inputs track with attention latching onto shortcut cues instead of spreading across the full linearized context, plus feed-forward layers that do not integrate the supplied knowledge and fall back to parametric memory. These patterns appear in single-hop, multi-hop, and tabular cases, which is the main new piece.

The work is useful for showing that the failures are systematic rather than noise, and the cross-format consistency could support detection ideas. The measurements of attention weights and representation similarity seem reproducible on the surface.

The soft spot is that the evidence stays observational. The abstract and reported results link the internal dynamics to hallucinated outputs but do not include activation patching, attention ablation, or editing experiments that would test whether fixing those dynamics actually lowers hallucination rates on held-out inputs. Without that step the claim that the dynamics cause the problem rather than co-occur with it stays untested.

Methods details such as exact datasets, controls, and statistical thresholds are not visible in the abstract, so the robustness of the associations cannot be judged yet. This is a standard limitation at the abstract stage but matters for the central argument.

The paper is aimed at researchers doing mechanistic interpretability on knowledge-augmented LLMs. It deserves a serious referee to check the experimental controls and push on the causality gap; the patterns are worth discussing even if the causal story needs more work.

Referee Report

2 major / 0 minor

Summary. The paper claims that hallucinations in LLMs when reasoning over linearized structured knowledge (graphs, tables) arise from systematic internal dynamics rather than random noise: attention disproportionately concentrates on shortcut-like structural cues instead of the full context, and feed-forward representations fail to ground the provided knowledge, causing reversion to parametric memory. Hallucination is consistently associated with failures in semantic grounding within feed-forward layers (with attention allocation showing greater task-dependent variability), and these patterns generalize beyond single-hop graphs to multi-hop and tabular settings, supporting hallucination detection.

Significance. If the reported associations hold under scrutiny, the work provides mechanistic insights into LLM failures on knowledge-grounded tasks and identifies patterns that could support improved detection across structured formats. The generalization claim across single-hop, multi-hop, and tabular settings is a potential strength if supported by consistent measurements. However, the absence of causal tests limits the ability to conclude that the identified dynamics drive hallucinations.

major comments (2)

[Abstract] Abstract: the central claims that hallucinations 'arise from' specific internal dynamics and that patterns 'generalize' and 'enable effective hallucination detection' are stated without any experimental details, dataset descriptions, controls, or statistical tests, so the robustness of the measurements cannot be assessed.
[Experimental results] Experimental results (throughout): the manuscript reports consistent associations between attention concentration on structural cues, feed-forward grounding failures, and hallucinated outputs across settings, but describes only observational measurements of attention weights and representation similarity; no activation patching, attention ablation, or representation editing experiments are described that would test whether altering those dynamics changes hallucination rates on held-out examples with fixed input knowledge.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. Our work is observational in nature, identifying consistent associations between internal model dynamics and hallucination rates. We address the points below by proposing targeted revisions to language and presentation while preserving the reported measurements.

read point-by-point responses

Referee: [Abstract] Abstract: the central claims that hallucinations 'arise from' specific internal dynamics and that patterns 'generalize' and 'enable effective hallucination detection' are stated without any experimental details, dataset descriptions, controls, or statistical tests, so the robustness of the measurements cannot be assessed.

Authors: We agree the abstract uses high-level phrasing without experimental specifics. We will revise it to replace causal language ('arise from') with 'are associated with', note the three settings studied (single-hop graphs, multi-hop graphs, tables), and indicate that detection results are based on the reported correlations. Full dataset descriptions, controls, and statistical details remain in Sections 3–5 of the main text. revision: yes
Referee: [Experimental results] Experimental results (throughout): the manuscript reports consistent associations between attention concentration on structural cues, feed-forward grounding failures, and hallucinated outputs across settings, but describes only observational measurements of attention weights and representation similarity; no activation patching, attention ablation, or representation editing experiments are described that would test whether altering those dynamics changes hallucination rates on held-out examples with fixed input knowledge.

Authors: The referee is correct that the study relies on observational measurements (attention weights, representation similarity) rather than causal interventions. We will revise the manuscript throughout to consistently describe the findings as associations and correlations, remove any implication that the dynamics are shown to drive hallucinations, and explicitly note the absence of patching or ablation experiments as a limitation. The strength of the work lies in the consistent patterns observed across three distinct structured-knowledge formats; we do not claim these patterns have been causally validated. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: purely empirical observational study

full rationale

The paper reports mechanistic observations from attention and representation measurements across graph and table tasks. No equations, parameter fits, derivations, or self-citation chains appear in the provided abstract or described results. All claims are framed as consistent associations between internal dynamics and hallucination outputs rather than quantities defined in terms of themselves or predictions forced by fitted inputs. This is the standard case of an empirical paper whose central content does not reduce to its own inputs by construction.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Abstract contains no mathematical derivations, free parameters, axioms, or newly postulated entities; insufficient information to populate the ledger.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Why LLMs Hallucinate on Structured Knowledge: A Mechanistic Analysis of Reasoning over Linearized Representations." pith.science (2026). https://pith.science/paper/BDCWMAH7

@misc{pith2026260526362,
  author       = {Pith},
  title        = {Pith review of: Why LLMs Hallucinate on Structured Knowledge: A Mechanistic Analysis of Reasoning over Linearized Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BDCWMAH7}},
  note         = {Machine review of arXiv:2605.26362}
}

read the original abstract

In many reasoning tasks, large language models (LLMs) rely on structured external knowledge, such as graphs and tables, which is typically linearized into sequential token representations. However, even when sufficient knowledge is available, LLMs can still produce hallucinated outputs, and the underlying mechanisms behind such failures remain poorly understood. We investigate these mechanisms and find that hallucinations arise from systematic internal dynamics rather than random noise. First, attention disproportionately concentrates toward shortcut-like structural cues rather than distributing across the full context. Second, feed-forward representations fail to ground the provided knowledge, causing the model to revert to parametric memory. Moreover, our results indicate that hallucination is consistently associated with failures in semantic grounding within feed-forward layers, while attention allocation exhibits greater task-dependent variability. Finally, we show that these mechanistic patterns generalize beyond single-hop graphs to multi-hop and tabular settings, enabling effective hallucination detection across structured knowledge formats.

Figures

Figures reproduced from arXiv: 2605.26362 by the authors.

**Figure 2.** Box plots of SSR and SAS scores for hallu [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Feature space and correlation. (Left) Scatter plot of SSR vs. SAS shows overlapping but distinguishable clusters. (Right) Correlation matrix shows weak dependencies between features and label, suggesting complementary information. 5.3 Quadrant-Based Case Study of Hallucination Patterns To test H3 (Joint Failure Modes), we examine the interaction between attention focus and semantic alignment by partitioning model o… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Hallucination Analysis via SSR × SAS Quadrant Classification. Each point represents a model output. Hallucinated answers are marked with “×” [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Prompt template for graph-based question an [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages

[1]

the moon is made of marshmallows

KG-GPT: A general framework for reasoning on knowledge graphs using large language models. InFindings of the Association for Computational Lin- guistics: EMNLP 2023, pages 9410–9421, Singapore. Association for Computational Linguistics. Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2024. Analyzing feed-forward blocks in transformers th...

work page arXiv 2023
[2]

Alon Talmor and Jonathan Berant

Redeep: Detecting hallucination in retrieval- augmented generation via mechanistic interpretabil- ity.arXiv preprint arXiv:2410.11414. Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Hum...

work page arXiv 2018

Pith tools

Reviewed June 29, 2026 · model on record in the stance chip above.

[1] [1]

the moon is made of marshmallows

KG-GPT: A general framework for reasoning on knowledge graphs using large language models. InFindings of the Association for Computational Lin- guistics: EMNLP 2023, pages 9410–9421, Singapore. Association for Computational Linguistics. Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2024. Analyzing feed-forward blocks in transformers th...

work page arXiv 2023

[2] [2]

Alon Talmor and Jonathan Berant

Redeep: Detecting hallucination in retrieval- augmented generation via mechanistic interpretabil- ity.arXiv preprint arXiv:2410.11414. Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Hum...

work page arXiv 2018