The LLM Fallacy: Misattribution in AI-Assisted Cognitive Workflows

Hanau Yi; Harin Yu; Hyunwoo Kim

arxiv: 2604.14807 · v2 · submitted 2026-04-16 · 💻 cs.AI · cs.CL

The LLM Fallacy: Misattribution in AI-Assisted Cognitive Workflows

Hyunwoo Kim , Harin Yu , Hanau Yi This is my paper

Pith reviewed 2026-05-10 10:29 UTC · model grok-4.3

classification 💻 cs.AI cs.CL

keywords LLM fallacycognitive attribution errorAI-assisted workflowsautomation biasself-perceptionhuman-AI collaborationAI literacycognitive offloading

0 comments

The pith

Individuals using large language models misinterpret AI-generated outputs as proof of their own independent competence, creating a gap between perceived and actual capability.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper defines the LLM fallacy as a cognitive attribution error where users credit LLM-assisted results in writing, coding, analysis, and other tasks to their personal skills instead of the model's contribution. This occurs because LLMs operate with high fluency and minimal visible effort, making the human-machine boundary hard to detect. A sympathetic reader would care because the error could distort self-assessment in learning, professional evaluation, and daily decision-making. The authors place this idea among related concepts like automation bias and cognitive offloading but treat it as distinct to AI workflows. They outline mechanisms, a typology across domains, and needed empirical checks without claiming new data.

Core claim

The LLM fallacy is a cognitive attribution error in which individuals misinterpret LLM-assisted outputs as evidence of their own independent competence, producing a systematic divergence between perceived and actual capability. The opacity, fluency, and low-friction interaction patterns of LLMs obscure the boundary between human and machine contribution, leading users to infer competence from outputs rather than from the processes that generate them. The paper situates the fallacy within literature on automation bias and human-AI collaboration while distinguishing it as an attributional distortion specific to generative AI, and it proposes a conceptual framework plus a typology of how the误误误

What carries the argument

The LLM fallacy itself, defined as the misattribution of AI outputs to personal competence due to obscured human-machine boundaries in fluent, low-friction interactions.

If this is right

Educational assessments must separate demonstrated process from final output to avoid inflated perceptions of student mastery.
Hiring practices that rely on AI-assisted samples may select for perceived rather than actual expertise.
AI literacy efforts need to include training on recognizing external contributions to one's own work products.
Professional fields using generative tools require updated metrics that track the human process alongside the delivered result.
Longer-term skill retention may decline if users consistently misread assisted performance as unaided ability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Unchecked, the fallacy could accelerate deskilling in domains where repeated LLM use replaces practice without users noticing the substitution.
Measurement of expertise in AI-augmented roles may require new protocols that log the sequence of human edits versus model generations.
The pattern resembles earlier cognitive offloading with calculators or search engines but may prove stronger because LLMs produce fluent prose and code rather than isolated facts.
Empirical validation could begin with before-and-after self-assessment surveys on narrowly scoped tasks such as essay revision or code debugging.

Load-bearing premise

LLMs' opacity, fluency, and low-friction design cause users to attribute the resulting outputs to their own independent effort rather than to the model.

What would settle it

A controlled study in which participants complete identical cognitive tasks with and without LLM assistance, then provide self-ratings of competence; the claim would be falsified if self-ratings show no systematic elevation in the LLM condition despite performance differences.

Figures

Figures reproduced from arXiv: 2604.14807 by Hanau Yi, Harin Yu, Hyunwoo Kim.

read the original abstract

The rapid integration of large language models (LLMs) into everyday workflows has transformed how individuals perform cognitive tasks such as writing, programming, analysis, and multilingual communication. While prior research has focused on model reliability, hallucination, and user trust calibration, less attention has been given to how LLM usage reshapes users' perceptions of their own capabilities. This paper introduces the LLM fallacy, a cognitive attribution error in which individuals misinterpret LLM-assisted outputs as evidence of their own independent competence, producing a systematic divergence between perceived and actual capability. We argue that the opacity, fluency, and low-friction interaction patterns of LLMs obscure the boundary between human and machine contribution, leading users to infer competence from outputs rather than from the processes that generate them. We situate the LLM fallacy within existing literature on automation bias, cognitive offloading, and human-AI collaboration, while distinguishing it as a form of attributional distortion specific to AI-mediated workflows. We propose a conceptual framework of its underlying mechanisms and a typology of manifestations across computational, linguistic, analytical, and creative domains. Finally, we examine implications for education, hiring, and AI literacy, and outline directions for empirical validation. We also provide a transparent account of human-AI collaborative methodology. This work establishes a foundation for understanding how generative AI systems not only augment cognitive performance but also reshape self-perception and perceived expertise.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper names an LLM-specific attribution error and sketches a typology, but offers only conceptual argument with no data or tests.

read the letter

The main takeaway is that this work labels a plausible pattern in LLM use: users credit themselves for outputs that the model largely produced, and it maps that pattern across task types while calling for real studies later. The authors pull from automation bias and cognitive offloading literature, then argue that LLM fluency and opacity make the machine contribution hard to notice, so people update their self-view upward from the finished product rather than from their own process. They add a typology covering computational, linguistic, analytical, and creative work plus notes on education and hiring effects. They also state upfront how they used AI in their own writing, which keeps the account clean. That framing is the useful part; it gives a compact way to talk about self-assessment distortion in current tools without claiming the idea is entirely new. The distinction from older bias work is drawn plainly enough to be usable in discussion. The clear limitation is the absence of any measurement. The claimed systematic gap between perceived and actual competence is reasoned from existing papers rather than shown here, and the listed mechanisms (opacity, low friction) stay at the level of plausible mediators without evidence on how much they actually drive the effect. No experiments, no user data, no checks on whether the divergence appears in practice. Because the paper itself flags the need for empirical validation, this is not hidden, but it means the central claim cannot be treated as established. Readers working on human-AI collaboration or on how tools affect learning and hiring will get a tidy organizing frame and a set of questions worth testing. Someone already deep in the cited bias literature will see an incremental extension rather than a break. I would bring it to a reading group to talk through the typology and what a first study might look like. It is not something I would cite yet as a result. It does deserve peer review so referees can press on the framework and suggest concrete validation steps.

Referee Report

0 major / 3 minor

Summary. The manuscript introduces the 'LLM fallacy' as a cognitive attribution error in which individuals misinterpret LLM-assisted outputs as evidence of their own independent competence, resulting in a divergence between perceived and actual capability. It situates this concept within literature on automation bias, cognitive offloading, and human-AI collaboration, proposes mechanisms involving opacity, fluency, and low-friction interactions, offers a typology of manifestations in computational, linguistic, analytical, and creative domains, discusses implications for education, hiring, and AI literacy, and outlines directions for empirical validation. The paper also includes a transparent description of its human-AI collaborative methodology.

Significance. If substantiated through empirical research, the LLM fallacy concept could significantly advance understanding of how generative AI affects self-perception and expertise attribution in cognitive workflows. This has potential implications for designing AI systems that promote accurate self-assessment, informing educational curricula on AI literacy, and refining hiring practices that may over-rely on AI-augmented outputs. The paper's strength lies in its clear conceptual framework and the explicit account of collaborative methodology, which enhances transparency in AI-related conceptual work.

minor comments (3)

The typology of manifestations across computational, linguistic, analytical, and creative domains is outlined at a high level; adding one or two concrete illustrative examples per domain would make the distinctions from existing biases more tangible and useful for guiding future empirical tests.
The implications section for education, hiring, and AI literacy would benefit from explicit cross-references to specific prior empirical studies on automation bias and cognitive offloading to better anchor the practical recommendations.
Ensure consistent terminology when referring to the core construct (e.g., 'LLM fallacy' versus 'attributional distortion') across the abstract, framework, and conclusion sections.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive and accurate summary of our manuscript, the recognition of its conceptual contribution, and the recommendation for minor revision. We appreciate the emphasis on the potential implications for AI literacy and self-assessment, as well as the note on methodological transparency.

Circularity Check

0 steps flagged

No significant circularity in conceptual proposal

full rationale

The manuscript is a conceptual proposal that defines and situates the LLM fallacy as a new attributional distortion, drawing on external literature on automation bias, cognitive offloading, and human-AI collaboration. It advances the central claim as an argument with proposed mechanisms and a typology, explicitly calling for future empirical validation rather than deriving results from equations, fitted parameters, or self-citations. No load-bearing steps reduce by construction to the paper's own inputs; the framework remains self-contained against external benchmarks with no self-definitional, fitted-prediction, or uniqueness-imported circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on the untested domain assumption that LLM interaction patterns reliably obscure human-machine boundaries and produce misattribution; no free parameters or invented physical entities are introduced, but the LLM fallacy itself functions as a new postulated cognitive mechanism without independent evidence.

axioms (1)

domain assumption LLM opacity, fluency, and low-friction patterns obscure the boundary between human and machine contribution
Invoked directly in the abstract as the mechanism producing the fallacy

invented entities (1)

LLM fallacy no independent evidence
purpose: To label and organize the described misattribution error across cognitive domains
Newly named conceptual entity introduced to distinguish from existing biases

pith-pipeline@v0.9.0 · 5545 in / 1261 out tokens · 51649 ms · 2026-05-10T10:29:19.850865+00:00 · methodology

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

The efficiency-gain illusion: People underestimate the rate of AI use and overestimate its benefits on simple tasks
cs.CY 2026-05 accept novelty 6.0

Three pre-registered studies with 2691 participants show people underestimate their AI usage rate and overestimate efficiency gains on simple tasks, with prior use entrenching further adoption.

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages · cited by 1 Pith paper

[1]

https://doi.org/10.17705/1thci. 00131. Sieck, W. R., & Arkes, H. R. (2005). The recalci- trance of overconfidence and its contribution to de- cision aid neglect.Journal of Behavioral Decision Making, 18(1), 29–53. https://doi.org/10. 1002/bdm.486. Sloman, S. A. (1996). The empirical case for two systems of reasoning.Psychological Bulletin, 119(1), 3–22. h...

work page doi:10.17705/1thci 2005
[2]

1996, ARA&A, 34, 645, doi:10.1146/annurev

https://doi.org/10.1146/annurev. psych.55.090902.141954. Zhang, Z., Wang, C., Wang, Y ., Shi, E., Ma, Y ., Zhong, W., Chen, J., Mao, M., & Zheng, Z. (2025). LLM hallucinations in practical code generation: Phe- nomena, mechanism, and mitigation.Proceedings of the ACM on Software Engineering, 2(ISSTA), 481– 503.https://doi.org/10.1145/3728894. 13

work page doi:10.1146/annurev 2025

[1] [1]

https://doi.org/10.17705/1thci. 00131. Sieck, W. R., & Arkes, H. R. (2005). The recalci- trance of overconfidence and its contribution to de- cision aid neglect.Journal of Behavioral Decision Making, 18(1), 29–53. https://doi.org/10. 1002/bdm.486. Sloman, S. A. (1996). The empirical case for two systems of reasoning.Psychological Bulletin, 119(1), 3–22. h...

work page doi:10.17705/1thci 2005

[2] [2]

1996, ARA&A, 34, 645, doi:10.1146/annurev

https://doi.org/10.1146/annurev. psych.55.090902.141954. Zhang, Z., Wang, C., Wang, Y ., Shi, E., Ma, Y ., Zhong, W., Chen, J., Mao, M., & Zheng, Z. (2025). LLM hallucinations in practical code generation: Phe- nomena, mechanism, and mitigation.Proceedings of the ACM on Software Engineering, 2(ISSTA), 481– 503.https://doi.org/10.1145/3728894. 13

work page doi:10.1146/annurev 2025