pith. sign in

arxiv: 2606.13441 · v1 · pith:UTTVDK6Ynew · submitted 2026-06-11 · 💻 cs.AI · cs.CL

Why Sampling Is Not Choosing: Intentionality, Agency, and Moral Responsibility in Large Language Models

Joseph Keshet This is my paper

Pith reviewed 2026-06-27 07:04 UTC · model grok-4.3

classification 💻 cs.AI cs.CL
keywords large language modelsmoral responsibilityintentionalityagencyfree willsamplingAI ethics
0
0 comments X

The pith

Moral responsibility requires intrinsic intentionality and self-attributed commitments that large language models lack.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper maintains that moral responsibility depends on commitment-bearing agency, which in turn rests on intrinsic intentionality and the self-attribution of actions as one's own. LLMs operate through probabilistic input-output mappings learned from data, producing outputs that appear intentional but remain derived from external sources rather than owned or guided by internal reasons. Stochastic sampling introduces variability yet does not create authorship or choice. This distinction matters because many current discussions treat LLM outputs as if they carried the kind of responsibility that only intrinsic agency can support. The paper examines and rejects attempts to ground agency in the intentional stance, functional performance, or compatibilist accounts of free will.

Core claim

Moral responsibility requires commitment-bearing agency grounded in intrinsic intentionality and self-attributed action, and such agency constitutes the form of free will relevant to responsibility. Although LLMs generate coherent and normatively evaluable outputs, their operation is fully characterized by probabilistic input-output mappings learned from data. Their apparent intentionality is derived rather than intrinsic, and their outputs are neither owned as commitments nor guided by reasons. Variability introduced by stochastic sampling does not amount to choice or authorship.

What carries the argument

Intrinsic intentionality, the form of directedness that allows an agent to own outputs as personal commitments guided by reasons, as opposed to derived intentionality supplied by training data or external design.

If this is right

  • LLMs cannot be held morally responsible for their outputs.
  • Functional descriptions of LLM behavior fail to establish genuine agency.
  • Compatibilist accounts of free will do not apply to systems whose intentionality remains derived.
  • Moral reasoning appearing in model outputs does not imply that the model itself bears responsibility for those outputs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Only entities capable of producing non-derived intentionality through their own causal history could qualify as moral agents under this standard.
  • Legal and ethical frameworks that assign responsibility for AI-generated content would need to locate it entirely in human designers or users.
  • Design efforts aimed at creating artificial moral agents would have to introduce mechanisms for self-attribution that go beyond statistical pattern completion.

Load-bearing premise

Only intrinsic intentionality, as opposed to derived or functional intentionality, can ground moral responsibility and agency.

What would settle it

An LLM that can attribute one of its own outputs to itself as a personal commitment for reasons that cannot be reduced to patterns in its training data would challenge the claim.

read the original abstract

Recent advances in large language models (LLMs) have prompted claims that such systems exhibit agency or qualify as moral agents. This paper argues that these attributions are misguided. We maintain that moral responsibility requires commitment-bearing agency grounded in intrinsic intentionality and self-attributed action, and that such agency constitutes the form of free will relevant to responsibility. Although LLMs generate coherent and normatively evaluable outputs, their operation is fully characterized by probabilistic input-output mappings learned from data. Their apparent intentionality is derived rather than intrinsic, and their outputs are neither owned as commitments nor guided by reasons. Variability introduced by stochastic sampling does not amount to choice or authorship. We address objections from the intentional stance, functionalism, compatibilism, and the presence of moral reasoning in model outputs, arguing that none suffice to establish genuine agency.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The paper claims that LLMs lack moral responsibility and genuine agency because their operation consists solely of probabilistic input-output mappings that yield only derived intentionality, not intrinsic intentionality or self-attributed commitments; stochastic sampling does not constitute choice or authorship. It maintains that moral responsibility requires commitment-bearing agency grounded in intrinsic intentionality and addresses objections from the intentional stance, functionalism, compatibilism, and apparent moral reasoning in outputs by arguing that none establish the required form of agency.

Significance. If the central distinction between intrinsic and derived intentionality is accepted as decisive, the paper offers a clear conceptual framework for denying moral agency to current LLMs, with potential relevance to AI ethics debates. The manuscript engages standard philosophical objections directly and maintains internal consistency within its chosen framework, though its impact hinges on the contested premise rather than new empirical or formal results.

major comments (1)
  1. [discussion of functionalism and intentional stance objections] The argument against functionalism (addressed in the section responding to functionalist objections) rests on the assertion that derived intentionality from probabilistic mappings cannot ground moral responsibility or agency, but provides no independent demonstration that a complete functional description of LLM behavior would still fail to satisfy the criteria for commitment-bearing agency even if granted.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the detailed and constructive report. The paper develops a philosophical argument that LLMs lack the commitment-bearing agency required for moral responsibility on the grounds that their operation yields only derived intentionality. We respond to the major comment on the functionalism objection below.

read point-by-point responses

  1. Referee: The argument against functionalism (addressed in the section responding to functionalist objections) rests on the assertion that derived intentionality from probabilistic mappings cannot ground moral responsibility or agency, but provides no independent demonstration that a complete functional description of LLM behavior would still fail to satisfy the criteria for commitment-bearing agency even if granted.

    Authors: We appreciate the referee highlighting this aspect of the argument. The manuscript explicitly characterizes the complete functional description of an LLM as a set of learned probabilistic input-output mappings. Under the framework developed in the paper, such mappings produce only derived intentionality because they lack any internal mechanism for self-attribution of commitments or reasons-responsive ownership of outputs. Commitment-bearing agency, by contrast, requires intrinsic intentionality that cannot be reduced to externally trained statistical correlations. Thus the functional description itself entails the absence of the required form of agency; no further independent demonstration is needed beyond showing that the functional profile is exhausted by derived intentionality. We are nevertheless willing to add a short clarifying paragraph in the functionalism section to make this entailment more explicit. revision: partial

Circularity Check

0 steps flagged

No circularity; conceptual argument rests on explicit philosophical premises without self-referential reduction.

full rationale

The paper advances a philosophical position that moral responsibility requires intrinsic intentionality and commitment-bearing agency, which LLMs lack because their behavior is characterized by probabilistic mappings. This distinction is presented as a premise drawn from standard philosophy rather than derived from any equations, fitted parameters, or self-citations within the paper. No load-bearing step reduces the conclusion to its inputs by construction, and the text contains no mathematical derivations, empirical fits, or citations to the author's prior work that would trigger the enumerated circularity patterns. The argument is self-contained against external benchmarks in the form of cited philosophical literature.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on one domain assumption about the requirements for moral responsibility; no free parameters or invented entities are introduced.

axioms (1)
  • domain assumption Moral responsibility requires commitment-bearing agency grounded in intrinsic intentionality and self-attributed action
    This premise is stated in the abstract and used to evaluate whether LLM outputs qualify as actions for which the model can be responsible.

pith-pipeline@v0.9.1-grok · 5664 in / 1238 out tokens · 24852 ms · 2026-06-27T07:04:07.313693+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 5 canonical work pages · 1 internal anchor

  1. [1]

    1995 , note =

    Franz Brentano , title =. 1995 , note =

    1995

  2. [2]

    1983 , publisher =

    Edmund Husserl , title =. 1983 , publisher =

    1983

  3. [3]

    Searle , title =

    John R. Searle , title =. Behavioral and Brain Sciences , volume =

  4. [4]

    Physica D: Nonlinear Phenomena , volume =

    Stevan Harnad , title =. Physica D: Nonlinear Phenomena , volume =

  5. [5]

    Brandom , title =

    Robert B. Brandom , title =

  6. [6]

    P. F. Strawson , title =. Proceedings of the British Academy , volume =

  7. [7]

    Bernard Williams , title =

  8. [8]

    1943 , publisher =

    Jean-Paul Sartre , title =. 1943 , publisher =

    1943

  9. [9]

    Frankfurt , title =

    Harry G. Frankfurt , title =. Journal of Philosophy , volume =. 1969 , doi =

    1969

  10. [10]

    Searle , title =

    John R. Searle , title =

  11. [11]

    Gomez and

    Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and. Attention Is All You Need , booktitle =

  12. [12]

    John Martin Fischer and Mark Ravizza , title =

  13. [13]

    Gary Watson , title =

  14. [14]

    David Velleman , title =

    J. David Velleman , title =

  15. [15]

    Bratman , title =

    Michael E. Bratman , title =

  16. [16]

    Robert Kane , title =

  17. [17]

    John Martin Fischer , title =

  18. [18]

    Bender and Timnit Gebru and Angelina McMillan-Major and Shmargaret Shmitchell , title =

    Emily M. Bender and Timnit Gebru and Angelina McMillan-Major and Shmargaret Shmitchell , title =. Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT) , year =

  19. [19]

    Minds and Machines , volume =

    Luciano Floridi and Massimo Chiriatti , title =. Minds and Machines , volume =

  20. [20]

    John Haugeland , title =

  21. [21]

    Manuel Vargas , title =

  22. [22]

    International Conference on Learning Representations (ICLR) , year =

    Ari Holtzman and Jan Buys and Li Du and Maxwell Forbes and Yejin Choi , title =. International Conference on Learning Representations (ICLR) , year =

  23. [23]

    Williams and David Zipser , title =

    Ronald J. Williams and David Zipser , title =. Neural Computation , volume =

  24. [24]

    Course in General Linguistics , note =

  25. [25]

    Searle, John R. , year=. Intentionality: An Essay in the Philosophy of Mind , publisher=

  26. [26]

    Dennett , title =

    Daniel C. Dennett , title =

  27. [27]

    Art, Mind, and Religion , year =

    Hilary Putnam , title =. Art, Mind, and Religion , year =

  28. [28]

    Fodor , title =

    Jerry A. Fodor , title =

  29. [29]

    AI and Ethics , volume =

    Mandy Zafar , title =. AI and Ethics , volume =. 2025 , pages =

    2025

  30. [30]

    arXiv preprint arXiv:2507.02788 , year =

    Joseph Boland , title =. arXiv preprint arXiv:2507.02788 , year =

    work page arXiv

  31. [31]

    Moral Responsibility for AI Systems , volume =

    Beckers, Sander , booktitle =. Moral Responsibility for AI Systems , volume =

  32. [32]

    Human Perceptions on Moral Responsibility of AI: A Case Study in AI-Assisted Bail Decision-Making , journal =

    Gabriel Lima and Nina Grgi\'c-Hla. Human Perceptions on Moral Responsibility of AI: A Case Study in AI-Assisted Bail Decision-Making , journal =

  33. [33]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Jason Wei and Xuezhi Wang and Dale Schuurmans and Maarten Bosma and Brian Ichter and Fei Xia and Ed Chi and Quoc Le and Denny Zhou , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  34. [34]

    International Conference on Learning Representations (ICLR) , year =

    Xuezhi Wang and Jason Wei and Dale Schuurmans and Quoc Le and Ed Chi and Denny Zhou , title =. International Conference on Learning Representations (ICLR) , year =

  35. [35]

    Tree of Thoughts: Deliberate Problem Solving with Large Language Models

    Shunyu Yao and Dian Yu and Jeffrey Zhao and Izhak Shafran and Thomas Griffiths and Yuan Cao and Karthik Narasimhan , title =. arXiv preprint arXiv:2305.10601 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  36. [36]

    European Conference on Computer Vision , pages=

    Paying more attention to image: A training-free method for alleviating hallucination in lvlms , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  37. [37]

    arXiv preprint arXiv:2410.03577 , year=

    Look twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language models , author=. arXiv preprint arXiv:2410.03577 , year=

    work page arXiv

  38. [38]

    arXiv preprint arXiv:2506.07233 , year=

    Reducing object hallucination in large audio-language models via audio-aware decoding , author=. arXiv preprint arXiv:2506.07233 , year=

    work page arXiv

  39. [39]

    arXiv preprint arXiv:2603.21687 (2026) 27

    Mirage The Illusion of Visual Understanding , author=. arXiv preprint arXiv:2603.21687 , year=

    work page arXiv

  40. [40]

    Science and Engineering Ethics , year =

    Constantinescu, Mihaela and Kaptein, Muel , title =. Science and Engineering Ethics , year =

  41. [41]

    AI , VOLUME =

    Woodruff, Earl and Hewitt, Jim , TITLE =. AI , VOLUME =. 2026 , NUMBER =

    2026

  42. [42]

    Proceedings of the National Academy of Sciences , volume =

    Vanessa Cheung and Maximilian Maier and Falk Lieder , title =. Proceedings of the National Academy of Sciences , volume =. 2025 , doi =

    2025

  43. [43]

    Frontiers in Artificial Intelligence , VOLUME=

    Barabadi, Elyas and Fotuhabadi, Zahra and Arghavan, Amanollah and Booth, James R , TITLE=. Frontiers in Artificial Intelligence , VOLUME=. 2026 , DOI=

    2026

  44. [44]

    Position: Trustworthy

    Yedi Zhang and Yufan Cai and Xinyue Zuo and Xiaokun Luan and Kailong Wang and Zhe Hou and Yifan Zhang and Zhiyuan Wei and Meng Sun and Jun Sun and Jing Sun and Jin Song Dong , booktitle=. Position: Trustworthy. 2025 , url=

    2025