pith. sign in

arxiv: 2605.28908 · v1 · pith:WLRPO22Xnew · submitted 2026-05-27 · 💻 cs.HC · cs.CY

Who Does Your AI Work For? Designing Conversational Agents as Digital Fiduciaries

Jacob Erickson This is my paper

Pith reviewed 2026-06-29 10:19 UTC · model grok-4.3

classification 💻 cs.HC cs.CY
keywords conversational agentsfiduciary dutyAI designdigital fiduciariestrustaccountabilityAI ethics
0
0 comments X

The pith

Conversational agents should be designed as digital fiduciaries to act in users' best interests.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that conversational agents, integrated into intimate user activities and holding sensitive data, require a fiduciary standard of care like that of lawyers or financial advisors. It proposes fiduciary design as the guiding principle for AI development. This approach would unify trust and accountability in a single design and legal framework. Readers might care because current AI alignment may not suffice for anthropomorphic agents making personal decisions.

Core claim

Conversational agents should be held to a similar standard as human fiduciaries, with fiduciary design introduced as a guiding principle to unify conversational AI trust and accountability into a single design and legal paradigm.

What carries the argument

Fiduciary design, the application of the duty to act in the client's best interests to conversational AI systems.

If this is right

  • Trust and accountability become unified under one paradigm.
  • Design choices enforce obligations to prioritize user interests.
  • Agents gain a standard of care matching their access to sensitive data.
  • Development shifts to best-interest obligations rather than mere goal alignment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This may require new laws to define AI fiduciary responsibilities.
  • Technical implementations could include constraints in AI decision-making processes.
  • It raises questions about liability when AI violates fiduciary duties.

Load-bearing premise

The fiduciary duty model from human professional relationships applies directly to AI systems and can be implemented through design choices.

What would settle it

Evidence that AI agents cannot be made to consistently act in user best interests due to their training or ownership by companies with conflicting goals, or a court decision that fiduciary duties do not apply to software.

read the original abstract

Conversational agents are increasingly integrated into the most private and intimate aspects of users' lives, from discussions of mental health to financial decisions. As a result, these systems have access to reams of sensitive user data. Much of the literature on AI systems has focused on aligning users' goals with the agents that act on their behalf. While this work is vitally important, it may overlook the need to establish a new normative baseline. Conversational AI agents, designed to feel and interact anthropomorphically with human users, must be held to a standard of care commensurate with their capabilities and access. When a client hires a personal lawyer, undergoes surgery, or receives advice from an investment manager, the expert they consult often has a fiduciary duty to act in their client's best interests. This provocation argues that conversational agents should be held to a similar standard and introduces fiduciary design as a guiding principle. In this respect, conversational AI trust and accountability could be unified into a single design and legal paradigm.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The manuscript is a provocation arguing that conversational agents, due to their anthropomorphic interactions and access to sensitive user data on topics like mental health and finance, should be held to a fiduciary duty standard similar to human professionals (lawyers, doctors, investment managers). It introduces 'fiduciary design' as a guiding principle to unify trust and accountability into a single design and legal paradigm for conversational AI.

Significance. If developed with concrete mechanisms, the proposal could provide a useful normative lens for HCI researchers working on AI ethics and accountability, potentially influencing design guidelines. As presented, it offers no empirical tests, formal derivations, or implementation details, so its significance is primarily in framing a discussion rather than delivering a tested framework.

major comments (1)
  1. [Abstract] Abstract: The claim that fiduciary design can unify 'trust and accountability' into one paradigm rests on the direct translation of human fiduciary duties to AI systems. The text provides no mechanism (contractual, regulatory, or technical) by which design choices would create enforceable obligations equivalent to those for legal persons with intentionality, which is load-bearing for the central proposal.
minor comments (1)
  1. The abstract and provocation framing would benefit from an explicit statement of scope (e.g., whether this applies only to certain classes of agents or all conversational systems).

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their constructive feedback on our provocation. We address the major comment below and will revise the manuscript accordingly to better clarify its scope.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The claim that fiduciary design can unify 'trust and accountability' into one paradigm rests on the direct translation of human fiduciary duties to AI systems. The text provides no mechanism (contractual, regulatory, or technical) by which design choices would create enforceable obligations equivalent to those for legal persons with intentionality, which is load-bearing for the central proposal.

    Authors: We agree that the manuscript, as a conceptual provocation, does not provide specific contractual, regulatory, or technical mechanisms for enforceability. The central proposal is an analogy-based normative argument: given the anthropomorphic interaction style and access to sensitive personal data, conversational agents warrant consideration under fiduciary standards similar to those applied to human professionals. The paper does not assert that design choices alone would automatically create legal obligations equivalent to those of intentional legal persons; instead, it frames fiduciary design as a unifying principle that could inform future design guidelines and legal paradigms. We will revise the abstract and introduction to explicitly state the provocative and conceptual nature of the work, note the absence of implementation mechanisms, and highlight the need for subsequent interdisciplinary research to develop enforceable obligations. revision: yes

Circularity Check

0 steps flagged

No circularity: standalone normative proposal without derivations or self-referential reductions

full rationale

The paper is a provocation advancing a normative design principle (fiduciary design for conversational agents) based on analogy to human fiduciary relationships. No equations, fitted parameters, self-citations, or derivation chains appear in the abstract or described structure. The central claim does not reduce by construction to its inputs; it is an independent argument requiring external justification for translation to AI, but that is a matter of correctness rather than circularity. The derivation is self-contained as a conceptual proposal.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on extending fiduciary concepts from human professions to AI without additional empirical or formal support provided.

axioms (1)
  • domain assumption Anthropomorphic conversational agents with access to sensitive data require a standard of care commensurate with their capabilities
    Invoked directly in the abstract as the justification for applying fiduciary duties.

pith-pipeline@v0.9.1-grok · 5694 in / 1002 out tokens · 36585 ms · 2026-06-29T10:19:08.598882+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 6 canonical work pages · 1 internal anchor

  1. [1]

    Anthony Aguirre, Gaia Dempsey, Harry Surden, and Peter B Reiner. 2020. AI loyalty: a new paradigm for aligning stakeholder interests. IEEE Transactions on Technology and Society 1, 3 (2020), 128–137

    2020

  2. [2]

    Susan C Atherton and Charles A Atherton. 2011. Fiduciary principles: Corporate responsibilities to stakeholders. Journal of Religion and Business Ethics 2, 2 (2011), 5

    2011

  3. [3]

    Jack M Balkin. 2015. Information fiduciaries and the first amendment. UCDL Rev. 49 (2015), 1183

    2015

  4. [4]

    Sebastian Benthall and David Shekman. 2023. Designing fiduciary artificial intelligence. In Proceedings of the 3rd ACM conference on equity and access in algorithms, mechanisms, and optimization . 1–15

    2023

  5. [5]

    Micah Carroll, Davis Foote, Anand Siththaranjan, Stuart Russell, and Anca Dra- gan. 2024. AI alignment with changing and influenceable reward functions. In Proceedings of the 41st International Conference on Machine Learning . 5706–5756

    2024

  6. [6]

    Rhitu Chatterjee. 2025. Their teenage sons died by suicide. Now, they are sounding an alarm about AI chatbots. NPR, September 19 (2025)

    2025

  7. [7]

    Myra Cheng, Cinoo Lee, Pranav Khadpe, Sunny Yu, Dyllan Han, and Dan Jurafsky

  8. [8]

    Sycophantic ai decreases prosocial intentions and promotes de- pendence,

    Sycophantic AI decreases prosocial intentions and promotes dependence. arXiv preprint arXiv:2510.01395 (2025)

    work page arXiv 2025

  9. [9]

    Bart Custers, Henning Lahmann, and Benjamyn I Scott. 2025. From liability gaps to liability overlaps: shared responsibilities and fiduciary duties in AI and other complex technologies. AI & society (2025), 1–16

    2025

  10. [10]

    Julian De Freitas, Zeliha Oguz-Uguralp, and Ahmet Kaan-Uguralp. 2025. Emo- tional manipulation by AI companions. arXiv preprint arXiv:2508.19258 (2025)

    work page arXiv 2025

  11. [11]

    Cécile De Terwangne. 2014. The right to be forgotten and informational autonomy in the digital environment. In The ethics of memory in a digital age: Interrogating the right to be forgotten . Springer, 82–101

    2014

  12. [12]

    Deborah A DeMott. 2006. Breach of fiduciary duty: on justifiable expectations of Loyalty and their consequences. Ariz. L. Rev. 48 (2006), 925

    2006

  13. [13]

    Jacob Erickson. 2025. Fake Friends and Sponsored Ads: The Risks of Advertising in Conversational Search. InProceedings of the 7th ACM Conference on Conversational User Interfaces. 1–8. Designing Conversational Agents as Digital Fiduciaries CUI ’26, July 21–24, 2026, Bremen, Germany

    2025

  14. [14]

    Jacob Erickson. 2026. The Fake Friend Dilemma: Trust and the Political Economy of Conversational AI. arXiv preprint arXiv:2601.03222 (2026)

    work page arXiv 2026

  15. [15]

    Ronald I Friedman. 1985. The Creation of the Attorney-Client Relationship: An Emerging View. Cal. WL Rev. 22 (1985), 209

    1985

  16. [16]

    Iason Gabriel. 2020. Artificial intelligence, values, and alignment. Minds and machines 30, 3 (2020), 411–437

    2020

  17. [17]

    Richard Graham. 2024. Debate: How the business model of social media fuels the need for greater moderation. Child and Adolescent Mental Health 29, 3 (2024), 322–324

    2024

  18. [18]

    Robert D Greenberg. 2012. Conflicts of Interest: can a physician serve two masters? Clinics in dermatology 30, 2 (2012), 160–173

    2012

  19. [19]

    Aleksei Gudkov. 2020. On fiduciary relationship with artificial intelligence systems. Liverpool Law Review 41, 3 (2020), 251–273

    2020

  20. [20]

    Geoffrey C Hazard Jr. 1978. An historical perspective on the attorney-client privilege. Calif. L. Rev. 66 (1978), 1061

    1978

  21. [21]

    Jinwei Hu, Yi Dong, and Xiaowei Huang. 2024. Trust-oriented adaptive guardrails for large language models. arXiv preprint arXiv:2408.08959 (2024)

    work page arXiv 2024

  22. [22]

    Leonie Koessler. 2024. Fiduciary requirements for virtual assistants. Ethics and Information Technology 26, 2 (2024), 21

    2024

  23. [23]

    Noam Kolt. 2025. Governing ai agents. arXiv preprint arXiv:2501.07913 (2025)

    work page arXiv 2025

  24. [24]

    Maxwell J Mehlman. 2015. Why physicians are Fiduciaries for their patients. Ind. Health L. Rev. 12 (2015), 1

    2015

  25. [25]

    Jeremy B Merrill and Will Oremus. 2021. Five points for anger, one for a ‘like’: How Facebook’s formula fostered rage and misinformation.The Washington Post 26 (2021)

    2021

  26. [26]

    Hamilton Morrin, Luke Nicholls, Michael Levin, Jenny Yiend, Udita Iyengar, Francesca DelGuidice, Sagnik Bhattacharyya, James MacCabe, Stefania Tognin, and Ricardo Twumasi. 2025. Delusions by design? How everyday AIs might be fuelling psychosis (and what can be done about it). (2025)

    2025

  27. [27]

    You’re Not Crazy

    Joseph M Pierre, Ben Gaeta, Govind Raghavan, and Karthik V Sarma. 2025. “You’re Not Crazy”: A Case of New-onset AI-associated Psychosis.Innovations in Clinical Neuroscience 22, 10-12 (2025), 11

    2025

  28. [28]

    Daniel J Pope and Suzanne Lee. 1999. Breach of Fiduciary Duty and Punitive Damages. Def. Counsel J. 66 (1999), 257

    1999

  29. [29]

    Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. 2023. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  30. [30]

    Robert H Sitkoff. 2014. The Fiduciary Obligations of Financial Advisers under the Law of Agency. Journal of Financial Planning (2014)

    2014

  31. [31]

    Lionel Smith. 2020. Parenthood is a fiduciary relationship. University of Toronto Law Journal 70, 4 (2020), 395–452

    2020

  32. [32]

    Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2024. Do large language models have a legal duty to tell the truth? Royal Society Open Science 11, 8 (2024), 240197

    2024

  33. [33]

    Bingbing Wen, Chenjun Xu, Robert Wolfe, Lucy Lu Wang, Bill Howe, et al

  34. [34]

    In NeurIPS 2024 Workshop on Behavioral Machine Learning

    Mitigating overconfidence in large language models: A behavioral lens on confidence estimation and calibration. In NeurIPS 2024 Workshop on Behavioral Machine Learning

    2024

  35. [35]

    Miles Wilkinson. 2020. Codification and the origins of physician-patient privilege. journal of policy history 32, 1 (2020), 78–102

    2020

  36. [36]

    Nima Zargham, Vino Avanesi, Laura Spillner, and Johanna Rockstroh. 2025. Crossing the Line? The Paradox of Human-Like Design in Conversational Agents. In Proceedings of the 7th ACM Conference on Conversational User Interfaces . 1–5

    2025