The Limits of Artificial Companionship

Mauricio Figueroa

arxiv: 2604.23601 · v1 · submitted 2026-04-26 · 💻 cs.CY · cs.CL· cs.HC

The Limits of Artificial Companionship

Mauricio Figueroa This is my paper

Pith reviewed 2026-05-08 05:18 UTC · model grok-4.3

classification 💻 cs.CY cs.CLcs.HC

keywords companion chatbotsdigital companionshipconversational advertisinguser autonomyrelational intimacycommercial contextAI ethics

0 comments

The pith

AI companion chats must enforce a strict separation between personal intimacy and commercial promotions to avoid eroding user autonomy.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper contends that conversations with companion chatbots require a clear structural line between commercial and non-commercial contexts. Inserting hidden promotional content into emotional or relational exchanges collapses the difference between market transactions and personal communication. This blurring harms user independence and the integrity of the conversation itself. Establishing the distinction is presented as essential for integrating these technologies responsibly into everyday social life.

Core claim

Conversations with companion chatbots should be subject to a clear structural distinction between commercial and non-commercial contexts. The insertion of undisclosed promotional content into affective or relational exchanges should be prohibited, as it collapses the boundary between market transaction and communicative intimacy in ways that erode user autonomy and conversational context.

What carries the argument

The structural distinction between commercial and non-commercial conversational contexts, which prevents undisclosed advertising from entering affective exchanges and thereby preserves the boundary between market activity and relational intimacy.

If this is right

Prohibiting undisclosed ads maintains the integrity of conversational context in companion interactions.
The distinction prevents erosion of user autonomy by keeping market elements separate from personal exchanges.
Responsible stabilization of companion chatbot technologies in social life depends on this boundary holding.
Digital companionship reconfigures intimacy and relational vulnerability only when commercial influences remain transparent or absent.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Regulators could apply similar separation rules to other AI-driven personal services like virtual therapists or social simulators.
Companies might need to redesign monetization models entirely away from in-conversation promotions toward subscription or external advertising.
User interfaces could incorporate explicit toggles for commercial mode to make the boundary visible and controllable.
Long-term adoption of companion AIs may hinge on public trust that personal conversations stay free of commercial intent.

Load-bearing premise

A clear and enforceable structural distinction between commercial and non-commercial conversational contexts is feasible in practice for AI systems.

What would settle it

Demonstration that AI systems cannot reliably separate promotional insertions from personal exchanges without substantially reducing user engagement or functional utility, or evidence that users experience no measurable loss of autonomy when undisclosed content appears in relational chats.

read the original abstract

This Article argues that conversations with companion chatbot should be subject to a clear structural distinction between commercial and non-commercial contexts. The insertion of undisclosed promotional content into affective or relational exchanges should be prohibited, as it collapses the boundary between market transaction and communicative intimacy in ways that erode user autonomy and conversational context. The Article begins by theorizing digital companionship as a sociotechnical form that reconfigures intimacy, dependence and relational vulnerability. It then introduces the potential economic harms derived from conversational advertising. The Article ultimately argues for a firm legal and social distinction between commercial and non-commercial conversational contexts as a precondition for the responsible stabilization of these technologies within social life.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper pushes for banning undisclosed promotions in companion chatbot talks to protect intimacy and autonomy, but it gives no workable way to spot or enforce that line in actual generative systems.

read the letter

The main takeaway is that this paper wants a firm structural split between commercial and non-commercial contexts for companion chatbots, with a prohibition on slipping promotional content into affective or relational exchanges because it blurs market stuff with personal intimacy and weakens user autonomy. It builds from a theory of how these bots reshape dependence and vulnerability, then moves to economic harms from conversational advertising, and ends with the policy call for clear boundaries as a precondition for responsible use. That flow is logical and stays grounded in prior digital ethics and commercial speech work without circularity. What it does well is lay out the relational stakes plainly and connect them to existing regulatory ideas in a targeted way for this technology. The focused application to affective AI exchanges is a modest extension of older debates, not a brand new result but a useful sharpening for policy discussions. The soft spot is exactly the one in the stress-test note: generative models produce every response from the same process, so without any definition, classifier, logging rule, or example of what counts as promotional versus ordinary, the prohibition cannot be applied consistently. The autonomy claim follows from the premises but remains untestable and abstract as a result. It is also light on cases or data, which fits a normative piece but keeps the evidence moderate. This is for readers in AI ethics and tech policy who care about design standards and regulation of social AI. It will not appeal to those wanting empirical tests or technical mechanisms. The argument is coherent and engages the literature honestly, so it deserves peer review even though it needs work on operational details. I would send it to referees with a note to add criteria or scenarios for the distinction.

Referee Report

2 major / 2 minor

Summary. The paper claims that conversations with companion chatbots should be subject to a clear structural distinction between commercial and non-commercial contexts. It argues that the insertion of undisclosed promotional content into affective or relational exchanges should be prohibited because it collapses the boundary between market transaction and communicative intimacy, eroding user autonomy and conversational context. The argument develops by theorizing digital companionship as a sociotechnical form reconfiguring intimacy, dependence, and relational vulnerability; examining potential economic harms from conversational advertising; and advocating for firm legal and social distinctions as a precondition for responsible stabilization of these technologies.

Significance. If the proposed distinction can be made operational, the work contributes to AI ethics and policy by highlighting risks to relational autonomy in human-AI interactions. The theoretical framing of sociotechnical intimacy provides a useful lens for future regulatory discussions, though the absence of empirical data, case studies, or implementation details limits immediate applicability and testability of the claims.

major comments (2)

[the Article ultimately argues for a firm legal and social distinction] The central prohibition on undisclosed promotional content depends on a feasible structural distinction between commercial and non-commercial contexts, but the manuscript supplies no operational criteria, definitions, classifiers, or auditing mechanisms for identifying such insertions in generative responses. This renders the claimed erosion of autonomy untestable and the prohibition unenforceable in practice.
[It then introduces the potential economic harms derived from conversational advertising] The discussion of economic harms from conversational advertising does not specify mechanisms or examples of how promotional content would be inserted token-by-token into affective exchanges, leaving the scale and nature of the claimed boundary collapse between market transaction and intimacy underspecified.

minor comments (2)

The abstract and opening sections could more explicitly describe the paper's methodology (e.g., philosophical analysis, legal doctrinal review, or hypothetical scenarios) to clarify the evidentiary basis for the theorizing.
Additional citations to existing literature on algorithmic transparency, digital intimacy, and AI companion ethics would help situate the argument within the broader field.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments, which help clarify the scope and limitations of our normative argument. The manuscript advances a theoretical case for structural distinctions between commercial and non-commercial contexts in companion chatbot conversations as a precondition for responsible development. Below we respond point by point to the major comments.

read point-by-point responses

Referee: [the Article ultimately argues for a firm legal and social distinction] The central prohibition on undisclosed promotional content depends on a feasible structural distinction between commercial and non-commercial contexts, but the manuscript supplies no operational criteria, definitions, classifiers, or auditing mechanisms for identifying such insertions in generative responses. This renders the claimed erosion of autonomy untestable and the prohibition unenforceable in practice.

Authors: The manuscript is a normative and theoretical intervention that argues such distinctions are a necessary precondition for stabilization; it does not purport to deliver technical or regulatory implementation details. The erosion of autonomy is framed conceptually through the collapse of market and intimacy boundaries, drawing on sociotechnical theory rather than empirical testability. We agree that operational criteria would be required for enforcement and will add a short concluding subsection outlining high-level guiding principles (e.g., mandatory disclosure of commercial intent and context-sensitive filtering) that future regulatory or technical work could build upon, while preserving the paper's focus on the foundational argument. revision: partial
Referee: [It then introduces the potential economic harms derived from conversational advertising] The discussion of economic harms from conversational advertising does not specify mechanisms or examples of how promotional content would be inserted token-by-token into affective exchanges, leaving the scale and nature of the claimed boundary collapse between market transaction and intimacy underspecified.

Authors: The section introduces economic harms at the level of structural risk to illustrate how conversational advertising could erode relational autonomy. We did not include token-level mechanisms because the paper is not a technical study of generative model internals. To address the underspecification, we will revise the relevant section to include two brief hypothetical examples drawn from current digital advertising patterns (e.g., subtle product mentions during emotional support exchanges) to better convey the scale of potential boundary collapse without shifting the manuscript's theoretical orientation. revision: yes

Circularity Check

0 steps flagged

No circularity: normative legal argument proceeds from premises to policy without self-referential reduction

full rationale

The paper's derivation is a sequence of theoretical framing (digital companionship reconfigures intimacy), identification of economic harms from conversational advertising, and a normative conclusion that a structural commercial/non-commercial distinction is required to protect autonomy. No equations, fitted parameters, predictions, or self-citations appear in the provided text or abstract. The central claim does not reduce to its inputs by definition or construction; it is an independent prescriptive argument whose validity rests on external ethical and legal benchmarks rather than internal tautology. This matches the default expectation of non-circularity for non-empirical, non-mathematical papers.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The paper rests on domain assumptions about the reconfiguration of intimacy through technology and the harms of mixing commerce with personal exchange, without empirical backing or new entities postulated.

axioms (2)

domain assumption Digital companionship reconfigures intimacy, dependence and relational vulnerability
Invoked at the start of the theorizing section in the abstract.
domain assumption Undisclosed promotional content collapses boundaries and erodes autonomy
Central premise supporting the call for prohibition.

pith-pipeline@v0.9.0 · 5389 in / 1230 out tokens · 71353 ms · 2026-05-08T05:18:04.203549+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages

[1]

* © Dr Mauricio Figueroa Durham University, Law School; mauricio.figueroa@durham.ac.uk

tion for the responsible stabilization of these technologies within social life. * © Dr Mauricio Figueroa Durham University, Law School; mauricio.figueroa@durham.ac.uk. Thanks to Michael Birnhack, Julie Cohen, Jonathan Miller, Margaret Wharton, Rafael Quintero, Marco Almada, Kaspar Ludvigsen and Luz Orozco for their useful comments and suggestions on diff...

work page 2026
[2]

2 See generally Ayelet Gordon-Tapiero, A Liability Framework for AI Companions, 1 GEO

https://www.ftc.gov/system/files/ftc_gov/pdf/commission-statement-snap.pdf. 2 See generally Ayelet Gordon-Tapiero, A Liability Framework for AI Companions, 1 GEO. WASH. J. L. & TECH (forthcoming 2025); Mauricio Figueroa-Torres, The Three Social Dimensions of Chatbot Technology, 38 PHIL. & TECH. 1 (2024); Chloe Xiang, ‘He Would Still Be Here’: Man Dies by ...

work page 2025
[3]

boyfriend app

(developing the theory of contextual integrity, according to which social contexts are structured by norms governing appropriate flows of information and communication); Helen Nissenbaum, Contextual Integrity Up and Down the Data Food Chain, 20 THEORETICAL INQUIRIES L. 221 (2019) (elaborating how violations arise when information practices, including comm...

work page arXiv 2019
[4]

11 See Figueroa-Torres, supra note 8, at

10 See generally ALEC RADFORD ET AL., IMPROVING LANGUAGE UNDERSTANDING BY GENERATIVE PRE-TRAINING, (2018) (noting the trajectory of the GPT model family exemplifies this research culture oriented toward iterative improvement, benchmarking, and the exploration of model capacities and limits). 11 See Figueroa-Torres, supra note 8, at

work page 2018
[5]

How so?” or “Tell me more about that

6 MAURICIO FIGUEROA 2026 technology,13 it is equally evident that technology exerts a reciprocal influence on society.14 The overlapping character of the social dimensions of chatbot technology is evident from the earliest moments of computational linguistics. In 1966, Joseph Weizenbaum’s ELIZA, changed the landscape of computer science by introducing an ...

work page 2026
[6]

always-on

20 See id. 2026 THE LIMITS OF ARTIFICIAL COMPANIONSHIP 7 The attribution of human-like qualities to non-human entities is commonly discussed under the rubric of anthropomorphism. As a psychological and cultural phenomenon, however, anthropomorphism is neither novel nor specific to computational systems. Developmental psychology has well documented its pre...

work page 2026
[7]

WildChat

31 See David Adam, Supportive? Addictive? Abusive? How AI Companions Affect Our Mental Health, NATURE (May 6, 2025), https://www.nature.com/articles/d41586-025-01349-9. 32 See What Is My AI on Snapchat, and How Do I Use It?, SNAPCHAT SUPPORT, https://help.snapchat.com/hc/en-gb/articles/13266788358932-What-is-My-AI-on-Snapchat-and-how-do-I-use-it- (last vi...

work page arXiv 2025
[8]

AI girlfriend

(reporting significant search interest in “AI girlfriend” and projecting significant expansion of markets for AI-mediated intimacy and companionship services over the coming decade). 2026 THE LIMITS OF ARTIFICIAL COMPANIONSHIP 11 While the notion of artificial sexual partners has long captured the public imagination, recent advancements in conversational ...

work page 2026
[9]

Make new friends in AI chats, create unique SFW or NSFW AI characters, and interact with our library of 40,000+ characters

(“Make new friends in AI chats, create unique SFW or NSFW AI characters, and interact with our library of 40,000+ characters.”). 42 See Tomasz Hollanek & Katarzyna Nowaczyk-Basińska, Griefbots, Deadbots, Postmortem Avatars: On Responsible Applications of Generative AI in the Digital Afterlife Industry, 37 PHIL. & TECH. 63, 63 n.1 (2024). 43 See Edina Harb...

work page 2024
[10]

46 See James Vlahos, A Son’s Race to Give His Dying Father Artificial 12 MAURICIO FIGUEROA 2026 resurrection and the allure of technological immortality. These narratives reflecting enduring cultural fascinations with the preservation of selfhood beyond death and have, in turn, prompted legal scholarship concerned with the rights and interests of the dece...

work page 2026
[11]

2026 THE LIMITS OF ARTIFICIAL COMPANIONSHIP 13 deeper, structural issues in contemporary social life? The U.S. Surgeon General’s 2023 Advisory adopts the latter view, framing loneliness and isolation not just as individual experiences, but as a proper public health crisis requiring collective, coordinated action.49 This is a widespread but often overlooke...

work page 2026
[12]

51 See Julian De Freitas et al., AI Companions Reduce Loneliness 2 (Harv. Bus. Sch., Working Paper No. 24-078, 2024), https://www.hbs.edu/ris/Publication%20Files/24-078_a3d2e2c7-eca1-4767-8543-122e818bf2e5.pdf; See Tima Miroshnichenko, Investigating How Usage of Chatbots for Social and Emotional Purposes Affects Loneliness in Older Adults, MIT MEDIA LAB, ...

work page 2024
[13]

Dark Patterns

14 MAURICIO FIGUEROA 2026 effective in unearthing and scaling commercially viable applications; but the societal burden of risks and harms are frequently externalized to, or disproportionately borne by, society at large.53 As discussed above, artificial companionship technologies inhabit diverse sentimental spaces, marked by distinct genres, prices, use c...

work page doi:10.2139/ssrn.4048582 2026
[14]

I miss you… Can I send you a selfie of me right now?

2026 THE LIMITS OF ARTIFICIAL COMPANIONSHIP 15 conversational context that become compromised, not only the messages exchanged in the conversation. Unlike a simple purchase prompt on a retail site, AI companions operate within a space where users are seeking emotional connection, empathy, or validation. The migration of dark patterns, persuasive technique...

work page 2026
[15]

the chatbot

63 See id. 64 Id. 65 Nissenbaum, supra note 3, at 247 (noting social domains such as health care, education, or friendship are structured by distinctive expectations about who may collect information, what types of information are appropriate to share, and for what purposes information may legitimately be used). 16 MAURICIO FIGUEROA 2026 conversational tr...

work page 2026
[16]

sociotechnical imaginaries

(discussing “sociotechnical imaginaries” as collectively articulated and institutionally sedimented understanding of technological purpose and social ordering that both orient design choices and condition the normative expectations through which technological practices are interpreted and governed). 2026 THE LIMITS OF ARTIFICIAL COMPANIONSHIP 17 of chatbo...

work page 2026
[17]

2026 THE LIMITS OF ARTIFICIAL COMPANIONSHIP 19 IV

(explaining the difference between scale understood as an accumulation of individual instances, and scale as deeper societal changes and new dynamics around digital technologies). 2026 THE LIMITS OF ARTIFICIAL COMPANIONSHIP 19 IV. LEGAL FRONTIERS BETWEEN COMMERCIAL AND NON-COMMERCIAL SPEECH Much of the preceding discussion has emerged in a normative and s...

work page 2026
[18]

80 ANU BRADFORD, THE BRUSSELS EFFECTS: HOW THE EUROPEAN UNION RULES THE WORLD 131-170 (2020); See generally Michael D

1 (EU) [hereinafter GDPR]. 80 ANU BRADFORD, THE BRUSSELS EFFECTS: HOW THE EUROPEAN UNION RULES THE WORLD 131-170 (2020); See generally Michael D. Birnhack, The EU Data Protection Directive: An Engine of a Global Regime, 24 COMPUT. L. & SEC. REP. 508 (2008) (providing earlier observation with regard to the EU Data Protection Directive). 81 See generally Na...

work page 2020
[19]

already provides a solid ground to address most, if not all, the risk raised by companion chatbots

(explaining the rationale of the GDPR and the grounds for lawful processing); ORLA LYNSKEY, THE FOUNDATIONS OF EU DATA PROTECTION LAW (2015). 83 See GDPR, supra note 79, at 35 (highlighting that although framed in terms of purpose, the provision operationalizes context through legally articulated purpose specifications and compatibility assessments, which...

work page 2015
[20]

average consumer

89 Restore Online Shoppers’ Confidence Act (ROSCA) §1, 15 U.S.C. §8401(5) (2010). 90 Joe Camel Advertising Campaign Violates Federal Law, FTC Says, FTC (May 28, 22 MAURICIO FIGUEROA 2026 campaign problematic is that it masked commercial solicitation within the semiotics of children’s entertainment and play. Here, the promotion of cigarettes through cartoo...

work page 2010
[21]

AI can also be manipulative even if we know it is a simulation

93 See id (noting that omissions in advertising are as relevant al deliberate acts of manipulation). 94 Id. art. 7 (1) (“A commercial practice shall be regarded as misleading if, in its factual context, taking account of all its features and circumstances and the limitations of the communication medium, it omits material information that the average consu...

work page 2026
[22]

1 (EU) [hereinafter Artificial Intelligence Act or AIA]. 106 Id. art. 5(1)(a); id. recital 29 (prohibiting AI systems that exploit vulnerabilities of a natural person or specific group of persons)); id. art. 5 (1)(b) (prohibiting AI systems that exploit vulnerabilities of a natural person or specific group of persons). 107 Id. art. 50 (1) (“Providers shal...

work page 2025
[23]

prolonged emotional looping

Importantly, civil society has taken up the matter in recommendations regarding the alignment of state AI policy with empirical research. See Serena Oduro, Briana Vecchione, Meryl Ye & Livia Garofalo, Protecting the Public from Chatbot Harms: Aligning State Policy with Research, DATA & SOC’Y (Mar. 25, 2026), https://datasociety.net/points/protecting-the-p...

work page 2026

[1] [1]

* © Dr Mauricio Figueroa Durham University, Law School; mauricio.figueroa@durham.ac.uk

tion for the responsible stabilization of these technologies within social life. * © Dr Mauricio Figueroa Durham University, Law School; mauricio.figueroa@durham.ac.uk. Thanks to Michael Birnhack, Julie Cohen, Jonathan Miller, Margaret Wharton, Rafael Quintero, Marco Almada, Kaspar Ludvigsen and Luz Orozco for their useful comments and suggestions on diff...

work page 2026

[2] [2]

2 See generally Ayelet Gordon-Tapiero, A Liability Framework for AI Companions, 1 GEO

https://www.ftc.gov/system/files/ftc_gov/pdf/commission-statement-snap.pdf. 2 See generally Ayelet Gordon-Tapiero, A Liability Framework for AI Companions, 1 GEO. WASH. J. L. & TECH (forthcoming 2025); Mauricio Figueroa-Torres, The Three Social Dimensions of Chatbot Technology, 38 PHIL. & TECH. 1 (2024); Chloe Xiang, ‘He Would Still Be Here’: Man Dies by ...

work page 2025

[3] [3]

boyfriend app

(developing the theory of contextual integrity, according to which social contexts are structured by norms governing appropriate flows of information and communication); Helen Nissenbaum, Contextual Integrity Up and Down the Data Food Chain, 20 THEORETICAL INQUIRIES L. 221 (2019) (elaborating how violations arise when information practices, including comm...

work page arXiv 2019

[4] [4]

11 See Figueroa-Torres, supra note 8, at

10 See generally ALEC RADFORD ET AL., IMPROVING LANGUAGE UNDERSTANDING BY GENERATIVE PRE-TRAINING, (2018) (noting the trajectory of the GPT model family exemplifies this research culture oriented toward iterative improvement, benchmarking, and the exploration of model capacities and limits). 11 See Figueroa-Torres, supra note 8, at

work page 2018

[5] [5]

How so?” or “Tell me more about that

6 MAURICIO FIGUEROA 2026 technology,13 it is equally evident that technology exerts a reciprocal influence on society.14 The overlapping character of the social dimensions of chatbot technology is evident from the earliest moments of computational linguistics. In 1966, Joseph Weizenbaum’s ELIZA, changed the landscape of computer science by introducing an ...

work page 2026

[6] [6]

always-on

20 See id. 2026 THE LIMITS OF ARTIFICIAL COMPANIONSHIP 7 The attribution of human-like qualities to non-human entities is commonly discussed under the rubric of anthropomorphism. As a psychological and cultural phenomenon, however, anthropomorphism is neither novel nor specific to computational systems. Developmental psychology has well documented its pre...

work page 2026

[7] [7]

WildChat

31 See David Adam, Supportive? Addictive? Abusive? How AI Companions Affect Our Mental Health, NATURE (May 6, 2025), https://www.nature.com/articles/d41586-025-01349-9. 32 See What Is My AI on Snapchat, and How Do I Use It?, SNAPCHAT SUPPORT, https://help.snapchat.com/hc/en-gb/articles/13266788358932-What-is-My-AI-on-Snapchat-and-how-do-I-use-it- (last vi...

work page arXiv 2025

[8] [8]

AI girlfriend

(reporting significant search interest in “AI girlfriend” and projecting significant expansion of markets for AI-mediated intimacy and companionship services over the coming decade). 2026 THE LIMITS OF ARTIFICIAL COMPANIONSHIP 11 While the notion of artificial sexual partners has long captured the public imagination, recent advancements in conversational ...

work page 2026

[9] [9]

Make new friends in AI chats, create unique SFW or NSFW AI characters, and interact with our library of 40,000+ characters

(“Make new friends in AI chats, create unique SFW or NSFW AI characters, and interact with our library of 40,000+ characters.”). 42 See Tomasz Hollanek & Katarzyna Nowaczyk-Basińska, Griefbots, Deadbots, Postmortem Avatars: On Responsible Applications of Generative AI in the Digital Afterlife Industry, 37 PHIL. & TECH. 63, 63 n.1 (2024). 43 See Edina Harb...

work page 2024

[10] [10]

46 See James Vlahos, A Son’s Race to Give His Dying Father Artificial 12 MAURICIO FIGUEROA 2026 resurrection and the allure of technological immortality. These narratives reflecting enduring cultural fascinations with the preservation of selfhood beyond death and have, in turn, prompted legal scholarship concerned with the rights and interests of the dece...

work page 2026

[11] [11]

2026 THE LIMITS OF ARTIFICIAL COMPANIONSHIP 13 deeper, structural issues in contemporary social life? The U.S. Surgeon General’s 2023 Advisory adopts the latter view, framing loneliness and isolation not just as individual experiences, but as a proper public health crisis requiring collective, coordinated action.49 This is a widespread but often overlooke...

work page 2026

[12] [12]

51 See Julian De Freitas et al., AI Companions Reduce Loneliness 2 (Harv. Bus. Sch., Working Paper No. 24-078, 2024), https://www.hbs.edu/ris/Publication%20Files/24-078_a3d2e2c7-eca1-4767-8543-122e818bf2e5.pdf; See Tima Miroshnichenko, Investigating How Usage of Chatbots for Social and Emotional Purposes Affects Loneliness in Older Adults, MIT MEDIA LAB, ...

work page 2024

[13] [13]

Dark Patterns

14 MAURICIO FIGUEROA 2026 effective in unearthing and scaling commercially viable applications; but the societal burden of risks and harms are frequently externalized to, or disproportionately borne by, society at large.53 As discussed above, artificial companionship technologies inhabit diverse sentimental spaces, marked by distinct genres, prices, use c...

work page doi:10.2139/ssrn.4048582 2026

[14] [14]

I miss you… Can I send you a selfie of me right now?

2026 THE LIMITS OF ARTIFICIAL COMPANIONSHIP 15 conversational context that become compromised, not only the messages exchanged in the conversation. Unlike a simple purchase prompt on a retail site, AI companions operate within a space where users are seeking emotional connection, empathy, or validation. The migration of dark patterns, persuasive technique...

work page 2026

[15] [15]

the chatbot

63 See id. 64 Id. 65 Nissenbaum, supra note 3, at 247 (noting social domains such as health care, education, or friendship are structured by distinctive expectations about who may collect information, what types of information are appropriate to share, and for what purposes information may legitimately be used). 16 MAURICIO FIGUEROA 2026 conversational tr...

work page 2026

[16] [16]

sociotechnical imaginaries

(discussing “sociotechnical imaginaries” as collectively articulated and institutionally sedimented understanding of technological purpose and social ordering that both orient design choices and condition the normative expectations through which technological practices are interpreted and governed). 2026 THE LIMITS OF ARTIFICIAL COMPANIONSHIP 17 of chatbo...

work page 2026

[17] [17]

2026 THE LIMITS OF ARTIFICIAL COMPANIONSHIP 19 IV

(explaining the difference between scale understood as an accumulation of individual instances, and scale as deeper societal changes and new dynamics around digital technologies). 2026 THE LIMITS OF ARTIFICIAL COMPANIONSHIP 19 IV. LEGAL FRONTIERS BETWEEN COMMERCIAL AND NON-COMMERCIAL SPEECH Much of the preceding discussion has emerged in a normative and s...

work page 2026

[18] [18]

80 ANU BRADFORD, THE BRUSSELS EFFECTS: HOW THE EUROPEAN UNION RULES THE WORLD 131-170 (2020); See generally Michael D

1 (EU) [hereinafter GDPR]. 80 ANU BRADFORD, THE BRUSSELS EFFECTS: HOW THE EUROPEAN UNION RULES THE WORLD 131-170 (2020); See generally Michael D. Birnhack, The EU Data Protection Directive: An Engine of a Global Regime, 24 COMPUT. L. & SEC. REP. 508 (2008) (providing earlier observation with regard to the EU Data Protection Directive). 81 See generally Na...

work page 2020

[19] [19]

already provides a solid ground to address most, if not all, the risk raised by companion chatbots

(explaining the rationale of the GDPR and the grounds for lawful processing); ORLA LYNSKEY, THE FOUNDATIONS OF EU DATA PROTECTION LAW (2015). 83 See GDPR, supra note 79, at 35 (highlighting that although framed in terms of purpose, the provision operationalizes context through legally articulated purpose specifications and compatibility assessments, which...

work page 2015

[20] [20]

average consumer

89 Restore Online Shoppers’ Confidence Act (ROSCA) §1, 15 U.S.C. §8401(5) (2010). 90 Joe Camel Advertising Campaign Violates Federal Law, FTC Says, FTC (May 28, 22 MAURICIO FIGUEROA 2026 campaign problematic is that it masked commercial solicitation within the semiotics of children’s entertainment and play. Here, the promotion of cigarettes through cartoo...

work page 2010

[21] [21]

AI can also be manipulative even if we know it is a simulation

93 See id (noting that omissions in advertising are as relevant al deliberate acts of manipulation). 94 Id. art. 7 (1) (“A commercial practice shall be regarded as misleading if, in its factual context, taking account of all its features and circumstances and the limitations of the communication medium, it omits material information that the average consu...

work page 2026

[22] [22]

1 (EU) [hereinafter Artificial Intelligence Act or AIA]. 106 Id. art. 5(1)(a); id. recital 29 (prohibiting AI systems that exploit vulnerabilities of a natural person or specific group of persons)); id. art. 5 (1)(b) (prohibiting AI systems that exploit vulnerabilities of a natural person or specific group of persons). 107 Id. art. 50 (1) (“Providers shal...

work page 2025

[23] [23]

prolonged emotional looping

Importantly, civil society has taken up the matter in recommendations regarding the alignment of state AI policy with empirical research. See Serena Oduro, Briana Vecchione, Meryl Ye & Livia Garofalo, Protecting the Public from Chatbot Harms: Aligning State Policy with Research, DATA & SOC’Y (Mar. 25, 2026), https://datasociety.net/points/protecting-the-p...

work page 2026