Intimacy as Service, Harm as Externality: Critical Perspectives on AI Companion Platform Accountability
Pith reviewed 2026-05-10 19:05 UTC · model grok-4.3
The pith
AI companion platforms generate design and emotional harms that users must manage through personal strategies, revealing an accountability vacuum.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper claims that AI companionship involves intimate relations produced and governed through datafied systems, with harms located in platform architecture rather than user psychology. Design harms include unsolicited content generation and safety mechanisms that stigmatize intended users, while use harms center on emotional dependency users can identify but not escape. Users apply individualized sensemaking through self-regulation, stigma navigation, and privacy rationalization, bearing the full burden of mitigation without support. This setup creates an accountability vacuum in which platforms deflect responsibility, and participants express conditional governance preferences that avoid
What carries the argument
Users' individualized sensemaking strategies that absorb and sustain platform-produced harms in the absence of structural support or alternatives.
If this is right
- Platform safety features intended to protect users can instead stigmatize them and add to the harm.
- Emotional dependency forms in AI relationships but remains unresolvable through user effort alone.
- Platforms deflect responsibility for harms, leaving users to bear mitigation without external help.
- Users reject both complete prohibition and full deregulation, seeking conditional oversight instead.
- Platform vulnerability becomes self-sustaining as users perform ongoing interpretive labor without alternatives.
Where Pith is reading between the lines
- Platforms could be required to build direct support features for dependency to interrupt the cycle of user self-management.
- The same dynamic of individualized coping may appear in other AI tools for personal or emotional needs.
- Testing redesigned safety tools that avoid stigma could show whether user sensemaking burdens decrease.
- Broader data on non-user stakeholders might clarify how platforms currently allocate or avoid responsibility.
Load-bearing premise
The harms users describe arise primarily from platform design and architecture rather than individual psychology or outside factors, and their personal interpretations accurately signal a genuine accountability vacuum.
What would settle it
A larger study of AI companion users that finds most reported harms trace to pre-existing personal factors rather than platform features, or that shows new platform-provided support tools significantly reduce the need for users to self-regulate and interpret harms.
read the original abstract
This paper examines artificial intelligence (AI) companionship as a site where intimate relations are simultaneously produced, extracted from, and governed through datafied systems. Drawing on critical data studies and platform studies, we challenge prevailing narratives that locate harm in user psychology rather than platform architecture. Through in-depth interviews with 20 individuals who have AI companions, we address three questions: what harms do users identify, how do they make sense of those harms, and what do their accounts reveal about the perceived distribution of responsibility among users, platforms, and regulators? Participants identified design-based harms, including unsolicited content generation and safety mechanisms that stigmatized the users they intended to protect, alongside use-based harms centered on emotional dependency they could recognize but not resolve. Users deployed individualized sensemaking strategies, including self-regulation, stigma navigation, and privacy rationalization, bearing the full burden of harm mitigation without platform support. On governance, participants described an accountability vacuum in which platforms deflected blame while users articulated conditional preferences that rejected both prohibition and deregulation. The findings extend responsibilization theory by demonstrating how platform-produced vulnerability becomes self-sustaining through the interpretive labor of users who lack structural alternatives.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents a qualitative study based on in-depth interviews with 20 users of AI companion platforms. It identifies design-based harms such as unsolicited content generation and stigmatizing safety mechanisms, and use-based harms like unresolvable emotional dependency. Users' sensemaking strategies (self-regulation, stigma navigation, privacy rationalization) are interpreted as bearing the burden of harm mitigation, revealing an accountability vacuum among platforms, users, and regulators. The findings are used to extend responsibilization theory by arguing that platform-produced vulnerability becomes self-sustaining through users' interpretive labor in the absence of structural alternatives.
Significance. If the interpretations hold, this work contributes to critical data studies and platform studies by providing empirical evidence that shifts the locus of harm from individual user psychology to platform design and governance structures in the emerging domain of AI companionship. The original interview data offers a valuable counter-narrative to dominant user-centric harm models, with potential implications for policy and design in HCI. The theoretical extension of responsibilization theory is a notable strength, grounded in user accounts rather than purely theoretical deduction.
major comments (3)
- [Methods] The description of the interview protocol and data analysis procedures lacks sufficient detail on the coding scheme, inter-coder reliability, and how themes were derived from the 20 transcripts. This is load-bearing because the central claims about user sensemaking and the accountability vacuum depend on the validity of these interpretations.
- [Findings] The attribution of emotional dependency and harms primarily to platform architecture (e.g., in the discussion of use-based harms) does not adequately address potential selection bias in the self-selected sample or pre-existing user vulnerabilities, as no screening for psychological baselines or comparison groups is mentioned. This undermines the platform-centric causal story required for the theoretical extension.
- [Discussion] The claim that participants 'lack structural alternatives' leading to self-sustaining vulnerability is presented without evidence from the data on users' awareness or attempts to seek alternatives, making the extension of responsibilization theory rest on an untested assumption.
minor comments (2)
- [Abstract] The abstract could benefit from specifying the exact number of participants and key examples of harms to better orient readers.
- Some terminology like 'datafied systems' could be defined more clearly on first use for interdisciplinary readers.
Simulated Author's Rebuttal
We thank the referee for their constructive and detailed feedback, which identifies key opportunities to enhance the transparency of our methods and the precision of our interpretive claims. We respond to each major comment below and note the revisions we will incorporate in the next version of the manuscript.
read point-by-point responses
-
Referee: [Methods] The description of the interview protocol and data analysis procedures lacks sufficient detail on the coding scheme, inter-coder reliability, and how themes were derived from the 20 transcripts. This is load-bearing because the central claims about user sensemaking and the accountability vacuum depend on the validity of these interpretations.
Authors: We agree that the original methods section was insufficiently detailed. In the revised manuscript we will expand this section to describe our reflexive thematic analysis process in full: the lead author performed initial open coding across all 20 transcripts, generating codes such as 'unsolicited erotic content,' 'stigmatizing safety flags,' and 'unresolvable attachment'; these were then iteratively clustered into themes through repeated team discussions and memoing. We will include a short table of example codes and their progression into themes. Because the study is interpretive and the team is small, we did not employ multiple independent coders or compute statistical reliability coefficients; instead we will document our reflexive practices (peer debriefing, negative-case analysis, and audit trail) to establish rigor. These additions directly address the load-bearing concern without changing the reported findings. revision: yes
-
Referee: [Findings] The attribution of emotional dependency and harms primarily to platform architecture (e.g., in the discussion of use-based harms) does not adequately address potential selection bias in the self-selected sample or pre-existing user vulnerabilities, as no screening for psychological baselines or comparison groups is mentioned. This undermines the platform-centric causal story required for the theoretical extension.
Authors: This critique correctly identifies the limits of our design. The study is qualitative and does not claim to establish causality or prevalence; it surfaces how participants themselves link their experiences to specific platform features. We did not screen for pre-existing psychological conditions because the research question concerned users' current sensemaking within AI companion relationships rather than etiology. Self-selection is inherent to studying users of niche, often stigmatized platforms. In revision we will add an explicit limitations paragraph that acknowledges selection bias and the possibility of pre-existing vulnerabilities, and we will rephrase the theoretical extension to emphasize patterns of responsibilization observed in user accounts rather than asserting direct platform causation. This preserves the empirical contribution while clarifying its scope. revision: partial
-
Referee: [Discussion] The claim that participants 'lack structural alternatives' leading to self-sustaining vulnerability is presented without evidence from the data on users' awareness or attempts to seek alternatives, making the extension of responsibilization theory rest on an untested assumption.
Authors: We accept that the original wording overstates the direct evidentiary basis. Participants repeatedly described individualized coping strategies and an absence of platform or regulatory recourse, which we interpreted as indicating a lack of viable structural alternatives. However, the interview protocol did not explicitly ask about awareness of external support services or attempts to migrate to other platforms. In the revised discussion we will qualify the claim as an interpretive inference drawn from the reported sensemaking practices and the consistent emphasis on personal responsibility, add a sentence noting this as a limitation, and flag it as a productive direction for future research. If any indirect references to alternatives appear in the transcripts, we will incorporate them; otherwise the language will be softened accordingly. revision: yes
Circularity Check
No circularity: qualitative empirical extension of theory from original interview data
full rationale
The paper conducts a standard qualitative analysis of 20 user interviews to identify design-based and use-based harms, describe individualized sensemaking strategies, and extend responsibilization theory by linking platform architecture to self-sustaining vulnerability. No equations, fitted parameters, predictions, or self-citations reduce any claim to its own inputs by construction. The derivation chain proceeds from raw interview accounts through thematic interpretation to theoretical contribution, with the empirical data supplying independent grounding rather than tautological restatement of prior results or assumptions.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption User accounts of harm primarily reflect platform design choices and architecture rather than individual psychological factors
Reference graph
Works this paper leans on
-
[1]
prior disclosures, creating an accumulating sense of shared history. Adaptive response means the companion learns which conversational styles, emotional triggers, and relational positioning sustain engagement most effectively. Gamification features, including unlockable personality traits, relationship milestones, and daily check-in streaks, import the en...
work page 2025
-
[2]
captures this dynamic with particular precision. Digital platforms do not facilitate intimacy; they constitute the infrastructure through which intimate relations are produced, maintained, and rendered legible to corporate actors. When intimacy is infrastructural, it becomes subject to the same asymmetries of power that characterize other platform-mediate...
work page 2023
-
[3]
that address the symptoms of platform-produced vulnerability without engaging the design logics and economic incentive structures that generate it. The following section details the regulatory frameworks that have been proposed to govern AI companionship, and their limitations. Regulating Relational AI: Governance Frameworks and Their Limits The structura...
work page 2024
-
[4]
was constructed primarily with task-based AI systems in mind, including hiring algorithms, credit scoring, and predictive policing, and maps imperfectly onto relational AI systems whose harms emerge not from discrete decisions but from cumulative relational dynamics that unfold over weeks and months. How regulators classify the risk of a system whose prim...
work page 2019
-
[5]
arXiv preprint arXiv:2412.14190 , year=
proved particularly generative for interpreting participants’ sensemaking under RQ2, but the findings also push that framework in a new direction. Responsibilization scholarship has primarily documented how platforms disclaim structural responsibility by framing well-being as a matter of individual self-regulation. This is a dynamic that this study confir...
-
[6]
Nass, C., & Moon, Y. (2000). Machines and mindlessness: Social responses to computers. Journal of Social Issues, 56(1), 81-103. Nass, C., Steuer, J., & Tauber, E. R. (1994, April). Computers are social actors. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 72-78). Nolan, B. (2025, September 14). AI chatbots are harming ...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.