A longitudinal qualitative study of 18 US users finds that LLMs deliver socioemotional support but also foster dependency, one-sided validation, and privacy risks because their designs prioritize engagement over well-being and lack care-based governance.
arXiv preprint arXiv:2407.11438 , year=
7 Pith papers cite this work. Polarity classification is still indexing.
7
Pith papers citing it
abstract
Measuring personal disclosures made in human-chatbot interactions can provide a better understanding of users' AI literacy and facilitate privacy research for large language models (LLMs). We run an extensive, fine-grained analysis on the personal disclosures made by real users to commercial GPT models, investigating the leakage of personally identifiable and sensitive information. To understand the contexts in which users disclose to chatbots, we develop a taxonomy of tasks and sensitive topics, based on qualitative and quantitative analysis of naturally occurring conversations. We discuss these potential privacy harms and observe that: (1) personally identifiable information (PII) appears in unexpected contexts such as in translation or code editing (48% and 16% of the time, respectively) and (2) PII detection alone is insufficient to capture the sensitive topics that are common in human-chatbot interactions, such as detailed sexual preferences or specific drug use habits. We believe that these high disclosure rates are of significant importance for researchers and data curators, and we call for the design of appropriate nudging mechanisms to help users moderate their interactions.
citation-role summary
background 3
citation-polarity summary
roles
background 3representative citing papers
Users show curiosity over concern toward LLM inferences of personal information, with acceptability depending on context, alignment with expectations, and who uses the inferences rather than just the content.
Changing only the prompter's age and gender in AI coding prompts produces statistically significant differences in generated website interface design, template content, and code structure across 800 generated websites and a 20-person user study.
LLM-based inference recovers user age, gender, and country from filtered ChatGPT logs at weighted F1 scores of 0.84-0.90, with median identification from the first 5% of history, driven by stereotype patterns.
PAAC aligns planner-executor decomposition with the device-cloud boundary via typed placeholders and on-device sanitization, delivering 15-36% higher accuracy and 2-6x lower leakage than prior device-cloud baselines on agentic benchmarks.
GroupGPT decouples intervention timing from response generation via edge-cloud collaboration for multi-user chats, scoring 4.72/5 on the new MUIR benchmark of 2500 segments while cutting token use by up to 3x and adding privacy sanitization.
An empirical audit of one web-scraped ML training dataset reveals persistent PII after sanitization, which the authors combine with legal analysis to highlight privacy risks and advocate redefining 'publicly available' data for AI training.
citing papers explorer
-
Engagement-Optimized Care: When LLMs become Mental Health Infrastructure
A longitudinal qualitative study of 18 US users finds that LLMs deliver socioemotional support but also foster dependency, one-sided validation, and privacy risks because their designs prioritize engagement over well-being and lack care-based governance.
-
When Are LLM Inferences Acceptable? User Reactions and Control Preferences for Inferred Personal Information
Users show curiosity over concern toward LLM inferences of personal information, with acceptability depending on context, alignment with expectations, and who uses the inferences rather than just the content.
-
Biased or Personalized? The Impact of Personal Information on AI-driven Development
Changing only the prompter's age and gender in AI coding prompts produces statistically significant differences in generated website interface design, template content, and code structure across 800 generated websites and a 20-person user study.
-
Inferential Privacy Leakage in Anonymized Conversational AI Logs
LLM-based inference recovers user age, gender, and country from filtered ChatGPT logs at weighted F1 scores of 0.84-0.90, with median identification from the first 5% of history, driven by stereotype patterns.
-
PAAC: Privacy-Aware Agentic Device-Cloud Collaboration
PAAC aligns planner-executor decomposition with the device-cloud boundary via typed placeholders and on-device sanitization, delivering 15-36% higher accuracy and 2-6x lower leakage than prior device-cloud baselines on agentic benchmarks.
-
GroupGPT: A Token-efficient and Privacy-preserving Agentic Framework for Multi-User Chat Assistant
GroupGPT decouples intervention timing from response generation via edge-cloud collaboration for multi-user chats, scoring 4.72/5 on the new MUIR benchmark of 2500 segments while cutting token use by up to 3x and adding privacy sanitization.
-
A Common Pool of Privacy Problems: Legal and Technical Lessons from a Large-Scale Web-Scraped Machine Learning Dataset
An empirical audit of one web-scraped ML training dataset reveals persistent PII after sanitization, which the authors combine with legal analysis to highlight privacy risks and advocate redefining 'publicly available' data for AI training.