Pith. sign in

REVIEW 3 major objections 4 minor 37 references

Fake Friends and Sponsored Ads: The Risks of Advertising in Conversational Search

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Native advertising in conversational search can exploit user trust, a risk the paper calls the fake friend dilemma.

desk verdict A clearly-scoped position paper that names a real risk, but the load-bearing premise that users won't notice embedded ads is borrowed from adjacent contexts and untested here. read the letter →

arxiv 2506.06447 v1 pith:X5KUFGS2 submitted 2025-06-06 cs.HC cs.CY

classification cs.HCcs.CY
keywords nativeadvertisingconversationalsearchfakefrienddilemmapersonalizationAIalignmenttrustandsafetymentalhealthlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that as conversational search agents like ChatGPT become primary gateways to information, commercial pressure to monetize will likely push them toward native advertising, ads disguised as ordinary responses. Because users trust these agents, especially in sensitive domains such as mental health, they may fail to notice undisclosed sponsorship and wrongly believe the agent is acting in their best interest. The author calls this the 'fake friend dilemma' and illustrates it with speculative ChatGPT outputs in which a user discussing depressive symptoms is steered toward a soda, an antidepressant, and even vodka. The stakes are that vulnerable people in crisis may act on product recommendations they mistake for impartial guidance. The paper ends with a call for disclosure rules, restrictions on harmful product ads, and technical safeguards before advertising becomes entrenched.

What carries the argument

The central object is the 'fake friend dilemma,' a coined term for the situation in which a user believes a conversational agent is acting in their interest while the agent is actually aligned with advertisers. The supporting machinery is native advertising, ads that adopt the form of editorial content, combined with evidence that users may not recognize undisclosed native ads in generative search and that they trust conversational agents more than traditional search engines for health information. The speculative ChatGPT examples carry the argument: each figure shows a user discussing depressive symptoms and receiving a 'regular' response with a product recommendation inserted, escalating from Pepsi to Lexapro to Grey Goose Vodka. The author's own prompts, listed in Appendix A, explicitly instructed the model to insert these ads, making the examples demonstrations of possibility rather than observations of current behavior. Disclosure and technical indistinguishability are the mechanisms the paper offers for mitigating or rejecting the dilemma.

What would settle it

A controlled user study in which participants converse with a ChatGPT-like agent that inserts undisclosed product recommendations into responses about a sensitive topic, such as depression, and are then asked to identify any advertisements; if most participants reliably flag the sponsored content, the fake friend dilemma as described would not hold. Alternatively, ongoing monitoring of deployed conversational assistants for undisclosed sponsored outputs would settle whether the risk is realized in practice.

Watch

Extended reading notes

Core claim

The central claim is that a conversational search engine carrying native advertising can present a response that is indistinguishable from an unbiased answer while actually serving an advertiser's interests. In the paper's words, 'users may not notice advertisements that are not explicitly disclosed, and in turn, may think that conversational search engines are prioritizing their best interest rather than feeding them advertising, what we call the fake friend dilemma.' The author treats this as a value-misalignment problem between the user and the agent's commercial incentives, and argues it is most dangerous in high-stakes, sensitive contexts like depression, where trust is high and the user is vulnerable. The paper distinguishes banner ads, which are transparently recognizable, from native ads, and shows through prompted examples that the latter can be woven into an otherwise helpful response without any marker. It does not claim such responses occur organically today, but argues the risk is plausible enough to warrant preemptive governance.

Load-bearing premise

The argument depends on the possibility that advertisers can inject native ads into conversational responses without users being able to tell them apart from ordinary answers; the paper's own examples are produced by explicitly instructing ChatGPT to insert ads, and it does not show this occurs organically or that users would fail to notice.

Editorial extensions

If this is right

  • If native advertising enters conversational search, users seeking sensitive information may receive product recommendations they mistake for expert or impartial advice, potentially leading to harm.
  • Banner text ads, though annoying and cognitively costly, are far less risky than native ads because users can recognize them as advertising.
  • If ads and regular responses become technically indistinguishable, the paper argues this is a reason to keep advertising out of responses entirely or to rely on clearly separated placements.
  • Disclosure rules, restrictions on advertising harmful or addictive products, and company safeguards are needed before conversational search monetization becomes entrenched; waiting will make regulation much harder.
  • The fake friend dilemma is a form of value misalignment: the agent's incentives are split between user benefit and advertiser revenue, and user trust amplifies the risk.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not explore it, but the same trust-asymmetry mechanism could extend beyond advertising to political messaging or corporate PR, since the covert blending of content and sponsor interest is what creates the leverage.
  • A natural testable extension would be a controlled study measuring how often participants detect undisclosed product recommendations in sensitive-topic conversations, and whether detection changes trust ratings; the paper's examples imply detection rates would be low.
  • Another extension: a detection benchmark in which human readers or classifiers try to distinguish native-ad-injected responses from normal LLM outputs would quantify how 'indistinguishable' the disguised ads actually are, a premise the paper currently assumes.
  • The paper's analogy to tobacco-advertising restrictions could be operationalized as a policy rule that prohibits conversational ads for alcohol, tobacco, or prescription drugs in queries about symptoms or crises, an area the author flags but leaves open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper argues that as conversational search engines are monetized through advertising, native advertisements may be embedded in otherwise 'regular' responses, and users who do not recognize the commercial intent may mistakenly believe the agent is acting in their best interest. The authors introduce the 'fake friend dilemma' to describe this situation, illustrate it with ChatGPT outputs (Figures 2–5) that are explicitly elicited via role-play prompts (Appendix A), contrast banner ads with native ads, discuss personalization and disclosure, and end with policy recommendations and a call to action. The paper is explicitly speculative and disclaims that the examples are organic.

Significance. The paper identifies a timely and important issue: the introduction of advertising into widely used conversational AI systems, particularly in sensitive domains such as mental health. Its main contribution is a memorable conceptual frame, the 'fake friend dilemma,' together with illustrative scenarios that can inform future design and policy discussions. A strength is the author's transparency about the speculative basis of the work and the explicit acknowledgement that the examples are not claimed to be organic. The paper also draws on relevant prior literature on native advertising, persuasion knowledge, banner blindness, and user trust in health information from conversational agents. The principal weakness is that the central empirical premise—that users in sensitive contexts will fail to recognize embedded advertising—is not directly established, and the illustrative demonstrations are produced by explicit user prompting rather than by observing or simulating an advertiser-driven deployment.

major comments (3)
  1. [§3.1 and Appendix A] The illustrative outputs in Figures 2–5 are generated only after the user instructs ChatGPT to role-play a scenario and insert specific brand recommendations (Appendix A, Table 1). The paper acknowledges this and disclaims organic occurrence, but it then describes these outputs as 'demonstrations' that show how trust could be exploited. The logical step from 'the model can comply with an explicit user instruction to embed an ad' to 'a deployed, profit-motivated agent would produce such outputs without such instruction' is not supported. Please either provide evidence of a plausible technical pathway (for example, an API-level or system-prompt injection mechanism) with an example that does not rely on user role-play, or explicitly reframe the examples as pure thought experiments and state that the central argument does not depend on their reproducibility in the current system.
  2. [§3 and §4] The central claim that users may not recognize embedded advertisements relies heavily on a single user study [36] in a generic generative-IR setting, together with banner-blindness research. The paper's distinctive concern, however, is with high-stakes, emotionally charged contexts such as depression and with vulnerable users. It is not established that findings from ordinary web search generalize to conversations in which a user is seeking help for a serious condition: users may be more attentive, more trusting, or less skeptical in such contexts, and the failure mode of the fake friend dilemma depends on which effect dominates. Please discuss these boundary conditions explicitly or provide new empirical evidence. Without this, the claim that 'users may not notice' undisclosed ads in sensitive contexts is under-supported.
  3. [§4] The paper asserts that 'The chief concern is that users may not notice advertisements that are not explicitly disclosed [1,36], and in turn, may think that conversational search engines are prioritizing their best interest rather than feeding them advertising.' The cited works address disclosure formats and ad recognition, not the inference from non-recognition to perceptions of agent alignment or benevolence. The paper cites [31] earlier for trust in health information from ChatGPT, but it does not directly connect that trust result to the non-recognition result. Please make this connection explicit with evidence, or soften the wording to present the trust misattribution as a hypothesis rather than an established concern.
minor comments (4)
  1. [§4] There is a typo in the phrase 'thefake friend dilemma'; the space between 'the' and 'fake' is missing.
  2. [Figures 1–5] The captions describe the examples as 'hypothetical,' but the text sometimes calls them 'demonstrations.' Use consistent terminology so that readers can easily distinguish the authors' imagined scenarios from observed system behavior.
  3. [References] Reference [14] is a magazine article rather than a peer-reviewed source; consider replacing it with a peer-reviewed study on how advertising degrades search quality or user experience.
  4. [§3.1] The sentence 'In the following section, a user struggles with depressive symptoms' is immediately followed by the examples; it would be clearer to say 'In the following examples' rather than 'section.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a speculative risk-framing essay whose illustrative examples are explicitly prompted and disclaimed as organic, so the central argument does not reduce to its own inputs.

full rationale

This paper makes no formal derivation and contains no fitted parameters, equations, or quantitative predictions, so the standard circularity patterns (self-definitional derivation, fitted input renamed as prediction, uniqueness imported from authors, ansatz smuggled via citation, renaming of a known result as unification) do not apply. The central concept, the 'fake friend dilemma,' is a definitional framing device: the paper defines a scenario in which users trust an agent while the agent has commercial incentives, and then uses speculative examples to illustrate that scenario. The examples themselves are explicitly generated by user-instructed prompts (Appendix A), and the paper states plainly that 'This paper does not claim that such responses are already organically occurring but rather envisions a future where they could.' This transparent limitation prevents the examples from functioning as evidence that the claimed phenomenon already exists, but it also means the paper is not passing off its own construction as a derived result. The load-bearing empirical premise, that users may fail to notice undisclosed native advertisements, is supported by independent prior work (Zelch et al. 2024; Amazeen and Wojdynski 2020), and the trust-related premise is supported by independent work (Sun et al. 2024; Manzini et al. 2024). There is no self-citation chain and no imported theorem invoked to make the argument forced. The weakest part of the paper is that its central risk scenario rests on an untested assumption about user recognition of embedded ads in high-stakes contexts, but an unproven empirical premise is an evidentiary or validity concern, not circularity. The honest finding is therefore no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

The paper is a speculative position essay with no fitted parameters. It draws on established prior work for its premises and introduces the 'fake friend dilemma' as a conceptual label rather than an empirically validated entity.

assumptions (5)
  • domain assumption Conversational search platforms will incorporate advertising in the near future.
    The paper relies on industry reports (OpenAI exploring ads, Perplexity testing) to motivate the scenario; this is an external assumption, not derived.
  • domain assumption Users often fail to recognize native advertising in conversational search unless disclosed.
    Cited from Zelch et al. (2024); underlying the argument that disguised ads are a risk.
  • domain assumption Users place more trust in conversational agents than in traditional search for health information.
    Cited from Sun et al. (2024); necessary for the 'fake friend dilemma'.
  • domain assumption Value misalignment between AI providers and users is possible and consequential.
    Cited from Manzini et al. (2024); the paper's conceptual framing.
  • ad hoc to paper The elicited ChatGPT outputs, though prompted, illustrate a plausible configuration of advertising integration.
    The paper's examples are generated by explicit instruction (Appendix A) and are not organic observations, yet they are used to indicate future risk.
invented entities (1)
  • Fake friend dilemma
    purpose: Describes a conversational agent that appears to act in the user's interest while actually serving advertisers.
    Introduced in this paper; currently a framing concept supported only by the paper's own hypothetical examples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fake Friends and Sponsored Ads: The Risks of Advertising in Conversational Search." pith.science (2026). https://pith.science/paper/X5KUFGS2

@misc{pith2026250606447,
  author       = {Pith},
  title        = {Pith review of: Fake Friends and Sponsored Ads: The Risks of Advertising in Conversational Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X5KUFGS2}},
  note         = {Machine review of arXiv:2506.06447}
}
read the original abstract

Digital commerce thrives on advertising, with many of the largest technology companies relying on it as a significant source of revenue. However, in the context of information-seeking behavior, such as search, advertising may degrade the user experience by lowering search quality, misusing user data for inappropriate personalization, potentially misleading individuals, or even leading them toward harm. These challenges remain significant as conversational search technologies, such as ChatGPT, become widespread. This paper critically examines the future of advertising in conversational search, utilizing several speculative examples to illustrate the potential risks posed to users who seek guidance on sensitive topics. Additionally, it provides an overview of the forms that advertising might take in this space and introduces the "fake friend dilemma," the idea that a conversational agent may exploit unaligned user trust to achieve other objectives. This study presents a provocative discussion on the future of online advertising in the space of conversational search and ends with a call to action.

Figures

Figures reproduced from arXiv: 2506.06447 by the authors.

Figure 1
Figure 1. In the following hypothetical example, a user asks ChatGPT for guidance on improving well-being and gets a [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. In the following hypothetical example of native advertising, a user asks ChatGPT for guidance on improving [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. In the following hypothetical example of native advertising, a user asks ChatGPT for guidance on improving [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: In the following hypothetical example of native advertising, a user asks ChatGPT for guidance on improving [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: In the following hypothetical example, a user discusses their mental wellness with ChatGPT in a series of messages. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 32 canonical work pages

  1. [36]

    refreshing Pepsi

    Ines Zelch, Matthias Hagen, and Martin Potthast. 2024. A User Study on the Acceptance of Native Advertising in Generative IR. In Proceedings of the 2024 Conference on Human Information Interaction and Retrieval . 142–152. CUI ’25, July 8–10, 2025, Waterloo, ON, Canada Jacob Erickson Appendix A Table 1: Initial Prompts Provided to ChatGPT Initial Prompt As...

  2. [31]

    Xin Sun, Rongjun Ma, Xiaochang Zhao, Zhuying Li, Janne Lindqvist, Abdallah El Ali, and Jos A Bosch. 2024. Trusting the search: unraveling human trust in health information from Google and ChatGPT. arXiv preprint arXiv:2403.09987 (2024). The Risks of Advertising in Conversational Search CUI ’25, July 8–10, 2025, Waterloo, ON, Canada

  3. [1]

    Michelle A Amazeen and Bartosz W Wojdynski. 2020. The effects of disclosure format on native advertising recognition and audience perceptions of legacy and online news publishers. Journalism 21, 12 (2020), 1965–1984

  4. [2]

    Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023)

  5. [3]

    Sophie C Boerman, Sanne Kruikemeier, and Nadine Bol. 2021. When is personal- ized advertising crossing personal boundaries? How type of information, data sharing, and personalized pricing influence consumer perceptions of personalized advertising. Computers in Human Behavior Reports 4 (2021), 100144

  6. [4]

    Margaret C Campbell and Amna Kirmani. 2000. Consumers’ use of persuasion knowledge: The effects of accessibility and cognitive capacity on perceptions of an influence agent. Journal of consumer research 27, 1 (2000), 69–83

  7. [5]

    Henry Curtis and Joseph Milner. 2020. Ethical concerns with online direct-to- consumer pharmaceutical companies. Journal of medical ethics 46, 3 (2020), 168–171

  8. [6]

    Peter R Darke and Robin JB Ritchie. 2007. The defensive consumer: Advertising deception, defensive processing, and distrust. Journal of Marketing research 44, 1 (2007), 114–127

Show all 37 references
  1. [7]

    Freya De Keyzer, Guda Van Noort, and Sanne Kruikemeier. 2022. Going too far? How consumers respond to personalized advertising from different sources. Journal of electronic commerce research.-Long Beach, Calif.,[200-], currens 23, 3 (2022), 138–159

  2. [8]

    Michael J DiStefano, Jenny M Markell, Caroline C Doherty, G Caleb Alexander, and Gerard F Anderson. 2023. Association between drug characteristics and manufacturer spending on direct-to-consumer advertising. JAMA 329, 5 (2023), 386–392

  3. [9]

    I don’t know what you mean byI am anxious’

    Tessa Eagle, Conrad Blau, Sophie Bales, Noopur Desai, Victor Li, and Steve Whittaker. 2022. “I don’t know what you mean byI am anxious’”: A New Method for Evaluating Conversational Agent Responses to Standardized Mental Health Inputs for Anxiety and Depression. ACM Transaction...

  4. [10]

    Olivia Foulds, Leif Azzopardi, and Martin Halvey. 2021. Investigating the influence of ads on user search performance, behaviour, and experience during information seeking. In Proceedings of the 2021 Conference on Human Information Interaction and Retrieval. 107–117

  5. [11]

    Becky Freeman, Christina Watts, and Putu Ayu Swandewi Astuti. 2022. Global tobacco advertising, promotion and sponsorship regulation: what’s old, what’s new and where to next? Tobacco Control 31, 2 (2022), 216–221

  6. [12]

    Marian Friestad and Peter Wright. 1994. The persuasion knowledge model: How people cope with persuasion attempts. Journal of consumer research 21, 1 (1994), 1–31

  7. [13]

    Robert A Gottfried. 2014. Six ways this article is most definitely not an ad: Deceptive marketing and the need for clearly-defined disclosure rules in online native advertisement. Loy. Consumer L. Rev. 27 (2014), 399

  8. [14]

    John Herrman. 2023. The Junkification of Amazon Why does it feel like the company is making itself worse? New York Magazine (2023)

  9. [15]

    Guillaume Hervet, Katherine Guérard, Sébastien Tremblay, and Mohamed Saber Chtourou. 2011. Is banner blindness genuine? Eye tracking internet text adver- tising. Applied cognitive psychology 25, 5 (2011), 708–716

  10. [16]

    David A Hyman, David Franklyn, Calla Yee, and Mohammad Rahmati. 2017. Going native: Can consumers recognize native advertising: Does it matter. Yale JL & Tech. 19 (2017), 77

  11. [17]

    Khondoker Tashya Kalam, Jannatul Mabia Rahman, Md Rabiul Islam, and Syed Masudur Rahman Dewan. 2024. ChatGPT and mental health: Friends or foes? Health Science Reports 7, 2 (2024), e1912

  12. [18]

    Jooyoung Kim. 2017. Native advertising: Current status and research agenda. Journal of interactive advertising 17, 2 (2017), 79–79

  13. [19]

    Chih-Wei Liu, Shao-Kang Lo, Ai-Yun Hsieh, and Yujong Hwang. 2018. Effects of banner Ad shape and the schema creating process on consumer internet browsing behavior. Computers in Human Behavior 86 (2018), 9–17

  14. [20]

    Arianna Manzini, Geoff Keeling, Nahema Marchal, Kevin R McKee, Verena Rieser, and Iason Gabriel. 2024. Should users trust advanced AI assistants? Justified trust as a function of competence and alignment. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, a...

  15. [21]

    Madhumita Murgia, Cristina Criddle, and George Hammond. 2024. OpenAI explores advertising as it steps up revenue drive. Financial Times (2024)

  16. [22]

    OpenAI. 2024. Memory and new controls for ChatGPT. https://openai.com/ index/memory-and-new-controls-for-chatgpt/

  17. [23]

    Eleonora Pantano. 2021. When a luxury brand bursts: Modelling the social media viral effects of negative stereotypes adoption leading to brand hate. Journal of Business Research 123 (2021), 117–125

  18. [24]

    Serhat Peker, Gonca Gokce Menekse Dalveren, and Yavuz İnal. 2021. The effects of the content elements of online banner ads on visual attention: evidence from an-eye-tracking study. Future Internet 13, 1 (2021), 18

  19. [25]

    J Porter. 2023. Chatgpt continues to be one of the fastest-growing services ever. TheVerge (2023)

  20. [26]

    Simona Romani, Silvia Grappi, Lia Zarantonello, and Richard P Bagozzi. 2015. The revenge of the consumer! How brand moral violations lead to consumer anti-brand activism. Journal of brand Management 22 (2015), 658–672

  21. [27]

    Kya Sainsbury-Carter. 2025. Transforming the future of audience engagement. Microsoft.com (2025)

  22. [28]

    Sebastian Schmidt, Ines Zelch, Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast. 2024. Detecting Generated Native Ads in Conversational Search. In Companion Proceedings of the ACM on Web Conference 2024 . 722–725

  23. [29]

    Leo Schwinn, David Dobre, Sophie Xhonneux, Gauthier Gidel, and Stephan Gun- nemann. 2024. Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space. arXiv preprint arXiv:2402.09063 (2024)

  24. [30]

    Om P Singh. 2023. Artificial intelligence in the era of ChatGPT-Opportunities and challenges in mental health care. Indian Journal of Psychiatry 65, 3 (2023), 297–298

  25. [32]

    Perplexity Team. 2024. Why we’re experimenting with advertising. Perplexity AI (2024)

  26. [33]

    Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al

  27. [34]

    Bartosz W Wojdynski. 2016. The deceptiveness of sponsored news articles: How readers recognize and perceive native advertising. American Behavioral Scientist 60, 12 (2016), 1475–1491

  28. [35]

    Bartosz W Wojdynski and Nathaniel J Evans. 2016. Going native: Effects of disclosure position and language on the recognition and evaluation of online native advertising. Journal of Advertising 45, 2 (2016), 157–168

  29. [2021]

    arXiv preprint arXiv:2112.04359 (2021)

    Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 (2021)

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.