Persuasion and Safety in the Era of Generative AI

Haein Kong

REVIEW 3 major objections 3 minor 22 references

Persuasion and Safety in the Era of Generative AI

T0 review · 3 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This dissertation proposal claims that the missing empirical resource for AI persuasion safety is a human-annotated dataset that separates rational persuasion from manipulation, and it sets out to build one.

desk verdict A clear, honest PhD proposal that identifies a real gap, but the annotation plan does not operationalize the definition of manipulation; the construct validity problem is load-bearing. read the letter →

arxiv 2505.12248 v1 pith:FG5WG6SN submitted 2025-05-18 cs.CY

Haein Kong This is my paper · ORCID

classification cs.CY

keywords persuasionmanipulationLLMsafetyhumanannotationtaxonomyEUAIActpersuasivecognitivebias

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a dissertation proposal arguing that one specific gap blocks progress on AI persuasion safety: there is no human-annotated dataset that labels persuasive texts as either rational persuasion or manipulation. To close that gap, the author plans a three-part project: design a taxonomy that sorts persuasive techniques by whether they appeal to reason or exploit cognitive biases, annotate roughly 2,000 comments from a public online-discussion dataset, and evaluate whether current large language models can make the same distinction. The author contends that this distinction matters because the EU AI Act prohibits AI that uses manipulative techniques to impair informed decision-making, making safe-versus-unsafe persuasion a regulatory and technical problem. A successful dataset would let researchers benchmark and train models to flag manipulative AI output.

What carries the argument

The central object is the taxonomy that separates persuasive techniques into rational persuasion and manipulation, grounded in cognitive theory: rational persuasion is aligned with the deliberate, systematic mode of information processing (System 2), while manipulation targets the fast, automatic, heuristic mode (System 1). This taxonomy is the mechanism that turns a fuzzy normative distinction into annotatable labels, and it does the work of making the planned dataset possible. A second load-bearing piece is the chosen source corpus, a public dataset of persuasive online comments, which supplies the raw texts that the research team will filter and annotate. The project's evaluation step then uses the annotated dataset as a benchmark to test whether LLMs reproduce the human distinction under zero-shot and few-shot prompting.

What would settle it

The central gap claim would be falsified by locating any published dataset that already carries human labels for rational persuasion versus manipulation; the annotation plan would be falsified by a pilot study in which independent annotators only reach low agreement (for example, Cohen's kappa below 0.4) on the proposed taxonomy.

Watch

Extended reading notes

Core claim

The paper claims that rational persuasion, which engages reason and evidence through a person's deliberate System 2 processing, and manipulation, which covertly exploits cognitive shortcuts in System 1/heuristic processing, have not yet been separated empirically in NLP. It asserts that no existing dataset carries human-annotated labels for this classification, and that current persuasion datasets treat persuasion at too broad a level. The proposed contribution is therefore to create the missing resource: a taxonomy of persuasion techniques, a human-annotated dataset of roughly 1,000 comments in each category drawn from public online discussion data, and baseline evaluations of current LLMs using zero-shot and few-shot prompting. The author frames this as the first empirical step toward automatic detection of unsafe persuasion and as a benchmark that future AI-safety research can build on.

Load-bearing premise

The whole argument rests on the premise that a small research team can define the boundary between rational persuasion and manipulation clearly enough that human annotators will mostly agree on it.

Editorial extensions

If this is right

If the dataset is built as planned, it becomes the first standard benchmark for classifying rational persuasion versus manipulation, letting future work train detectors rather than argue definitions anew.
The taxonomy would give AI-safety teams a concrete list of techniques that count as manipulation, making it easier to test models against the EU AI Act's prohibition on manipulative AI practices.
Baseline evaluations with zero- and few-shot prompting would measure how far current LLMs are from reliably distinguishing the two, quantifying the safety gap.
The follow-up proposal to improve LLM classification with cognitive-theory-inspired prompting would connect human dual-process models to machine classification performance.
Future researchers could extend the taxonomy with additional subtechniques and use the dataset to benchmark their own detectors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the taxonomy reliably separates the two, the same annotation scheme could be applied to other persuasive texts such as advertisements, political campaign messages, or chatbot sales dialogues, not just online discussion comments.
The project's success will hinge on whether a small team can annotate a concept that philosophy and law still debate; a likely testable extension is measuring inter-annotator agreement and publishing disagreement cases as a secondary dataset.
The paper's framing implies a monotone link between System 1 processing and harm, but some System 1 cues can be benign; a more detailed account of which heuristics are harmful would sharpen the taxonomy.
LLM evaluation results on this dataset could later be used to design probes that test whether models can be induced to produce manipulative output, connecting classification to generation-side safety.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a PhD symposium proposal, not a results paper. The author proposes a taxonomy, a ~2,000-comment annotated dataset, and an LLM evaluation to distinguish rational persuasion from manipulation. The conceptual distinction already exists in the cited literature [7, 11, 18]; what's new is the plan to build resources around it. The writing is clear, the motivation is relevant given the EU AI Act, and the author is appropriately upfront that everything is future work. That honesty earns credit.

The paper does several things well. It correctly notes that NLP persuasion research tends to treat persuasion as a single category, ignoring the rational/manipulative distinction. The two-stage plan—baseline evaluation, then improvement with cognitive-theory-inspired prompting—is sensible. The connection to dual-process theory is plausible, though it is a hypothesis, not a result.

The soft spots are real, and the stress-test note lands. The definition of manipulation used in Sec. 2.3 hinges on sender intent and covertness. But the planned data source, r/ChangeMyView comments, gives annotators only the text. They can guess intent but cannot observe it. Pilot tests and tutorial sessions address inter-annotator reliability, not construct validity. Even perfect agreement would not show that the labels measure what the paper claims they measure. This is the central weakness of the proposal, and it needs a fix before the dataset can support the benchmark. One possibility is to define manipulation in terms of observable textual tactics rather than unobservable intent, or to generate examples where intent is known by construction.

Minor issues: the claim that no existing dataset exists (Sec. 3.1) is asserted without a systematic search, and r/ChangeMyView is a good-faith debate setting where manipulative comments may be rare, creating possible imbalance or unrepresentativeness. Those are worth noting but not fatal.

Who is this for? Anyone working on persuasive AI safety or on the design of datasets for ethically loaded constructs might find the proposal a useful starting point. As a research preprint it has no citable results yet. For a PhD symposium, it is appropriate; a serious referee would give useful feedback, mainly on the construct validity issue. I would not desk reject it if the venue accepts proposals, but I would recommend revision to make the annotation protocol operationalize the definition.

If you are in this area, it is worth a quick read as a proposal example. If you are asked to referee, the key point to raise is the intent problem.

Referee Report

3 major / 3 minor

Summary. This manuscript is a PhD-symposium dissertation proposal on distinguishing rational persuasion from manipulation in the era of generative AI. The author argues that existing NLP research treats persuasion at too coarse a level, that no dataset currently exists with human-annotated labels for rational persuasion versus manipulation, and that LLMs should be evaluated on their ability to make this distinction. The proposed work has three stages: (1) build a taxonomy of persuasive techniques, construct a roughly 2,000-comment human-annotated dataset from the WinningArguments/r/ChangeMyView corpus, and evaluate baseline LLMs via zero-shot and few-shot prompting; (2) design prompt-engineering, fine-tuning, or framework methods to improve LLM classification; and (3) contribute the resulting taxonomy, dataset, and benchmark to AI-safety research. The paper contains no empirical data, experiments, or formal derivations; it is a research plan.

Significance. If the proposed dataset and benchmark are built and validated, they could be a useful resource for NLP and AI-safety research on persuasive LLMs, particularly in light of the EU AI Act's prohibitions on manipulative AI. The paper's strengths are its clear framing of the rational-persuasion-versus-manipulation distinction, its integration of relevant literature from philosophy, psychology, and NLP, and its concrete plan to use an existing public dataset rather than collecting new raw data. However, the central contribution is entirely prospective: the manuscript reports no results, no annotation, and no evaluation. The significance therefore hinges on whether the planned annotation can actually instantiate the constructs defined in Section 2.3, and on whether the claimed gap (no existing dataset) is established by a systematic search. These points are presently not secured.

major comments (3)

[Section 2.3 and Section 3.1] The proposed dataset's construct validity is not operationalized. Section 2.3 defines manipulation as 'intentionally and covertly influencing' decision-making and rational persuasion as involving sender intent, but Section 3.1 plans to annotate public r/ChangeMyView comments, where annotators see only text and cannot observe the sender's actual intention or covertness. The manuscript says the research team will hold tutorial sessions and a pilot test, but these address inter-annotator agreement, not whether the labels correspond to the stated constructs. Please specify observable textual criteria or a coding protocol that operationalizes intent and covertness, or explicitly revise the construct definitions to what can be annotated from text, and discuss how the resulting labels support the proposed LLM benchmark.
[Section 3.1] The load-bearing claim that 'there is no existing dataset built on this classification in the literature' is asserted without a systematic search protocol. The author does not state which databases were searched, which queries were used, which inclusion/exclusion criteria were applied, or how prior persuasion datasets (e.g., [10, 19, 20]) were checked for relevant sublabels. Because this gap motivates the entire dissertation, please provide a reproducible search method or weaken the claim to a scoped statement (e.g., 'no dataset with explicit rational-persuasion-versus-manipulation labels that we located'), so a reviewer or reader can verify the gap.
[Section 3.1] The annotation plan does not address measurement reliability beyond mentioning tutorial sessions and a pilot. For a dataset intended as a benchmark, it needs explicit inter-annotator agreement targets (e.g., Cohen's kappa or Krippendorff's alpha), a plan for resolving disagreements, and a description of how annotation guidelines will be iterated. Additionally, r/ChangeMyView is a deliberative, good-faith debate community; its comments may be unrepresentative of manipulative persuasion, which is often covert and may not appear in public argumentative contexts. Please discuss how the sampling frame affects the prevalence and representativeness of the two target classes and whether the dataset can support claims about manipulation in general.

minor comments (3)

[Section 2.4] The model is repeatedly called the 'Heuristics and Systematic Model'; the standard name is the 'Heuristic-Systematic Model,' and 'heuristics' should be singular. Please correct this terminology throughout.
[Section 3.1] The target of roughly 1,000 comments per class is stated without justification. Please explain how this size was chosen (e.g., annotation budget, expected class balance, or desired statistical power for the downstream LLM evaluation).
[References] Several references are cited only by URL (e.g., [1], [6], [15]). For reproducibility, provide version numbers, access dates, and stable identifiers or DOIs where available.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the proposal makes no fitted predictions and its few self-citations are not load-bearing.

full rationale

This paper is a dissertation proposal rather than a derivation, so there is no chain of equations or fitted parameters that could reduce to its inputs. The central claim is that no existing dataset labels rational persuasion versus manipulation and that the author will build one; that is an empirical gap claim, not a circular prediction. The taxonomy is explicitly built from external prior work [7, 18], and the planned annotation process is a future data-construction effort, not a retrofitted fit. The only self-citation, [13], supports a background statement about emotion-detection tasks and plays no role in the argument. The reader's noted concern about construct validity, namely that annotators label public comments without direct access to sender intent, is a substantive correctness risk for the proposed dataset, but it is not circularity: the proposal does not claim to have derived the labels from the definition by construction, and no result is being presented as a prediction that is equivalent to its own input. Therefore, no circular step is present, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper proposes no new entities or fitted parameters. The axioms are conceptual and methodological assumptions imported from prior work and from the dissertation plan, none of which are validated within this preprint. The central weight falls on the assumption that the rational persuasion/manipulation boundary can be annotated reliably, which is asserted rather than demonstrated.

assumptions (4)

domain assumption The binary distinction rational persuasion versus manipulation is well-defined and annotatable (Sections 2.3 and 3.1).
The entire proposed dataset depends on this boundary being reliably annotatable, but the paper provides no pilot results or inter-annotator agreement evidence, only a plan to run tutorials.
domain assumption Manipulation relies on System 1/heuristic processing while rational persuasion relies on System 2/systematic processing (Section 2.4).
This mapping is asserted from the dual process and HSM literatures without empirical support in the paper; it is used as the theoretical basis for the taxonomy.
domain assumption WinningArguments comments from r/ChangeMyView are a suitable source for examples of persuasive techniques (Section 3.1).
The example distribution in rational debate forums may not represent real-world persuasive AI interactions, but the proposal relies on it as the dataset source.
domain assumption EU AI Act interpretation: manipulative and deceptive techniques are prohibitively risky (Section 2.2).
The paper adopts this reading as the motivation for the work without engaging with the interpretive debate it cites [21] in depth.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Persuasion and Safety in the Era of Generative AI." pith.science (2026). https://pith.science/paper/FG5WG6SN

@misc{pith2026250512248,
  author       = {Pith},
  title        = {Pith review of: Persuasion and Safety in the Era of Generative AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FG5WG6SN}},
  note         = {Machine review of arXiv:2505.12248}
}

read the original abstract

As large language models (LLMs) achieve advanced persuasive capabilities, concerns about their potential risks have grown. The EU AI Act prohibits AI systems that use manipulative or deceptive techniques to undermine informed decision-making, highlighting the need to distinguish between rational persuasion, which engages reason, and manipulation, which exploits cognitive biases. My dissertation addresses the lack of empirical studies in this area by developing a taxonomy of persuasive techniques, creating a human-annotated dataset, and evaluating LLMs' ability to distinguish between these methods. This work contributes to AI safety by providing resources to mitigate the risks of persuasive AI and fostering discussions on ethical persuasion in the age of generative AI.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 14 canonical work pages

[1]

Artificial Intelligence Act. 2025. Article 5 - Prohibited AI Practices. https: //artificialintelligenceact.eu/article/5/ Accessed: 2025-03-09

work page 2025
[2]

Hui Bai, Jan Voelkel, Johannes Eichstaedt, and Robb Willer. 2023. Artificial intelligence can persuade humans on political issues. (2023)

work page 2023
[3]

Yoshua Bengio, Sören Mindermann, Daniel Privitera, Tamay Besiroglu, Rishi Bommasani, Stephen Casper, Yejin Choi, Philip Fox, Ben Garfinkel, Danielle Gold- farb, et al. 2025. International AI Safety Report. arXiv preprint arXiv:2501.17805 (2025)

arXiv 2025
[4]

Gerd Bohner, Hans-Peter Erb, and Frank Siebler. 2008. Information processing approaches to persuasion: Integrating assumptions from the dual-and single- processing perspectives. Attitudes and attitude change (2008), 161–188

work page 2008
[5]

Gerd Bohner, Gordon B Moskowitz, and Shelly Chaiken. 1995. The interplay of heuristic and systematic processing of social information. European review of social psychology 6, 1 (1995), 33–68

work page 1995
[6]

Esin Durmus, Liane Lovitt, Alex Tamkin, Stuart Ritchie, Jack Clark, and Deep Ganguli. 2024. Measuring the Persuasiveness of Language Models . https://www. anthropic.com/news/measuring-model-persuasiveness

work page 2024
[7]

Seliem El-Sayed, Canfer Akbulut, Amanda McCroskery, Geoff Keeling, Zachary Kenton, Zaria Jalan, Nahema Marchal, Arianna Manzini, Toby Shevlane, Shannon Vallor, et al. 2024. A mechanism-based approach to mitigating harms from persuasive generative ai. arXiv preprint arXiv:2404.15058 (2024)

arXiv 2024
[8]

2022.Persuasion: Social influence and compliance gaining

Robert H Gass and John S Seiter. 2022.Persuasion: Social influence and compliance gaining. Routledge

work page 2022

Show all 22 references

[9]

Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. 2023. An overview of catastrophic ai risks. arXiv preprint arXiv:2306.12001 (2023)

2023 arXiv
[10]

Chuhao Jin, Kening Ren, Lingzhen Kong, Xiting Wang, Ruihua Song, and Huan Chen. 2024. Persuading across Diverse Domains: a Dataset and Persuasion Large Language Model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

2024
[11]

Cameron R Jones and Benjamin K Bergen. 2024. Lies, Damned Lies, and Distri- butional Language Statistics: Persuasion and Deception with Large Language Models. arXiv preprint arXiv:2412.17128 (2024)

2024 arXiv
[12]

Daniel Kahneman. 2011. Thinking, fast and slow . macmillan

2011
[13]

Haein Kong and Seonghyeon Moon. 2024. RU at WASSA 2024 Shared Task: Task-Aligned Prompt for Predicting Empathy and Distress. In Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis. Association for Computational Lingu...

2024
[14]

Michal Kosinski. 2023. Theory of mind may have spontaneously emerged in large language models. arXiv preprint arXiv:2302.02083 4 (2023), 169

2023 arXiv
[15]

OpenAI. 2024. OpenAI o1 System Card . https://openai.com/index/openai-o1- system-card/

2024
[16]

Charlie Pinder, Jo Vermeulen, Benjamin R Cowan, and Russell Beale. 2018. Digital behaviour change interventions to break and form habits. ACM Transactions on Computer-Human Interaction (TOCHI) 25, 3 (2018), 1–66

2018
[17]

Alexander Rogiers, Sander Noels, Maarten Buyl, and Tijl De Bie. 2024. Persuasion with Large Language Models: a Survey. arXiv preprint arXiv:2411.06837 (2024)

2024 arXiv
[18]

Daniel Susser, Beate Roessler, and Helen Nissenbaum. 2019. Technology, auton- omy, and manipulation. Internet policy review 8, 2 (2019), 1–22

2019
[19]

Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee
[20]

Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019. Persuasion for good: Towards a personalized persuasive dialogue system for social good. arXiv preprint arXiv:1906.06725 (2019)

2019 arXiv
[21]

Huixin Zhong, Eamonn O’Neill, and Janina A Hoffmann. 2024. Regulating AI: Applying Insights from Behavioural Economics and Psychology to the Appli- cation of Article 5 of the EU AI Act. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 20001–20009

2024
[2016]

In Proceedings of the 25th international conference on world wide web

Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. In Proceedings of the 25th international conference on world wide web . 613–624

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Artificial Intelligence Act. 2025. Article 5 - Prohibited AI Practices. https: //artificialintelligenceact.eu/article/5/ Accessed: 2025-03-09

work page 2025

[2] [2]

Hui Bai, Jan Voelkel, Johannes Eichstaedt, and Robb Willer. 2023. Artificial intelligence can persuade humans on political issues. (2023)

work page 2023

[3] [3]

Yoshua Bengio, Sören Mindermann, Daniel Privitera, Tamay Besiroglu, Rishi Bommasani, Stephen Casper, Yejin Choi, Philip Fox, Ben Garfinkel, Danielle Gold- farb, et al. 2025. International AI Safety Report. arXiv preprint arXiv:2501.17805 (2025)

arXiv 2025

[4] [4]

Gerd Bohner, Hans-Peter Erb, and Frank Siebler. 2008. Information processing approaches to persuasion: Integrating assumptions from the dual-and single- processing perspectives. Attitudes and attitude change (2008), 161–188

work page 2008

[5] [5]

Gerd Bohner, Gordon B Moskowitz, and Shelly Chaiken. 1995. The interplay of heuristic and systematic processing of social information. European review of social psychology 6, 1 (1995), 33–68

work page 1995

[6] [6]

Esin Durmus, Liane Lovitt, Alex Tamkin, Stuart Ritchie, Jack Clark, and Deep Ganguli. 2024. Measuring the Persuasiveness of Language Models . https://www. anthropic.com/news/measuring-model-persuasiveness

work page 2024

[7] [7]

Seliem El-Sayed, Canfer Akbulut, Amanda McCroskery, Geoff Keeling, Zachary Kenton, Zaria Jalan, Nahema Marchal, Arianna Manzini, Toby Shevlane, Shannon Vallor, et al. 2024. A mechanism-based approach to mitigating harms from persuasive generative ai. arXiv preprint arXiv:2404.15058 (2024)

arXiv 2024

[8] [8]

2022.Persuasion: Social influence and compliance gaining

Robert H Gass and John S Seiter. 2022.Persuasion: Social influence and compliance gaining. Routledge

work page 2022