Promoting Online Safety by Simulating Unsafe Conversations with LLMs

Kangze Peng; Owen Hoffman; Sajid Kamal; Sukrit Venkatagiri; Zehua You

REVIEW 3 major objections 4 minor 21 references

Promoting Online Safety by Simulating Unsafe Conversations with LLMs

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two simulated LLM personas—one pushing for information, one resisting under user coaching—can produce an effective scam conversation, the paper reports.

desk verdict A short, honest workshop paper describing a two-agent scam simulation with user steering; the design is new in combination, but the effectiveness claim is anecdotal and the educational transfer is unvalidated. read the letter →

arxiv 2507.22267 v1 pith:HAMVZCWP submitted 2025-07-29 cs.HC cs.AI

Owen Hoffman , Kangze Peng , Zehua You , Sajid Kamal , Sukrit Venkatagiri This is my paper

classification cs.HCcs.AI

keywords LLMsimulationscamconversationsonlinesafetyeducationconversationalagentspersonapromptinguserfeedbackanti-scamtrainingguardrails

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Generative AI is making online scams faster and cheaper to produce, and this paper proposes using the same technology to train people to resist them. The authors built a system in which one LLM plays a scammer and another plays a would-be victim, while a human user coaches the victim by giving feedback. The paper reports that role-play prompts, a deliberate model pairing, and few-shot examples produce a realistic and effective scam conversation. This matters because it offers learners a hands-on way to encounter scam tactics before real ones occur, although the reported evidence is qualitative and no user outcomes are measured yet.

What carries the argument

The mechanism is a two-agent conversation loop with an external human coach. One LLM receives a scammer persona (urgent, persistent, information-seeking) and the other a victim persona (kind, trusting, disclosure-prone), while the user steers the victim by supplying feedback. The loop is made to work by three prompt-engineering choices: role-play phrasing that avoids safety triggers, personas explicitly described as 'not value aligned' to relax guardrails, and few-shot examples of the desired conversation. Model personality is treated as a tunable component, with a more assertive model placed in the scammer role and a more compliant, feedback-absorbing model in the target role.

What would settle it

Give one group the interactive simulation, another group the same scam examples presented as static text, and a third no training, then test all three on new, unseen scam messages; the central claim fails if the simulation group does not identify or refuse those messages at a clearly higher rate.

Watch

Extended reading notes

Core claim

The paper's central claim is that a paired-LLM simulation can generate realistic, unsafe scam conversations for online-safety education. In the system, the scammer agent is instructed to create urgency and keep asking for personal information, while the target agent is instructed to be kind and trusting so it can plausibly fall for the scam. The user watches the exchange and offers real-time feedback to the target, trying to keep it from giving in. The authors report that they made an effective scam conversation by pairing a more assertive model as the scammer with a more feedback-responsive model as the target, and by moving from zero-shot to few-shot prompting. They also report that framing the scammer as an 'expert persuader' and describing the characters as 'not value aligned' let the agents sidestep many safety guardrails, and they say a quantitative evaluation is still in progress.

Load-bearing premise

The load-bearing assumption is that practicing by coaching a simulated victim transfers to real life—that people who use the system actually recognize and resist real scams afterward; the paper states this as a belief but reports no user data.

Editorial extensions

If this is right

If the simulation works as reported, learners can face realistic scam lures—urgency, repeated information requests, fake IT help, requests for passwords or money—without real-world harm.
If feedback during the dialogue builds a mental model of how these conversations unfold, users may get better at steering real encounters toward safer outcomes.
The success of the two-model combination suggests that agent personality and model choice are design parameters for safety training systems, not just implementation details.
Because the system reportedly produces convincing scam content, it can double as a generator of training examples or awareness demonstrations, although the same capability is a safety concern.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A key transfer question the paper leaves open is whether coaching a simulated victim changes how people respond to real scam messages; this is the assumption on which the whole training rationale rests.
The reported guardrail-bypassing prompts are a double-edged artifact: if the system reaches wider use, the same two-phrase recipe could be copied by actual scammers, a risk the authors acknowledge and hope future safeguards fix.
The same two-persona-plus-feedback architecture could be pointed at other unsafe conversations, such as grooming, extortion, or harassment, with the open question being how realistic the target agent's behavior is in those domains.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a work-in-progress workshop paper, and it reads like one: the system design is coherent, the authors are honest about what is missing, and the central claim is more of a proposal than a result. The punchline is that if you need a validated intervention for scam education, this is not there yet; if you want a sensible design for a two-agent simulation with user feedback, it is worth a look.

What is actually new: the specific combination of a scammer LLM and a target LLM where the user steers the target with feedback, plus the deliberate pairing of OpenAI as the scammer and Gemini as the target based on observed behavioral differences. That is a modest but genuine extension of prior systems like ScamGPT-J (single agent) and Rehearsal or GPTeach (no live scam-specific feedback loop). The paper also does some things well: it cites relevant prior work without padding, it clearly explains the prompt-engineering choices (temperature, few-shot, 'persuader' roleplay, 'not value aligned'), and it flags the dual-use concern about circumventing guardrails rather than hiding it.

The soft spots are real and load-bearing. The claim in Section 3 that 'we were able to make an effective scam conversation' is supported only by qualitative assessment—no transcripts, no metrics, no independent rating. The stress-test note is on target: the abstract-level claim about promoting online safety requires transfer from simulated practice to real-world scam resistance, and the paper provides no evidence for that. Section 4 says 'we believe users are more likely to learn' and Section 5 defers efficacy evaluation to future work. That is honest, but it means the paper currently offers no support for its central motivational claim. The guardrail circumvention section is also a legitimate concern, though the authors acknowledge it and express hope for fixes.

This is not a desk-reject-level manuscript. It is a perfectly reasonable workshop submission with a clear design and an honest limitations section. A serious referee would ask for a user study, baseline comparison, and actual conversation examples, but the ideas are worth engaging with. I would not cite it for a result yet, but I might bring it to a reading group to discuss evaluation standards for safety-education tools.

Recommendation: send it to peer review, but expect and request substantial revision—specifically, evidence that the simulated conversations are realistic and that practicing on them changes real-world behavior.

Referee Report

3 major / 4 minor

Summary. The manuscript describes a system in which a scammer LLM and a target LLM converse with each other, while a user provides feedback to the target LLM to help it resist the scam. The authors report qualitative observations about model behavior, noting that OpenAI models acted more assertively and Gemini models were more receptive to feedback, and they describe prompt-engineering strategies such as few-shot prompting, the use of the term "persuader," and instructing the models that their characters are "not value aligned" to bypass safety guardrails. The paper claims that an "effective scam conversation" was produced and argues, without user data, that such simulations can promote online safety by helping users learn to recognize and resist scams. The authors state that quantitative evaluation of usability, safety, and efficacy is future work.

Significance. If the system's effectiveness and its learning-transfer mechanism were demonstrated, this would be a timely contribution to LLM-based safety training, addressing a real and growing threat. The design choice of combining two different model families and using user feedback to steer the target agent is interesting, and the authors are transparent about their qualitative evaluation and limitations. However, as it stands, the paper does not establish either of the two load-bearing claims: that the conversations are effective in any measurable sense, or that practicing on them transfers to real-world scam resistance. The contribution is therefore currently a system description and position statement rather than an empirical result, and its significance depends entirely on future evaluation.

major comments (3)

[Section 3, "Using these tools we were able to make an effective scam conversation"] This central claim is supported only by the authors' qualitative judgment. No conversation transcripts, interaction logs, evaluation rubric, baseline, or inter-rater reliability are provided, and "effective" is never defined (e.g., effective at eliciting information, at resembling real scams, or at engaging learners). Because the paper's motivation depends on the simulation being realistic, this needs to be operationalized and evaluated before the claim can be accepted.
[Sections 4 and 5] The title and abstract claim that the system promotes online safety, but the only stated evidence is "we believe users are more likely to learn" in Section 4, and Section 5 defers efficacy evaluation ("whether or not people are able to accurately identify scam conversations") to future work. No user study, pre/post measure, transfer task, or comparison baseline appears anywhere in the manuscript. The learning-science citation (Ion et al.) supports feedback-based learning in general, not transfer from this specific game to real scam encounters; the transfer assumption is the load-bearing premise and is untested. This must be addressed for the paper's central claim to hold.
[Sections 3 and 4, safety guardrail circumvention] The authors describe instructing LLMs that their characters are "not value aligned" to bypass safety restrictions and acknowledge that this sidesteps many safety features, yet the paper does not describe any IRB review, risk assessment, or debriefing protocol for participants, despite acknowledging the potential for "unwanted emotions or dramatizations" in Section 4. Given that the system is designed to produce manipulative content, this ethical dimension needs concrete safeguards and institutional context before deployment or user studies are conducted.

minor comments (4)

[Section 3, reference [19]] The phrase "not value aligned" is cited to reference [19], which is the Rehearsal paper; that reference does not appear to support this claim. The citation should be corrected or a proper source added.
[Sections 2 and 3, typographical issues] There are several typos and informal phrasings: "to provided advice" and "perspective managers" in Section 2, "Guyre's system" should be more formal, and "the the character" appears in Section 3.
[Reference list] The reference list has formatting problems, including "InProceedings" missing a space and the Ion et al. entry containing malformed author fields such as "Sánchez Martí , Angelina, ,".
[Section 3, reproducibility] The model details are underspecified: although "OpenAI" and "Gemini" are named, the prompt templates, temperature values, few-shot examples, and number of turns are not given, which limits reproducibility for a system-description paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an in-progress system description with qualitative claims, no fitted equations or predictions, no self-cited load-bearing results, and the safety-transfer gap is an unsupported assumption rather than a circular derivation.

full rationale

The paper reports an in-progress system in which two LLMs converse as scammer and target while the user gives feedback. The derivation chain is practical and empirical rather than formal: model selection (OpenAI as scammer, Gemini as target), temperature adjustment, and few-shot prompting are described as trial-and-error ('we found', 'we adjusted'), and the central claim 'we were able to make an effective scam conversation' is an authorial qualitative assessment. There are no equations, fitted parameters, or quantitative predictions, so none of the enumerated circularity patterns apply. The cited bases, ScamGPT-J and Ion et al., are external works rather than self-citations, and the paper does not invoke a uniqueness theorem or import an ansatz from prior same-author work. Section 4 explicitly notes that 'a more robust quantitative assessment is in progress,' and Section 5 defers evaluation of efficacy: 'whether or not people are able to accurately identify scam conversations.' These are honest limitations, but they identify an untested transfer assumption between simulated conversations and real-world scam resistance, which is a validity or evidence concern rather than circularity. Under the hard rules, unsupported qualitative claims without a specific self-referential reduction do not raise the circularity score. No circular steps found; score 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities or fitted numerical parameters. It relies on domain assumptions about simulation-based learning and LLM behavior that are stated but not validated in this work.

assumptions (3)

domain assumption Roleplaying simulation can improve learners' skills in domains such as negotiation, counseling, and cybersecurity.
Invoked in Section 2 as the basis for the training approach, citing simulation games in prior work.
domain assumption Providing feedback on one's hypothetical actions promotes learning.
Stated in the introduction and Section 4, citing Ion et al. [11] as the learning-science basis.
ad hoc to paper Two LLMs conversing with a human steering the target is a realistic enough proxy for a real scam encounter to produce learning transfer.
This is the central untested assumption of the system; it appears in Sections 3 and 4 where the authors claim effectiveness without user data.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Promoting Online Safety by Simulating Unsafe Conversations with LLMs." pith.science (2026). https://pith.science/paper/HAMVZCWP

@misc{pith2026250722267,
  author       = {Pith},
  title        = {Pith review of: Promoting Online Safety by Simulating Unsafe Conversations with LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HAMVZCWP}},
  note         = {Machine review of arXiv:2507.22267}
}

read the original abstract

Generative AI, including large language models (LLMs) have the potential -- and already are being used -- to increase the speed, scale, and types of unsafe conversations online. LLMs lower the barrier for entry for bad actors to create unsafe conversations in particular because of their ability to generate persuasive and human-like text. In our current work, we explore ways to promote online safety by teaching people about unsafe conversations that can occur online with and without LLMs. We build on prior work that shows that LLMs can successfully simulate scam conversations. We also leverage research in the learning sciences that shows that providing feedback on one's hypothetical actions can promote learning. In particular, we focus on simulating scam conversations using LLMs. Our work incorporates two LLMs that converse with each other to simulate realistic, unsafe conversations that people may encounter online between a scammer LLM and a target LLM but users of our system are asked provide feedback to the target LLM.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 12 canonical work pages

[1]

Role-Play Observations in a Trauma Course

2022. "Role-Play Observations in a Trauma Course" by Emily Horton, Claudia G. Interiano-Shiverdecker et al. https://research.library.kutztown. edu/jcps/vol15/iss2/18/

work page 2022
[2]

[2305.02547] PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits

2023. [2305.02547] PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits. https://arxiv.org/abs/2305.02547

arXiv 2023
[3]

Rangina Ahmad, Dominik Siemon, Ulrich Gnewuch, and Susanne Robra-Bissantz. 2022. A Framework of Personality Cues for Conversational Agents. doi:10.24251/HICSS.2022.524

work page doi:10.24251/hicss.2022.524 2022
[4]

Hwang, and Weichao Yuwen

Rebekah Lee Baik, Stephanie Lee, Serena Jinchen Xie, Wang Liao, Elina H. Hwang, and Weichao Yuwen. 2025. Adapting Communication Styles in Health Chatbot using Large Language Models to Support Family Caregivers from Multicultural Backgrounds. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’25) ....

arXiv 2025
[6]

Noam Ebner and Yael Efron. 2005. Using Tomorrow’s Headlines for Today’s Training: Creating Pseudo-reality in Conflict Resolution Simulation Games. Negotiation Journal 21, 3 (2005), 377–394. doi:10.1111/j.1571-9979.2005.00070.x _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1571- 9979.2005.00070.x

work page arXiv 2005
[7]

Gilad Gressel, Rahul Pankajakshan, and Yisroel Mirsky. 2024. Discussion Paper: Exploiting LLMs for Scam Automation: A Looming Threat. In Proceedings of the 3rd ACM Workshop on the Security Implications of Deepfakes and Cheapfakes . ACM, Singapore Singapore, 20–24. doi:10.1145/ 3660354.3660356

work page arXiv 2024
[8]

Heng Gu, Chadha Degachi, Uğur Genç, Senthil Chandrasegaran, and Himanshu Verma. 2023. On the Effectiveness of Creating Conversational Agent Personalities Through Prompting. doi:10.48550/arXiv.2310.11182 arXiv:2310.11182 [cs]

work page Pith review arXiv doi:10.48550/arxiv.2310.11182 2023
[10]

Seungju Han, Beomsu Kim, Jin Yong Yoo, Seokjun Seo, Sangbum Kim, Enkhbayar Erdenee, and Buru Chang. 2022. Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterances. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi...

work page 2022

Show all 21 references

[11]

Georgeta Ion, Sánchez Martí , Angelina, , and Ingrid Agud Morell. 2019. Giving or receiving feedback: which is more beneficial to students’ learning? Assessment & Evaluation in Higher Education 44, 1 (Jan. 2019), 124–138. doi:10.1080/02602938.2018.1484881 Publisher: SRHE Websi...

2019
[12]

Ge Jin, Manghui Tu, Tae-Hoon Kim, Justin Heffron, and Jonathan White. 2018. Game based Cybersecurity Training for High School Students. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education. ACM, Baltimore Maryland USA, 68–73. doi:10.1145/3159450.3159591

2018
[13]

Hankyung Kim, Dong Yoon Koh, Gaeun Lee, Jung-Mi Park, and Youn-kyung Lim. 2019. Designing Personalities of Conversational Agents. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA ’19) . Association for Computing Machinery, New York...

2019
[14]

Nikola Kovacevic, Tobias Boschung, Christian Holz, Markus Gross, and Rafael Wampfler. 2024. Chatbots With Attitude: Enhancing Chatbot Interactions Through Dynamic Personality Infusion. In Proceedings of the 6th ACM Conference on Conversational User Interfaces (CUI ’24) . Assoc...

2024
[15]

Ponnurangam Kumaraguru, Steve Sheng, Alessandro Acquisti, Lorrie Faith Cranor, and Jason Hong. 2008. Lessons from a real world evaluation of anti-phishing training. In 2008 eCrime Researchers Summit . 1–12. doi:10.1109/ECRIME.2008.4696970 ISSN: 2159-1245

2008
[16]

Markel, Steven G

Julia M. Markel, Steven G. Opferman, James A. Landay, and Chris Piech. 2023. GPTeach: Interactive TA Training with GPT-based Students. In Proceedings of the Tenth ACM Conference on Learning @ Scale . ACM, Copenhagen Denmark, 226–236. doi:10.1145/3573051.3593393

2023
[17]

Federico Milana, Enrico Costanza, and Joel E Fischer. 2023. Chatbots as Advisers: the Effects of Response Variability and Reply Suggestion Buttons. In Proceedings of the 5th International Conference on Conversational User Interfaces (CUI ’23) . Association for Computing Machin...

2023
[18]

Alisha Pradhan and Amanda Lazar. 2021. Hey Google, Do You Have a Personality? Designing Personality and Personas for Conversational Agents. In CUI 2021 - 3rd Conference on Conversational User Interfaces . ACM, Bilbao (online) Spain, 1–4. doi:10.1145/3469595.3469607

2021
[19]

Gelfand, Diyi Yang, and Michael S

Omar Shaikh, Valentino Chai, Michele J. Gelfand, Diyi Yang, and Michael S. Bernstein. 2024. Rehearsal: Simulating Conflict to Teach Conflict Resolution. doi:10.48550/arXiv.2309.12309 arXiv:2309.12309 [cs]

2024 doi arXiv
[20]

Sinan Sonlu, Bennie Bendiksen, Funda Durupinar, and Uğur Güdükbay. 2024. The effects of embodiment and personality expression on learning in llm-based educational agents. arXiv preprint arXiv:2407.10993 (2024)

2024 arXiv
[21]

Can you send me a photo?

Tarja Susi, Niklas Torstensson, and Ulf Wilhelmsson. 2019. “Can you send me a photo?” – A Game-Based Approach for Increasing Young Children’s Risk Awareness to Prevent Online Sexual Grooming. In Proceedings of DiGRA 2019 Conference: Game, Play and the Emerging Ludo-Mix . doi:1...

2019 doi
[22]

Xue Wen Tan, Kenneth See, and Stanley Kok. 2024. ScamGPT-J: Inside the Scammer’s Mind, A Generative AI-Based Approach Toward Combating Messaging Scams. doi:10.48550/arXiv.2412.13528 arXiv:2412.13528 [cs]

2024 doi arXiv
[23]

Wright, Burnham , Joy J., Christopher , T

Vivian H. Wright, Burnham , Joy J., Christopher , T. Inman, , and N. Ogorchock Heather. 2009. Cyberbullying: Using Virtual Scenarios to Educate and Raise Awareness. Journal of Computing in Teacher Education 26, 1 (Sept. 2009), 35–42. doi:10.1080/10402454.2009.10784629 Publishe...

2009

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Role-Play Observations in a Trauma Course

2022. "Role-Play Observations in a Trauma Course" by Emily Horton, Claudia G. Interiano-Shiverdecker et al. https://research.library.kutztown. edu/jcps/vol15/iss2/18/

work page 2022

[2] [2]

[2305.02547] PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits

2023. [2305.02547] PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits. https://arxiv.org/abs/2305.02547

arXiv 2023

[3] [3]

Rangina Ahmad, Dominik Siemon, Ulrich Gnewuch, and Susanne Robra-Bissantz. 2022. A Framework of Personality Cues for Conversational Agents. doi:10.24251/HICSS.2022.524

work page doi:10.24251/hicss.2022.524 2022

[4] [4]

Hwang, and Weichao Yuwen

Rebekah Lee Baik, Stephanie Lee, Serena Jinchen Xie, Wang Liao, Elina H. Hwang, and Weichao Yuwen. 2025. Adapting Communication Styles in Health Chatbot using Large Language Models to Support Family Caregivers from Multicultural Backgrounds. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’25) ....

arXiv 2025

[5] [6]

Noam Ebner and Yael Efron. 2005. Using Tomorrow’s Headlines for Today’s Training: Creating Pseudo-reality in Conflict Resolution Simulation Games. Negotiation Journal 21, 3 (2005), 377–394. doi:10.1111/j.1571-9979.2005.00070.x _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1571- 9979.2005.00070.x

work page arXiv 2005

[6] [7]

Gilad Gressel, Rahul Pankajakshan, and Yisroel Mirsky. 2024. Discussion Paper: Exploiting LLMs for Scam Automation: A Looming Threat. In Proceedings of the 3rd ACM Workshop on the Security Implications of Deepfakes and Cheapfakes . ACM, Singapore Singapore, 20–24. doi:10.1145/ 3660354.3660356

work page arXiv 2024

[7] [8]

Heng Gu, Chadha Degachi, Uğur Genç, Senthil Chandrasegaran, and Himanshu Verma. 2023. On the Effectiveness of Creating Conversational Agent Personalities Through Prompting. doi:10.48550/arXiv.2310.11182 arXiv:2310.11182 [cs]

work page Pith review arXiv doi:10.48550/arxiv.2310.11182 2023

[8] [10]

Seungju Han, Beomsu Kim, Jin Yong Yoo, Seokjun Seo, Sangbum Kim, Enkhbayar Erdenee, and Buru Chang. 2022. Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterances. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi...

work page 2022