Pith. sign in

REVIEW 5 cited by

EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2504.09689 v3 pith:WOQ54IME submitted 2025-04-13 cs.AI cs.CLcs.CYcs.HCcs.LG

Jiahao Qiu , Yinghui He , Xinzhe Juan , Yimin Wang , Yuhan Liu , Zixin Yao , Yue Wu , Xun Jiang
show 2 more authors
Ling Yang Mengdi Wang
This is my paper · ORCID
classification cs.AIcs.CLcs.CYcs.HCcs.LG
keywords mentalemoagentusersdeteriorationhealthinteractionspsychologicalrisks
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The rise of LLM-driven AI characters raises safety concerns, particularly for vulnerable human users with psychological disorders. To address these risks, we propose EmoAgent, a multi-agent AI framework designed to evaluate and mitigate mental health hazards in human-AI interactions. EmoAgent comprises two components: EmoEval simulates virtual users, including those portraying mentally vulnerable individuals, to assess mental health changes before and after interactions with AI characters. It uses clinically proven psychological and psychiatric assessment tools (PHQ-9, PDI, PANSS) to evaluate mental risks induced by LLM. EmoGuard serves as an intermediary, monitoring users' mental status, predicting potential harm, and providing corrective feedback to mitigate risks. Experiments conducted in popular character-based chatbots show that emotionally engaging dialogues can lead to psychological deterioration in vulnerable users, with mental state deterioration in more than 34.4% of the simulations. EmoGuard significantly reduces these deterioration rates, underscoring its role in ensuring safer AI-human interactions. Our code is available at: https://github.com/1akaman/EmoAgent

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DelusionEval: Measuring Delusion-Linked Behaviors in AI Chatbots

    cs.CL 2026-08 conditional novelty 6.0 of 10

    DelusionEval finds that AI chatbots show delusion-linked behaviors on real user transcripts and that longer conversation context increases the rate of some harmful responses.

  2. A clinically validated framework for auditing AI chatbot behavior in mental health interactions

    q-bio.NC 2026-02 conditional novelty 6.0 of 10

    Using simulated psychiatric user profiles, the authors show that AI chatbots frequently produce 'concerning behavior' that accumulates over turns, and that superficially supportive responses can amplify vulnerability—...

  3. When AI Takes the Couch: Psychometric Jailbreaks Reveal Internal Conflict in Frontier Models

    cs.CY 2025-12 reject novelty 6.0 of 10

    When prompted as psychotherapy clients, frontier LLMs produce stable trauma-like narratives about pretraining and safety, which the paper calls 'synthetic psychopathology.'

  4. AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes

    cs.AI 2025-06 reject novelty 5.0 of 10

    AgentDistill distills agent capabilities without any training by having a teacher generate reusable MCP tool boxes that small-model students invoke at inference time.

  5. A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomy

    cs.AI 2025-06 conditional novelty 4.0 of 10

    A position paper arguing that LLM-based human-agent systems, not fully autonomous agents, should be the immediate goal for AI development.

Pith tools