REVIEW
Open Problems in Chatbots and Mental Health
Computation and Language · Submitted 7 Nov 2026 · 184 references
Pith's verdict: A well-structured, transparent agenda-setting paper that synthesizes known problems into a coherent framework; no empirical novelty but valuable for its scope and clarity.
T0 · AI review
Why Pith says this
Highlight any text to comment on it.
Referee report
The paper
The paper claims
The most critical open problems in chatbots and mental health cluster into three interdependent domains: controlled evaluation, real-world evidence, and policy.
The reading
What carries the argument
The paper's central machinery is a tripartite framework that identifies controlled evaluation, real-world evidence, and policy as the three interacting pillars needed to understand and improve chatbot mental health effects. Within each pillar, the authors distill specific recommendations (12 total) and open questions (12 total) that serve as actionable targets for stakeholders. The framework itself highlights dependencies: evaluation needs real-world validation, real-world evidence requires governed data access and independent oversight, and policy must be informed by both.
Extended reading notes
Core claim
The central claim of this paper is that the most urgent challenges in chatbot mental health cluster into three interacting domains: (1) controlled evaluation, which must move from static benchmarks to ecologically valid, multi-turn assessments linked to clinical outcomes; (2) real-world evidence, which requires studying deployed products, their interfaces and human escalation systems, and collecting longitudinal chat log data under strong privacy protections; and (3) policy, which needs evidence-based standards of care, user protections, and governed data access. The authors argue that these domains are mutually dependent—valid evaluations require real-world validation, and effective policy depends on independently produced evidence—and that no single discipline can address them alone. The paper organizes the existing literature to distill 12 concrete recommendations and 12 open questions as actionable targets for researchers, developers, clinicians, users, and policymakers.
Load-bearing premise
The paper's selection of open problems and recommendations is based on a non-systematic literature review shaped by the author team's composition and experience; if other critical problems exist outside this framing, the research agenda would be incomplete.
Editorial extensions
If this is right
- Mental health chatbot benchmarks will need to specify constructs, test reliability, and validate against real-world outcomes rather than relying on static datasets alone.
- User simulations used in safety evaluations must be validated against real user behavior using clinically relevant fidelity metrics before they can predict deployment risks.
- Policy interventions such as required crisis-response reporting, component standards, and governed data access will become testable through natural experiments across jurisdictions.
- Longitudinal chat log data, if shared under privacy-preserving governance models resembling genomic repositories, could provide new evidence on psychopathology and risk signals not available from clinical settings.
- Human review and escalation practices by chatbot companies must be studied rigorously to determine which interventions actually connect users to care.
Reading between the lines
- The paper's framework implicitly elevates independent evaluation as a bottleneck: without governed access to proprietary system components and real user logs, neither academic nor regulatory bodies can verify safety claims. This suggests that data governance agreements may be a prerequisite for progress in the other two domains.
- The emphasis on general-purpose chatbots (rather than purpose-built mental health apps) broadens the regulatory challenge: these systems are not marketed as medical devices, yet the paper documents how they can drift into therapeutic roles, creating a liability gap that existing frameworks do not close.
- A testable consequence of the paper's thesis is that in jurisdictions requiring systematic safety evaluations, measurable harms (e.g., crisis-related hospitalizations attributed to chatbot use) will be lower than in jurisdictions without such requirements, after controlling for baseline mental health outcomes.
Original abstract
General-purpose chatbots are increasingly used for emotional and mental health support. They can be useful, but issues such as sycophancy, reinforcement of delusional beliefs, and poor crisis response create risks that are difficult for researchers to study and evaluate. In this paper, we review the most critical open problems for researchers and policymakers in the use of general-purpose chatbots and their effects on mental health. First, there are tractable problems in the controlled evaluation of chatbots: establishing the construct and ecological validity of evaluations, evaluating multi-turn interactions, and connecting evaluations to improvements in chatbot behavior. For example, conventional benchmarks that primarily rely on static datasets cannot establish chatbot safety, and novel simulation methods need validation. Second, researchers must connect their studies of chatbots and mental health to real-world evidence. This includes analyzing the chatbot products that AI companies and mental health practitioners have deployed, including their user interfaces and human review and escalation practices for user welfare concerns. However, clinical analysis opportunities afforded by real chat log data sharing are complicated by significant privacy concerns. Finally, we argue that researchers should take an active role in the construction and refinement of evidence-based policy through corporate and governmental decisions such as user protections, standards of care, protected access to user chat logs, data sharing between institutions, incident reporting, and post-deployment monitoring. Such behavioral evaluations, studies of real-world outcomes, and data governance depend on each other and must be addressed in tandem. We believe a coordinated research effort would have substantial real-world impact.
The compiled record
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
The abstract, sentence by sentence
General-purpose chatbots are increasingly used for emotional and mental health support. They can be useful, but issues such as sycophancy, reinforcement of delusional beliefs, and poor crisis response create risks that are difficult for researchers to study and evaluate. In this paper, we review the most critical open problems for researchers and policymakers in the use of general-purpose chatbots and their effects on mental health. First, there are tractable problems in the controlled evaluation of chatbots: establishing the construct and ecological validity of evaluations, evaluating multi-turn interactions, and connecting evaluations to improvements in chatbot behavior. For example, conventional benchmarks that primarily rely on static datasets cannot establish chatbot safety, and novel simulation methods need validation. Second, researchers must connect their studies of chatbots and mental health to real-world evidence. This includes analyzing the chatbot products that AI companies and mental health practitioners have deployed, including their user interfaces and human review and escalation practices for user welfare concerns. However, clinical analysis opportunities afforded by real chat log data sharing are complicated by significant privacy concerns. Finally, we argue that researchers should take an active role in the construction and refinement of evidence-based policy through corporate and governmental decisions such as user protections, standards of care, protected access to user chat logs, data sharing between institutions, incident reporting, and post-deployment monitoring. Such behavioral evaluations, studies of real-world outcomes, and data governance depend on each other and must be addressed in tandem. We believe a coordinated research effort would have substantial real-world impact.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | General-purpose chatbots are increasingly used for emotional and mental health support. | background 1.00 | 0.05 | 0.02 | asserted 0.99 |
| s1 | They can be useful, but issues such as sycophancy, reinforcement of delusional beliefs, and poor crisis response create risks that are difficult for researchers to study and evaluate. | background 0.85 | 0.06 | 0.04 | asserted 0.87 |
| s2 | In this paper, we review the most critical open problems for researchers and policymakers in the use of general-purpose chatbots and their effects on mental health. | contribution 1.00 | 0.55 | 0.17 | asserted 0.96 |
| s3 | First, there are tractable problems in the controlled evaluation of chatbots: establishing the construct and ecological validity of evaluations, evaluating multi-turn interactions, and connecting evaluations to improvements in chatbot behavior. | future work 0.62 | 0.12 | 0.08 | asserted 1.00 |
| s4 | For example, conventional benchmarks that primarily rely on static datasets cannot establish chatbot safety, and novel simulation methods need validation. | gap 0.52 | 0.09 | 0.03 | asserted 0.99 |
| s5 | Second, researchers must connect their studies of chatbots and mental health to real-world evidence. | future work 0.93 | 0.06 | 0.06 | asserted 0.97 |
| s6 | This includes analyzing the chatbot products that AI companies and mental health practitioners have deployed, including their user interfaces and human review and escalation practices for user welfare concerns. | future work 0.79 | 0.07 | 0.03 | asserted 0.94 |
| s7 | However, clinical analysis opportunities afforded by real chat log data sharing are complicated by significant privacy concerns. | limitation 0.57 | 0.09 | 0.03 | asserted 0.99 |
| s8 | Finally, we argue that researchers should take an active role in the construction and refinement of evidence-based policy through corporate and governmental decisions such as user protections, standards of care, protected access to user chat logs, data sharing between institutions, incident reporting, and post-deployment monitoring. | future work 0.99 | 0.09 | 0.33 | asserted 0.66 |
| s9 | Such behavioral evaluations, studies of real-world outcomes, and data governance depend on each other and must be addressed in tandem. | future work 0.63 | 0.11 | 0.37 | asserted 1.00 |
| s10 | We believe a coordinated research effort would have substantial real-world impact. | future work 0.72 | 0.09 | 0.39 | hedged possibility 0.85 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
Figures
Figures reproduced from arXiv:2611.10997 by the authors.
References
Discussion
Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.