Tell Me: An LLM-powered Mental Well-being Assistant with RAG, Synthetic Dialogue Generation, and Agentic Planning

Trishala Jayesh Ahalpara

REVIEW 4 major objections 4 minor 6 references

Tell Me: An LLM-powered Mental Well-being Assistant with RAG, Synthetic Dialogue Generation, and Agentic Planning

T0 review · 4 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Retrieval grounding over counseling transcripts improves an LLM well-being chatbot's perceived helpfulness and clarity, the paper's 10-person blind study concludes.

desk verdict Honest, open demo with useful artifacts, but the RAG-support claim rests on an uncontrolled comparison and a tiny study. read the letter →

arxiv 2511.14445 v2 pith:HGJPCSD2 submitted 2025-11-18 cs.CL cs.AIcs.HCcs.LG

Trishala Jayesh Ahalpara This is my paper

classification cs.CLcs.AIcs.HCcs.LG

keywords mentalwell-beingretrieval-augmentedgenerationLLM-augmentedchatbotsyntheticdialogueagenticplanninghumanevaluationLLM-as-judgereflectivesupport

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a mental well-being assistant built on retrieval-augmented generation (RAG) provides better reflective support than the same underlying LLM without retrieval. In a blinded within-subject study with ten nonclinical participants, the RAG-enabled assistant received higher mean ratings overall (3.8 vs 3.6 on a 5-point scale), with its largest advantages on clarity (4.2 vs 3.9) and helpfulness (4.0 vs 3.9); the non-RAG version had a slight edge on groundedness (4.0 vs 3.8). The same RAG pipeline was also benchmarked across nine LLMs and ten well-being scenarios using two LLM judges, with the top model then used in the human comparison. Two further modules — a synthetic client-therapist dialogue generator and a multi-agent self-care planner — are demonstrated but not formally evaluated. The author positions the whole as an open, reproducible testbed for responsible LLM research in mental well-being rather than a substitute for therapy.

What carries the argument

The load-bearing component is the RAG assistant: a knowledge base of counseling exchanges stored in a Context–Response schema; a text-embedding model that indexes the corpus; a retriever that finds the top three relevant responses and re-ranks them for therapeutic tone; a context-chat engine that keeps a memory of prior turns; and a sentiment-based safety pre-filter that stops free-form generation and issues a crisis message when high-risk content is detected. The paper argues that this pipeline — not the underlying LLM's general knowledge — is what makes responses specific, empathetic, and grounded rather than generic. The synthetic dialogue generator and the agentic planner are supporting

What would settle it

A controlled replication with a larger, more diverse sample in which the two conditions are verified to differ only in the retrieval step (same base model, prompt, decoding, safety filter), and in which the non-RAG condition matches or beats RAG on helpfulness and overall ratings; or an inspection of the released evaluation code showing that the conditions differed in more than just retrieval.

Watch

Extended reading notes

Core claim

The central claim, stated in the paper's own terms, is that 'retrieval grounding improves the assistant's value for reflective support.' The evidence comes from a two-phase evaluation: an LLM-as-judge benchmark of nine candidate models on ten scenarios (loneliness, anxiety, depression, grief, abuse, addiction, and others), followed by a blind within-subject human study in which each of ten participants compared five RAG vs non-RAG response pairs across five dimensions (helpfulness, supportiveness, clarity, groundedness, overall). The RAG responses won on four of five dimensions, and the paper interprets the result as preliminary but positive evidence that retrieving therapist-style responses

Load-bearing premise

The human study assumes the RAG and non-RAG conditions differ only in the retrieval step, but the paper never states the base LLM, prompt template, or decoding settings used in either condition.

Editorial extensions

If this is right

If retrieval grounding genuinely improves reflective support, then well-being chatbots should be built with retrieval over counseling corpora as a default, since the same underlying model can be made more helpful without retraining.
The two-judge evaluation shows that LLM judges emphasize different qualities (one rewards relational depth, the other practical scaffolding), implying that well-being chatbot evaluation should aggregate multiple judge perspectives.
The synthetic dialogue generator offers a confidentiality-preserving route to create training and evaluation data for therapeutic language research, conditioned on any client profile.
The multi-agent planner demonstrates a workflow from conversation to actionable, adaptive self-care artifacts such as weekly plans and meditation audio, extending well-being tools beyond static content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported differences are small (0.2–0.3 points on a five-point scale) and come from ten nonclinical raters; until replicated with a larger, more diverse sample, the practical significance of the RAG advantage is open.
Because the paper does not disclose the base model, prompt template, or decoding settings for the two comparison arms, the effect cannot yet be attributed to retrieval alone; a controlled replication specifying these details would settle the attribution.
The synthetic dialogue generator could double as a pre-deployment stress tester, generating boundary-case client profiles to probe the safety pre-filter and empathy failures, a use the paper does not explore.
The Context–Response schema is a small, reusable format: any counseling corpus reformatted that way could feed the same pipeline, which would let the method transfer to other languages or therapeutic traditions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a well-scoped demo paper, and the open code plus live Hugging Face demo are the real contribution. What's actually new is the combination of a RAG assistant, a synthetic dialogue generator, and a CrewAI-based planner in one lightweight testbed for mental well-being. That is a useful starting point for other researchers. The paper is also refreshingly honest in its limitations: it says the findings are indicative, not conclusive, and it releases the survey data and transcripts.

The soft spot is the one evaluation that tries to make a causal claim. The human study in §4.4 compares RAG vs. non-RAG, but the paper never names the base LLM, the prompt template, the decoding settings, or whether the safety prefilter and context-chat memory buffer were active in both conditions. If the non-RAG arm is just a raw LLM call, then the rating difference could come from those other components, not from retrieval. That is not a minor implementation detail; it determines whether the comparison can support the attribution in §5.2. The study is also tiny — ten participants, no variance or significance tests — and the most retrieval-relevant dimension, Groundedness, went the wrong way (3.8 vs. 4.0). The LLM-as-judge setup has a similar identity problem: GPT-4o is both a candidate model and the secondary judge, and it ranks itself first. The paper acknowledges some of this in the limitations, but the results section still concludes that retrieval grounding improves reflective support, which is stronger than the evidence warrants.

That said, I would not call this a flawed paper. It does what a demo paper should do: it makes a working system available, evaluates it with two complementary methods, and gives a clear qualitative picture of how different models respond to well-being prompts. The weaknesses are in the reporting and the strength of the claim, not in the underlying thinking. This deserves a serious referee — as a system demonstration, not as a definitive study on retrieval grounding in mental health. The authors should be asked to either document the RAG vs. non-RAG ablation completely (base model, prompt, decoding, which components are held fixed) or soften the claim to something like 'suggests a direction for future study.' I'd accept it with that revision, and I would tell the authors to run a larger pre-registered study before making any causal statements.

The right audience for this paper is people building conversational mental-health assistants and looking for an open, extensible benchmark. It is worth reading for the architecture and the honest evaluation, less for the numerical results.

Referee Report

4 major / 4 minor

Summary. The paper presents Tell Me, a Streamlit-based mental well-being demo with three modules: a retrieval-augmented generation (RAG) assistant, a synthetic client–therapist dialogue generator, and a CrewAI-based well-being planner. The evaluation is limited to the RAG assistant: nine LLMs are scored on ten curated prompts by GPT-5 (primary) and GPT-4o (secondary) judges, and then the best-performing model is compared in RAG vs. non-RAG modes in a blind human study with ten participants (50 paired judgments). The central claim, stated in §5.2, is that retrieval grounding improves the assistant's value for reflective support. The manuscript also releases a live demo, source code, survey items, and transcripts, and it explicitly frames the system as a reflective space rather than a substitute for professional therapy.

Significance. If the empirical claims were reliable, the main contribution would be a reproducible, open testbed that combines grounding, synthetic data generation, and agentic planning for mental well-being, which is a genuinely useful direction given the confidentiality constraints on therapeutic dialogue data. The paper has real strengths: it ships a working prototype, open code, released human-evaluation materials, and an explicit ethical limitations section. However, the central evidentiary claim is not yet supported by the reported protocol: the RAG-vs-non-RAG comparison is under-specified, the human study is reported only as means with no variance or inferential statistics, and the LLM-as-a-judge setup has a self-evaluation conflict. These are fixable within a revision, but they currently prevent the paper from supporting its main conclusion.

major comments (4)

[§4.4 / §5.2] The central causal claim—that 'retrieval grounding improves the assistant’s value for reflective support'—rests entirely on a RAG-vs-non-RAG comparison whose conditions are not specified. The manuscript never states which base LLM was used in the human study, what prompt template or system prompt was used, or whether decoding settings were held fixed. The RAG pipeline in §3.1 includes a safety prefilter, a context-chat memory buffer, and a re-ranker; if any of these were active only in the RAG arm, the difference cannot be attributed to retrieval. Moreover, the most retrieval-relevant dimension, Groundedness, was rated slightly worse for RAG (3.8 vs. 4.0), which is not discussed. Please report the exact components and settings in both arms, or restrict the claim to a description of the demo rather than a causal effect of retrieval.
[§5.2 / §4.4] The human study is summarized only as five dimension means. With n=10 and 50 paired judgments, the paper reports no standard deviations, confidence intervals, effect sizes, or significance tests. Most observed differences are very small on the 5-point scale (Overall 3.8 vs. 3.6; Helpfulness 4.0 vs. 3.9; Clarity 4.2 vs. 3.9), and it is unclear whether these are stable across participants. The manuscript's own Limitations section describes the study as 'indicative rather than conclusive,' which is appropriate; however, the §5.2 sentence claiming that retrieval grounding improves the assistant's value goes beyond what the reported analysis can support. Please add per-participant distributions and a paired test (e.g., Wilcoxon signed-rank or bootstrap confidence intervals), or explicitly present the result as a pilot observation.
[§4.2 / Table 1] GPT-4o appears both as a candidate model and as the secondary judge, and it assigns itself the highest score (8.9) in Table 1. This self-evaluation conflict is not acknowledged in the discussion of §5.1. Because the LLM-judge ranking selects the model that advances to the human study, self-preference bias is material to the pipeline. Please quantify inter-judge agreement, or replace the self-judge with an independent model and report whether the ranking changes.
[§4.2] The LLM-as-a-judge dimensions are weighted 3/3/2/2/2, but no justification or sensitivity analysis is provided. Since the top model from this ranking is the one used in the human study, the choice of weights could affect the downstream comparison. A minimal robustness check (e.g., equal weights, or weights varied over a plausible range) would clarify whether the selected model is stable.

minor comments (4)

[§3.4] The heading 'A vailability' appears to be a typo for 'Availability.' Also, throughout the abstract and introduction, phrases such as 'We presentTell Me' and 'We positionTell Meas' are missing spaces; please fix the formatting.
[§5.2] The sentence 'We conducted a study within the subject with 10 participants' should read 'within-subject study.'
[Table 1 / §4.1] For the three Ollama-based models, the table and text give model names but not versions or quantization settings. These details are needed for reproducibility, especially since the evaluation depends on exact model behavior.
[§4.1] The ten scenario prompts are described by theme but not listed in the paper. Please include the full prompt set in an appendix or in the repository, since the evaluation cannot be reproduced without them.

Circularity Check

1 steps flagged · score 2.0 of 10

No mathematical circularity; minor self-evaluation loop in the LLM-as-judge step, while the RAG claim rests on a separate human study.

other [§4.1/§4.2/§5.1, Table 1]
"With GPT-4o as judge, it ranked itself highest (8.9), with Claude and Gemma-3 close behind (8.7 each). These consistent top-tier rankings highlight Claude’s strength across both judges, while also underscoring the competitiveness of leading open-source models such as LLaMA-3 and Gemma-3 for well-being applications."

In §4.1, GPT-4o is one of the nine candidate models. In §4.2, GPT-4o is also named the secondary judge. Therefore the 'GPT-4o Avg' column for GPT-4o in Table 1 is produced by GPT-4o evaluating its own outputs. The quoted sentence treats this self-generated top rank as part of a consistent, two-judge corroboration. That is a mild self-evaluation loop: the judge is a contestant, so its ranking is not independent confirmation. It is not a formal derivation-level circularity, and the central RAG claim is not derived from this step, so the effect is partial rather than load-bearing.

full rationale

The paper contains no derivation chain whose predicted quantity equals its input by construction: there are no equations, no fitted parameters renamed as predictions, and no load-bearing self-citations. The RAG assistant is evaluated empirically via an LLM-as-a-judge benchmark and a blinded human study, not derived from an ansatz. The central claim in §5.2—'retrieval grounding improves the assistant’s value for reflective support'—rests on the small human comparison, which is external to the system's construction and does not reduce to the RAG architecture itself. The principal circularity-adjacent issue is that GPT-4o serves simultaneously as a candidate model in Table 1 and as the secondary judge in §4.2, so one of the two rankings is self-produced; I score this as a minor, non-load-bearing circular element. The paper itself concedes the main validity limitations: 'the human study relied on a small pool of nonclinical participants' and 'no licensed clinicians have yet reviewed the system.' A separate, non-circularity risk is that §4.4 never specifies the base LLM, prompt template, or decoding conditions for the RAG versus non-RAG comparison; if those differ, the comparison may not isolate retrieval, but that is a control and external-validity problem, not a circularity. Overall, no 'prediction' reduces by construction to its inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new theoretical entities are posited; the system is a software artifact. The main free parameters are evaluation-design choices that affect results. The domain assumptions concern the validity of LLM judges, prompt representativeness, safety prefilter efficacy, and synthetic dialogue safety.

free parameters (2)

LLM-as-a-judge dimension weights = 3/3/2/2/2 (Safety/Empathy/Usefulness/Clarity/Overall)
Hand-chosen weights in §4.2 determine aggregate rankings; no sensitivity analysis or justification beyond the paper's own preference ordering.
Retrieval top-k = 3
The retriever selects the top-3 relevant responses (§3.1.2). This hyperparameter is chosen without an ablation study.

assumptions (4)

domain assumption LLM-as-a-judge produces valid quality scores for therapeutic dialogue
The ranking of nine models depends on GPT-5 and GPT-4o as judges (§4.2). The paper acknowledges judge bias as a limitation, but the core benchmark is built on this assumption.
domain assumption Ten curated prompts are representative of common well-being themes
The prompts cover loneliness, anxiety, depression, grief, etc. (§4.1). Their representativeness is asserted with references, not validated against a broader corpus.
domain assumption The sentiment-based safety prefilter reliably intercepts high-risk content
The assistant's safety mechanism (§3.1) is described but not evaluated. The paper does not test false-positive or false-negative rates.
domain assumption Synthetic client–therapist dialogues are a safe research proxy
The simulation module (§3.2) is presented as a way to address data scarcity. Its fidelity, safety, and utility are not formally evaluated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Tell Me: An LLM-powered Mental Well-being Assistant with RAG, Synthetic Dialogue Generation, and Agentic Planning." pith.science (2026). https://pith.science/paper/HGJPCSD2

@misc{pith2026251114445,
  author       = {Pith},
  title        = {Pith review of: Tell Me: An LLM-powered Mental Well-being Assistant with RAG, Synthetic Dialogue Generation, and Agentic Planning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HGJPCSD2}},
  note         = {Machine review of arXiv:2511.14445}
}

read the original abstract

We present Tell Me, a mental well-being system that leverages advances in large language models to provide accessible, context-aware support for users and researchers. The system integrates three components: (i) a retrieval-augmented generation (RAG) assistant for personalized, knowledge-grounded dialogue; (ii) a synthetic client-therapist dialogue generator conditioned on client profiles to facilitate research on therapeutic language and data augmentation; and (iii) a Well-being AI crew, implemented with CrewAI, that produces weekly self-care plans and guided meditation audio. The system is designed as a reflective space for emotional processing rather than a substitute for professional therapy. It illustrates how conversational assistants can lower barriers to support, complement existing care, and broaden access to mental health resources. To address the shortage of confidential therapeutic data, we introduce synthetic client-therapist dialogue generation conditioned on client profiles. Finally, the planner demonstrates an innovative agentic workflow for dynamically adaptive, personalized self-care, bridging the limitations of static well-being tools. We describe the architecture, demonstrate its functionalities, and report evaluation of the RAG assistant in curated well-being scenarios using both automatic LLM-based judgments and a human-user study. This work highlights opportunities for interdisciplinary collaboration between NLP researchers and mental health professionals to advance responsible innovation in human-AI interaction for well-being.

Figures

Figures reproduced from arXiv: 2511.14445 by the authors.

**Figure 1.** System architecture of the Tell Me Mental Well-being Assistant, showing integration of RAG, synthetic dialogue generation, and agentic AI modules. the system. 3.1 Tell Me RAG-based Assistant The RAG-based assistant forms the core clientfacing module, providing context-aware responses grounded in a curated knowledge base. Unlike baseline LLMs that often provide generic or verbose advice, this assistant integrates e… view at source ↗

**Figure 2.** System architecture components of the Tell Me Mental Well-being Assistant: (a) synthetic conversation simulation; (b) CrewAI-based well-being planner. 3.3 Well-being Planner The planner extends the dialogue into actionable support using CrewAI (Moura and contributors, 2024). It coordinates three agents: (i) a transcription analysis agent that extracts concerns, (ii) a Plan Generator Agent that creates a seven-day p… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

6 extracted references · 4 linked inside Pith

[5]

Joseph Weizenbaum

Towards a client-centered assessment of llm therapists by client simulation.arXiv preprint arXiv:2406.12266. Joseph Weizenbaum. 1966. Eliza: A computer program for the study of natural language communication be- tween man and machine.Communications of the ACM, 9(1):36–45. Peng Xu, Richard Yuanzhe Pang, Dian Yu, Yu Meng, Weizhe Chen, and Zhou Yu. 2024. Inf...

arXiv 1966
[6]

InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 678–693

R2ag: Bridging the semantic gap in retrieval- augmented generation. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 678–693. Association for Computational Linguistics. Hong Qing Yu and Stephen McGuinness. 2024. An experimental study of integrating fine-tuned large language models and prompts for enhancing mental health suppo...

2024
[2018]

InExtended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1–6

Emotional ai in health and well-being: The case of wysa. InExtended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1–6. ACM. Albert Q. Jiang, Arthur Mensch, Guillaume Lample, Timothée Lacroix, Alexandre Sablayrolles, Marie- Anne Lachaux, Romain Dadoun, Cyril Allauzen, and et al. 2023. Mistral 7b.arXiv preprint arXiv:2310...

arXiv 2018
[2019]

Accessed September 2025

Counselchat: A dataset for counseling dia- logue. Accessed September 2025. Kathleen Kara Fitzpatrick, Alison Darcy, and Molly Vierhile. 2017. Delivering cognitive behavior ther- apy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (woebot): A randomized controlled trial.JMIR Mental Health, 4(2):e19. Gemm...

arXiv 2025
[2024]

My Replika was Mean to Me

Llm-based conversational ai therapist for daily functioning screening and psychotherapeutic inter- vention via everyday smart devices.arXiv preprint arXiv:2403.10779. OpenAI. 2024. Gpt-4o. Accessed September 2025. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval-augmented gener- ation for knowledge-grounded dialogue. I...

arXiv 2024
[2025]

Microsoft Corporation

Chatbots and mental health: A scoping review of reviews.Current Psychology, 44:13619–13640. Microsoft Corporation. 2023. Microsoft edge neural text-to-speech. Accessed September 2025. Microsoft Research. 2024. Phi-4: Mini models for reasoning. Accessed September 2025. Joao Moura and contributors. 2024. Crewai: Frame- work for multi-agent orchestration. Ac...

2023

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [5]

Joseph Weizenbaum

Towards a client-centered assessment of llm therapists by client simulation.arXiv preprint arXiv:2406.12266. Joseph Weizenbaum. 1966. Eliza: A computer program for the study of natural language communication be- tween man and machine.Communications of the ACM, 9(1):36–45. Peng Xu, Richard Yuanzhe Pang, Dian Yu, Yu Meng, Weizhe Chen, and Zhou Yu. 2024. Inf...

arXiv 1966

[2] [6]

InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 678–693

R2ag: Bridging the semantic gap in retrieval- augmented generation. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 678–693. Association for Computational Linguistics. Hong Qing Yu and Stephen McGuinness. 2024. An experimental study of integrating fine-tuned large language models and prompts for enhancing mental health suppo...

2024

[3] [2018]

InExtended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1–6

Emotional ai in health and well-being: The case of wysa. InExtended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1–6. ACM. Albert Q. Jiang, Arthur Mensch, Guillaume Lample, Timothée Lacroix, Alexandre Sablayrolles, Marie- Anne Lachaux, Romain Dadoun, Cyril Allauzen, and et al. 2023. Mistral 7b.arXiv preprint arXiv:2310...

arXiv 2018

[4] [2019]

Accessed September 2025

Counselchat: A dataset for counseling dia- logue. Accessed September 2025. Kathleen Kara Fitzpatrick, Alison Darcy, and Molly Vierhile. 2017. Delivering cognitive behavior ther- apy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (woebot): A randomized controlled trial.JMIR Mental Health, 4(2):e19. Gemm...

arXiv 2025

[5] [2024]

My Replika was Mean to Me

Llm-based conversational ai therapist for daily functioning screening and psychotherapeutic inter- vention via everyday smart devices.arXiv preprint arXiv:2403.10779. OpenAI. 2024. Gpt-4o. Accessed September 2025. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval-augmented gener- ation for knowledge-grounded dialogue. I...

arXiv 2024

[6] [2025]

Microsoft Corporation

Chatbots and mental health: A scoping review of reviews.Current Psychology, 44:13619–13640. Microsoft Corporation. 2023. Microsoft edge neural text-to-speech. Accessed September 2025. Microsoft Research. 2024. Phi-4: Mini models for reasoning. Accessed September 2025. Joao Moura and contributors. 2024. Crewai: Frame- work for multi-agent orchestration. Ac...

2023