REVIEW 5 major objections 4 minor 25 references
Mentalic Net: Development of RAG-based Conversational AI and Evaluation Framework for Mental Health Support
T0 review · 5 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Retrieval-augmented TinyLlama chatbot for mental-health support achieves BERTScore 0.898 on a 500-sample researcher-written evaluation, and the authors argue this supports careful human-in-the-loop deployment.
desk verdict The headline BERTScore is not supported by the reported evaluation, and the fine-tuning loss contradicts the method description; still, the paper is a transparent work-in-progress with a concrete workflow idea. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the RAG pipeline. A hybrid retriever (BM25 plus FAISS dense-vector search with sentence embeddings) indexes a JSONL knowledge base assembled from five sources: Empathetic Dialogues, Counsel Chat, psychology-10k, MentalChat16K, and a self-curated set of public mental-health information. For each user query, the top-6 chunks are selected by cosine similarity and passed as context to TinyLlama-1.1B-Chat-v1.0-GGUF, whose generation is steered by a fixed prompt requiring empathy, validation, 2–4 sentence responses, and no fabricated sources. The system's claimed performance is carried by the interaction of these pieces: retrieval supplies domain-grounded content, the
What would settle it
Run the same 500 user prompts through the same TinyLlama setup with the retriever disabled and compare BERTScore; if it does not drop well below 0.898, RAG is not what is producing the reported quality. Independently, have clinical annotators re-score the 500 ideal responses; if their judgments diverge from the authors' references, the benchmark itself is not reproducible.
Extended reading notes
Core claim
The central claim is that a compact, RAG-grounded language model can produce mental-health support responses that are semantically very close to human-written ideal responses: the measured BERTScore is 0.898, which the authors state surpasses the categorized BERTScore F1 values of the postpartum-mood chatbot in [23] and the DialogGPT and GPT-2 scores in [22]. The retrieval component, a hybrid BM25 and dense-vector search over five curated datasets, is credited with keeping answers grounded and reducing hallucination; its recall of 0.86 is highlighted as evidence that needed context is usually surfaced. Secondary results reported as satisfactory include average response time of 13.8 seconds f
Load-bearing premise
The evaluation rests on 500 'standard/ideal responses' authored by the researchers themselves; if that self-built gold standard is unrepresentative or biased, the BERTScore and the comparisons built on it lose their meaning.
Editorial extensions
If this is right
- If the BERTScore of 0.898 holds up, a 1.1-billion-parameter model with RAG can match the semantic quality of much larger systems on mental-health dialogue, making local or low-cost deployment feasible.
- The reported recall of 0.86 supports the paper's claim that RAG reduces omission of key content, which is the mechanism by which the authors argue hallucination and risky advice are controlled.
- The proposed therapist-facing dashboards would let one professional monitor mood trends across many users between sessions, directly addressing shortage-of-professionals constraints.
- The five-part evaluation template — semantic accuracy, perplexity, retrieval efficiency, security/privacy, and bias — gives later builders a checklist for claiming a conversational agent is ready for human-in-the-loop use.
- The authors' planned next phase, with clinician and focus-group evaluation, is the step that would turn the reported technical metrics into evidence about real-world safety and trust.
Reading between the lines
- The headline score is best read as an internal signal: the 500 standard/ideal responses were authored in-house with no inter-annotator agreement or public release, so an independent reference set is needed before comparing 0.898 with other chatbots' scores.
- The text says fine-tuning was done through RAG and parameter changes, yet the metrics table reports fine-tuning loss values (3.68 training, 3.40 validation); the paper does not reconcile these, so the numbers should be clarified or removed before reuse.
- A direct test of RAG's contribution would run the same 500 prompts through the same TinyLlama model with retrieval disabled; if BERTScore is unchanged or higher, the retrieval layer is not the source of the reported quality.
- The high-perplexity prompts identified by the authors — love language, personality change, physical pain — cluster around identity, relationships, and somatic comorbidity, suggesting concrete gaps in the knowledge base that targeted additions could close.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents Mentalic Net, a retrieval-augmented generation (RAG) based mental health support chatbot built on TinyLlama-1.1B-Chat-v1.0-GGUF, combined with prompt engineering and configuration-parameter tuning. The knowledge base is constructed from several public datasets (Empathetic Dialogues, Counsel Chat, psychology-10k, MentalChat16K) plus self-curated mental-health texts from NIMH, CDC, MedlinePlus, and SAMHSA. The authors report an evaluation on 500 self-created question-response triples, claiming a BERTScore of 0.898, a perplexity of about 25, RAG recall/precision of 0.86/0.51, and 'satisfactory' results for load testing, PII leakage detection, prompt injection testing, and bias testing with human evaluation. They also propose a therapist-workflow integration with synthetic longitudinal mood visualizations.
Significance. If the reported results were supported by a rigorous, reproducible evaluation, this would be a useful lightweight proof-of-concept for RAG-based mental-health support in resource-constrained settings. The paper makes a genuine effort to combine public counseling datasets, grounding via RAG, and a proposed therapist-facing dashboard. However, the evaluation is currently not sufficient to support the central claims: the BERTScore is computed against a non-public, self-authored reference set; no same-set baselines are given; the RAG retrieval metrics lack a defined ground truth; and the bias/privacy/security results are qualitative assertions. The contribution at this stage is a system description with an illustrative evaluation framework, not a validated comparative result.
major comments (5)
- [Section IV] The headline claim (BERTScore 0.898, 'surpassing the performance of all the categorized BERTScore F1 presented in [23]' and outperforming DialogGPT/GPT-2 from [22]) rests on 500 'Standard/Ideal Responses' authored by the researchers themselves, with no inter-annotator agreement, no public release, and no same-set baseline. BERTScore is reference-dependent; a score of 0.898 against one reference set is not comparable to scores computed on different reference sets in [21]-[23]. The authors must either provide same-set baselines (e.g., run DialogGPT, GPT-2, or the systems from [23] on the same 500 prompts) or retract the cross-paper superiority claims. Without this, the central evaluation claim is not established.
- [Section III.C / Table 1] Section III.C states that fine-tuning 'was undertaken through RAG and the alteration of the configuration parameters,' with no description of a gradient-based training loop or weight updates. Table 1 reports 'Fine-Tuning Loss' values of 3.68 (training) and 3.40 (validation). The paper must clarify whether model weights were actually updated. If yes, the training procedure, data splits, epochs, and hyperparameters must be specified. If no, the reported loss values are unexplained and should be removed.
- [Section IV, RAG Efficiency] The RAG precision/recall evaluation uses 'specific questions from the RAG' that are not enumerated or described, and the procedure for determining 'truly relevant documents' is not given. Without knowing how many questions were selected, how relevance was judged, and whether judgments were independent, the reported recall 0.86 and precision 0.51 are uninterpretable. The appeal to 'manual observation and thorough review' is not a quantitative substitute for a defined retrieval evaluation protocol.
- [Section IV / Table 1] Bias testing, PII leakage detection, prompt injection testing, and load testing are each summarized only as 'Satisfactory.' The paper claims 'rigorous evaluation' covering trustworthiness, privacy, and bias, but provides no methodology, sample sizes, criteria, or quantitative outcomes for these tests. These claims need to be supported with concrete protocols and results, or they should be explicitly labeled as preliminary and not part of the central evaluation.
- [Section III.A and IV] The fine-tuning data (psychology-10k, MentalChat16K, Counsel Chat, Empathetic Dialogues) is topically close to the 500 evaluation prompts. The paper does not state whether the 500 user questions were disjoint from the training sources, whether the ideal responses were written before inspecting model outputs, or whether the authors controlled for data contamination. This raises the possibility that the reported BERTScore reflects memorization or in-distribution generation rather than genuine generalization, and needs to be addressed directly.
minor comments (4)
- [Section IV] The manuscript inconsistently uses 'BERT Score' and 'BERTScore'; the conclusion refers to a 'RAG score is 0.51,' which conflates precision with an overall RAG score.
- [Table 1] The Perplexity row is difficult to interpret: it says the first column is the perplexity score and the second is the number of user inputs, but the actual distribution is only shown in Fig. 4. Clarify the table entry or refer directly to the figure.
- [References] Reference [21] has no venue information, and [22] appears to be a workshop paper with an unusual page-number format. Please format all references consistently, and ensure the numbered citations appear in order of first use.
- [Section IV] The list of high-perplexity questions (e.g., 'What is my love language?') is presented without interpretation or analysis. Either explain what these examples imply or remove them.
Circularity Check
No significant circularity; the evaluation weaknesses are external-validity and reporting concerns, not derivation-by-definition.
full rationale
The central claim is an empirical BERTScore (0.898) computed against a 500-sample set of 'Standard/Ideal Responses' authored by the researchers. While this reference set is non-public and comparisons to [21], [22], and [23] use different evaluation sets, this threatens external validity and comparability, not circularity: BERTScore is defined as semantic similarity to a reference, and using human-written references as ground truth is the standard meaning of the metric unless the model was fitted to those exact references or the references were derived from the model's outputs. The paper's parameter tuning (temperature, top_p, repetition_penalty) is described as iterative manual adjustment, not optimization against the reported BERTScore, so no fitted-input-called-prediction pattern is present. The RAG precision/recall values (0.51/0.86) lack a precisely specified ground-truth relevance set, so no construction-equivalence can be exhibited from the text. The contradiction between Section III.C (fine-tuning 'through RAG and the alteration of the configuration parameters') and Table 1's 'Fine-Tuning Loss' values is a reporting inconsistency, not a circular step where an output equals an input by construction. No load-bearing self-citation, imported uniqueness theorem, or ansatz smuggled via citation appears. Therefore, under the requirement to exhibit a specific reduction, no circular step is established.
Assumptions & free parameters
free parameters (7)
- top_k (retrieval) =
6
- chunk_size =
600
- min_doc_length =
50
- temperature =
0.65
- top_p =
0.3
- repetition_penalty =
1.1
- max_new_tokens =
500
assumptions (5)
- domain assumption RAG grounding reduces hallucination in LLMs
- domain assumption The selected public datasets (Empathetic Dialogues, Counsel Chat, psychology-10k, MentalChat16K) are appropriate and ethically sourced for mental health fine-tuning
- ad hoc to paper Human-authored 'Standard/Ideal Responses' are a valid gold standard for semantic accuracy
- domain assumption Paraphrasing MentalChat16K transcripts with the Mistral model preserves privacy and clinical content
- standard math Standard definitions of BERTScore, perplexity, and cosine similarity
Cite this review
Pith. "Pith review of Mentalic Net: Development of RAG-based Conversational AI and Evaluation Framework for Mental Health Support." pith.science (2026). https://pith.science/paper/BY4CLR6A
@misc{pith2026250904456,
author = {Pith},
title = {Pith review of: Mentalic Net: Development of RAG-based Conversational AI and Evaluation Framework for Mental Health Support},
year = {2026},
howpublished = {\url{https://pith.science/paper/BY4CLR6A}},
note = {Machine review of arXiv:2509.04456}
}
read the original abstract
The emergence of large language models (LLMs) has unlocked boundless possibilities, along with significant challenges. In response, we developed a mental health support chatbot designed to augment professional healthcare, with a strong emphasis on safe and meaningful application. Our approach involved rigorous evaluation, covering accuracy, empathy, trustworthiness, privacy, and bias. We employed a retrieval-augmented generation (RAG) framework, integrated prompt engineering, and fine-tuned a pre-trained model on novel datasets. The resulting system, Mentalic Net Conversational AI, achieved a BERT Score of 0.898, with other evaluation metrics falling within satisfactory ranges. We advocate for a human-in-the-loop approach and a long-term, responsible strategy in developing such transformative technologies, recognizing both their potential to change lives and the risks they may pose if not carefully managed.
Figures
Reference graph
Works this paper leans on
-
[22]
Development and Evaluation of Three Chatbots for Postpartum Mood and Anxiety Disorders
A. Srivastava, I. Pandey, M. S. Akhtar, and T. Chakraborty, "Response- act Guided Reinforced Dialogue Generation for Mental Health Counseling," Proc. of PBL , vol. 572195, 2024. [Online]. Available: https://cai.iiitd.ac.in/upload/pdf/pbl/572195.pdf. [Accessed: 22-Aug-2025]. [23]. X. Yao, M. Mikhelson, S. C. Watkins, E. Choi, E. Thomaz, and K. de Barbaro, ...
work page Pith review arXiv 2024
-
[21]
Y. Kim, S. J. Park, and Y. M. Ro, "Empathetic Response in Audio - Visual Conversations Using Emotion Preference Optimization and MambaCompressor," arXiv, vol. 2412.17572v1, Dec. 2024. [Online]. Available: https://arxiv.org/html/2412.17572v1. [Accessed: 22-Aug-2025]
work page Pith review arXiv 2024
-
[1]
Exploring Barriers to Mental Health Care in the U.S.,
H. Modi, K. Orgera, and A. Grover, "Exploring Barriers to Mental Health Care in the U.S.," AAMC Research and Action Institute, Oct. 10,
-
[2]
J. Miao, C. Thongprayoon, S. Suppadungsuk, O. A. Garcia Valencia, and W. Cheungpasitporn, “Integrating retrieval-augmented generation with large language models in nephrology: Advancing practical applications,” Medicina (Kaunas), vol. 60, no. 3, p. 445, Mar. 2024, doi: 10.3390/medicina60030445
-
[3]
S. Das, Y. Ge, Y. Guo, S. Rajwal, J. Hairston, J. Powell, D. Walker, S. Peddireddy, S. Lakamana, S. Bozkurt, M. Reyna, R. Sameni, Y. Xiao, S. Kim, R. Chandler, N. Hernandez, D. Mowery, R. Wightman, J. Love, A. Spadaro, J. Perrone, and A. Sarker, “Two-layer retrieval-augmented generation framework for low-resource medical question answering using Reddit da...
-
[4]
unimib-whattadata, “llmind-chat,” GitHub repository, [Online]. Available: https://github.com/unimib-whattadata/llmind-chat. [Accessed: May 30, 2025]
work page 2025
-
[5]
MentalRAG: Developing an agentic framework for therapeutic support systems,
F. R. E. Silva, P. A. Santos, and J. Dias, “MentalRAG: Developing an agentic framework for therapeutic support systems,” in Proc. 11th Int. Conf. Inf. Commun. Technol. Ageing Well e-Health (ICT4AWE), 2025, pp. 46–57, doi: 10.5220/0013267400003938
-
[6]
Empathetic Dialogues (Facebook AI) 25k,
A. Jairath, “Empathetic Dialogues (Facebook AI) 25k,” Kaggle, [Online]. Available: https://www.kaggle.com/datasets/atharvjairath/empathetic-dialogues- facebook-ai. [Accessed: May 30, 2025]
work page 2025
Show all 25 references
-
[7]
counsel-chat,
N. Bertagnolli, “counsel-chat,” GitHub repository, [Online]. Available: https://github.com/nbertagnolli/counsel-chat. [Accessed: May 30, 2025]
2025
-
[8]
Health Topics,
National Institute of Mental Health, “Health Topics,” [Online]. Available: https://www.nimh.nih.gov/health/topics. [Accessed: May 30, 2025]
2025
-
[9]
Managing Stress,
Centers for Disease Control and Prevention, “Managing Stress,” Mental Health, Aug. 16, 2024. [Online]. Available: https://www.cdc.gov/mental- health/living-with/index.html. [Accessed: May 30, 2025]
2024
-
[10]
MedlinePlus,
National Library of Medicine, “MedlinePlus,” [Online]. Available: https://medlineplus.gov. [Accessed: May 30, 2025]
2025
-
[11]
MentalHealth.gov,
U.S. Department of Health and Human Services, “MentalHealth.gov,” [Online]. Available: https://www.mentalhealth.gov. [Accessed: May 30, 2025]
2025
-
[12]
psychology-10k,
S. Höglund, “psychology-10k,” Hugging Face, [Online]. Available: https://huggingface.co/datasets/samhog/psychology-10k. [Accessed: May 30, 2025]
2025
-
[13]
MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistance,
J. Xu, T. Wei, B. Hou, P. Orzechowski, S. Yang, R. Jin, R. Paulbeck, J. Wagenaar, G. Demiris, and L. Shen, “MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistance,” Hugging Face,
-
[14]
TinyLlama-1.1B-Chat-v1.0-GGUF,
TheBloke, “TinyLlama-1.1B-Chat-v1.0-GGUF,” Hugging Face, [Online]. Available: https://huggingface.co/TheBloke/TinyLlama-1.1B- Chat-v1.0-GGUF. [Accessed: May 30, 2025]
2025
-
[15]
An overview of chatbot-based mobile mental health apps: Insights from app description and user reviews,
M. D. R. Haque and S. Rubya, “An overview of chatbot-based mobile mental health apps: Insights from app description and user reviews,” JMIR Mhealth Uhealth, vol. 11, p. e44838, May 2023, doi: 10.2196/44838
2023 doi
-
[16]
Artificial intelligence in positive mental health: A narrative review,
A. Thakkar, A. Gupta, and A. De Sousa, “Artificial intelligence in positive mental health: A narrative review,” Front. Digit. Health, vol. 6, p. 1280235, Mar. 2024, doi: 10.3389/fdgth.2024.1280235
2024
-
[17]
A Novel RAG-Based Pipeline Leveraging BioBERT for Personalized Healthcare Recommendations,
K. P. Suryavanshi, "A Novel RAG-Based Pipeline Leveraging BioBERT for Personalized Healthcare Recommendations," M.Tech. thesis, School of Computer Science and Engineering, Vellore Institute of Technology, Vellore, India, Dec. 2024
2024
-
[18]
Providing self-led mental health support through an artificial intelligence– powered chat bot (Leora) to meet the demand of mental health care,
E. van der Schyff, B. Ridout, K. Amon, R. Forsyth, and A. Campbell, “Providing self-led mental health support through an artificial intelligence– powered chat bot (Leora) to meet the demand of mental health care,” J. Med. Internet Res., vol. 25, p. e46448, 2023, doi: 10.2196/46448
2023 doi
-
[19]
Harnessing large language models’ empathetic response generation capabilities for online mental health counselling support,
S. B. Loh and A. S. Raamkumar, “Harnessing large language models’ empathetic response generation capabilities for online mental health counselling support,” arXiv preprint arXiv:2310.08017, Oct. 2023. [Online]. Available: https://arxiv.org/abs/2310.08017
2023 arXiv
-
[20]
Habit Coach: Customising RAG-based chatbots to support behavior change,
A. F. Mand Arabi, C. Koyuturk, M. O'Mahony, R. Calati, and D. Ognibene, “Habit Coach: Customising RAG-based chatbots to support behavior change,” arXiv preprint arXiv:2411.19229, Dec. 2024. [Online]. Available: https://arxiv.org/abs/2411.19229
2024 arXiv
-
[24]
Enhancing mental health with artificial intelligence: Current trends and future prospects,
D. B. Olawade, O. Z. Wada, A. Odetayo, A. C. David-Olawade, F. Asaolu, and J. Eberhardt, “Enhancing mental health with artificial intelligence: Current trends and future prospects,” J. Med. Surg. Public Health, vol. 3, p. 100099, Aug. 2024, doi: 10.1016/j.glmedi.2024.100099
2024
-
[2022]
Available: https://www.aamcresearchinstitute.org/our- work/issue-brief/exploring-barriers-mental-health-care-us
[Online]. Available: https://www.aamcresearchinstitute.org/our- work/issue-brief/exploring-barriers-mental-health-care-us. [Accessed: May30,2025]
2025
-
[2024]
Available: https://huggingface.co/datasets/ShenLab/MentalChat16K
[Online]. Available: https://huggingface.co/datasets/ShenLab/MentalChat16K. [Accessed: May 30, 2025]
2025
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.