REVIEW 3 major objections 4 minor 33 references
KokoroChat: A Japanese Psychological Counseling Dialogue Dataset Collected via Role-Playing by Trained Counselors
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A role-play dataset of 6,589 Japanese counseling dialogues, built by trained counselors, improves both response generation and automated evaluation.
desk verdict KokoroChat is a genuinely useful new dataset and the paper deserves a serious referee; the main caveat is that the client-feedback scores are role-play labels whose validity as counseling-quality measures is never tested. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The role-playing collection protocol combined with the 20-item client feedback instrument. Trained professional and trainee counselors, 480 in total and all with 10 hours of online text-counseling training, alternate as counselor and client in hour-long text sessions; after each session the client-role player scores the counselor on 10 overall-impression items and 10 counseling-skill items, with three screening flags that can zero or halve the total. This design supplies both the dataset's authenticity claim and the supervision signal used for fine-tuning response generation on high-versus-low score partitions and for training the score-prediction evaluator.
What would settle it
Run an inter-rater reliability study where several trained counselors each play the client role for the same recorded counselor session and independently complete the 20-item feedback form; if the scores disagree substantially across raters, the feedback signal that powers both fine-tuning and score prediction is not a stable measure of counseling quality.
Extended reading notes
Core claim
The paper's central claim is that a large-scale role-play protocol staffed by trained counselors can produce counseling dialogues that carry detailed per-session client feedback, and that this data improves both generation and evaluation of counseling responses. Each KokoroChat dialogue is a roughly one-hour text exchange, with client-role players rating counselor-role players on 20 items covering overall impression and professional skills on a 0-to-5 scale. The paper reports that fine-tuning Llama-3.1-Swallow-8B on the high-scoring subset (Kokoro-High) outperforms fine-tunes on low-scoring and full datasets in human pairwise evaluation, and that a score-prediction model trained on the feedback exceeds GPT-4o in accuracy (35.35 vs 30.92) and mean absolute error (0.828 vs 1.015). The claim is that the dataset's size, dialogue length (averaging 91.2 utterances), topic coverage, and itemized scores make it a stronger resource for Japanese counseling dialogue systems than existing LLM-augmented or smaller human-collected datasets.
Load-bearing premise
The dataset's value rests on the premise that role-playing by trained counselors produces dialogues and client feedback scores that faithfully represent real counseling interactions, a premise the paper asserts but does not validate against real counseling data or an external benchmark.
Editorial extensions
If this is right
- Fine-tuning on KokoroChat, especially the high-scoring subset, improves the quality of counseling responses generated by an 8-billion-parameter open-source Japanese LLM.
- A dialogue evaluation model trained on KokoroChat's client feedback predicts the 20 per-dimension scores with higher accuracy and lower error than GPT-4o in zero-shot mode.
- Client word count shows the strongest positive correlation with feedback scores (rho = 0.42), suggesting that encouraging client expression matters more for positive evaluation than counselor verbosity.
- Counselor response time correlates negatively with feedback scores, implying that faster replies may contribute to a better counseling experience.
- The dataset's size and per-item scores enable future work such as dialogue-act annotation, cross-lingual comparison, and fine-grained analysis of questioning strategies.
Reading between the lines
- If the score-prediction model proves reliable, it could serve as an automated coaching tool that gives novice counselors itemized feedback without requiring a human supervisor.
- The authenticity claim rests on an unvalidated premise; comparing role-play transcripts with de-identified real counseling sessions would be a direct test of whether the dialogues faithfully represent real client-counselor interactions.
- The strong correlation among feedback items such as gaining new insights, feeling hopeful, and perceiving value suggests that a model optimized only for empathy may miss the insight-oriented components that clients associate with a valuable session.
- Translating or adapting KokoroChat into other languages would allow direct comparison with English and Chinese counseling datasets and would test whether the score distributions and item correlations hold across cultures.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces KokoroChat, a Japanese psychological counseling dialogue dataset of 6,589 role-played dialogues between trained professional and trainee counselors, each accompanied by a 20-item client feedback score (0-100). The authors claim KokoroChat is the largest human-collected counseling dialogue dataset to date. They fine-tune Llama-3.1-Swallow on the dataset and report that it improves both response generation quality (automatic metrics and human pairwise evaluation) and automatic dialogue evaluation (score prediction accuracy, soft accuracy, and MAE compared with Llama-3.1 and GPT-4o). The dataset is released publicly.
Significance. If the underlying data and scores are valid, KokoroChat fills a clear gap as a large Japanese-language counseling resource with long, realistic-length dialogues and fine-grained evaluation labels. The paper's strengths include the scale of human collection (6,589 dialogues, 480 participants), the expert-designed 20-item feedback instrument, the use of trained counselors as role players, a public release, and careful documentation of fine-tuning and inference settings (QLoRA, seeds, deterministic decoding). The human evaluation by five professional counselors is a useful independent check. However, the significance of the automatic-evaluation contribution rests entirely on the validity of the role-play client feedback scores, which are not externally validated, and the response-generation experiments are evaluated on a narrow high-score subset of the data.
major comments (3)
- [3.3 and 5.2] The automatic-evaluation claim in Section 5.2 is load-bearing and depends on the 20-item client feedback scores being valid measures of counseling quality, but the paper provides no external validation of these scores. The raters are trained counselors acting as simulated clients, the scores are immediately shared with the counselor-role player, and the platform administrator monitors the process; these conditions may introduce social-desirability or peer-evaluation bias. The paper asserts authenticity in Sections 1 and 3.1 but never compares the proxy-client scores against real-client perceptions, independent expert ratings, or an established counseling-quality instrument. The paper's own Ethical Considerations state that the dialogues are not real counseling sessions. Consequently, Table 5 demonstrates only that a fine-tuned model can predict the role-play clients' scores better than GPT-4o; it does not establish better evaluation of counseling quality. The authors should either add an external validation study or substantially soften the abstract's claim about 'automatic evaluation of counseling dialogues.'
- [5.1.1] The response-generation test set consists exclusively of 118 dialogues with client feedback scores of 99 or 100, while all training variants use dialogues with scores at or below 98. This restricts evaluation to the extreme upper tail of the score distribution and does not reflect performance on typical dialogues (the mean score is 63.58, median 64.00). Moreover, the Kokoro-High vs Kokoro-Low comparison is confounded by the mismatch between the score distribution of each training set and the test set: neither training set contains dialogues from the 99-100 score range. To support the general claim that fine-tuning on KokoroChat improves response quality, the evaluation should include a test set sampled from the full score range, or the conclusions should be limited to the high-score regime.
- [5.1.3 and Figure 5] The human pairwise evaluation reports only raw win/lose/tie percentages without significance tests, bootstrap confidence intervals, or inter-annotator agreement. With 100 responses per model and only 10 dialogues, margins of a few percentage points (e.g., in the first comparison in Figure 5, the win and lose rates differ by only a few percentage points) are within sampling noise, yet the text concludes that 'even when using only the lower-scoring portions of KokoroChat, it still enhances open-source LLMs.' The authors should report statistical significance or confidence intervals for each pairwise comparison, or refrain from strong conclusions where the margin is not significant.
minor comments (4)
- [2.2] The dataset name 'C ACTUS' has an extra space and should be written as 'CACTUS' (also in Table 1).
- [5.1.1 (footnote 3)] The justification for the score threshold of 70 ('to ensure balanced data segmentation') is not substantiated; the authors should report the number of dialogues and utterances in each partition to show balance.
- [Table 5 and Appendix D] Standard deviations are reported only for the fine-tuned model (averaged over five seeds), while the baseline Llama-3.1 and GPT-4o results appear to be single runs, making the stability comparison asymmetric; reporting multiple runs for the baselines or clearly stating the limitation would be fairer.
- [4.1] The dialogue-topic distribution is produced by GPT-4o-mini without any human validation or agreement measure; the diversity claims based on Figure 2 should be treated as preliminary unless a sample of the topic labels is verified.
Circularity Check
No significant circularity; the claims rest on held-out supervised learning and independent human evaluation, with only a minor non-load-bearing self-citation.
full rationale
The paper's central claims are empirical rather than derivational. The score-prediction experiment (§5.2) fine-tunes a model on an 8:1:1 split and evaluates on a held-out test set, so the reported ACC/ACCsoft/MAE values measure generalization to unseen dialogues, not a quantity that was fitted into the model. The response-generation experiment (§5.1) uses held-out high-scoring dialogues as a test set and is evaluated both by standard metrics and by pairwise human judgments from five professional counselors, which are external to the client-feedback labels used to partition the training data. The related-work citation to Inaba et al. (2024) involves overlapping authors, but it is background context about GPT-4's counseling ability, not a load-bearing premise for KokoroChat's dataset or experimental conclusions. The main substantive concern is the unvalidated role-play client feedback as ground truth for counseling quality, which the paper itself acknowledges as a limitation regarding direct comparability with existing research; that is a validity or external-benchmark issue, not circular reasoning. No equation or parameter is defined in terms of the result it predicts, and no conclusion reduces by construction to its own inputs.
Assumptions & free parameters
free parameters (3)
- Score threshold 70 for high/low data split =
70
- Test set criterion: client feedback scores 99 or 100 =
>=99
- Data filtering rules =
>=30 utterances, >=30 minutes, not all items rated 3
assumptions (3)
- domain assumption Role-play by trained counselors produces dialogues representative of real counseling interactions
- domain assumption Client feedback from role-playing clients is a reliable and valid measure of counseling quality
- standard math Standard automatic metrics (BLEU, ROUGE) are meaningful proxies for counseling response quality
Cite this review
Pith. "Pith review of KokoroChat: A Japanese Psychological Counseling Dialogue Dataset Collected via Role-Playing by Trained Counselors." pith.science (2026). https://pith.science/paper/T5JLWTZN
@misc{pith2026250601357,
author = {Pith},
title = {Pith review of: KokoroChat: A Japanese Psychological Counseling Dialogue Dataset Collected via Role-Playing by Trained Counselors},
year = {2026},
howpublished = {\url{https://pith.science/paper/T5JLWTZN}},
note = {Machine review of arXiv:2506.01357}
}
read the original abstract
Generating psychological counseling responses with language models relies heavily on high-quality datasets. Crowdsourced data collection methods require strict worker training, and data from real-world counseling environments may raise privacy and ethical concerns. While recent studies have explored using large language models (LLMs) to augment psychological counseling dialogue datasets, the resulting data often suffers from limited diversity and authenticity. To address these limitations, this study adopts a role-playing approach where trained counselors simulate counselor-client interactions, ensuring high-quality dialogues while mitigating privacy risks. Using this method, we construct KokoroChat, a Japanese psychological counseling dialogue dataset comprising 6,589 long-form dialogues, each accompanied by comprehensive client feedback. Experimental results demonstrate that fine-tuning open-source LLMs with KokoroChat improves both the quality of generated counseling responses and the automatic evaluation of counseling dialogues. The KokoroChat dataset is available at https://github.com/UEC-InabaLab/KokoroChat.
Figures
Figures from the paper (11 more)
Reference graph
Works this paper leans on
-
[1]
Asma Ben Abacha, Wen wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen, Fei Xia, and Thomas Lin. 2024. https://arxiv.org/abs/arXiv:2412.19260 Medec: A benchmark for medical error detection and correction in clinical notes
arXiv 2024
-
[2]
Tim Althoff, Kevin Clark, and Jure Leskovec. 2016. https://doi.org/10.1162/tacl_a_00111 Large-scale analysis of counseling conversations: An application of natural language processing to mental health . Transactions of the Association for Computational Linguistics, 4:463--476
-
[3]
Yirong Chen, Xiaofen Xing, Jingkai Lin, Huimin Zheng, Zhenyu Wang, Qi Liu, and Xiangmin Xu. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.83 S oul C hat: Improving LLM s' empathy, listening, and comfort abilities through fine-tuning with multi-turn empathy conversations . In Findings of the Association for Computational Linguistics: EMNLP 2023, pa...
-
[4]
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. https://arxiv.org/abs/arXiv:2305.14314 Qlora: Efficient finetuning of quantized llms
arXiv 2023
-
[5]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, and Angela Fan et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783
arXiv 2024
-
[6]
Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. Continual pre-training for cross-lingual llm adaptation: Enhancing japanese language capabilities. In Proceedings of the First Conference on Language Modeling, COLM, University of Pennsylvania, USA
2024
-
[7]
Michimasa Inaba, Mariko Ukiyo, and Keiko Takamizo. 2024. https://arxiv.org/abs/2402.12738 Can large language models be used to provide psychological counselling? an analysis of gpt-4-generated responses using role-play dialogues . In The 14th International Workshop on Spoken Dialogue Systems Technology
arXiv 2024
-
[8]
Suyeon Lee, Sunghwan Kim, Minju Kim, Dongjin Kang, Dongil Yang, Harim Kim, Minseok Kang, Dayi Jung, Min Hee Kim, Seungbeen Lee, Kyong-Mee Chung, Youngjae Yu, Dongha Lee, and Jinyoung Yeo. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.832 Cactus: Towards psychological counseling conversations using cognitive behavioral theory . In Findings of the A...
Show all 33 references
-
[9]
Anqi Li, Lizhi Ma, Yaling Mei, Hongliang He, Shuai Zhang, Huachuan Qiu, and Zhenzhong Lan. 2023. https://doi.org/10.18653/v1/2023.acl-long.577 Understanding client reactions in online mental health counseling . In Proceedings of the 61st Annual Meeting of the Association for C...
2023 doi
-
[10]
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. https://doi.org/10.18653/v1/N16-1014 A diversity-promoting objective function for neural conversation models . In Proceedings of the 2016 Conference of the North A merican Chapter of the Association f...
2016 doi
-
[11]
Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics
2004
-
[12]
Liu, Donghao Li, He Cao, Tianhe Ren, Zeyi Liao, and Jiamin Wu
June M. Liu, Donghao Li, He Cao, Tianhe Ren, Zeyi Liao, and Jiamin Wu. 2023. https://arxiv.org/abs/arXiv:2309.15461 Chatcounselor: A large language models for mental health support
2023 arXiv
-
[13]
Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. 2021. https://doi.org/10.18653/v1/2021.acl-long.269 Towards emotional support dialog systems . In Proceedings of the 59th Annual Meeting of the Association for Computational ...
2021 doi
-
[14]
Naoaki Okazaki, Kakeru Hattori, Hirai Shota, Hiroki Iida, Masanari Ohi, Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Rio Yokota, and Sakae Mizuki. 2024. Building a large japanese web corpus for large language models. In Proceedings of the First Conference on Language Modeling,...
2024
-
[15]
OpenAI. 2023. https://arxiv.org/abs/arXiv:2303.08774 Gpt-4 technical report
2023 arXiv
-
[16]
OpenAI. 2024. https://arxiv.org/abs/arXiv:2410.21276 Gpt-4o system card
2024 arXiv
-
[17]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 Bleu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311...
2002
-
[18]
Huachuan Qiu, Hongliang He, Shuai Zhang, Anqi Li, and Zhenzhong Lan. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.34 SMILE : Single-turn to multi-turn inclusive language expansion via C hat GPT for mental health support . In Findings of the Association for Computation...
2024 doi
-
[19]
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. https://doi.org/10.18653/v1/P19-1534 Towards empathetic open-domain conversation models: A new benchmark and dataset . In Proceedings of the 57th Annual Meeting of the Association for Computational Lingu...
2019 doi
-
[20]
SAMHSA. 2015. https://www.samhsa.gov/data/sites/default/files/NSDUH-FRR1-2014/NSDUH-FRR1-2014.pdf Behavioral health trends in the united states: Results from the 2014 national survey on drug use and health . Annual report, Substance Abuse and Mental Health Services Administration
2015
-
[21]
Ashish Sharma, Adam Miner, David Atkins, and Tim Althoff. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.425 A computational approach to understanding empathy expressed in text-based mental health support . In Proceedings of the 2020 Conference on Empirical Methods in Natur...
2020 doi
-
[22]
WHO. 2022. https://www.who.int/publications/i/item/9789240049338 World mental health report: Transforming mental health for all
2022
-
[23]
Zixiu Wu, Simone Balloccu, Vivek Kumar, Rim Helaoui, Ehud Reiter, Diego Reforgiato Recupero, and Daniele Riboni. 2022. https://doi.org/10.1109/ICASSP43922.2022.9746035 Anno-mi: A dataset of expert-annotated counselling dialogues . In ICASSP 2022 - 2022 IEEE International Confe...
2022
-
[24]
Mengxi Xiao, Qianqian Xie, Ziyan Kuang, Zhicheng Liu, Kailai Yang, Min Peng, Weiguang Han, and Jimin Huang. 2024. https://doi.org/10.18653/v1/2024.acl-long.93 H eal M e: Harnessing cognitive reframing in large language models for psychotherapy . In Proceedings of the 62nd Annu...
2024 doi
-
[25]
Jing Ye, Lu Xiang, Yaping Zhang, and Chengqing Zong. 2025. https://arxiv.org/abs/2412.08389 Sweetiechat: A strategy-enhanced role-playing framework for diverse scenarios handling emotional support agent . In Proceedings of the 31th International Conference on Computational Lin...
2025 arXiv
-
[26]
Chenhao Zhang, Renhao Li, Minghuan Tan, Min Yang, Jingwei Zhu, Di Yang, Jiahao Zhao, Guancheng Ye, Chengming Li, and Xiping Hu. 2024 a . https://doi.org/10.18653/v1/2024.findings-acl.830 CP sy C oun: A report-based multi-turn dialogue reconstruction and evaluation framework fo...
2024 doi
-
[27]
Tenggan Zhang, Xinjie Zhang, Jinming Zhao, Li Zhou, and Qin Jin. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.723 ESC o T : Towards interpretable emotional support dialogue systems . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisti...
2024 doi
-
[28]
Haiquan Zhao, Lingyu Li, Shisong Chen, Shuqi Kong, Jiaan Wang, Kexin Huang, Tianle Gu, Yixu Wang, Jian Wang, Liang Dandan, Zhixu Li, Yan Teng, Yanghua Xiao, and Yingchun Wang. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.883 ESC -eval: Evaluating emotion support conversat...
2024 doi
-
[29]
Chujie Zheng, Yong Liu, Wei Chen, Yongcai Leng, and Minlie Huang. 2021. https://doi.org/10.18653/v1/2021.findings-acl.72 C o MAE : A multi-factor hierarchical framework for empathetic response generation . In Findings of the Association for Computational Linguistics: ACL-IJCNL...
2021 doi
-
[30]
Chujie Zheng, Sahand Sabour, Jiaxin Wen, Zheng Zhang, and Minlie Huang. 2023. https://doi.org/10.18653/v1/2023.findings-acl.99 A ug ESC : Dialogue augmentation with large language models for emotional support conversation . In Findings of the Association for Computational Ling...
2023 doi
-
[31]
Zhonghua Zheng, Lizi Liao, Yang Deng, Libo Qin, and Liqiang Nie. 2024. https://doi.org/10.18653/v1/2024.acl-long.611 Self-chats from large language models make small emotional support chatbot better . In Proceedings of the 62nd Annual Meeting of the Association for Computation...
2024 doi
-
[32]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[33]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.