{"id":"ee6ca309-5282-4a8c-83ce-184900ed1120","arxiv_id":"2412.17651","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A system combining LLM-extracted features with traditional ML classifiers claims 90% accuracy for multi-label anxiety and depression detection from chatbot dialogues, but the evaluation has leakage and missing baselines.","lead":"This paper presents a multi-label system that classifies chatbot dialogues as indicating anxiety, depression, both, or neither, using GPT-4o-mini to extract psychological and linguistic features that feed random forest, decision tree, and naive Bayes classifiers. The authors report 90% accuracy on their private dataset, but the evaluation setup has serious methodological weaknesses that make the performance claim unreliable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 90% accuracy is not evidence of dialogue-based detection because the GADS/YGDS questionnaires used to create labels are embedded in the session text fed to the LLM feature extractor (Sec. 3.1, Listing 1); the model can read the label source directly.","rationale":"The reader's REJECT verdict is appropriate, but the most load-bearing concern is more fundamental than session non-independence. The paper's intended contribution is a system that detects anxiety and depression from natural dialogue using LLM-extracted features and ML classifiers. For that claim to hold, the classifier's accuracy must reflect generalization from conversational language to mental-health labels, not from reading the diagnostic instruments. Section 3.1 says the chatbot embeds GADS/YGDS questionnaires into the conversation every three months and uses the answers as labels. Section 4.2 and Listing 1 confirm that the LLM feature extractor runs on the complete session text, so the feature vector contains information about the exact questionnaire responses that define the target. Any reasonably flexible classifier, especially the random forest on 56 features, can exploit these markers and report high accuracy without learning anything about spontaneous language. The reuse of the same 10% subset for feature selection and hyperparameter tuning, and the lack of a user-disjoint held-out split, compound the problem, but even a clean split would not fix the input/label overlap. The central claimed accuracy therefore does not support the paper's stated contribution. I would keep the REJECT verdict, with the rationale strengthened by this direct leakage argument.","tokens_in":13755,"tokens_out":5087,"duration_ms":49737,"concrete_test":"Strip every GADS/YGDS questionnaire turn (both the chatbot's questions and the user's answers) from the session text before running the Listing 1 prompt, then re-run feature selection, hyperparameter tuning, and RF classification on a user-disjoint evaluation (e.g., GroupKFold by user ID or a temporal split on session date). If the held-out accuracy on the cleaned dialogues does not remain near 90% (e.g., falls below 80% or toward the majority-class baseline), the headline result is attributable to label leakage from the embedded questionnaires.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the system detects anxiety and depression from free-form dialogue is undermined by a structural input-label overlap. Section 3.1 states that every three months the chatbot embeds the GADS and YGDS questionnaires into the conversation flow and uses those responses as the user's labels. Section 4.2 confirms that sessions are complete conversations, and Listing 1 shows the LLM feature extractor receives the full session text. Consequently, the engineered features—such as 'anguish', 'negative emotion', and 'health issues'—are computed from text that includes the very questionnaire items and answers that define the target labels. A classifier can achieve high accuracy by recognizing those embedded responses rather than by learning general linguistic markers of mental health. This explains why no user-disjoint or temporal split is reported: session-level splits cannot remove the leakage, because the leakage is in the input features themselves. Even if the statistical issues raised by the reader (non-independence of sessions, label noise) were fixed, the headline result would still not support the claim of detecting mental-health conditions from natural dialogue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-label classifier for anxiety and depression from free-form chatbot dialogues. It uses GPT-4o-mini to extract 14 psychological and linguistic features from each user session via a prompt, expands each feature with sliding-window statistics, selects features on a 10% cold-start subset, and trains Naive Bayes, decision tree, and random forest classifiers. Reported results in Table 4 claim up to 89.43% accuracy and 84.72% exact match for the random forest in scenario 1, which the authors state improves on prior literature. An explainability dashboard is also presented. The main technical claim is that an LLM-feature-based ML pipeline can jointly detect anxiety and depression from natural conversation.","tokens_in":14002,"tokens_out":3898,"duration_ms":39163,"significance":"If the central claim were supported, the paper would offer a practical combination of LLM feature extraction with interpretable multi-label classifiers and an explainability dashboard, addressing a real need for scalable mental-health screening. The use of a real chatbot dataset rather than social media, and the effort to provide clinician-facing explanations, are welcome. However, the reported 90% accuracy is not currently credible as evidence of dialogue-based detection, because the labels are derived from questionnaires embedded in the same session text that is fed to the LLM feature extractor. The evaluation also lacks a proper held-out test set, user-level splits, baselines, and error bars. The paper's contribution is therefore potentially useful but its headline result is unsupported.","major_comments":[{"comment":"There is a structural label-leakage problem: the GADS and YGDS questionnaires are embedded in the conversation flow every three months, and the responses to those questionnaires are used as the target labels. The LLM feature extractor in Listing 1 receives the complete session text, which therefore includes the questionnaire items and the user's answers. Features such as \"anguish\" and \"health issues\" can be read directly from the user's questionnaire responses rather than inferred from free-form dialogue. The high accuracy in Table 4 is thus fully compatible with the model recognizing the label source inside the input, and it does not support the claim that the system detects anxiety and depression from free-form conversation. The authors need to re-run feature extraction after removing all questionnaire-related text from the session input, or otherwise demonstrate that the questionnaire content is not accessible to the feature extractor.","section":"Sec. 3.1, Sec. 4.2, Listing 1"},{"comment":"The evaluation does not use a proper held-out test set. Feature selection is performed on 10% of the dataset (Sec. 4.4), and hyperparameters are optimized with GridSearchCV on the same 10% subset (Sec. 4.5). The paper does not state that the final models are evaluated on a disjoint test set, so the results in Table 4 likely reflect training-data performance or at least tuning leakage. A nested cross-validation or a fixed train/validation/test split with feature selection and hyperparameter tuning performed entirely within the training folds is required before the accuracy claims can be accepted.","section":"Sec. 4.4, Sec. 4.5, Table 4"},{"comment":"The dataset consists of 2,186 sessions from only 32 users, with an average of 68 sessions per user, and labels are re-assigned every three months from the embedded questionnaires. The experiments do not group sessions by user or by time period. Random session-level splits will almost certainly place sessions from the same user in both training and test folds, and temporally adjacent sessions from the same user have strongly correlated labels and features. This can substantially inflate accuracy. The authors should use user-disjoint and temporally split evaluation, e.g., leaving out entire users or entire time periods, to estimate performance on independent individuals.","section":"Sec. 4.1, Sec. 4.2"},{"comment":"The paper claims the results \"improve those in the prior literature,\" but it provides no quantitative baselines on the same dataset or protocol. There is no comparison against simple text baselines such as bag-of-words with logistic regression, or against transformer or LLM classifiers, and no confidence intervals or significance tests are reported. Without these, the 90% accuracy figure cannot be interpreted. The explainability dashboard is also presented without any user study or quantitative evaluation of explanation quality.","section":"Sec. 4.5, Sec. 4.6, Table 4"}],"minor_comments":[{"comment":"Hamming Loss is defined as 1 - Accuracy, but the standard Hamming Loss is the fraction of incorrectly predicted labels across all instances and labels. This nonstandard definition should be corrected or renamed.","section":"Sec. 3.4, Eq. (6)"},{"comment":"The table reports both macro and micro precision, recall, and F-measure but does not define the microaveraging formulas; please add definitions or cite a standard reference.","section":"Table 4"},{"comment":"The prompt template contains typos such as \"emp hasi zed_ term s\" and an inconsistent feature name; these should be corrected.","section":"Listing 5"},{"comment":"The statement that intentional concealment is reduced because the tests are embedded in the dialogues is an empirical claim with no supporting evidence; it should be softened or justified.","section":"Sec. 3.1"},{"comment":"The explainability module is described but not evaluated; a brief qualitative assessment or sample explanations would help verify that the generated text is coherent and faithful to the features.","section":"Sec. 4.6"},{"comment":"The keyword list includes \"eXplainable Artificial Intelligence\" and \"Large Language Models\" but the paper uses lowercase abbreviations; please standardize the notation throughout.","section":"Keywords"}],"recommendation":"major_revision","confidential_remarks":"The structural label leakage described in the major comments is the most serious issue, and it is not merely a statistical artifact: the feature extractor can directly read the questionnaire responses that define the labels. That said, the issue is in principle fixable by filtering questionnaire content out of the session text and re-running the experiments, so I do not view it as an irreparable flaw. The remaining evaluation problems (lack of a held-out set, user-level splits, baselines, and error bars) need to be addressed in a revised submission. I would encourage the editor to send the paper back for major revision rather than reject it outright, provided the authors can demonstrate that the 90% accuracy persists after removing the leakage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nI don't buy the headline result, and the stress-test note pinpoints the real reason. The GADS/YGDS questionnaires that generate the labels are embedded in the conversation every three months, and the LLM feature extractor receives the full session text (Listing 1). So the features like 'anguish' and 'health issues' are computed from the same text that determines the labels. The model can be reading the answer key. That makes the 90% accuracy uninformative about detecting mental health from natural dialogue, no matter how clean the train/test split is.\n\nWhat's actually new is the pipeline: LLM-as-feature-extractor feeding traditional ML classifiers in a multi-label anxiety-plus-depression setup, plus an explainability dashboard. That's a legitimate extension of earlier work, and the dashboard is a sensible concession to clinical needs. The literature review is competent and positions the work reasonably against BERT-based and LLM-based approaches.\n\nThe reader's statistical concerns are real too: feature selection and hyperparameter tuning share the same 10% cold-start subset, there's no separate test set, sessions from 32 users are not independent, and there are no baselines or error bars. Those would be enough to reject the accuracy claim on their own. But the leakage is the deeper problem, and it persists even after fixing the split.\n\nI'd also note the authors are transparent about embedding the questionnaires—they think it reduces concealment—so this is a design flaw, not a hidden one. To salvage the idea, they'd need to hold the label-generating text out of the feature extraction, or explicitly test whether the model is keying on the questionnaire passages.\n\nThis paper is useful as a cautionary example for anyone building LLM feature pipelines with clinical instruments, but I wouldn't cite it as evidence. I'd desk-reject it, explaining the leakage and the evaluation issues, rather than send it to referees. If they redo the study with separated label acquisition and a proper evaluation, the pipeline idea might be worth a second look.","headline":"The 90% accuracy claim is invalid: the questionnaire responses used as labels are in the same session text the LLM feature extractor sees, so the model is partly reading the label source.","tokens_in":14495,"tokens_out":3278,"would_cite":false,"duration_ms":41369,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-label system detects anxiety and depression from chatbot dialogues with 90% accuracy, pairing LLM feature extraction with random forest classification.","keywords":["anxiety detection","depression detection","multi-label classification","large language models","explainable AI","chatbot dialogues","mental health screening","natural language processing"],"falsifier":"Run a leave-one-user-out cross-validation on the same dataset: if accuracy on users never seen in training drops substantially below the reported 89.43%, the performance is partly user memorization rather than general detection. A second check is to re-label a sample of sessions with the same questionnaires at different times and measure how often the label changes for the same user's sessions; high instability would show the model is learning noisy targets.","tokens_in":13562,"feed_emoji":"🧠","tokens_out":4897,"duration_ms":43569,"temperature":0.7,"pith_summary":"This paper claims that anxiety and depression can be detected together, as a multi-label problem, from free-form conversations with a chatbot. The proposed system uses a large language model to score fourteen emotional and linguistic cues in each session, then feeds those scores plus their statistics over past sessions into standard machine-learning classifiers. On 2,186 real sessions from the Celia chatbot, the best classifier, Random Forest, reaches 89.43% accuracy with a 10.57% Hamming loss, outperforming the prior literature the authors compare against. If true, this would make early screening scalable and less reliant on rigid questionnaires, because the assessment happens naturally inside an ongoing dialogue and the model explains its decision in plain language.","feed_headline":"Chatbot chats expose anxiety and depression together at 90% accuracy","feed_subtitle":"A language model scores emotional cues from free conversation; a random forest then flags both conditions at once.","key_machinery":"The machinery is a two-stage pipeline. First, a prompt-engineered LLM (gpt-4o-mini with temperature 0) reads each full dialogue session and outputs scores from 0.0 to 1.0 for 14 features covering the user's cognitive state (insecurity, loneliness, negative/positive emotion, sadness, anguish, health issues) and the discourse (catastrophic terms, emphasized terms, repeated concepts, interjections, negative adverbs, negative terms, polarity). Each feature is expanded with its average and three quartiles over a sliding window of the last 30 sessions, yielding 56 features; a tree-based impurity selector keeps about 39% of them. The selected features train Naive Bayes, Decision Tree, and Random Forest classifiers under a multi-class transformation of the two clinical labels, and the best model's prediction is turned into a natural-language explanation by a second prompt template that also fills a dashboard for caregivers and users.","core_discovery":"The central claim is that coupling an LLM feature-engineering stage with a traditional multi-label classifier yields accurate joint detection of anxiety and depression from real chatbot dialogues, with explainability preserved. The authors treat the task as four mutually exclusive categories—no condition, depression only, anxiety only, and both—using the multi-class transformation strategy on binary labels derived from Spanish versions of the Goldberg Anxiety and Depression Scales and the Yesavage Geriatric Depression Scale administered inside the chatbot every three months. They report that Random Forest attains 89.43% accuracy, 93.50% macro precision, 84.18% macro recall, and 84.72% exact-match ratio in the full-session scenario, and that these results beat the closest related multi-label baseline, which used activity-tracker data and binary-relevance classifiers.","pith_inferences":["The paper does not report user-level cross-validation; the natural next test is leave-one-user-out evaluation, because with only 32 users, standard cross-validation may mix sessions from the same user in both training and test sets.","The quarterly re-labeling creates a temporal dependency: the same user's sessions share a label for up to three months, so adjacent sessions are not independent, and a stricter evaluation would group sessions by labeling period.","The fourteen features are generic enough that the same prompt-template approach could be tested on other disorders or other languages, but that would require retraining and revalidating labels.","A practical extension the authors do not report is calibration: knowing when the confidence score is unreliable could matter more than average accuracy in a pre-screening tool that decides who gets referred."],"forward_implications":["If the 90%-level accuracy holds, a health system could deploy the pipeline as a pre-screening layer that flags users for formal assessment without making them fill out anxiety and depression questionnaires separately.","Because the LLM is used only for feature extraction, the final classification is done by an interpretable ML model, so the system avoids relying on the LLM as a black-box judge and still gives a confidence score.","The sliding-window statistics mean the model uses the user's history, not just one isolated chat, so a single atypical session should not by itself change the assessment.","The explainability dashboard could let clinicians see which of the fourteen cues drove the prediction, supporting accountability under AI-regulation expectations that users receive clear explanations."],"supporting_citations":[{"why":"Supplies the multi-class transformation strategy used to turn the two binary clinical labels into four categories.","marker":"[35]"},{"why":"The closest multi-label baseline; the paper compares its accuracy against this activity-tracker study.","marker":"[2]"},{"why":"Shows a similar design of extracting BERT features to feed traditional ML models, which this work extends with LLM features and explainability.","marker":"[27]"},{"why":"A related stacking proposal for anxiety, depression, and comorbidity from social media that the multi-label setting builds on.","marker":"[31]"},{"why":"Documents poor performance of a non-fine-tuned LLM for depression classification, motivating the LLM-feature-extraction approach.","marker":"[1]"},{"why":"Tests LLMs directly for mental-health prediction; the authors contrast their method by adding explainability and a multi-label setting.","marker":"[15]"}],"fun_headline_variants":["Chatbot talk exposes anxiety and depression together at 90%","One model, two conditions: anxiety and depression from chat","AI flags both anxiety and depression from natural dialogue","Explainable AI detects anxiety and depression in chatbot chats","Jointly detecting anxiety and depression from dialogue with 90% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the 2,186 sessions from 32 users are independent and that the labels, assigned every three months from questionnaires embedded in the chatbot, are correct and stable; if users' repeated sessions or label noise are not handled, the reported accuracy overstates how the system performs on independent individuals.","fun_headline_variants_meta":{"raw":{"variants":["Chatbot talk exposes anxiety and depression together at 90%","One model, two conditions: anxiety and depression from chat","AI flags both anxiety and depression from natural dialogue","Explainable AI detects anxiety and depression in chatbot chats","Jointly detecting anxiety and depression from dialogue with 90% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000751,"raw_usage":{"total_tokens":3324,"prompt_tokens":907,"completion_tokens":2417,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":2336}},"tokens_in":523,"tokens_out":2417,"duration_ms":18400,"temperature":1.0,"reasoning_tokens":2336,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:17:39.488549+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a leave-one-user-out cross-validation on the same dataset: if accuracy on users never seen in training drops substantially below the reported 89.43%, the performance is partly user memorization rather than general detection. A second check is to re-label a sample of sessions with the same questionnaires at different times and measure how often the label changes for the same user's sessions; high instability would show the model is learning noisy targets.","supporting_citations":[{"cited_title":"Rivolli, J","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-class transformation strategy used to turn the two binary clinical labels into four categories."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The closest multi-label baseline; the paper compares its accuracy against this activity-tracker study."},{"cited_title":"Sarkar, A","cited_arxiv_id":null,"evidence_quote":"Shows a similar design of extracting BERT features to feed traditional ML models, which this work extends with LLM features and explainability."}],"review_version":1}