{"id":"8f16743f-8b2d-492b-b283-9bdcfd9f878e","arxiv_id":"2506.06616","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM-generated summary embeddings improve supervised depression severity classification, but zero-shot GPT-4o wins on binary depression and differential diagnosis.","lead":"This paper compares zero-shot GPT-4o with simple classifiers trained on raw embeddings, psychological word counts, and embeddings of GPT-4o-written mental health summaries. It finds the summaries help for severity prediction, but zero-shot GPT-4o wins on binary depression and differential diagnosis.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central comparison rests on a single 70/30 split despite abstract claiming five-fold cross-validation; reported gains lack error bars and may reflect one favorable partition.","rationale":"The central claim's truth condition is a fair, reliable comparison between feature representations. The manuscript explicitly contradicts itself on the evaluation design: the abstract and Section IV say five-fold cross-validation, while Section III-A describes a single 70/30 partition. The results are reported only as point estimates, and the differences that support the claim are small enough to be explained by one lucky split. This concern is more concrete and more directly load-bearing than label noise, because it is textually verifiable and because the same single-split issue affects every reported comparison. I agree with the reader's overall conditional verdict and with the mention of the cross-validation contradiction and missing error bars in the rationale, but the reader's stated 'weakest assumption' focuses on label reliability rather than the evaluation protocol. My stress test shifts the emphasis to the protocol: rerunning with repeated splits would settle whether the claimed improvement is real. The label-noise concern is legitimate but secondary here, since a comparative evaluation can remain valid under label noise as long as the noise affects both feature sets similarly. Thus the verdict remains CONDITIONAL, pending the concrete rerun.","tokens_in":7787,"tokens_out":3200,"duration_ms":36485,"concrete_test":"Replicate all three tasks using five-fold stratified cross-validation as the abstract states, computing per-fold accuracy and F1 for every feature set and classifier. Compare summary-embedding models against Text+LIWC models with a paired statistical test, such as McNemar's test over the test sets or a Wilcoxon signed-rank test over folds. If summary embeddings do not win on a clear majority of folds, or if the paired difference is not significant at p<0.05, the central claim is not supported by the evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central comparative claim—that LLM-derived summary embeddings improve over raw text embeddings—is evaluated under an inconsistent protocol. The abstract and Section IV describe five-fold cross-validation, but Section III-A specifies a single 70% train / 30% test split with no overlap, and all reported numbers appear to come from that one split. No standard deviations, confidence intervals, or significance tests are provided anywhere in the results. The differences supporting the claim are small in several key comparisons (e.g., binary F1 0.95 vs 0.91 for Logistic Regression; severity accuracy 58% without variance), and on a single split these gaps can easily arise from chance. If the observed ranking is an artifact of one favorable partition, the headline claim does not survive. This is load-bearing because the entire contribution is a comparison; an unsound comparison cannot support it. The label-noise concern is secondary: even with noisy labels, the comparison can be fair if the noise affects all feature sets similarly, but the missing repeated evaluation leaves the comparison unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a feature-engineering approach for social-media-based depression detection: an LLM (GPT-4o) is prompted to generate a one-to-two-sentence mental-health summary of each post, the summary is embedded with a sentence encoder (all-mpnet-base-v2), and these summary embeddings are fed to traditional classifiers (logistic regression, SVM, random forest). The approach is compared against classifiers using raw text embeddings plus LIWC features and against zero-shot GPT-4o classification on three tasks: binary depression detection, depression severity grading, and differential diagnosis among depression, anxiety, and PTSD. The paper reports that summary embeddings improve over raw-text features in binary and severity tasks, while zero-shot LLM classification is strongest in binary detection but weak in severity.","tokens_in":7969,"tokens_out":4872,"duration_ms":50459,"significance":"If the reported results are reliable, the paper makes a useful, easily reproducible contribution: it shows that using an LLM as a semantic interpreter before embedding can preserve or sharpen task-relevant signal, and it provides an interpretability path via the intermediate summaries. Strengths include the use of five public datasets, a simple and transparent pipeline, standard classifiers, and a clear research question. The central comparative claim, however, rests on an evaluation protocol that is internally inconsistent (abstract states five-fold cross-validation, methods describe a single 70/30 split) and lacks any measure of variance or statistical testing. These issues are load-bearing because the entire contribution is a comparison; without a sound evaluation, the headline result is not yet established.","major_comments":[{"comment":"The abstract and Section IV state that five-fold cross-validation was used, but Section III-A specifies that \"the corresponding dataset is partitioned into a 70% training set and a 30% test set with no overlap,\" and all reported numbers appear to come from that single split. No standard deviations, confidence intervals, or significance tests are provided anywhere in the results. This matters because the observed improvements are small in several key comparisons (e.g., Table II: LR summary accuracy 0.93 vs. text+LIWC 0.89; RF summary 0.92 vs. 0.91), and on a single split these gaps could easily arise from a favorable partition. Please clarify the exact protocol and provide repeated evaluation (e.g., 5-fold CV with per-fold metrics, or bootstrap confidence intervals) and appropriate significance tests.","section":"Section III-A / Abstract"},{"comment":"The abstract claims that supervised models trained on LLM summary embeddings \"often achieve more accurate and consistent performance, particularly for multi-class and ordinal classification tasks,\" but Section IV-C reports that the zero-shot LLM classifier achieved the highest accuracy (65%) on the multi-class differential diagnosis task, outperforming the best summary-embedding model (logistic regression, 59%). This directly contradicts the abstract's implication for multi-class tasks. The conclusion also states that summary embeddings lead to \"more accurate and consistent performance\" specifically for severity, not for multi-class diagnosis. Please align the abstract with the actual results, or provide additional evidence to support the multi-class superiority claim.","section":"Section IV-C / Abstract"},{"comment":"The claim that the logistic-regression summary-embedding model performed \"significantly outperforming the zero-shot LLM-based approach\" on depression severity is unsupported: no accuracy figure for the zero-shot LLM is reported, no statistical test is conducted, and the word \"significantly\" is used without evidence. Please report exact accuracy and per-class F1 for all models on this task, and perform a paired statistical test (e.g., McNemar's test for model predictions on the same test set) to substantiate the claim.","section":"Section IV-B"},{"comment":"The construction of the binary depression label from HelaDepDet is unclear. Table I lists HelaDepDet labels as \"Depression, Minimum, Mild, Moderate, Severe,\" but the binary classification task requires a positive/negative distinction. Are all severity levels (minimum, mild, moderate, severe) mapped to the positive \"depression\" class, or only some? If the severity levels are ordinal, the mapping to binary labels needs explicit description, because it affects the class balance and the resulting test set, and therefore the comparison in Table II.","section":"Section III-A / Table I"}],"minor_comments":[{"comment":"There is a typo: \"accurante\" should be \"accurate.\"","section":"Section V"},{"comment":"Reference [21] is cited as \"Qian el al.\"; the correct spelling is \"Qian et al.\"","section":"Section II-B"},{"comment":"The description of [12] contains \"V ADER\" with an extra space; it should read \"VADER.\"","section":"Section II-A"},{"comment":"The phrase \"with no overlap between the two\" is ambiguous: it likely means the training and test sets are disjoint, but it could also be read as no posts being shared. Please clarify.","section":"Section III-A"},{"comment":"The confusion matrix in Figure 1 is described as showing \"a slight tendency to classify non-depressive posts as depressive,\" but the normalized percentages are not given in the text, making the magnitude of this bias difficult to assess.","section":"Section IV-A / Figure 1"},{"comment":"The violin plots in Figures 2a and 2b are not fully described; the axes (predicted severity level vs. true severity level) are implied but not labeled clearly in the text, which makes it hard to interpret the distributions.","section":"Section IV-B / Figure 2"},{"comment":"The paper reports overall accuracy for differential diagnosis but does not report class-wise accuracy or balanced accuracy; given that the classes may be imbalanced (e.g., RMHD has only 658 posts), macro-averaged metrics or balanced accuracy would be more informative.","section":"Section IV-C"},{"comment":"The paper does not discuss the computational cost or API usage of GPT-4o for generating summaries across all datasets, which is relevant for reproducibility and practical deployment.","section":"Section III-D"},{"comment":"The paper assumes that the labels in the five public datasets are reliable ground truth. Some of these datasets are based on self-disclosure or crowdsourced annotations; a discussion of label noise and its potential impact on feature rankings would strengthen the evaluation.","section":"Section III-A"}],"recommendation":"major_revision","confidential_remarks":"The central idea is promising and the paper is generally well organized, but the inconsistency between the abstract's five-fold cross-validation claim and the single 70/30 split described in the methods is a serious issue that must be resolved before the paper can be considered for publication. The differential-diagnosis result contradicts the abstract's main claim, which suggests the abstract was written too broadly. I would recommend requiring the authors to either fix the protocol and re-run experiments with repeated cross-validation and confidence intervals, or explicitly revise the abstract and conclusions to match the actual single-split results. The lack of any statistical testing is also a recurring problem throughout the results section. If the authors can address these points, the paper could become a solid empirical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a straightforward empirical paper with a useful idea and an evaluation that doesn't yet support its headline. The core recipe—prompt GPT-4o to write a two-sentence mental-health summary of a social media post, embed that summary with all-mpnet-base-v2, and train a logistic regression on the embedding—is simple, practical, and genuinely interpretable in a clinical sense: the summaries themselves could be shown to a clinician. The comparison across five public datasets and three tasks is a legitimate contribution, and the finding that summary embeddings help most on severity prediction is plausible and worth testing further. The related-work coverage is solid and the writing is clear.\n\nThe problems are in the evaluation, and they are load-bearing. The abstract says five-fold cross-validation, but Section III-A describes a single 70/30 split, and the reported numbers appear to come from that one split. No confidence intervals, standard deviations, or significance tests appear anywhere. The differences in the key comparisons are sometimes small (binary F1 0.95 vs 0.91; severity accuracy 58% with no variance), so on one favorable partition the ranking could easily be noise. On top of that, the differential diagnosis results actually put zero-shot LLM on top (65% vs 59%), which undercuts the abstract's claim that summary embeddings are particularly strong for multi-class tasks. The paper also uses a 10th/90th percentile text length filter without sensitivity analysis, and no code or data release is mentioned, so reproducing the exact numbers is not possible.\n\nNone of this means the underlying idea is wrong. The summary-embedding approach is sensible, and the paper honestly reports the differential-diagnosis numbers. But the central empirical claim—that LLM summaries improve over raw text embeddings—is not verified until the evaluation is repeated over multiple splits and reported with error bars and the abstract and methods are aligned.\n\nWho this is for: researchers working on applied clinical NLP who want a cheap, transparent feature-engineering trick. It is a conference-workshop-level paper in its current form, but it deserves a serious referee if the authors can fix the evaluation. I would not cite the specific numbers, though I might cite the idea once it is properly validated.","headline":"Useful recipe, shaky evaluation: the summary-embedding idea is worth trying, but the headline comparison rests on a single split and the abstract overclaims.","tokens_in":8460,"tokens_out":1984,"would_cite":false,"duration_ms":23454,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that embeddings of LLM-written mental-health summaries beat raw-text embeddings for supervised depression classification.","keywords":["depression detection","large language models","summary embeddings","social media text","zero-shot classification","mental health NLP","sentence embeddings","interpretability"],"falsifier":"Re-annotate a random sample of posts from MHB, CAMS, HelaDepDet, RMHD, and DepressionEmo with clinician-reviewed labels, then rerun the same 70/30 comparison; if classifiers trained on raw text embeddings match or exceed those trained on LLM summaries under the cleaner labels, the paper's central claim is false.","tokens_in":7604,"feed_emoji":"🧠","tokens_out":5329,"duration_ms":51238,"temperature":0.7,"pith_summary":"The paper asks whether it is better to feed a classifier raw social-media text, or to first have a large language model write a one-to-two-sentence clinical interpretation of the user's mental state and then embed that summary. The authors' central claim is that these LLM-derived summary embeddings outperform conventional text embeddings and psycholinguistic features for supervised depression classification. The evidence comes from five public social-media datasets and three tasks: binary depression detection, four-level severity grading, and differential diagnosis among depression, anxiety, and PTSD. The paper also finds that a zero-shot GPT-4o classifier is very accurate at the binary task but falls behind supervised classifiers on fine-grained severity prediction. The practical payoff is a hybrid recipe: use the LLM as an interpreter that produces readable summaries, and a lightweight classifier as the decision maker.","feed_headline":"LLM summaries beat raw text for depression detection","feed_subtitle":"Prompting GPT-4o to interpret a post before embedding it lifts supervised classifiers on severity and diagnosis tasks.","key_machinery":"The central mechanism is the prompted summary-embedding pipeline: a mental-health-oriented prompt forces GPT-4o to paraphrase the post into a short clinical interpretation, and the sentence encoder embeds that summary instead of the raw text. This does two jobs at once: it filters irrelevant surface detail and emphasizes affective and cognitive cues, and it produces a human-readable intermediate artifact that a clinician could inspect. The same sentence encoder is used for both raw text and summaries, so the only difference is the LLM-mediated interpretation inserted before embedding.","core_discovery":"The paper claims that prompting an LLM to summarize a social media post's mental state before embedding it yields feature vectors that carry more task-relevant signal than embeddings of the raw post. Concretely, it proposes a pipeline: GPT-4o receives a mental-health-expert prompt and writes one or two sentences describing emotional tone, cognitive state, and possible signs of conditions; the sentence encoder all-mpnet-base-v2 turns that summary into a 768-dimensional vector; and logistic regression, SVM, or random forest is trained on those vectors. Across the three tasks, summary-embedding classifiers generally match or beat classifiers trained on raw text embeddings plus LIWC features, and the logistic-regression variant reaches the best severity accuracy (58%). The zero-shot LLM is the best binary classifier (96% accuracy) but the weakest severity predictor, showing that the LLM's raw judgment and its summarization ability are different capabilities.","pith_inferences":["Editorial: because GPT-4o is itself a strong classifier, part of the summary-embedding gain may be inherited from the model's prior knowledge; testing with a smaller open LLM or with summaries generated to exclude diagnostic words would isolate the summarization effect.","Editorial: the binary control set consists of AITA moral-dilemma posts from a different community than the depression datasets, so the 96% binary result may partly reflect domain differences rather than depression detection; a matched control sample would sharpen the estimate.","Editorial: severity predictions could likely be improved further by ordinal-aware losses that treat the four severity labels as ordered, or by training on summary embeddings plus the zero-shot LLM's predicted probabilities.","Editorial: the summary-embedding approach should transfer to other mental-health labels such as suicidal ideation or eating disorders, and to languages beyond English, since the summary is natural language that can be inspected for faithfulness."],"forward_implications":["LLM-derived summary embeddings make lightweight linear models competitive with, and on severity tasks better than, direct LLM classification, so expensive LLM calls can be spent once per post rather than per prediction.","Because summary embeddings beat raw text plus LIWC features while using the same classifiers, the representation, not the classifier, drives most of the gain.","The readable summary gives a built-in explanation channel: a clinician can read why the model produced its feature vector, which directly addresses interpretability concerns.","The zero-shot LLM's strength on binary detection but weakness on ordinal tasks suggests that LLMs should be assigned to screening, while supervised summary-embedding models handle severity triage."],"supporting_citations":[{"why":"This dataset supplies MHB forum posts labeled for depression, anxiety, and PTSD, used in the binary and differential-diagnosis tasks.","marker":"[23]"},{"why":"This dataset supplies CAMS posts labeled for depression, used to build the binary training set.","marker":"[24]"},{"why":"This dataset supplies HelaDepDet posts with four ordinal depression-severity labels, which define the severity task and contribute binary examples.","marker":"[25]"},{"why":"This dataset supplies RMHD posts labeled depression and anxiety, used in binary and differential-diagnosis tasks.","marker":"[26]"},{"why":"This dataset supplies DepressionEmo posts labeled for depression, used in the binary task.","marker":"[27]"},{"why":"This dataset supplies AITA posts as the non-depression control class in the binary task.","marker":"[28]"},{"why":"This model provides the all-mpnet-base-v2 sentence encoder that produces both the raw text and the LLM-summary embeddings.","marker":"[29]"},{"why":"This library computes the accuracy, precision, recall, and F1 metrics reported for all classifiers.","marker":"[30]"}],"fun_headline_variants":["LLM summaries boost depression severity classifiers","Summaries, not raw text, improve mental health AI","Prompt LLMs to interpret, then classify for depression","Depression detection: LLM summaries outperform raw text","GPT-4o summaries aid supervised depression models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the public datasets' labels are trustworthy enough that a better feature representation is what makes summary embeddings win, and that GPT-4o's summaries preserve rather than distort the cues needed for each task.","fun_headline_variants_meta":{"raw":{"variants":["LLM summaries boost depression severity classifiers","Summaries, not raw text, improve mental health AI","Prompt LLMs to interpret, then classify for depression","Depression detection: LLM summaries outperform raw text","GPT-4o summaries aid supervised depression models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1278,"prompt_tokens":925,"completion_tokens":353,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":280}},"tokens_in":541,"tokens_out":353,"duration_ms":3883,"temperature":1.0,"reasoning_tokens":280,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:52:33.255369+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of posts from MHB, CAMS, HelaDepDet, RMHD, and DepressionEmo with clinician-reviewed labels, then rerun the same 70/30 comparison; if classifiers trained on raw text embeddings match or exceed those trained on LLM summaries under the cleaner labels, the paper's central claim is false.","supporting_citations":[{"cited_title":"Leveraging mental health forums for user-level depression detection on social media,","cited_arxiv_id":null,"evidence_quote":"This dataset supplies MHB forum posts labeled for depression, anxiety, and PTSD, used in the binary and differential-diagnosis tasks."},{"cited_title":"CAMS: an annotated corpus for causal analysis of mental health issues in social media posts,","cited_arxiv_id":null,"evidence_quote":"This dataset supplies CAMS posts labeled for depression, used to build the binary training set."},{"cited_title":"HelaDepDet: a novel multi-class classification model for detecting the severity of human depression,","cited_arxiv_id":null,"evidence_quote":"This dataset supplies HelaDepDet posts with four ordinal depression-severity labels, which define the severity task and contribute binary examples."},{"cited_title":"From posts to knowledge: annotating a pandemic-Era Reddit dataset to navigate mental health narratives,","cited_arxiv_id":null,"evidence_quote":"This dataset supplies RMHD posts labeled depression and anxiety, used in binary and differential-diagnosis tasks."},{"cited_title":"De- pressionEmo: a novel dataset for multilabel classification of depression emotions,","cited_arxiv_id":null,"evidence_quote":"This dataset supplies DepressionEmo posts labeled for depression, used in the binary task."},{"cited_title":"‘Am I the Bad One’? predicting the moral judgement of the crowd using pre–trained language models,","cited_arxiv_id":null,"evidence_quote":"This dataset supplies AITA posts as the non-depression control class in the binary task."},{"cited_title":"MPNet: masked and permuted pre-training for language understanding,","cited_arxiv_id":null,"evidence_quote":"This model provides the all-mpnet-base-v2 sentence encoder that produces both the raw text and the LLM-summary embeddings."},{"cited_title":"Scikit-learn: machine learning in Python,","cited_arxiv_id":null,"evidence_quote":"This library computes the accuracy, precision, recall, and F1 metrics reported for all classifiers."}],"review_version":1}