{"id":"09d84f9d-a527-4973-90fe-1f2dfe623a85","arxiv_id":"2303.15727","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Zero-shot ChatGPT classified social media posts for stress, depression and suicidality above majority-class baselines, with F1 of 0.73, 0.86 and 0.37 respectively.","lead":"ChatGPT, asked to sort Reddit posts into mental health categories with no training examples, scored F1 0.73 for stress, 0.86 for depression and 0.37 for a five-way suicidality task, each above a majority-class baseline. The numbers are an early look at whether a general chat model can stand in for purpose-built mental health classifiers.","discovery_kind":"new_application","skeptic_critique":{"model":"selfhosted/deepseek-v4.1-flash","headline":"The depression F1 of 0.86 is the paper's strongest number, but the 548 negative-class posts have no stated subreddit provenance; if they come from unrelated communities, the score measures topic separation rather than depression.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern I would raise: the depression negative class has no stated subreddit provenance, so the 0.86 may measure topic separation rather than mood. I agree with that diagnosis and with the CONDITIONAL verdict. The paper's claim is deliberately weak ('indicates a potential use'), and the evidence does support that weak claim: ChatGPT beats the majority baseline on all three tasks, and the confusion matrices are shown. The paper is not rejected on its central claim. It is conditional because the depression number, which is the strongest of the three, is the one most exposed to a construct-validity confound that the paper neither controls nor acknowledges. The stress result is better protected because Dreaddit [11] draws both classes from the same subreddits, and the suicidality result is weak but above baseline. Secondary reporting issues the reader already noted (the 0.72/0.73 prose-versus-table discrepancy, the unstated count of dropped posts, the single API call with no temperature or seed, the post-hoc 'None' bucket) are real but do not move the verdict on their own. I also note that the paper's own cited in-domain models (BERT at 0.81 on stress in [10], CNN at 0.65 on suicidality in [14]) beat or approach ChatGPT, which limits the practical significance of the zero-shot numbers; the paper acknowledges this, so I do not re-raise it as a new finding. The concrete test above would settle whether the depression concern lands without requiring new experiments beyond retrieving the source subreddits and fitting a topic-only baseline. If it lands, the verdict should stay CONDITIONAL with the depression result flagged; if it does not, the paper's central claim stands as stated.","tokens_in":7427,"tokens_out":7503,"duration_ms":195585,"concrete_test":"Retrieve the 548 non-depression posts from [12], record their source subreddits, and fit a topic-only baseline (TF-IDF plus logistic regression on subreddit identity, or a bag-of-words classifier with no mental-health features) on the same 1841 posts. If that baseline reaches approximately 0.86 weighted F1, the depression result is explained by topic separation and the concern lands; if it falls near the 0.60 majority baseline, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that zero-shot ChatGPT beats a majority-class baseline on three mental health tasks, which 'indicates a potential use' (abstract:s6). The depression result (0.86 vs 0.60) carries the largest margin and is the paper's best evidence. But §II-A-2 describes the negative class only as 'Reddit posts from the non-depression category in the dataset [12]' and gives no subreddit provenance for those 548 posts. If they come from unrelated communities (e.g., r/AskReddit, r/relationships), a classifier can separate the two classes by topic and style alone, and 0.86 reflects domain separation rather than depressive content. The paper offers no lexical or topic baseline to rule this out; the only comparison is a majority-class baseline, which any topic-sensitive model clears. The stress dataset is less exposed because Dreaddit [11] draws stress and non-stress posts from the same subreddits, but no such control is stated for the depression data. This is not among the limitations the Discussion lists (prompt setting, single backend, dataset size, annotation dependence), so it is unresolved rather than acknowledged. If the concern lands, the depression number is not a measure of depression detection, and the 'potential use' claim rests on the weaker stress and suicidality results.","agreement_with_reader":"agree"},"referee_report":{"model":"selfhosted/deepseek-v4.1-flash","summary":"The manuscript reports a zero-shot evaluation of ChatGPT (gpt-3.5-turbo backend) on three text-based mental health classification tasks drawn from public social media datasets: binary stress detection (Dreaddit-derived, 715 test posts), binary depression detection (1293 depression vs. 548 non-depression Reddit posts), and five-way suicidality detection (500 users). Each post is submitted once to the ChatCompletions API with a single fixed prompt naming the candidate classes, and the first reply is compared against the dataset annotation. The reported weighted F1 scores are 0.73, 0.86 and 0.37 against majority-class baselines of 0.35, 0.60 and 0.19, with balanced accuracies of 0.73, 0.85 and 0.33. The authors conclude that zero-shot classification accuracy 'indicates a potential use of language models for mental health classification tasks' (abstract:s6) and discuss fine-tuning, prompt variation, and annotation quality as future work.","tokens_in":7664,"tokens_out":4039,"duration_ms":91707,"significance":"If the numbers hold, the result is a useful and cheap data point: an off-the-shelf chat model with a one-line prompt, no training data and no task-specific features, clears a majority-class baseline on all three tasks and comes within roughly eight F1 points of a fine-tuned BERT on stress detection (0.73 vs. 0.81 in [10]) despite never seeing in-domain labels. The study is easy to replicate in principle, uses public datasets, and reports balanced accuracy alongside F1, which is the right instinct under the severe class imbalance in all three datasets. The authors are also candid about several limitations (single backend, single prompt, small datasets, annotation dependence). The contribution is a benchmark snapshot rather than a methodological advance, and its value depends on the reported margins being attributable to the construct of interest rather than to dataset artifacts.","major_comments":[{"comment":"The depression result (F1 0.86 vs. 0.60 baseline) carries the largest margin and is the paper's strongest evidence, but the negative class is described only as 'the Reddit posts from the non-depression category in the dataset [12]', with no subreddit provenance for the 548 posts. If those posts come from communities unrelated to depression (e.g., general-interest or relationship subreddits), a model can separate the two classes largely on topic and style, and 0.86 measures domain separation rather than depressive content. The only comparison offered is a majority-class baseline, which any topic-sensitive method clears. Please (i) report the subreddit distribution of both classes, and (ii) add at least one trivial lexical baseline (e.g., TF-IDF plus logistic regression) evaluated on the same posts. If the lexical baseline also reaches roughly 0.86, the headline number should be reinterpreted and the abstract adjusted accordingly. This concern is not among the limitations listed in §IV, so it is currently unresolved rather than acknowledged.","section":"§II-A-2 and Table I","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The evaluation protocol does not report the number of posts actually scored per task or per class, yet §III states that 'some posts in the dataset could not be evaluated due to errors in reading these posts from the dataset or invoking the API'. Without those counts the reader cannot tell whether the reported F1 is computed over the full test set or a filtered subset, and selective API failures could bias the result in either direction. Table I should report N evaluated and the per-class support for each task, together with the number of API and parsing failures. In the same place, please state the exact model snapshot (e.g., gpt-3.5-turbo-0301), the temperature and any other sampling parameters, and the date of collection; 'gpt-3.5-turbo' is a moving alias and the numbers are not reproducible without the snapshot.","section":"§II-B and Table I","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The prompt instructs the model to 'Only reply with answers from one of the classes', but the manuscript reports that the model sometimes did not, and these replies are assigned to a 'None' class that is defined by the model's own output rather than by the annotation scheme. The manuscript does not say how None predictions enter the weighted F1 of 0.37: if they are scored as errors the number is conservative, if they are excluded from the average the number is inflated. Please state the scoring rule explicitly, and also state the response-parsing rule (case normalization, substring versus exact matching, handling of multi-class or explanatory replies), since that rule is an unconstrained analyst choice that directly determines the label assigned to each post.","section":"§II-B and Figure 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"A majority-class baseline is too weak to support the inference in abstract:s6 that the model 'can infer a user's mental health state to an extent'. Any method sensitive to topic, sentiment or post length clears that bar, so the comparison establishes only that ChatGPT is not degenerate. A single additional baseline — bag-of-words or TF-IDF with logistic regression, trained on the in-domain training portions where they exist — would make the claim informative at modest cost, and would directly address the depression concern raised above. Where the published in-domain numbers already cited in §IV ([10] BERT 0.81, [14] CNN 0.65) are used as reference points, please restate in the same sentence that those models were trained on in-domain labels while ChatGPT was not, so the comparison is not read as like-for-like.","section":"§III and Table I","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The abstract and Table I report an F1 of 0.73 for stress detection, while the first sentence of §III says 'We obtained an F1 score of 0.72 for the stress detection task'. Please reconcile the two values.","section":"Abstract vs. §III","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Table I would be substantially more useful with columns for N evaluated, class distribution, and per-class precision/recall, particularly for the five-way suicidality task where the weighted F1 of 0.37 and balanced accuracy of 0.33 cannot be interpreted without the class support.","section":"Table I","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The two example posts are quoted verbatim from Reddit, including specific personal circumstances. Verbatim quotation of mental-health-related social media text carries a re-identification risk via search, and the manuscript does not state whether the data use was reviewed or whether posts were altered. Paraphrasing the examples, or stating the data-handling and ethics position explicitly, would be appropriate for a journal in this field.","section":"§II-A-1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Please give the exact prompt string as issued, including any system message and the verbatim class names, and note whether the class names matched the dataset label strings ('stressed'/'non-stressed' versus 'stress'/'non-stress'). Small wording differences in the label set are a plausible source of the out-of-label replies discussed in Figure 3.","section":"§II-B","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Three of the datasets are cited only as GitHub repositories. Please add access dates and, where the repository corresponds to a published paper or dataset descriptor, cite that as well, so that readers can verify the annotation protocol and class definitions.","section":"References [10], [12], [13]","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The discussion of future work would benefit from one or two sentences on deployment risk, in particular the asymmetry of errors in the suicidality task, where the confusion matrix in Figure 3 shows most inputs collapsing into Suicidal Ideation. A statement that the present results do not support clinical use would guard against over-reading abstract:s6.","section":"§IV","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Two typographical corrections: 'defacto' should read 'de facto', and 'MultMedQA' should read 'MultiMedQA'.","section":"§I","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as a short empirical note rather than a full research article, and its shelf life is tied to a specific API snapshot that has already been superseded. The stress and suicidality results are modest but honestly reported; the depression result is the one that carries the paper and is the one most exposed to a dataset-provenance confound. I would want the subreddit provenance and a lexical baseline before the 0.86 is presented as a depression-detection result. The authors' own limitation list is unusually candid, which is in the paper's favour."},"author_rebuttal":null,"desk_editor":{"model":"selfhosted/deepseek-v4.1-flash","letter":"Short version: this is a four-page empirical note reporting zero-shot gpt-3.5-turbo F1 on three public mental health datasets. The numbers are what they say they are, with one caveat about the depression set that the paper does not address. What is new is narrow and honest: nobody had published these particular zero-shot numbers, with this prompt, on these three datasets. No new method, no new data, no new annotation. The framing matches the contribution, and 'indicates a potential use' [abstract:s6] is about right for a single-prompt, single-call evaluation. Credit where due: the confusion matrices are shown, the comparison to fine-tuned BERT at 0.81 in [10] and the CNN at 0.65 in [14] is fair and not spun, and the Discussion does list four real limitations, namely prompt setting, single backend, small datasets, and annotation dependence. The Pith record says no limitations were found; that is wrong, they are in Section IV. Soft spots, in order of weight. The depression negative class is the big one. Section II-A-2 says only 'Reddit posts from the non-depression category in the dataset [12]' and gives no subreddit provenance for those 548 posts. If they come from unrelated communities, 0.86 measures topic separation rather than mood, and the majority baseline is the only control, which any topic-sensitive model clears. This is checkable against the public dataset, so it is a reporting gap rather than a hidden flaw, but it is not in the limitations list and it should be. Second, reporting discipline: 0.72 in the Results prose against 0.73 in Table I and the abstract; the number of posts dropped for read or API errors is never stated; one call per post with no temperature, seed, or snapshot. Third, the 'None' bucket in Figure 3 is defined by the model's own refusals, so the five-class weighted F1 is partly a measure of abstention rate. Minor: the suicidality margin, 0.37 against 0.19, is thin enough that I would want the per-class breakdown, not just weighted F1. Who it is for: people who need a quick baseline before committing to a fine-tuned model, and anyone writing the GPT-4 follow-up. It is workshop-tier, not a main-conference result. I would send it out, because the claim is modest, the data is public, and the fixes are all in the writing.","headline":"A thin but honest zero-shot baseline report; the numbers are checkable, but the depression negative class has no stated provenance and that is the one thing that could undercut the headline 0.86.","tokens_in":8342,"tokens_out":1812,"would_cite":false,"duration_ms":44106,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"pith_extraction":{"msc":[],"pacs":[],"model":"selfhosted/deepseek-v4.1-flash","headline":"An off-the-shelf chat model, given only a one-line prompt, classifies social media posts for stress and depression well above a majority-class guess, and does noticeably worse on a five-way suicidality task.","keywords":["ChatGPT","large language models","mental health","stress detection","depression detection","suicidality detection","zero-shot classification","social media text"],"falsifier":"Re-run the depression task with negative-class posts drawn from the same subreddits and topic distribution as the positive posts; if F1 falls from 0.86 toward the 0.60 majority baseline, the original score was measuring topic separation rather than depression.","tokens_in":8699,"feed_emoji":"🧠","tokens_out":1966,"duration_ms":52303,"temperature":0.7,"pith_summary":"The paper asks a simple, practical question: without any fine-tuning, training data, or task-specific engineering, how far does a general-purpose chat model get on text-based mental health classification? The author takes three already-annotated public datasets of social media posts — stress versus non-stress, depression versus non-depression, and a five-way expert-labelled suicidality scheme — and asks ChatGPT (gpt-3.5-turbo backend) to assign each post to one of the named classes, with a single prompt and a single API call per post. The reported weighted F1 scores are 0.73, 0.86 and 0.37, against majority-class baselines of 0.35, 0.60 and 0.19. The point a sympathetic reader should take away is not that the model is clinically useful, but that a general language model already carries enough signal about distress in ordinary text that a bare prompt clears the trivial baseline on all three tasks, which is the precondition for treating such models as a default backend for language-understanding work in mental health applications.","feed_headline":"ChatGPT beats a majority guess on mental health text","feed_subtitle":"Zero-shot prompting scores F1 0.86 for depression and 0.73 for stress, but only 0.37 across five suicidality classes.","key_machinery":"The mechanism is minimal by design: a fixed natural-language prompt of the form 'Which of the classes can the following post be assigned to? Only reply with answers from one of the classes: CLASS NAMES', issued through the ChatComplete API with the gpt-3.5-turbo backend, with the first response taken as the prediction and compared against the dataset's existing annotations. The evaluation apparatus — weighted F1 for the multi-class case, balanced accuracy, and confusion matrices against a majority-class baseline — is what carries the argument, since the baseline is the only thing standing between the numbers and a claim of trivial class prior exploitation.","core_discovery":"The central claim is that zero-shot classification by ChatGPT substantially exceeds a dominant-class baseline on all three mental health detection tasks, and that this gap is evidence of a usable latent ability to infer mental state from social media text. Concretely, the paper reports weighted F1 of 0.73 for two-class stress detection (baseline 0.35), 0.86 for two-class depression detection (baseline 0.60), and 0.37 for five-class suicidality detection (baseline 0.19), with balanced accuracies of 0.73, 0.85 and 0.33. The confusion matrices show where the ability thins out: on the five-way suicidality task most inputs collapse into the Suicidal Ideation category, and the author notes that the same collapse appears in a supervised CNN model trained on the same data, which he reads as evidence that the class boundaries themselves overlap rather than as a pure failure of the model. The author also places the numbers against in-domain supervised results — a fine-tuned BERT reaching 0.81 on stress, and a feature-engineered neural model reaching 0.93 on depression — and argues that since his numbers are zero-shot, closing that gap by fine-tuning is a plausible next step rather than a hope.","pith_inferences":["The headline depression number is the one most exposed to dataset construction: if the 548 negative posts come from unrelated subreddits, a strong F1 can be earned by detecting community topic and writing style rather than depressive content, so a same-community negative control would be the decisive check.","A prompt-sensitivity sweep is cheap and would sharpen the claim considerably: if F1 moves by several points across paraphrases of the same class list, the reported numbers describe one prompt rather than the model's ability.","Repeated sampling per post would convert the single-response point estimate into a distribution, and the spread itself may be informative — high-variance posts are plausibly the ones near the class boundary.","Because the model was never shown examples, its errors on the suicidality task are a partial signal about where human category boundaries are thin, which is useful annotation feedback independent of any deployment plan."],"forward_implications":["If the reported gaps hold, a general chat model can serve as a zero-effort first-pass filter for stress and depression text before any labelled data or trained model is available.","The suicidality result sets a realistic expectation: five-way severity schemes are where prompt-only classification breaks down, so applications needing that granularity still require supervised training or expert annotation.","Because the model was evaluated zero-shot, the gap to fine-tuned BERT (0.81 on stress) and to feature-engineered models (0.93 on depression) becomes a concrete target for adaptation rather than a ceiling.","The agreement between ChatGPT's confusion pattern and that of a supervised CNN on the same suicidality data suggests the five-class label scheme itself may have overlapping boundaries, independent of which model is applied.","Re-annotating the stress and depression datasets with multiple experts would test whether model errors track genuine annotator disagreement rather than model failure."],"supporting_citations":[{"why":"Supplies the stress detection dataset used for the first task and the fine-tuned BERT result of 0.81 that the zero-shot number is compared against.","marker":"[10]"},{"why":"The Dreaddit stress dataset that inspired the stress data, establishing the provenance and labelling scheme of the positive and negative classes.","marker":"[11]"},{"why":"Supplies the depression detection dataset, both the depression-class posts and the non-depression negative class the 0.86 F1 is computed against.","marker":"[12]"},{"why":"Supplies the suicidality dataset with the five expert-assigned classes used in the third task.","marker":"[13]"},{"why":"Supplies the 500-user subset used in this evaluation and the supervised CNN baseline of 0.65 whose confusion pattern the author compares with ChatGPT's.","marker":"[14]"},{"why":"The BERT model whose fine-tuned stress-detection performance frames how far zero-shot prompting sits below in-domain supervised training.","marker":"[15]"},{"why":"Reports the 0.93 depression-detection F1 that the author uses as the supervised benchmark the 0.86 zero-shot score approaches.","marker":"[16]"}],"fun_headline_variants":["ChatGPT clears baseline on stress and depression text, not on five-way suicidality","Zero-shot ChatGPT F1: 0.73 stress, 0.86 depression, 0.37 suicidality","Five-way suicidality detection stalls at F1 0.37 as ChatGPT collapses to one class","ChatGPT beats a dominant-class baseline in all three mental health text tasks","Fine-tuned BERT still leads on stress at 0.81 versus ChatGPT zero-shot 0.73"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The claim rests on the assumption that each dataset's positive and negative classes differ in the mental health construct being measured rather than in topic, community, or writing style, which matters most for the depression task where the negative posts are described only as coming from a non-depression category.","fun_headline_variants_meta":{"raw":{"variants":["ChatGPT clears baseline on stress and depression text, not on five-way suicidality","Zero-shot ChatGPT F1: 0.73 stress, 0.86 depression, 0.37 suicidality","Five-way suicidality detection stalls at F1 0.37 as ChatGPT collapses to one class","ChatGPT beats a dominant-class baseline in all three mental health text tasks","Fine-tuned BERT still leads on stress at 0.81 versus ChatGPT zero-shot 0.73"]},"model":"selfhosted/deepseek-v4.1-flash","effort":"low","attempts":1,"cost_usd":0.0,"raw_usage":{"total_tokens":1567,"prompt_tokens":986,"completion_tokens":581,"prompt_tokens_details":null,"completion_tokens_details":{"reasoning_tokens":470}},"tokens_in":986,"tokens_out":581,"duration_ms":13539,"image_pages":[],"temperature":1.0,"reasoning_budget":50,"reasoning_tokens":470,"transport_provider":"selfhosted","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-11T19:25:12.995154+00:00","model_set":{"reader":"selfhosted/deepseek-v4.1-flash"},"falsifier":"Re-run the depression task with negative-class posts drawn from the same subreddits and topic distribution as the positive posts; if F1 falls from 0.86 toward the 0.60 majority baseline, the original score was measuring topic separation rather than depression.","supporting_citations":[{"cited_title":"Insight stress analysis,","cited_arxiv_id":null,"evidence_quote":"Supplies the stress detection dataset used for the first task and the fine-tuned BERT result of 0.81 that the zero-shot number is compared against."},{"cited_title":"Dreaddit: A Reddit Dataset for Stress Analysis in Social Media","cited_arxiv_id":"1911.00133","evidence_quote":"The Dreaddit stress dataset that inspired the stress data, establishing the provenance and labelling scheme of the positive and negative classes."},{"cited_title":"Identifying depression,","cited_arxiv_id":null,"evidence_quote":"Supplies the depression detection dataset, both the depression-class posts and the non-depression negative class the 0.86 F1 is computed against."},{"cited_title":"Suicide risk assessment using reddit,","cited_arxiv_id":null,"evidence_quote":"Supplies the suicidality dataset with the five expert-assigned classes used in the third task."},{"cited_title":"Knowledge-aware assessment of severity of suicide risk for early intervention,","cited_arxiv_id":null,"evidence_quote":"Supplies the 500-user subset used in this evaluation and the supervised CNN baseline of 0.65 whose confusion pattern the author compares with ChatGPT's."},{"cited_title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","cited_arxiv_id":"1810.04805","evidence_quote":"The BERT model whose fine-tuned stress-detection performance frames how far zero-shot prompting sits below in-domain supervised training."},{"cited_title":"Detection of depression- related posts in reddit social media forum,","cited_arxiv_id":null,"evidence_quote":"Reports the 0.93 depression-detection F1 that the author uses as the supervised benchmark the 0.86 zero-shot score approaches."}],"review_version":1}