{"id":"e50b51e8-04d1-4fe6-9cfb-f9ffa0373b69","arxiv_id":"2508.15440","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"M-HELP is a new expert-annotated Reddit dataset for detecting help-seeking posts, mental health disorders, and their causes, benchmarked across 14 models.","lead":"This paper introduces M-HELP, a Reddit-based dataset annotated by mental health professionals with help-seeking labels, nine mental health disorders, and root causes, benchmarked across many NLP models. Generalists should care because an automated help-seeking detector could help NGOs and platforms route at-risk users to support services.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Help-seeking label reliability is unproven and Appendix C shows a clear counterexample, so M-HELP's central labels may be inconsistent.","rationale":"The single most load-bearing issue is not the novelty claim or reproducibility but the validity of the labels themselves. The central claim is that M-HELP enables detection of help-seeking; if the help-seeking labels are wrong, no model result can be meaningful. The Appendix C example is direct evidence of label inconsistency: the post contains multiple help-seeking signals that the paper's own definition explicitly includes, yet it is labeled 0. This is more decisive than the lack of per-label kappa for disorders, because it undermines the primary binary task, not a secondary multi-label one. The reader's weakest assumption focused on the general difficulty of diagnosing from posts; I narrow it to an internal inconsistency that is observable in the manuscript. A re-annotation audit with blinded clinicians would settle whether this is a one-off error or a systemic annotation bias. If the audit finds many false negatives, the dataset's primary labels are unreliable and the headline results should be interpreted with caution; if the audit confirms the labels, the Appendix C example would be an outlier and the dataset may be usable. The paper's other weaknesses (copy-paste artifacts, missing comparisons to MentalHelp, code release) are real but do not threaten the central claim as directly. Therefore the verdict remains conditional: the paper is promising but must provide label validation before the central claim is accepted.","tokens_in":9900,"tokens_out":4091,"duration_ms":38766,"concrete_test":"Independently re-annotate a stratified random sample of 200 M-HELP posts (spanning train/val/test splits) using two licensed clinicians blinded to the original labels and to each other. Apply only the paper's Section 2.2a help-seeking definition. Compute Fleiss kappa for the help-seeking label on this sample, and the false-negative rate among posts originally labeled 0 that both re-annotators label 1 (focusing on posts containing explicit markers like 'need help', 'don't know what to do', 'can't take it', 'disappear'). If kappa < 0.6 or false-negative rate > 15%, the help-seeking labels are unreliable and the central claim fails. Also compute per-label agreement for the 9 disorder labels and the cause labels on the same sample to validate the rest of the annotation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that M-HELP is the first dataset for detecting help-seeking and that models can identify help-seekers—rests on the reliability of the binary help-seeking labels. The paper reports one Fleiss kappa (0.8355) for help-seeking but gives no agreement for the nine disorder labels or the cause labels, and the kappa is asserted without error bars or per-item breakdown. More damningly, Appendix C shows a post in which the user says \"I don't know what to do\", \"I rly need to find a job\", \"I don't know how long I can handle it\", and \"I've been starting to think more seriously about disappearing\"; the paper reports \"Actual label: 0\". This contradicts the Section 2.2a guideline that indirect statements like \"I don't know what to do\" count as help-seeking. If this is a single annotation error, the dataset still lacks validation; if it reflects the annotators' operational rule, then the label misses the most acute help-seeking signals. Either way, the help-seek classification results in Table 3 are not a reliable measure of help-seeking detection, and the disorder/cause labels, produced by the same process, are equally unvalidated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces M-HELP, a Reddit-based dataset for detecting mental-health help-seeking behavior. 1,853 posts from r/MentalHealthSupport and r/mentalillness were annotated by three mental health professionals with (i) a binary help-seeking label, (ii) multi-label DSM-5/ICD-10 disorder labels from nine categories, and (iii) a free-text cause label. The authors benchmark encoder, long-context, encoder-decoder, decoder-only, and closed-source models on disorder classification and help-seeking classification, reporting best weighted F1 values of 76.59% (Gemini-1.5, disorder) and 60.18% (Mental-BERT, help-seeking). The paper claims this is the first dataset focused on help-seeking detection and that models trained on it can address three tasks: identifying help-seekers, diagnosing mental health conditions, and uncovering root causes.","tokens_in":10130,"tokens_out":4409,"duration_ms":50054,"significance":"If the annotations are reliable, M-HELP would fill a genuine gap: most prior social-media mental health datasets target disorder detection or counseling summarization, not the identification of users who are actively or indirectly seeking help. The multi-label disorder annotation and the inclusion of causes are useful additions, and the broad model comparison provides a reasonable starting benchmark. The paper also ships the dataset and code, which is a strength. However, the central value of the resource depends on label quality, and the manuscript currently provides insufficient evidence for that quality; one of its own appendix examples appears to contradict the annotation guidelines. The absence of any evaluation of the third claimed task (cause classification) further limits the contribution.","major_comments":[{"comment":"The post in Appendix C contains 'I don't know what to do in live', 'I've been starting to think more seriously about disappearing', and 'I just don't know how long I can handle it'. The paper reports 'Actual label: 0'. Section 2.2a explicitly states that indirect statements such as 'I feel lost and don't know what to do' should count as help-seeking. This example is therefore not merely an LLM confusion; it shows that the gold-standard help-seeking label misses acute distress. The authors must re-examine this instance, report the corrected label, and provide a detailed adjudication procedure for similar borderline cases.","section":"Appendix C vs. Section 2.2a"},{"comment":"Inter-annotator agreement is reported only for the binary help-seeking label (Fleiss kappa = 0.8355), with no confidence interval and no clarification of whether all posts were independently annotated by all three annotators. No agreement is reported for the nine disorder labels or the cause labels. Since Table 3 presents disorder classification results as a central contribution, the lack of reliability evidence for those labels makes the results uninterpretable. Please report per-label agreement (e.g., per-disorder kappa or prevalence-adjusted bias-adjusted kappa) and the disagreement-resolution process.","section":"Section 2.2"},{"comment":"The paper defines a cause label and lists 'uncovering the root causes of issues' as one of three tasks, yet Table 3 contains no cause classification results, and no cause label distribution is given anywhere. The only cause-related artifact is a word cloud. The claim that M-HELP supports three tasks is therefore unsupported. Either add cause-classification experiments and summary statistics, or explicitly reframe the contribution as two tasks with causes as an auxiliary annotation.","section":"Section 2.2c and Table 3"},{"comment":"Contribution 1 claims 'the first dataset focused on detecting help-seeking behavior on social media.' However, the reference list itself includes MentalHelp (Raihan et al., 2024), a multi-task dataset for mental health in social media, whose title suggests a closely related objective. The novelty claim is not established without an explicit comparison to MentalHelp and a broader search of help-seeking datasets (e.g., CLPsych shared tasks). Please add a related-work comparison and soften the 'first' claim if needed.","section":"Section 1 and References"},{"comment":"The paper repeatedly uses clinical terms such as 'diagnosing mental health conditions' and assigns DSM-5/ICD-10 disorder labels to individual Reddit posts without any clinical interaction. A single post cannot support a formal diagnosis, and labeling a post with 'MDD' or 'PTSD' risks overstating clinical validity. At minimum, the task should be described as 'disorder mention classification' or 'symptom-based labeling,' and the limitations of using DSM-5/ICD-10 in this setting should be stated clearly. This affects how readers interpret the disorder classification results.","section":"Section 2.2b and Table 3"}],"minor_comments":[{"comment":"The header uses 'PSTD' instead of 'PTSD'.","section":"Table 1"},{"comment":"The final sentence says the configuration 'allowed us to effectively train and evaluate our models across various mental health meme classification tasks.' This appears to be a copy-paste error from a different paper; the tasks here are help-seeking and disorder classification, not meme classification.","section":"Appendix A"},{"comment":"The text says 'Table 4 presents the examples of the datasets,' but the examples are in Table 5, not Table 4.","section":"Appendix D"},{"comment":"The table caption uses 'Require Help' for label 1, while Section 2.2 uses 'Help-Seeking.' Please standardize the terminology.","section":"Table 2"},{"comment":"The caption is grammatically unclear: 'violet and green for train and test splits' should specify which color corresponds to which split.","section":"Figure 1a"},{"comment":"The ethics statement mentions compliance with Reddit's API terms, but it does not discuss anonymization, the risk of re-identification, or the potential harm to individuals whose posts are included. Given the sensitive nature of suicidal ideation in the data, a more detailed ethical discussion is warranted.","section":"Ethics Statement"}],"recommendation":"major_revision","confidential_remarks":"The Appendix C example is the most serious issue: if the 'Actual label: 0' reflects the annotation rule, then the help-seeking labels are not reliable for the most acute cases, which would undermine the dataset's core purpose. The missing cause-classification evaluation is also a substantial gap, not a minor omission. I would not reject solely on the novelty claim, but the authors need to either validate and correct the labels or substantially narrow their claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: M-HELP is a plausible dataset idea—binary help-seeking labels plus multi-label DSM-5 disorders plus free-text causes on Reddit posts—and the authors have done a reasonable job of benchmarking a wide range of models. But the central label, help-seeking, is not trustworthy on the evidence in the paper, and the 'first dataset' framing is unverified against MentalHelp, which they themselves cite.\n\nWhat's new: no existing dataset in their cited literature combines these three annotation layers. The data is on Hugging Face, so it's accessible. The model comparison is broad (BERT, Longformer, BART, Llama, GPT-4o, Gemini) with detailed metrics, and the authors are honest that current scores are too low for real-world use. The task framing—active help-seeking rather than disorder presence—addresses a real gap.\n\nThe soft spots are serious. Appendix C shows a post where the user says \"I don't know what to do,\" \"I rly need to find a job,\" and \"I've been starting to think more seriously about disappearing\"—and the gold label is 0. This directly contradicts their stated guideline that indirect statements like \"I don't know what to do\" count as help-seeking. Either this is an annotation error, in which case the dataset needs re-annotation, or it's not an error, in which case the operational rule is wrong. Either way, the help-seeking labels can't be taken at face value. The reported Fleiss kappa (0.8355) is only for the binary help-seeking category; there's no agreement reported for the nine disorder labels or the cause labels, which are produced by the same process and are equally unvalidated. Given that diagnosing a disorder from a single post is clinically dubious, the disorder classification results in Table 3 should be treated as unvalidated.\n\nThe novelty claim is also unsupported. MentalHelp (Raihan et al., 2024) is cited in related work but never compared. The paper needs to show what help-seeking labels, if any, exist there and why M-HELP differs. There are also signs of sloppy drafting: Appendix A mentions \"mental health meme classification\" and a \"proposed model\" that doesn't exist in the paper, and the hyperparameter table lists LoRA settings while the experiments never mention LoRA. These don't kill the resource but they hurt reproducibility.\n\nWho this is for: researchers working on mental health NLP, specifically help-seeking detection, could get value from the dataset once the labels are fixed. It deserves a serious referee because the idea is useful and the problems are addressable, but I would not rely on the current release.","headline":"M-HELP is a useful dataset idea and benchmark run, but the help-seeking labels are not reliable as released, and the 'first dataset' claim needs explicit comparison with MentalHelp.","tokens_in":10626,"tokens_out":3130,"would_cite":false,"duration_ms":29507,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reddit posts that ask for help—directly or between the lines—carry labels for nine mental-health disorders and their causes in the new M-HELP dataset, which the authors argue is the first to target help-seeking behavior.","keywords":["mental health","help-seeking detection","social media","Reddit","multi-label classification","DSM-5","LLM benchmark","dataset annotation"],"falsifier":"Take a sample of M-HELP posts and ask an independent panel of clinicians not involved in the annotation to assign DSM-5/ICD-10 labels; if agreement among them is no better than chance, the disorder labels are not valid clinical targets. Alternatively, compare model predictions against posts written by users who later confirm their actual diagnosis.","tokens_in":9791,"feed_emoji":"🧠","tokens_out":6328,"duration_ms":57506,"temperature":0.7,"pith_summary":"M-HELP is a new dataset of 1,853 first-person posts taken from mental-health subreddits, each one annotated for three things: whether the poster is seeking help (even indirectly), which of nine DSM-5/ICD-10 mental-health disorders the post reflects (more than one label is allowed), and the underlying cause or trigger mentioned in the post. The paper's central claim is that this is the first dataset aimed specifically at detecting help-seeking behavior, not just at classifying disorders, and that models trained on it can address three tasks at once: identifying help-seekers, assigning disorder labels, and identifying root causes. The authors benchmark a wide range of large language models on these tasks and report that mental-health fine-tuned models such as Mental-BART perform comparably to closed-source Gemini-1.5 on disorder classification, while all tested models still underperform on help-seeking detection. If the claim holds, M-HELP gives NGOs, researchers, and platform moderators a structured way to surface posts that need human follow-up, with context about why.","feed_headline":"Dataset M-HELP flags who seeks help in Reddit posts","feed_subtitle":"Each post is labeled for help-seeking, nine disorders, and root cause to route social-media users to support.","key_machinery":"The load-bearing object is M-HELP itself, a dataset of 1,853 Reddit posts annotated with three layers of labels: a binary help-seeking label; a multi-label set of nine disorders (MDD, GAD, PTSD, SZD, BPD, OCD, AND, SUD, ASD) drawn from DSM-5/ICD-10 criteria; and a cause label naming the trigger or stressor (e.g., relationship challenges, financial hardship, trauma). The annotation was done by three mental-health professionals working to shared guidelines, with a reported Fleiss kappa of 0.8355 for the help-seeking category. The evaluation machinery consists of encoder-only, long-context, encoder-decoder, decoder-only, and closed-source models, all fine-tuned or prompted on the dataset; the c","core_discovery":"M-HELP is presented as the first dataset focused on detecting help-seeking behavior on social media. Each of its 1,853 posts carries three annotation layers: a binary help-seeking label, one or more of nine mental-health disorder labels from DSM-5/ICD-10, and a cause label pointing to the event or condition that triggered the distress. The authors report that three mental-health professionals annotated the data with near-perfect inter-rater agreement (Fleiss kappa = 0.8355), and they benchmark encoder, long-context, encoder-decoder, decoder-only, and closed-source models, finding that mental-health fine-tuned BART achieves the best open-source disorder classification while Mental-BERT leads","pith_inferences":["A direct validation experiment—taking a sample of M-HELP posts and comparing the disorder labels against structured clinical assessment (or against the poster's confirmed diagnosis) would test whether single-post DSM-5 labels are stable outside the annotating team.","The reported kappa covers only the binary help-seeking label; whether the nine multi-label disorders and the cause labels are equally reliable is left open, so treating 0.8355 as a global reliability figure is an extrapolation.","Extending the same three-layer annotation to other social platforms and languages would show whether help-seeking cues generalize beyond the specific Reddit communities used here.","The paper's example of a 'motivation' post that models frequently misread as a mental-health crisis suggests a useful refinement: an explicit 'demotivation vs. clinical distress' distinction in the annotation guidelines."],"forward_implications":["Social-media monitoring systems could use a help-seeking classifier to surface posts that a human counselor or crisis-resource bot follows up on, rather than only flagging posts that mention diagnostic terms.","Multi-label disorder and cause annotations let downstream systems distinguish 'anxiety caused by financial stress' from 'anxiety caused by relationship conflict,' which changes the kind of resource that would help.","Mental-health fine-tuned open models (Mental-BART, Mental-BERT) match or beat closed APIs on these tasks, making the benchmark usable without proprietary costs.","Because all tested models still score too low for reliable triage on help-seeking, the dataset functions more as a challenge benchmark than as a finished screening tool."],"supporting_citations":[{"why":"Supplies the kappa formula used to report the 0.8355 inter-annotator agreement for the help-seeking label.","marker":"Fleiss, 1971"},{"why":"Supplies the DSM-5 diagnostic criteria the annotators use to define the nine disorder labels.","marker":"Regier et al., 2013"},{"why":"Supplies the ICD-10 classification criteria that, together with DSM-5, define the disorder labels.","marker":"Manchikanti et al., 2011"},{"why":"MentalHelp, the closest prior multi-task mental-health dataset; the paper contrasts M-HELP against it to claim novelty on help-seeking.","marker":"Raihan et al., 2024"},{"why":"Prior explainable depression-detection dataset that includes cause-related features; the cause-label design extends this line of work.","marker":"Adarsh et al., 2023"},{"why":"Prior annotated dataset of interpersonal risk factors; supports the cause-label layer and the claim that causes are missing in prior work.","marker":"Garg et al., 2023"},{"why":"Explainable depression detection with large language models on social media; one of the prior efforts M-HELP positions itself against.","marker":"Wang et al., 2024"}],"fun_headline_variants":["M-HELP dataset flags help-seeking, disorder, and cause in Reddit posts","First dataset to spot active mental health help-seeking on social media","M-HELP: three-layer labels reveal who seeks help and why on Reddit","Benchmarking shows fine-tuned BART leads disorder classification on M-HELP","Detect help-seeking signals, disorders, and triggers with M-HELP dataset"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The paper assumes that three mental-health professionals can assign valid DSM-5/ICD-10 disorder labels to single Reddit posts without meeting the poster, and that the 0.8355 agreement measured for the binary help-seeking category also holds for the nine disorder labels and the cause labels.","fun_headline_variants_meta":{"raw":{"variants":["M-HELP dataset flags help-seeking, disorder, and cause in Reddit posts","First dataset to spot active mental health help-seeking on social media","M-HELP: three-layer labels reveal who seeks help and why on Reddit","Benchmarking shows fine-tuned BART leads disorder classification on M-HELP","Detect help-seeking signals, disorders, and triggers with M-HELP dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1388,"prompt_tokens":625,"completion_tokens":763,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":369,"completion_tokens_details":{"reasoning_tokens":672}},"tokens_in":369,"tokens_out":763,"duration_ms":8473,"temperature":1.0,"reasoning_tokens":672,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:52:10.868682+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of M-HELP posts and ask an independent panel of clinicians not involved in the annotation to assign DSM-5/ICD-10 labels; if agreement among them is no better than chance, the disorder labels are not valid clinical targets. Alternatively, compare model predictions against posts written by users who later confirm their actual diagnosis.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DSM-5 diagnostic criteria the annotators use to define the nine disorder labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ICD-10 classification criteria that, together with DSM-5, define the disorder labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MentalHelp, the closest prior multi-task mental-health dataset; the paper contrasts M-HELP against it to claim novelty on help-seeking."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior explainable depression-detection dataset that includes cause-related features; the cause-label design extends this line of work."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior annotated dataset of interpersonal risk factors; supports the cause-label layer and the claim that causes are missing in prior work."}],"review_version":1}