{"id":"2a7f2993-1523-4d7b-a678-d6aa371c2e72","arxiv_id":"2411.15586","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Feature-based machine learning models detect ADHD from social media text with F1 around 0.77 in-domain and generalize to Twitter better than fine-tuned transformers.","lead":"This paper tests whether machine learning can spot ADHD from people's social media posts, comparing simple feature-based models, a BiLSTM, and transformer models. The authors find that feature-based models keep their performance better when moving from Reddit to Twitter data, and identify linguistic markers such as readability, negative emotion, and self-reference.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central ADHD-detection claim rests on a control group that excludes all mental-health-related posting; until models are tested against psychiatric controls, the reported F1 may reflect mental-health discourse, not ADHD-specific language.","rationale":"The reader's weakest-assumption analysis correctly identifies the control-group confound as the most load-bearing issue. My independent reading of the manuscript confirms that Section 3.1's control definition—no mental-health-related subreddit activity and no mention of ADHD or similar mental health terms—makes the classification task partly a proxy for 'does this user discuss mental health?' The ADHD group necessarily discusses mental health; the control group is selected to avoid it. This confound directly threatens the strongest claim, since the reported F1 of 0.77 and the cross-platform robustness results could be explained by topic/discourse detection rather than ADHD-specific signal. The feature ablation results in Table 5 are consistent with this interpretation: the most important features are emotion terms and health topics, which are typical of any mental-health-related discussion. I do not see a stronger competing concern. The lack of error bars is important but secondary, because even with error bars the central interpretation would still be confounded. The proprietary feature tool is a reproducibility issue, but it does not by itself undermine the validity of the reported comparisons. The proposed psychiatric-control experiment is a direct, feasible test: if ADHD-versus-psychiatric-control performance collapses, the central claim must be revised; if it remains high, the confound is less severe than feared. The reader's CONDITIONAL verdict is appropriate, and my analysis does not move it to a different category, so I recommend UNCHANGED.","tokens_in":12331,"tokens_out":2913,"duration_ms":30798,"concrete_test":"Run the identical BiLSTM and Gradient Boosting pipelines on a modified training/evaluation setup where the control group is replaced by psychiatric controls: users self-reporting depression, anxiety, or bipolar disorder from SMHD or Twitter-STMHD, matched on number of posts, posting activity, and general mental-health discourse volume. If the F1 score drops substantially from 0.77 toward chance, the original model was detecting mental-health-related discourse rather than ADHD-specific language. As a complementary check, retrain on ADHD users after removing all mental-health-related posts (keeping only non-mental-health posts) versus the original clean controls; if performance collapses, the label-discourse coupling is the main driver.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is that social media language carries ADHD-specific signal, with BiLSTM and MentalRoBERTa reaching F1 0.77 and feature-based models generalizing better out-of-domain. The load-bearing weakness is the control-group construction in Section 3.1. ADHD users are identified by self-reported diagnosis phrases, and their retained posts include mental-health-related content. Control users are defined as those who 'never posted or commented in mental health-related subreddits and never mentioned ADHD or similar mental health terms.' This couples the label almost perfectly with the presence of mental-health-related discourse. The preprocessing step that removes posts directly referencing ADHD diagnoses does not remove this coupling, because mental-health-related posts are deliberately retained. Consequently, a model can achieve high F1 by detecting whether a user discusses mental health at all, rather than by detecting ADHD-specific linguistic patterns. The feature-importance results in Table 5 reinforce this concern: the top features include emotion categories such as loathing, disgust, terror, rage, grief, anger, and fear, plus Topic Health and cohesion measures, all of which are natural correlates of mental-health-related posting. The out-of-domain experiment on Twitter-STMHD inherits the same issue in weaker form, since ADHD users are self-reported via anchor tweets while controls have no reported mental health conditions. This concern does not invalidate the paper's internal implementation, but it does undercut the external validity of the central claim: the reported F1 scores cannot be interpreted as evidence of ADHD detection unless the models also separate ADHD users from users with other mental health conditions. This is not a disagreement with the field's consensus; it is a correctness risk in the experimental design.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents user-level ADHD detection from social media text, comparing shallow feature-based classifiers (Logistic Regression, Random Forest, SVM, Gradient Boosting), a BiLSTM trained on 344 hand-crafted linguistic features, and transformer baselines (RoBERTa, MentalRoBERTa). The authors report that BiLSTM and MentalRoBERTa achieve the best in-domain F1 (0.77) on a new Reddit dataset, that feature-based models degrade less than transformers in an out-of-domain Twitter evaluation, and that SP-LIME and MDI analyses identify interpretable linguistic markers of ADHD.","tokens_in":1204,"tokens_out":1402,"duration_ms":52720,"significance":"If the central claims hold, the paper would make a useful contribution to interpretable digital phenotyping for ADHD: it reimplements and extends the SMHD collection methodology, uses a relatively large user-level sample, includes an explicit out-of-distribution evaluation across platforms, and compares transparent models against strong transformer baselines. The manuscript is weakened, however, by a control-group construction that couples the ADHD label with the presence of mental-health-related discourse, and by the absence of confidence intervals or significance tests for the reported performance differences. The feature-attribution experiments are also mislabeled as ablations. These issues are addressable but require additional experiments and reanalysis.","major_comments":[{"comment":"The control group is defined as users who never posted or commented in mental health-related subreddits and never mentioned ADHD or similar mental health terms. Because the ADHD group is identified by self-reported diagnosis phrases and its retained posts deliberately include mental-health-related content (the paper states that posts directly referencing ADHD diagnoses were removed while mental health-related posts were retained), the label is coupled with the presence or absence of mental-health discourse. A model can therefore achieve high F1 by detecting mental-health-related language generally rather than ADHD-specific linguistic patterns. This confound also affects the feature-importance interpretation in Tables 3 and 5, where emotion categories and health topics dominate. The authors should evaluate against a control group of users who post about other mental health conditions, or otherwise demonstrate that the classifier separates ADHD from non-ADHD mental-health discourse.","section":"Section 3.1"},{"comment":"The key comparative claims, that BiLSTM and MentalRoBERTa lead with F1 0.77 and that feature-based models are more robust than transformers out-of-domain, are reported without confidence intervals or significance tests. The reported differences are small (0.01 in the in-domain comparison and 0.07-0.08 in the out-of-domain comparison) and, in the absence of uncertainty estimates on user-level predictions, do not establish model ordering or a robustness advantage. Bootstrap confidence intervals or paired significance tests should be added for both the Reddit and Twitter evaluations.","section":"Section 4.1 / Section 4.3, Tables 2 and 4"},{"comment":"The experiments labeled feature ablation do not remove feature groups and retrain the models; SP-LIME is a post-hoc global explanation method, and MDI is a similar attribution measure from the Random Forest. The claimed contribution Feature Ablation and the conclusion that specific feature groups play a significant role are therefore not supported by the reported experiments. Either retrain the models with feature groups excluded and report the resulting performance changes, or rename the section and the corresponding claims to feature attribution.","section":"Section 4.2, Table 3"}],"minor_comments":[{"comment":"There is a typo in the citation Zhou et al., 2022, with a doubled comma and space; this should be corrected.","section":"Introduction"},{"comment":"The Limitations section lists several threats to generalizability but does not acknowledge the control-group selection confound described in my major comment; this omission should be addressed explicitly in a revision.","section":"Section 7, Limitations"},{"comment":"The threshold that discards posts with fewer than three sentences is not accompanied by exclusion counts for the ADHD and control groups; because the ADHD group has substantially more texts per user (317,073 vs. 174,765), the threshold could affect the two groups differentially, and this should be reported.","section":"Appendix, Pre-processing"},{"comment":"Hyperparameter tuning is described only as grid search and sequential tuning without listing the search ranges; reporting the grids would be necessary for reproducibility.","section":"Section 3.3"},{"comment":"The reference Loshchilov, 2017 is incomplete; the paper should cite the actual AdamW publication (Loshchilov and Hutter, ICLR 2019).","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The control-group confound is the main barrier to acceptance; it is fixable through new experiments with psychiatric or mental-health-discourse control groups. The feature ablation mislabeling and missing uncertainty quantification also need to be addressed. I would not reject the paper because the dataset, the cross-platform evaluation, and the interpretability focus are valuable, but the current version overstates what the evidence supports."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent applied NLP study with a real empirical contribution, but its headline claim—that models detect ADHD-specific language—rests on a control group that makes the result hard to interpret. The ADHD users are selected by self-reported diagnosis phrases; controls are users who never posted in any mental-health subreddit and never mentioned ADHD or similar terms. That couples the label almost perfectly with mental-health-related discourse. A model can score 0.77 F1 by detecting 'talks about mental health' rather than ADHD. The feature list reinforces the worry: top MDI features include loathing, disgust, terror, rage, grief, anger, fear, plus cohesion measures—exactly what you'd expect from the confound. The preprocessing that removes posts directly referencing an ADHD diagnosis does not fix this, because other mental-health posts are retained.\n\nWhat's genuinely good: the paper builds a new 12k+12k user ADHD dataset from Reddit using SMHD-style patterns, benchmarks seven models (shallow, BiLSTM, two transformers), runs a proper out-of-domain Reddit→Twitter evaluation, and reports SP-LIME and MDI feature analyses. That's real work, and the OOD setup is more than most MIDAS papers do. The in-domain F1 differences between models are small (0.74–0.77) and no confidence intervals are given, so the 'BiLSTM and MentalRoBERTa tie' and 'feature-based models generalize better' conclusions are weaker than the prose suggests. Still, the qualitative pattern—feature-based models degrade less OOD—is plausible and worth checking.\n\nThe confound is not a minor weakness; it is load-bearing. Until the models are tested against psychiatric controls (e.g., users with depression or anxiety, or at least a control group that posts about mental health), the reported F1 cannot be interpreted as ADHD detection. The authors' limitations section mentions single-disorder focus and social-media-only data, but never addresses the control selection. The proprietary feature tool CYMO also blocks replication until a detailed feature specification is released.\n\nBottom line: this deserves a serious referee, but the referee should push for a differential/control-group experiment and variance estimates. If the authors can show the classifier separates ADHD from other mental-health discourse, the paper would be a useful contribution to the MIDAS literature. As is, I'd treat the central claim as conditional.","headline":"A solid, honest applied NLP study whose central ADHD-detection claim is undermined by a control group that confounds ADHD with mental-health discourse in general.","tokens_in":13208,"tokens_out":1722,"would_cite":false,"duration_ms":14396,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A feature-based BiLSTM matches the best transformer for ADHD detection on Reddit, and feature models transfer more robustly to Twitter.","keywords":["ADHD detection","social media NLP","interpretable machine learning","BiLSTM","transformer generalization","feature ablation","Reddit","Twitter"],"falsifier":"Take the same feature pipeline and retrain the classifiers with a control group sampled from users who post actively in mental-health subreddits about conditions other than ADHD, matching the ADHD group on posting volume and mental-health topic mentions; if the F1-score drops to chance, the reported signal is mental-health discourse, not ADHD.","tokens_in":12111,"feed_emoji":"🧠","tokens_out":8256,"duration_ms":66750,"temperature":0.7,"pith_summary":"The paper sets out to show that ADHD can be picked out from ordinary social-media text with moderate accuracy, and that this does not force a choice between transparency and performance. On Reddit data, a BiLSTM trained on 344 interpretable linguistic features reaches an F1-score of 0.77, matching the domain-adapted transformer MentalRoBERTa. When the same models are tested on Twitter, the feature-based classifiers degrade less (F1 drop of 0.06–0.12) than transformers (drop of 0.13–0.15), suggesting that human-readable features are more portable across platforms. The paper also identifies which feature groups carry the signal, with readability, grammatical categories, and emotion leading the rankings. If correct, these results make a case for building explainable digital screening tools whose decisions clinicians can inspect.","feed_headline":"F1 of 0.77 for ADHD detection from social text","feed_subtitle":"Feature-based BiLSTM matches transformers and transfers better to Twitter.","key_machinery":"The load-bearing machinery is a sentence-level feature extraction system that computes 344 human-interpretable measures in eight groups: Syntactic Complexity, Lexical Richness/Complexity, Cohesion, Stylistics, Readability, Grammatical Categories, Topical Categories, and Emotion Categories. For each user, these sentence vectors are averaged for shallow classifiers or fed as sequences into a three-layer Bidirectional Long Short-Term Memory (BiLSTM) network. Interpretability comes from SP-LIME, a global aggregation of LIME local linear explanations that ranks the eight feature groups, and from random-forest mean decrease in impurity, which ranks individual features. The feature-first design is what lets the authors trace predictions back to specific linguistic dimensions.","core_discovery":"The central claim is that linguistic features engineered from eight dimensions of verbal behavior are sufficient for ADHD detection at levels comparable to fine-tuned transformers, while remaining interpretable. The paper reports in-domain F1-scores of 0.74–0.77 across shallow and deep models, with BiLSTM and MentalRoBERTa both at 0.77; BiLSTM achieves the highest recall (0.79). In out-of-distribution transfer from Reddit to Twitter, Gradient Boosting is the most robust (F1 0.70, drop 0.06), whereas RoBERTa and MentalRoBERTa lose recall heavily and land at F1 0.63 and 0.62. Feature ablations using SP-LIME rank Readability, Grammatical Categories, and Topical Categories as the most informative groups, and the paper links these to measurable linguistic patterns: lower readability with higher syntactic and lexical complexity, greater use of self-referential pronouns, more negative emotion words, and more health-related topics. The conclusion is that interpretable feature-based models generalize better than black-box transformers for this task.","pith_inferences":["Because the control group never discusses mental health, the model may be learning topic rather than ADHD; a stricter control that also posts about mental-health topics (e.g., anxiety or depression) would reveal how much of the signal is ADHD-specific.","The 0.77 ceiling may reflect noisy self-reported labels on Reddit; the same feature framework with verified clinical labels could plausibly score higher or lower, testing the validity of the digital phenotype.","The same eight feature groups could be applied to autism or anxiety detection; if the same groups (readability, emotion, self-reference) dominate, they may be shared distress markers rather than ADHD-specific.","A direct test of the readability finding would be to compare clause-per-sentence and vocabulary measures against word count alone, to see whether ADHD language is structurally different or just longer and more verbose."],"forward_implications":["A screening tool built on these features could show clinicians which linguistic dimensions pushed a given user into the ADHD-positive group, unlike a transformer's attention weights.","The feature-ranking results give candidate digital biomarkers—low readability, high self-referential pronoun use, elevated negative-emotion words—that can be tested in clinical interviews or diary text.","The small OOD degradation of feature models (0.06–0.12 F1 drop) suggests the framework may transfer to other social platforms or messaging apps without full retraining.","The large recall drop of transformers on Twitter (0.49) indicates that transformer embeddings pick up platform-specific phrasing, so cross-platform deployment should favor feature-based models.","The reported 0.77 F1 can serve as a baseline for future multimodal or LLM-based ADHD detection on Reddit data."],"supporting_citations":[{"why":"Supplies the SMHD data-collection methodology that the ADHD dataset reimplements and refines.","marker":"(Cohan et al., 2018)"},{"why":"Prior finding on language of ADHD in social media that the paper's feature results are compared against.","marker":"(Guntuku et al., 2019)"},{"why":"MentalRoBERTa, the domain-adapted transformer baseline that ties with BiLSTM in-domain.","marker":"(Ji et al., 2021)"},{"why":"Evidence that mental-health models generalize poorly, motivating the cross-platform evaluation.","marker":"(Harrigian et al., 2020)"},{"why":"Provides the Twitter-STMHD dataset used for the out-of-distribution test.","marker":"(Singh et al., 2022)"},{"why":"LIME, on which SP-LIME builds for global feature-group explanation.","marker":"(Ribeiro et al., 2016)"},{"why":"Random forests and mean decrease in impurity used as the secondary feature-importance method.","marker":"(Breiman, 2001)"}],"fun_headline_variants":["Interpretable ADHD detection rivals black-box transformers","BiLSTM matches transformers, transfers better to Twitter","ADHD screening: transparent models match deep learning","Feature-based ADHD detection generalizes across platforms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The weakest assumption is that control users who never mention mental health are a fair comparison for ADHD users, who by definition discuss ADHD and frequently other mental-health topics; the models may be detecting mental-health discourse rather than ADHD-specific language.","fun_headline_variants_meta":{"raw":{"variants":["Interpretable ADHD detection rivals black-box transformers","BiLSTM matches transformers, transfers better to Twitter","ADHD screening: transparent models match deep learning","Feature-based ADHD detection generalizes across platforms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000142,"raw_usage":{"total_tokens":1152,"prompt_tokens":910,"completion_tokens":242,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":184}},"tokens_in":526,"tokens_out":242,"duration_ms":3179,"temperature":1.0,"reasoning_tokens":184,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:07:48.277467+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same feature pipeline and retrain the classifiers with a control group sampled from users who post actively in mental-health subreddits about conditions other than ADHD, matching the ADHD group on posting volume and mental-health topic mentions; if the F1-score drops to chance, the reported signal is mental-health discourse, not ADHD.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SMHD data-collection methodology that the ADHD dataset reimplements and refines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior finding on language of ADHD in social media that the paper's feature results are compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Twitter-STMHD dataset used for the out-of-distribution test."}],"review_version":1}