{"id":"fe2426ec-9913-4f1e-bdb0-e2450f431d75","arxiv_id":"2411.15404","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"RoBERTa outperforms other transformers and BERT-embedded LSTMs on a new 37,821-post Reddit dataset for suicidal-ideation detection, but the evaluation has annotation and significance gaps.","lead":"This paper compares five transformer models and eight LSTM variants for detecting suicidal ideation in Reddit posts, finding RoBERTa best at 93.22% accuracy. The value for a general reader is the new 37,821-post corpus and benchmark, though the dataset's label validation is flawed.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 93% accuracy is likely driven by subreddit/topic separation rather than suicidal-ideation detection; the negative class includes r/geopolitics and other topically distant subreddits, making the classification task trivial and undermining the central comparative claim.","rationale":"I agree with the reader that the dataset's label construction is a serious weakness, but I find a more specific and load-bearing aspect of the same dataset problem to be the deliberate selection of topically distant negative subreddits, especially r/geopolitics. Even if every r/SuicideWatch post really is suicidal and every r/geopolitics post really is non-suicidal—which the paper's validation does not establish—the binary classification task is nearly trivial because the classes differ in topic and style, not just in suicidal content. All models, including LSTMs with random embeddings, would be expected to separate such classes well, so the reported 93% accuracy is not a meaningful measure of suicidal-ideation detection. The paper's limited validation (inter-annotator kappa on 2% of posts and LDA topics on the positive class) does not directly address this confound. The proposed concrete test—retraining on a negative class drawn from mental-health-adjacent subreddits—would calibrate how much of the reported performance stems from topic separation. Since the reader's conditional verdict already demands further validation, I do not recommend a change in the verdict, but the specific conditions should include this test to establish task validity.","tokens_in":10642,"tokens_out":7350,"duration_ms":64925,"concrete_test":"Retrain the top models (RoBERTa, BERT, and the best BERT-LSTM) on a version of the dataset where the negative class is replaced by posts from mental-health-adjacent subreddits such as r/depression and r/Anxiety, keeping the class balance and all other pipeline settings identical. If the accuracy or F1 metrics drop by more than five percentage points relative to Table II, the original results are largely an artifact of topical domain separation rather than suicidal-ideation detection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV.A.1 states that negative posts come from r/socialanxiety, r/TrueOffMyChest, r/bipolar, r/confidence, and r/geopolitics, selected because they are 'not related to suicidal factors at all.' This design makes negative examples topically separable from r/SuicideWatch: a classifier can achieve high accuracy simply by recognizing politics, confidence, or other topics, without detecting any expression of suicidal ideation. The reported ~93% accuracy for RoBERTa (Table II) is therefore not evidence of detecting suicidal ideation; it is evidence of distinguishing subreddit communities. The authors' validation (Sections IV.C.2, IV.C.3) only inspects the suicidal class via LDA topic modeling and measures inter-annotator agreement on a 2% sample; neither step checks whether negative posts are truly free of suicidal ideation or whether positive posts are not just topically different. If the negative class were drawn from mental-health-adjacent subreddits (e.g., r/depression) or from generic social media posts, the gap between classes would shrink and the reported accuracy would likely fall, changing or invalidating the central claim that RoBERTa is the most effective model for suicidal-ideation detection.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a binary text-classification study for detecting suicidal ideation in Reddit posts. The authors collect 37,821 posts from six subreddits over three months, label all 18,566 posts from r/SuicideWatch as suicidal and all posts from the other five subreddits as non-suicidal, and compare five transformer models (BERT, RoBERTa, DistilBERT, ALBERT, ELECTRA) with several LSTM variants using BERT, GloVe, and Word2Vec embeddings. Evaluation is by 5-fold stratified cross-validation on 80% of the data with a held-out 20% test set, and the reported headline result is that RoBERTa achieves 93.22% accuracy and 93.14% F1, with a BERT-embedded LSTM with attention close behind at 92.65%/92.69%. The paper also describes linguistic analysis, LDA topic modeling of the positive class, and a manual annotation check on a 2% sample, and it makes the dataset available on request.","tokens_in":10853,"tokens_out":3440,"duration_ms":31952,"significance":"If the labels were valid and the evaluation protocol were clean, the paper would provide a useful comparative benchmark of transformer and LSTM models for a socially important task, and the training-time observation about DistilBERT would be practical. The paper has concrete strengths: a relatively large dataset, a systematic comparison across several transformer and LSTM architectures, explicit hyperparameter settings for the LSTM models, and a held-out test protocol with majority voting across CV folds. However, the central claim that RoBERTa is the most effective model for detecting suicidal ideation is currently not supported by the evidence as presented, because the label-validity check does not validate the subreddit-derived labels and because the negative class is constructed to be topically distant from the positive class. These are load-bearing issues, not presentation defects.","major_comments":[{"comment":"The label validation is logically incomplete. The paper states that all r/SuicideWatch posts are suicidal and all posts from the other five subreddits are non-suicidal (Section IV.A.2), and then validates this by reporting Cohen's kappa between two annotators on a random 2% sample (Section IV.C.3). This only measures inter-annotator agreement on the sample; it does not measure agreement between the subreddit-derived labels and manual judgments. The appropriate check would be to compare the automatic subreddit-based labels against the manual labels for the same sample and report the confusion matrix or kappa between the two labeling schemes. As written, even near-perfect annotator agreement is compatible with the subreddit-derived labels being wrong for a large fraction of posts. Additionally, the reported 'both Cohen's kappa values of Author 1, 0.897, and Author 2, 0.854' is incoherent: Cohen's kappa is a single pairwise statistic, not a per-annotator score.","section":"IV.A.2 and IV.C.3"},{"comment":"The construction of the negative class makes the classification task substantially easier than suicidal-ideation detection and undermines the reported accuracies. The negative posts come from r/socialanxiety, r/TrueOffMyChest, r/bipolar, r/confidence, and r/geopolitics, which the authors select because they are 'not related to suicidal factors at all' (Section IV.A.1). In particular, r/geopolitics is topically very distant from r/SuicideWatch, so a model can achieve high accuracy by recognizing topic or community rather than by detecting expressions of suicidal ideation. The validation steps in Sections IV.C.2 and IV.C.3 examine only the suicidal class via LDA topic modeling and measure annotator agreement; they do not test whether negative posts are free of suicidal content or whether positive posts are distinguishable from mental-health-adjacent non-suicidal posts. To support the central claim, the authors need to evaluate on a negative class that includes mental-health-adjacent subreddits (e.g., r/depression) or on manually relabeled data, and report how accuracy changes when the topic confound is removed.","section":"IV.A.1 and V.A"},{"comment":"The claim that RoBERTa is the most effective model is not supported with statistical evidence. The accuracy differences among RoBERTa (93.22%), ELECTRA (92.57%), BERT (92.39%), and the BERT LSTM variants (around 92.5-92.7%) are small, but the paper reports no confidence intervals, standard deviations across folds, or significance tests. The phrase 'average cross-validation accuracy' is also ambiguous: it is unclear whether Table II reports averages over the five validation folds, performance on the held-out 20% test set after majority voting, or some other aggregation. The authors should state the evaluation protocol precisely and report variability or significance testing before drawing the comparative conclusion.","section":"Table II and Section V.A"},{"comment":"The topic-modeling analysis does not provide evidence about label quality for the negative class. LDA is run only on r/SuicideWatch posts (Section IV.C.2), so it can at most show that the positive class is thematically coherent around suicide-related language. It cannot detect suicidal ideation in the negative subreddits, nor can it establish that the positive and negative classes are separated by suicidal content rather than by topic or community. This should be acknowledged, or additional analyses of the negative class should be provided.","section":"IV.A.1 and IV.C.2"}],"minor_comments":[{"comment":"The acronym 'SVN' appears in the sentence about early work using Support Vector Machines; this should be 'SVM'.","section":"II (Related Work)"},{"comment":"The text refers to 'GolVe' embeddings in the sentence about the superiority of BERT embeddings; this should be 'GloVe'.","section":"V.B"},{"comment":"The heading and text use 'Cohens’ Kappa' and 'Both Cohen's kappa values'; the phrasing should be corrected to refer to a single pairwise kappa statistic between the two annotators.","section":"IV.C.3"},{"comment":"The sentence 'We divided the whole dataset into 20% to test the models’ overall performance' is awkward and should be rephrased for clarity, for example as 'We held out 20% of the dataset for testing...'.","section":"V (Experiments)"},{"comment":"The hyperparameter description covers the LSTM models but does not report batch size, number of epochs, maximum sequence length, or other training details for the transformer models; providing these would improve reproducibility.","section":"V (Experiments)"},{"comment":"Some references are incomplete or inconsistently formatted; for example, the RoBERTa citation gives only an arXiv identifier and the Ray reference is a general website link. A uniform reference style throughout would improve presentation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core experimental setup is straightforward and the paper is readable, but the validity of the dependent variable is the central issue. In my view this is fixable within the scope of a revision: the authors could re-annotate a substantial sample manually, compare the subreddit-derived labels with manual labels, include a mental-health-adjacent negative class or matched non-suicidal posts, and report variability. Without those changes, the comparative claim is not yet established. I would not reject outright, because the paper's methodology is otherwise standard and the authors have already collected a large dataset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper builds a new 37,821-post Reddit corpus and runs a broad, cleanly described benchmark of five transformers plus LSTM variants with BERT/GloVe/Word2Vec embeddings. That part is useful. But the headline claim — RoBERTa detects suicidal ideation at 93% — doesn't survive reading the design. Labels come straight from subreddit membership, and the negative class is drawn from r/geopolitics, r/confidence, r/socialanxiety, r/bipolar, and r/TrueOffMyChest. A classifier can hit high accuracy just by learning 'is this about politics or confidence vs. SuicideWatch,' without any genuine understanding of suicidal language. The stress-test note is right: the task is close to subreddit classification.\n\nThe validation section is the softest spot. The reported Cohen's kappa of 0.897/0.854 is between two authors manually labeling 2% of the data. That measures inter-annotator agreement, not whether the subreddit-derived labels are correct. The two annotators agree with each other, not with the original labeling. The LDA topic modeling is done only on the suicidal posts, so it can't say anything about whether the negative posts are free of suicidal content. Neither step addresses the actual labeling assumption.\n\nOn the positive side, the experimental setup is standard: 5-fold stratified CV with a 20% held-out test, grid search via Ray-Tune, and consistent hyperparameters across models. The finding that BERT-embedded LSTMs match transformers while GloVe/Word2Vec LSTMs drop 15-25 points is a clean, reproducible result within the paper. The DistilBERT training-time observation is a practical data point.\n\nWhat's missing: error bars. They trained five folds per model; reporting mean accuracy without standard deviation or a significance test makes the RoBERTa-vs-ELECTRA 0.65% gap uninterpretable. And the data is only 'available upon request,' which, with Pushshift shut down, will be hard to reproduce.\n\nNet: this is a useful dataset-plus-benchmark paper with a flawed evaluation design. The absolute accuracies should not be trusted, and the relative comparisons are only meaningful once the negative class is made realistic and the labels are properly validated. I'd send it to review — it deserves referee time — but with the expectation of major revision: resample negatives from mental-health-adjacent or general subreddits, get external annotation or at least show the subreddit labels match manual judgment, report variance, and release the data. As is, I wouldn't cite the accuracy numbers.","headline":"The new corpus is a real resource, but the subreddit-sourced labels and topically distant negative class make the 93% accuracy a measure of topic separation, not suicidal-ideation detection.","tokens_in":11430,"tokens_out":2325,"would_cite":false,"duration_ms":20929,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned RoBERTa is the most effective model for detecting suicidal ideation in Reddit posts, reaching 93.22% average accuracy and a 93.14% F1 score.","keywords":["suicidal ideation detection","Reddit","transformer models","RoBERTa","LSTM","BERT embeddings","text classification","mental health screening"],"falsifier":"Have the two authors (or independent annotators) label the 756 held-out posts and compare their labels directly against the subreddit-derived labels; if agreement is below about 0.8 kappa, or if models trained on subreddit labels perform markedly worse on a fully manually-labeled test set than the reported 93% accuracy, the central claim fails.","tokens_in":10377,"feed_emoji":"🧠","tokens_out":11318,"duration_ms":84458,"temperature":0.7,"pith_summary":"This paper tries to establish that modern deep-learning language models can reliably separate Reddit posts that express suicidal ideation from ordinary posts, with fine-tuned RoBERTa the most accurate of the five transformers tested. On a self-collected dataset of 37,821 posts — every r/SuicideWatch post labeled suicidal, posts from five other subreddits labeled non-suicidal — RoBERTa reaches 93.22% accuracy and 93.14% F1, while all other transformers stay above 91%. The paper also shows that LSTM models using BERT embeddings nearly match the transformers (best at 92.65% accuracy), but the same LSTM architecture with GloVe or Word2Vec embeddings falls far behind, indicating the embedding choice drives performance. If these results hold, automated screening of social media for suicide risk becomes practical.","feed_headline":"RoBERTa detects suicidal Reddit posts at 93% accuracy","feed_subtitle":"RoBERTa wins; BERT-embedded LSTMs trail by under a point, making automated screening feasible.","key_machinery":"The mechanism that carries the argument is a binary text-classification pipeline trained on a dataset whose labels come from subreddit membership: 18,566 posts from r/SuicideWatch are labeled suicidal and 19,255 posts from five other subreddits are labeled non-suicidal. The models — BERT, RoBERTa, DistilBERT, ALBERT, ELECTRA, and LSTM variants with BERT, GloVe, or Word2Vec embeddings — are fine-tuned and compared under a 5-fold stratified cross-validation with majority voting across folds. To defend the labels, the paper uses LDA topic modeling (which extracts suicidal phrasings like 'ending tonight' and 'want die') and a Cohen's-kappa analysis on a 2% random sample manually annotated by the two authors. The load-bearing hinge is that the subreddit-of-origin label is treated as ground truth.","core_discovery":"On the paper's own terms, the central discovery is a performance ranking: fine-tuned RoBERTa is the most effective model, with 93.22% average cross-validation accuracy and 93.14% F1, followed closely by ELECTRA and BERT above 92%, and DistilBERT and ALBERT just above 91%. The second discovery is that a bidirectional LSTM with attention plus BERT embeddings performs nearly as well (92.65% accuracy), whereas identical LSTM models with GloVe or Word2Vec embeddings drop to 77% or lower, and in the Word2Vec case to the mid-50s. The authors take this as evidence that contextual, bidirectional representations are what make suicidal-ideation detection work, and that transformer-based models, in particular, offer a strong foundation for mental-health monitoring tools.","pith_inferences":["If subreddit-membership labels are accurate enough, this study demonstrates a low-cost annotation strategy: entire communities serve as weak labels, which could scale to other mental-health conditions (depression, anxiety) and other platforms.","A natural stress test is to apply the best model to posts from subreddits not in the training set; if accuracy drops sharply, the model may be learning subreddit style rather than suicidal content.","The paper's precision-recall balance (about 91-93% for transformers) means false negatives are roughly as common as false positives in this setting; a deployment would need to decide which error is costlier, something the paper does not address.","Adding user-level features (post history, posting frequency) is a testable extension that could either improve accuracy or reveal that the current model already saturates the information in individual posts."],"forward_implications":["Automated screening is within reach: a fine-tuned RoBERTa flagging at-risk posts could operate at scale on public Reddit data, with precision and recall both around 93%.","The near-tie between RoBERTa and BERT-embedded LSTM suggests that the transformer encoder's contextual embeddings, rather than the fine-tuned classification head, may carry most of the signal.","DistilBERT's half training time with only about 1.3 points lower accuracy makes it the pragmatic choice for organizations with limited compute.","The released dataset gives the community a shared English Reddit benchmark for suicide-ideation detection, enabling direct comparison of future models."],"supporting_citations":[{"why":"Defines BERT, the base model of the five transformer variants fine-tuned in the comparison.","marker":"[19]"},{"why":"Provides the Cohen's kappa scale (0.81-1.00 as almost perfect) used to interpret the 2% manual annotation validation.","marker":"[29]"},{"why":"Defines RoBERTa, the model reported as the most accurate (93.22% accuracy, 93.14% F1).","marker":"[30]"},{"why":"Defines DistilBERT, the distilled variant that achieves competitive accuracy with about half the training time.","marker":"[31]"},{"why":"Defines ALBERT, a lightweight BERT variant evaluated in the model comparison.","marker":"[32]"},{"why":"Defines ELECTRA, the second-strongest transformer in the reported results.","marker":"[33]"},{"why":"Supports the paper's claim that BERT embeddings outperform other word embeddings for text representation.","marker":"[38]"}],"fun_headline_variants":["RoBERTa tops suicide detection on Reddit at 93%","RoBERTa wins, but LSTM with BERT embeddings nearly ties","Contextual embeddings boost both transformers and LSTMs for suicide posts","Suicide ideation posts: RoBERTa leads, BERT-embedded LSTM second","For suicide posts, RoBERTa best; LSTM needs BERT to approach it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole accuracy story depends on the assumption that every r/SuicideWatch post is suicidal and every post from the other five subreddits is not; the manual check on 2% of posts only shows that the two authors label consistently with each other, not that the subreddit-derived labels match the concept.","fun_headline_variants_meta":{"raw":{"variants":["RoBERTa tops suicide detection on Reddit at 93%","RoBERTa wins, but LSTM with BERT embeddings nearly ties","Contextual embeddings boost both transformers and LSTMs for suicide posts","Suicide ideation posts: RoBERTa leads, BERT-embedded LSTM second","For suicide posts, RoBERTa best; LSTM needs BERT to approach it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000376,"raw_usage":{"total_tokens":2008,"prompt_tokens":954,"completion_tokens":1054,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":954}},"tokens_in":570,"tokens_out":1054,"duration_ms":9731,"temperature":1.0,"reasoning_tokens":954,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:20:27.916461+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have the two authors (or independent annotators) label the 756 held-out posts and compare their labels directly against the subreddit-derived labels; if agreement is below about 0.8 kappa, or if models trained on subreddit labels perform markedly worse on a fully manually-labeled test set than the reported 93% accuracy, the central claim fails.","supporting_citations":[],"review_version":1}