REVIEW 4 major objections 8 minor 24 references
RSD-15K: A Large-Scale User-Level Annotated Dataset for Suicide Risk Detection on Social Media
T0 review · 4 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read RSD-15K, a 14,613-post user-level Reddit dataset with four-level suicide risk labels and timestamps, is presented as a benchmark on which DeBERTa reaches 76.0% accuracy and 77.0% macro F1.
desk verdict A genuinely useful timestamped user-level suicide-risk dataset, but the quality evidence for 70% of the labels is missing and the 'largest' claim doesn't survive contact with the paper's own comparison table. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The object that carries the argument is the dataset itself. RSD-15K is organized at the user level: each of 1,265 users contributes multiple posts, and the user's risk label is the label of the latest post in a time window, so posting order and intervals are part of the prediction signal. The label scheme adapts the Columbia Suicide Severity Rating Scale into four classes: Indicator (no direct personal risk, e.g., discussing third parties), Ideation (suicidal thoughts without action), Behavior (preparatory or self-harm actions), and Attempt (reported previous suicide attempt). The quality-control mechanism combines three trained annotators, a reported Fleiss Kappa of 0.7206 on the 30% of posts labeled by consensus, independent labeling for the remaining 70%, an uncertainty-reporting policy for hard cases, and daily review of randomly selected labels.
What would settle it
Take a random sample of the 10,229 posts that were labeled independently and re-label them with the same three-person voting procedure used for the 30% consensus subset. If the agreement between the re-labeled sample and the released labels is materially lower than the 0.7206 Kappa reported for the consensus portion, the dataset-quality claim would be undermined.
Extended reading notes
Core claim
The central claim is that RSD-15K is a large-scale, user-level suicide risk assessment dataset that is reliable enough and rich enough to support automatic risk classification. It contains 14,613 posts from 1,265 Reddit users, each post assigned one of four risk labels derived from the Columbia Suicide Severity Rating Scale — Indicator, Ideation, Behavior, Attempt — with posting timestamps preserved so that a user's history can be studied as a sequence. The annotation process used three trained annotators, with reported Fleiss Kappa of 0.7206 on a 30% consensus subset and independent labeling for the remaining 70%, plus uncertainty reporting and daily expert inspection. On the user-level task, DeBERTa reaches 76.0% accuracy and 77.0% macro F1, and the authors also report that a base-sized DeBERTa trained on the 15K dataset outperforms a fully optimized DeBERTa-large trained on 500 posts, which they take as evidence that data scale compensates for model size. The paper concludes that the dataset effectively supports automatic suicide risk assessment and is positioned as a public benchmark resource.
Load-bearing premise
The dataset-quality claim depends on assuming that the 70% of posts labeled independently by single annotators are as reliable as the 30% labeled by three-person consensus, even though the reported agreement score was calculated only on that smaller consensus portion.
Editorial extensions
If this is right
- RSD-15K gives researchers a single public benchmark with 14,613 posts, 1,265 users, four risk levels, and timestamps, so future suicide-risk models can be compared on the same data.
- Because the dataset is user-level and time-ordered, it supports models that use posting sequences and intervals, not just single-post text, to estimate a user's current risk level.
- The reported result that a base-sized DeBERTa on 15K posts outperforms a fully optimized large DeBERTa on 500 posts, if it holds, suggests data scale can offset model capacity in this setting.
- The feature-importance analysis indicates that time-based signals such as posting intervals and nighttime posting contribute most to risk prediction, which would make timestamps a core part of future data collection.
- The four-level label scheme allows models to distinguish thinking about suicide, preparing for it, and reporting past attempts, a finer output space than binary risk detection.
Reading between the lines
- A natural extension the authors do not develop is using the timestamps to model within-user transitions between risk levels, such as whether Ideation posts tend to precede Behavior posts; this would turn the dataset from a classification benchmark into a trajectory resource.
- Because all posts come from one self-selected crisis community, the vocabulary and label distribution are community-specific, so transfer to general social media or clinical populations would require re-validation.
- The scale-versus-capacity conclusion rests on a single model comparison, so an external replication with several model families and multiple train/validation/test splits would be needed before treating dataset size as a reliable substitute for model size.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces RSD-15K, a user-level dataset for suicide risk detection on social media, consisting of 14,613 Reddit posts from 1,265 users, labeled into four risk categories (Attempt, Behavior, Ideation, Indicator) adapted from the C-SSRS. The dataset preserves posting timestamps and order, which the authors argue supports temporal modeling of suicide risk. The paper describes the annotation process, reports Fleiss' Kappa on a subset, compares the dataset with prior resources, and presents benchmark results for XGBoost, BiLSTM, HiGRU, RoBERTa, and DeBERTa on a user-level risk classification task. The main claim is that the dataset effectively supports automatic suicide risk assessment, evidenced by DeBERTa reaching 76.0% accuracy and 77.0% macro-F1. The paper also discusses privacy and ethical considerations.
Significance. If the label quality is properly substantiated, RSD-15K would be a valuable community resource: it is among the largest user-level datasets with four-level suicide risk labels, it preserves temporal ordering of user posts, and it is publicly available upon request. The authors are transparent about several annotation procedures, including annotator training thresholds, an uncertainty reporting policy, and a daily inspection rule. The benchmark suite spans traditional models, deep learning models, and large pre-trained language models, providing a useful reference for future work. However, the significance is conditional on resolving the missing quality evidence for the single-annotator majority of the data and on correcting the internal inconsistencies in the benchmark analysis.
major comments (4)
- [II-C1 / II-B] The annotation quality of the 70% single-annotator subset is not established. Section II-C1 reports Fleiss' Kappa = 0.7206 only for the 4,384 samples (30%) that were jointly labeled, while Section II-B states that the remaining 10,229 posts (70%) were labeled independently without joint review. No agreement metric, adjudication result, or re-review outcome is reported for that majority. Because the training, validation, and test splits are drawn from the full dataset, the benchmark results in Table III and the dataset-quality claim depend on the unverified labels of these 10,229 posts. Please report an inter-annotator agreement statistic on a random sample of the independent portion, or the daily-inspection pass rates and the number of corrected labels for both annotation portions.
- [III-B / Table IV] The comparison across dataset sizes in Table IV is confounded and internally inconsistent. The small-scale condition uses DeBERTa-Large with hyperparameter optimization and data balancing, while the large-scale condition uses DeBERTa-Base without optimization; the model size and the optimization protocol change simultaneously, so the conclusion that 'the increase in data volume has a decisive influence' is not supported. In addition, Table IV reports a macro-F1 of 0.70 for the 15K/Base condition, whereas Table III reports a macro-F1 of 77.0 for DeBERTa on the same task; the discrepancy must be resolved. The authors should compare the same model architecture and protocol at the two dataset sizes, or run an ablation that isolates dataset size.
- [III-B] The benchmark results are reported without variance or statistical testing. The statement that 'all models performed stably across multiple experimental runs' is not supported by any reported standard deviation, number of runs, or significance test; without this information, the gap between DeBERTa (76.0% accuracy) and RoBERTa (71.0%) could be within run-to-run noise. Also, model performance stability does not by itself demonstrate annotation quality, so the sentence linking stability to 'high quality data annotation and reliable datasets' is not a valid inference. Please report mean and standard deviation over at least three seeds and, if superiority claims are made, a significance test.
- [III-A / III-B] The claimed advantage of the temporal information is not demonstrated. The paper lists the preservation of posting time sequences as a key contribution, but no baseline is included that removes timestamps or post order, or that uses only the latest post as input. Since the user-level label is defined as the risk level of the user's latest post, a model that ignores history could in principle perform similarly; the authors should include an ablation (e.g., feeding only the latest post, or shuffling the order) to show that the temporal structure contributes to the reported performance.
minor comments (8)
- [Abstract / Title] The abstract says '15,000 user-level posts' but the dataset contains 14,613 posts; please align the wording or define '15K' as approximate.
- [II-B] In the last paragraph of the annotation process, the sentence 'Only when the accuracy of the reviewed data exceeds 85%.' is grammatically incomplete; specify what happens when the threshold is not met.
- [Footnote 1] The repository name in the footnote contains a space ('RSD 15K'); please provide the working URL.
- [Table IV] The column headers ('Data', 'Model', 'Opt.', '1', 'No') are cryptic; expand them so the experimental conditions are self-explanatory.
- [References] Reference [24] (Vabalas et al.) is about validation with limited sample sizes; the claim that large datasets compensate for model capacity would be better supported by a reference on scaling laws or data-centric scaling.
- [IV] The statement 'After this anonymization process, there is no way to re-identify users from the data' is an absolute claim; given the known limitations of de-identification, please soften this to a risk statement.
- [III-A2] Section III-A2 contains a duplicated passage: the sentence beginning 'This mechanism integrates temporal features...' appears twice.
- [II-C1] Section II-C1 describes the 70% as 'labeled separately by three annotators', which is ambiguous because the 30% subset was also labeled by three annotators with voting; clarify that the 70% were labeled independently by one annotator each.
Circularity Check
No circularity found: the benchmark results are evaluated on human labels with disjoint train/test users, and no fitted parameter is renamed as a prediction.
full rationale
The paper makes no mathematical derivation; its central claim is that the RSD-15K dataset supports suicide-risk classification, evidenced by benchmark accuracies (DeBERTa 76.0% accuracy and 77.0% macro-F1, Table III). The labels are human annotations, not outputs of the evaluated models. The paper explicitly partitions users into disjoint train/validation/test sets: "we randomly divide all users into training set (80%), validation set (10%), and test set (10%) to ensure that the users from the training set and test set are entirely disjoint to prevent data leakage risks," so test performance is not forced by construction. The label scheme is adapted from the external C-SSRS scale and from prior datasets [3, 6], transparently cited. No parameter is fitted to a subset and then reported as a prediction on that same subset; no uniqueness theorem or ansatz is imported from the authors' own prior work, and the reference list shows no author overlap with the cited prior datasets. The most notable weakness is data-quality related, not circular: Fleiss' Kappa = 0.7206 was computed only on the 30% consensus subset (4,384 samples), while 70% of labels (10,229 posts) were labeled independently by single annotators, so the reliability of the majority portion is less evidenced. That is a validity and ground-truth quality concern, not a circularity of the derivation chain. Accordingly, no circular step can be quoted and no specific reduction exists.
Assumptions & free parameters
free parameters (3)
- daily inspection accuracy threshold =
85%
- annotator training accuracy threshold =
95%
- time window size =
5 (stable version)
assumptions (5)
- domain assumption Posts from r/SuicideWatch are a valid representation of users' suicide risk states.
- domain assumption C-SSRS-derived textual categories (Attempt, Behavior, Ideation, Indicator) can be assigned reliably from posts alone.
- ad hoc to paper The 30% consensus-labeled subset's agreement (Kappa=0.7206) generalizes to the 70% independently labeled subset.
- domain assumption The label of the user's latest post is a sufficient user-level risk ground truth.
- domain assumption Publicly posted Reddit content can be redistributed in anonymized form while making re-identification impossible.
Cite this review
Pith. "Pith review of RSD-15K: A Large-Scale User-Level Annotated Dataset for Suicide Risk Detection on Social Media." pith.science (2026). https://pith.science/paper/H5EBYUFE
@misc{pith2026250711559,
author = {Pith},
title = {Pith review of: RSD-15K: A Large-Scale User-Level Annotated Dataset for Suicide Risk Detection on Social Media},
year = {2026},
howpublished = {\url{https://pith.science/paper/H5EBYUFE}},
note = {Machine review of arXiv:2507.11559}
}
read the original abstract
In recent years, cognitive and mental health (CMH) disorders have increasingly become an important challenge for global public health, especially the suicide problem caused by multiple factors such as social competition, economic pressure and interpersonal relationships among young and middle-aged people. Social media, as an important platform for individuals to express emotions and seek help, provides the possibility for early detection and intervention of suicide risk. This paper introduces a large-scale dataset containing 15,000 user-level posts. Compared with existing datasets, this dataset retains complete user posting time sequence information, supports modeling the dynamic evolution of suicide risk, and we have also conducted comprehensive and rigorous annotations on these datasets. In the benchmark experiment, we systematically evaluated the performance of traditional machine learning methods, deep learning models, and fine-tuned large language models. The experimental results show that our dataset can effectively support the automatic assessment task of suicide risk. Considering the sensitivity of mental health data, we also discussed the privacy protection and ethical use of the dataset. In addition, we also explored the potential applications of the dataset in mental health testing, clinical psychiatric auxiliary treatment, etc., and provided directional suggestions for future research work.
Figures
Reference graph
Works this paper leans on
-
[3]
Suicide risk level prediction and suicide trigger detection: A bench- mark dataset,
J. Li, X. Chen, Z. Lin, K. Yang, H. V . Leong, N. X. Yu, and Q. Li, “Suicide risk level prediction and suicide trigger detection: A bench- mark dataset,” HKIE Transactions Hong Kong Institution of Engineers , vol. 29, no. 4, pp. 268–282, 2022
work page 2022
-
[14]
L. Cao, H. Zhang, L. Feng, Z. Wei, X. Wang, N. Li, and X. He, “Latent suicide risk detection on microblog via suicide-oriented word embeddings and layered attention,” arXiv preprint arXiv:1910.12038 , 2019
work page Pith review arXiv 1910
-
[1]
T. G. Sami Hamdan and G. Zalsman, “The clinical sequelae of the covid-19 pandemic: Loneliness, depression, excessive alcohol use, social media addiction, and risk for suicide ideation,” Archives of Suicide Research, vol. 29, no. 1, pp. 238–251, 2025, pMID: 38756019. [Online]. Available: https://doi.org/10.1080/13811118.2024.2345170
-
[2]
Suicidal ideation detection on social media: a review of machine learning methods,
A. Abdulsalam and A. Alhothali, “Suicidal ideation detection on social media: a review of machine learning methods,” Social Network Analysis and Mining , vol. 14, no. 1, p. 188, September 2024. [Online]. Available: https://doi.org/10.1007/s13278-024-01348-0
-
[4]
Reddit Inc., “Reddit api,” 2025, accessed: 2024. [Online]. Available: https://www.reddit.com/dev/api/
work page 2025
-
[5]
Columbia-suicide severity rating scale (c-ssrs),
K. Posner, D. Brent, C. Lucas, M. Gould, B. Stanley, G. Brown, P. Fisher, J. Zelazny, A. Burke, M. Oquendo et al. , “Columbia-suicide severity rating scale (c-ssrs),” New York, NY: Columbia University Medical Center, vol. 10, p. 2008, 2008
work page 2008
-
[6]
Knowledge-aware assessment of severity of suicide risk for early intervention,
M. Gaur, A. Alambo, J. P. Sain, U. Kursuncu, K. Thirunarayan, R. Kavuluru, A. Sheth, R. Welton, and J. Pathak, “Knowledge-aware assessment of severity of suicide risk for early intervention,” in The world wide web conference , 2019, pp. 514–525
work page 2019
-
[7]
Label Studio: Data labeling software,
M. Tkachenko, M. Malyuk, A. Holmanyuk, and N. Liubimov, “Label Studio: Data labeling software,” 2020-2025, open source software available from https://github.com/HumanSignal/label-studio. [Online]. Available: https://github.com/HumanSignal/label-studio
2020
Show all 24 references
-
[8]
Measuring nominal scale agreement among many raters
J. L. Fleiss, “Measuring nominal scale agreement among many raters.” Psychological bulletin, vol. 76, no. 5, p. 378, 1971
1971
-
[9]
The feeling of knowing: Some metatheoretical implications for consciousness and control,
A. Koriat, “The feeling of knowing: Some metatheoretical implications for consciousness and control,” Consciousness and cognition , vol. 9, no. 2, pp. 149–171, 2000
2000
-
[10]
The impact of evidence reliability on sensitivity and bias in decision confidence
A. Boldt, V . De Gardelle, and N. Yeung, “The impact of evidence reliability on sensitivity and bias in decision confidence.” Journal of experimental psychology: human perception and performance , vol. 43, no. 8, p. 1520, 2017
2017
-
[11]
Confirmation bias: A ubiquitous phenomenon in many guises,
R. S. Nickerson, “Confirmation bias: A ubiquitous phenomenon in many guises,” Review of general psychology , vol. 2, no. 2, pp. 175–220, 1998
1998
-
[12]
Suicide and depression detection,
N. Komati, “Suicide and depression detection,” https://www.kaggle.com/ datasets/nikhileswarkomati/suicide-watch/data, 2021, accessed: 2025- 02-25
2021
-
[13]
Supervised learning for suicidal ideation detection in online user content,
S. Ji, C. P. Yu, S.-f. Fung, S. Pan, and G. Long, “Supervised learning for suicidal ideation detection in online user content,” Complexity, vol. 2018, no. 1, p. 6157249, 2018
2018
-
[15]
# suicidal-a multipronged approach to identify and explore suicidal ideation in twitter,
P. P. Sinha, R. Mishra, R. Sawhney, D. Mahata, R. R. Shah, and H. Liu, “# suicidal-a multipronged approach to identify and explore suicidal ideation in twitter,” in Proceedings of the 28th ACM international conference on information and knowledge management , 2019, pp. 941– 950
2019
-
[16]
Expert, crowdsourced, and machine assessment of suicide risk via online postings,
H.-C. Shing, S. Nair, A. Zirikly, M. Friedenberg, H. Daum ´e III, and P. Resnik, “Expert, crowdsourced, and machine assessment of suicide risk via online postings,” in Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clin...
2018
-
[17]
CLPsych 2019 shared task: Predicting the degree of suicide risk in Reddit posts,
A. Zirikly, P. Resnik, ¨O. Uzuner, and K. Hollingshead, “CLPsych 2019 shared task: Predicting the degree of suicide risk in Reddit posts,” in Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology , K. Niederhoffer, K. Hollingshead, P. Resnik, R...
2019
-
[18]
Characterization of time-variant and time-invariant assessment of suicidality on reddit using c-ssrs,
M. Gaur, V . Aribandi, A. Alambo, U. Kursuncu, K. Thirunarayan, J. Beich, J. Pathak, and A. Sheth, “Characterization of time-variant and time-invariant assessment of suicidality on reddit using c-ssrs,” PloS one, vol. 16, no. 5, p. e0250448, 2021
2021
-
[19]
Depression and suicide risk detection on social media using fasttext embedding and xgboost classifier,
S. Ghosal and A. Jain, “Depression and suicide risk detection on social media using fasttext embedding and xgboost classifier,” Procedia Com- puter Science , vol. 218, pp. 1631–1639, 2023, international Conference on Machine Learning and Data Engineering. [Online]. Available: ...
2023
-
[20]
Sentiment classification on suicide notes using bi-lstm model,
R. Beniwal and A. Dobhal, “Sentiment classification on suicide notes using bi-lstm model,” in Proceedings of Data Analytics and Manage- ment, A. Swaroop, Z. Polkowski, S. D. Correia, and B. Virdee, Eds. Singapore: Springer Nature Singapore, 2024, pp. 1–10
2024
-
[21]
Higru: Hierarchical gated recurrent units for utterance-level emotion recognition,
W. Jiao, H. Yang, I. King, and M. R. Lyu, “Higru: Hierarchical gated recurrent units for utterance-level emotion recognition,” 2019. [Online]. Available: https://arxiv.org/abs/1904.04446
2019 arXiv
-
[22]
Roberta: A robustly optimized bert pretraining approach,
Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” 2019. [Online]. Available: https://arxiv.org/abs/1907.11692
2019 arXiv
-
[23]
Deberta: Decoding- enhanced bert with disentangled attention,
P. He, X. Liu, J. Gao, and W. Chen, “Deberta: Decoding- enhanced bert with disentangled attention,” 2021. [Online]. Available: https://arxiv.org/abs/2006.03654
2021 arXiv
-
[24]
Machine learning algorithm validation with a limited sample size,
A. Vabalas, E. Gowen, E. Poliakoff, and A. J. Casson, “Machine learning algorithm validation with a limited sample size,” PLOS ONE, vol. 14, no. 11, pp. 1–20, 11 2019. [Online]. Available: https://doi.org/10.1371/journal.pone.0224365
2019 doi
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.