Pith. sign in

REVIEW 4 major objections 8 minor 24 references

RSD-15K: A Large-Scale User-Level Annotated Dataset for Suicide Risk Detection on Social Media

T0 review · 4 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read RSD-15K, a 14,613-post user-level Reddit dataset with four-level suicide risk labels and timestamps, is presented as a benchmark on which DeBERTa reaches 76.0% accuracy and 77.0% macro F1.

desk verdict A genuinely useful timestamped user-level suicide-risk dataset, but the quality evidence for 70% of the labels is missing and the 'largest' claim doesn't survive contact with the paper's own comparison table. read the letter →

arxiv 2507.11559 v1 pith:H5EBYUFE submitted 2025-07-14 cs.CY cs.SI

classification cs.CYcs.SI
keywords suicideriskdetectionuser-leveldatasetRedditsocialmediamentalhealthfour-levelannotationtemporalmodelingtextclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces RSD-15K, a dataset of 14,613 posts from 1,265 Reddit users on r/SuicideWatch, each post labeled with one of four suicide risk levels — Indicator, Ideation, Behavior, Attempt — and each user retaining the full posting time sequence. The authors' claim is that this is currently the largest user-level suicide risk assessment dataset and that its combination of fine-grained labels, temporal information, and expert annotation makes it a reliable benchmark for automatic risk detection. On the user-level classification task, DeBERTa reaches 76.0% accuracy and 77.0% macro F1, which the authors read as evidence that the dataset can support automatic assessment. The practical motivation is early detection and intervention: a dataset that preserves how a user's posts evolve over time could let models flag rising risk rather than classify a single post in isolation.

What carries the argument

The object that carries the argument is the dataset itself. RSD-15K is organized at the user level: each of 1,265 users contributes multiple posts, and the user's risk label is the label of the latest post in a time window, so posting order and intervals are part of the prediction signal. The label scheme adapts the Columbia Suicide Severity Rating Scale into four classes: Indicator (no direct personal risk, e.g., discussing third parties), Ideation (suicidal thoughts without action), Behavior (preparatory or self-harm actions), and Attempt (reported previous suicide attempt). The quality-control mechanism combines three trained annotators, a reported Fleiss Kappa of 0.7206 on the 30% of posts labeled by consensus, independent labeling for the remaining 70%, an uncertainty-reporting policy for hard cases, and daily review of randomly selected labels.

What would settle it

Take a random sample of the 10,229 posts that were labeled independently and re-label them with the same three-person voting procedure used for the 30% consensus subset. If the agreement between the re-labeled sample and the released labels is materially lower than the 0.7206 Kappa reported for the consensus portion, the dataset-quality claim would be undermined.

Watch

Extended reading notes

Core claim

The central claim is that RSD-15K is a large-scale, user-level suicide risk assessment dataset that is reliable enough and rich enough to support automatic risk classification. It contains 14,613 posts from 1,265 Reddit users, each post assigned one of four risk labels derived from the Columbia Suicide Severity Rating Scale — Indicator, Ideation, Behavior, Attempt — with posting timestamps preserved so that a user's history can be studied as a sequence. The annotation process used three trained annotators, with reported Fleiss Kappa of 0.7206 on a 30% consensus subset and independent labeling for the remaining 70%, plus uncertainty reporting and daily expert inspection. On the user-level task, DeBERTa reaches 76.0% accuracy and 77.0% macro F1, and the authors also report that a base-sized DeBERTa trained on the 15K dataset outperforms a fully optimized DeBERTa-large trained on 500 posts, which they take as evidence that data scale compensates for model size. The paper concludes that the dataset effectively supports automatic suicide risk assessment and is positioned as a public benchmark resource.

Load-bearing premise

The dataset-quality claim depends on assuming that the 70% of posts labeled independently by single annotators are as reliable as the 30% labeled by three-person consensus, even though the reported agreement score was calculated only on that smaller consensus portion.

Editorial extensions

If this is right

  • RSD-15K gives researchers a single public benchmark with 14,613 posts, 1,265 users, four risk levels, and timestamps, so future suicide-risk models can be compared on the same data.
  • Because the dataset is user-level and time-ordered, it supports models that use posting sequences and intervals, not just single-post text, to estimate a user's current risk level.
  • The reported result that a base-sized DeBERTa on 15K posts outperforms a fully optimized large DeBERTa on 500 posts, if it holds, suggests data scale can offset model capacity in this setting.
  • The feature-importance analysis indicates that time-based signals such as posting intervals and nighttime posting contribute most to risk prediction, which would make timestamps a core part of future data collection.
  • The four-level label scheme allows models to distinguish thinking about suicide, preparing for it, and reporting past attempts, a finer output space than binary risk detection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors do not develop is using the timestamps to model within-user transitions between risk levels, such as whether Ideation posts tend to precede Behavior posts; this would turn the dataset from a classification benchmark into a trajectory resource.
  • Because all posts come from one self-selected crisis community, the vocabulary and label distribution are community-specific, so transfer to general social media or clinical populations would require re-validation.
  • The scale-versus-capacity conclusion rests on a single model comparison, so an external replication with several model families and multiple train/validation/test splits would be needed before treating dataset size as a reliable substitute for model size.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper introduces RSD-15K, a user-level dataset for suicide risk detection on social media, consisting of 14,613 Reddit posts from 1,265 users, labeled into four risk categories (Attempt, Behavior, Ideation, Indicator) adapted from the C-SSRS. The dataset preserves posting timestamps and order, which the authors argue supports temporal modeling of suicide risk. The paper describes the annotation process, reports Fleiss' Kappa on a subset, compares the dataset with prior resources, and presents benchmark results for XGBoost, BiLSTM, HiGRU, RoBERTa, and DeBERTa on a user-level risk classification task. The main claim is that the dataset effectively supports automatic suicide risk assessment, evidenced by DeBERTa reaching 76.0% accuracy and 77.0% macro-F1. The paper also discusses privacy and ethical considerations.

Significance. If the label quality is properly substantiated, RSD-15K would be a valuable community resource: it is among the largest user-level datasets with four-level suicide risk labels, it preserves temporal ordering of user posts, and it is publicly available upon request. The authors are transparent about several annotation procedures, including annotator training thresholds, an uncertainty reporting policy, and a daily inspection rule. The benchmark suite spans traditional models, deep learning models, and large pre-trained language models, providing a useful reference for future work. However, the significance is conditional on resolving the missing quality evidence for the single-annotator majority of the data and on correcting the internal inconsistencies in the benchmark analysis.

major comments (4)
  1. [II-C1 / II-B] The annotation quality of the 70% single-annotator subset is not established. Section II-C1 reports Fleiss' Kappa = 0.7206 only for the 4,384 samples (30%) that were jointly labeled, while Section II-B states that the remaining 10,229 posts (70%) were labeled independently without joint review. No agreement metric, adjudication result, or re-review outcome is reported for that majority. Because the training, validation, and test splits are drawn from the full dataset, the benchmark results in Table III and the dataset-quality claim depend on the unverified labels of these 10,229 posts. Please report an inter-annotator agreement statistic on a random sample of the independent portion, or the daily-inspection pass rates and the number of corrected labels for both annotation portions.
  2. [III-B / Table IV] The comparison across dataset sizes in Table IV is confounded and internally inconsistent. The small-scale condition uses DeBERTa-Large with hyperparameter optimization and data balancing, while the large-scale condition uses DeBERTa-Base without optimization; the model size and the optimization protocol change simultaneously, so the conclusion that 'the increase in data volume has a decisive influence' is not supported. In addition, Table IV reports a macro-F1 of 0.70 for the 15K/Base condition, whereas Table III reports a macro-F1 of 77.0 for DeBERTa on the same task; the discrepancy must be resolved. The authors should compare the same model architecture and protocol at the two dataset sizes, or run an ablation that isolates dataset size.
  3. [III-B] The benchmark results are reported without variance or statistical testing. The statement that 'all models performed stably across multiple experimental runs' is not supported by any reported standard deviation, number of runs, or significance test; without this information, the gap between DeBERTa (76.0% accuracy) and RoBERTa (71.0%) could be within run-to-run noise. Also, model performance stability does not by itself demonstrate annotation quality, so the sentence linking stability to 'high quality data annotation and reliable datasets' is not a valid inference. Please report mean and standard deviation over at least three seeds and, if superiority claims are made, a significance test.
  4. [III-A / III-B] The claimed advantage of the temporal information is not demonstrated. The paper lists the preservation of posting time sequences as a key contribution, but no baseline is included that removes timestamps or post order, or that uses only the latest post as input. Since the user-level label is defined as the risk level of the user's latest post, a model that ignores history could in principle perform similarly; the authors should include an ablation (e.g., feeding only the latest post, or shuffling the order) to show that the temporal structure contributes to the reported performance.
minor comments (8)
  1. [Abstract / Title] The abstract says '15,000 user-level posts' but the dataset contains 14,613 posts; please align the wording or define '15K' as approximate.
  2. [II-B] In the last paragraph of the annotation process, the sentence 'Only when the accuracy of the reviewed data exceeds 85%.' is grammatically incomplete; specify what happens when the threshold is not met.
  3. [Footnote 1] The repository name in the footnote contains a space ('RSD 15K'); please provide the working URL.
  4. [Table IV] The column headers ('Data', 'Model', 'Opt.', '1', 'No') are cryptic; expand them so the experimental conditions are self-explanatory.
  5. [References] Reference [24] (Vabalas et al.) is about validation with limited sample sizes; the claim that large datasets compensate for model capacity would be better supported by a reference on scaling laws or data-centric scaling.
  6. [IV] The statement 'After this anonymization process, there is no way to re-identify users from the data' is an absolute claim; given the known limitations of de-identification, please soften this to a risk statement.
  7. [III-A2] Section III-A2 contains a duplicated passage: the sentence beginning 'This mechanism integrates temporal features...' appears twice.
  8. [II-C1] Section II-C1 describes the 70% as 'labeled separately by three annotators', which is ambiguous because the 30% subset was also labeled by three annotators with voting; clarify that the 70% were labeled independently by one annotator each.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the benchmark results are evaluated on human labels with disjoint train/test users, and no fitted parameter is renamed as a prediction.

full rationale

The paper makes no mathematical derivation; its central claim is that the RSD-15K dataset supports suicide-risk classification, evidenced by benchmark accuracies (DeBERTa 76.0% accuracy and 77.0% macro-F1, Table III). The labels are human annotations, not outputs of the evaluated models. The paper explicitly partitions users into disjoint train/validation/test sets: "we randomly divide all users into training set (80%), validation set (10%), and test set (10%) to ensure that the users from the training set and test set are entirely disjoint to prevent data leakage risks," so test performance is not forced by construction. The label scheme is adapted from the external C-SSRS scale and from prior datasets [3, 6], transparently cited. No parameter is fitted to a subset and then reported as a prediction on that same subset; no uniqueness theorem or ansatz is imported from the authors' own prior work, and the reference list shows no author overlap with the cited prior datasets. The most notable weakness is data-quality related, not circular: Fleiss' Kappa = 0.7206 was computed only on the 30% consensus subset (4,384 samples), while 70% of labels (10,229 posts) were labeled independently by single annotators, so the reliability of the majority portion is less evidenced. That is a validity and ground-truth quality concern, not a circularity of the derivation chain. Accordingly, no circular step can be quoted and no specific reduction exists.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The annotation process and benchmarks rest on several unverified assumptions, most critically that single-annotator labels for 70% of the data are as reliable as the consensus-labeled 30%, and that a latest-post label captures user-level risk. No fitting parameters were used in constructing the dataset, but the hand-chosen thresholds (95%, 85%) and window size (5) are arbitrary.

free parameters (3)
  • daily inspection accuracy threshold = 85%
    Chosen by hand in Section II-B2 as the pass/fail threshold for the 10% daily quality review; affects the claimed quality assurance but no sensitivity analysis is given.
  • annotator training accuracy threshold = 95%
    Chosen by hand in Section II-B2 as the required accuracy on 100 pre-annotation samples before formal labeling begins; impacts annotator selection.
  • time window size = 5 (stable version)
    Section III states 'the stable version has 5 window elements' for the user-level sequential task; the number of recent posts used as context is fixed arbitrarily.
assumptions (5)
  • domain assumption Posts from r/SuicideWatch are a valid representation of users' suicide risk states.
    The dataset is built entirely from that subreddit (Section II-A1); if the community is not representative of at-risk populations, the dataset's utility is limited.
  • domain assumption C-SSRS-derived textual categories (Attempt, Behavior, Ideation, Indicator) can be assigned reliably from posts alone.
    Annotators label text without clinical interviews (Section II-B); the paper assumes the adapted scale is valid for text-only annotation.
  • ad hoc to paper The 30% consensus-labeled subset's agreement (Kappa=0.7206) generalizes to the 70% independently labeled subset.
    Section II-C1 reports Kappa only on the 30% jointly labeled data; no quality evidence is given for the 70% majority, yet benchmarks use the full dataset.
  • domain assumption The label of the user's latest post is a sufficient user-level risk ground truth.
    Section III defines the user label as the latest post's risk level; the paper does not validate this proxy against clinician assessment or longitudinal outcomes.
  • domain assumption Publicly posted Reddit content can be redistributed in anonymized form while making re-identification impossible.
    Section IV asserts that removing identifiers makes re-identification impossible, which is a strong claim not empirically supported for free-text social media data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RSD-15K: A Large-Scale User-Level Annotated Dataset for Suicide Risk Detection on Social Media." pith.science (2026). https://pith.science/paper/H5EBYUFE

@misc{pith2026250711559,
  author       = {Pith},
  title        = {Pith review of: RSD-15K: A Large-Scale User-Level Annotated Dataset for Suicide Risk Detection on Social Media},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H5EBYUFE}},
  note         = {Machine review of arXiv:2507.11559}
}
read the original abstract

In recent years, cognitive and mental health (CMH) disorders have increasingly become an important challenge for global public health, especially the suicide problem caused by multiple factors such as social competition, economic pressure and interpersonal relationships among young and middle-aged people. Social media, as an important platform for individuals to express emotions and seek help, provides the possibility for early detection and intervention of suicide risk. This paper introduces a large-scale dataset containing 15,000 user-level posts. Compared with existing datasets, this dataset retains complete user posting time sequence information, supports modeling the dynamic evolution of suicide risk, and we have also conducted comprehensive and rigorous annotations on these datasets. In the benchmark experiment, we systematically evaluated the performance of traditional machine learning methods, deep learning models, and fine-tuned large language models. The experimental results show that our dataset can effectively support the automatic assessment task of suicide risk. Considering the sensitivity of mental health data, we also discussed the privacy protection and ethical use of the dataset. In addition, we also explored the potential applications of the dataset in mental health testing, clinical psychiatric auxiliary treatment, etc., and provided directional suggestions for future research work.

Figures

Figures reproduced from arXiv: 2507.11559 by the authors.

Figure 1
Figure 1. Distribution of Posts per User. It can be observed that [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Figure3. The data after annotation contains 14,613 valid data [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. Risk Level Distribution for Most Active Users (Top [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 18 canonical work pages

  1. [3]

    Suicide risk level prediction and suicide trigger detection: A bench- mark dataset,

    J. Li, X. Chen, Z. Lin, K. Yang, H. V . Leong, N. X. Yu, and Q. Li, “Suicide risk level prediction and suicide trigger detection: A bench- mark dataset,” HKIE Transactions Hong Kong Institution of Engineers , vol. 29, no. 4, pp. 268–282, 2022

  2. [14]

    Latent Suicide Risk Detection on Microblog via Suicide-Oriented Word Embeddings and Layered Attention

    L. Cao, H. Zhang, L. Feng, Z. Wei, X. Wang, N. Li, and X. He, “Latent suicide risk detection on microblog via suicide-oriented word embeddings and layered attention,” arXiv preprint arXiv:1910.12038 , 2019

  3. [1]

    The clinical sequelae of the covid-19 pandemic: Loneliness, depression, excessive alcohol use, social media addiction, and risk for suicide ideation,

    T. G. Sami Hamdan and G. Zalsman, “The clinical sequelae of the covid-19 pandemic: Loneliness, depression, excessive alcohol use, social media addiction, and risk for suicide ideation,” Archives of Suicide Research, vol. 29, no. 1, pp. 238–251, 2025, pMID: 38756019. [Online]. Available: https://doi.org/10.1080/13811118.2024.2345170

  4. [2]

    Suicidal ideation detection on social media: a review of machine learning methods,

    A. Abdulsalam and A. Alhothali, “Suicidal ideation detection on social media: a review of machine learning methods,” Social Network Analysis and Mining , vol. 14, no. 1, p. 188, September 2024. [Online]. Available: https://doi.org/10.1007/s13278-024-01348-0

  5. [4]

    Reddit api,

    Reddit Inc., “Reddit api,” 2025, accessed: 2024. [Online]. Available: https://www.reddit.com/dev/api/

  6. [5]

    Columbia-suicide severity rating scale (c-ssrs),

    K. Posner, D. Brent, C. Lucas, M. Gould, B. Stanley, G. Brown, P. Fisher, J. Zelazny, A. Burke, M. Oquendo et al. , “Columbia-suicide severity rating scale (c-ssrs),” New York, NY: Columbia University Medical Center, vol. 10, p. 2008, 2008

  7. [6]

    Knowledge-aware assessment of severity of suicide risk for early intervention,

    M. Gaur, A. Alambo, J. P. Sain, U. Kursuncu, K. Thirunarayan, R. Kavuluru, A. Sheth, R. Welton, and J. Pathak, “Knowledge-aware assessment of severity of suicide risk for early intervention,” in The world wide web conference , 2019, pp. 514–525

  8. [7]

    Label Studio: Data labeling software,

    M. Tkachenko, M. Malyuk, A. Holmanyuk, and N. Liubimov, “Label Studio: Data labeling software,” 2020-2025, open source software available from https://github.com/HumanSignal/label-studio. [Online]. Available: https://github.com/HumanSignal/label-studio

Show all 24 references
  1. [8]

    Measuring nominal scale agreement among many raters

    J. L. Fleiss, “Measuring nominal scale agreement among many raters.” Psychological bulletin, vol. 76, no. 5, p. 378, 1971

  2. [9]

    The feeling of knowing: Some metatheoretical implications for consciousness and control,

    A. Koriat, “The feeling of knowing: Some metatheoretical implications for consciousness and control,” Consciousness and cognition , vol. 9, no. 2, pp. 149–171, 2000

  3. [10]

    The impact of evidence reliability on sensitivity and bias in decision confidence

    A. Boldt, V . De Gardelle, and N. Yeung, “The impact of evidence reliability on sensitivity and bias in decision confidence.” Journal of experimental psychology: human perception and performance , vol. 43, no. 8, p. 1520, 2017

  4. [11]

    Confirmation bias: A ubiquitous phenomenon in many guises,

    R. S. Nickerson, “Confirmation bias: A ubiquitous phenomenon in many guises,” Review of general psychology , vol. 2, no. 2, pp. 175–220, 1998

  5. [12]

    Suicide and depression detection,

    N. Komati, “Suicide and depression detection,” https://www.kaggle.com/ datasets/nikhileswarkomati/suicide-watch/data, 2021, accessed: 2025- 02-25

  6. [13]

    Supervised learning for suicidal ideation detection in online user content,

    S. Ji, C. P. Yu, S.-f. Fung, S. Pan, and G. Long, “Supervised learning for suicidal ideation detection in online user content,” Complexity, vol. 2018, no. 1, p. 6157249, 2018

  7. [15]

    # suicidal-a multipronged approach to identify and explore suicidal ideation in twitter,

    P. P. Sinha, R. Mishra, R. Sawhney, D. Mahata, R. R. Shah, and H. Liu, “# suicidal-a multipronged approach to identify and explore suicidal ideation in twitter,” in Proceedings of the 28th ACM international conference on information and knowledge management , 2019, pp. 941– 950

  8. [16]

    Expert, crowdsourced, and machine assessment of suicide risk via online postings,

    H.-C. Shing, S. Nair, A. Zirikly, M. Friedenberg, H. Daum ´e III, and P. Resnik, “Expert, crowdsourced, and machine assessment of suicide risk via online postings,” in Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clin...

  9. [17]

    CLPsych 2019 shared task: Predicting the degree of suicide risk in Reddit posts,

    A. Zirikly, P. Resnik, ¨O. Uzuner, and K. Hollingshead, “CLPsych 2019 shared task: Predicting the degree of suicide risk in Reddit posts,” in Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology , K. Niederhoffer, K. Hollingshead, P. Resnik, R...

  10. [18]

    Characterization of time-variant and time-invariant assessment of suicidality on reddit using c-ssrs,

    M. Gaur, V . Aribandi, A. Alambo, U. Kursuncu, K. Thirunarayan, J. Beich, J. Pathak, and A. Sheth, “Characterization of time-variant and time-invariant assessment of suicidality on reddit using c-ssrs,” PloS one, vol. 16, no. 5, p. e0250448, 2021

  11. [19]

    Depression and suicide risk detection on social media using fasttext embedding and xgboost classifier,

    S. Ghosal and A. Jain, “Depression and suicide risk detection on social media using fasttext embedding and xgboost classifier,” Procedia Com- puter Science , vol. 218, pp. 1631–1639, 2023, international Conference on Machine Learning and Data Engineering. [Online]. Available: ...

  12. [20]

    Sentiment classification on suicide notes using bi-lstm model,

    R. Beniwal and A. Dobhal, “Sentiment classification on suicide notes using bi-lstm model,” in Proceedings of Data Analytics and Manage- ment, A. Swaroop, Z. Polkowski, S. D. Correia, and B. Virdee, Eds. Singapore: Springer Nature Singapore, 2024, pp. 1–10

  13. [21]

    Higru: Hierarchical gated recurrent units for utterance-level emotion recognition,

    W. Jiao, H. Yang, I. King, and M. R. Lyu, “Higru: Hierarchical gated recurrent units for utterance-level emotion recognition,” 2019. [Online]. Available: https://arxiv.org/abs/1904.04446

  14. [22]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” 2019. [Online]. Available: https://arxiv.org/abs/1907.11692

  15. [23]

    Deberta: Decoding- enhanced bert with disentangled attention,

    P. He, X. Liu, J. Gao, and W. Chen, “Deberta: Decoding- enhanced bert with disentangled attention,” 2021. [Online]. Available: https://arxiv.org/abs/2006.03654

  16. [24]

    Machine learning algorithm validation with a limited sample size,

    A. Vabalas, E. Gowen, E. Poliakoff, and A. J. Casson, “Machine learning algorithm validation with a limited sample size,” PLOS ONE, vol. 14, no. 11, pp. 1–20, 11 2019. [Online]. Available: https://doi.org/10.1371/journal.pone.0224365

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.