REVIEW 4 major objections 6 minor 10 references
The methodology of Constructing the Large-Scale Dataset for Detecting Presuicidal and Anti-Suicidal Signals in Social Media Texts in Russian
T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read A documented annotation pipeline produced a large-scale Russian dataset that separates presuicidal from anti-suicidal signals in social-media text — 57,810 public examples.
desk verdict Useful new Russian dataset for suicide-risk signals, but the fine-grained label reliability is not actually demonstrated—only ternary signal direction is verified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the annotation protocol: a 45-class signal taxonomy inside a documented workflow. Presuicidal classes number 33, grouped into seven categories (clinical manifestations, destructive behavior, family problems, external-world problems, death, feelings, and other); 12 anti-suicidal classes describe protective factors such as love, positive connections, help-seeking, and positive dynamics. The protocol's moving parts: an iteratively revised instruction with class tables; development rounds that produced 28 guideline revisions; soft majority voting, the aggregation variant that raised label coverage from 71% to 93% at equal agreement; control sets of 9.2% of each block,
What would settle it
Take 1,000 posts at random from the released training set, have fresh annotators label them using only the published instructions, and measure their agreement with the released labels. If that agreement is no better than the original annotators' mutual agreement (alpha around 0.4–0.5), the ground truth is too noisy to sustain the reported classifier gains, and the model-guided correction stage is the prime suspect.
Extended reading notes
Core claim
The paper claims that a deliberately engineered annotation pipeline can yield a large-scale, multi-label Russian dataset for suicide-risk signals in social-media text, and demonstrates this with 57,810 examples: 38,406 presuicidal, 9,702 anti-suicidal, and 9,702 irrelevant. The taxonomy — 33 presuicidal classes in seven groups plus 12 anti-suicidal protective-factor classes — was synthesized from suicidology literature and prior work, and the pipeline adds instruction-validation rounds, soft majority voting for multi-label texts, parallel control-set verification at a 15% mismatch threshold, and a reannotation pass guided by dataset cartography and model errors. The authors are explicit abou
Load-bearing premise
Everything rests on the assumption that the annotation instructions, soft-voting aggregation, and model-guided reannotation yield ground truth reliable enough to train on, despite the paper's own figures: an inter-annotator agreement score of 0.542 on the test set and an acknowledged high level of task subjectivity.
Editorial extensions
If this is right
- A public fine-grained Russian suicide-risk resource now exists, so volunteer services can train classifiers that flag at-risk posts instead of reading feeds manually.
- Granularity is the dominant performance lever: three-way direction labels support about 0.70 F1 and binary relevant/irrelevant 0.71, while 26-class presuicidal classification falls to 0.53 — so users can choose the label level that matches their tolerance for noise.
- Because the presuicidal taxonomy is intentionally atomic, other groups can remap it onto their own theoretical models or adapt it to other languages without redoing the synthesis step.
- The documented failure modes — third-person texts, overlapping lexical triggers, and the irrelevant class absorbing subjective signals — give future dataset builders a concrete checklist of what to control during annotation.
Reading between the lines
- If the released labels carry the agreement level the paper reports (alpha ≈ 0.54), every classifier trained on them inherits that noise; the reported F1 scores should be read as performance against a noisy ground truth, not as the models' true detection ceiling.
- The correction stage is the riskiest step: examples were reannotated precisely where models and annotators disagreed, which can conceal the real labeling difficulty; an independent fresh-annotator replication would reveal whether the corrected labels are actually more reliable.
- The entanglement of the irrelevant class with subjective feeling classes is likely not language-specific, so the same taxonomy, applied to other languages, should be expected to produce the same confusion — and the same need for a correction stage.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a methodology for constructing a Russian-language social-media dataset for detecting presuicidal and anti-suicidal signals. It describes the development of a fine-grained class taxonomy (33 presuicidal classes in 7 groups; 12 anti-suicidal classes), annotator recruitment and training, sampling schemes, a five-iteration annotation workflow, control-set verification, and a post-hoc reannotation stage guided by dataset cartography and model-error analysis. The resulting dataset contains 57,810 annotated texts, with 38,406 presuicidal, 9,702 anti-suicidal, and 9,702 irrelevant examples. The authors train RuBERT classifiers at several label granularities and report F1 scores from 0.53 to 0.71. The paper claims to provide a detailed, replicable methodology and a large-scale public resource.
Significance. If the fine-grained labels are reliable, this would be the first large Russian multi-label dataset for suicide-risk signals, with a class taxonomy grounded in suicidology literature and direct relevance to volunteer-based rescue efforts. The manuscript has notable strengths: the dataset, code, and annotation materials are publicly available; the authors give concrete attention to annotator well-being; they acknowledge and analyze subjectivity; and they use MASI for multi-label inter-annotator agreement. The significance, however, depends on evidence that the fine-grained labels are trustworthy. As the manuscript currently stands, the reliability evidence covers only ternary signal direction and one moderate alpha value for a test subset, which is insufficient to support the central claim of a fine-grained multi-label resource.
major comments (4)
- [§3.6, p. 199] Verification compares primary and control annotations only by signal direction (anti-suicidal / pre-suicidal / irrelevant), and the 15% acceptance threshold is defined at that coarse level. Since the dataset is advertised as fine-grained multilabel (33 presuicidal and 12 anti-suicidal classes), a block can pass verification while a substantial fraction of exact class labels are wrong. The only class-level reliability number reported is final Krippendorff alpha 0.542 for the presuicidal test part (§4), which is moderate and does not cover the training set or the reannotated anti-suicidal set. Please report per-class agreement or restrict the dataset's claim to ternary direction.
- [§3.7] The post-hoc reannotation selects examples using dataset cartography (hard-to-learn region x<0.4, y<0.4 in Fig. 1) and model-error analysis, and then checks reannotators every 500 examples with a 50-example batch at a 15% error threshold. This creates a loop in which model predictions influence what becomes 'gold' data; no independent audit demonstrates that reannotation removes noise rather than aligning labels with model preferences. I ask for before/after agreement statistics and a manual audit of corrected labels to establish that reannotation actually improves label quality.
- [§5, Table 6] The evaluation removes all multiclass examples (26.1% of the dataset) and classes with fewer than 100 examples, and the paper states that 'the results might be biased' because irrelevant examples are shared. Without error bars or significance tests, the reported F1 values (0.53 exact presuicidal, 0.63 anti-suicidal) cannot substantiate the claim of 'promising performance'. Since the dataset's unique contribution is its fine-grained multi-label taxonomy, please report metrics on the full multilabel setting (e.g., exact match, micro/macro F1) and include confidence intervals.
- [§3.2, §4] Development inter-annotator agreement is low (alpha 0.39–0.45), and the choice of soft majority voting is based on only 30 examples per method (91% agreement vs. 91% for standard voting, but 93% vs. 71% coverage). No confidence intervals are given. The final alpha 0.542 is reported only for the presuicidal test part; alpha for the anti-suicidal set and for the training/validation parts is omitted. Please provide agreement statistics for all released subsets and justify the aggregation choice with a larger audit.
minor comments (6)
- [§3.4] The standard sampling criteria are not operationalized: 'balanced text length distribution' has no binning or tolerance, and 'presence of first-person pronouns' is binary without handling of indirect speech. Please specify parameters.
- [§4, Tables 4–5] The phrase 'counter distribution' should be 'count distribution'. Also, class names differ between Table 3 and §6.5.1 (e.g., 'Feelings/mental emptiness, depression, longing, sadness' vs. 'Feelings/Emotional suffering, emptiness, melancholy, sadness'); please unify terminology.
- [§5, Table 6] The presuicidal 'exact' model has 26 classes in Table 6, but §6.5.1 lists 33 classes. Clarify exactly which classes were removed and how this affects the comparison.
- [§3.1] The sentence 'The parsers available on the Github' is ungrammatical; reference [6] also lacks full repository metadata. The paper needs a careful language edit throughout.
- [§6.3, principle 4] The instruction prohibits interpreting texts ('You must not attempt to interpret texts in any way'), yet the paper repeatedly emphasizes subjectivity and provides examples where annotators interpreted figurative language differently. This tension should be addressed with explicit guidance on metaphor, sarcasm, and indirect expression.
- [References] Several references are incomplete or nonstandard, e.g., [22], [23], [25] lack full bibliographic details, and [29] points to a nonstandard HF mirror (hf.global-rail.com). Please provide canonical citations.
Circularity Check
No significant circularity: the dataset is human-annotated, and model-guided steps are sampling/reannotation aids rather than fitted predictions.
full rationale
After walking the claimed derivation chain, I find no step where a reported result reduces by construction to its inputs. The paper's central contribution is a human-annotated dataset; the class taxonomy is transparently sourced from prior suicidology literature ([24], [25]) and from the authors' own earlier guidelines [18], but that prior work is used as a provenance/history statement, not as a load-bearing proof of correctness. The model-in-the-loop procedures in Sections 3.6 and 3.7 use confusion matrices and dataset cartography only to select examples for human reannotation and to motivate a revised anti-suicidal class table; final labels are still assigned by annotators and ML specialists, so the dataset is not defined by model outputs. The only use of the word 'predictions' is the optional sampling criterion 'Predictions from a suicidal signal detection model [29]' in Section 3.4, which influences sample composition but does not determine ground-truth labels and is never reported as a predictive result. Reported inter-annotator agreement (e.g., final Krippendorff alpha 0.542) is a quality measure, not a fitted parameter renamed as a prediction. The paper's own admissions of low alpha and subjectivity, and the fact that final verification checks only ternary signal direction, are validity/quality limitations, not circularity. No equation equates an output to an input, and no uniqueness claim is imported from the authors' prior work. The derivation is therefore self-contained with respect to the dataset-construction claim.
Assumptions & free parameters
free parameters (5)
- Acceptable mismatch threshold =
15%
- Control set proportion =
9.2% (185/2000)
- Hard-to-learn region threshold =
x < 0.4 and y < 0.4
- Class frequency cutoff =
100 examples
- Soft majority vote threshold =
at least 2 votes
assumptions (4)
- domain assumption Presuicidal and anti-suicidal signals can be reliably identified in short social media texts by human annotators following instructions.
- domain assumption The class taxonomy synthesized from literature [18,24,25] is complete and non-overlapping enough for annotation.
- domain assumption The used datasets and social media sources are representative of Russian-speaking individuals at risk.
- standard math Krippendorff's alpha with MASI is a valid measure of agreement for multi-label annotation.
Cite this review
Pith. "Pith review of The methodology of Constructing the Large-Scale Dataset for Detecting Presuicidal and Anti-Suicidal Signals in Social Media Texts in Russian." pith.science (2026). https://pith.science/paper/DHKGSSXI
@misc{pith2026260800497,
author = {Pith},
title = {Pith review of: The methodology of Constructing the Large-Scale Dataset for Detecting Presuicidal and Anti-Suicidal Signals in Social Media Texts in Russian},
year = {2026},
howpublished = {\url{https://pith.science/paper/DHKGSSXI}},
note = {Machine review of arXiv:2608.00497}
}
read the original abstract
The suicide is a terrifying act of a person who is misled by his own mental state. This problem arises across many countries. Sadly, Russia also has quite high number of persons who committed suicide. Luckily, a subset of these people writes their struggles in social media, allowing a way to find them and help. However, these valuable texts disappearing in many irrelevant texts which is considerably slowing down the decision process about person's suicidal risk. To tackle this problem, in this work we have presented a detailed methodology of building the dataset for detecting texts that describe presuicidal and anti-suicidal signals. This methodology describes the process of instruction and class table creation, the process of annotation, verification and post-annotation correction. Guiding by this methodology, we collect and annotate a large-scale Russian dataset with more than 50 thousand texts from social media. We provide a count statistic of the dataset as well as common problems in annotation. We also conduct basic experiments of building the classification models to show the on go performance on different levels of annotation. Furthermore, we make the dataset, code and all materials publicly available.
Reference graph
Works this paper leans on
-
[1]
[2]. Suicide and its prevention in Russia, 2019: general facts // Demoscope URL: https://www.demoscope.ru/weekly/2020/0869/suicide.php (accessed: 18.05.2025). [3]. Bollen J. et al. Historical language records reveal a surge of cognitive distortions in recent decades. Proc Natl Acad Sci USA,
work page 2019
-
[2]
Kitoboy // Github URL: https://github.com/psytechlab/kitoboy (accessed: 18.05.2025)
[6]. Kitoboy // Github URL: https://github.com/psytechlab/kitoboy (accessed: 18.05.2025). [7]. Glen Coppersmith et al. From ADHD to SAD: Analyzing the Language of Mental Health on Twitter through Self-Reported Diagnoses // Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, Denv...
work page 2025
-
[29]
Computational Linguistics and Intellectual Technologies
[16]. Stankevich M., Smirnov I. et al. Predicting Depression from Essays in Russian // Proceedings of “Computational Linguistics and Intellectual Technologies” DIALOGUE, 2019, pp. 637-647. [17]. Литвинова Т.А., Литвинова О.А. Языковые особенности русскоязычных текстов лиц, совершивших суицид, и лиц с высоким риском аутоагрессивного поведения // Studia Hum...
work page 2019
-
[2001]
272 с. / Tatabatina N. V. A workshop on the psychology of post -traumatic stress. 1 edition, SPb.: Piter, 2001, 272 p. Buyanov I.O., Yaskova D.V., Serenko D.S., Shkereda D.N., Yaskov A.D., Sochenkov I.V. The methodology of constructing the large -scale dataset for detecting presuicidal and anti-suicidal signals in social media texts in Russian. Trudy ISP ...
work page 2001
-
[2009]
[29]. Astromis Presuicidal RuBERT // Astromis HF URL: https://hf.global- rail.com/astromis/presuisidal_rubert (accessed: 18.05.2025). [30]. RuBERT-Tiny2 Russian Emotion Detection // Hugging Face URL: https://huggingface.co/Djacon/rubert- tiny2-russian-emotion-detection (accessed: 18.05.2025). [31]. Blanchefort RuBERT Base Cased Sentiment // Blanchefort HF...
work page 2025
-
[2018]
[35]. Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics // ArXiv URL: https://arxiv.org/abs/2009.10795 (accessed: 18.05.2025). [36]. Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language // ArXiv URL: https://arxiv.org/abs/1905.07213 (accessed: 18.05.2025). [37]. RoBERTa: A Robustly Optimized BERT Pretra...
arXiv 2009
-
[2019]
Research interests: natural language processing, named entity recognition in specific domains, text augmentation methods. Данил Сергеевич СЕРЕНКО является студентом кафедры математического моделирования и искусственного интеллекта РУДН имени Патриса Лумумбы, научным сотрудником Федерального исследовательского центра "Информатика и управление" Российской а...
work page 2025
-
[2022]
No. 21, pp. 81-92. [19]. VK // VK URL: https://vk.com/ (accessed: 18.05.2025). [20]. X (Twitter) // X URL: https://x.com/ (accessed: 18.05.2025). [21]. Suicide Forum // Suicide Forum URL: http://www.suicide-forum.com/ (accessed: 18.05.2025). [22]. A. Aluoja, J. Shlik, V. Vasar, K. Luuk, M. Leinsalu , The Emotional Well -being Questionnaire (EEK)
work page 2025
Show all 10 references
-
[2025]
pp. 191-210. 208 [24]. Пакулина С.А. Психодиагностика суицидального поведения детей и подростков. 1 изд., Челябинск: 2014 / Pakulina S. A. Psychodiagnostics of suicidal behavior in children and adolescents. 1 edition , Chelabinsk:
2014
-
[9822]
Sean MacAvaney et al
[11]. Sean MacAvaney et al. Community -level Research on Suicidality Prediction in a Secure Environment: Overview of the CLPsych 2021 Shared Task // Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology. Online, 2021, pp. 70-80. [12]. Reading...
2021 arXiv
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.