{"id":"dbdc95c6-a637-41e6-ad75-dff5b71f5db4","arxiv_id":"2509.03530","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A model trained on earlier, non-suicidal forum interactions predicts whether a user will later write a post about suicidal thoughts, reaching 0.73 balanced accuracy on a Dutch youth forum.","lead":"The authors trained a model that reads an anonymous teenager's earlier forum posts and replies to predict whether that teen will later post about suicidal thoughts. On a Dutch youth forum, it reached a balanced accuracy of 0.73, suggesting moderators might spot at-risk users earlier.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"User-level labels are generated by the authors' own detector, so the headline balanced accuracy may be predicting detector artifacts rather than real SIB; the detector's published F1 is measured on a 28%-positive test set, not the low-prevalence full forum.","rationale":"The reader's weakest assumption identifies exactly the load-bearing point: the user-level ground truth is produced by the authors' own detection model, making the evaluation partially self-referential. I agree with this assessment and find no additional concern that is more fundamental. The paper is otherwise well-executed: the architecture is sensible, ablations are informative, and the zero-shot baselines are weaker as expected. But the headline claim -- that SIB can be predicted before explicit disclosure with balanced accuracy 0.73 -- requires that the labels used for training and evaluation correspond to real SIB disclosure. The reported detection F1 is measured on a heavily enriched post-level test set (28% positive), which does not transfer automatically to the full forum's low-prevalence setting. Label noise from base-rate shift can both introduce false-positive users and leave true SIB posts in the input stream, directly undermining the task definition. The Limitations section acknowledges the issue, but the paper does not quantify how detector errors propagate to the 0.73 figure, nor does it provide an external validation set. Thus the correct verdict remains CONDITIONAL: the method is promising, but the central claim is not yet established against true SIB labels. No change to the reader's verdict is needed.","tokens_in":17158,"tokens_out":4643,"duration_ms":45822,"concrete_test":"Draw a stratified random sample of users from the user-level dataset (e.g., 200 predicted-SIB and 200 predicted-No-SIB), have two independent annotators label each user's full history as 'ever disclosed SIB' using the C-SSRS criteria, then recompute EARLY-SIB's balanced accuracy on this expert-labeled subset (or retrain on expert labels). If balanced accuracy drops materially below 0.73 or is not significantly above chance, the central claim is unsupported. A cheaper complementary check: take a random sample of forum posts (e.g., 500), obtain expert SIB labels, and compare against the detector's labels to estimate full-corpus precision and recall; if precision is far below 0.92, the user-level labels are too noisy to support the headline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the user-level SIB labels used for training and evaluation are valid proxies for real suicidal ideation or behavior. Section 3.2 generates these labels by applying the authors' own fine-tuned LLaMA-3-8B detection model to every forum post, and Section 6.1 reports F1 = 0.96 on a post-level test set with 28% SIB prevalence. However, the full-forum prevalence is far lower: only 3.8% of users are labeled SIB, and per-post prevalence is lower still. Under this base-rate shift, the detector's precision and recall on the full corpus are unknown and could be substantially worse than the reported 0.92/0.91. False positives create 'SIB users' who never disclosed; false negatives leave true SIB posts in the input history, partially violating the paper's central no-self-disclosure framing. Because the downstream model is trained and evaluated against these same inferred labels, the 0.73 balanced accuracy is self-referential: it may reflect how well the model predicts the detector's biases rather than future SIB. The Limitations section acknowledges the inference but does not quantify label noise or bound its effect on the headline metric.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces Early-SIB, a transformer-based model to predict whether a user on a Dutch adolescent forum (Kindertelefoon) will write a post containing suicidal ideation or behavior (SIB), using only the user's preceding posts and replies that do not contain SIB. The authors create a post-level annotated dataset (569 SIB, 1,438 No-SIB posts), train a LLaMA-3-8B sequence classifier to detect SIB posts, apply it to the entire forum to pseudo-label users, and then train Early-SIB on the user-level pseudo-labels. They report a balanced accuracy of 0.73 for early prediction, alongside ablation studies and SHAP-based explanations. The paper frames the work as moving suicide prediction earlier than prior methods, which typically rely on explicit disclosure.","tokens_in":17406,"tokens_out":9525,"duration_ms":88318,"significance":"The task and framing are novel, and the engineering is careful: the authors provide code, a restricted-access dataset, a well-designed post-level annotation protocol with high inter-annotator agreement, and a thorough ablation study. The ethical discussion is thoughtful and the deployment concerns (precision vs. intervention) are addressed explicitly. However, the central evaluation is undermined by the fact that the user-level labels are generated by the authors' own post-level detector and no independent validation is provided. If the label-noise issue is resolved, the approach would be a meaningful step toward early detection of suicidal ideation in social media. As it stands, the reported 0.73 balanced accuracy remains an internal assessment of predicting the detector's output, rather than a verified prediction of future SIB.","major_comments":[{"comment":"The user-level target is defined by the authors' own fine-tuned LLaMA-3-8B post-level classifier, which is applied to all forum posts. This creates a self-referential evaluation: the EARLY-SIB model is trained and assessed against labels produced by a sibling model. The post-level confusion matrix (Figure 3, left) shows a false positive rate of 0.03 (row-normalized for actual No-SIB posts). With a low base rate of SIB users (284/7,540 = 3.8%), this per-post FPR implies that a substantial number of users labeled SIB are false positives; for a user with the median 3 pre-SIB posts, the probability of at least one false-positive post label is 1 - (0.97)^3 ≈ 0.087. Moreover, false negatives leave true SIB posts in the input history, violating the paper's central no-self-disclosure assumption. The Limitations section acknowledges that labels were inferred, but it does not quantify the label noise or bound its effect on the 0.73 balanced accuracy, which is a load-bearing claim. I recommend adding an independent user-level validation set (e.g., a random sample of users manually labeled for SIB disclosure) or a sensitivity analysis that simulates plausible label noise.","section":"Section 3.2 / Figure 3"},{"comment":"The post-level detection model's F1 is reported on a test set with 28% SIB prevalence, which is very different from the full-forum distribution. The paper does not report precision and recall of the detection model under the actual base rate, nor does it evaluate the user-level labels against any ground truth. Because the early-prediction model is trained and evaluated on the same pseudo-labels, the 0.73 balanced accuracy may reflect the detector's systematic biases (e.g., over-flagging certain topics) rather than the true signal of future SIB. The authors should either (a) construct a human-annotated user-level test set, or (b) provide a detailed analysis of the detector's performance on a random sample of the full forum, including the user-level false positive and false negative rates. Without this, the central claim is not supported.","section":"Section 5.1 / Table 1"}],"minor_comments":[{"comment":"The rows are all labeled 'BTL' with only numeric values, so it is unclear which components are included or removed in each ablation. Please use checkmarks or explicit labels (e.g., 'No LSTM') for each row.","section":"Table 3"},{"comment":"The phrase 'meaningful addition to traditional methods' is stronger than the evidence supports, given the very low precision (0.10) at the reported operating point in Table 2. Consider tempering the claim or adding an explicit discussion of the precision/base-rate trade-off.","section":"Abstract"},{"comment":"'C-SRSS' is a typo for 'C-SSRS'.","section":"Section 3.1"},{"comment":"The reference for Hadzic et al. in Section 2.3 appears corrupted in the reference list; please fix the author names and formatting.","section":"References"},{"comment":"The confusion matrices are row-normalized; presenting also column-normalized values or absolute counts would help assess the effects of class imbalance.","section":"Figure 3"},{"comment":"The paper states 'weighted F1-score' without defining the weighting; please specify whether it is macro, micro, or support-weighted.","section":"Section 6.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is in a highly sensitive area, and the authors have taken appropriate ethical and legal steps, including a collaboration agreement, ethics approval, and pseudonymization. The main concern is the self-referential label generation, which makes the headline metric unverifiable. If the authors can add an independent user-level validation set or a convincing sensitivity analysis, the paper could become a strong contribution. In the meantime, I recommend major revision rather than rejection, since the issue is load-bearing but potentially addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one for the task framing, not for the headline number. Predicting suicidal ideation before a user explicitly writes about it, using only their earlier forum interactions, is a genuine gap in the CLPsych-style literature. That part is real and worth building on.\n\nWhat the paper does well: clean experimental design, a reasonable architecture (BERTje per post, LSTM over the sequence, a separate title/tag stream), thorough baselines including the Zhang depression model, and a careful ablation. The context-window analysis is useful. The SHAP analysis is a nice addition, and the authors are transparent about the data being a single Dutch adolescent forum. The post-level annotation process is careful, with decent inter-annotator agreement.\n\nThe soft spot is the user-level ground truth. Every user's SIB label comes from the authors' own fine-tuned LLaMA-3 detector applied to all forum posts. That detector's 0.96 F1 is measured on a test set with 28% SIB prevalence; the full forum is much sparser, so precision and recall on the real distribution are unknown. If the detector misses true SIB posts, those posts stay in the input history and leak the outcome into the 'pre-disclosure' window. If it hallucinates false positives, the early model is learning to predict the detector's quirks. So the balanced accuracy of 0.73 is best read as 'we can predict our detector's future labels,' not 'we can predict real SIB.' The authors do acknowledge the inferred labels in the Limitations, but they don't quantify the noise or bound its effect.\n\nThe precision of 0.10 also undercuts the abstract's claim about a 'meaningful addition to traditional methods.' In a population with 4% prevalence, a positive prediction is wrong nine times out of ten. That's not necessarily fatal for a screening tool—you can raise the threshold—but the paper doesn't discuss that trade-off.\n\nI don't think this is a desk reject. The task is novel, the methods are sound, and the failure modes are the kind the field needs to see addressed. A good reviewer at CLPsych or a health-NLP track would push for human-validated user labels or a noise analysis, but the paper deserves that chance.\n\nWould I cite it? Probably not for the headline result, but yes for the task framing and the negative result about zero-shot LLaMA on this task.\n\nFor your reading group: maybe—it's a good case study in how label noise propagates into downstream evaluation.","headline":"Novel task framing and clean experiments, but the 0.73 balanced accuracy is computed against the authors' own detector's labels, so the real-world claim is weaker than it looks.","tokens_in":17947,"tokens_out":5487,"would_cite":true,"duration_ms":45427,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Model predicts teen suicidal ideation from pre-crisis forum posts","keywords":["suicidal ideation prediction","early detection","social media","adolescent mental health","transformer model","Dutch youth forum","balanced accuracy","SHAP explainability"],"falsifier":"Re-run the user-level prediction with expert-annotated user labels on a held-out set of the same forum, and compare balanced accuracy with the 0.73 obtained from model-inferred labels; a large drop would show the reported signal is partly an artifact of the detection model. Alternatively, stratify the detection model's false negatives and check whether the early-prediction signal is concentrated in users whose SIB posts the detector got wrong.","tokens_in":16969,"feed_emoji":"💬","tokens_out":7982,"duration_ms":78050,"temperature":0.7,"pith_summary":"The paper sets out to show that suicidal ideation and behavior (SIB) can be detected in an adolescent's ordinary forum activity before the user ever posts about it explicitly. On a Dutch youth help forum, the EARLY-SIB model reads the posts and replies that precede any SIB disclosure and predicts whether the user will later write a SIB post, achieving balanced accuracy of 0.73. If the finding holds, platforms could flag at-risk adolescents earlier than self-disclosure, catching some of the many young people who never contact mental-health services. The authors frame the system as a support tool for human moderators rather than a clinical diagnosis.","feed_headline":"Model predicts teen suicidal ideation from pre-crisis forum posts","feed_subtitle":"On a Dutch youth forum, a transformer hits balanced accuracy 0.73 using only pre-disclosure posts.","key_machinery":"The central object is EARLY-SIB, a two-stream transformer architecture for whole-history sequence classification. One stream passes each prior interaction (title plus body for posts, body for replies) through BERTje and collects the CLS vectors; a bidirectional LSTM then processes these vectors in order, followed by an attention layer and mean pooling. The second stream concatenates all of a user's titles and tags into one string, encodes it with a second BERTje, and uses the CLS vector; the two streams are combined in a fully connected layer that outputs the future-SIB probability. The architecture matters because more than half of user histories exceed the input limits of standard BERT models, while truncating to the most recent posts hurts performance; a 30-interaction window covers over 90% of users, and the title-and-tag stream carries a large share of the signal.","core_discovery":"The central claim is that future suicidal disclosure is already legible in a user's pre-disclosure language, and a sequence model can read it. EARLY-SIB encodes each prior interaction with a Dutch BERT model, BERTje, runs the resulting per-interaction representations through a bidirectional LSTM with attention, and fuses them with a second BERT stream over the user's post titles and tags before classifying the user as future-SIB or not. On the Kindertelefoon forum it reaches balanced accuracy 0.73 ± 0.02 with recall 0.71 ± 0.07, outperforming the earlier architecture it adapts (0.67 ± 0.04) and clearly beating zero-shot language-model baselines. SHAP analysis indicates the model spreads its attention across many interactions, and the most influential interaction typically occurs within ten days of the first SIB post.","pith_inferences":["A natural test is to rebuild the user-level labels with expert annotation or clinical records; if balanced accuracy drops far below 0.73, the reported performance is partly an artifact of the detection model's own errors rather than a true early-warning signal.","The SHAP result that risk is spread across many interactions suggests lightweight keyword or single-post screeners will underperform sequence models like EARLY-SIB, a concrete architectural constraint for any future deployment.","The same pre-disclosure framing could be applied to other delayed disclosures, such as self-harm, eating disorders, or substance use, where ordinary online activity may precede the first explicit admission.","Since the most predictive interactions cluster in the days just before first SIB disclosure, the practical utility of the approach depends on fast, near-real-time processing; a latency-aware deployment study would test whether the lead time is actionable."],"forward_implications":["If the balanced accuracy transfers to other platforms, moderators of youth forums could be alerted to at-risk users based only on pre-disclosure posts, potentially reaching adolescents who never seek clinical care.","Precision is only about 0.10 on the highly imbalanced data, so most flagged users would be false positives under this model alone; any deployed system would need human review and a clear privacy and stigma protocol.","The model's dependence on titles and tags means self-chosen metadata is a substantial part of the early risk signal, not just the body text.","Because the model receives no SIB content at input, the task directly tests whether warning signs are distributed across ordinary interactions rather than concentrated in a single explicit post."],"supporting_citations":[{"why":"Supplies BERTje, the Dutch BERT model that initializes both the body and title/tag transformers in EARLY-SIB.","marker":"(Vries et al., 2019)"},{"why":"Provides the multi-post architecture that EARLY-SIB adapts and the main comparison baseline.","marker":"Zhang et al. (2024)"},{"why":"LLaMA-3-8B is fine-tuned to label every forum post with SIB/No-SIB, generating the user-level ground truth, and also serves as a zero-shot baseline.","marker":"Touvron et al. (2023)"},{"why":"C-SSRS guidelines define what counts as suicidal ideation and behavior for the post-level annotations.","marker":"(Posner et al., 2010)"},{"why":"SHAP is used to attribute each interaction's contribution to the early-prediction output.","marker":"Lundberg and Lee (2017)"},{"why":"Establishes the failure of risk-factor-based prediction, the motivation for attempting earlier data-driven detection.","marker":"(Franklin et al., 2017)"}],"fun_headline_variants":["AI predicts teen suicidal thoughts from earlier posts","Model reads pre-crisis forum posts to foresee teen suicidal risk","Early-SIB: spotting suicidal ideation before it's spoken","Pre-crisis posts reveal future suicidal ideation in teens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result rests on trusting the machine-generated labels that decide which users count as having future suicidal ideation; if the detection model systematically misses or invents SIB posts, the reported balanced accuracy is not measuring real suicidal ideation, a limitation the authors acknowledge.","fun_headline_variants_meta":{"raw":{"variants":["AI predicts teen suicidal thoughts from earlier posts","Model reads pre-crisis forum posts to foresee teen suicidal risk","Early-SIB: spotting suicidal ideation before it's spoken","Pre-crisis posts reveal future suicidal ideation in teens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000493,"raw_usage":{"total_tokens":2400,"prompt_tokens":905,"completion_tokens":1495,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1429}},"tokens_in":521,"tokens_out":1495,"duration_ms":12560,"temperature":1.0,"reasoning_tokens":1429,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:57:58.296084+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the user-level prediction with expert-annotated user labels on a held-out set of the same forum, and compare balanced accuracy with the 0.73 obtained from model-inferred labels; a large drop would show the reported signal is partly an artifact of the detection model. Alternatively, stratify the detection model's false negatives and check whether the early-prediction signal is concentrated in users whose SIB posts the detector got wrong.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"C-SSRS guidelines define what counts as suicidal ideation and behavior for the post-level annotations."}],"review_version":2}