{"id":"c68ed35d-f3c9-47b0-bdb5-2a025c1c706c","arxiv_id":"2507.00693","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multimodal system combining acoustic, textual, and LLM-extracted features reports 74% accuracy for adolescent suicide risk detection on the SW1 challenge test set.","lead":"This paper describes a suicide risk detection system for the 1st SpeechWellness Challenge, combining acoustic speech models, text models, and large language model extracted behavioral indicators. The system reports 74% accuracy on the challenge test set and says it ranked first.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.740 test accuracy is not shown to be statistically robust: the winning ensemble was chosen on the dev set, and the margin over baseline could be within chance on a ~100-participant test set.","rationale":"The paper has genuine external support: it reports a first-place result on the SW1 challenge's held-out test set, which is stronger evidence than a purely private split. The method is also clearly described and uses standard, reproducible building blocks. The most load-bearing weakness, however, is not the semantic validity of the five DeepSeek-R1 flags per se, but the statistical credibility of the 0.740 point estimate. The authors report several voting combinations, select the best on dev, and then present its test accuracy without any uncertainty quantification. The dev-to-test collapse of the acoustic-only ensemble (0.710 to 0.540) demonstrates that small-sample variability in this dataset is large. A 13-point improvement over a 0.610 baseline on a ~100-item test set is suggestive but not decisive, especially after model selection. If the per-sample predictions were available and McNemar's test showed p ≥ 0.05, the central claim of 'substantially surpassing' the baselines would no longer be supported. The reader identified a related concern in the rationale (missing error bars and dev-based selection) but highlighted the unvalidated LLM indicators as the weakest assumption; I agree that this is a real limitation, yet the accuracy claim is even more directly threatened by the lack of significance testing and selection-bias control. Therefore the verdict should remain conditional pending release of the test predictions or an independent evaluation.","tokens_in":8354,"tokens_out":6182,"duration_ms":72179,"concrete_test":"Obtain the per-participant test-set predictions for combination (1,2,5,6,9,13,16) and for Baseline-bonus, then run McNemar's exact test on the paired disagreement counts and a 10,000-sample bootstrap 95% CI for the accuracy difference. If p ≥ 0.05 or the CI includes 0, the headline margin is not statistically meaningful and the claim of 'substantially surpassing' should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the third voting combination reaches 0.740 test accuracy and 'substantially surpasses' the baselines—rests on one point estimate from a small held-out test set. Table 3 shows the acoustic-only ensemble (1,2,5,6,8,9) reached 0.710 on dev but only 0.540 on test, a 17-point collapse indicating high variance and dev-set overfitting. The headline combination (1,2,5,6,9,13,16) was the best of three on dev (0.770), and no confidence interval, bootstrap, or paired significance test is reported for the test result. With a test set of roughly 100 participants, the 13-point gap over Baseline-bonus (0.610) has an approximate standard error of 6–7 points, so the observed margin is around 1.8–2 standard errors; after selecting among multiple ensembles on dev, the effective significance is weaker. The unvalidated DeepSeek-R1 indicators make this worse: the interpretable-feature branch is fit and weighted using the same dev labels, so its contribution to the 0.740 point estimate cannot currently be separated from selection bias. Unless per-sample test predictions are released, the claim of substantial superiority is not statistically established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes a system for the 1st SpeechWellness Challenge (SW1) that combines acoustic embeddings from Hubert, Wav2Vec2, and Whisper; text embeddings from BERT and XLM-RoBERTa; and five interpretable indicators extracted from emotional-regulation transcripts by DeepSeek-R1. Each branch is classified separately (MLP for embeddings, Random Forest for LLM indicators), and a weighted voting ensemble yields 0.740 accuracy and F1 on the held-out test set, ranked first in the challenge. The authors claim this substantially surpasses the challenge baselines.","tokens_in":8742,"tokens_out":3170,"duration_ms":33823,"significance":"As a challenge-system paper, the work is of moderate significance. Its positive aspects are the external held-out test benchmark, the rule-based prompt design grounded in cited psychology literature, and the explicit limitation statement about the MINI-KID assessment framework. However, the headline result rests on a single point estimate without statistical inference, and the LLM-extracted features lack validation, which limits the strength of the claims. The interpretability angle is useful but does not offset these verification gaps.","major_comments":[{"comment":"The central claim that the third voting combination 'substantially surpasses' the baselines rests on a single point estimate on the held-out test set. The test set contains roughly 100 participants (Section 2), and no confidence interval, bootstrap, or paired significance test is reported. The winning combination (1,2,5,6,9,13,16) was selected as the best of three on the dev set (0.770 dev vs 0.768 for the alternative), and the acoustic-only ensemble (1,2,5,6,8,9) drops from 0.710 dev to 0.540 test, indicating high variance and dev-set overfitting. With an approximate standard error of 4 to 5 points on the test accuracy, the 13-point margin over Baseline-bonus is about two to three standard errors, but after selection among multiple ensembles on dev, the effective significance is weaker. Please provide confidence intervals, bootstrap results, or per-sample test predictions so that the superiority claim can be evaluated properly.","section":"Table 3 and Section 4.2"},{"comment":"The five DeepSeek-R1 indicators (self-harm, pressure, social support, unhealthy outlets, exercise) are extracted from ER transcripts with no reported manual validation, inter-annotator agreement, or error analysis. Since the voting weight of the DeepSeek-R1 branch was increased and the ensemble was chosen on the dev set, the contribution of these unvalidated indicators to the 0.740 test accuracy cannot be separated from selection bias. Please provide a manual evaluation on a random sample of transcripts with agreement metrics (e.g., Cohen's kappa) or, at minimum, a detailed error analysis showing that the extracted flags are reliable.","section":"Section 3.4 and Figure 2"},{"comment":"The paper reports the final test accuracy without any measure of variance, and it does not state the number of test participants used in Table 3. This makes it impossible to assess whether the 0.740 result is statistically distinct from the 0.610 Baseline-bonus. Please report the test set size, the exact voting rule (including how ties are broken), and a significance test between the proposed ensemble and the baselines on the same test partition.","section":"Section 4.1 and 4.2"}],"minor_comments":[{"comment":"The word 'Sementic' should be 'Semantic'.","section":"Section 3.3 heading"},{"comment":"The heading contains a typo: 'V oting-Based' should be 'Voting-Based'.","section":"Table 3 heading"},{"comment":"The dataset section describes only training and development sets; the test set used in Table 3 is never described, including its size and whether it was provided by the challenge organizers.","section":"Section 2"},{"comment":"The prompt instruction says '1{3 examples'; this appears to be a typo for '1-3 examples'.","section":"Figure 2 prompt text"},{"comment":"The manuscript does not mention whether the DeepSeek-R1 API is accessed deterministically (e.g., temperature setting), which is relevant for reproducibility of the indicator extraction.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper reads like a short challenge description rather than a full journal article. The core result may be valid, but the lack of statistical inference, code/data release, and validation of the LLM-based features makes it unsuitable for acceptance in its current form. The authors should be encouraged to strengthen the evidence base or reposition the paper as a challenge report with clearly softened claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a decent challenge write-up. The new idea that earns its keep is the DeepSeek-R1 prompt that extracts five interpretable binary indicators (self-harm, pressure, social support, unhealthy outlets, exercise) from the ER transcripts, complete with short verbatim quotes, and feeds them through a random forest into a voting ensemble with acoustic and text embeddings. The held-out test result (0.740 accuracy and F1, first in SW1) is a concrete external evaluation, and the authors deserve credit for being explicit in the conclusion that the MINI-KID labels measure current risk, not future suicidal behavior.\n\nThe prompt design is genuinely thoughtful, grounded in the psychology literature on protective and risk factors, and the example output in Figure 2 shows a real interpretability mechanism. That is the paper's genuine contribution.\n\nThe soft spots are statistical, and they are not minor. The test set is on the order of 100 participants. The winning ensemble was chosen as the best of three on dev; the purely acoustic ensemble (1,2,5,6,8,9) went from 0.710 dev to 0.540 test, a 17-point collapse that tells you how noisy this small sample is. The headline 0.740 has no confidence interval, no significance test, and no per-sample predictions, so we cannot tell whether the margin over Baseline-bonus (0.610) is real or within chance. A rough standard error of 6-7 points puts the 13-point gap at about 1.8-2 SEs; after dev-set selection, the effective strength is weaker. On top of that, the five LLM indicators themselves are unvalidated—no manual scoring, no inter-annotator agreement, no error analysis—so the voting advantage attributed to the interpretable branch could in principle be noise that happens to line up with the dev labels.\n\nNone of this kills the idea. The ensemble approach is reasonable for a challenge, and the interpretable LLM feature extraction is a useful contribution to a field that badly needs transparency. But the claim of 'substantially surpassing' the baselines is not statistically established by the evidence in the paper. Anyone building on this should treat the 0.740 as a promising point estimate, not a proven result.\n\nThe audience is people working on speech-based mental health screening and challenge participants. Worth a serious referee, because the external evaluation and the interpretability angle deserve engagement. A reviewer should ask for error bars or per-sample outputs, and for some validation of the LLM extraction step.","headline":"A sensible challenge system with a genuinely interpretable LLM feature-extraction idea, but the headline 0.740 accuracy is a single point estimate from a small test set and is not statistically robust.","tokens_in":9107,"tokens_out":2795,"would_cite":false,"duration_ms":31462,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A voting ensemble that adds LLM-extracted interpretable indicators to acoustic and text models achieves 74 percent accuracy and F1 for adolescent suicide-risk detection.","keywords":["suicide risk detection","spontaneous speech","large language model","multimodal fusion","interpretability","voting ensemble","adolescent mental health","DeepSeek-R1"],"falsifier":"Replace the five DeepSeek-R1 flags with random binary flags matched to the same base rates, keep everything else fixed, and re-run the voting ensemble; if test accuracy stays near 0.740, the LLM features are not carrying the signal the paper attributes to them.","tokens_in":8167,"feed_emoji":"🗣️","tokens_out":8826,"duration_ms":99117,"temperature":0.7,"pith_summary":"The paper tries to establish that a multimodal speech-based system can detect current adolescent suicide risk more accurately than the challenge baselines by adding large-language-model features to conventional acoustic and text models. It reports that a weighted voting ensemble of three speech encoders, one text encoder, and five interpretable indicators extracted by DeepSeek-R1 from an emotion-regulation transcript reaches 74 percent accuracy and F1 on the test set, the top result in the SW1 challenge. This matters because speech can be collected outside clinical settings, and the LLM component returns short verbatim quotes that let a clinician see which risk factors triggered a label.","feed_headline":"Adding LLM clues lifts suicide-risk detection to 74%","feed_subtitle":"The weighted ensemble beats both challenge baselines by fusing acoustic, text, and interpretable LLM flags.","key_machinery":"The load-bearing mechanism is a structured prompt that asks DeepSeek-R1 to mark five binary indicators in each transcript of the Emotional Regulation task, an open-ended answer about managing extreme distress, namely self-harm behavior, pressure, social support, unhealthy outlets, and exercise, and to quote the supporting text for each. A Random Forest classifies those flags, and its score enters a weighted voting ensemble alongside MLP classifiers trained on HuBERT, Wav2Vec2, Whisper, BERT, and RoBERTa embeddings. The prompt's requirement of verbatim translated quotes is what makes the semantic signal interpretable rather than a black-box score.","core_discovery":"On the paper's own terms, the central discovery is that interpretable, LLM-extracted semantic indicators, fused by voting with acoustic and textual predictions, supply information that neither modality alone provides. The winning combination weights outputs from three speech encoders, a multilingual text encoder, and a Random Forest over five flags extracted by DeepSeek-R1 from the Emotional Regulation task, and it reaches 0.740 accuracy and 0.740 F1 on the held-out test set, compared with 0.510 and 0.610 accuracy for the two challenge baselines. The LLM indicator pipeline alone reaches 0.652 accuracy and 0.679 F1, outperforming every single acoustic and text model reported. The authors read this as evidence that transparent, semantically grounded features can carry much of the predictive signal in suicide-risk screening.","pith_inferences":["Nothing in the paper pins down how much of the 0.740 comes from the five LLM flags versus the random forest weighting; ablating each indicator and reweighting the vote would be the direct test.","The five indicators were chosen from prior correlational findings, so the same prompting recipe could be extended to other protective factors; a neighbouring experiment would be to compare prompt-derived flags against human-rated labels on the same transcripts.","Because the conclusion explicitly restricts itself to the MINI-KID current-risk label, a longitudinal follow-up would be needed to know whether a 0.740 decision reflects a stable trait or a momentary state; the paper itself flags this boundary.","The transcripts are Chinese with English-quoted evidence, so the system's portability to other languages is an open question that a cross-lingual replication would settle."],"forward_implications":["An audio-only voting ensemble overfits the development set (0.710 dev, 0.540 test), while adding RoBERTa and the LLM flags raises test accuracy to 0.740, so the interpretable text channel is what stabilises generalization.","DeepSeek-R1's five flags extracted from one ER transcript already beat every single acoustic and text model, so a transcript-only LLM pipeline is a viable lightweight screener.","Because the ensemble returns verbatim transcript snippets for each flagged indicator, a clinician can trace the basis of a high-risk decision instead of receiving an opaque score.","The method ranks first in the SW1 challenge, so on this dataset the combination of pretrained speech encoders, a multilingual text encoder, and LLM-based feature extraction is a workable recipe for spontaneous-speech risk screening."],"supporting_citations":[{"why":"Supplies DeepSeek-R1, the LLM that extracts the five interpretable suicide-risk indicators from ER transcripts.","marker":"[26]"},{"why":"Provides the SW1 dataset, the challenge baselines, and the held-out test set on which the 0.740 accuracy is reported.","marker":"[20]"},{"why":"Earlier LLM-based suicide risk detection that the paper contrasts as lacking interpretability.","marker":"[18]"},{"why":"Supports the claim that fusing multiple modalities improves mental-health diagnostics and motivates the voting design.","marker":"[19]"},{"why":"Establishes social support as a protective factor, justifying one of the five prompt indicators.","marker":"[27, 28, 29]"},{"why":"Establishes exercise as a protective factor, justifying another of the five prompt indicators.","marker":"[30, 31]"},{"why":"Supplies Wav2Vec2-large-xlsr-53, one of the acoustic encoders in the winning voting combination.","marker":"[21]"},{"why":"Supplies Whisper-large-v3, used both for ASR transcription and as one of the three acoustic encoders.","marker":"[22]"},{"why":"Supplies Hubert-large-ls960-ft, one of the acoustic encoders in the winning voting combination.","marker":"[23]"},{"why":"Supplies XLM-RoBERTa-base, the text encoder whose predictions are combined with the LLM flags in the winning vote.","marker":"[25]"}],"fun_headline_variants":["LLM semantic clues push suicide-risk detection to 74% accuracy","First-place suicide-risk detection via LLM and acoustic fusion","LLM features alone beat acoustic models in suicide-risk screening","Interpretable LLM flags drive suicide-risk speech screening to 74%","Weighted LLM ensemble wins suicide-risk challenge at 74% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole voting advantage rests on the five DeepSeek-R1 flags being accurate, but the paper reports no human validation or inter-annotator agreement on those flags; if the flags are wrong or driven by the prompt, the gain over baselines could be an accident of noise.","fun_headline_variants_meta":{"raw":{"variants":["LLM semantic clues push suicide-risk detection to 74% accuracy","First-place suicide-risk detection via LLM and acoustic fusion","LLM features alone beat acoustic models in suicide-risk screening","Interpretable LLM flags drive suicide-risk speech screening to 74%","Weighted LLM ensemble wins suicide-risk challenge at 74% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1374,"prompt_tokens":834,"completion_tokens":540,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":451}},"tokens_in":450,"tokens_out":540,"duration_ms":5551,"temperature":1.0,"reasoning_tokens":451,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:08:26.702968+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the five DeepSeek-R1 flags with random binary flags matched to the same base rates, keep everything else fixed, and re-run the voting ensemble; if test accuracy stays near 0.740, the LLM features are not carrying the signal the paper attributes to them.","supporting_citations":[{"cited_title":"Multimodal sensing for depression risk detection: integrating audio, video, and text data,","cited_arxiv_id":null,"evidence_quote":"Supplies DeepSeek-R1, the LLM that extracts the five interpretable suicide-risk indicators from ER transcripts."},{"cited_title":"Validating machine learning algorithms for twitter data against established measures of suicidality,","cited_arxiv_id":null,"evidence_quote":"Provides the SW1 dataset, the challenge baselines, and the held-out test set on which the 0.740 accuracy is reported."},{"cited_title":"Specific coping behaviors in relation to adolescent depression and suicidal ideation,","cited_arxiv_id":null,"evidence_quote":"Earlier LLM-based suicide risk detection that the paper contrasts as lacking interpretability."},{"cited_title":"Using large language models to detect depression from user-generated diary text data as a novel approach in digital mental health screening: Instrument validation study,","cited_arxiv_id":null,"evidence_quote":"Supplies Wav2Vec2-large-xlsr-53, one of the acoustic encoders in the winning voting combination."},{"cited_title":"Linguistic Features Extracted by GPT-4 Improve Alzheimer's Disease Detection based on Spontaneous Speech","cited_arxiv_id":"2412.15772","evidence_quote":"Supplies Whisper-large-v3, used both for ASR transcription and as one of the three acoustic encoders."},{"cited_title":"Suicidal intention de- tection in tweets using bert-based transformers,","cited_arxiv_id":null,"evidence_quote":"Supplies Hubert-large-ls960-ft, one of the acoustic encoders in the winning voting combination."},{"cited_title":"Spontaneous Speech-Based Suicide Risk Detection Using Whisper and Large Language Models","cited_arxiv_id":"2406.03882","evidence_quote":"Supplies XLM-RoBERTa-base, the text encoder whose predictions are combined with the LLM flags in the winning vote."}],"review_version":1}