{"id":"3da1e5b1-a26e-4b57-963d-87dd86ae450c","arxiv_id":"2412.00057","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A machine learning model trained on voice audio alone is reported to classify mental-health helpline call priority as high or low with 92% balanced accuracy across 459 calls.","lead":"This paper reports 92% balanced accuracy for a machine learning model that predicts high or low priority of mental health helpline calls from callers' voices. The result is promising, but the paper does not describe the model or data processing well enough for anyone to verify it.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4 and Table 7 give contradictory class distributions (209 vs 212 high-priority; 250 vs 247 low-priority), making the confusion matrix and the 92% balanced accuracy unverifiable.","rationale":"The reader's weakest assumption focuses on the clinical validity of the triage labels as ground truth. While that is a relevant limitation, the more immediately load-bearing concern is that the paper's own numerical records are internally inconsistent: the train/test split in Table 4 implies 209 high-priority and 250 low-priority calls, while Table 7 reports 212 and 247 respectively. Since the confusion matrix and the headline 92% balanced accuracy are built from the test counts in Table 4, an unresolved discrepancy means the primary result cannot be verified from the manuscript. This is a concrete, checkable problem, not a matter of interpretation. If the authors can reconcile the counts, the central claim may stand subject to further scrutiny; if not, the claim is unsupported. I therefore agree with the REJECT verdict but for a different, more specific reason than the reader's label-validity concern.","tokens_in":10150,"tokens_out":13466,"duration_ms":117043,"concrete_test":"Request from the authors the raw per-call triage labels (A-G) and the exact train/test partition, or the counts per triage category in train and test. Recompute Table 4 and Table 7. If the high-priority test count is not 38 or the total high-priority count is not 212, the confusion matrix and the 92% balanced accuracy are not supported by the reported data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of 92% balanced accuracy rests on the test-set confusion matrix (Table 5), which uses 38 high-priority and 44 low-priority test calls as reported in Table 4. However, these counts do not reconcile with the demographic breakdown in Table 7. Table 4 splits 171 high + 38 high = 209 high-priority calls and 206 low + 44 low = 250 low-priority calls. Table 7 reports A-D total = 212 high-priority calls and E-G total = 247 low-priority calls. Both sum to 459, but the class distributions differ by 3 per class. This means at least one table misreports the data. If Table 4's split is wrong, the confusion matrix and all derived metrics (sensitivity 92%, specificity 91%, balanced accuracy 92%) may not correspond to any real test set. The manuscript also contains unfilled placeholders (Section 3.4: 'XX males', 'XX females') and no methodological details, but the Table 4/Table 7 mismatch is a specific, checkable internal inconsistency that directly undermines trust in the headline number. The authors must supply per-category train/test counts to resolve this.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a feasibility study in which machine learning is applied to audio recordings of 459 calls to an Australian mental health helpline. Clinician-assigned UK Mental Health Triage Scale codes (A–G) are dichotomized into high priority (A–D) and low priority (E–G), and a hierarchical deep neural network classifier is trained to predict the binary priority from the caller's voice. On a held-out test set of 82 calls, the authors report a confusion matrix with 35/38 high-priority and 40/44 low-priority calls correctly classified, yielding a balanced accuracy of 92%, sensitivity 92%, specificity 91%, and precision 90%. The paper also describes a prototype application interface for real-time decision support. The central claim is that voice-only deep learning can accurately estimate mental health triage priority, but the manuscript lacks the methodological detail needed to verify or reproduce the result.","tokens_in":10294,"tokens_out":5112,"duration_ms":45742,"significance":"If the reported result were fully supported, the paper would provide a useful feasibility signal for voice-based triage support in mental health helplines, a domain where prior work is sparse. The authors deserve credit for obtaining ethics approval, using manually de-identified real helpline recordings, and reporting a held-out test set rather than retraining on the test labels; the confusion matrix and most derived metrics are internally computable. However, the significance is currently undermined by two load-bearing problems: the model is described only as a 'hierarchical classifier using deep learning neural networks' with no architecture, input representation, training, or validation details, and the reported data counts in Tables 4 and 7 are mutually inconsistent. With no confidence intervals, no repeated trials, and incomplete demographic reporting, the headline 92% figure cannot currently be credited as a reproducible or well-calibrated estimate.","major_comments":[{"comment":"The model is described only as 'a hierarchical classifier using deep learning neural networks' with no architecture, input representation, audio front-end, segmentation scheme, training procedure, validation strategy, or hyperparameters. The central claim of 92% balanced accuracy therefore cannot be independently assessed, checked for leakage, or reproduced; this is a load-bearing reporting gap.","section":"§3.3"},{"comment":"Table 4 counts 171+38=209 high-priority and 206+44=250 low-priority calls, while Table 7 reports 212 high-priority and 247 low-priority calls; both tables sum to 459, so at least one table misreports the data. Since Table 4 defines the test set underlying the confusion matrix, this 3-call per-class discrepancy makes the test-set composition and all derived metrics unverifiable. The authors should provide per-triage-category train/test counts and reconcile the tables.","section":"§3.1, Tables 4 and 7"},{"comment":"The results section contains unfilled placeholders ('XX males', 'XX females', 'mean age of XX'), so the demographic table is incomplete and the representativeness of the sample cannot be evaluated. This is an incomplete-reporting issue that must be fixed before the results can be interpreted.","section":"§3.4"},{"comment":"No confidence intervals or repeated-run variability are reported for a single test set of only 82 calls; for example, sensitivity of 35/38 has a wide 95% confidence interval, so reporting '92%' as a point estimate conveys false precision. In addition, the reported F1 score of 91.2% does not match the confusion matrix, which gives 2*35/(2*35+3+4)=90.9%.","section":"§3.4, Table 6"},{"comment":"The ground-truth labels are clinician-assigned triage codes made during the same calls whose audio is used as model input, so the model may be learning correlates of clinician judgment rather than objective risk. The Discussion's claim that the tool is 'free from the biases that often influence human decisions' is not supported by the evidence; the authors should report inter-rater reliability of the triage labels or validate the model against independent clinical outcomes.","section":"§3.1, Table 2"}],"minor_comments":[{"comment":"There is a typo in 'They key idea of this paper' that should read 'The key idea'.","section":"§3.1"},{"comment":"The header 'T rain Samples' should read 'Train Samples'.","section":"Table 4"},{"comment":"The sentence 'Most studies use audio transcription, with only studies using audio and text separately' is grammatically unclear and should be rewritten.","section":"§2.3"},{"comment":"Section 3.5 describes an application that updates priority and confidence 'throughout the call', while Section 4 states the analysis 'did not provide this in real time to the listener'; this contradiction should be resolved.","section":"§3.4 / §4"},{"comment":"The statement 'there have been no previous studies conducted on mental health helpline data' is asserted without a systematic literature search and should be either supported or softened.","section":"§3.4"},{"comment":"The UK Mental Health Triage Scale is attributed to Sands et al. [2001], but the cited reference appears to describe an interactive computer program for triage competence; the citation should be verified.","section":"Table 2 citation"}],"recommendation":"reject","confidential_remarks":"The manuscript is not ready for review in its current form: the core methodology is unspecified, the demographic results contain placeholders, and the train/test counts do not reconcile with the category-level totals. If the authors can supply a full model description, correct the tables, and add uncertainty quantification, a resubmission could be considered, but the current version does not meet the bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuine first application of audio-only deep learning to mental-health helpline triage priority, on a real 459-call dataset, but as written the central result is not checkable. The methods section names the model only as \"a hierarchical classifier using deep learning neural networks\" — no architecture, audio representation, training/validation split procedure, hyperparameters, or inference details. There are no confidence intervals and no baselines. I can't tell whether 92% is stable or a lucky draw on 82 test calls.\n\nWhat it does well: the problem is real and worth attacking. The authors identify a gap in the literature — emergency-call severity prediction largely uses transcripts, while audio-only approaches are rare — and they place their work against relevant prior art, with Abi Kanaan et al. 2023 as the closest comparison. They also acknowledge the ethical and clinical limits of a decision aid and describe a prototype that keeps a human in the loop. The labeling scheme, UK Mental Health Triage Scale collapsed into high/low, is clearly documented.\n\nSoft spots, in order of severity. First, the data tables don't reconcile. Table 4 gives 171+38=209 high-priority and 206+44=250 low-priority calls; Table 7 reports 212 and 247. Both sum to 459, but the class split differs by three in each direction. Since the confusion matrix and all derived metrics rely on the 38/44 test split, at least one table is wrong, and the headline number cannot be trusted until this is fixed. Second, the methods are a black box. With no architecture or training details, no code/data release, and no cross-validation or repeated runs, the result is not reproducible. Third, the demographics section has unfilled \"XX\" placeholders, which suggests the manuscript was posted before it was ready. Fourth, the label construct deserves scrutiny: the model learns to predict clinicians' triage codes from the same calls, so it may be capturing correlates of clinician judgment rather than objective risk. That is not fatal for a decision-support tool, but it should be discussed with data on inter-rater reliability.\n\nWho is this for? Researchers working on speech-based mental-health triage. In current form, I would not send it to referees; I would ask the authors for a full methods supplement, corrected and reconciled data tables, baselines, and uncertainty estimates, then treat a resubmission as a serious candidate.","headline":"Genuine first application of audio-only deep learning to mental-health helpline triage, but the 92% balanced accuracy is unverifiable from the text and the train/test counts contradict the demographic table.","tokens_in":10917,"tokens_out":3079,"would_cite":false,"duration_ms":29619,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a deep-learning model can predict whether a mental-health helpline call is high priority from the caller's voice alone, with 92% balanced accuracy on 459 recorded calls.","keywords":["mental health triage","call priority prediction","deep learning","voice analysis","speech-based assessment","helpline calls","triage scale","balanced accuracy"],"falsifier":"Re-triage a sample of the 459 calls with independent clinicians and compute inter-rater agreement on the high/low split; then test the trained model on calls that receive unanimous labels. If agreement is poor, or if accuracy on unanimously labelled calls falls well below 92%, the central claim fails. A second decisive test is external validation: train on this dataset and evaluate on recordings from a different helpline or a later time period, where the same accuracy would not be expected to hold.","tokens_in":9874,"feed_emoji":"📞","tokens_out":4978,"duration_ms":41767,"temperature":0.7,"pith_summary":"This paper tests whether a machine-learning model can predict the priority of a mental-health helpline call from the caller's voice alone, without transcribing what is said. On 459 recorded calls labelled with the UK Mental Health Triage Scale, the authors collapse the seven triage codes into high-priority (A-D) and low-priority (E-G) classes and train a deep neural network on the audio. The model reaches 92% balanced accuracy on a held-out test set, with 92% sensitivity for high-priority calls and 91% specificity. The stated purpose is not to replace clinicians but to give call operators a real-time, objective signal that supports triage decisions and helps avoid delayed assistance for high-risk callers.","feed_headline":"Voice alone predicts mental-health call priority at 92% accuracy","feed_subtitle":"A 459-call trial shows a deep-learning model can flag high-risk helpline calls from speech alone, aiding triage decisions.","key_machinery":"The load-bearing mechanism is a hierarchical classifier built from deep neural networks that learns a representation directly from the audio recording of the call, rather than from transcribed text or manually specified acoustic features such as pitch and articulation rate. The learned representation is fed into a classifier that outputs a priority level, and the authors wrap this in a prototype application that displays a running priority score and confidence estimate to the call taker. The other essential component is the triage labelling itself: the seven-level UK Mental Health Triage Scale is collapsed into two classes, high priority (A-D) and low priority (E-G), so the model is trained as a binary predictor of urgency.","core_discovery":"The central claim is that call priority can be estimated from vocal properties of the caller's speech with clinically useful accuracy. On a test set of 82 calls (38 high priority, 44 low priority), the hierarchical deep-learning classifier correctly identified 35 of 38 high-priority calls and 40 of 44 low-priority calls, yielding a balanced accuracy of 92%, sensitivity of 92%, specificity of 91%, precision of 90%, and an F1 score of 91.2%. The authors position this as a feasibility demonstration: because the model uses learned audio representations rather than hand-crafted acoustic features or transcripts, it can run while a call is in progress and provide continuous priority estimates. They note that no previous study has applied this approach to mental-health helpline audio, and they treat the result as evidence that voice-based triage support is worth developing further.","pith_inferences":["A more demanding test than balanced accuracy on one dataset would be external validation: if the model is trained on one helpline's recordings and tested on another's, the accuracy may fall because of different recording equipment, accents, and caller demographics.","The A-D/E-G split is clinically coarse: an A emergency call and a D moderate-risk call are grouped together, so the 92% figure says little about whether the model can discriminate the most urgent calls from the merely urgent ones.","A natural extension the authors do not test is calibration: for a triage aid, the confidence score should track the true probability of high priority, and miscalibrated confidence could mislead call operators even if ranking accuracy is good.","The label reliability question could be turned into an experiment: ask several clinicians to re-triage the same calls and measure inter-rater agreement; if agreement is low, the reported accuracy overstates the model's actual predictive ceiling."],"forward_implications":["Voice-based triage support is feasible for mental-health helplines: a model can produce a usable priority signal from audio without needing call transcripts.","Call operators could receive a continuously updating priority and confidence display during a call, allowing them to focus attention on callers the model judges high-risk.","Because the model relies on vocal delivery rather than spoken content, it can assist triage without requiring callers to state specific symptoms.","The reported false-negative rate of 8% means roughly 1 in 12 high-priority calls would be missed; the authors explicitly aim to minimize this class of error in future versions.","Deployment would require clinical oversight: the paper frames the output as decision support, not an autonomous triage decision."],"supporting_citations":[{"why":"Supplies the UK Mental Health Triage Scale used to label every call and define the high/low split.","marker":"Sands et al. [2001]"},{"why":"Establishes the broader evidence that speech can carry signals of psychiatric disorders, motivating the audio-only approach.","marker":"Low et al. [2021]"},{"why":"Provides the closest prior result, predicting emergency call severity from speech only, which this paper extends to mental-health triage.","marker":"Abi Kanaan et al. [2023]"},{"why":"Is the earlier NLP-based call-priority study that this work contrasts with by using audio instead of text.","marker":"Ceklic et al. [2022]"},{"why":"Reviews clinical decision support systems for emergency telephone triage, framing the application context.","marker":"Michel et al. [2024]"},{"why":"Reviews mental health triage tools and supports the paper's claim that triage in mental health access is understudied.","marker":"Watson et al. [2022]"}],"fun_headline_variants":["Voice only: ML predicts mental-health call priority at 92% accuracy","92% accurate: AI sorts mental-health crisis calls from voice alone","Voice-based ML flags high-risk helpline calls at 92% balanced accuracy","Feasibility shown: ML reads caller voice for triage at 92% accuracy","AI hears caller voice risk, triage at 92% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire result depends on the clinician-assigned triage codes being accurate and reliable, because the model is trained to predict those codes, and on the A-D versus E-G split preserving the distinction that matters for call handling.","fun_headline_variants_meta":{"raw":{"variants":["Voice only: ML predicts mental-health call priority at 92% accuracy","92% accurate: AI sorts mental-health crisis calls from voice alone","Voice-based ML flags high-risk helpline calls at 92% balanced accuracy","Feasibility shown: ML reads caller voice for triage at 92% accuracy","AI hears caller voice risk, triage at 92% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001129,"raw_usage":{"total_tokens":4670,"prompt_tokens":902,"completion_tokens":3768,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":3670}},"tokens_in":518,"tokens_out":3768,"duration_ms":25686,"temperature":1.0,"reasoning_tokens":3670,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:36:44.769662+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-triage a sample of the 459 calls with independent clinicians and compute inter-rater agreement on the high/low split; then test the trained model on calls that receive unanimous labels. If agreement is poor, or if accuracy on unanimously labelled calls falls well below 92%, the central claim fails. A second decisive test is external validation: train on this dataset and evaluate on recordings from a different helpline or a later time period, where the same accuracy would not be expected to hold.","supporting_citations":[{"cited_title":"King, P., Bourke-Finn, K., & Brunning, D","cited_arxiv_id":null,"evidence_quote":"Supplies the UK Mental Health Triage Scale used to label every call and define the high/low split."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the broader evidence that speech can carry signals of psychiatric disorders, motivating the audio-only approach."},{"cited_title":"A methodology for emergency calls severity prediction: from pre-processing to bert-based classifiers","cited_arxiv_id":null,"evidence_quote":"Provides the closest prior result, predicting emergency call severity from speech only, which this paper extends to mental-health triage."},{"cited_title":"Ambulance dispatch prioritisation for traffic crashes using machine learning: a natural language approach","cited_arxiv_id":null,"evidence_quote":"Is the earlier NLP-based call-priority study that this work contrasts with by using audio instead of text."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reviews clinical decision support systems for emergency telephone triage, framing the application context."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reviews mental health triage tools and supports the paper's claim that triage in mental health access is understudied."}],"review_version":1}