{"id":"f75d14f6-433c-42a5-aabf-20d2336272df","arxiv_id":"2511.04476","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PTTSD, a probabilistic LSTM-attention model, predicts PHQ-8 scores and calibrated uncertainties from utterance sequences in clinical interviews.","lead":"A new model, PTTSD, predicts depression severity from clinical interview text and outputs a confidence interval for each prediction. It reports state-of-the-art results among text-only systems on E-DAIC, but the comparison is affected by re-transcribed audio.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Re-transcribed input text invalidates the state-of-the-art comparison; PTTSD's MAE/RMSE gains may stem from WhisperX transcripts, not the model.","rationale":"The reader identified re-transcription as the weakest assumption, and my analysis confirms it is the most load-bearing concern. The paper's central claim of state-of-the-art performance is contingent on fair comparison with baselines that used the official transcripts. Without a control experiment on the original transcripts, the reported improvement cannot be attributed to the model. This does not invalidate the entire paper—the probabilistic framework and calibration analysis are still of interest—but it means the headline SOTA claim is conditional on a check the authors did not perform. The reader's conditional verdict is appropriate; my concern does not shift the verdict, only reinforces the need for the missing control. I considered the per-timestep calibration issue as an alternative, but the re-transcription is more directly tied to the main quantitative claim and is the first thing a skeptical reader would question.","tokens_in":16135,"tokens_out":3700,"duration_ms":37731,"concrete_test":"Rerun the PTTSD sequence-to-sequence all-MiniLM-L6-v2 configuration (the variant that achieved MAE=3.85 on E-DAIC) using the original official E-DAIC transcripts while keeping all preprocessing, model architecture, training hyperparameters, and evaluation metrics identical. Compare the test MAE/RMSE against the reported 3.85/4.52. If MAE rises above 4.22 (the best text-only baseline by Sadeghi et al. 2024 on Whisper transcripts), the state-of-the-art claim is not supported. If MAE stays below 4.22, the re-transcription is unlikely to be the sole cause of the gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that PTTSD achieves state-of-the-art text-only performance on E-DAIC (MAE=3.85, RMSE=4.52) rests on an unfair comparison. Section 3.2 states that all audio was re-transcribed with WhisperX, while every baseline in Table 1 (e.g., Sadeghi et al. 2024, Fang et al. 2023b) was evaluated on the original official transcripts. The comparison therefore conflates model quality with transcript quality. If WhisperX transcripts are easier to predict from (e.g., contain fewer errors, better speaker attribution, or different content), then the MAE/RMSE improvement may be entirely due to the input text rather than PTTSD's architecture or probabilistic training. This is not a minor protocol detail: it directly undermines the paper's headline quantitative contribution. The paper itself acknowledges re-transcription in the Abstract and Section 3.2, but does not run any control using the official transcripts, leaving the effect unmeasured. The calibration claim is also affected because calibration is evaluated on the same WhisperX text, so the input change could influence uncertainty quality as well. This is the single most load-bearing assumption: if the gain disappears on official transcripts, the SOTA claim is void, and the paper reduces to a standard probabilistic LSTM with an unclear evaluation protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PTTSD, a probabilistic framework for predicting PHQ-8 depression severity from utterance-level transcripts, with sequence-to-sequence and sequence-to-one variants built on bidirectional LSTMs, self-attention, residual connections, and Gaussian or Student-t output heads trained with NLL. The authors report state-of-the-art text-only results on E-DAIC (MAE 3.85, RMSE 4.52) and DAIC (MAE 3.55, RMSE 4.77), claim well-calibrated uncertainty via ECE and coverage analysis, and present ablations, hyperparameter sensitivity, and case studies. The manuscript is clearly written and includes a limitations section, but the central SOTA claim rests on an uncontrolled input-text change, and the sequence-to-sequence evaluation protocol is not precisely defined.","tokens_in":16431,"tokens_out":8320,"duration_ms":71654,"significance":"If the SOTA claim were validated, PTTSD would be a useful contribution to clinical NLP: it offers a reproducible, prompt-free, fully probabilistic model with uncertainty estimates, three-run averaged results, public code, and a multi-part calibration analysis. The uncertainty-calibration component is a valuable direction, and the paper's framing around clinical decision support is appropriate. However, the re-transcription of all audio with WhisperX changes the model's input relative to most baselines, so the headline accuracy gains cannot currently be attributed to the model. The sequence-to-sequence evaluation also needs clarification before the numbers in Table 1 can be compared with single-session baselines. The underlying ideas are defensible, but the evidence as presented does not yet establish the paper's strongest claims.","major_comments":[{"comment":"The state-of-the-art claim is undermined by an uncontrolled input change. Section 3.2 states that original E-DAIC audio was re-transcribed with WhisperX, and Section 4.1 states that all audio is re-transcribed. PTTSD is therefore evaluated on text that differs from the official E-DAIC/DAIC-WOZ transcripts used by most baselines in Table 1 (with the exception of Sadeghi et al. 2024's Whisper-based variant). No control experiment is reported in which PTTSD is run on the official transcripts, nor are any baselines run on the WhisperX transcripts. The observed gains in MAE/RMSE may thus be due to the re-transcribed input rather than to PTTSD's architecture or probabilistic training. This is load-bearing for the paper's headline contribution; please provide a controlled comparison or explicitly downgrade the claim.","section":"§3.2, §4.2, Table 1"},{"comment":"The evaluation protocol for the sequence-to-sequence variant is not defined tightly enough for a fair comparison. In seq2seq mode, the session-level PHQ-8 label is used as the target at every utterance, and Section 4.1 says metrics are computed over all valid (non-padded) utterances. This means each participant contributes multiple predictions, and a per-utterance aggregate weights longer interviews more heavily than a per-session baseline. The paper does not state whether Table 1 reports per-utterance MAE/RMSE, a per-participant average of the T predictions, or another aggregation. The same repeated-label structure affects the calibration and error-uncertainty correlation analyses, where per-utterance predictions are treated as independent samples, inflating the effective sample size and significance. Please clarify the aggregation and justify comparability with single-session baselines","section":"§3.4–3.5, §4.1, Table 1"},{"comment":"Ablation results do not support the claim that probabilistic modeling improves accuracy. Removing the variance head and training with MSE yields MAE 5.98 and RMSE 7.21, which are better than the full model's 6.32 and 8.10. The abstract and conclusion state that ablations confirm the value of probabilistic modeling, but Table 3 shows the opposite for point-error metrics. Moreover, the ablation full model is trained for only 20 epochs and reaches MAE 6.32/RMSE 8.10, far worse than the main full model's 3.85/4.52 in Table 1, so the ablation is not performed at a comparable training budget. Please retrain all ablation variants to convergence and separate the effect of loss change from removal of the variance head.","section":"§4.3, Table 3"}],"minor_comments":[{"comment":"The sentence 'We represent each utterance using pretrained sentence encoders.' is duplicated verbatim.","section":"§3.3"},{"comment":"For DAIC, PTTSD's RMSE 4.77 is slightly worse than Fang et al.'s 4.76; the statement that PTTSD 'matches the best test RMSE' is imprecise.","section":"§4.2, Table 1"},{"comment":"The text says MSE 'yields the highest test RMSE among all probabilistic losses,' but MSE is not a probabilistic loss and its test RMSE (4.876) is lower than Gaussian NLL (5.0219) and Student-t (5.1488). Please rephrase or correct.","section":"§4.3, Table 2"},{"comment":"Calibration analysis is reported on the DAIC test set, while the headline SOTA result is on E-DAIC. Please report E-DAIC calibration or justify the choice.","section":"§4.5, Figure 3"},{"comment":"The uncertainty ranges differ across figures (up to 35 in Figure 7, up to 60 in Figure 3), and Figure 6 contains the placeholder text 'Multiple model sources not available.' Please clarify which configuration/split each figure uses and remove placeholder text.","section":"§E.1, §E.2, Figures 6–7"},{"comment":"Fang et al. 2023a and 2023b appear to be the same publication; the table and related work use both labels. The related-work table also has a typo, 'Oureshi et al. (2021)' for 'Qureshi et al. (2021)'.","section":"References and Table 1"},{"comment":"Appendix B.1 states that E-DAIC is already a re-transcribed and quality-controlled extension of DAIC-WOZ; Section 3.2 then says the authors re-transcribed the audio again with WhisperX. Please clarify the relationship between the official E-DAIC transcripts, the WhisperX transcripts, and the input actually used for each dataset.","section":"§3.2, Appendix B.1"}],"recommendation":"major_revision","confidential_remarks":"The re-transcription issue is the main risk to the paper. The SOTA claim is currently not grounded in a controlled comparison. A revision that includes PTTSD evaluated on the official transcripts (or a baseline run on WhisperX transcripts) would substantially strengthen the paper. The seq2seq evaluation protocol must also be nailed down. The uncertainty-calibration analysis is a promising direction but is entangled with the input change. I would not support acceptance without these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the model is sensible and the calibration work is worth a look, but the headline \"state-of-the-art\" number is not clean. The stress-test note has it right: the paper retranscribes all audio with WhisperX, while the baselines in Table 1 mostly ran on official transcripts or standard Whisper. That changes the input, so the MAE/RMSE gains could come from better transcripts rather than the model. The paper is upfront about the retranscription, which I credit, but it never runs the obvious controls: PTTSD on official transcripts, or at least one baseline on WhisperX text. Without those, the SOTA claim is unsupported.\n\nWhat's genuinely new: a sequence-to-sequence LSTM with attention that outputs a predictive distribution at every utterance, trained with NLL, for text-only PHQ-8 regression. The three-part calibration analysis (ECE, coverage plots, error-uncertainty correlation) is more thorough than most papers in this area, and the ECE of 0.022 on the DAIC test set is decent if it holds up. The limitations section is candid about the small dataset, the Wizard-of-Oz interviews, and the lack of clinical evaluation. The GitHub link is a good sign, though I haven't verified the code runs.\n\nThe soft spots, in proportion: First, the SOTA comparison is the load-bearing issue. Even though Sadeghi et al. used Whisper rather than official transcripts, WhisperX is not Whisper; differences in speaker diarization and alignment can affect what the model sees. This is fixable, but as it stands the comparison conflates transcript quality with model quality. Second, the seq2seq evaluation repeats the same PHQ-8 label at every utterance and then averages per-utterance metrics. That inflates the effective sample size, and it makes the reported r=0.88 between uncertainty and error look stronger than it is—those aren't independent observations. Third, the ablation shows that removing the variance head actually improves MAE and RMSE. The paper frames this as a trade-off, which is fair, but it means the \"value of probabilistic modeling\" is about calibration, not accuracy. The abstract should say that.\n\nThe core idea—uncertainty counts in clinical decision support—still holds. The paper is worth engaging with, but it needs proper controls and a more careful evaluation protocol before its claims can be trusted. I'd send it to reviewers, with the explicit ask that they require the control experiments.","headline":"Reasonable probabilistic model for PHQ-8 prediction, but the SOTA claim rests on an input change (WhisperX retranscription) that isn't controlled for.","tokens_in":16873,"tokens_out":2608,"would_cite":false,"duration_ms":28167,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Probabilistic text model predicts depression severity with calibrated uncertainty from clinical interviews.","keywords":["depression detection","PHQ-8","uncertainty quantification","textual time series","probabilistic neural network","clinical NLP","calibration","LSTM self-attention"],"falsifier":"Run PTTSD and the strongest baselines on the same transcript text — either official E-DAIC transcripts or the re-transcribed version — and check whether the MAE/RMSE gap persists; if PTTSD loses its margin on matched input, the state-of-the-art claim is an artifact of transcription.","tokens_in":16040,"feed_emoji":"🧠","tokens_out":4615,"duration_ms":44808,"temperature":0.7,"pith_summary":"The paper argues that depression severity from clinical interviews is best treated as a probabilistic textual time series: each utterance updates a predictive distribution over the PHQ-8 score rather than a single point estimate. By training LSTMs with self-attention and Gaussian or Student-t output heads on negative log-likelihood, the authors obtain a system that is both accurate and uncertainty-aware. On E-DAIC it reports MAE 3.85 and RMSE 4.52, beating previous text-only systems, while producing well-calibrated intervals. The authors see this as evidence that probabilistic modeling and temporal structure belong in clinical NLP, not just point-prediction pipelines.","feed_headline":"Text-only depression model sets new accuracy record","feed_subtitle":"PTTSD reports MAE 3.85 on E-DAIC and outputs confidence intervals that track error.","key_machinery":"The engine is a bidirectional LSTM over utterance embeddings, followed by multi-head self-attention with residual connections, and a probabilistic output head. The Gaussian head predicts mean and variance per timestep; the Student-t head additionally predicts degrees of freedom. Training minimizes negative log-likelihood, optionally reweighted, and seq2seq mode lets confidence be read off at every utterance despite a single session-level target. Attention masks and log-transformed targets support stable training on variable-length interviews.","core_discovery":"The central claim is that modeling PHQ-8 regression as a fully probabilistic sequence problem — predicting a Gaussian or Student-t distribution over the score at each utterance (seq2seq) or for the whole interview (seq2one) — yields both competitive point accuracy and calibrated confidence estimates. The authors report state-of-the-art text-only results on E-DAIC (MAE 3.85, RMSE 4.52) and competitive results on DAIC, with an ECE of 0.022 and 66.2% empirical coverage at the nominal 68% level for the Gaussian NLL model. They interpret this as showing that uncertainty estimates track prediction quality, with a Pearson correlation of 0.88 between predicted uncertainty and absolute error.","pith_inferences":["The comparison is not apples-to-apples: all baselines were scored on official transcripts, while PTTSD was scored on automatically re-transcribed audio; if transcription quality is the driver, the state-of-the-art claim may not survive matched transcripts.","The same probabilistic time-series treatment could be applied to other single-session clinical ratings, and the temporal confidence signal could feed adaptive data collection or active learning in digital therapy.","Because the corpus has fewer than 300 sessions and uses a virtual interviewer, calibration quality on genuine clinician-patient speech remains an open, testable question.","The strong uncertainty-error correlation suggests predicted variance could serve as a screening gate, sending only high-uncertainty cases for human review; the paper does not test this."],"forward_implications":["If the reported numbers hold, a simple, prompt-free text-only pipeline can match or beat far more elaborate multimodal and prompt-based systems on E-DAIC.","Clinicians could use the predicted interval width as a flag for when the model is unsure, rather than receiving only a point score.","The seq2seq variant shows how confidence evolves during the interview, potentially indicating which parts of a session are most informative.","The ablation implies that self-attention and residual connections are load-bearing for accuracy, while the variance head costs a little raw error in exchange for calibrated uncertainty.","Default Gaussian NLL with balanced weights appears to be the right training objective for sharp and calibrated intervals."],"fun_headline_variants":["Depression model reports its own uncertainty and stays accurate","Probabilistic depression detection: scores plus calibrated confidence","Text-based depression prediction with honest confidence intervals","Uncertainty-aware depression scoring matches top text-only accuracy","PTTSD: depression severity with error bars you can trust"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The system is evaluated on its own re-transcribed text rather than the official transcripts used by all comparison models, so the reported accuracy advantage could come from better transcription, not from the probabilistic architecture.","fun_headline_variants_meta":{"raw":{"variants":["Depression model reports its own uncertainty and stays accurate","Probabilistic depression detection: scores plus calibrated confidence","Text-based depression prediction with honest confidence intervals","Uncertainty-aware depression scoring matches top text-only accuracy","PTTSD: depression severity with error bars you can trust"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001448,"raw_usage":{"total_tokens":5656,"prompt_tokens":715,"completion_tokens":4941,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":459,"completion_tokens_details":{"reasoning_tokens":4865}},"tokens_in":459,"tokens_out":4941,"duration_ms":40159,"temperature":1.0,"reasoning_tokens":4865,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T23:38:34.483116+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PTTSD and the strongest baselines on the same transcript text — either official E-DAIC transcripts or the re-transcribed version — and check whether the MAE/RMSE gap persists; if PTTSD loses its margin on matched input, the state-of-the-art claim is an artifact of transcription.","supporting_citations":[],"review_version":1}