{"id":"5a2f53dd-bead-4149-ab52-71a96be8a76b","arxiv_id":"2412.18614","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Mismatch between acoustic and textual emotion, learned by a cross-attention model, improves automatic depression severity detection on a private counseling dataset.","lead":"This paper adds a new signal to automatic depression detection: whether a person's voice and words carry mismatched emotions, such as speaking sad words in a calm voice. On a private counseling dataset, adding this emotional inconsistency feature raised three-class depression severity accuracy from about 76% to 81%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported improvement may stem from the extra trainable ATEI branch or from sentiment content in its supervision rather than from the inconsistency construct; a label-permuted control is required.","rationale":"The reader's weakest assumption is the construct validity of the binary sentiment-agreement label (Section 4.1/4.3.2). I agree that no inter-annotator agreement or ECI validation is provided, but I would sharpen the load-bearing problem: even if the labels were a valid operationalization, the experiments cannot attribute the accuracy gain to the inconsistency information because the ATEI branch is a large jointly trained module absent from the baseline. The A+T+E configuration improves over A+T, but this could come from added capacity, from the auxiliary consistency task, or from the sentiment content that the labels carry. The proposed random-label control directly tests whether the consistency supervision is the active ingredient. This is compatible with a conditional verdict: the paper's result is interesting but the causal claim is unproven until the control is run and significance is established. The reader already reached CONDITIONAL with moderate confidence, and my concern does not move that verdict; it specifies a concrete missing experiment that would settle it.","tokens_in":17817,"tokens_out":6898,"duration_ms":74166,"concrete_test":"Run the identical five-fold protocol with a null-control ATEI branch: within each training fold, permute the consistent/inconsistent labels before computing L_ATEI while keeping architecture, joint loss, scaling, and hyperparameters fixed. If subject-level accuracy stays near 81.25%, the gain does not depend on the inconsistency labels and the central claim is not supported. If accuracy falls back to the 75.74% baseline, a second control should replace the consistency target with acoustic sentiment (or text sentiment) as the auxiliary label; if that recovers most of the gain, the feature is sentiment leakage rather than inconsistency. Report per-fold accuracies, standard errors, and a paired significance test (e.g., McNemar on the 272 subject-level outcomes) for each control.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim that ATEI embeddings cause the +5.51% subject-level accuracy gain (Table 5) is not supported by the experimental design. In Section 4.3.2, e(E) is the FC2 output of a multimodal cross-attention network supervised by consistency labels derived in Section 4.1, and Equation (16) sets L_Total = L_Depression + L_ATEI during joint training. The ATEI branch is therefore a large additional trainable module (separate Transformer stacks over X(A) and X(T), cross-attention, and FC layers) whose gradients are also driven by the depression label, while the A+T baseline in Table 2 has no analogous branch. This confound is not addressed by comparing with [19]*, since that system uses a different architecture and a single acoustic-sentiment auxiliary task. Moreover, the acoustic sentiment labels come from five annotators listening to audio that contains the words, so the 'acoustic' label is not independent of lexical content; the binary consistency label may encode absolute sentiment polarity or annotator noise rather than the ECI-style calm-negative mismatch. No ablation isolates the inconsistency signal: the paper never trains the same pipeline with shuffled or alternative auxiliary labels.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Transformer-based framework for automatic depression detection that incorporates Acoustic-Textual Emotional Inconsistency (ATEI) as an auxiliary signal. ATEI is defined by label disagreement between acoustic sentiment (majority vote of five annotators) and textual sentiment (HowNet lexicon score). A multimodal cross-attention network extracts ATEI embeddings, which are fused with acoustic and textual features via addition, multiplication, or concatenation, optionally modulated by a learnable scaling factor. On a private counseling dataset of 272 subjects, the best configuration (A+T+E with concatenation and scaling) achieves 81.25% subject-level accuracy, a 5.51% absolute improvement over the authors' A+T baseline. The paper also claims to be the first to use emotional expression inconsistency for depression detection.","tokens_in":18079,"tokens_out":7118,"duration_ms":65518,"significance":"If validated, the idea of using cross-modal emotional inconsistency as a depression biomarker is novel and clinically motivated, drawing on Emotion Context-Insensitivity theory. The manuscript is systematic in its ablations: it compares four SSL acoustic extractors, two text encoders, three fusion strategies, three embedding layers, and a scaling factor, which is a strength. The authors are also explicit that ATEI alone is not sufficient (Section 4.3.2), which is a useful negative result. However, the current empirical evidence does not yet establish that the ATEI construct, rather than the additional trainable module or the depression-label gradient flowing into it, drives the reported gains. The evaluation rests on a private dataset with no error bars or significance tests, and the baseline is the authors' own reimplementation, so the external validity of the numbers is unclear. These issues are addressable with additional control experiments and statistical reporting, but they are load-bearing for the central claim.","major_comments":[{"comment":"The central comparison A+T+E vs. A+T does not isolate the ATEI construct. The ATEI branch in Fig. 3 is a large trainable module (Transformer stacks over X(A) and X(T), cross-attention, FC layers) that is jointly optimized with the depression loss L_Total = L_Depression + L_ATEI (Eq. 16). The A+T baseline has no analogous branch, so the +5.51% accuracy gain (Table 5) could be caused by the extra model capacity or by the gradient of L_Depression flowing into the ATEI branch, rather than by the inconsistency information itself. The paper's own report that an ATEI-only system reaches only ~40% accuracy (Section 4.3.2) supports the concern that the raw ATEI embedding is weak and that the gain comes from joint task-specific optimization. To support the claim, the authors should include a control in which the same architecture is trained with (a) shuffled consistency labels, (b) a non-inconsistency auxiliary label (e.g., acoustic sentiment only), or (c) a frozen/random ATEI branch. The comparison with [19]* in Table 3 is not this control because [19]* uses a different architecture.","section":"Section 4.3.2, Eq. (16), Tables 3–5"},{"comment":"The operational definition of emotional inconsistency is binary label disagreement: the acoustic sentiment is the majority vote of five annotators and the text sentiment is a HowNet lexicon score, and a segment is 'consistent' if the two labels match. The paper provides no inter-annotator agreement statistic and no validation that this label-disagreement proxy corresponds to the ECI-based construct of emotional blunting cited in the introduction. Moreover, because the audio annotators listen to speech that contains the words, the 'acoustic' sentiment is not independent of lexical content, so the consistency label may reflect general sentiment polarity or annotator noise rather than a calm–negative mismatch. Please report annotation reliability (e.g., Fleiss' kappa) and, ideally, a comparison of the learned ATEI features against the pilot-study findings in [20].","section":"Section 4.1 and Section 4.3.2"},{"comment":"All reported results are single estimates from five-fold cross-validation, with no standard deviations, confidence intervals, or significance tests. Differences as large as 1.84% (Table 5) could plausibly fall within cross-fold variance for a dataset of 272 subjects. The authors should report per-fold results or at least the variance across folds, and apply a paired significance test (e.g., paired t-test or Wilcoxon test) for the key comparisons between the best ATEI system and the A+T baseline and between the scaled and unscaled variants.","section":"Section 5.2, Tables 3–5"}],"minor_comments":[{"comment":"The phrase 'To best of our knowledge' should be 'To the best of our knowledge' in both the abstract and the introduction.","section":"Abstract and Section 1"},{"comment":"The notation FC1, FC2, FC3 is ambiguous: Fig. 3 shows FC1, FC2, FC3 followed by an output layer, while the text says 'three fully connected layers (FC1, FC2, and FC3)' and Table 3 uses FC3 as an embedding layer. Please clarify which layer is the output layer and which layers are used for the Embedding representation.","section":"Section 4.3.2 and Fig. 3"},{"comment":"The constraint sum(α_i)=1 is stated but no parameterization or training details are given for the learnable scaling factor. Also, since e(E) is used in element-wise addition and multiplication with e(A) and e(T), the scaling may interact with those fusion operations; please clarify how α is applied in each fusion case.","section":"Section 4.4, Eq. (15)"},{"comment":"The SDS/SAS score range for healthy controls is 25–45; if these are raw scores, the range overlaps with elevated depression/anxiety levels. Please specify the instrument versions (e.g., SDS raw vs. index score) and the clinical cutoffs used.","section":"Table 1"},{"comment":"The hyperparameters (learning rate, batch size, epochs) are given, but the model selection procedure is not described; specifically, whether a validation set was used for early stopping or hyperparameter tuning, and how the best epoch was chosen for each fold.","section":"Section 5.1"},{"comment":"The t-SNE plots are qualitative; please provide a quantitative measure of cluster separability (e.g., silhouette score) to support the claim that ATEI features improve class separation.","section":"Figure 4"},{"comment":"The term 'SOTA baseline' is used for what is essentially the authors' own architecture. Since no comparison with published state-of-the-art results on public benchmarks (e.g., DAIC-WOZ) is given, the 'SOTA' label is not justified.","section":"Table 2 and Section 5.2.1"},{"comment":"The pilot study [20] is cited as evidence for the ATEI phenomenon, but its methods and quantitative findings are not summarized. A brief description would help readers assess the validity of the operationalization used here.","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about label permutation is well-founded and I echo it. The paper is likely within the scope of this journal, but the experimental design needs the control experiments described in the major comments before the central claim can be accepted. I would not reject on the basis of private data alone; the field often accepts such studies if the analyses are rigorous. The lack of significance testing is also a recurring issue in this area, but it is fixable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"New here is the application of acoustic-textual emotional inconsistency to depression detection; prior work stopped at bipolar disorder (Niu et al.) and the authors' own pilot analysis (Wu et al.). That novelty is real and worth exploring. The paper is also systematic: ablations across SSL acoustic models, BERT/RoBERTa, three fusion strategies, several ATEI representations, and a scaling factor. The reported improvements are internally consistent across tables, and the authors are honest that ATEI alone gives only ~40% accuracy, so it is auxiliary.\n\nThe soft spot is the main empirical claim. The A+T+E system adds a substantial trainable branch: two Transformer stacks, cross-attention, FC layers, and its own auxiliary loss (Eq. 16). The A+T baseline has no analogous branch. So the +5.5% absolute gain could come from extra capacity or from task-specific depression supervision inside the branch, not from the inconsistency construct. The paper never runs a control with shuffled consistency labels or with a similarly sized branch trained on single-sentiment labels. The stress-test concern lands. Also, the acoustic sentiment labels come from five annotators listening to the same audio that contains the words, so the acoustic label is not acoustically pure; without inter-annotator statistics we cannot tell if the consistency labels are signal or noise. The text sentiment from the HowNet lexicon is crude. This matters because the operational definition of inconsistency is label disagreement.\n\nMinor issues: no error bars or significance tests across folds; the learned scaling factor alpha with sum-to-one constraint is stated without justification; and the dataset is private, so the numbers cannot be reproduced.\n\nFor a reader in affective computing or computational psychiatry, this is a plausible idea with a reasonable engineering effort. I would not accept the central claim on current evidence, but I would send it to peer review because the idea is novel and the gaps are addressable. A referee should ask for a label-permuted control, ablation of the branch architecture, significance tests, and preferably code/data release before this becomes a reliable reference.","headline":"New application of emotional inconsistency to depression detection, but the +5.5% gain is confounded by an extra trainable branch and unreleased data.","tokens_in":18573,"tokens_out":2916,"would_cite":true,"duration_ms":27812,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that emotional inconsistency between speech and text is a learnable signal for depression severity, and reports that adding it lifts subject-level accuracy from 75.74% to 81.25%.","keywords":["depression detection","acoustic-textual emotional inconsistency","multimodal cross-attention","sentiment label disagreement","emotional expression","counseling conversations","Transformer","severity classification"],"falsifier":"Feed the two sentiment labels directly into the depression classifier without the cross-attention embedding; if that baseline matches or beats 81.25% subject-level accuracy, the reported gain is carried by the sentiment information rather than by the ATEI mismatch representation.","tokens_in":17641,"feed_emoji":"🗣️","tokens_out":7851,"duration_ms":69506,"temperature":0.7,"pith_summary":"This paper tries to establish that the mismatch between how a person sounds and what their words say carries diagnostic information for depression, and that this mismatch can be extracted automatically from counseling conversations. The authors define acoustic-textual emotional inconsistency (ATEI) as disagreement between the sentiment label of an audio segment, obtained by majority vote of five annotators, and the sentiment label of its transcript, obtained from a HowNet lexicon. A multimodal cross-attention network is trained to predict this consistency label, and its intermediate embeddings are fused with acoustic and textual features in a Transformer depression classifier. On the authors' counseling conversational dataset, adding a scaled ATEI embedding raises subject-level three-class depression severity accuracy from 75.74% to 81.25%.","feed_headline":"A speech-text emotion mismatch lifts depression detection to 81%","feed_subtitle":"Adding a learned cross-modal inconsistency feature improves three-class accuracy from 75.7% to 81.3%.","key_machinery":"The central object is the Acoustic-Textual Emotional Inconsistency (ATEI) embedding, produced by a multimodal cross-attention network. The network is supervised by a consistent or inconsistent label: a segment pair is consistent if the majority-vote acoustic sentiment and the HowNet text sentiment agree, and inconsistent otherwise. Cross-attention computes $\\mathbf{X}^{(AT)} = \\mathrm{softmax}(\\mathbf{Q}^{(A)}\\mathbf{K}^{(T)\\top}/\\sqrt{d_k})\\mathbf{V}^{(T)}$ and the reverse $\\mathbf{X}^{(TA)}$, so each modality's emotional content is re-expressed through the other; the pooled concatenation of the original and cross-attended sequences is passed through fully connected layers, and the FC2 output serves as the ATEI representation. The depression classifier then concatenates acoustic, textual, and ATEI features, scales the ATEI part by a learnable $\\alpha$ with $\\sum_i \\alpha_i = 1$, and is trained jointly with the consistency classification loss.","core_discovery":"The paper's central claim is that emotional expression inconsistency across acoustic and textual modalities is a usable, depression-relevant signal, and that it can be captured without manual feature engineering by a cross-attention network supervised by a binary consistency label. The network reads frame-level self-supervised speech features and word-level text features, computes cross-domain attention in both directions, pools and concatenates the outputs, and uses a middle fully connected layer as the ATEI embedding. When that embedding is concatenated with acoustic and textual depression features and reweighted by a learnable scaling factor whose components sum to one, the resulting system reaches 81.25% subject-level accuracy for healthy, mild, and moderate classification, an absolute gain of 5.51 points over the acoustic-textual baseline. The paper also claims that this is the first depression detection system to use emotional expression inconsistency across modalities.","pith_inferences":["Editorial extension: Because the consistency label is defined as sentiment disagreement, replacing the ATEI embedding with the two sentiment labels themselves in the same fusion architecture would reveal how much of the gain is specifically the mismatch signal rather than extra sentiment information.","Editorial extension: The learnable scaling factor with $\\sum_i \\alpha_i = 1$ can be inspected post hoc; if high-weight dimensions correspond to known acoustic-textual disagreement patterns, the model doubles as a hypothesis generator for which inconsistency types track depression severity.","Editorial extension: The method should transfer to public interview corpora such as DAIC-WOZ; the paper expects less affective variability there, so a smaller gain in that setting would actually be consistent with the mechanism rather than a contradiction."],"forward_implications":["The ATEI embedding is data-driven and does not require hand-designed features, so the same labeled segments can be reused to obtain an extra diagnostic signal.","ATEI information is complementary but not sufficient on its own: the paper reports roughly 40% accuracy when ATEI embeddings are used alone, versus the full gain when they are fused with acoustic and textual features.","Among the fusion strategies tested, concatenation outperforms addition and multiplication, and applying the learnable scaling factor to the ATEI embedding adds a further absolute gain of 1.84 points in the acoustic-textual-ATEI configuration.","The middle fully connected layer (FC2) of the cross-attention network yields the best ATEI representation, indicating that intermediate hidden states carry more depression-relevant inconsistency information than the binary consistency label or its output probability.","Because evaluation uses speaker-disjoint folds and majority voting across segments, the reported 81.25% subject-level accuracy is a claim about unseen speakers rather than about memorized segments."],"supporting_citations":[{"why":"Supplies the pilot evidence that depressed speakers show inconsistency between spoken and semantic emotion in conversation.","marker":"[20]"},{"why":"Precedent for using acoustic-textual emotional mismatch as a feature, developed manually for bipolar disorder mood identification.","marker":"[21]"},{"why":"Emotion Context-Insensitivity theory, the psychological rationale that depressed patients show constrained emotional variation.","marker":"[22]"},{"why":"HowNet lexicon used to assign the textual sentiment labels that define the consistency and inconsistency supervision.","marker":"[54]"},{"why":"Baseline system using emotional features from a single acoustic sentiment label; the comparison point for the advantage of ATEI.","marker":"[19]"},{"why":"Earlier demonstration that auxiliary emotion information from a single acoustic sentiment label improves depression estimation.","marker":"[16]"}],"fun_headline_variants":["Speech-text emotion mismatch: a new predictor for depression","Depression detection gets a 5.5-point boost from emotion inconsistency","First system to use cross-modal emotion mismatch for depression detection","Cross-modal emotion inconsistency: a new depression marker with 81% accuracy","Emotional mismatch in speech and text detects depression with 81% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that binary disagreement between the majority-vote acoustic sentiment label and the HowNet text sentiment label faithfully captures the emotional inconsistency that distinguishes depressed from healthy speakers.","fun_headline_variants_meta":{"raw":{"variants":["Speech-text emotion mismatch: a new predictor for depression","Depression detection gets a 5.5-point boost from emotion inconsistency","First system to use cross-modal emotion mismatch for depression detection","Cross-modal emotion inconsistency: a new depression marker with 81% accuracy","Emotional mismatch in speech and text detects depression with 81% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001704,"raw_usage":{"total_tokens":6736,"prompt_tokens":925,"completion_tokens":5811,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":5722}},"tokens_in":541,"tokens_out":5811,"duration_ms":36501,"temperature":1.0,"reasoning_tokens":5722,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:57:26.776179+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed the two sentiment labels directly into the depression classifier without the cross-attention embedding; if that baseline matches or beats 81.25% subject-level accuracy, the reported gain is carried by the sentiment information rather than by the ATEI mismatch representation.","supporting_citations":[{"cited_title":"Depression enhances internal inconsistency between spoken and semantic emotion: Evidence from the anal- ysis of emotion expression in conversation,","cited_arxiv_id":null,"evidence_quote":"Supplies the pilot evidence that depressed speakers show inconsistency between spoken and semantic emotion in conversation."},{"cited_title":"Capturing mismatch between textual and acoustic emotion expressions for mood identification in bipolar disorder,","cited_arxiv_id":null,"evidence_quote":"Precedent for using acoustic-textual emotional mismatch as a feature, developed manually for bipolar disorder mood identification."},{"cited_title":"Mood and emotion in major depres- sion,","cited_arxiv_id":null,"evidence_quote":"Emotion Context-Insensitivity theory, the psychological rationale that depressed patients show constrained emotional variation."},{"cited_title":"Hownet-a hybrid language and knowledge resource,","cited_arxiv_id":null,"evidence_quote":"HowNet lexicon used to assign the textual sentiment labels that define the consistency and inconsistency supervision."},{"cited_title":"A transformer-based depression detec- tion network leveraging speech emotional expression cues,","cited_arxiv_id":null,"evidence_quote":"Baseline system using emotional features from a single acoustic sentiment label; the comparison point for the advantage of ATEI."},{"cited_title":"Improving depression level estimation by concur- rently learning emotion intensity,","cited_arxiv_id":null,"evidence_quote":"Earlier demonstration that auxiliary emotion information from a single acoustic sentiment label improves depression estimation."}],"review_version":1}