{"id":"52fb5fc1-458b-4140-9ded-fbead7031f00","arxiv_id":"2505.24609","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"MHIM, a masking training method, is applied to attention-based depression detection models, producing modest improvements in accuracy and attention-based explainability on Thai and English datasets.","lead":"This paper applies Masked Hard Instance Mining, a training trick from medical image analysis, to text-based depression detection to make model attention less concentrated and more explainable. The authors report improved prediction and explainability on Thai and English interview datasets, though the gains are small and not statistically tested.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MHIM's explainability gain is confounded with its entropy increase: Recall@k has no precision or entropy-matched baseline, and the DAIC/overall-HAM-D drop proxy only shows model reliance. The central explainability claim needs a controlled comparison before it can stand.","rationale":"The reader's weakest assumption was already that high-entropy attention must correspond to clinically relevant features and that the proxy evaluation must be valid. I agree with that diagnosis and sharpen it into a specific, testable confound: entropy itself can inflate Recall@k, especially at high k, and the paper reports no precision, no error bars, and no entropy-matched baseline. The drop-based proxy in Figures 2–3 is circular because it uses model attention both to select 'important' sentences and to measure the model's output change, which only demonstrates model reliance. The proposed entropy-matched permutation test isolates whether MHIM's attention is genuinely better at identifying human-labeled salient sentences at the low k values where the explainability claim is most meaningful. If the low-k Recall gains survive the control, the central claim is supported; if not, the remaining evidence is a small, statistically unsupported change in prediction metrics, and the explainability contribution is unproven. Because the reader already assigned CONDITIONAL with high correctness risk, my analysis does not move the verdict; it strengthens the reasons for requiring controlled evidence before acceptance.","tokens_in":8128,"tokens_out":5121,"duration_ms":66096,"concrete_test":"On the Thai HAM-D1 test folds, compute Recall@k and precision@k for MHIM versus a baseline that randomly permutes MHIM's per-interview attention scores (preserving each interview's entropy), averaged over many permutations. If MHIM's Recall@k at k=10% and k=20% is not significantly higher than this entropy-matched random baseline under a paired bootstrap test, the explainability improvement is explained by attention spreading rather than by identification of clinically salient sentences.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that MHIM improves prediction and explainability. The prediction gain is already weak (Table 1: Thai overall MAE worsens 2.47→2.52; all RMSE differences are within reported standard deviations; DAIC RMSE shifts only 0.733→0.717), but the load-bearing novelty is explainability. There the evidence has two gaps. First, Table 2 reports Recall@k on HAM-D1 only, without precision, error bars, or significance testing, and the method's explicit mechanism is to spread attention (entropy rises from 2.406 to 2.496 and from 2.158 to 2.819 in Table 1). Recall@k at high k mechanically favors spread attention; at low k the MHIM gains are small (0.13→0.17 at k=10%, 0.21→0.27 at k=20%). No random, uniform, or entropy-matched baseline is provided, so the gains cannot be attributed to selecting clinically salient sentences rather than to a broader attention distribution. Second, for overall HAM-D and DAIC-WOZ, Figures 2–3 replace ISL with 'drop the model's top-attended sentences and measure the RMSE rise.' This measures the model's reliance on those sentences, not their clinical relevance, making it circular as an explainability claim; the figures also lack error bars. Section 5.2 itself concedes the model is 'prone to cases where a single high-ISL sentence may lead to a false positive.' Because the abstract asserts that MHIM 'significantly improves' explainability, and the only non-circular evidence is one labeled item without controls, the central claim is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Masked Hard Instance Mining (MHIM), a two-phase training procedure for attention-based depression detection. In phase 1 a Dual Encoder is trained to predict depression severity; in phase 2 a second model is trained with attention masking derived from the phase-1 model: t top-attended and b bottom-attended sentence embeddings are masked, forcing the receiver to distribute attention more broadly. The method is evaluated on Thai-Maywe (HAM-D) and DAIC-WOZ (PHQ-8 item 2), comparing Dual Encoder, Single Encoder, and LLM prompting baselines on RMSE/MAE and on attention-based explainability metrics: Recall@k against human Importance Sentence Labels for HAM-D1, and a drop-top-attention sensitivity analysis for overall HAM-D and DAIC-WOZ. The authors claim that MHIM significantly improves both prediction accuracy and explainability.","tokens_in":8473,"tokens_out":4015,"duration_ms":42213,"significance":"If the claims held, MHIM would be a cheap and appealing regularizer for clinical NLP: training to spread attention would improve both accuracy and interpretability in low-resource settings, an important problem. The paper has useful strengths: it evaluates on two languages/corpora, reports attention entropy, makes one direct comparison with human Importance Sentence Labels (Thai HAM-D1), and includes baselines from prior work. However, the current evidence does not establish the central claims: prediction gains are within noise, and the explainability evidence is either confounded with entropy or circular. With additional controlled experiments and significance testing, the ideas could become a solid contribution.","major_comments":[{"comment":"The claim that MHIM 'significantly improves' prediction is not supported by inferential statistics; the reported gains are mostly within one standard deviation (e.g., Thai HAM-D1 RMSE 0.54±0.28 vs 0.48±0.28, MAE 0.37±0.19 vs 0.33±0.19), and overall HAM-D MAE actually worsens from 2.47±0.76 to 2.52±0.60, while the DAIC-WOZ results in Section 5.3 (RMSE 0.733 to 0.717) are reported without error bars or significance tests.","section":"Section 5.1, Table 1"},{"comment":"Recall@k against human ISL labels is reported only for HAM-D1, with no precision, error bars, or significance tests, and no random/uniform or entropy-matched baseline; because the method's explicit mechanism is to increase attention entropy (Section 3, Table 1 shows entropy rising from 2.406 to 2.496 and 2.158 to 2.819), the large gains at k=50-90% are mechanically expected for any spread-out attention distribution, so the table cannot currently distinguish 'better explanations' from 'broader attention'.","section":"Section 5.2, Table 2"},{"comment":"The 'drop top-attended sentences and measure RMSE rise' analysis measures model reliance, not clinical relevance, making it circular as an explainability claim for overall HAM-D and DAIC-WOZ; this is not an external validation, especially since the paper concedes in Section 5.2 that the model is 'prone to cases where a single high-ISL sentence may lead to a false positive.'","section":"Section 5.2, Figures 2-3"},{"comment":"Because the phase-2 training objective explicitly encourages more spread attention and the same attention weights are then used as the explanation, the explainability evaluation needs to control for entropy (e.g., match entropy between baseline and MHIM or report precision at fixed recall); without such a control, the self-referential loop between training objective and evaluation metric remains unresolved.","section":"Section 3 and Section 5.2"}],"minor_comments":[{"comment":"'PHD-Q' appears to be a typo for 'PHQ-8' (twice), and 'HAM-D or PHD-Q score' should be corrected.","section":"Sections 3 and 4.1"},{"comment":"'The masks comes from two sets' should read 'The masks come from two sets'; additionally, Table 1 shows '0.542.52' with a missing separator between MAE and entropy for the MHIM row.","section":"Section 3"},{"comment":"The phrase 'adjacency to human attention' is vague; consider clarifying what property of attention is being claimed.","section":"Section 2.4"},{"comment":"The notation uses n for both the number of sentences and the denominator sum; if the index i runs over sentences, clarify that and define n accordingly.","section":"Section 4.3, Equation 2"}],"recommendation":"major_revision","confidential_remarks":"I see no evidence of misconduct. The main concerns are statistical rigor and the conflation of attention spread with explanation quality; both are fixable with additional analysis, so I support a major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read arXiv:2505.24609. The core idea is a faithful transfer of Masked Hard Instance Mining from whole-slide pathology to text-based depression detection: train a donor Dual Encoder, then use its top-attention masks to force a second encoder to spread attention. That's a legitimate extension, and the authors are honest about the MIL framing and about the weak points. The direct evaluation on Thai HAM-D1 against externally provided ISL labels is the right kind of check, and the paper reports the entropy increase that the method induces.\n\nThe problem is that the central claim — that MHIM improves explainability — does not survive a controlled look. The Recall@k gains at high k (e.g., 0.52 to 0.62 at k=50%) are mechanically favored by the method's explicit goal of spreading attention. With no precision, no error bars, no significance test, and no uniform or entropy-matched baseline, those numbers cannot distinguish 'finds clinically salient sentences' from 'just attends to more sentences.' For overall HAM-D and DAIC-WOZ, the drop-proxy (remove top-attended sentences, measure RMSE rise) measures model reliance, which is the same attention the model was trained to spread. That's circular for explainability purposes. The paper itself concedes a false-positive failure mode, which is good, but it doesn't rescue the evidence.\n\nPrediction-wise, the abstract says 'significantly improves,' but the tables don't support that: Thai overall MAE worsens (2.47→2.52), and all RMSE differences are within one standard deviation. DAIC is reported without error bars (0.733→0.717). No significance testing anywhere. The LLM baseline is handled passably, but the star handling for recall at low k is a minor issue.\n\nWhat's good: the two-phase donor-receiver design is clearly described, the entropy hypothesis is explicit, and using an external human label (ISL) is a genuine attempt to measure interpretability rather than just reporting attention as proof. That's more than many explainability papers do.\n\nWho should read it: anyone working on attention-based clinical NLP, or on explainability evaluation pitfalls. It's a useful cautionary example of why entropy regularization can confound explainability metrics.\n\nMy recommendation: this deserves a serious referee. The method is a real extension, and the ISL evaluation gives it enough substance that an editor should not desk-reject. The authors need to add significance tests, report variance on DAIC, add an entropy-matched or random-attention baseline, and either find a non-circular proxy for overall HAM-D or drop that part of the claim. I would not cite it as-is.","headline":"A legitimate transfer of MHIM to text depression detection with the right instinct for external labels, but the explainability claim is confounded by the method's own entropy increase and lacks the controlled baselines it needs.","tokens_in":8980,"tokens_out":2164,"would_cite":false,"duration_ms":25033,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that masking a depression detector's most-attended sentences during a second training phase yields attention that is more evenly spread and more clinically aligned, improving prediction and explainability on both Thai…","keywords":["depression detection","explainable AI","attention mechanism","multi-instance learning","masked hard instance mining","low-resource NLP","Thai clinical interviews","DAIC-WOZ"],"falsifier":"Evaluate the baseline and MHIM models on a held-out set of clinical interviews where expert annotators label every sentence's relevance to the target symptom, then compare Recall@k of each model's attention rankings; if the MHIM receiver does not beat the baseline donor at multiple k values, or if the RMSE-drop curves are reproduced by randomly chosen sentence removals, the central explainability claim fails.","tokens_in":7905,"feed_emoji":"🧠","tokens_out":8643,"duration_ms":94823,"temperature":0.7,"pith_summary":"Depression screening models typically output a score without a trustworthy explanation. This paper argues that a model can be trained to give both. The proposed method, Masked Hard Instance Mining (MHIM), works in two phases: a first Dual Encoder learns normally, then a second is trained with the first model's highest-attended sentences masked out, forcing its attention to spread across more of the interview. On Thai (Thai-Maywe) and English (DAIC-WOZ) clinical interviews, the masked model improves prediction error, attention entropy, and agreement with clinician-labeled important sentences, while the same attention weights become better at locating sentences whose removal changes the prediction. If the claim holds, a cheap training-time regularizer can improve both accuracy and explainability in low-resource clinical NLP.","feed_headline":"Masking top attention spots improves depression detection explanations","feed_subtitle":"Blindfolding over-attended sentences during training lifts accuracy and explainability in depression screening.","key_machinery":"The load-bearing mechanism is a training-time attention mask inside the Multiple Instance Learning aggregation layer, where each interview is a bag of sentence instances and attention weights serve as the pooling operation. In phase 1, a donor Dual Encoder is trained normally. In phase 2, a second Dual Encoder is trained while the donor's top attention weights are masked out of the receiver, so the receiver cannot rely on the same few sentences and must find alternative evidence. The paper tracks attention entropy as a direct measure of diversification, and uses Recall@k against clinician labels, plus an RMSE-after-dropping sensitivity analysis, as measures of explainability.","core_discovery":"On the paper's own terms, the discovery is that forcing attention diversification through MHIM makes the attention mechanism a better explanation device without sacrificing prediction quality. Starting from the Dual Encoder, which pools sentence representations with attention, the authors train a donor model, obtain its attention scores, and then train a second model that is blinded to the donor's top-attended sentence embeddings, plus a few random low-attention masks. At inference the mask is removed, yet the receiver's learned attention has higher entropy and better predictive performance on both datasets. On Thai HAM-D 1, where clinician-provided Importance Sentence Labels exist, the MHIM model's Recall@k beats the baseline at every cutoff and the qualitative top-3 sentences show more symptom-related content; on overall HAM-D and DAIC-WOZ PHQ-8, where no sentence labels exist, the authors use the proxy that removing the model's top-attended sentences raises RMSE more than random removal, which they interpret as evidence that the top attention identifies important evidence.","pith_inferences":["A natural one-phase alternative would be to add an explicit attention-entropy regularizer during training; if entropy gain is the true cause, a single run could capture most of MHIM's benefit at lower cost.","The RMSE-drop proxy measures sensitivity, not clinical validity, so an independent human study of whether the model's chosen sentences are genuinely more useful to clinicians would strengthen or refute the explanation claim.","Because MHIM only requires attention weights from a donor model, it could transfer to other weakly labeled bag-classification tasks, such as pathology slide analysis or social-media screening, where attention overfits to a few instances.","The paper's observation that the LLM baseline underperforms on longer explanation lists hints that attention-based MIL may be better suited than prompting for exhaustive explanations."],"forward_implications":["On Thai-Maywe, adding MHIM to the Dual Encoder lowers RMSE on HAM-D 1 from 0.54 to 0.48 and on overall HAM-D from 3.54 to 3.15, while MAE improves on HAM-D 1.","Attention entropy rises with MHIM, from 2.406 to 2.496 on HAM-D 1 and from 2.158 to 2.819 on overall HAM-D, supporting the claim that attention spreads more evenly.","Against clinician-provided Importance Sentence Labels on HAM-D 1, MHIM raises Recall@k at every reported cutoff, for example from 0.52 to 0.62 at k=50% and from 0.85 to 0.90 at k=80%.","On the English DAIC-WOZ dataset, MHIM keeps prediction RMSE comparable or slightly better (0.733 to 0.717) while the attention-based dropping analysis shows improved sensitivity to important sentences.","The two-phase masking also improves the Single Encoder variant, so the benefit does not depend on the Dual Encoder architecture specifically."],"supporting_citations":[{"why":"Provides the Dual Encoder architecture and training scheme used as the phase-1 donor and the main baseline.","marker":"[8]"},{"why":"Introduces Masked Hard Instance Mining, the masking procedure the paper adapts to attention-based depression detection.","marker":"[14]"},{"why":"Supplies the Thai-Maywe dataset, including the Importance Sentence Labels used for Recall@k explainability evaluation.","marker":"[10]"},{"why":"Supplies the English DAIC-WOZ corpus with PHQ-8 labels used to test the method on a second language and dataset.","marker":"[11]"},{"why":"Provides the LLM prompting baseline and the finding that prompted explanations underperform fine-tuned models, against which MHIM is compared.","marker":"[9]"},{"why":"Provides the attention-based Multiple Instance Learning formulation that frames dialogue turns as a bag of instances.","marker":"[13]"},{"why":"Provides the Single Encoder architecture used as an additional baseline to show MHIM's benefit is not limited to the Dual Encoder.","marker":"[24]"}],"fun_headline_variants":["Masking top-attended sentences improves depression detection clarity","Attention masking makes depression screening models explain better","Training with masked attention diversifies focus in depression models","Blinding key sentences during training lifts depression detection explanations","Wider attention after masking boosts depression detection and explainability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gain rests on the assumption that forcing attention to spread out during training makes attention weights at inference genuinely better at pointing to clinically relevant sentences, and that the evaluation metrics, namely Recall@k on one labeled item plus the RMSE-after-dropping proxy, actually measure that relevance.","fun_headline_variants_meta":{"raw":{"variants":["Masking top-attended sentences improves depression detection clarity","Attention masking makes depression screening models explain better","Training with masked attention diversifies focus in depression models","Blinding key sentences during training lifts depression detection explanations","Wider attention after masking boosts depression detection and explainability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1271,"prompt_tokens":843,"completion_tokens":428,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":459,"completion_tokens_details":{"reasoning_tokens":353}},"tokens_in":459,"tokens_out":428,"duration_ms":5678,"temperature":1.0,"reasoning_tokens":353,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:17:07.622037+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the baseline and MHIM models on a held-out set of clinical interviews where expert annotators label every sentence's relevance to the target symptom, then compare Recall@k of each model's attention rankings; if the MHIM receiver does not beat the baseline donor at multiple k values, or if the RMSE-drop curves are reproduced by randomly chosen sentence removals, the central explainability claim fails.","supporting_citations":[{"cited_title":"Depression and other common mental disorders: global health estimates,","cited_arxiv_id":null,"evidence_quote":"Provides the Dual Encoder architecture and training scheme used as the phase-1 donor and the main baseline."},{"cited_title":"Xa-bilstm: a deep learning approach for depression detection in imbalanced data,","cited_arxiv_id":null,"evidence_quote":"Introduces Masked Hard Instance Mining, the masking procedure the paper adapts to attention-based depression detection."},{"cited_title":"A text classification framework for simple and effective early depression detection over social media streams,","cited_arxiv_id":null,"evidence_quote":"Supplies the Thai-Maywe dataset, including the Importance Sentence Labels used for Recall@k explainability evaluation."},{"cited_title":"Real-time acoustic based depression detection using machine learning techniques,","cited_arxiv_id":null,"evidence_quote":"Supplies the English DAIC-WOZ corpus with PHQ-8 labels used to test the method on a second language and dataset."},{"cited_title":"Detecting linguistic traces of depression in topic-restricted text: Attending to self-stigmatized depression with nlp,","cited_arxiv_id":null,"evidence_quote":"Provides the attention-based Multiple Instance Learning formulation that frames dialogue turns as a bag of instances."},{"cited_title":"Multi instance neural networks,","cited_arxiv_id":null,"evidence_quote":"Provides the Single Encoder architecture used as an additional baseline to show MHIM's benefit is not limited to the Dual Encoder."}],"review_version":1}