{"id":"ec97eb58-a788-454e-a52a-390961458a6e","arxiv_id":"2501.09687","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"U-Fair, a gender-specific uncertainty reweighting for multitask depression detection, improves fairness over an uncertainty baseline but not consistently over unitask or vanilla multitask, and the PHQ-8 difficulty link is weakly supported.","lead":"This paper tests a multitask depression detector in which the eight PHQ-8 questionnaire items are separate prediction tasks and per-gender uncertainty weights rescale the losses. It reports fairness gains on one dataset, mixed results on another, and a claimed match between model task difficulty and clinical item discrimination that is not statistically supported.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Learned per-gender uncertainty weights are the linchpin, but Table 5 contradicts the claimed PHQ-8 alignment: de la Torre's top-3 DC items are PHQ-1,2,6 while 1/sigma^2 top-3 are PHQ-1,2,8, with no seeds or confidence intervals to stabilize either the sigma estimates or the fairness ratios.","rationale":"The reader's weakest assumption pointed at the reliability of sigma^s_t, and this is indeed the load-bearing element of the paper: the proposed reweighting, the fairness improvements, and the claimed alignment with de la Torre et al. all depend on these uncertainty estimates. I agree with the reader that small samples and single runs make these estimates suspect. However, the more precise and damaging issue is that the paper's own Table 5 contradicts the claimed alignment: the top-3 items by 1/sigma^2 are PHQ-1, 2, and 8, while de la Torre's top-3 are PHQ-1, 2, and 6. The text explicitly says PHQ-1,2,6 are the most discriminative and then claims alignment with 'our results where PHQ-1,2 and 8 are easier,' which is internally inconsistent. This is not merely a missing statistical test; the qualitative pattern the authors claim does not appear to hold. Because the same sigma estimates are used to justify the U-Fair loss, this inconsistency also undermines the fairness claims: if the uncertainty weights do not track meaningful task difficulty differences, the mechanism has no principled grounding. I additionally note that Eq. 3 (Unitask) and Eq. 4 with w_t=1 (Multitask) are algebraically identical, yet Tables 2/3 show different numbers, which suggests a mis-specified baseline or an unreported difference in training; this is a secondary but real concern for the 'multitask helps' narrative. The conditional verdict is still appropriate because the paper can be revised by adding multi-seed experiments, paired tests, and an honest assessment of the alignment (or by removing the alignment claim). A rejection would be too strong since the underlying MTL+fairness question is worth investigating and the method is not logically impossible. Therefore the reader's CONDITIONAL verdict remains unchanged, with the additional requirement that the PHQ-8 alignment claim be either statistically supported or removed.","tokens_in":19108,"tokens_out":6865,"duration_ms":66636,"concrete_test":"Run each method (Unitask, Multitask, Baseline UW, U-Fair) for at least 10 random seeds and report the mean and 95% CI for (i) the learned sigma^s_t for each gender/task, (ii) the four fairness ratios MSP, MEOpp, MEOdd, MEAcc, and (iii) the binary F1. Then compute a Spearman rank correlation between the de la Torre DC column and each of the four 1/sigma^2 columns in Table 5, and also compare the top-3 sets. If U-Fair does not significantly beat Baseline UW on a paired test over seeds (e.g., Wilcoxon signed-rank), or if the Spearman rho is not significant at p<0.05 (or the top-3 overlap is not better than chance), the headline claims of fairness improvement and PHQ-8 alignment should be weakened or withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that U-Fair (Eq. 6) improves fairness and alleviates negative transfer rests entirely on the learned per-gender noise parameters sigma^s_t being reliable, stable estimates of task difficulty. These are estimated from single training runs on small gender subgroups (DAIC-WOZ: 100 males/85 females; E-DAIC: 168 males/103 females, with smaller test splits). The paper reports no multi-seed variance, no confidence intervals, and no statistical test for the fairness ratios, many of which are extreme (e.g., Table 7 Baseline UW MEOdd = 8.21) and are ratios of small counts. The internal evidence is more damning: Section 5.3 says de la Torre et al.'s most discriminative items are PHQ-1,2,6, and claims agreement with U-Fair's results, but Table 5 shows the highest 1/sigma^2 values are PHQ-1,2,8 in every column, while PHQ-6 has low 1/sigma^2 values (0.59–0.73). Thus the top-3 sets disagree on 2 of 3 items, and the explicit 'alignment' claim is contradicted by the paper's own numbers. Since the same sigma estimates drive both the fairness claims and the difficulty-alignment claim, the observed mismatch is concrete evidence that the uncertainty weights may be dominated by noise rather than reflecting population-level item discrimination. Additionally, the unitask formulation in Eq. 3 is mathematically identical to the multitask loss with w_t=1 (Eq. 4), yet Tables 2 and 3 report different results, indicating an unexplained discrepancy in the baseline definitions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether multitask learning over the eight PHQ-8 subitems improves or worsens performance and gender fairness in depression detection. The authors propose U-Fair, a gender-conditioned extension of Kendall et al.'s (2018) uncertainty-weighted multitask loss, which learns per-gender, per-task aleatoric uncertainty weights sigma_t^s and reweights each gender's task losses (Eq. 6). Experiments on DAIC-WOZ and E-DAIC compare unitask, vanilla MTL (w_t = 1), uncertainty-weighted MTL (UW), and U-Fair on accuracy, F1, recall, UAR, and four ratio-based fairness measures. The paper reports that vanilla MTL improves over unitask on DAIC-WOZ but shows negative transfer on E-DAIC; that U-Fair improves fairness and the fairness-accuracy Pareto frontier relative to UW; and that per-task difficulties 1/sigma^2 align with the PHQ-8 item discrimination capacities reported by de la Torre et al. (2023).","tokens_in":19479,"tokens_out":16976,"duration_ms":148335,"significance":"The U-Fair idea, using gender-conditioned aleatoric uncertainty to reweight the PHQ-8 subtask losses, is simple, principled, and well aligned with the clinician's summation-based scoring procedure; if the fairness gains are confirmed with proper uncertainty quantification, it would be a useful contribution to machine learning for mental health. The paper also provides unusually detailed per-item, per-fairness-measure result tables (Tables 6-7) and explicitly reports negative transfer and a reduced Pareto frontier, which are important and often omitted phenomena in this literature. The most novel claim, that learned task difficulties match the psychometric discrimination capacities of the PHQ-8 items, is attractive but is contradicted by the paper's own Table 5 as it currently stands, so the significance of the work hinges on the re-analysis called for below.","major_comments":[{"comment":"The claimed agreement with de la Torre et al. (2023) is contradicted by the numbers in Table 5: de la Torre's three most discriminative items are PHQ-1, PHQ-2, and PHQ-6 (DC 3.06, 3.42, 2.86), whereas the highest 1/sigma^2 values in all four columns (DW-F, DW-M, ED-F, ED-M) are PHQ-1, PHQ-2, and PHQ-8. PHQ-6 has near-bottom 1/sigma^2 values (0.59-0.73) in every column, and PHQ-8, ranked fifth of eight by discrimination capacity, is the easiest item in three of four columns. The sentence in Section 5.3 that these results are 'in alignment' while listing two different top-3 sets is internally inconsistent. Please replace this qualitative claim with a rank-correlation analysis (e.g., Spearman rho between DC and 1/sigma^2 in each column) and report per-item agreement; if the mismatch persists, the claim should be substantially weakened.","section":"5.3, Table 5"},{"comment":"The unitask loss in Eq. (3) and the multitask loss in Eq. (4) with w_t = 1 are the same sum of per-task KL losses, yet Tables 2 and 3 report different unitask and multitask results (e.g., DAIC-WOZ accuracy 0.66 vs 0.70; E-DAIC F1 0.51 vs 0.45). Moreover, Eq. (4) says L_t is 'the single task loss ... as defined in Equation 3,' but Eq. (3) already contains a sum over t, so L_t is ambiguous if read literally. Please state precisely how the unitask model differs from the multitask baseline (e.g., independent per-task training, a direct binary prediction head, or a different backbone-sharing scheme); as written, the negative-transfer and MTL-versus-STL comparisons that motivate the paper are not interpretable.","section":"3.3-3.4, Tables 2-3"},{"comment":"All results come from a single training run per condition, with no standard deviations, confidence intervals, or significance tests, and the fairness ratios are computed on small test partitions (the datasets contain roughly 85 and 103 females in total, with smaller test splits). Extreme ratios such as E-DAIC MEOdd = 8.21 for UW and 5.00 for U-Fair are consistent with small-count instability. The claims in Section 5.2 that U-Fair improved fairness 'across all fairness measures' and improved the Pareto frontier (Figs. 2-3) require multi-seed runs (at least five) reported as mean +/- standard deviation with a bootstrap or paired significance test on the fairness metrics.","section":"5, Tables 2-3 and 6-7"},{"comment":"The fairness improvement is benchmarked against the baseline UW, but relative to the unitask approach U-Fair is farther from parity on E-DAIC for MSP (0.65 to 1.67) and MEOdd (0.75 to 5.00). The abstract's unqualified statement that U-Fair 'improves performance and fairness' is therefore too strong; please qualify the claim by dataset and by fairness measure and state the comparison baseline explicitly.","section":"5.2, Tables 2-3"},{"comment":"The evidence for negative transfer on E-DAIC is metric-dependent: multitask accuracy (0.58) and UAR (0.67) are higher than unitask (0.55 and 0.63), while F1 and recall are lower. Please state explicitly on which performance metrics negative transfer is observed, or restrict the claim that 'a unitask approach seems to perform better' to those metrics.","section":"5.1, Table 3"}],"minor_comments":[{"comment":"The text says 'A PHQ-8 TS of >= 10 belongs to the depressed class (Y=1) whereas TS <= 10 belongs to the non-depressed class (Y=0)'; with integer scores, TS = 10 is assigned to both classes, so the second condition should be TS < 10.","section":"3.1"},{"comment":"Please specify how the per-gender task loss L_t^s is computed (e.g., restricted to gender-s samples in the batch) and state whether gender information is used during inference; Eq. (6) as written only defines the training objective.","section":"3.6, Eq. (6)"},{"comment":"The claim that the baseline UW 'managed to further improve performance' holds for accuracy but not for F1 (UW drops F1 from 0.53 to 0.29 on DAIC-WOZ); performance claims should be reported per metric.","section":"5.1, Tables 2-3"},{"comment":"Only overall gender counts are given for DAIC-WOZ and E-DAIC; please report the train/dev/test counts per gender so that the stability of the fairness ratios can be assessed.","section":"Appendix A.1"},{"comment":"The text says the fairness results are normalised, but the axes appear to use raw ratio values; please explain how values such as 1.7 or 8.21 are mapped onto the displayed range.","section":"Figures 2-3"},{"comment":"Because the fairness measures are defined as ratios P(s0)/P(s1), values above and below 1 are both deviations from parity; please evaluate distance from parity with |log(ratio)| rather than raw proximity so that 1.67 and 0.60 are treated symmetrically.","section":"Section 4.2"},{"comment":"The statement that PHQ-3 and PHQ-5 are the two most difficult items holds in three of four columns; in DW-M the lowest 1/sigma^2 is PHQ-6 (0.59), so the statement should be qualified.","section":"5.3, Table 5"},{"comment":"There are several typos and grammatical slips: 'appraoch' (Section 3.5), 'using a a gender-based' (ML for Healthcare Implication paragraph), 'has the greatest ability' (Section 5.3), and 'results in different PHQ-8 distribution' (Section 1).","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is the published PMLR/ML4H 2024 proceedings version of this work; if the submission venue is a journal, the authors should clarify in a cover letter what has been added beyond the conference version, since the current text does not state any extension. The authors also cite their own prior work (Cheong et al., 2021-2024) extensively in the literature review; this is defensible given the topic, but the novelty of U-Fair relative to FairRefuse (Cheong et al., 2024a) and to the uncertainty-as-fairness line (Kuzucu et al., 2024) should be made explicit. My recommendation is driven by substantive but fixable issues in the evidence presented in Tables 2-5; a careful revision, especially the re-analysis of Section 5.3 and the addition of multi-seed results, could bring the manuscript to an acceptable standard."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plain take: the paper has a sensible new framing and a clean loss, but its headline claims are stronger than its own numbers. Per-gender uncertainty reweighting on PHQ-8 subtasks is a reasonable idea for depression detection, and showing that vanilla MTL can hurt fairness is a useful, non-obvious observation. But Section 5.3's 'agreement with de la Torre' is contradicted by Table 5, and the baseline definitions don't survive close reading.\n\nWhat's new and what works: treating each of the eight PHQ-8 subitems as an MTL task is a clinically grounded framing I have not seen elsewhere for this problem. U-Fair (Eq. 6) is a simple per-group application of Kendall et al.'s uncertainty weighting; not a new loss family, but the combination is untested and cheap to try. The systematic comparison across unitask, vanilla MTL, uncertainty weighting, and U-Fair with four fairness measures is a reasonable empirical contribution, and the negative-transfer warning on E-DAIC is worth heeding.\n\nSoft spots, roughly in order of severity. First, the alignment claim. Section 5.3 says de la Torre et al.'s most discriminative items are PHQ-1, 2 and 6, and calls that 'in alignment' with U-Fair's results; Table 5 shows the paper's easiest items are PHQ-1, 2 and 8 in every column, with PHQ-6 near the bottom (0.59 in the male DAIC-WOZ column). The top-3 sets disagree on two of three items, and the abstract advertises this as 'the very first tangible evidence' linking ML findings to population studies. That claim is contradicted by the paper's own table. Second, the loss definitions. Equation 3 sums the per-task KL losses, making unitask mathematically identical to the multitask loss in Eq. 4 with w_t = 1; yet Tables 2 and 3 report different results for those rows. The paper never defines the unitask baseline it actually ran. Third, all results are single runs. The per-gender sigma estimates come from roughly 85-103 females per dataset, no seeds, no confidence intervals, no significance tests, and the fairness ratios are extreme (baseline MEOdd of 8.21 improving to 5.00). On E-DAIC, U-Fair is farther from parity than the unitask baseline on MSP and MEOdd, so the 'fairer' claim only holds against the very unfair uncertainty-weighting baseline. No code is released. The self-citation pattern is heavy, but the central claims don't rest on those works, so I don't see circularity.\n\nWho it's for: ML4Health and affective-computing readers interested in fairness under small clinical cohorts. The question is important enough and the method plausible enough that a serious editor should send it to reviewers; the revision ask is substantial—fix or drop the de la Torre claim, clarify the unitask baseline, add multi-seed runs with intervals, and compare against FairRefuse and Zanna et al. I would not cite it in its current form.","headline":"A plausible per-gender uncertainty reweighting idea for depression detection whose headline alignment claim is contradicted by its own Table 5 and whose baselines don't survive close reading.","tokens_in":20012,"tokens_out":13995,"would_cite":false,"duration_ms":118877,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that gender-based uncertainty reweighting of the eight PHQ-8 subtasks improves the fairness-accuracy trade-off in depression detection and links learned task difficulty to clinical discrimination capacity.","keywords":["multitask learning","fairness","depression detection","PHQ-8","aleatoric uncertainty","gender bias","multimodal learning","negative transfer"],"falsifier":"Re-run U-Fair and the uncertainty-weighted baseline on the same DAIC-WOZ and E-DAIC splits with multiple random seeds and cross-validation, then check whether the per-gender sigma ordering, such as PHQ-3 and PHQ-5 being hardest, and the direction of the fairness ratios reproduce. If the item-difficulty ranking flips between seeds or the fairness ratios do not move closer to 1 than the baseline, the paper's central claims are not supported.","tokens_in":18918,"feed_emoji":"⚖️","tokens_out":7211,"duration_ms":63209,"temperature":0.7,"pith_summary":"This paper tries to establish that multitask learning can be made fairer for depression detection by treating each of the eight PHQ-8 questionnaire items as a separate task and weighting those tasks by per-gender uncertainty estimates. The authors argue that, compared with predicting a single binary depressed/not-depressed label, a multitask approach generally improves performance and fairness, but a naive version can suffer negative transfer and shrink the accuracy-fairness Pareto frontier, which is a serious problem in healthcare. They propose U-Fair, which replaces the usual task weighting with gender-specific aleatoric uncertainty weights, and report that it improves fairness metrics while mostly preserving accuracy, and that the learned task difficulties align with the item discrimination capacity found in the largest PHQ-8 population study. If these results hold, the method offers a clinician-grounded way to reduce gender bias in automated depression screening, and it provides a bridge between machine-learned task difficulty and established psychometric evidence.","feed_headline":"Per-gender uncertainty weights make PHQ-8 depression screening fairer","feed_subtitle":"Per-gender uncertainty weights improve the fairness-accuracy trade-off and echo clinical PHQ-8 findings","key_machinery":"The load-bearing object is the gender-based aleatoric uncertainty reweighting loss in Equation (6): LU-Fair = (1/|S|) sum_s sum_t (1/($\\sigma$^s_t)^2) L^s_t + log $\\sigma$^s_t, where s indexes gender and t indexes the eight PHQ-8 items. Each $\\sigma$^s_t is a learned parameter that the model can raise to downweight a task that is noisy or difficult for that gender, while the log $\\sigma$ term stops it from inflating $\\sigma$ arbitrarily. This is the same uncertainty-weighting idea used in the Kendall et al. baseline, but applied separately per gender so that the joint representation can adapt to male/female differences in PHQ-8 symptom profiles; the mechanism also yields the per-gender task-difficulty estimates that are compared with clinical discrimination capacity.","core_discovery":"On the paper's own terms, the central discovery is that the fairness-accuracy trade-off in multimodal depression detection can be improved by reweighting each PHQ-8 subitem task separately for males and females using learned aleatoric uncertainty values. With tasks t=1..8 corresponding to the eight items, U-Fair's loss sums per-gender terms 1/($\\sigma$^s_t)^2 L^s_t + log $\\sigma$^s_t, so difficult or noisy items are downweighted and the model is not forced to sacrifice accuracy for parity. The authors show that this recovers performance and fairness where a plain multitask baseline fails: on DAIC-WOZ, U-Fair raises F1 from 0.29 to 0.54 relative to the uncertainty-weighted baseline while bringing four fairness ratios closer to 1; on E-DAIC it improves the Pareto frontier, especially on equal opportunity. They also report that the ordering of task difficulty from the learned $\\sigma$ values matches the discrimination capacity of PHQ-8 items reported in de la Torre et al. (2023), calling this the first tangible link between ML task-difficulty findings and large-scale population evidence on PHQ-8.","pith_inferences":["If the per-gender sigma values are stable across runs, the same approach could be applied to other clinician-administered questionnaires, such as PHQ-9 or GAD-7, by treating each item as a task and using demographic-group uncertainty.","The paper only uses gender as the sensitive attribute; an obvious extension is intersectional groups such as gender by age or race, which would require datasets larger than DAIC-WOZ and E-DAIC.","A stronger test of the PHQ-8 link would be to compare learned task difficulties against item response theory parameters from independent population samples; if the ordering reproduces, uncertainty estimates could serve as cheap psychometric proxies.","Repeated-seed and cross-validated runs are needed to determine whether the per-gender sigma values in Table 5 are signal or noise; until then the difficulty-fairness link is suggestive rather than established."],"forward_implications":["Treating the eight PHQ-8 items as separate tasks can outperform single-label depression prediction, but only if task weights are handled carefully; naive multitask learning risks negative transfer.","Gender-based uncertainty reweighting makes the fairness metrics (statistical parity, equal opportunity, equalized odds, equal accuracy) move toward 1 on both DAIC-WOZ and E-DAIC while keeping accuracy competitive.","The learned ordering of task difficulty, with PHQ-3 and PHQ-5 hardest and PHQ-1, PHQ-2, and PHQ-8 easiest, agrees with the discrimination capacity reported in the largest PHQ-8 population study.","Improvements to the fairness-accuracy Pareto frontier are more visible on the harder E-DAIC dataset, suggesting the benefit grows when the unitask baseline is weaker."],"supporting_citations":[{"why":"Supplies the uncertainty-based multitask weighting formula (1/sigma^2 L + log sigma) that U-Fair adapts per gender.","marker":"Kendall et al. (2018)"},{"why":"Provides the large-scale PHQ-8 item discrimination-capacity results that the paper's learned task difficulties are compared against.","marker":"de la Torre et al. (2023)"},{"why":"Defines the PHQ-8 questionnaire and its eight subitems, which form the task structure of the method.","marker":"Kroenke et al. (2009)"},{"why":"Supplies the attention-based multimodal fusion architecture with eight output heads used in all experiments.","marker":"Wei et al. (2022)"},{"why":"Provides the DAIC-WOZ dataset with per-subitem PHQ-8 ground truth and train/validation/test splits.","marker":"Valstar et al. (2016)"},{"why":"Provides the E-DAIC corpus, the second and harder benchmark used to test generalization.","marker":"Ringeval et al. (2019)"},{"why":"Frames the fairness-accuracy trade-off in multitask learning that U-Fair is designed to improve.","marker":"Wang et al. (2021b)"},{"why":"Prior bias-reducing multitask learning for mental health and the source of the 0.80/1.20 fairness bounds used in evaluation.","marker":"Zanna et al. (2022)"}],"fun_headline_variants":["Uncertainty-weighted multitask learning cuts depression screening bias","Gender-specific uncertainty weights improve depression detection fairness","U-Fair reweights tasks by uncertainty for fairer depression screening","New method aligns ML task difficulty with PHQ-8 clinical findings","Uncertainty-based reweighting makes depression AI fairer and accurate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that the learned per-gender uncertainty weights $\\sigma$^s_t in Equation (6) reliably measure how difficult each PHQ-8 item is for each gender; if those weights are unstable noise, the fairness gains and the agreement with clinical discrimination capacity collapse.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty-weighted multitask learning cuts depression screening bias","Gender-specific uncertainty weights improve depression detection fairness","U-Fair reweights tasks by uncertainty for fairer depression screening","New method aligns ML task difficulty with PHQ-8 clinical findings","Uncertainty-based reweighting makes depression AI fairer and accurate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000911,"raw_usage":{"total_tokens":3943,"prompt_tokens":1004,"completion_tokens":2939,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":2854}},"tokens_in":620,"tokens_out":2939,"duration_ms":19840,"temperature":1.0,"reasoning_tokens":2854,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:45:22.861161+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run U-Fair and the uncertainty-weighted baseline on the same DAIC-WOZ and E-DAIC splits with multiple random seeds and cross-validation, then check whether the per-gender sigma ordering, such as PHQ-3 and PHQ-5 being hardest, and the direction of the fairness ratios reproduce. If the item-difficulty ranking flips between seeds or the fairness ratios do not move closer to 1 than the baseline, the paper's central claims are not supported.","supporting_citations":[{"cited_title":"Avec 2016: Depression, mood, and emotion recognition workshop and chal- lenge","cited_arxiv_id":null,"evidence_quote":"Provides the DAIC-WOZ dataset with per-subitem PHQ-8 ground truth and train/validation/test splits."}],"review_version":1}