REVIEW 5 major objections 8 minor 17 references
U-Fair: Uncertainty-based Multimodal Multitask Learning for Fairer Depression Detection
T0 review · 5 major / 8 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims that gender-based uncertainty reweighting of the eight PHQ-8 subtasks improves the fairness-accuracy trade-off in depression detection and links learned task difficulty to clinical discrimination capacity.
desk verdict A plausible per-gender uncertainty reweighting idea for depression detection whose headline alignment claim is contradicted by its own Table 5 and whose baselines don't survive close reading. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the gender-based aleatoric uncertainty reweighting loss in Equation (6): LU-Fair = (1/|S|) sum_s sum_t (1/($\sigma$^s_t)^2) L^s_t + log $\sigma$^s_t, where s indexes gender and t indexes the eight PHQ-8 items. Each $\sigma$^s_t is a learned parameter that the model can raise to downweight a task that is noisy or difficult for that gender, while the log $\sigma$ term stops it from inflating $\sigma$ arbitrarily. This is the same uncertainty-weighting idea used in the Kendall et al. baseline, but applied separately per gender so that the joint representation can adapt to male/female differences in PHQ-8 symptom profiles; the mechanism also yields the per-gender task-difficulty estimates that are compared with clinical discrimination capacity.
What would settle it
Re-run U-Fair and the uncertainty-weighted baseline on the same DAIC-WOZ and E-DAIC splits with multiple random seeds and cross-validation, then check whether the per-gender sigma ordering, such as PHQ-3 and PHQ-5 being hardest, and the direction of the fairness ratios reproduce. If the item-difficulty ranking flips between seeds or the fairness ratios do not move closer to 1 than the baseline, the paper's central claims are not supported.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that the fairness-accuracy trade-off in multimodal depression detection can be improved by reweighting each PHQ-8 subitem task separately for males and females using learned aleatoric uncertainty values. With tasks t=1..8 corresponding to the eight items, U-Fair's loss sums per-gender terms 1/($\sigma$^s_t)^2 L^s_t + log $\sigma$^s_t, so difficult or noisy items are downweighted and the model is not forced to sacrifice accuracy for parity. The authors show that this recovers performance and fairness where a plain multitask baseline fails: on DAIC-WOZ, U-Fair raises F1 from 0.29 to 0.54 relative to the uncertainty-weighted baseline while bringing four fairness ratios closer to 1; on E-DAIC it improves the Pareto frontier, especially on equal opportunity. They also report that the ordering of task difficulty from the learned $\sigma$ values matches the discrimination capacity of PHQ-8 items reported in de la Torre et al. (2023), calling this the first tangible link between ML task-difficulty findings and large-scale population evidence on PHQ-8.
Load-bearing premise
The central assumption is that the learned per-gender uncertainty weights $\sigma$^s_t in Equation (6) reliably measure how difficult each PHQ-8 item is for each gender; if those weights are unstable noise, the fairness gains and the agreement with clinical discrimination capacity collapse.
Editorial extensions
If this is right
- Treating the eight PHQ-8 items as separate tasks can outperform single-label depression prediction, but only if task weights are handled carefully; naive multitask learning risks negative transfer.
- Gender-based uncertainty reweighting makes the fairness metrics (statistical parity, equal opportunity, equalized odds, equal accuracy) move toward 1 on both DAIC-WOZ and E-DAIC while keeping accuracy competitive.
- The learned ordering of task difficulty, with PHQ-3 and PHQ-5 hardest and PHQ-1, PHQ-2, and PHQ-8 easiest, agrees with the discrimination capacity reported in the largest PHQ-8 population study.
- Improvements to the fairness-accuracy Pareto frontier are more visible on the harder E-DAIC dataset, suggesting the benefit grows when the unitask baseline is weaker.
Reading between the lines
- If the per-gender sigma values are stable across runs, the same approach could be applied to other clinician-administered questionnaires, such as PHQ-9 or GAD-7, by treating each item as a task and using demographic-group uncertainty.
- The paper only uses gender as the sensitive attribute; an obvious extension is intersectional groups such as gender by age or race, which would require datasets larger than DAIC-WOZ and E-DAIC.
- A stronger test of the PHQ-8 link would be to compare learned task difficulties against item response theory parameters from independent population samples; if the ordering reproduces, uncertainty estimates could serve as cheap psychometric proxies.
- Repeated-seed and cross-validated runs are needed to determine whether the per-gender sigma values in Table 5 are signal or noise; until then the difficulty-fairness link is suggestive rather than established.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies whether multitask learning over the eight PHQ-8 subitems improves or worsens performance and gender fairness in depression detection. The authors propose U-Fair, a gender-conditioned extension of Kendall et al.'s (2018) uncertainty-weighted multitask loss, which learns per-gender, per-task aleatoric uncertainty weights sigma_t^s and reweights each gender's task losses (Eq. 6). Experiments on DAIC-WOZ and E-DAIC compare unitask, vanilla MTL (w_t = 1), uncertainty-weighted MTL (UW), and U-Fair on accuracy, F1, recall, UAR, and four ratio-based fairness measures. The paper reports that vanilla MTL improves over unitask on DAIC-WOZ but shows negative transfer on E-DAIC; that U-Fair improves fairness and the fairness-accuracy Pareto frontier relative to UW; and that per-task difficulties 1/sigma^2 align with the PHQ-8 item discrimination capacities reported by de la Torre et al. (2023).
Significance. The U-Fair idea, using gender-conditioned aleatoric uncertainty to reweight the PHQ-8 subtask losses, is simple, principled, and well aligned with the clinician's summation-based scoring procedure; if the fairness gains are confirmed with proper uncertainty quantification, it would be a useful contribution to machine learning for mental health. The paper also provides unusually detailed per-item, per-fairness-measure result tables (Tables 6-7) and explicitly reports negative transfer and a reduced Pareto frontier, which are important and often omitted phenomena in this literature. The most novel claim, that learned task difficulties match the psychometric discrimination capacities of the PHQ-8 items, is attractive but is contradicted by the paper's own Table 5 as it currently stands, so the significance of the work hinges on the re-analysis called for below.
major comments (5)
- [5.3, Table 5] The claimed agreement with de la Torre et al. (2023) is contradicted by the numbers in Table 5: de la Torre's three most discriminative items are PHQ-1, PHQ-2, and PHQ-6 (DC 3.06, 3.42, 2.86), whereas the highest 1/sigma^2 values in all four columns (DW-F, DW-M, ED-F, ED-M) are PHQ-1, PHQ-2, and PHQ-8. PHQ-6 has near-bottom 1/sigma^2 values (0.59-0.73) in every column, and PHQ-8, ranked fifth of eight by discrimination capacity, is the easiest item in three of four columns. The sentence in Section 5.3 that these results are 'in alignment' while listing two different top-3 sets is internally inconsistent. Please replace this qualitative claim with a rank-correlation analysis (e.g., Spearman rho between DC and 1/sigma^2 in each column) and report per-item agreement; if the mismatch persists, the claim should be substantially weakened.
- [3.3-3.4, Tables 2-3] The unitask loss in Eq. (3) and the multitask loss in Eq. (4) with w_t = 1 are the same sum of per-task KL losses, yet Tables 2 and 3 report different unitask and multitask results (e.g., DAIC-WOZ accuracy 0.66 vs 0.70; E-DAIC F1 0.51 vs 0.45). Moreover, Eq. (4) says L_t is 'the single task loss ... as defined in Equation 3,' but Eq. (3) already contains a sum over t, so L_t is ambiguous if read literally. Please state precisely how the unitask model differs from the multitask baseline (e.g., independent per-task training, a direct binary prediction head, or a different backbone-sharing scheme); as written, the negative-transfer and MTL-versus-STL comparisons that motivate the paper are not interpretable.
- [5, Tables 2-3 and 6-7] All results come from a single training run per condition, with no standard deviations, confidence intervals, or significance tests, and the fairness ratios are computed on small test partitions (the datasets contain roughly 85 and 103 females in total, with smaller test splits). Extreme ratios such as E-DAIC MEOdd = 8.21 for UW and 5.00 for U-Fair are consistent with small-count instability. The claims in Section 5.2 that U-Fair improved fairness 'across all fairness measures' and improved the Pareto frontier (Figs. 2-3) require multi-seed runs (at least five) reported as mean +/- standard deviation with a bootstrap or paired significance test on the fairness metrics.
- [5.2, Tables 2-3] The fairness improvement is benchmarked against the baseline UW, but relative to the unitask approach U-Fair is farther from parity on E-DAIC for MSP (0.65 to 1.67) and MEOdd (0.75 to 5.00). The abstract's unqualified statement that U-Fair 'improves performance and fairness' is therefore too strong; please qualify the claim by dataset and by fairness measure and state the comparison baseline explicitly.
- [5.1, Table 3] The evidence for negative transfer on E-DAIC is metric-dependent: multitask accuracy (0.58) and UAR (0.67) are higher than unitask (0.55 and 0.63), while F1 and recall are lower. Please state explicitly on which performance metrics negative transfer is observed, or restrict the claim that 'a unitask approach seems to perform better' to those metrics.
minor comments (8)
- [3.1] The text says 'A PHQ-8 TS of >= 10 belongs to the depressed class (Y=1) whereas TS <= 10 belongs to the non-depressed class (Y=0)'; with integer scores, TS = 10 is assigned to both classes, so the second condition should be TS < 10.
- [3.6, Eq. (6)] Please specify how the per-gender task loss L_t^s is computed (e.g., restricted to gender-s samples in the batch) and state whether gender information is used during inference; Eq. (6) as written only defines the training objective.
- [5.1, Tables 2-3] The claim that the baseline UW 'managed to further improve performance' holds for accuracy but not for F1 (UW drops F1 from 0.53 to 0.29 on DAIC-WOZ); performance claims should be reported per metric.
- [Appendix A.1] Only overall gender counts are given for DAIC-WOZ and E-DAIC; please report the train/dev/test counts per gender so that the stability of the fairness ratios can be assessed.
- [Figures 2-3] The text says the fairness results are normalised, but the axes appear to use raw ratio values; please explain how values such as 1.7 or 8.21 are mapped onto the displayed range.
- [Section 4.2] Because the fairness measures are defined as ratios P(s0)/P(s1), values above and below 1 are both deviations from parity; please evaluate distance from parity with |log(ratio)| rather than raw proximity so that 1.67 and 0.60 are treated symmetrically.
- [5.3, Table 5] The statement that PHQ-3 and PHQ-5 are the two most difficult items holds in three of four columns; in DW-M the lowest 1/sigma^2 is PHQ-6 (0.59), so the statement should be qualified.
- [Throughout] There are several typos and grammatical slips: 'appraoch' (Section 3.5), 'using a a gender-based' (ML for Healthcare Implication paragraph), 'has the greatest ability' (Section 5.3), and 'results in different PHQ-8 distribution' (Section 1).
Circularity Check
Unitask and multitask baselines are identical by construction, undermining the MTL-improvement claim; the central U-Fair evaluation remains independent.
-
self definitional
[Section 3.3 (Eq. 3), Section 3.4 (Eq. 4), and results in Tables 2 and 3.]
"For our single task approach, we use a Kullback-Leibler (KL) Divergence loss as follows: L_STL = Σ_{t∈T} p_t(x) log(p_t(x)/q_t(x)). (3) ... For our baseline multitask approach, we extend the loss function in Equation 3 to arrive at the following generalisation: L_MTL = Σ_{t∈T} w_t L_t. (4) ... We set w_t = 1 in our experiments."
With w_t = 1, Eq. (4) is term-by-term identical to Eq. (3), so the paper's 'multitask' baseline is the same objective as its 'unitask' baseline by construction. Tables 2 and 3 nonetheless report different Unitask and Multitask results (e.g., DAIC-WOZ accuracy 0.66 vs 0.70; E-DAIC MSP 0.65 vs 1.25), and Section 5.1 claims that 'a multitask approach generally improves results compared to a unitask approach' and that there is evidence of negative transfer. Because the two losses are identical under the stated definitions, the observed differences cannot be attributed to multitask learning without an unstated implementation change, so the claimed MTL-versus-unitask comparison and the negative-transfer interpretation are not well-defined from the formalism as written.
full rationale
No load-bearing self-citation chain is present: the Cheong et al. references appear as background, motivation, or metric choices, and the central U-Fair claim is an empirical evaluation of a learned reweighting loss on standard benchmarks, not a result forced by the authors' prior theorems. The main contribution therefore does not reduce to its inputs. However, the unitask/multitask comparison is self-definitional: Eq. (4) with w_t = 1 is identically Eq. (3), yet Tables 2 and 3 report different numbers for the two rows, making the claimed multitask improvement and negative-transfer phenomenon not well-defined under the stated equations. The PHQ-8 difficulty-alignment claim in Section 5.3 is not circular, but it is internally inconsistent: the text states that de la Torre et al.'s most discriminative items are PHQ-1, 2 and 6 and claims agreement, while Table 5 shows the highest 1/sigma^2 values as PHQ-1, 2 and 8; this is a correctness risk rather than a construction-level circularity. Overall, there is partial circularity in one baseline comparison, but the central U-Fair evaluation retains independent content, so the score is moderate rather than high.
Assumptions & free parameters
free parameters (1)
- Gender-task aleatoric uncertainty weights sigma^s_t =
Learned during training; 1/sigma^2 values in Table 5 (e.g., 1.50 to 1.72 for PHQ-8, 0.51 to 0.64 for PHQ-3)
assumptions (3)
- domain assumption Treating the eight PHQ-8 subitems as independent tasks and summing model scores to form the binary outcome replicates the clinician's PHQ-8 protocol
- domain assumption The learned per-gender sigma^s_t capture stable aleatoric uncertainty differences rather than noise from small subgroups
- domain assumption Male/female is the only relevant sensitive attribute and the binary grouping is appropriate
Cite this review
Pith. "Pith review of U-Fair: Uncertainty-based Multimodal Multitask Learning for Fairer Depression Detection." pith.science (2026). https://pith.science/paper/ZROACQ34
@misc{pith2026250109687,
author = {Pith},
title = {Pith review of: U-Fair: Uncertainty-based Multimodal Multitask Learning for Fairer Depression Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZROACQ34}},
note = {Machine review of arXiv:2501.09687}
}
read the original abstract
Machine learning bias in mental health is becoming an increasingly pertinent challenge. Despite promising efforts indicating that multitask approaches often work better than unitask approaches, there is minimal work investigating the impact of multitask learning on performance and fairness in depression detection nor leveraged it to achieve fairer prediction outcomes. In this work, we undertake a systematic investigation of using a multitask approach to improve performance and fairness for depression detection. We propose a novel gender-based task-reweighting method using uncertainty grounded in how the PHQ-8 questionnaire is structured. Our results indicate that, although a multitask approach improves performance and fairness compared to a unitask approach, the results are not always consistent and we see evidence of negative transfer and a reduction in the Pareto frontier, which is concerning given the high-stake healthcare setting. Our proposed approach of gender-based reweighting with uncertainty improves performance and fairness and alleviates both challenges to a certain extent. Our findings on each PHQ-8 subitem task difficulty are also in agreement with the largest study conducted on the PHQ-8 subitem discrimination capacity, thus providing the very first tangible evidence linking ML findings with large-scale empirical population studies conducted on the PHQ-8.
Figures
Reference graph
Works this paper leans on
-
[5]
Multimodal gender fairness in de- pression prediction: Insights on data from the usa & china
Joseph Cameron, Jiaee Cheong, Micol Spitale, and Hatice Gunes. Multimodal gender fairness in de- pression prediction: Insights on data from the usa & china. arXiv preprint arXiv:2408.04026 ,
-
[7]
Emer- gent unfairness in algorithmic fairness-accuracy trade-off research
A Feder Cooper, Ellen Abrams, and Na Na. Emer- gent unfairness in algorithmic fairness-accuracy trade-off research. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Soci- ety, pages 46–54,
work page 2021
-
[8]
A systematic study of bias amplification
Melissa Hall, Laurens van der Maaten, Laura Gustafson, Maxwell Jones, and Aaron Adcock. A systematic study of bias amplification. arXiv preprint arXiv:2201.11706,
-
[9]
Bias mitigation for ma- chine learning classifiers: A comprehensive survey
Max Hort, Zhenpeng Chen, Jie M Zhang, Federica Sarro, and Mark Harman. Bias mitigation for ma- chine learning classifiers: A comprehensive survey. arXiv preprint arXiv:2207.07068 ,
-
[10]
Trustsleepnet: A trustable deep multimodal network for sleep stage classification
Guanjie Huang and Fenglong Ma. Trustsleepnet: A trustable deep multimodal network for sleep stage classification. In 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), pages 01–04. IEEE,
work page 2022
-
[13]
Siyang Song, Linlin Shen, and Michel Valstar. Human behaviour-based automatic depression analysis us- ing hand-crafted statistics and deep learned spec- tral features. In FG 2018 , pages 158–165. IEEE,
work page 2018
-
[14]
Micol Spitale, Jiaee Cheong, and Hatice Gunes. Un- derneath the numbers: Quantitative and qualita- tive gender fairness in llms for depression predic- tion. arXiv preprint arXiv:2406.08183 ,
-
[16]
Fair classi- fication with group-dependent label noise
Jialu Wang, Yang Liu, and Caleb Levy. Fair classi- fication with group-dependent label noise. In Pro- ceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages 526–536, 2021a. Jingying Wang, Lei Zhang, Tianli Liu, Wei Pan, Bin Hu, and Tingshao Zhu. Acoustic differences between healthy and depressed people: a cross- situatio...
work page 2021
Show all 17 references
-
[2001]
Model multiplicity: Opportunities, concerns, and solutions
Emily Black, Manish Raghavan, and Solon Barocas. Model multiplicity: Opportunities, concerns, and solutions. In Proceedings of the 2022 ACM Con- ference on Fairness, Accountability, and Trans- parency, pages 850–863,
2022
-
[2014]
Avec 2016: Depression, mood, and emotion recognition workshop and chal- lenge
Michel Valstar, Jonathan Gratch, Bj¨ orn Schuller, Fabien Ringeval, Denis Lalanne, Mercedes Tor- res Torres, Stefan Scherer, Giota Stratou, Roddy Cowie, and Maja Pantic. Avec 2016: Depression, mood, and emotion recognition workshop and chal- lenge. pages 3–10,
2016
-
[2016]
Investigating bias and fairness in fa- cial expression recognition
Tian Xu, Jennifer White, Sinan Kalkan, and Hat- ice Gunes. Investigating bias and fairness in fa- cial expression recognition. In Computer Vision– ECCV 2020 Workshops: Glasgow, UK, August 23– 28, 2020, Proceedings, Part VI 16 , pages 506–523. Springer,
2020
-
[2018]
Gender bias in depression detection using audio features
Andrew Bailey and Mark D Plumbley. Gender bias in depression detection using audio features. EU- SIPCO 2021 ,
2021
-
[2020]
Multi-task learning with dynamic re-weighting to achieve fairness in healthcare pre- dictive modeling
Can Li, Sirui Ding, Na Zou, Xia Hu, Xiaoqian Jiang, and Kai Zhang. Multi-task learning with dynamic re-weighting to achieve fairness in healthcare pre- dictive modeling. Journal of Biomedical Informat- ics, 143:104399, 2023a. Can Li, Dejian Lai, Xiaoqian Jiang, and Kai Zhang. ...
-
[2021]
Class un- certainty: A measure to mitigate class imbalance
Zeynep Sonat Baltaci, Kemal Oksuz, Selim Kuzucu, Kivanc Tezoren, Berkin Kerim Konar, Alpay Ozkan, Emre Akbas, and Sinan Kalkan. Class un- certainty: A measure to mitigate class imbalance. arXiv preprint arXiv:2311.14090 ,
-
[2022]
it’s not fair!
Jiaee Cheong, Sinan Kalkan, and Hatice Gunes. Causal structure learning of bias for fair affect recognition. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vi- sion, pages 340–349, 2023a. Jiaee Cheong, Selim Kuzucu, Sinan Kalkan, and Hat- ice Gu...
2023
-
[2023]
Fair resource alloca- tion in multi-task learning
Hao Ban and Kaiyi Ji. Fair resource alloca- tion in multi-task learning. arXiv preprint arXiv:2402.15638,
-
[2024]
Multi-task learning for depression detection in di- alogs
Chuyuan Li, Chlo´ e Braud, and Maxime Amblard. Multi-task learning for depression detection in di- alogs. arXiv preprint arXiv:2208.10250 ,
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.