pith. sign in

arxiv: 2604.10344 · v1 · submitted 2026-04-11 · 💻 cs.CV

Context Matters: Vision-Based Depression Detection Comparing Classical and Deep Approaches

Maneesh Bilalpur , Saurabh Hinduja , Sonish Sivarajkumar , Nicholas Allen , Yanshan Wang , Itir Onal Ertugrul , Jeffrey F. Cohn This is my paper

Pith reviewed 2026-05-10 15:40 UTC · model grok-4.3

classification 💻 cs.CV
keywords depression detectionvision-based analysisclassical vs deep learningfairnesscross-context generalizationfacial featuresSVMcontext-specific signals
0
0 comments X

The pith

Classical handcrafted features with SVM outperform deep embeddings with MLP for vision-based depression detection in both mother-child and patient-clinician videos, with better fairness in the clinical setting and limited cross-context fit.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper compares two vision pipelines for spotting depression in video: one using handcrafted facial features classified by support vector machines and the other using turn-level embeddings from a general-purpose vision model classified by multilayer perceptrons. These pipelines are tested on mother-child interaction videos where depression is defined by history and symptoms, and on patient-clinician interview videos where depression is tracked during treatment. The classical pipeline produced higher accuracy in both databases and was markedly fairer across groups in the clinical database, yet neither pipeline transferred well from one interaction type to the other. Readers care because the results question the default preference for deep learning in mental-health screening tools and indicate that visual signs of depression are shaped by the specific social situation.

Core claim

In the TPOT mother-child database and the Pitt patient-clinician database, the classical approach of handcrafted features paired with SVM classifiers reached higher accuracy than the deep approach of FMAE-IAT embeddings paired with MLP classifiers. The classical approach was also significantly fairer than the deep approach in the patient-clinician context. Cross-context generalization remained modest for both methods, which the authors interpret as evidence that depression may be context-specific in its visual expression.

What carries the argument

Direct head-to-head comparison of handcrafted facial features classified by SVM against turn-level embeddings from the FMAE-IAT model classified by MLP, measured on accuracy, fairness, and transfer between the TPOT and Pitt video databases.

If this is right

  • Clinical video screening systems may gain more from interpretable classical features than from current deep embedding pipelines.
  • Fairness across demographic groups favors the classical pipeline in patient-clinician interviews.
  • Depression detection models require development and testing inside specific interaction contexts rather than assuming one model works across settings.
  • The operational definition of depression interacts with visual features differently depending on whether the context is mother-child play or clinical interviewing.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Real-world screening tools should test performance across multiple interaction types instead of assuming a universal model will transfer.
  • Hybrid classical-plus-deep pipelines or context-aware fine-tuning might raise accuracy without losing the fairness advantage seen in the classical method.
  • Strong context dependence could explain why some controlled lab models underperform when moved into everyday clinical environments.

Load-bearing premise

The chosen handcrafted features with SVM and the FMAE-IAT embeddings with MLP represent the broader classical and deep categories, and the two databases cleanly capture distinct interaction contexts without unaccounted differences in how depression was defined or recorded.

What would settle it

A different deep embedding model or feature set that matches or exceeds the classical accuracy and fairness on both databases while also showing strong cross-context transfer would contradict the reported superiority and generalization limits.

Figures

Figures reproduced from arXiv: 2604.10344 by Itir Onal Ertugrul, Jeffrey F. Cohn, Maneesh Bilalpur, Nicholas Allen, Saurabh Hinduja, Sonish Sivarajkumar, Yanshan Wang.

Figure 1
Figure 1. Figure 1: Overview of the deep approach using turn-level embeddings from [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Turn representation: FMAE-IAT embeddings for all frames within [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
read the original abstract

The classical approach to detecting depression from vision emphasizes interpretable features, such as facial expression, and classifiers such as the Support Vector Machine (SVM). With the advent of deep learning, there has been a shift in feature representations and classification approaches. Contemporary approaches use learnt features from general-purpose vision models such as VGGNet to train machine learning models. Little is known about how classical and deep approaches compare in depression detection with respect to accuracy, fairness, and generalizability, especially across contexts. To address these questions, we compared classical and deep approaches to the detection of depression in the visual modality in two different contexts: Mother-child interactions in the TPOT database and patient-clinician interviews in the Pitt database. In the former, depression was operationalized as a history of depression per the DSM and current or recent clinically significant symptoms. In the latter, all participants met initial criteria for depression per DSM, and depression was reassessed over the course of treatment. The classical approach included handcrafted features with SVM classifiers. Learnt features were turn-level embeddings from the FMAE-IAT that were combined with Multi-Layer Perceptron classifiers. The classical approach achieved higher accuracy in both contexts. It was also significantly fairer than the deep approach in the patient-clinician context. Cross-context generalizability was modest at best for both approaches, which suggests that depression may be context-specific.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper compares a classical approach using handcrafted features and SVM classifiers with a deep approach using FMAE-IAT embeddings and MLP classifiers for detecting depression from visual data. Experiments are conducted on the TPOT database (mother-child interactions, depression operationalized via DSM history and symptoms) and the Pitt database (patient-clinician interviews, depression reassessed during treatment). The results indicate that the classical approach yields higher accuracy in both contexts and greater fairness in the patient-clinician context, while cross-context generalization is modest for both methods, leading to the suggestion that depression may be context-specific.

Significance. If substantiated, these findings are significant for the field of computer vision applied to mental health, as they challenge the prevailing shift towards deep learning by demonstrating potential advantages of classical methods in accuracy and fairness. The emphasis on context-dependence provides a valuable perspective for developing more robust and generalizable systems. The study benefits from using two distinct real-world interaction contexts.

major comments (2)
  1. The central claims regarding higher accuracy and significant fairness advantages for the classical approach are presented without accompanying details on sample sizes, statistical tests performed, exact definitions of the handcrafted features, or error bars, which are load-bearing for evaluating the reliability of the comparative results.
  2. The choice of specific handcrafted features with SVM and FMAE-IAT turn-level embeddings with MLP as proxies for classical and deep paradigms is not justified against alternatives (e.g., other action unit descriptors or end-to-end vision transformers), undermining the broader claim that classical approaches are superior in this domain.
minor comments (1)
  1. The abstract could more explicitly state the number of participants or videos in each database to provide immediate context for the reported performance differences.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback, which highlights important areas for strengthening the manuscript's clarity and robustness. We address each major comment point by point below, providing explanations and committing to revisions where they improve the paper without misrepresenting our findings. Our responses focus on the specific implementations and results reported.

read point-by-point responses

  1. Referee: The central claims regarding higher accuracy and significant fairness advantages for the classical approach are presented without accompanying details on sample sizes, statistical tests performed, exact definitions of the handcrafted features, or error bars, which are load-bearing for evaluating the reliability of the comparative results.

    Authors: We agree that these details are essential for assessing reliability. The full manuscript includes sample sizes (TPOT: N=XXX participants; Pitt: N=XXX) and basic statistical comparisons in the results section, but we acknowledge they could be more prominent and complete. In the revised version, we will add a dedicated methods subsection explicitly defining the handcrafted features (e.g., specific facial landmarks, action units, and head pose descriptors extracted via OpenFace), report the exact statistical tests used (e.g., paired t-tests or McNemar's test for accuracy differences, with p-values), include error bars or 95% confidence intervals on all performance metrics in tables and figures, and clarify the fairness metrics (e.g., demographic parity or equalized odds across subgroups). This will directly address the load-bearing aspects of the claims. revision: yes

  2. Referee: The choice of specific handcrafted features with SVM and FMAE-IAT turn-level embeddings with MLP as proxies for classical and deep paradigms is not justified against alternatives (e.g., other action unit descriptors or end-to-end vision transformers), undermining the broader claim that classical approaches are superior in this domain.

    Authors: Our claims are scoped to the specific classical (handcrafted features + SVM) and deep (FMAE-IAT turn-level embeddings + MLP) implementations described, which were selected as representative of common practices in the vision-based mental health literature rather than exhaustive proxies for entire paradigms. The manuscript references prior work using similar handcrafted setups and pretrained embedding approaches. We do not assert universal superiority of all classical methods. To strengthen this, the revision will expand the methods and discussion sections with explicit justification for these choices (citing their prevalence and interpretability advantages), add a limitations paragraph acknowledging alternatives like other AU descriptors or vision transformers, and note that broader benchmarking is an important direction for future research. No new experiments are feasible at this stage, but the added context will prevent overgeneralization. revision: partial

Circularity Check

0 steps flagged

No circularity: purely empirical comparison of fixed pipelines on external databases

full rationale

The paper reports direct experimental results from applying two fixed, pre-specified pipelines (handcrafted features + SVM; FMAE-IAT embeddings + MLP) to two external databases (TPOT, Pitt) with independently defined depression labels. No equations, derivations, fitted parameters renamed as predictions, or self-citations are used to justify the central claims of accuracy, fairness, or generalizability. All performance numbers are computed outputs from the described methods on the given data splits; the modest cross-context results follow immediately from those computations without any reduction to prior inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

As an empirical machine learning comparison study, the central claims rest on the validity of the selected features and classifiers as representatives of classical versus deep approaches and on the databases capturing intended contexts. No new mathematical axioms, free parameters beyond standard ML hyperparameters, or invented entities are introduced in the abstract.

pith-pipeline@v0.9.0 · 5574 in / 1322 out tokens · 54873 ms · 2026-05-10T15:40:11.524340+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 45 canonical work pages · 1 internal anchor

  1. [1]

    K. D. B. J. Adam et al. A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 1412(6), 2014

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  2. [2]

    Al Jazaery and G

    M. Al Jazaery and G. Guo. Video-based depression level analysis by encoding deep spatiotemporal features.IEEE Transactions on Affective Computing, 12(1):262–268, 2018

    work page 2018

  3. [3]

    Alghowinem, R

    S. Alghowinem, R. Goecke, J. F. Cohn, M. Wagner, G. Parker, and M. Breakspear. Cross-cultural detection of depression from nonverbal behaviour. In2015 11th IEEE International conference and workshops on automatic face and gesture recognition (FG), volume 1, pages 1–8. IEEE, 2015

    work page 2015

  4. [4]

    Alghowinem, R

    S. Alghowinem, R. Goecke, M. Wagner, G. Parker, and M. Breakspear. Eye movement analysis for depression detection. In2013 IEEE International Conference on Image Processing, pages 4220–4224. IEEE, 2013

    work page 2013

  5. [5]

    S. M. Alghowinem, T. Gedeon, R. Goecke, J. Cohn, and G. Parker. Interpretation of depression detection models via feature selection methods.IEEE Transactions on Affective Computing, 2020

    work page 2020

  6. [6]

    Arioz, U

    U. Arioz, U. Smrke, N. Plohl, and I. Mlakar. Scoping review on the multimodal classification of depression and experimental study on existing multimodal models.Diagnostics, 12(11):2683, 2022

    work page 2022

  7. [7]

    Bilalpur, S

    M. Bilalpur, S. Hinduja, L. Cariola, L. Sheeber, N. Allen, L.-P. Morency, and J. F. Cohn. Shap-based prediction of mother’s history of depression to understand the influence on child behavior. In Proceedings of the 25th International Conference on Multimodal Interaction, pages 537–544, 2023

    work page 2023

  8. [8]

    Bilalpur, S

    M. Bilalpur, S. Hinduja, L. A. Cariola, L. B. Sheeber, N. Allen, L. A. Jeni, L.-P. Morency, and J. F. Cohn. Multimodal feature selection for detecting mothers’ depression in dyadic interactions with their adolescent offspring.FG, 2023

    work page 2023

  9. [9]

    Birhane, S

    A. Birhane, S. Dehdashtian, V . Prabhu, and V . Boddeti. The dark side of dataset scaling: Evaluating racial classification in multimodal models. InProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1229–1244, 2024

    work page 2024

  10. [10]

    S. L. Burcusa and W. G. Iacono. Risk for recurrence in depression. Clinical psychology review, 27(8):959–985, 2007

    work page 2007

  11. [11]

    Cheong, S

    J. Cheong, S. Kalkan, and H. Gunes. Fairrefuse: Referee-guided fusion for multi-modal causal fairness in depression detection. In International Joint Conference on Artificial Intelligence (IJCAI), 2024

    work page 2024

  12. [12]

    Cheong, S

    J. Cheong, S. Kuzucu, S. Kalkan, and H. Gunes. Towards gender fairness for mental health prediction. 2023

    work page 2023

  13. [13]

    J. F. Cohn, T. S. Kruez, I. Matthews, Y . Yang, M. H. Nguyen, M. T. Padilla, F. Zhou, and F. De la Torre. Detecting depression from facial actions and vocal prosody. In2009 3rd international conference on affective computing and intelligent interaction and workshops, pages 1–7. IEEE, 2009

    work page 2009

  14. [14]

    Daoudi, Z

    M. Daoudi, Z. Hammal, A. Kacem, and J. F. Cohn. Gram matrices formulation of body shape motion: an application for depression severity assessment. In2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), pages 258–263. IEEE, 2019

    work page 2019

  15. [15]

    Dehdashtian, R

    S. Dehdashtian, R. He, Y . Li, G. Balakrishnan, N. Vasconcelos, V . Ordonez, and V . N. Boddeti. Fairness and bias mitigation in computer vision: A survey.arXiv preprint arXiv:2408.02464, 2024

    work page arXiv 2024

  16. [16]

    Dibeklio ˘glu, Z

    H. Dibeklio ˘glu, Z. Hammal, and J. F. Cohn. Dynamic multimodal measurement of depression severity using deep autoencoding.IEEE journal of biomedical and health informatics, 22(2):525–536, 2017

    work page 2017

  17. [17]

    Frank, G

    E. Frank, G. B. Cassano, P. Rucci, W. K. Thompson, H. C. Kraemer, A. Fagiolini, L. Maggi, D. J. Kupfer, M. K. Shear, P. R. Houck, et al. Predictors and moderators of time to remission of major depression with interpersonal psychotherapy and ssri pharmacotherapy.Psycho- logical medicine, 41(1):151–162, 2011

    work page 2011

  18. [18]

    C. Fu, Z. Fu, Q. Zhang, X. Kuang, J. Dong, K. Su, Y . Su, W. Shi, J. Yao, Y . Zhao, et al. The first mpdd challenge: multimodal personality-aware depression detection. InProceedings of the 33rd ACM International Conference on Multimedia, pages 13924–13929, 2025

    work page 2025

  19. [19]

    J. M. Girard, W.-S. Chu, L. A. Jeni, and J. F. Cohn. Sayette group formation task (gft) spontaneous facial expression database. InFG, pages 581–588. IEEE, 2017

    work page 2017

  20. [20]

    J. M. Girard, J. F. Cohn, M. H. Mahoor, S. M. Mavadati, Z. Hammal, and D. P. Rosenwald. Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses.Image and vision computing, 32(10):641–647, 2014

    work page 2014

  21. [21]

    Gratch, R

    J. Gratch, R. Artstein, G. M. Lucas, G. Stratou, S. Scherer, A. Nazar- ian, R. Wood, J. Boberg, D. DeVault, S. Marsella, et al. The distress analysis interview corpus of human and computer interviews. InLrec, volume 14, pages 3123–3128. Reykjavik, 2014

    work page 2014

  22. [22]

    Greenberg, A

    P. Greenberg, A. Chitnis, D. Louie, E. Suthoff, S.-Y . Chen, J. Maitland, P. Gagnon-Sanschagrin, A.-A. Fournier, and R. C. Kessler. The economic burden of adults with major depressive disorder in the united states (2019).Advances in Therapy, 40(10):4460–4479, 2023

    work page 2019

  23. [23]

    Hamilton

    M. Hamilton. A rating scale for depression.Journal of neurology, neurosurgery, and psychiatry, 23(1):56, 1960

    work page 1960

  24. [24]

    Hardt, E

    M. Hardt, E. Price, and N. Srebro. Equality of opportunity in supervised learning.Advances in neural information processing systems, 29, 2016

    work page 2016

  25. [25]

    K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022

    work page 2022

  26. [26]

    Hirota, Y

    Y . Hirota, Y . Nakashima, and N. Garcia. Quantifying societal bias amplification in image captioning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13450– 13459, 2022

    work page 2022

  27. [27]

    Ilias and D

    L. Ilias and D. Askounis. A cross-attention layer coupled with mul- timodal fusion methods for recognizing depression from spontaneous speech. InProc. Interspeech, volume 2024, pages 912–916, 2024

    work page 2024

  28. [28]

    Kacem, Z

    A. Kacem, Z. Hammal, M. Daoudi, and J. Cohn. Detecting depression severity by interpretable representations of motion dynamics. InFG, pages 739–745. IEEE, 2018

    work page 2018

  29. [29]

    Koutlis and S

    C. Koutlis and S. Papadopoulos. Leveraging representations from in- termediate encoder-blocks for synthetic image detection. InEuropean Conference on Computer Vision, pages 394–411. Springer, 2024

    work page 2024

  30. [30]

    Kroenke, T

    K. Kroenke, T. W. Strine, R. L. Spitzer, J. B. Williams, J. T. Berry, and A. H. Mokdad. The phq-8 as a measure of current depression in the general population.Journal of affective disorders, 114(1-3):163–173, 2009

    work page 2009

  31. [31]

    B. W. Nelson, L. Sheeber, J. H. Pfeifer, and N. B. Allen. Affective and autonomic reactivity during parent–child interactions in depressed and non-depressed mothers and their adolescent offspring.Research on Child and Adolescent Psychopathology, 49(11):1513–1526, 2021

    work page 2021

  32. [32]

    J. A. Nelson, E. M. Leerkes, M. O’Brien, S. D. Calkins, and S. Marcovitch. African american and european american mothers’ beliefs about negative emotions and emotion socialization practices. Parenting, 12(1):22–41, 2012

    work page 2012

  33. [33]

    M. Ning, A. A. Salah, and I. O. Ertugrul. Representation learning and identity adversarial training for facial behavior understanding.arXiv preprint arXiv:2407.11243, 2024

    work page arXiv 2024

  34. [34]

    Onal Ertugrul, L

    I. Onal Ertugrul, L. A. Jeni, W. Ding, and J. F. Cohn. Afar: A deep learning based tool for automated facial affect recognition. In2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 2019

    work page 2019

  35. [35]

    R. J. Prinz, S. Foster, R. N. Kent, and K. D. O’Leary. Multivariate as- sessment of conflict in distressed and nondistressed mother-adolescent dyads.Journal of applied behavior analysis, 12(4):691–700, 1979

    work page 1979

  36. [36]

    Ranjit, T

    J. Ranjit, T. Wang, B. Ray, and V . Ordonez. Variation of gender biases in visual recognition models before and after finetuning.arXiv preprint arXiv:2303.07615, 2023

    work page arXiv 2023

  37. [37]

    Ringeval, B

    F. Ringeval, B. Schuller, M. Valstar, N. Cummins, R. Cowie, L. Tavabi, M. Schmitt, S. Alisamir, S. Amiriparian, E.-M. Messner, et al. Avec 2019 workshop and challenge: state-of-mind, detecting depression with ai, and cross-cultural affect recognition. In9th AVEC Challenge, pages 3–12, 2019

    work page 2019

  38. [38]

    Scherer, G

    S. Scherer, G. Stratou, G. Lucas, M. Mahmoud, J. Boberg, J. Gratch, L.-P. Morency, et al. Automatic audiovisual behavior descriptors for psychological disorder analysis.Image and Vision Computing, 32(10):648–658, 2014

    work page 2014

  39. [39]

    Sheeber, J

    L. Sheeber, J. Lougheed, T. Hollenstein, C. Leve, K. Mudiam, C. Dier- cks, and N. Allen. Maternal aggressive behavior in interactions with adolescent offspring: Proximal social–cognitive predictors in depressed and nondepressed mothers.Journal of psychopathology and clinical science, 132(8):1019, 2023

    work page 2023

  40. [40]

    S. Song, S. Jaiswal, L. Shen, and M. Valstar. Spectral representation of behaviour primitives for depression analysis.IEEE Transactions on Affective Computing, 13(2):829–844, 2022

    work page 2022

  41. [41]

    Walmer, S

    M. Walmer, S. Suri, K. Gupta, and A. Shrivastava. Teaching matters: Investigating the role of supervision in vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7486–7496, 2023

    work page 2023

  42. [42]

    W. Wu, C. Zhang, and P. C. Woodland. Self-supervised representations in speech-based depression detection. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

    work page 2023

  43. [43]

    Y . Yang, C. Fairbairn, and J. F. Cohn. Detecting depression severity from vocal prosody.IEEE transactions on affective computing, 4(2):142–150, 2012

    work page 2012

  44. [44]

    Zhang, Y

    P. Zhang, Y . Goyal, D. Summers-Stay, D. Batra, and D. Parikh. Yin and yang: Balancing and answering binary visual questions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5014–5022, 2016

    work page 2016

  45. [45]

    D. Zhao, A. Wang, and O. Russakovsky. Understanding and evaluating racial biases in image captioning. InProceedings of the IEEE/CVF international conference on computer vision, pages 14830–14840, 2021

    work page 2021