Pith. sign in

REVIEW 5 major objections 5 minor 57 references

A Multimodal Emotion Recognition System: Integrating Facial Expressions, Body Movement, Speech, and Spoken Language

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Combining facial, vocal, language, and body cues yields 96.43% emotion-recognition accuracy in a simulated interview setting, beating every single modality.

desk verdict A well-documented open-source integration undone by a circular language-labeling step and an evaluation that doesn't support the headline accuracy. read the letter →

arxiv 2412.17907 v1 pith:RAV62CFM submitted 2024-12-23 cs.HC cs.CLcs.CVcs.LGcs.MMcs.SDeess.AS

Kris Kraack This is my paper · ORCID
classification cs.HCcs.CLcs.CVcs.LGcs.MMcs.SDeess.AS
keywords multimodalemotionrecognitionfacialexpressionspeechspokenlanguageunderstandingbodymovementanalysisfusionpsychologicalevaluationdeeplearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a computer can recognise emotional state more reliably by fusing four channels at once—facial expressions, body movement, tone of voice, and the meaning of spoken words—than by relying on any one channel. In a simulated real-world test with 52 participants acting six emotions, the fused system reached 96.43% overall accuracy, with the weakest category, neutral, still at 92.31%, while each single modality scored between 82.97% and 94.72%. The system is proposed as a second-opinion aid for psychologists and clinicians, aimed at catching subtle cues that human evaluators may miss and reducing the risk of mis- and overdiagnosis. The author explicitly notes that the test used simulated scenarios with volunteers rather than actual mental-health patients or clinicians, so the result is preliminary evidence rather than proof of clinical utility.

What carries the argument

The load-bearing mechanism is late fusion of per-modality probability distributions. Each component first outputs its own distribution over the emotion set: a convolutional neural network for facial expressions, a pose-estimation module that sorts body movement into low, medium, or high intensity, a convolutional model trained on audio features for vocal emotion, and a bidirectional sequence model over transcribed text for spoken language. The system then weights and sums these distributions into one unified emotional profile, and it is this weighted summation that the paper credits with lifting accuracy above every single modality.

What would settle it

Record real clinical interviews in which two or more trained clinicians independently label the emotional state of each segment, run the system on the same segments, and compare agreement; if the system matches the clinicians no better than chance or no better than the best single modality, the central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that multimodal integration outperforms single modalities in emotion recognition. On model-evaluation data the best facial-expression model reached 85.21%, the speech model 99.63%, and the spoken-language model 97.96%, yet in the simulated real-world test their standalone accuracies were 82.97%, 85.71%, and 87.64%, with body movement alone at 94.72%. Fusing all four channels produced 96.43% overall accuracy, only 13 misclassifications across 351 predictions, and no emotion category below 92.31%. The paper argues that this fusion compensates for specific unimodal failures—'fear' frequently missed by facial expressions, 'neutral' by spoken language—and that the result supports using the system as a quantitative complement to human clinical judgment.

Load-bearing premise

The accuracy claim assumes that volunteers' self-reported true/false feedback on deliberately acted emotions in an ideal simulated interview is a valid stand-in for how the system would perform with real patients and clinicians in actual clinical interviews.

Editorial extensions

If this is right

  • A clinician could use the fused profile as a second-opinion layer during structured interviews, letting the system flag emotional states that a single modality would miss.
  • People with reduced facial expression, such as those with some neurodevelopmental conditions, would still be assessed through speech, language, and movement channels, which the paper singles out as a key benefit.
  • Because the speech and language components can process audio after a session ends, the same pipeline can review recorded interviews retrospectively as well as in real time.
  • The weakest fused category is neutral at 92.31%, so flat or neutral affect is where a clinician would still need to pay the closest attention.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 96.43% figure measures agreement with participants' own true/false judgments of acted emotions; a stronger test would compare the system against independent clinician labels on the same interviews, and the paper does not report that comparison.
  • A natural extension is to test the fusion when one or more channels are missing, for example a phone-based screening with no video; the modular design implies the remaining channels would carry the prediction, but the paper does not test this.
  • Because participants were instructed to create an ideal recording environment, real clinics would need equivalent lighting, camera placement, and microphone conditions for the accuracy to transfer, so deployment should include an environment checklist.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents a multimodal emotion recognition system combining facial expression recognition, body movement analysis, speech recognition, and spoken language analysis, intended as a supportive tool for psychological evaluation. The system is built from public datasets, with a CNN for faces, a CNN for speech features, a BiLSTM for textual emotion classification, and a MediaPipe-based body movement intensity classifier. The authors report per-modality evaluation metrics and a simulated real-world test in which the multimodal integration achieves 96.43% overall accuracy. The paper claims that the multimodal approach outperforms single modalities and that the system can help mitigate subjectivity and diagnostic errors in clinical settings.

Significance. If the reported 96.43% multimodal accuracy were valid, this would be a practically significant result for affective computing and clinical decision support. The paper addresses an important application area, draws on multiple established datasets, and makes its code openly available through several linked repositories, which is a strength for reproducibility. However, the central empirical claims are currently compromised by a circular training/evaluation scheme in the spoken language modality, an unspecified fusion method, and internally contradictory evaluation metrics. As presented, the results do not establish that the multimodal system outperforms its components or that it would transfer to real clinical interviews.

major comments (5)
  1. [Section III-D] The spoken language component is trained in a circular manner. The authors state that after preprocessing, the text data were "classified in terms of emotional state observations using on the keywords dictionary," and this keyword-derived classification was then used as the training target for the BiLSTM. The reported 97.96% validation accuracy therefore largely measures how well the model reproduces the keyword heuristic, not how well it understands emotion in language. In the real-world test (Section V), participants spoke pre-defined sentences in Phase 1 and self-composed sentences designed to evoke specific emotions in Phase 2, both of which are likely to contain the same explicit emotion vocabulary used to build the dictionary. The 87.64% spoken-language accuracy and its contribution to the 96.43% multimodal result are thus not evidence of generalizable language understanding.
  2. [Section III-E] The multimodal fusion method is not specified. The text says only that outputs were aggregated "by weighing and synthesising their respective probabilities," with no equations, algorithm, or description of how the weights were chosen. Without a reproducible fusion rule, the reported 96.43% accuracy in Table IV cannot be independently verified, and it cannot be determined whether fusion adds information or merely propagates the scores of the circular language channel.
  3. [Section IV and Appendix Table III] The model evaluation metrics are internally inconsistent. The main text reports for the FER-2013 model a precision of 67.92%, recall of 68.35%, and F1 of 66.80%, while Appendix Table III lists an overall precision of 65%, recall of 58.29%, and F1 of 60%. For RAF-DB, the main text reports 85.21% accuracy, 85.04% precision, 85.21% recall, and 84.96% F1, whereas the appendix lists 81.71% precision, 74.86% recall, and 77.71% F1. The combined FER-2013 & RAF-DB numbers also differ substantially. These discrepancies are not acknowledged or explained, so the reliability of the evaluation section is undermined.
  4. [Section V] The primary evidence for the system's effectiveness, the simulated real-world test, relies on participants' self-reported true/false feedback as ground truth. Participants knew which emotion they intended to express and then judged whether the system recognized it, which is not an independent measure of emotional state. In addition, Section VI correctly notes that the test did not involve clinicians or mental health patients. The paper's central claim that the system can support psychological evaluation is therefore not supported by the presented evidence, since acted, self-labeled emotion expressions in an idealized setting do not replicate clinical discourse.
  5. [Section VIII] The body movement analysis component is not an emotion recognizer: it classifies physical activity as low, medium, or high, and Section VIII admits it "lacks a meaningful correlation with emotional states." Including this channel in the fused multimodal output cannot, by itself, improve emotion recognition unless the fusion learns some mapping from movement intensity to emotional state, and no such mapping is described. The reported 94.72% body-movement accuracy is thus for movement intensity, not emotion, and does not contribute to validating the multimodal emotion accuracy claim.
minor comments (5)
  1. [Section II] The section numbering is duplicated: "II. RELATED WORK" is followed by another "II. LIMITATIONS OF TRADITIONAL EVALUATION AND TECHNOLOGICAL SOLUTION." Subsequent sections should be renumbered accordingly.
  2. [Section IV] Figure numbering is inconsistent: both the RAF-DB learning curves and the speech model learning curves are labeled "Figure 2," and the subsequent figures are numbered out of sequence. Figures should be renumbered sequentially in order of appearance.
  3. [Section III.C] The phrase "a dataset comprising of 39,458 audio samples" contains a grammatical error; "comprising of" should be "comprising." Similar minor wording issues appear throughout, such as "utilising" with missing objects.
  4. [Section III.D] The sentence "The preprocessed text data were then classified in terms of emotional state observations using on the keywords dictionary" contains a typo ("using on") and should be rephrased for clarity.
  5. [References] The in-text citations for TensorFlow and Dlib appear swapped relative to the reference list: the text cites Dlib as [41] and TensorFlow as [42], but the reference list assigns [41] to TensorFlow and [42] to Dlib.

Circularity Check

1 steps flagged · score 6.0 of 10

The spoken-language model is trained on labels produced by a keyword dictionary built from the same texts, so its 97.96% validation accuracy and its contribution to the 96.43% multimodal result reduce to keyword matching rather than emotion understanding.

  1. fitted input called prediction [Section III-D (Spoken Language Recognition), results in Section IV and Section V-B]
    "The preprocessing stage entailed the implementation of Regular Expression operations (Regex) to harmonise the data, the creation of custom stop-words in observations, and the retrieval of the most frequent keywords for each emotional state (anger, disgust, joy, sadness, surprise, fear) to create a custom keyword dictionary, with the custom stop-words filtered out. The preprocessed text data were then classified in terms of emotional state observations using on the keywords dictionary."

    The BiLSTM's training and validation labels are not independent human annotations; they are generated by a keyword dictionary derived from the most frequent words in the same texts the model sees as input. A model can reach the reported 97.96% validation accuracy by learning to reproduce that dictionary's keyword-to-emotion mapping, so the metric measures self-consistency with the heuristic, not emotion recognition. The real-world language test then used pre-defined sentences expressing each emotion (Phase 1) and personal sentences 'designed to evoke specific emotions' (Phase 2), which can be passed by the same keyword-spotting shortcut.

full rationale

The only concrete circularity is in the spoken-language component. Section III-D constructs a keyword dictionary from the training texts, classifies those same texts with it, and then trains the BiLSTM on those derived labels; the 97.96% validation accuracy therefore measures how well the model reproduces the dictionary, not how well it recognizes emotion. Because the real-world test uses pre-defined or self-composed sentences explicitly designed to express the target emotions, the language channel can score highly from emotionally explicit vocabulary alone, and the unspecified fusion ('weighing and synthesising respective probabilities') leaves the 96.43% multimodal result potentially dominated by that channel. The other components use standard datasets with external labels, and the author's GitHub self-citations are resource links rather than load-bearing evidence. The evaluation's reliance on self-reported ground truth and absence of clinical participants is a validity limitation, not a circularity. Overall, the central claim of multimodal superiority is partially undermined, but not wholly reducible to a single fitted parameter across all modalities.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claims depend on a large set of unchecked data assumptions: label compatibility across datasets, self-reported ground truth from non-clinical participants, and unspecified fusion weights. No new theoretical entities are introduced.

free parameters (4)
  • Multimodal fusion weights = unspecified
    Section III.E says outputs are 'aggregated by weighing and synthesising their respective probabilities' but no weights or method are given; the 96.43 percent multimodal result depends on these hand-chosen or tuned weights.
  • Keyword dictionary for emotion labeling = unspecified thresholds
    Section III.D constructs labels by retrieving 'most frequent keywords for each emotional state'; the cutoff and keyword set are hand-chosen and directly determine the training labels for the BiLSTM.
  • Data preprocessing filters = unspecified
    Section III.A filters FER-2013 and RAF-DB with cropping, Haar/Dlib face detection, and Laplacian blur filtering; thresholds are not quantified and affect which training samples survive.
  • Train/validation/test splits = unspecified
    Section IV reports test accuracy but not the split ratios or stratification for any dataset; the 99.63 percent speech result is sensitive to how segments from the same speaker or conversation are divided.
assumptions (4)
  • domain assumption Emotion labels in FER-2013, RAF-DB, CK+, CREMA-D, RAVDESS, SAVEE, TESS, ESD, MELD, GoEmotions, and Sentiment140 are mutually compatible after manual harmonization.
    Sections III.C and III.D harmonize labels across datasets with different taxonomies, for example 28 GoEmotions classes collapsed to 6; if the mapping is wrong, every downstream accuracy is meaningless.
  • domain assumption Participant self-report of whether the system recognized the intended emotion is a valid ground truth.
    Section V asks participants to act emotions and give true/false feedback; this assumes actors can reliably produce and judge the target emotion, which was not validated.
  • domain assumption The simulated ideal recording environment matches the intended clinical interview setting.
    Section V instructs participants to create an ideal testing environment; Section VI admits the target audience was not included. The transfer of the 96.43 percent result to clinics is an unverified assumption.
  • standard math Standard deep learning optimization and generalization assumptions hold.
    The CNN and BiLSTM training use Adam and cross-entropy loss with batch normalization and dropout; no novel math is introduced, but the usual i.i.d. and no-leakage assumptions are relied on and likely violated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Multimodal Emotion Recognition System: Integrating Facial Expressions, Body Movement, Speech, and Spoken Language." pith.science (2026). https://pith.science/paper/RAV62CFM

@misc{pith2026241217907,
  author       = {Pith},
  title        = {Pith review of: A Multimodal Emotion Recognition System: Integrating Facial Expressions, Body Movement, Speech, and Spoken Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RAV62CFM}},
  note         = {Machine review of arXiv:2412.17907}
}
read the original abstract

Traditional psychological evaluations rely heavily on human observation and interpretation, which are prone to subjectivity, bias, fatigue, and inconsistency. To address these limitations, this work presents a multimodal emotion recognition system that provides a standardised, objective, and data-driven tool to support evaluators, such as psychologists, psychiatrists, and clinicians. The system integrates recognition of facial expressions, speech, spoken language, and body movement analysis to capture subtle emotional cues that are often overlooked in human evaluations. By combining these modalities, the system provides more robust and comprehensive emotional state assessment, reducing the risk of mis- and overdiagnosis. Preliminary testing in a simulated real-world condition demonstrates the system's potential to provide reliable emotional insights to improve the diagnostic accuracy. This work highlights the promise of automated multimodal analysis as a valuable complement to traditional psychological evaluation practices, with applications in clinical and therapeutic settings.

Figures

Figures reproduced from arXiv: 2412.17907 by the authors.

Figure 1
Figure 1. System architecture diagram of the multimodal integration. IV. MODEL EVALUATION In this section, an evaluation of the system is conducted of each classification model performance. This excludes body movement analysis and the multimodal integration. The evaluation considers performance metrics on unseen test data, such as accuracy, loss, precision, recall, F1 score, and AUC score. In the context of facial expression … view at source ↗
Figure 2
Figure 2. Training and validation accuracy and loss curves over epochs for the RAF-DB model. The speech recognition model was trained on a harmonised dataset synthesised from multiple sources, including CREMA￾D, RAVDESS, SAVEE, TESS, ESD, and MELD. The model demonstrated excellent generalisation capabilities, achieving notable outcomes with an accuracy of 99.63%, a test loss of 0.0124, a precision of 99.63%, a recall of 99.63… view at source ↗
Figure 3
Figure 3. Training and validation accuracy and loss curves over epochs for the spoken language model [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 6
Figure 6. Figure 6: Pie-chart diagrams of the demographic composition of the test group. A. Phase 1: Single Modal Emotion Recognition First phase of the real-world testing, centered on single modal emotion recognition of individual components. Participants were tasked with providing true/…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 40 canonical work pages

  1. [1]

    Clinical judgment and decision making,

    H. N. Garb, "Clinical judgment and decision making," Annu. Rev. Clin. Psychol., vol. 1, pp. 67 –89, Apr. 2005, doi: 10.1146/annurev.clinpsy.1.102803.143810

    work page arXiv 2005

  2. [2]

    Recognizing and reducing cognitive bias in clinical and forensic neurology,

    S. Satya -Murti and J. Lockhart, “Recognizing and reducing cognitive bias in clinical and forensic neurology,” Neurol. Clin. Pract., vol. 5, no. 5, pp. 389-396, Oct. 2015, doi: 10.1212/CPJ.0000000000000181

    work page doi:10.1212/cpj.0000000000000181 2015

  3. [3]

    2001, PMID: 20669516

    Office of the Surgeon General (US), Center for Mental Health Services (US), and National Institute of Mental Health (US), Mental Health: Culture, Race, and Ethnicity: A Supplement to Mental Health: A Report of the Surgeon General, Rockville, MD: Substance Abuse and Mental Health Services Administration (US), Aug. 2001, PMID: 20669516

    work page 2001

  4. [4]

    Deep Facial Expression Recognition: A Survey,

    S. Li and W. Deng, "Deep Facial Expression Recognition: A Survey," in IEEE Transactions on Affective Computing, vol. 13, no. 3, pp. 1195 - 1215, 1 July-Sept. 2022, doi: 10.1109/TAFFC.2020.2981446

    arXiv 2022

  5. [5]

    Facial Expression Recognition Using Facial Movement Features,

    L. Zhang and D. Tjondronegoro, "Facial Expression Recognition Using Facial Movement Features," in IEEE Transactions on Affective Computing, vol. 2, no. 4, pp. 219 -229, Oct.-Dec. 2011, doi: 10.1109/T - AFFC.2011.13

    doi:10.1109/t 2011

  6. [6]

    Face Expression Recognition and Analysis: The State of the Art

    V. Bettadapura, “Face Expression Recognition and Analysis: The State of the Art,” arXiv, Mar. 2012, [Online]. Available: https://doi.org/10.48550/arXiv.1203.6722

    work page Pith review arXiv doi:10.48550/arxiv.1203.6722 2012

  7. [7]

    The first facial expression recognition and analysis challenge,

    M. F. Valstar, B. Jiang, M. Mehu, M. Pantic and K. Scherer, "The first facial expression recognition and analysis challenge," 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), Santa Barbara, CA, USA, 2011, pp. 921 -926, doi: 10.1109/FG.2011.5771374

    work page arXiv 2011

  8. [8]

    Methods for Facial Expression Recognition with Applications in Challenging Situations,

    A. A. Pise, M. A. Alqahtani, P. Verma, K. P., D. A. Karras, S. P., and A. Halifa, “Methods for Facial Expression Recognition with Applications in Challenging Situations,” Comput. Intell. Neurosci., vol. 2022, art. no. 9261438, May 25, 2022, doi: 10.1155/2022/9261438

    work page doi:10.1155/2022/9261438 2022

Show all 57 references
  1. [9]

    A Comprehensive Review of Speech Emotion Recognition Systems,

    T. M. Wani, T. S. Gunawan, S. A. A. Qadri, M. Kartiwi and E. Ambikairajah, "A Comprehensive Review of Speech Emotion Recognition Systems," in IEEE Access, vol. 9, pp. 47795 -47814, 2021, doi: 10.1109/ACCESS.2021.3068045

    2021

  2. [10]

    Emotion Recognition from Speech Signals using Excitation Source and Spectral Features,

    A. R. Choudhury, A. Ghosh, R. Pandey and S. Barman, "Emotion Recognition from Speech Signals using Excitation Source and Spectral Features," 2018 IEEE Applied Signal Processing Conference (ASPCON), Kolkata, India, 2018, pp. 257 -261, doi: 10.1109/ASPCON.2018.8748626

    2018

  3. [11]

    Learning Salient Features for Speech Emotion Recognition Using Convolutional Neural Networks,

    Q. Mao, M. Dong, Z. Huang and Y. Zhan, "Learning Salient Features for Speech Emotion Recognition Using Convolutional Neural Networks," in IEEE Transactions on Multimedia, vol. 16, no. 8, pp. 2203-2213, Dec. 2014, doi: 10.1109/TMM.2014.2360798

    2014

  4. [12]

    LSTM Based Feature Learning and CNN Based Classification for Speech Emotion Recognition,

    H. Kalra, "LSTM Based Feature Learning and CNN Based Classification for Speech Emotion Recognition," 2023 International Conference on Data Science and Network Security (ICDSNS), Tiptur, India, 2023, pp. 1-6, doi: 10.1109/ICDSNS58469.2023.10244802

    2023

  5. [13]

    Emotion Detection using Natural Language Processing,

    D. SakethNath, H. Kaur and A. Singh, "Emotion Detection using Natural Language Processing," 2022 5th International Conference on Contemporary Computing and Informatics (IC3I), Uttar Pradesh, India, 2022, pp. 1237-1242, doi: 10.1109/IC3I56241.2022.10072841

    2022

  6. [14]

    Multi -Class Twitter Emotion Classification: A New Approach,

    R. C. Balabantaray, M. Mohammad, and N. Sharma, “Multi -Class Twitter Emotion Classification: A New Approach,” Int. J. Appl. Inf. Syst., vol. 4, no. 1, pp. 48-53, Sept. 2012, doi: 10.5120/ijais12-450651

    2012 doi

  7. [15]

    Multi -class sentiment analysis on twitter: Classification performance and challenges,

    M. Bouazizi and T. Ohtsuki, "Multi -class sentiment analysis on twitter: Classification performance and challenges," in Big Data Mining and Analytics, vol. 2, no. 3, pp. 181 -194, September 2019, doi: 10.26599/BDMA.2019.9020002

    2019

  8. [16]

    Multimodal Emotion Recognition Using Deep Learning Techniques,

    S. Jerald James and L. Jacob, "Multimodal Emotion Recognition Using Deep Learning Techniques," 2022 4th International Conference on Advances in Computing, Communication Control and Networking (ICAC3N), Greater Noida, India, 2022, pp. 903 -908, doi: 10.1109/ICAC3N56670.2022.10074512

    2022

  9. [17]

    Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning,

    S. Tripathi, S. Tripathi, and H. Beigi, “Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning,” arXiv, Apr. 2018. [Online]. Available: https://doi.org/10.48550/arXiv.1804.05788

    2018 doi arXiv

  10. [18]

    FAF: A novel multimodal emotion recognition approach integrating face, body and text,

    Z. Fang, A. He, Q. Yu, B. Gao, W. Ding, T. Zhang, and L. Ma, “FAF: A novel multimodal emotion recognition approach integrating face, body and text,” arXiv, Nov. 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2211.15425

    2022 doi arXiv

  11. [19]

    Multimodal mixed emotion detection,

    A. S. Patwardhan, "Multimodal mixed emotion detection," 2017 2nd International Conference on Communication and Electronics Systems (ICCES), Coimbatore, India, 2017, pp. 139 -143, doi: 10.1109/CESYS.2017.8321250

    2017

  12. [20]

    Multimodal Emotion Recognition using Deep Learning,

    S. M. S. Abdullah, S. Y. A. Ameen, M. A. M. Sadeeq, and S. Zeebaree, “Multimodal Emotion Recognition using Deep Learning,” J. Adv. Sci. Technol. Trends (JASTT), vol. 2, no. 01, pp. 73 –79, May 2021, doi: 10.38094/jastt20291

    2021 doi

  13. [21]

    Exploring Fusion Methods for Multimodal Emotion Recognition with Missing Data,

    J. Wagner, E. Andre, F. Lingenfelser and J. Kim, "Exploring Fusion Methods for Multimodal Emotion Recognition with Missing Data," in IEEE Transactions on Affective Computing, vol. 2, no. 4, pp. 206 -218, Oct.-Dec. 2011, doi: 10.1109/T-AFFC.2011.12

    2011 doi

  14. [22]

    Kraack, Face Expression Recognition, GitHub repository, Nov

    K. Kraack, Face Expression Recognition, GitHub repository, Nov. 2024. [Online]. Available: https://github.com/kraack- tech/face_expression_recognition

    2024

  15. [23]

    Kraack, Body Movement Analysis , GitHub repository, Nov

    K. Kraack, Body Movement Analysis , GitHub repository, Nov. 2024. [Online]. Available: https://github.com/kraack- tech/body_movement_analysis

    2024

  16. [24]

    Kraack, Voice Emotion Detection , GitHub repository, Nov

    K. Kraack, Voice Emotion Detection , GitHub repository, Nov. 2024. [Online]. Available: https://github.com/kraack - tech/voice_emotion_recognition. Model Class Precision Recall F1 FER-2013 Anger 54% 54% 54% Disgust 64% 43% 52% Fear 65% 40% 50% Happy 85% 92% 88% Neutral 57% 81%...

    2024

  17. [25]

    Kraack, NLP Emotion Recognition, GitHub repository, Nov

    K. Kraack, NLP Emotion Recognition, GitHub repository, Nov. 2024. [Online]. Available: https://github.com/kraack- tech/NLP_emotion_recognition

    2024

  18. [26]

    Kraack, Multimodal Emotion Detection, GitHub repository, Nov

    K. Kraack, Multimodal Emotion Detection, GitHub repository, Nov

  19. [27]

    An ingroup disadvantage in recognizing micro -expressions,

    Q. Wu, K. Peng, Y. Xie, Y. Lai, X. Liu, and Z. Zhao, “An ingroup disadvantage in recognizing micro -expressions,” Front. Psychol., vol. 13, art. no. 1050068, Nov. 25, 2022, doi: 10.3389/fpsyg.2022.1050068

    2022

  20. [28]

    Life Events and Mental Status: A Longitudinal Study,

    J. K. Myers, J. J. Lindenthal, M. P. Pepper, and D. R. Ostrander, “Life Events and Mental Status: A Longitudinal Study,” Journal of Health and Social Behavior, vol. 13, no. 4, pp. 398 –406, 1972, doi: 10.2307/2136832

    1972 doi

  21. [29]

    Compassion fatigue and global compassion fatigue in practitioner psychologists: A qualitative study,

    K. Stevens and M. Al -Abbadey, “Compassion fatigue and global compassion fatigue in practitioner psychologists: A qualitative study,” Curr. Psychol., vol. 43, pp. 7259 –7274, 2024, doi: 10.1007/s12144-023- 04908-3

    2024 doi

  22. [30]

    Fatigue among clinicians and the safety of patients,

    D. M. Gaba and S. K. Howard, “Fatigue among clinicians and the safety of patients,” N. Engl. J. Med., vol. 347, no. 16, pp. 1249 –1255, Oct. 17, 2002, doi: 10.1056/NEJMsa020846

    2002 doi

  23. [31]

    Breaking the taboo: Eight Swedish clinical psychologists’ experiences of compassion fatigue,

    M. N. Harling, E. Högman, and E. Schad, “Breaking the taboo: Eight Swedish clinical psychologists’ experiences of compassion fatigue,” Int. J. Qual. Stud. Health Well -Being, vol. 15, no. 1, 2020, doi: 10.1080/17482631.2020.1785610

    2020

  24. [32]

    Anatomy of an epidemic: Psychiatric drugs and the astonishing rise of mental illness in America,

    R. Whitaker, “Anatomy of an epidemic: Psychiatric drugs and the astonishing rise of mental illness in America,” Ethical Hum. Psychol. Psychiatry, vol. 7, no. 1, pp. 23–35, Spring 2005

    2005

  25. [33]

    Diagnostic error in mental health: A review,

    A. Bradford, A. N. D. Meyer, S. Khan, T. D. Giardina, and H. Singh, “Diagnostic error in mental health: A review,” BMJ Qual. Saf., vol. 33, no. 10, pp. 663–672, Sep. 19, 2024, doi: 10.1136/bmjqs-2023-016996

    2024 doi

  26. [34]

    Overdiagnosis of mental disorders in children and adolescents (in developed countries),

    E. C. Merten, J. C. Cwik, J. Margraf, et al., “Overdiagnosis of mental disorders in children and adolescents (in developed countries),” Child Adolesc. Psychiatry Ment. Health, vol. 11, no. 5, 2017, doi: 10.1186/s13034-016-0140-5

    2017 doi

  27. [35]

    Deep convolution network based emotion analysis towards mental health care,

    Z. Fei, E. Yang, D. D. -U. Li, S. Butler, W. Ijomah, X. Li, and H. Zhou, “Deep convolution network based emotion analysis towards mental health care,” Neurocomputing, vol. 388, pp. 212 –227, 2020, doi: 10.1016/j.neucom.2020.01.034

    2020 doi

  28. [36]

    Spoken Language Derived Measures for Detecting Mild Cognitive Impairment,

    B. Roark, M. Mitchell, J. -P. Hosom, K. Hollingshead and J. Kaye, "Spoken Language Derived Measures for Detecting Mild Cognitive Impairment," in IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 7, pp. 2081 -2090, Sept. 2011, doi: 10.1109/TASL.2011.2112351

    2011

  29. [37]

    Challenges in representation learning: A report on three machine learning contests,

    I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D. -H. Lee, Y. Zhou, C. Ramaiah, F. Feng, R. Li, X. Wang, D. Athanasakis, J. Shawe -Taylor, M. Milakov, J. Park, R. Ionescu, M. Popescu, C. Grozea, J. Bergstra, J. X...

    2015 doi

  30. [38]

    Reliable Crowdsourcing and Deep Locality - Preserving Learning for Unconstrained Facial Expression Recognition,

    S. Li and W. Deng, "Reliable Crowdsourcing and Deep Locality - Preserving Learning for Unconstrained Facial Expression Recognition," in IEEE Transactions on Image Processing, vol. 28, no. 1, pp. 356 -370, Jan. 2019, doi: 10.1109/TIP.2018.2868382

    2019

  31. [39]

    The Extended Cohn -Kanade Dataset (CK+): A complete dataset for action unit and emotion -specified expression,

    P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar and I. Matthews, "The Extended Cohn -Kanade Dataset (CK+): A complete dataset for action unit and emotion -specified expression," 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshop...

    2010

  32. [40]

    OpenCV: Open Source Computer Vision Library,

    OpenCV Development Team, “OpenCV: Open Source Computer Vision Library,” [Online]. Available: https://opencv.org/

  33. [41]

    TensorFlow: An open -source machine learning framework,

    TensorFlow, “TensorFlow: An open -source machine learning framework,” [Online]. Available: https://www.tensorflow.org/

  34. [42]

    dlib: A toolkit for machine learning,

    D. King, "dlib: A toolkit for machine learning," [Online]. Available: http://dlib.net/

  35. [43]

    MediaPipe’s pose estimation model,

    MediaPipe, “MediaPipe’s pose estimation model,” [Online]. Available: https://ai.google.dev/edge/mediapipe/solutions/vision/pose_landmarker

  36. [44]

    CREMA -D: Crowd -Sourced Emotional Multimodal Actors Dataset,

    H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova and R. Verma, "CREMA -D: Crowd -Sourced Emotional Multimodal Actors Dataset," in IEEE Transactions on Affective Computing, vol. 5, no. 4, pp. 377-390, 1 Oct.-Dec. 2014, doi: 10.1109/TAFFC.2014.2336244

    2014

  37. [45]

    The Ryerson Audio -Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English,

    S. R. Livingstone and F. A. Russo, “The Ryerson Audio -Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English,” PLOS ONE, vol. 13, no. 5, May 16, 2018, doi: 10.1371/journal.pone.0196391

    2018 doi

  38. [46]

    Surrey Audio -Visual Expressed Emotion (SAVEE) Database,

    P. Jackson and S. Haq, Apr. 2, 2015. “Surrey Audio -Visual Expressed Emotion (SAVEE) Database,” distributed by University of Surrey, [Online]. Available: http://kahlan.eps.surrey.ac.uk/savee/Database.html

    2015

  39. [47]

    Toronto Emotional Speech Set (TESS),

    K. Dupuis and M. K. Pichora -Fuller, 2010. “Toronto Emotional Speech Set (TESS),” distributed by University of Toronto, Psychology Department, [Online]. Available: https://hdl.handle.net/1807/24487

    2010

  40. [48]

    Emotional voice conversion: theory, databases, and ESD,

    K. Zhou, B. Sisman, R. Liu, and H. Li, “Emotional voice conversion: theory, databases, and ESD,” distributed by National University of Singapore and Singapore University of Technology and Design, [Online]. Available: https://hltsingapore.github.io/ESD/index.html

  41. [49]

    Seen and Unseen Emotional Style Transfer for Voice Conversion with A New Emotional Speech Dataset,

    K. Zhou, B. Sisman, R. Liu and H. Li, "Seen and Unseen Emotional Style Transfer for Voice Conversion with A New Emotional Speech Dataset," ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada, 2021, pp. 920-9...

    2021

  42. [50]

    MELD: A multimodal multi -party dataset for emotion recognition in conversation,

    S. Poria, D. Hazarika, N. Majumder, G. Naik, R. Mihalcea, and E. Cambria, “MELD: A multimodal multi -party dataset for emotion recognition in conversation,” 2018. Available: https://affective - meld.github.io/

    2018

  43. [51]

    EmotionLines: An emotion corpus of multi -party conversations,

    S. Y. Chen, C. C. Hsu, C. C. Kuo, and L. W. Ku, “EmotionLines: An emotion corpus of multi -party conversations,” arXiv preprint arXiv:1802.08379, 2018. Available: https://arxiv.org/abs/1802.08379

    2018 arXiv

  44. [52]

    GoEmotions: A dataset of fine -grained emotions,

    D. Demszky, D. Movshovitz -Attias, J. Ko, A. Cowen, G. Nemade, and S. Ravi, “GoEmotions: A dataset of fine -grained emotions,” in Proc. 58th Annu. Meet. Assoc. Comput. Linguistics (ACL), 2020, doi: 10.18653/v1/2020.acl-main.59

    2020 doi

  45. [53]

    Emotion dataset for NLP tutorial,

    A. W. Ibrahim, “Emotion dataset for NLP tutorial,” Kaggle, [Online]. Available: https://www.kaggle.com/datasets/abdallahwagih/emotion- dataset/data

  46. [54]

    Emotions,

    N. Elgiriye withana, 2024. “Emotions,” Kaggle, [Online]. Available: https://doi.org/10.34740/KAGGLE/DSV/7563141

    2024

  47. [55]

    Twitter sentiment classification using distant supervision,

    A. Go, R. Bhayani, and L. Huang, 2009. “Twitter sentiment classification using distant supervision,” CS224N Project Report, Stanford, vol. 1, 2009, p. 12. Available: https://huggingface.co/datasets/stanfordnlp/sentiment140

    2009

  48. [56]

    Whisper: A general -purpose speech recognition model,

    OpenAI, “Whisper: A general -purpose speech recognition model,” [Online]. Available: https://openai.com/research/whisper

  49. [2024]

    Available: https://github.com/kraack - tech/multimodal_emotion_detection

    [Online]. Available: https://github.com/kraack - tech/multimodal_emotion_detection

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.