Pith. sign in

REVIEW 4 major objections 6 minor 42 references

This paper introduces the InCarEmo multimodal in-cabin dataset—RGB and infrared video, audio, and dialogue text—and reports a lightweight model, CAMEL, reaching 82.25% emotion-recognition accuracy, above fine-tuned 7B baselines.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 01:20 UTC pith:WV5RY7SS

load-bearing objection A genuinely missing in-cabin multimodal dataset, worth refereeing, but the acted-data validity, small test sets, and an overclaim about fusion all need work. the 4 major comments →

arxiv 2607.14683 v1 pith:WV5RY7SS submitted 2026-07-16 cs.AI

InCarEmo: A Multimodal Dataset for In-Cabin Emotion Recognition and Driver State Monitoring

Hao Yang , Yanyan Zhao , Kewei Zhao , Hongbo Zhang , Tian Zheng , Yusheng Liu , Xing Fu , Bichen Wang
show 6 more authors
Yu Zhang Hao He Zhen Wu Xuda Zhi Yongbo Huang Bing Qin
This is my paper
classification cs.AI
keywords in-cabin emotion recognitiondriver state monitoringmultimodal datasetRGB and infrared videodialogue textfatigue detectiondistraction detectionCAMEL
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces InCarEmo, a public dataset for studying drivers' emotions and attentional state inside a vehicle cabin. Its central claim is that emotion recognition in this setting needs more than video: combining RGB and infrared face video with in-cabin audio and the text of driver–passenger or driver–assistant dialogue captures cues that visual-only datasets miss. To back this, the authors collected 3,600 Chinese emotional clips plus fatigue and distraction samples from 25 trained participants in parked cars, annotated them through expert-reviewed, LLM-assisted labels, and built an auxiliary English version. They also propose CAMEL, a lightweight multimodal model that reaches 82.25% accuracy on the six-class emotion task, above fine-tuned 7B-parameter baselines. A sympathetic reader would care because a reliable multimodal in-cabin benchmark is a precondition for safer, more adaptive driver-vehicle interaction.

Core claim

On its own terms, the paper claims to provide the first in-cabin dataset that simultaneously includes RGB video, infrared/thermal video, audio, and dialogue text for emotion understanding in conversational driving settings. The dataset supports three tasks—emotion recognition over six classes (neutral, happy, sad, angry, surprised, anxious), fatigue detection, and distraction monitoring—with a subject-independent split and expert-verified labels (Cohen's kappa 0.87). The accompanying CAMEL model first fine-tunes each modality encoder separately, then aligns the modalities with cross-modal contrastive loss, and finally combines predictions from several lightweight classifiers by soft voting.

What carries the argument

The load-bearing object is the multimodal in-cabin sample X_t=(V,A,T): synchronized visual (RGB and infrared face crops), audio waveform, and dialogue transcript within a five-second window. Carrying the argument is the three-stage CAMEL pipeline: modality-specific fine-tuning of pretrained vision, speech, and language encoders; a cross-modal contrastive alignment loss that pulls embeddings of the same clip together across modality pairs; and multi-classifier soft voting over unimodal and fused representations at inference. For fatigue and distraction, the mechanism is classical facial geometry—eye aspect ratio, mouth aspect ratio, and PERCLOS eye-closure rate—fed into a lightweight classifi

Load-bearing premise

The dataset's usefulness rests on the assumption that scripted, acted emotional dialogues performed in parked cars by 25 participants, with LLM-generated and expert-checked labels, faithfully represent the emotions and behaviors of real drivers in actual traffic.

What would settle it

Record the same drivers' natural in-cabin behavior during real drives (or a high-fidelity driving simulator), apply models trained on InCarEmo, and compare their predictions and the distribution of observed emotions with the dataset; if accuracy drops to near chance or predictions are systematically biased toward acted expression styles, the benchmark does not transfer to real driving.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the claim holds, in-cabin emotion systems should fuse audio and dialogue text with face video; trimodal input beats the best single modality by a wide margin.
  • A model of under a billion parameters can outperform fine-tuned 7B emotion LLMs on this benchmark, which matters for on-device deployment in vehicles.
  • Infrared/thermal input is a strong complementary channel, particularly for low-light conditions where ordinary video degrades.
  • The auxiliary English benchmark indicates that cross-lingual transfer is nontrivial: accuracy drops mainly in audio-involved settings.
  • The dataset gives the community a common split and protocol for comparing emotion, fatigue, and distraction methods in the cabin.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the clips are scripted and acted in parked vehicles, the dataset's predictive value for real road driving is an open empirical question; naturalistic on-road recordings would be the most direct test.
  • The English benchmark uses synthesized speech, with roughly 40% of samples discarded for quality; observed language gaps may partly reflect synthesis artifacts rather than genuine cross-lingual emotion differences.
  • The contrastive-alignment plus voting recipe is not cabin-specific; it could be transferred to other conversational emotion settings, but that extension is untested here.
  • Fatigue and distraction labels are based on acted behavioral displays such as yawning and phone use; real drowsiness has physiological signatures that acted video may not capture.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces InCarEmo, a Chinese-centered multimodal in-cabin dataset combining RGB video, infrared (IR) video, in-cabin audio, and dialogue text from scripted scenarios with 25 participants in parked vehicles. It defines three tasks — six-class emotion recognition, fatigue detection, and distraction monitoring — and provides an auxiliary English benchmark built via LLM translation and TTS synthesis with a multi-stage filter. The authors present extensive unimodal/bimodal/trimodal baselines and propose CAMEL, a lightweight fusion model that achieves 82.25% accuracy on emotion recognition, outperforming fine-tuned 7B models, and 84.58%/81.06% accuracy on fatigue/distraction detection. They further analyze modality-missing and audio-synthesis conditions.

Significance. If the dataset's validity and benchmark results hold, InCarEmo would be a useful public resource, being one of the few in-cabin datasets that combines RGB, IR, audio, and dialogue text and that explicitly targets conversational driver emotion. The release of data, code, and model weights, together with a reasonably broad set of baselines including zero-shot and fine-tuned large models, would support reproducibility and comparability in in-cabin affective computing. The CAMEL baseline is lightweight relative to the 7B-class baselines, which is valuable for embedded deployment. However, the paper's central empirical claims — that multimodal fusion consistently helps, and that CAMEL-state is best for fatigue/distraction — are weakened by small test sets and an overgeneralized statement contradicted by the paper's own English results. The construct validity of the acted, GPT-4o-scripted emotion labels is also an external-validity risk that the authors need to confront directly.

major comments (4)
  1. [§6.4, Table 4] The text states 'Combining modalities consistently improves performance, with all bimodal variants surpassing unimodal ones.' This is not true for the English benchmark in the same table: English VA achieves 69.47% accuracy while unimodal V achieves 72.52%, a 3.05-point drop. Since Table 4 is the main evidence for the 'benefits of multimodal fusion' claim, the sentence must either be restricted to the Chinese split or the English exception must be acknowledged and analyzed.
  2. [§6.3, Table 3] The fatigue and distraction sets contain 230 and 270 clips total (Table 1), so with the 80/10/10 split the test sets are only 23 and 27 samples. Table 3 reports large accuracy gaps (e.g., 84.58 vs. 66.08 for fatigue) without confidence intervals, significance tests, or exact test-set sizes. With n=23, differences of this magnitude are within the standard error, so the claim that CAMEL-state outperforms Qwen2.5-VL-7B on these tasks is not supported as presented. Please report CIs/bootstraps and exact n, or use leave-one-subject-out/other cross-validation that makes fuller use of the small samples.
  3. [§3.1–3.2] Ground-truth emotion labels are anchored to GPT-4o-written scripts that participants read/act in parked vehicles; expert annotators check only 'consistency between the expressed emotions and the intended labels.' There is no validation against naturalistic driver affect, self-report, or physiological signals. Consequently, the predictive value of InCarEmo for real-world in-cabin driver monitoring is not established. The paper should explicitly state this limitation and temper the real-world claims, and ideally provide a small naturalistic validation set or a transfer experiment to an existing naturalistic/on-road corpus.
  4. [§4.2, §5.2] The fatigue and distraction tasks are defined on multimodal inputs X_t=(V_t,A_t,T_t), but the proposed CAMEL-state model and Table 3 use only visual features — facial landmarks and CLIP visual embeddings. This means the paper presents no multimodal baseline for two of the three core tasks. Please either clarify that the state-monitoring benchmark is unimodal visual, or add audio/text/trimodal ablations for these tasks so that the multimodal claim is consistent across all tasks.
minor comments (6)
  1. [§6.2, Table 2 text] The text says Qwen3-0.6B achieves 7.69% F1, but Table 2 reports 14.52% F1. The numbers should be reconciled.
  2. [§3.2] The sentence 'we constructed 3.6K RGB and IR video clips' is ambiguous: does each clip have both RGB and IR versions, or do the two modalities total 3.6K clips? Please state the modality composition explicitly.
  3. [§6.1] In the first stage, text/audio/visual encoders are described as 'frozen' in some places and 'fine-tuned' in others. Clarify which components receive gradient updates, especially for CLIP and Chinese-HuBERT.
  4. [Table 2] HumanOmni-5B is listed with 1.37B params, which is inconsistent with the '5B' designation. Please verify the model name/parameter count.
  5. [§3.2] The inter-annotator agreement is reported as Cohen's kappa = 0.87. Cohen's kappa is defined for two raters; please report pairwise/Fleiss kappa or specify the exact computation for three annotators.
  6. [§6.4] The English benchmark results are dismissed as 'supplemental evidence,' but the main conclusion about bilingual/multilingual transfer is based on a small filtered TTS corpus; no English human-collected data are used. The wording 'preliminary cross-lingual evaluation' in the abstract should be reflected in the conclusion.

Circularity Check

1 steps flagged

No significant circularity: dataset construction and CAMEL evaluation are self-contained; one minor self-referential element involves GPT-4o generating test content and then serving as a baseline.

specific steps
  1. other [Section 3.2 (Script Generation) and Section 6.2 / Table 2 (GPT-4o baseline)]
    "we used GPT-4o [21] to generate approximately 2,000 multi-turn dialogue scripts that reflect natural interactions between drivers and passengers or between drivers and in-vehicle assistants. Each dialogue was annotated with fine-grained emotional labels by a LLM to capture subtle emotional states. ... GPT-4o achieves the best zero-shot accuracy (75.58%)."

    The test-set gold labels are traceable to GPT-4o-generated scripts and LLM-assigned emotion labels (after human review), and GPT-4o is then evaluated zero-shot against those same labels. A portion of its 75.58% accuracy may reflect self-generation rather than independent emotion recognition. This is benchmark/test contamination, not a fitted-parameter prediction or definitional equivalence; it does not force the central dataset or CAMEL results, which are evaluated on held-out subjects with independently trained encoders.

full rationale

InCarEmo is an empirical dataset paper with no formal derivation chain whose predictions reduce to inputs by construction. The central claims are: (1) release of a new multimodal in-cabin dataset, (2) benchmark results showing fusion helps, and (3) CAMEL's strong accuracy. None of these depend on an equation that equates output to input. The only in-scope concern is that GPT-4o both generated the scripted dialogues/emotion labels and appears as a zero-shot baseline, so its reported 75.58% accuracy is not a fully independent probe of the dataset; this is a provenance/validity caveat rather than a circular derivation. The one self-citation ([30], a survey by the same group) is cited only as general progress context and is not load-bearing. CAMEL's 82.25% is obtained through standard supervised training on a subject-independent split and is not fitted to the test labels. I therefore find no significant circularity, scoring 2 for the minor self-referential GPT-4o evaluation element.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

This is an empirical dataset/benchmark paper; there is no theoretical derivation, so the central claim rests on data-collection and evaluation assumptions rather than free parameters in a model. The listed parameters are hand-chosen design choices that affect the reported benchmark numbers; the axioms concern the validity of the dataset as a proxy for real driving.

free parameters (4)
  • temporal window w = 5 seconds
    Section 5.1 fixes w=5s for all models; emotion and state classification is defined over this window, affecting what cues are captured.
  • contrastive temperature tau = not specified
    Equation (2) defines a cross-modal contrastive loss with temperature tau; the value is not given, yet it scales all embedding similarities and affects alignment.
  • PERCLOS threshold = not specified
    Section 5.2 computes PERCLOS as the proportion of frames with EAR below a 'predefined threshold'; the threshold is not reported, directly determining fatigue labels.
  • English audio filtering thresholds (ASR WER and duration) = kept ~60%, rejected ~40%
    Section 4.3 discards samples where ASR WER or duration mismatch exceeds an unreported threshold; this selects which English samples appear in the benchmark.
axioms (4)
  • domain assumption Acted, scripted emotional performances are representative of natural in-cabin driver emotions.
    Sections 3.1-3.2 describe recording participants acting from GPT-4o-generated scripts in parked vehicles; the entire dataset's validity as a driver-emotion resource assumes this transfer.
  • domain assumption LLM-generated emotion labels, after expert review, are correct ground truth.
    Section 3.2 assigns final emotion labels based on LLM annotation plus expert consistency checks; this assumes the labels reflect the driver's actual expressed emotion.
  • domain assumption Synthesized English TTS audio preserves the emotional content of the original Chinese utterances.
    Section 4.3 constructs the English benchmark by translating text and synthesizing speech with higgs-audio and filtering by ASR/duration; the benchmark's meaning depends on emotion preservation in TTS.
  • domain assumption A subject-independent split with 25 participants gives statistically reliable evaluation.
    Section 6.1 uses an 80/10/10 split with only 25 participants; test sets for fatigue and distraction are ~23 and ~27 clips, so reported accuracy may be dominated by a few subjects.

pith-pipeline@v1.3.0-alltime-deepseek · 12109 in / 15681 out tokens · 153767 ms · 2026-08-02T01:20:51.964960+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of InCarEmo: A Multimodal Dataset for In-Cabin Emotion Recognition and Driver State Monitoring." pith.science (2026). https://pith.science/paper/WV5RY7SS

@misc{pith2026260714683,
  author       = {Pith},
  title        = {Pith review of: InCarEmo: A Multimodal Dataset for In-Cabin Emotion Recognition and Driver State Monitoring},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WV5RY7SS}},
  note         = {Machine review of arXiv:2607.14683}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Understanding driver emotion and state is critical for the next generation of intelligent in-cabin systems that ensure safety and enhance human-vehicle interaction. However, existing public datasets for in-cabin affective computing are largely limited to visual modalities and rarely include conversational information, making it difficult to capture the linguistic and interactive cues underlying driver emotion. To address these gaps, we introduce InCarEmo, a multimodal dataset for in-cabin emotion recognition and driver state monitoring. InCarEmo integrates RGB and infrared video, in-cabin audio, and dialogue text collected from scripted in-cabin scenarios designed to simulate realistic driver behaviors, covering diverse lighting conditions and driving contexts. The dataset supports three primary tasks: 1) multimodal emotion recognition, 2) fatigue detection, and 3) distraction monitoring. In addition to the original Chinese data, we construct an auxiliary English benchmark to support preliminary cross-lingual evaluation. We provide a unified benchmark with extensive baseline results across unimodal and multimodal methods, including analyses under modality-missing and noise conditions. Experimental results demonstrate the benefits of multimodal fusion and reveal remaining challenges under real-world noise and low-light conditions. By releasing InCarEmo, we aim to establish a comprehensive foundation for robust, interpretable, and human-centric in-cabin affective understanding, promoting safer and more empathetic driver-vehicle interaction.

Figures

Figures reproduced from arXiv: 2607.14683 by Bichen Wang, Bing Qin, Hao He, Hao Yang, Hongbo Zhang, Kewei Zhao, Tian Zheng, Xing Fu, Xuda Zhi, Yanyan Zhao, Yongbo Huang, Yusheng Liu, Yu Zhang, Zhen Wu.

Figure 1
Figure 1. Figure 1: Overview of the InCarEmo dataset construction process. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Emotion and topic distributions of the dataset. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: An example of prompting a Video LLM. of driver actions. Given a visual clip 𝑉𝑡 , the encoder produces a feature embedding: 𝑧 𝑣 𝑡 = 𝑓 𝑣 (𝑉𝑡 ), (5) which is then passed through a task-specific classification head: 𝑦ˆ dis 𝑡 = 𝑔𝜙 (𝑧 𝑣 𝑡 ). (6) This classifier is trained using cross-entropy supervision and is designed to identify common in-vehicle distracted behaviors. 6 Experiment 6.1 Experimental Setup We ado… view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of each emotion category recognition [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 1 canonical work pages

  1. [1]

    Shabnam Abtahi, Mona Omidyeganeh, Shervin Shirmohammadi, and Behnoosh Hariri. 2014. YawDD: a yawning detection dataset. InProceedings of the 5th ACM Multimedia Systems Conference(Singapore, Singapore)(MMSys ’14). Association for Computing Machinery, New York, NY, USA, 24–28. doi:10.1145/2557642. 2563678

    work page doi:10.1145/2557642 2014

  2. [2]

    Pongtep Angkititrakul, Matteo Petracca, Amardeep Sathyanarayana, and John HL Hansen. 2007. UTDrive: Driver behavior and speech interactive systems for in- vehicle environments. In2007 IEEE Intelligent Vehicles Symposium. IEEE, 566–569

    2007

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al . 2025. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923(2025)

    Pith/arXiv arXiv 2025

  4. [4]

    Jonathan Binas, Daniel Neil, Shih-Chii Liu, and Tobi Delbruck. 2017. DDD17: End-to-end DAVIS driving dataset.arXiv preprint arXiv:1711.01458(2017)

    Pith/arXiv arXiv 2017

  5. [5]

    Boson AI. 2025. Higgs Audio V2: Redefining Expressiveness in Audio Genera- tion. https://github.com/boson-ai/higgs-audio. GitHub repository. Release blog available at https://www.boson.ai/blog/higgs-audio-v2

    2025

  6. [6]

    Karel A Brookhuis, Dick De Waard, and Wiel H Janssen. 2001. Behavioural impacts of advanced driver assistance systems–an overview.European Journal of Transport and Infrastructure Research1, 3 (2001)

    2001

  7. [7]

    Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. IEMOCAP: Interactive emotional dyadic motion capture database.Language resources and evaluation42, 4 (2008), 335–359

    2008

  8. [8]

    Nan Cao, Yu-Ru Lin, Xiaohua Sun, David Lazer, Shixia Liu, and Huamin Qu

  9. [9]

    Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. 2024. Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning.Advances in Neural Information Processing Systems37 (2024), 110805–110853

    2024

  10. [10]

    Qi Fan, Yutong Li, Yi Xin, Xinyu Cheng, Guanglai Gao, and Miao Ma. 2024. Lever- aging contrastive learning and self-training for multimodal emotion recognition with limited labeled samples. InProceedings of the 2nd International Workshop on Multimodal and Responsible Affective Computing. 72–77

    2024

  11. [11]

    Niki Maria Foteinopoulou and Ioannis Patras. 2024. Emoclip: A vision-language method for zero-shot video facial expression recognition. In2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG). IEEE, 1–10

    2024

  12. [12]

    Adam Geitgey. 2017. face_recognition: Simple facial recognition library for Python. https://github.com/ageitgey/face_recognition. Based on dlib’s state-of- the-art face recognition built with deep learning

    2017

  13. [13]

    Pengcheng Guo and Shixing Liu. 2022. chinese_speech_pretrain. https://github. com/TencentGameMate/chinese_speech_pretrain

    2022

  14. [14]

    Imen Jegham, Anouar Ben Khalifa, Ihsen Alouani, and Mohamed Ali Mahjoub

  15. [15]

    Mira Jeong and Byoung Chul Ko. 2018. Driver’s facial expression recognition in real-time for safe driving.Sensors18, 12 (2018), 4270

    2018

  16. [16]

    Davis E. King. 2009. Dlib-ml: A Machine Learning Toolkit.Journal of Machine Learning Research10 (2009), 1755–1758

    2009

  17. [17]

    Schuller, and Jianhua Tao

    Zheng Lian, Rui Liu, Kele Xu, Bin Liu, Xuefei Liu, Yazhou Zhang, Xin Liu, Yong Li, Zebang Cheng, Haolin Zuo, Ziyang Ma, Xiaojiang Peng, Xie Chen, Ya Li, Erik Cam- bria, Guoying Zhao, Björn W. Schuller, and Jianhua Tao. 2025. MER 2025: When Affective Computing Meets Large Language Models. arXiv:2504.19423 [cs.HC] https://arxiv.org/abs/2504.19423

    Pith/arXiv arXiv 2025

  18. [18]

    Zheng Lian, Haiyang Sun, Licai Sun, Jiangyan Yi, Bin Liu, and Jianhua Tao

  19. [19]

    C Lugaresi, J Tang, H Nash, C McClanahan, E Uboweja, M Hays, F Zhang, CL Chang, MG Yong, J Lee, et al. 1906. Mediapipe: A framework for building percep- tion pipelines. arXiv 2019.arXiv preprint arXiv:1906.081725 (1906)

    Pith/arXiv arXiv 1906

  20. [20]

    Quentin Massoz, Thomas Langohr, Clémentine François, and Jacques G Verly

  21. [21]

    OpenAI. 2024. GPT-4o. https://openai.com/index/hello-gpt-4o. Accessed: 2025- 11-14

    2024

  22. [22]

    2020.DMD: A Large- Scale Multi-modal Driver Monitoring Dataset for Attention and Alertness Analysis

    Juan Diego Ortega, Neslihan Kose, Paola Cañas, Min-An Chao, Alexander Un- nervik, Marcos Nieto, Oihana Otaegui, and Luis Salgado. 2020.DMD: A Large- Scale Multi-modal Driver Monitoring Dataset for Attention and Alertness Analysis. Springer International Publishing, 387–405. doi:10.1007/978-3-030-66823-5_23

    doi:10.1007/978-3-030-66823-5_23 2020

  23. [23]

    Walaa Othman, Alexey Kashevnik, Ammar Ali, and Nikolay Shilov. 2022. Driver- mvt: In-cabin dataset for driver monitoring including video and vehicle telemetry information.Data7, 5 (2022), 62

    2022

  24. [24]

    Sanghyuk Park, Fei Pan, Sunghun Kang, and Chang D Yoo. 2016. Driver drowsi- ness detection system based on feature representation learning using various deep networks. InAsian conference on computer vision. Springer, 154–164

    2016

  25. [25]

    Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2018. Meld: A multimodal multi-party dataset for Conference’17, July 2017, Washington, DC, USA Yang et al. emotion recognition in conversations.arXiv preprint arXiv:1810.02508(2018)

    Pith/arXiv arXiv 2018

  26. [26]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. InInternational conference on machine learning. PmLR, 8748–8763

    2021

  27. [27]

    Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. InProceedings of the conference. Association for computational linguistics. Meeting, Vol. 2019. 6558

    2019

  28. [28]

    Huafei Xiao, Wenbo Li, Guanzhong Zeng, Yingzhang Wu, Jiyong Xue, Juncheng Zhang, Chengmou Li, and Gang Guo. 2022. On-road driver emotion recognition using facial expression.Applied Sciences12, 2 (2022), 807

    2022

  29. [29]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

    Pith/arXiv arXiv 2025

  30. [30]

    Hao Yang, Yanyan Zhao, Yang Wu, Shilong Wang, Tian Zheng, Hongbo Zhang, Zongyang Ma, Wanxiang Che, Shijin Wang, Si Wei, et al. 2025. Large language models meet text-centric multimodal sentiment analysis: A survey.Science China Information Sciences68, 10 (2025), 1–29

    2025

  31. [31]

    Lie Yang, Haohan Yang, Bin-Bin Hu, Yan Wang, and Chen Lv. 2023. A robust driver emotion recognition method based on high-purity feature separation.IEEE Transactions on Intelligent Transportation Systems24, 12 (2023), 15092–15104

    2023

  32. [32]

    Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis.arXiv preprint arXiv:1707.07250(2017)

    Pith/arXiv arXiv 2017

  33. [33]

    Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos.arXiv preprint arXiv:1606.06259(2016)

    Pith/arXiv arXiv 2016

  34. [34]

    AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis- Philippe Morency. 2018. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2236–2246

    2018

  35. [35]

    Sebastian Zepf, Javier Hernandez, Alexander Schmitt, Wolfgang Minker, and Rosalind W Picard. 2020. Driver emotion recognition for intelligent vehicles: A survey.ACM Computing Surveys (CSUR)53, 3 (2020), 1–30

    2020

  36. [36]

    Jiaxing Zhao, Xihan Wei, and Liefeng Bo. 2025. R1-omni: Explainable omni- multimodal emotion recognition with reinforcement learning.arXiv preprint arXiv:2503.05379(2025)

    Pith/arXiv arXiv 2025

  37. [37]

    Jiaxing Zhao, Qize Yang, Yixing Peng, Detao Bai, Shimin Yao, Boyuan Sun, Xiang Chen, Shenghao Fu, Xihan Wei, Liefeng Bo, et al. 2025. Humanomni: A large vision-speech language model for human-centric video understanding.arXiv preprint arXiv:2501.15111(2025)

    Pith/arXiv arXiv 2025

  38. [38]

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. 2023. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852(2023)

    Pith/arXiv arXiv 2023

  39. [2012]

    Whisper: Tracing the spatiotemporal process of information diffusion in real time.IEEE transactions on visualization and computer graphics18, 12 (2012), 2649–2658

    2012

  40. [2016]

    In2016 IEEE Winter Conference on Applications of Computer Vision (W ACV)

    The ULg multimodality drowsiness database (called DROZY) and examples of use. In2016 IEEE Winter Conference on Applications of Computer Vision (W ACV). IEEE, 1–7

  41. [2020]

    doi:10.1016/j.image.2020.115960

    A novel public dataset for multimodal multiview and multispectral driver distraction analysis: 3MDAD.Signal Processing: Image Communication88 (2020), 115960. doi:10.1016/j.image.2020.115960

    arXiv 2020

  42. [2024]

    AffectGPT: Dataset and framework for explainable multimodal emotion recognition.arXiv preprint arXiv:2407.07653(2024)

    Pith/arXiv arXiv 2024