Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Asking a multimodal language model to first describe emotion cues in audio and video, then fine-tuning it on those descriptions, improves affective computing across the benchmarked sentiment, emotion, and humor tasks.

desk verdict A plausibly first trimodal MLLM benchmark for affective computing, but the arXiv version omits all quantitative results and the prompting gain is not isolated from a length/text control. read the letter →

arxiv 2508.02429 v1 pith:6R3HMRXO submitted 2025-08-04 cs.AI cs.LG

Miaosen Luo , Jiesen Long , Zequn Li , Yunying Yang , Yuncheng Jiang , Sijie Mai This is my paper
classification cs.AIcs.LG
keywords multimodalaffectivecomputinglargelanguagemodelsgenerativeknowledgepromptingsupervisedfine-tuninglow-rankadaptationsentimentanalysisemotionrecognitionhumordetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal affective computing tries to infer emotion from text, audio, and video together, and this paper asks whether open-source multimodal large language models (MLLMs) can do that end-to-end on established benchmarks. The paper benchmarks seven such models on six datasets spanning sentiment analysis, emotion recognition, and humor detection, and argues that model architecture and dataset properties explain much of the performance spread. To improve accuracy, it proposes a hybrid strategy: first let the model, without training, generate written descriptions of emotion-relevant cues in the raw audio and video, then feed the original data together with those descriptions into a knowledge-augmented prompt and fine-tune the model with low-rank adapters. The central claim is that this integrated approach beats supervised fine-tuning alone across the benchmarked tasks, which would give practitioners a cheap, prompt-based route to better affective computing without new architectures or new datasets.

What carries the argument

The mechanism is a two-stage generate-then-fine-tune setup. In stage one, the untuned MLLM is prompted zero-shot to produce explicit descriptions of emotional cues from the raw audio and video streams, such as a changing facial expression, a shouting voice, or background noise. In stage two, those descriptions are inserted, together with the original audio, video, and conversational text, into a knowledge-augmented prompt; supervised fine-tuning with low-rank adapter matrices then adjusts only a small fraction of the model's weights. The benchmark part of the paper treats the model zoo itself as an instrument: comparing MLLMs with different visual encoders, audio encoders, and fusion mechanisms, and correlating performance with dataset properties such as modality dominance.

What would settle it

Run the proposed pipeline on one benchmark dataset, then re-run it with the same model and fine-tuning but with the model-generated audio and video descriptions replaced by descriptions taken from other clips; if accuracy does not drop, the content of the descriptions is not the load-bearing ingredient. A second check is to regenerate the descriptions with a differently trained MLLM and see whether the gains survive the switch.

Watch

Extended reading notes

Core claim

On its own terms, the paper makes two claims. The first is descriptive: on trimodal input, current MLLMs vary widely, and the variation tracks the fit between each model's pretraining choices, especially the audio encoder, and the downstream task, as well as the modality that dominates each dataset. The second is constructive: the proposed generate-then-fine-tune pipeline improves performance. The model first uses its zero-shot ability to produce explicit descriptions of emotional cues in the audio and video streams; these descriptions are then aggregated with the original audiovisual inputs and the dialogue text into one prompt; and supervised fine-tuning with low-rank adapters aligns the model to the task. The paper reports that this pipeline outperforms fine-tuning without generated descriptions on several datasets, with the gains concentrated where the audio or visual modality carries the discriminative signal, and with little or no gain on text-dominated datasets where the extra descriptions are redundant.

Load-bearing premise

The method only works if the descriptions the model writes about the audio and video are accurate and emotionally relevant; if those descriptions are noisy or biased, the measured gains could come from the prompting format rather than from real knowledge.

Editorial extensions

If this is right

  • Practitioners can apply the recipe directly: feed raw audio and video to an MLLM, collect its written emotion-cue descriptions, add them to the prompt, and fine-tune only low-rank adapters.
  • Model selection for affective computing can be guided by the match between an encoder's pretraining objective and the target task; audio towers trained for speech recognition transfer better to sentiment tasks.
  • The size of the prompting gain is dataset-dependent, so gains should be expected mainly where audio or visual information, not text, carries the emotion signal.
  • The prompting boost is additive on top of supervised fine-tuning for several of the benchmarked sentiment, emotion, and humor detection tasks, rather than an either-or alternative.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the descriptions come from the model itself, nothing in the method prevents using a stronger model to generate descriptions offline and feeding them to a weaker model at inference time; that would test whether the knowledge, rather than the model's own parameters, is doing the work.
  • A pure-prompting ablation, generative knowledge prompting without any fine-tuning, would separate the contribution of the prompt pattern from the contribution of the parameter update, and the paper's design makes that a natural next experiment.
  • The audio-encoder finding suggests a cheap selection heuristic for practical systems: among models of similar size, prefer one whose audio encoder was trained for speech-to-text, because that pretraining task appears to transfer well to affective judgments; this is testable by swapping audio encoders within one architecture.
  • The generated descriptions could also serve as external supervision: a large MLLM's descriptions of a dataset could be used to train a smaller, cheaper model that lacks multimodal input, effectively distilling the listening-and-watching step into text.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript benchmarks seven open-source multimodal large language models (MLLMs) that jointly process text, audio, and video on six established affective computing datasets (CMU-MOSI, CMU-MOSEI, CH-SIMS, CH-SIMS v2, MELD, UR-FUNNY v2). It compares these models with traditional multimodal learning baselines and analyzes how model architecture and dataset properties affect performance. To improve MLLM performance, the paper proposes a hybrid strategy that first uses zero-shot MLLM generation to produce textual descriptions of audio and video emotional cues, then incorporates these descriptions into the prompt and applies supervised fine-tuning with LoRA. The abstract and conclusion claim that this integrated approach significantly improves performance across tasks. The manuscript also releases code at a public repository.

Significance. If the claimed results are substantiated, the benchmark would be a useful contribution: it is among the first systematic evaluations of trimodal open-source MLLMs on standard affective computing benchmarks, and the proposed generative-knowledge-prompting-plus-SFT strategy is simple and potentially transferable. The analysis of audio-encoder pretraining alignment with downstream tasks also offers actionable guidance. However, the current manuscript does not contain the quantitative results needed to assess these claims, so the significance cannot yet be evaluated. The paper's stated strengths—open code, multiple models, multiple datasets—are conditional on the actual evidence being present and correct.

major comments (4)
  1. [Abstract, Section IV, Appendix] The central claim of significant performance improvement is not supported by any numerical results in the manuscript. No accuracy, F1, MAE, correlation, or error-bar values appear for any model or dataset. Section IV says that detailed evaluation information is in the Appendix, but the Appendix only defines metrics and describes model architectures; it contains no result tables. The abstract's 'significantly improves performance' and the conclusion's 'significantly enhanced' therefore cannot be checked. This is load-bearing: the paper is an empirical benchmark and boosting study, and without the actual numbers the entire contribution is unverifiable. Please add complete results tables with per-model, per-dataset scores for all baselines and for the proposed method, together with statistical significance or error bars where applicable.
  2. [Section III.C.2, Figure 2, Section V.C] The proposed method is not experimentally isolated from confounds. The SFT+knowledge condition differs from the SFT baseline both in the addition of generated descriptions and in the length and content of the prompt text. The example in Figure 2 and Section V.C shows that the generated descriptions contain emotion words such as 'fear', 'pain', 'alarm', and 'sense of urgency.' These words may directly leak the target label or provide extra contextual cues that are unrelated to faithful affective knowledge extraction. Without a control condition using length-matched generic or emotion-free text, and without any evaluation of the quality of the generated descriptions (e.g., human ratings or agreement with human annotations), the observed gains cannot be attributed to the proposed knowledge-prompting mechanism. Please add such a control and a description-quality analysis.
  3. [Section IV.B] The hyperparameter reporting is incomplete and one sentence is truncated. The text reads 'the training epoch was selected from' and then breaks off; no epoch values, learning rates, LoRA ranks, or optimization details are reported for the main experiments. Without these details, the experiments are not reproducible, and the claimed results cannot be independently verified. Please provide the complete experimental setup for every model and dataset, including prompt templates and split information.
  4. [Section V.D] The analysis of modality contributions relies on Figures 3 and 4, but these figures are not present in the manuscript. The claims about audio-encoder pretraining alignment and visual encoder advantages rest entirely on these missing figures. Please include the figures with clear axes, legends, and captions, and ensure the quantitative values they display are consistent with the results tables.
minor comments (4)
  1. [Section IV.A] There is a typo in the second paragraph: 'or the CMU-MOSI and CMU-MOSEI datasets' should be 'for the CMU-MOSI and CMU-MOSEI datasets.'
  2. [General] Several citations in the reference list contain formatting artifacts, such as 'V olume' in [43] and the missing space in 'arXiv e-prints , pp. arXiv–2502' in [39]. Please proofread the bibliography.
  3. [Section III.A] The dataset statistics mentioned at the start of Section III.A are not provided in the main text or appendix; a small table of dataset sizes, modalities, languages, and label distributions would help readers assess the benchmark scope.
  4. [Section V.D] The text states that 'all MLLMs achieve superior performance' on text modality, but without the underlying numbers this statement is too vague. Please tie this qualitative claim to specific metrics from the results tables.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the benchmark and SFT comparisons are anchored to external labeled datasets and prior published baselines, and the generative-knowledge step is self-distillation rather than a definitional reduction.

full rationale

The paper's central comparisons are anchored to externally labeled datasets (CMU-MOSI, CMU-MOSEI, CH-SIMS, CH-SIMS v2, MELD, UR-FUNNY v2) and to external baseline methods, so the benchmark itself is not circular. The proposed method first extracts zero-shot descriptions from the MLLM and then uses those descriptions as auxiliary input during supervised fine-tuning; the training target remains the original human-annotated label. This is a form of self-distillation or input augmentation, not a case where the predicted quantity is defined in terms of the fitted quantity or where a fitted parameter is renamed as a prediction. The self-citations among the comparison baselines (e.g., MOAC, C-MIB, MGT, SemanticMAC) are prior published methods used as benchmark comparators; they do not supply assumptions that force the paper's conclusion, and no uniqueness or ansatz-justifying theorem is imported from the authors' own prior work. The manuscript's failure to include result tables and its truncated hyperparameter sentence make the claimed 'significant improvements' unverifiable, but that is an evidential or completeness problem rather than a circularity problem. No specific reduction of the claimed result to its own inputs by construction is present in the text.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new theoretical quantities are introduced. The central empirical claims rest on standard datasets, public model weights, and a small set of fine-tuning hyperparameters and prompting assumptions.

free parameters (2)
  • SFT training epochs = unspecified in visible text
    The paper reports candidate epoch ranges for Emotion-LLaMA and PandaGPT but the final selected epochs are not given. This choice directly affects the reported performance difference between SFT alone and knowledge-prompt SFT.
  • LoRA rank and learning rate = not reported in visible text
    The rank-r matrices A and B in Eq. 1 are central to the LoRA update, yet the rank and learning rate are not specified. These hyperparameters can change the magnitude of the reported improvements.
assumptions (4)
  • domain assumption Human annotations in the six MAC datasets are treated as ground truth labels for sentiment, emotion, and humor.
    The entire benchmark score depends on the validity of these labels as affective ground truth; the paper does not audit label noise or inter-annotator agreement.
  • ad hoc to paper The MLLMs' zero-shot generated descriptions of audio and video faithfully capture emotionally relevant cues.
    The proposed knowledge-prompting strategy assumes these descriptions are accurate and useful; if they are noisy or biased, the reported improvement could be an artifact. This enters in Section III.C.2 and Figure 2.
  • domain assumption Open-source MLLM checkpoints reproduce the capabilities reported in their original papers.
    The benchmark compares models as packaged in their public repositories, so any discrepancy between the released weights and the original model descriptions affects all comparisons.
  • standard math Eq. 1 (W_new = W + A x B) describes a valid low-rank adaptation that preserves the pre-trained model's representational capacity during SFT.
    The LoRA update is the standard formulation introduced in the cited literature; the paper uses it without modification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting." pith.science (2026). https://pith.science/paper/6R3HMRXO

@misc{pith2026250802429,
  author       = {Pith},
  title        = {Pith review of: Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6R3HMRXO}},
  note         = {Machine review of arXiv:2508.02429}
}
read the original abstract

Multimodal Affective Computing (MAC) aims to recognize and interpret human emotions by integrating information from diverse modalities such as text, video, and audio. Recent advancements in Multimodal Large Language Models (MLLMs) have significantly reshaped the landscape of MAC by offering a unified framework for processing and aligning cross-modal information. However, practical challenges remain, including performance variability across complex MAC tasks and insufficient understanding of how architectural designs and data characteristics impact affective analysis. To address these gaps, we conduct a systematic benchmark evaluation of state-of-the-art open-source MLLMs capable of concurrently processing audio, visual, and textual modalities across multiple established MAC datasets. Our evaluation not only compares the performance of these MLLMs but also provides actionable insights into model optimization by analyzing the influence of model architectures and dataset properties. Furthermore, we propose a novel hybrid strategy that combines generative knowledge prompting with supervised fine-tuning to enhance MLLMs' affective computing capabilities. Experimental results demonstrate that this integrated approach significantly improves performance across various MAC tasks, offering a promising avenue for future research and development in this field. Our code is released on https://github.com/LuoMSen/MLLM-MAC.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. C2F-Thinker: Coarse-to-Fine Reasoning with Hint-Guided Reinforcement Learning for Multimodal Sentiment Analysis

    cs.CL 2026-03 unverdicted novelty 6.0 of 10

    C2F-Thinker combines structured coarse-to-fine chain-of-thought reasoning with hint-guided GRPO reinforcement learning to achieve competitive fine-grained sentiment regression and superior cross-domain generalization ...

  2. QASA: Quality-Aware Semantic Augmentation for Robust Multimodal Sentiment Analysis

    cs.LG 2026-01 reject novelty 5.0 of 10

    Diffusion-generated video/audio samples weighted by a learned quality scorer are claimed to improve multimodal sentiment analysis on CH-SIMS, CMU-MOSI, and MUStARD.

Reference graph

Works this paper leans on

74 extracted references · 39 canonical work pages · cited by 2 Pith papers

  1. [1]

    A review of affective computing: From unimodal analysis to multimodal fusion,

    S. Poria, E. Cambria, R. Bajpai, and A. Hussain, “A review of affective computing: From unimodal analysis to multimodal fusion,” Information fusion, vol. 37, pp. 98–125, 2017

    2017

  2. [2]

    A systematic review on affective computing: Emotion models, databases, and recent advances,

    Y . Wang, W. Song, W. Tao, A. Liotta, D. Yang, X. Li, S. Gao, Y . Sun, W. Ge, W. Zhang et al. , “A systematic review on affective computing: Emotion models, databases, and recent advances,” Information Fusion , vol. 83, pp. 19–52, 2022

    work page 2022

  3. [3]

    An effective data fusion methodology for multi-modal emotion recognition: A survey,

    S. R. Sanku and B. Sandhya, “An effective data fusion methodology for multi-modal emotion recognition: A survey,” International Journal , vol. 12, no. 7, 2024

    work page 2024

  4. [4]

    Surveying the mllm landscape: A meta-review of current surveys,

    M. Li, K. Chen, Z. Bi, M. Liu, B. Peng, Q. Niu, J. Liu, J. Wang, S. Zhang, X. Pan et al., “Surveying the mllm landscape: A meta-review of current surveys,” arXiv preprint arXiv:2409.18991 , 2024

    arXiv 2024

  5. [5]

    Learning by comparing: Boosting multi- modal affective computing through ordinal learning,

    S. Mai, Y . Zeng, and H. Hu, “Learning by comparing: Boosting multi- modal affective computing through ordinal learning,” in Proceedings of the ACM on Web Conference 2025 , 2025, pp. 2120–2134

    work page 2025

  6. [6]

    Llm-based nlg evaluation: Current status and challenges,

    M. Gao, X. Hu, X. Yin, J. Ruan, X. Pu, and X. Wan, “Llm-based nlg evaluation: Current status and challenges,” Computational Linguistics , pp. 1–27, 2025

    work page 2025

  7. [7]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision),

    Z. Yang, L. Li, K. Lin, J. Wang, C.-C. Lin, Z. Liu, and L. Wang, “The dawn of lmms: Preliminary explorations with gpt-4v (ision),” arXiv preprint arXiv:2309.17421, vol. 9, no. 1, p. 1, 2023

    arXiv 2023

  8. [8]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems , vol. 36, pp. 34 892– 34 916, 2023. 8

    work page 2023

Show all 74 references
  1. [9]

    Gemini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al., “Gemini: a family of highly capable multimodal models,”arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  2. [10]

    Qwen-vl: A versatile vision-language model for understanding, localization,

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A versatile vision-language model for understanding, localization,” Text Reading, and Beyond , vol. 2, p. 1, 2023

    2023

  3. [11]

    A survey on multimodal large language models,

    S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, “A survey on multimodal large language models,” National Science Review , vol. 11, no. 12, p. nwae403, 2024

    2024

  4. [12]

    Emotionqueen: A benchmark for evaluating empathy of large language models,

    Y . Chen, H. Wang, S. Yan, S. Liu, Y . Li, Y . Zhao, and Y . Xiao, “Emotionqueen: A benchmark for evaluating empathy of large language models,” arXiv preprint arXiv:2409.13359 , 2024

    2024 arXiv

  5. [13]

    Eemo-bench: A benchmark for multi-modal large lan- guage models on image evoked emotion assessment,

    L. Gao, Z. Jia, Y . Zeng, W. Sun, Y . Zhang, W. Zhou, G. Zhai, and X. Min, “Eemo-bench: A benchmark for multi-modal large lan- guage models on image evoked emotion assessment,” arXiv preprint arXiv:2504.16405, 2025

    2025

  6. [14]

    Can large language models help multimodal language analysis? mmla: A comprehensive benchmark,

    H. Zhang, Z. Li, Y . Zhu, H. Xu, P. Wang, H. Zhu, J. Zhou, and J. Zhang, “Can large language models help multimodal language analysis? mmla: A comprehensive benchmark,” arXiv preprint arXiv:2504.16427 , 2025

    2025 arXiv

  7. [15]

    The future of mllm prompting is adaptive: A comprehensive experimental evaluation of prompt engineering methods for robust multimodal performance,

    A. Mohanty, V . B. Parthasarathy, and A. Shahid, “The future of mllm prompting is adaptive: A comprehensive experimental evaluation of prompt engineering methods for robust multimodal performance,” arXiv preprint arXiv:2504.10179, 2025

    2025 arXiv

  8. [16]

    Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos,

    A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, “Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos,” arXiv preprint arXiv:1606.06259 , 2016

    2016 arXiv

  9. [17]

    Ch- sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality,

    W. Yu, H. Xu, F. Meng, Y . Zhu, Y . Ma, J. Wu, J. Zou, and K. Yang, “Ch- sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality,” in Proceedings of the 58th annual meeting of the association for computational linguistics , 2020, pp. 3718–3727

    2020

  10. [18]

    Make acoustic and visual cues matter: Ch-sims v2. 0 dataset and av-mixup consistent module,

    Y . Liu, Z. Yuan, H. Mao, Z. Liang, W. Yang, Y . Qiu, T. Cheng, X. Li, H. Xu, and K. Gao, “Make acoustic and visual cues matter: Ch-sims v2. 0 dataset and av-mixup consistent module,” in Proceedings of the 2022 international conference on multimodal interaction , 2022, pp. 247–258

    2022

  11. [19]

    Meld: A multimodal multi-party dataset for emotion recognition in conversations,

    S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihal- cea, “Meld: A multimodal multi-party dataset for emotion recognition in conversations,” arXiv preprint arXiv:1810.02508 , 2018

    2018 arXiv

  12. [20]

    Ur-funny: A multimodal language dataset for understanding humor,

    M. K. Hasan, W. Rahman, A. Zadeh, J. Zhong, M. I. Tanveer, L.- P. Morency et al. , “Ur-funny: A multimodal language dataset for understanding humor,” arXiv preprint arXiv:1904.06618 , 2019

    1904 arXiv

  13. [21]

    Generated knowledge prompting for commonsense reasoning,

    J. Liu, A. Liu, X. Lu, S. Welleck, P. West, R. L. Bras, Y . Choi, and H. Hajishirzi, “Generated knowledge prompting for commonsense reasoning,” arXiv preprint arXiv:2110.08387 , 2021

    2021 arXiv

  14. [22]

    Self-attentive feature-level fusion for multimodal emotion detection,

    D. Hazarika, S. Gorantla, S. Poria, and R. Zimmermann, “Self-attentive feature-level fusion for multimodal emotion detection,” in 2018 IEEE Conference on multimedia information processing and retrieval (MIPR) . IEEE, 2018, pp. 196–201

    2018

  15. [23]

    Emotion recognition using feature-level fusion of facial expressions and body gestures,

    T. Keshari and S. Palaniswamy, “Emotion recognition using feature-level fusion of facial expressions and body gestures,” in 2019 international conference on communication and electronics systems (ICCES) . IEEE, 2019, pp. 1184–1189

    2019

  16. [24]

    Decision-level fusion method for emotion recognition using multimodal emotion recog- nition information,

    K.-S. Song, Y .-H. Nho, J.-H. Seo, and D.-s. Kwon, “Decision-level fusion method for emotion recognition using multimodal emotion recog- nition information,” in 2018 15th international conference on ubiquitous robots (UR). IEEE, 2018, pp. 472–476

    2018

  17. [25]

    Deep learning-based late fusion of multi- modal information for emotion classification of music video,

    Y . R. Pandeya and J. Lee, “Deep learning-based late fusion of multi- modal information for emotion classification of music video,” Multime- dia Tools and Applications , vol. 80, no. 2, pp. 2887–2905, 2021

    2021

  18. [26]

    A joint cross-attention model for audio-visual fusion in dimensional emotion recognition,

    R. G. Praveen, W. C. de Melo, N. Ullah, H. Aslam, O. Zeeshan, T. Denorme, M. Pedersoli, A. L. Koerich, S. Bacon, P. Cardinal et al. , “A joint cross-attention model for audio-visual fusion in dimensional emotion recognition,” in Proceedings of the IEEE/CVF conference on comput...

    2022

  19. [27]

    Speech emotion recognition with co-attention based multi-level acoustic information,

    H. Zou, Y . Si, C. Chen, D. Rajan, and E. S. Chng, “Speech emotion recognition with co-attention based multi-level acoustic information,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022, pp. 7367–7371

    2022

  20. [28]

    Omni-emotion: Extending video mllm with detailed face and audio modeling for multimodal emotion analysis,

    Q. Yang, D. Bai, Y .-X. Peng, and X. Wei, “Omni-emotion: Extending video mllm with detailed face and audio modeling for multimodal emotion analysis,” arXiv preprint arXiv:2501.09502 , 2025

    2025 arXiv

  21. [29]

    Omnivox: Zero-shot emotion recognition with omni-llms,

    J. Murzaku and O. Rambow, “Omnivox: Zero-shot emotion recognition with omni-llms,” arXiv preprint arXiv:2503.21480 , 2025

    2025 arXiv

  22. [30]

    Mellm: Exploring llm-powered micro-expression understanding enhanced by subtle motion perception,

    Z. Zhang, S. Zhao, S. Liu, S. Yin, X. Mao, T. Xu, and E. Chen, “Mellm: Exploring llm-powered micro-expression understanding enhanced by subtle motion perception,” arXiv preprint arXiv:2505.07007 , 2025

    2025

  23. [31]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PmLR, 2021, pp. 8748–8763

    2021

  24. [32]

    Videomae: Masked autoen- coders are data-efficient learners for self-supervised video pre-training,

    Z. Tong, Y . Song, J. Wang, and L. Wang, “Videomae: Masked autoen- coders are data-efficient learners for self-supervised video pre-training,” Advances in neural information processing systems , vol. 35, pp. 10 078– 10 093, 2022

    2022

  25. [33]

    Beats: Audio pre-training with acoustic tokenizers,

    S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, and F. Wei, “Beats: Audio pre-training with acoustic tokenizers,” arXiv preprint arXiv:2212.09058, 2022

    2022 arXiv

  26. [34]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” in International conference on machine learning . PMLR, 2023, pp. 19 730–19 742

    2023

  27. [35]

    Salmonn: Towards generic hearing abilities for large language models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “Salmonn: Towards generic hearing abilities for large language models,” arXiv preprint arXiv:2310.13289 , 2023

    2023 arXiv

  28. [36]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models,

    Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models,” arXiv preprint arXiv:2311.07919 , 2023

    2023 arXiv

  29. [37]

    Minicpm-v: A gpt-4v level mllm on your phone,

    Y . Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. He et al. , “Minicpm-v: A gpt-4v level mllm on your phone,” arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  30. [38]

    Humanomni: A large vision-speech language model for human-centric video understanding,

    J. Zhao, Q. Yang, Y . Peng, D. Bai, S. Yao, B. Sun, X. Chen, S. Fu, X. Wei, L. Bo et al. , “Humanomni: A large vision-speech language model for human-centric video understanding,” arXiv preprint arXiv:2501.15111, 2025

    2025 arXiv

  31. [39]

    Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment,

    Z. Liu, Y . Dong, J. Wang, Z. Liu, W. Hu, J. Lu, and Y . Rao, “Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment,” arXiv e-prints , pp. arXiv–2502, 2025

    2025

  32. [40]

    Qwen2. 5-omni technical report,

    J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y . Fan, K. Danget al., “Qwen2. 5-omni technical report,” arXiv preprint arXiv:2503.20215, 2025

    2025 arXiv

  33. [41]

    Learning emotional prompt features with multiple views for visual emotion analysis,

    Q. Xu, Y . Wei, S. Yuan, J. Wu, L. Wang, and C. Wu, “Learning emotional prompt features with multiple views for visual emotion analysis,” Information Fusion , vol. 108, p. 102366, 2024

    2024

  34. [42]

    Visual and textual prompts in vllms for enhancing emotion recognition,

    Z. Wang, Q. Zhang, P. Zhang, W. Niu, K. Zhang, R. Sankaranarayana, S. Caldwell, and T. Gedeon, “Visual and textual prompts in vllms for enhancing emotion recognition,” IEEE Transactions on Circuits and Systems for Video Technology , 2025

    2025

  35. [43]

    Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph,

    A. B. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency, “Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long ...

    2018

  36. [44]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms,

    Z. Cheng, S. Leng, H. Zhang, Y . Xin, X. Li, G. Chen, Y . Zhu, W. Zhang, Z. Luo, D. Zhao et al. , “Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms,” arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

  37. [45]

    Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning,

    Z. Cheng, Z.-Q. Cheng, J.-Y . He, K. Wang, Y . Lin, Z. Lian, X. Peng, and A. Hauptmann, “Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning,” Advances in Neural Information Processing Systems, vol. 37, pp. 110 805–110 853, 2024

    2024

  38. [46]

    Pandagpt: One model to instruction-follow them all,

    Y . Su, T. Lan, H. Li, J. Xu, Y . Wang, and D. Cai, “Pandagpt: One model to instruction-follow them all,” arXiv preprint arXiv:2305.16355 , 2023

    2023 arXiv

  39. [47]

    Lora: Low-rank adaptation of large language models

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chen et al. , “Lora: Low-rank adaptation of large language models.” ICLR, vol. 1, no. 2, p. 3, 2022

    2022

  40. [48]

    Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations,

    S. Mai, Y . Zeng, and H. Hu, “Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations,” IEEE Transactions on Multimedia , vol. 25, pp. 4121–4134, 2022

    2022

  41. [49]

    Injecting multimodal informa- tion into pre-trained language model for multimodal sentiment analysis,

    S. Mai, Y . Zeng, A. Xiong, and H. Hu, “Injecting multimodal informa- tion into pre-trained language model for multimodal sentiment analysis,” IEEE Transactions on Affective Computing , 2025

    2025

  42. [50]

    Towards explainable fusion and balanced learning in multimodal sentiment analysis,

    M. Luo, Y . Jiang, and S. Mai, “Towards explainable fusion and balanced learning in multimodal sentiment analysis,” arXiv preprint arXiv:2504.12151, 2025

    2025 arXiv

  43. [51]

    Hgtfm: Hierarchical gating-driven transformer fusion model for robust multimodal sentiment analysis,

    C. Yang, Z. Liang, D. Yan, Z. Hu, and T. Wu, “Hgtfm: Hierarchical gating-driven transformer fusion model for robust multimodal sentiment analysis,” IEEE Access , 2025

    2025

  44. [52]

    End-to-end semantic- centric video-based multimodal affective computing,

    R. Lin, Y . Zeng, S. Mai, and H. Hu, “End-to-end semantic- centric video-based multimodal affective computing,” arXiv preprint arXiv:2408.07694, 2024

    2024 arXiv

  45. [53]

    Llamafactory: Unified efficient fine-tuning of 100+ language models,

    Y . Zheng, R. Zhang, J. Zhang, Y . Ye, Z. Luo, Z. Feng, and Y . Ma, “Llamafactory: Unified efficient fine-tuning of 100+ language models,” 9 in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 3: System Demonstrations) . Bangkok,...

    2024 arXiv

  46. [54]

    Flashattention-2: Faster attention with better parallelism and work partitioning,

    T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning,” arXiv preprint arXiv:2307.08691 , 2023

    2023 arXiv

  47. [55]

    Divide, conquer and combine: Hierarchical feature fusion network with local and global perspectives for multimodal affective computing,

    S. Mai, H. Hu, and S. Xing, “Divide, conquer and combine: Hierarchical feature fusion network with local and global perspectives for multimodal affective computing,” in Proceedings of the 57th annual meeting of the association for computational linguistics , 2019, pp. 481–492

    2019

  48. [56]

    Qwen2. 5-coder technical report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu et al. , “Qwen2. 5-coder technical report,” arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  49. [57]

    Robust speech recognition via large-scale weak supervi- sion,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervi- sion,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518

    2023

  50. [58]

    Qwen2. 5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang et al. , “Qwen2. 5-vl technical report,” arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  51. [59]

    A survey on vision transformer,

    K. Han, Y . Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y . Tang, A. Xiao, C. Xu, Y . Xuet al., “A survey on vision transformer,” IEEE transactions on pattern analysis and machine intelligence , vol. 45, no. 1, pp. 87–110, 2022

    2022

  52. [60]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” in Proceedings of the IEEE/CVF international conference on computer vision , 2023, pp. 11 975–11 986

    2023

  53. [61]

    Llava-onevision: Easy visual task transfer,

    B. Li, Y . Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y . Li, Z. Liu et al. , “Llava-onevision: Easy visual task transfer,” arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  54. [62]

    Oryx mllm: On- demand spatial-temporal understanding at arbitrary resolution,

    Z. Liu, Y . Dong, Z. Liu, W. Hu, J. Lu, and Y . Rao, “Oryx mllm: On- demand spatial-temporal understanding at arbitrary resolution,” arXiv preprint arXiv:2409.12961, 2024

    2024 arXiv

  55. [63]

    Qwen2 technical report,

    Q. Team, “Qwen2 technical report,” arXiv preprint arXiv:2407.10671 , 2024

    2024 arXiv

  56. [64]

    Imagebind: One embedding space to bind them all,

    R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V . Alwala, A. Joulin, and I. Misra, “Imagebind: One embedding space to bind them all,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 15 180–15 190

    2023

  57. [65]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM transactions on audio, speech, and language processing , vol. 29, pp. 3451–3460, 2021

    2021

  58. [66]

    Mae-dfer: Efficient masked au- toencoder for self-supervised dynamic facial expression recognition,

    L. Sun, Z. Lian, B. Liu, and J. Tao, “Mae-dfer: Efficient masked au- toencoder for self-supervised dynamic facial expression recognition,” in Proceedings of the 31st ACM International Conference on Multimedia , 2023, pp. 6110–6121

    2023

  59. [67]

    Eva: Exploring the limits of masked visual representation learning at scale,

    Y . Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y . Cao, “Eva: Exploring the limits of masked visual representation learning at scale,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2023, pp. 19 358–19 369

    2023

  60. [68]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023. APPENDIX A. Evaluation Metrics We use different evalua...

    2023 arXiv

  61. [69]

    CMU-MOSI and CMU-MOSEI datasets: We use the following evaluation metrics to measure the performance of the model: (1) Acc7: the accuracy of classifying sentiment scores into seven discrete classes (predictions rounded to nearest integer in [-3, 3]); (2) Acc2: the accuracy for ...

  62. [70]

    CH-SIMS and CH-SIMS v2 datasets.: We use the fol- lowing metrics: (1) Acc5: the accuracy of dividing emotional scores into five discrete categories (predictions rounded to nearest integer in [-1, 1]); (2) Acc3: the accuracy in categoriz- ing emotions into three types (positive...

  63. [71]

    MELD dataset: We use the following metrics: (1) w- Acc: the weighted accuracy; (2) w-F1: the weighted average F1 score

  64. [72]

    UR-FUNNY v2 dataset: We use the following metrics: (1) w-Precision: the weighted precision; (2) w-Recall: the weighted recall; (3) w-Acc; (4) w-F1. B. Multimodal Large Language Models

  65. [73]

    Its core architecture follows the Thinker-Talker design

    Qwen2.5Omni: The Qwen2.5-Omni [40] is an end-to- end multimodal model capable of processing a variety of modalities, including text, image, audio, and video, while simultaneously generating text and natural speech responses. Its core architecture follows the Thinker-Talker des...

  66. [74]

    Its key innovation lies in the ability to simultaneously process visual and speech information in human-centric scenes

    HumanOmni: The HumanOmni [38] is a large vision- speech language model designed to focus on human-centric video understanding. Its key innovation lies in the ability to simultaneously process visual and speech information in human-centric scenes. The model comprises three spec...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.