Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

Emotion-Qwen: A Unified Framework for Emotion and Vision Understanding

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Emotion-Qwen claims that emotional understanding and general vision-language reasoning can coexist in a single 7B multimodal model, and reports state-of-the-art scores on both families of benchmarks.

desk verdict A solid system paper with a genuinely new dataset (VER) and a plausible connector design, but the headline SOTA numbers are not independently checkable until the fine-tuning/evaluation split is documented and error bars are added. read the letter →

arxiv 2505.06685 v3 pith:R22TTCKV submitted 2025-05-10 cs.MM cs.CV

Dawei Huang , Qing Li , Chuan Yan , Zebang Cheng , Zihao Han , Yurong Huang , Xiang Li , Bin Li
show 4 more authors
Xiaohui Wang Zheng Lian Zhi-Qi Cheng Xiaojiang Peng
This is my paper · ORCID
classification cs.MMcs.CV
keywords multimodalemotionrecognitionvision-languagemodelsmixtureofexpertscatastrophicforgettingreasoningVideodatasetinstructiontuningfacialexpression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that emotion understanding and general vision-language ability do not have to be traded off in a multimodal model. It introduces Emotion-Qwen, a 7B vision-and-text model whose visual tokens are routed through two experts—one tuned for emotional content, one for general scenes—and which is pre-trained in three stages so the emotion expert can be added without erasing general knowledge. To support fine-grained reasoning, the authors build the Video Emotion Reasoning (VER) dataset, over 40,000 bilingual clips with context-aware annotations. The reported results are state-of-the-art on emotion benchmarks (DFEW, EMER, EmoSet, MER2024) while MMBench, TextVQA, and ScienceQA remain competitive with the best open general LMMs. If correct, this gives a practical recipe for emotion-specialized assistants that do not lose general competence.

What carries the argument

The load-bearing mechanism is the Hybrid Compressor, a mixture-of-experts projector with two MLP experts—an Emotion Expert and a General Expert—whose outputs are combined by an attention-based gating network: $V_{out} = G \odot V_{emo} + (1-G) \odot V_{gen}$. The gate decides per input how much of the visual representation should be processed emotionally versus generally, and an ablation records task-dependent routing (for example, 0.63 of the gate weight goes to the Emotion Expert on MER2024 versus 0.37 on MMBench). Around this sits the Facial Emotion Capture module, which keeps key facial-expression frames and masks backgrounds, and a three-stage pretraining schedule that first aligns the general expert, then warms the emotion expert, then fine-tunes the LLM on instruction data. During emotion instruction tuning, separate low-rank adapters (LoRA) per dataset let the Qwen2.5 backbone specialize without full retraining.

What would settle it

Run the instruction-tuning evaluation on a cleaned version of the benchmarks from which every clip that resembles a VER training clip (same source video, scene, or speaker from MAFW or MER2024) has been removed; if the reported margins over Emotion-LLaMA and MMA-DFER shrink or disappear, the balanced-capability claim is not established. Independently, collect human ratings on a random sample of EMER responses and compare them with the ChatGPT-assigned Clue/Label scores; low agreement would put the emotion-reasoning metric in doubt.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a single 7B model can specialize in emotion without generalizing worse: Emotion-Qwen reaches 87.3 on MMBench, 87.9 on TextVQA, and 77.2 on ScienceQA in zero-shot evaluation, and after instruction fine-tuning with per-dataset LoRA adapters it reports 78.31 WAR on DFEW, 8.25 Clue and 8.16 Label overlap on EMER, 85.49 accuracy on EmoSet, and 85.47/79.67 on MER2024 SEMI/NOISE. This is achieved with vision and text only—no audio—beating audio-capable emotion models. The mechanism is a dynamic routing of compressed visual tokens between an emotion expert and a general expert, guided by an attention-based gate, so the model can keep scene-level reasoning while adding facial-emotion analysis. The authors credit the new VER dataset and the staged pretraining pipeline for making the emotion expert learn without damaging the general one.

Load-bearing premise

The load-bearing premise is that the evaluation uses clean train/test separation, so the VER training clips drawn from MAFW and MER2024 do not overlap the DFEW, EmoSet, MER2024, or EMER test sets, and that the ChatGPT-assigned Clue/Label overlap scores faithfully measure emotional reasoning quality.

Editorial extensions

If this is right

  • A 7B open-source model can match or beat far larger systems on both emotion and general vision-language benchmarks, so emotion-specialized assistants no longer have to sacrifice general competence.
  • Fine-grained emotion reasoning can be trained and evaluated as a vision-and-text task, and the VER dataset provides over 40,000 bilingual clips with human-verified explanations for doing so.
  • Per-dataset LoRA adapters on a frozen backbone make it practical to extend the model to new emotion domains without full retraining.
  • Task-dependent gating means one deployed model can serve mixed workloads, weighting the emotion expert for affective queries and the general expert for scene-level ones.
  • Reported gains on DFEW, EMER, EmoSet, and MER2024, if the evaluation splits are clean, indicate that catastrophic forgetting during emotion fine-tuning is avoidable rather than inevitable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The routing weights suggest a softer, input-dependent specialization than a hard switch; probing the experts' internal representations for facial-action versus scene-layout features would test whether the two experts truly encode complementary information.
  • Because the model uses vision and text only, adding an audio expert is a natural next experiment; if the bottleneck is modality coverage, EMER and MER2024 NOISE scores should rise with speech input.
  • Since Clue and Label overlap scores come from an LLM judge, a human-agreement study on a sample of EMER outputs would tell whether the metric reflects human judgments of emotional reasoning quality.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Emotion-Qwen, a 7B video-capable large multimodal model that combines a Facial Emotion Capture (FEC) module with a Mixture-of-Experts Hybrid Compressor to route visual tokens through emotion-specialized and general-purpose experts. The authors introduce a three-stage pretraining pipeline, a new bilingual Video Emotion Reasoning (VER) dataset of over 40K clips, and a multi-LoRA instruction fine-tuning strategy. They report state-of-the-art results on emotion benchmarks (DFEW, MER2024, EMER, EmoSet) while retaining competitive performance on general vision-language benchmarks such as MMBench and TextVQA.

Significance. If the reported results are reproducible and the evaluation protocol is clean, the paper makes a useful contribution: it demonstrates a concrete architecture for mitigating catastrophic forgetting when adapting large multimodal models to emotion-centric tasks, contributes a substantial new emotion-reasoning dataset, and provides component-level ablations for the FEC module, the Hybrid Compressor, and the multi-LoRA training strategy. The release of code and weights is a further strength, and the routing-pattern analysis in Table 7 is a valuable sanity check for the MoE design. However, the central SOTA claims are not independently checkable as written because the instruction fine-tuning and evaluation split protocol is not documented, and the EMER metric relies on ChatGPT scoring without reported human validation.

major comments (3)
  1. [Emotional Instruction Fine-tuning / Instruction Fine-tuning Evaluation, Tables 2 and 4] The paper never documents how the instruction fine-tuning data and the evaluation data are disjoint. The fine-tuning section states that VER (built from MER2024 and MAFW) is used together with DFEW and EmoViT, and the evaluation section then reports MER2024 SEMI/NOISE, DFEW WAR/UAR, EMER, and EmoSet accuracy. Since VER is constructed from MER2024, and since DFEW and EmoSet are used in fine-tuning, there is a direct route for training/evaluation overlap to inflate the headline numbers (85.47/79.67, 78.31, 85.49). The manuscript must specify the exact train/test splits, state which video IDs were excluded from VER, and describe which LoRA adapter is used for each evaluation benchmark. Without this, the SOTA claims in Table 4 are not verifiable.
  2. [Table 2 note and Instruction Fine-tuning Evaluation] The EMER Clue and Label Overlap scores are assessed by ChatGPT, but the paper reports no human agreement, no blinding protocol, and no correlation between ChatGPT scores and human judgments. The 8.25/8.16 EMER results are central to the claim of state-of-the-art emotional reasoning; if the metric rewards stylistic fluency rather than substantive emotional inference, the comparison is not meaningful. The authors should add a human-evaluation study or report the official EMER scoring procedure and its reliability, and ideally compare ChatGPT scores against human annotations on a subset.
  3. [General experimental reporting, Tables 2, 4, 5, and 8] No error bars, confidence intervals, or multiple-seed results are reported anywhere. Several headline margins are small (e.g., 78.31 vs. 77.51 WAR on DFEW in Table 4), and the multi-LoRA vs. single-LoRA comparison in Table 8 could depend on initialization and adapter selection. The authors should report at least three runs for the fine-tuning experiments and, where applicable, for the ablations, so that the claimed improvements can be distinguished from run-to-run variance.
minor comments (5)
  1. [Ablation Study, Table 5] The text states that FEC yields a 1.92% gain on DFEW, but Table 5 shows only +0.10 for WAR and +1.82 for UAR; the reported number appears to be the sum of the two metric gains. Please clarify whether the text refers to a combined score or to a single metric.
  2. [Emotional Instruction Fine-tuning] The phrase 'such as DFEW and EmoViT' calls EmoViT a dataset, but elsewhere EmoViT is presented as a model (Xie et al., 2024). The authors should clarify which concrete emotion datasets are used for instruction fine-tuning (e.g., EmoSet, AffectNet, or another resource) and cite the corresponding dataset papers.
  3. [Table 3] The table header 'H-Params' should be 'Hyperparameters', and 'DeepSpeed Zero2' should be written as 'ZeRO-2' for consistency with the cited DeepSpeed paper.
  4. [Table 7] The table has a typo ('Emoiton Expert') and the column headers are not fully legible in the submitted text; please format the table so that each benchmark has clear 'Emo' and 'Gen' subcolumns.
  5. [Reproducibility] Since code and weights are promised, please include the exact dataset versions, filtering thresholds, and the prompts used for VER construction and for ChatGPT-based EMER scoring in the supplementary material.

Circularity Check

1 steps flagged · score 2.0 of 10

No formal derivation-level circularity; the main risk is that VER is built from MER2024 and instruction fine-tuning also uses DFEW/EmoViT, while the SOTA tables report those same benchmarks without a documented split protocol.

  1. fitted input called prediction [Emotional Instruction Fine-tuning and Instruction Fine-tuning Evaluation / Tables 2 and 4]
    "The VER dataset was constructed leveraging publicly available data from MAFW (Liu et al. 2022) and MER2024 (Lian et al. 2024b), selecting 8,034 and 36,357 annotated samples, respectively. ... we utilize our constructed VER dataset along with additional emotion-focused datasets, such as DFEW (Jiang et al. 2020) and EmoViT (Xie et al. 2024), for instruction-based fine-tuning. ... Emotion-Qwen attains scores of 85.47 and 79.67 on the SEMI and NOISE tracks of MER2024, respectively"

    The fine-tuning corpus VER was built from 36,357 MER2024 samples, and the same evaluation section reports MER2024 SEMI/NOISE scores as SOTA. The paper does not state that VER's MER2024-derived clips were excluded from the MER2024 evaluation split, nor that DFEW/EmoSet LoRA training samples were held out from the reported test folds. If the splits overlap, the reported numbers are memory of training samples rather than predictions, so the claimed SOTA reduces by data construction. Without a stated split protocol the result is not independently checkable.

full rationale

Aside from the undocumented instruction-tuning/evaluation disjointness, the paper's framework is evaluated against external benchmarks and no formal derivation equates an input to an output. The Hybrid Compressor, FEC module, and three-stage pretraining are compared with ablations on the same external tasks; the central SOTA claims depend on clean splits, which are not specified. The AffectGPT, MER2024, and EMER citations are self-citations of the same group, but they are used for dataset construction and evaluation protocols, not as a uniqueness theorem or ansatz that forces the result, so they are not load-bearing. The ChatGPT-assessed EMER Clue/Label scores may reward LLM-like phrasing, but that is a measurement-validity concern rather than a circular derivation. Overall, this is a moderate evaluation-transparency risk, not a proof-level circularity; score 2.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

All learned network weights are treated as the model itself, not as free parameters in the physics sense. The entries above are hand-chosen or unreported configuration choices that the reported performance depends on. The axioms capture the unverified assumptions about data and evaluation that the SOTA claims rest on.

free parameters (4)
  • FEC emotion confidence threshold = not reported
    Controls selection of key emotional frames; not specified in the paper.
  • Number of compressed visual tokens N2 = not reported
    Compression ratio of Hybrid Compressor; not specified.
  • LoRA rank (r) and alpha = 64
    Set manually for all emotion fine-tuning; affects how much task-specific adaptation occurs.
  • Max video resolution and sampling rate = 448x448, 3 FPS
    Preprocessing choices that determine the information available to the model.
assumptions (4)
  • domain assumption DeepFace emotion predictions are accurate enough to select salient emotional frames.
    FEC module relies on DeepFace landmark and emotion classifier without error analysis.
  • domain assumption ChatGPT-assigned CLUE and LABEL overlap scores are a valid measure of emotion reasoning quality.
    EMER evaluation uses ChatGPT scoring; no human agreement is reported.
  • domain assumption Standard train/test partitions are used and VER (built from MAFW and MER2024) does not leak into evaluation sets.
    Split protocol is never stated, so this unverified premise is load-bearing.
  • ad hoc to paper The three-stage pretraining plus multi-LoRA recipe transfers emotion reasoning without catastrophic forgetting.
    This is the core proposed training strategy; it is validated only on the reported benchmarks, not derived from first principles.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Emotion-Qwen: A Unified Framework for Emotion and Vision Understanding." pith.science (2026). https://pith.science/paper/R22TTCKV

@misc{pith2026250506685,
  author       = {Pith},
  title        = {Pith review of: Emotion-Qwen: A Unified Framework for Emotion and Vision Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R22TTCKV}},
  note         = {Machine review of arXiv:2505.06685}
}
read the original abstract

Accurate emotion understanding in videos necessitates effectively recognizing and interpreting emotional states by integrating visual, textual, auditory, and contextual cues. Although recent Large Multimodal Models (LMMs) have exhibited significant progress in general vision-language (VL) tasks, their performance often deteriorates in emotion-specific scenarios, exhibiting catastrophic forgetting when fine-tuned on emotion-centric tasks. To overcome these limitations, we propose Emotion-Qwen, a unified multimodal framework designed to simultaneously enable robust emotion understanding and preserve general VL reasoning capabilities. Emotion-Qwen introduces a novel Hybrid Compressor based on a Mixture-of-Experts (MoE) architecture, dynamically routing inputs to optimally balance emotion-specific processing and general multimodal reasoning. We further propose a carefully structured three-stage pre-training pipeline, leveraging extensive general and emotion-focused datasets to strengthen multimodal representation robustness and model adaptability. Additionally, we develop the Video Emotion Reasoning (VER) dataset, a large-scale bilingual resource containing over 40K video clips annotated with detailed context-aware emotional descriptions, significantly facilitating research on fine-grained emotional reasoning. Extensive experiments confirm that Emotion-Qwen achieves state-of-the-art performance across multiple emotion recognition and reasoning benchmarks, while maintaining highly competitive results in general VL tasks.

Figures

Figures reproduced from arXiv: 2505.06685 by the authors.

Figure 1
Figure 1. Motivation behind Emotion-Qwen (zoom in for detailed Q&A): (a) Current state-of-the-art emotional Large Multimodal Models (LMMs) experience severe catastrophic forgetting, causing inaccurate emotion recognition, ineffective dialogue comprehension, and limited multimodal reasoning abilities. (b) In contrast, Emotion-Qwen effectively addresses these limitations, balancing fine-grained emotion reasoning with robust gen… view at source ↗
Figure 2
Figure 2. Overview of Emotion-Qwen’s overall training pipeline and model architecture. The Facial Emotion Capture (FEC) Module identi￾fies key emotional cues from input videos, while the Hybrid Compressor employs an attention-based Gating Network to dynamically integrate emotion-specific and general multimodal features. The detailed structure of the Gating Network is illustrated on the right. stage training to significantly i… view at source ↗
Figure 3
Figure 3. Construction pipeline of the Video Emotion Reasoning (VER) dataset. Outputs from multiple models are synthesized and human-verified to reduce bias and hallucination. mitigate model-induced biases, outputs from these models were integrated and synthesized using DeepSeek (Liu et al. 2024a), producing fine-grained emotional annotations. Sub￾sequently, 12 professional analysts rigorously evaluated the annotations on a 0… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Balanced performance of Large Multimodal Models (LMMs). Our Emotion-Qwen demonstrates a superior balance be￾tween fine-grained emotion reasoning and general vision-language understanding. Evaluation details are provided in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Illustration of different projector architectures evaluated in Emotion-Qwen: MLP Projector, Fusion Projector, and the pro￾posed Hybrid Compressor with a Gating Network. General Benchmarks Emotion Tasks MME POPE MER2024 EmoSet Emo Gen Emo Gen Emo Gen Emo Gen 0.38 0.62 0…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. E$^3$mo-Bench: A Scalable Benchmark for Multimodal Evoked and Expressed Emotion Understanding via Bayesian Pairwise Alignment

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A new video benchmark jointly evaluates expressed and evoked emotion understanding in multimodal LLMs using perception, open-vocabulary recognition, and VAD rating tasks, with Bayesian pairwise alignment for scalable ...

  2. OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction

    cs.HC 2026-08 conditional novelty 6.0 of 10

    A 4.5B-parameter multimodal model trained on a new 130K reasoning-trajectory dataset with a multi-task reinforcement learning recipe reports near-commercial performance across eight affective tasks.

  3. MVEI & EmObserver: Empowering MLLM-Oriented Visual Emotional Intelligence via Emotion Statement Judgement

    cs.CV 2026-07 conditional novelty 6.0 of 10

    ESJ statement verification, the INSETS-462k corpus, the MVEI benchmark, and the EmObserver model together provide a scalable way to evaluate and improve MLLM visual emotional intelligence.

  4. Why Do Vision Language Models Struggle To Recognize Human Emotions?

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    VLMs fail at video emotion recognition because they collapse rare emotions into common ones and process frames as an unordered bag; summarizing skipped frames in text partially recovers temporal cues.

Reference graph

Works this paper leans on

43 extracted references · 14 canonical work pages · cited by 4 Pith papers

  1. [1]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    L.; and Parikh, D

    Antol, S.; Agrawal, A.; Lu, J.; Mitchell, M.; Batra, D.; Zitnick, C. L.; and Parikh, D. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, 2425--2433

    2015

  3. [3]

    Baevski, A.; Zhou, H.; Mohamed, A.; and Auli, M. 2020. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. arXiv:2006.11477

    arXiv 2020

  4. [4]

    Bai, J.; Bai, S.; Chu, Y.; Cui, Z.; Dang, K.; Deng, X.; Fan, Y.; Ge, W.; Han, Y.; Huang, F.; et al. 2023 a . Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  5. [5]

    Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023 b . Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966

    arXiv 2023

  6. [6]

    M.; and Zisserman, A

    Cao, Q.; Shen, L.; Xie, W.; Parkhi, O. M.; and Zisserman, A. 2018. VGGFace2: A dataset for recognising faces across pose and age. arXiv:1710.08092

    arXiv 2018

  7. [7]

    Cheng, Z.; Cheng, Z.-Q.; He, J.-Y.; Wang, K.; Lin, Y.; Lian, Z.; Peng, X.; and Hauptmann, A. 2024. Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning. In Globerson, A.; Mackey, L.; Belgrave, D.; Fan, A.; Paquet, U.; Tomczak, J.; and Zhang, C., eds., Advances in Neural Information Processing Systems, volume 37, 110805--1108...

    work page 2024

  8. [8]

    Dao, T. 2023. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. arXiv:2307.08691

    arXiv 2023

Show all 43 references
  1. [9]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    2020 arXiv

  2. [10]

    et al., A. G. 2024. The Llama 3 Herd of Models. arXiv:2407.21783

    2024 arXiv

  3. [11]

    Fu, C.; Chen, P.; Shen, Y.; Qin, Y.; Zhang, M.; Lin, X.; Yang, J.; Zheng, X.; Li, K.; Sun, X.; et al. 2023. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394

    2023 arXiv

  4. [12]

    Hendrycks, D.; and Gimpel, K. 2023. Gaussian Error Linear Units (GELUs). arXiv:1606.08415

    2023 arXiv

  5. [13]

    Huang, D.; Yan, C.; Li, Q.; and Peng, X. 2024. From Large Language Models to Large Multimodal Models: A Literature Review. Applied Sciences, 14(12)

    2024

  6. [14]

    Jiang, X.; Zong, Y.; Zheng, W.; Tang, C.; Xia, W.; Lu, C.; and Liu, J. 2020. DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild. In Proceedings of the 28th ACM International Conference on Multimedia, 2881--2889

    2020

  7. [15]

    Lei, S.; Dong, G.; Wang, X.; Wang, K.; Qiao, R.; and Wang, S. 2024. InstructERC: Reforming Emotion Recognition in Conversation with Multi-task Retrieval-Augmented Large Language Models. arXiv:2309.11911

    2024 arXiv

  8. [16]

    Li, X.; Ren, W.; Qin, W.; Wang, L.; Zhao, T.; and Hong, R. 2025. Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1--5. IEEE

    2025

  9. [17]

    Li, Y.; Du, Y.; Zhou, K.; Wang, J.; Zhao, Wayne Xin ; and Wen, Ji-Rong . 2023. Evaluating Object Hallucination in Large Vision-Language Models. In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  10. [18]

    Lian, Z.; Chen, H.; Chen, L.; Sun, H.; Sun, L.; Ren, Y.; Cheng, Z.; Liu, B.; Liu, R.; Peng, X.; Yi, J.; and Tao, J. 2025. AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models. arXiv:2501.16566

    2025 arXiv

  11. [19]

    W.; and Tao, J

    Lian, Z.; Sun, H.; Sun, L.; Chen, K.; Xu, M.; Wang, K.; Xu, K.; He, Y.; Li, Y.; Zhao, J.; Liu, Y.; Liu, B.; Yi, J.; Wang, M.; Cambria, E.; Zhao, G.; Schuller, B. W.; and Tao, J. 2023 a . MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning. In Proc...

    2023

  12. [20]

    Lian, Z.; Sun, H.; Sun, L.; Gu, H.; Wen, Z.; Zhang, S.; Chen, S.; Xu, M.; Xu, K.; Chen, K.; Chen, L.; Liang, S.; Li, Y.; Yi, J.; Liu, B.; and Tao, J. 2024 a . Explainable Multimodal Emotion Recognition. arXiv:2306.15401

    2024 arXiv

  13. [21]

    W.; and Tao, J

    Lian, Z.; Sun, H.; Sun, L.; Wen, Z.; Zhang, S.; Chen, S.; Gu, H.; Zhao, J.; Ma, Z.; Chen, X.; Yi, J.; Liu, R.; Xu, K.; Liu, B.; Cambria, E.; Zhao, G.; Schuller, B. W.; and Tao, J. 2024 b . MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emo...

    2024 arXiv

  14. [22]

    Lian, Z.; Sun, L.; Sun, H.; Chen, K.; Wen, Z.; Gu, H.; Chen, S.; Liu, B.; and Tao, J. 2023 b . Gpt-4v with emotion: A zero-shot benchmark for multimodal emotion understanding. CoRR

    2023

  15. [23]

    Liu, A.; Feng, B.; Xue, B.; Wang, B.; Wu, B.; Lu, C.; Zhao, C.; Deng, C.; Zhang, C.; Ruan, C.; et al. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  16. [24]

    Liu, C.; Kang, Y.; Wang, S.; Qing, L.; Zhao, F.; Wu, C.; Sun, C.; Kuang, K.; and Wu, F. 2024 b . More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLM s. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Proceedings of the 2024 Confere...

    2024

  17. [25]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual instruction tuning. Advances in neural information processing systems, 36: 34892--34916

    2023

  18. [26]

    Liu, Y.; Dai, W.; Feng, C.; Wang, W.; Yin, G.; Zeng, J.; and Shan, S. 2022. MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild. In Proceedings of the 30th ACM International Conference on Multimedia, MM '22, 24–32...

    2022

  19. [27]

    Liu, Y.; Duan, H.; Zhang, Y.; Li, B.; Zhang, S.; Zhao, W.; Yuan, Y.; Wang, J.; He, C.; Liu, Z.; et al. 2024 c . MMBench: Is your multi-modal model an all-around player? In European conference on computer vision, 216--233. Springer

    2024

  20. [28]

    Lu, P.; Mishra, S.; Xia, T.; Qiu, L.; Chang, K.-W.; Zhu, S.-C.; Tafjord, O.; Clark, P.; and Kalyan, A. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 2507--2521

    2022

  21. [29]

    K.; and Chakraborty, A

    Mishra, A.; Shekhar, S.; Singh, A. K.; and Chakraborty, A. 2019. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), 947--952. IEEE

    2019

  22. [30]

    Poria, S.; Hazarika, D.; Majumder, N.; Naik, G.; Cambria, E.; and Mihalcea, R. 2019. MELD : A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. In Korhonen, A.; Traum, D.; and M \`a rquez, L., eds., Proceedings of the 57th Annual Meeting of the Associati...

    2019

  23. [31]

    Rajbhandari, S.; Rasley, J.; Ruwase, O.; and He, Y. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, 1--16. IEEE

    2020

  24. [32]

    Singh, A.; Natarajan, V.; Shah, M.; Jiang, Y.; Chen, X.; Batra, D.; Parikh, D.; and Rohrbach, M. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8317--8326

    2019

  25. [33]

    Taigman, Y.; Yang, M.; Ranzato, M.; and Wolf, L. 2014. DeepFace: Closing the Gap to Human-Level Performance in Face Verification. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, 1701--1708

    2014

  26. [34]

    K.; Qian, X.; Shou, M

    Tao, R.; Pan, Z.; Das, R. K.; Qian, X.; Shou, M. Z.; and Li, H. 2021. Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection. In Proceedings of the 29th ACM International Conference on Multimedia, MM '21, 3927–3935. New York, NY, U...

    2021

  27. [35]

    Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; Rodriguez, A.; Joulin, A.; Grave, E.; and Lample, G. 2023 a . LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971

    2023 arXiv

  28. [36]

    Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  29. [37]

    Wang, P.; Bai, S.; Tan, S.; Wang, S.; Fan, Z.; Bai, J.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Fan, Y.; Dang, K.; Du, M.; Ren, X.; Men, R.; Liu, D.; Zhou, C.; Zhou, J.; and Lin, J. 2024. Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution. arXiv...

    2024 arXiv

  30. [38]

    Xie, H.; Peng, C.-J.; Tseng, Y.-W.; Chen, H.-J.; Hsu, C.-F.; Shuai, H.-H.; and Cheng, W.-H. 2024. EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuning. arXiv:2404.16670

    2024 arXiv

  31. [39]

    Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Li, C.; Liu, D.; Huang, F.; Wei, H.; et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  32. [40]

    Yao, Y.; Yu, T.; Zhang, A.; Wang, C.; Cui, J.; Zhu, H.; Cai, T.; Li, H.; Zhao, W.; He, Z.; et al. 2024. MiniCPM-V: A GPT-4V Level MLLM on Your Phone. arXiv preprint arXiv:2408.01800

    2024 arXiv

  33. [41]

    Ye, J.; He, J.; Peng, X.; Wu, W.; and Qiao, Y. 2020. Attention-Driven Dynamic Graph Convolutional Network for Multi-label Image Recognition. In Vedaldi, A.; Bischof, H.; Brox, T.; and Frahm, J.-M., eds., Computer Vision -- ECCV 2020, 649--665. Cham: Springer International Publ...

    2020

  34. [42]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  35. [43]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.