{"id":"adc29476-8bb0-47a9-be9e-bcda6d4ec0c8","arxiv_id":"2501.09502","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Omni-Emotion combines face, audio, and video features in a large language model to achieve state-of-the-art scores on emotion recognition and emotion reasoning benchmarks.","lead":"Researchers built a video emotion model by adding a dedicated face encoder and an audio encoder to a standard video language model, then trained it on 24,137 machine-reviewed and 3,500 human-checked video clips with detailed emotion descriptions. The model reports top scores on emotion recognition benchmarks and produces written explanations of the emotional cues it uses.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EMER reasoning SOTA rests on an unvalidated ChatGPT overlap score that is confounded by the GPT-style annotations used to train the model; without human rescoring, the 8.22 clue-overlap claim does not establish better emotion reasoning.","rationale":"The reader's weakest assumption matches the most load-bearing point: the EMER reasoning evaluation is only as valid as the ChatGPT-based overlap score. Because the model was trained on GPT-3.5-filtered, Qwen2-VL-generated rationales, the evaluator and the training-target distribution are from the same model family, making stylistic mimicry a serious confound. My proposed human-rescoring check would settle this directly. I also considered the appended correction about EMER/MER24 provenance (Section 9) as a possible train/eval contamination risk, but the paper states that SRE/HRE are built from training splits, and the exact relationship between EMER and MER24 is too ambiguous to make leakage the primary concern. The reader's CONDITIONAL verdict is appropriate: the recognition results provide partial support, but the headline reasoning claim should remain conditional until the judge metric is validated. I therefore keep the verdict unchanged rather than moving to reject, while noting that a failed human-rescoring check would push the verdict toward rejection of the reasoning claim.","tokens_in":19450,"tokens_out":11153,"duration_ms":110796,"concrete_test":"Select a random subset of, say, 60 EMER test videos. Obtain paired outputs from Omni-Emotion and Emotion-LLaMA (or from the paper's re-implementation), and have at least three human annotators, blind to model identity, score emotion-clue overlap using a fixed rubric that separates content from style. Also have ChatGPT score the same outputs. Report mean scores, the paired Omni-Emotion minus Emotion-LLaMA difference, human-ChatGPT agreement (e.g., ICC), and inter-annotator agreement. If the Omni-Emotion advantage is not significant under human scoring, or if human-ChatGPT agreement is weak, the 8.22 clue-overlap result should be treated as a style artifact rather than evidence of superior emotion reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of state-of-the-art emotion reasoning depends entirely on the ChatGPT-based clue-overlap metric in Section 5.2 (Table 2). The training annotations in SRE/HRE were produced by Qwen2-VL and then filtered and corrected by GPT-3.5 (Section 3.1), so Omni-Emotion is explicitly instruction-tuned to emit long, GPT-style, clue-enumerating rationales. ChatGPT is then used to score how well those rationales overlap with ground-truth EMER descriptions. This creates a potential circularity: the judge may reward stylistic similarity, template phrasing, response length, and confident wording rather than genuine emotional insight. The paper reports no judge prompt, no judge variance, no inter-rater reliability, and no correlation of ChatGPT scores with human judgments. The reported margin over Emotion-LLaMA (8.22 vs 7.83) is small enough to be explained by such a style confound. If the metric is biased, the 'SOTA in both emotion recognition and reasoning' claim reduces to a recognition-only claim; the DFEW/MAFW recognition results are less affected by this particular issue. The Table 3 selection of the fusion module on the same EMER benchmarks compounds the risk of optimistic evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces two instruction-tuning datasets for multimodal emotion analysis, SRE (24,137 samples filtered by GPT-3.5 alignment scores) and HRE (3,500 manually verified samples), and proposes Omni-Emotion, a video MLLM built on LLaVA-OneVision-7B with an added FaceXFormer facial encoder and a Whisper-large-v3 audio encoder. The model is trained in three phases: audio alignment, facial feature alignment, and multimodal instruction tuning. The authors report state-of-the-art results on DFEW (UAR 68.80), MAFW (UAR 53.81), EMER-OV (average score 65.9), and EMER reasoning (clue overlap 8.22), and include ablations on modality contributions and visual feature fusion strategies.","tokens_in":19728,"tokens_out":5406,"duration_ms":55293,"significance":"If the reported recognition numbers are reproducible, they would be a useful advance: the DFEW and MAFW results beat several strong prior multimodal emotion models, and the dataset construction pipeline is scalable. The ablation study in Table 1 and the fusion comparison in Table 3 provide informative engineering evidence about the value of explicit facial and audio encoders in a video MLLM. However, the central 'SOTA in emotion reasoning' claim is not yet established: the reasoning metric in Section 5.2 is an unvalidated ChatGPT overlap score that is plausibly confounded with the GPT-style training annotations, and the open-vocabulary result in Section 5.1 compares only one baseline without error bars. No code or model weights are linked beyond a project page, so reproducibility is not currently verifiable. These issues are addressable, but they are load-bearing for the paper's headline claims.","major_comments":[{"comment":"The emotion-reasoning evaluation uses ChatGPT to score overlap between model predictions and EMER ground-truth descriptions, but the paper reports no judge prompt, no score distribution, no inter-rater reliability, and no correlation of ChatGPT scores with human judgments. This is a genuine concern because the SRE/HRE training annotations were generated by Qwen2-VL and filtered/corrected by GPT-3.5 (Section 3.1), so Omni-Emotion is explicitly instructed to produce long, clue-enumerating, GPT-style rationales. The reported margin over Emotion-LLaMA (8.22 vs 7.83) is small enough to be explained by stylistic mimicry rather than better emotion understanding. Please add a blind human rescoring of a random sample, report the exact scoring prompt, and include a non-LLM or differently-instructed judge as a control.","section":"Section 5.2, Table 2"},{"comment":"The EMER-OV claim rests on a single baseline (AffectGPT), and the EMER reasoning comparison is selected by ablating the fusion module on the same EMER test set whose final numbers are then reported as the main result. No error bars or significance tests are provided for any EMER result, so the statement that the model 'significantly outperforms' AffectGPT is not supported. Please add more baselines, including the generic video MLLMs already listed in Table 2 and at least one recent emotion MLLM, use a validation/holdout split for model selection, and report mean and standard deviation over multiple runs or a paired significance test.","section":"Section 5.1, Table 1 and Section 5.2, Table 3"},{"comment":"The DFEW and MAFW recognition results are the most convincing evidence in the paper, but the table lacks confidence intervals or significance tests, and Emotion-LLaMA is not evaluated on MAFW, so the comparison is incomplete. The text also states that both datasets 'containing audio information' without specifying whether DFEW actually provides audio tracks or how the audio modality is used for DFEW. Please clarify the evaluation protocol and report variance or statistical significance, especially for the DFEW margin (68.80 vs 66.01), which is much smaller than the MAFW margin.","section":"Section 5.4, Table 5"},{"comment":"The dataset construction relies on a GPT-3.5 alignment score threshold of 5 to filter training samples, but no evidence is provided that this score correlates with human judgments, and the threshold is treated as a free parameter. The HRE dataset is described as human-reviewed, but the paper does not report the annotation procedure, the number of annotators, or inter-annotator agreement. Since the SRE/HRE datasets are central contributions, please provide a human validation of the filtering criterion and basic annotation statistics for the human-reviewed subset.","section":"Section 3.1 and Section 3.2"}],"minor_comments":[{"comment":"The self-correction that EMER data should be attributed to MER24 rather than EMER must be propagated to the main text, and related citation errors should be checked; for example, the introduction references 'EmoLLM [32]' where [32] is AffectGPT, and the text is garbled.","section":"Section 9"},{"comment":"There are frequent typographical and formatting errors, including 'RA VDESS', 'LLaV A-OneVision', 'Explainable reason:The', and 'emotion reason analysis'; these should be corrected before publication.","section":"Throughout"},{"comment":"The three-phase training recipe is clear, but the exact amount of audio alignment data (1.5M instructions) and facial alignment data (130K) is only reported in the appendix; consider moving these numbers, as well as the modality-specific tokens and frame sampling details, into the main experimental section.","section":"Section 4.4"},{"comment":"The precision/recall definitions on grouped open-vocabulary labels should be specified more carefully; in particular, state whether Y and Y-hat are sets or multisets and how ties and empty predictions are handled.","section":"Section 5.1, Equation (2)"},{"comment":"The claim of 'state-of-the-art performance in both emotion recognition and reasoning tasks' is stronger than what the evidence supports; the abstract and conclusion should be tempered until the reasoning metric is validated against human judgments.","section":"Abstract and Conclusion"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a respectable systems paper that makes a real architectural contribution — adding a dedicated face encoder to a video MLLM — and the DFEW/MAFW recognition numbers look plausible. But the emotion-reasoning SOTA claim is weaker than advertised: the ChatGPT overlap score in Section 5.2 is likely confounded by the fact that the training annotations were themselves generated and filtered by GPT-family models.\n\nWhat's new: the explicit FaceXFormer encoder for fine-grained facial cues, combined with Whisper audio features, is a genuine departure from prior work like Emotion-LLaMA and AffectGPT, which use generic vision encoders. The SRE/HRE datasets (24k self-reviewed, 3.5k human-reviewed) are large, and the pipeline of cross-modal consistency checking with GPT-3.5 and manual review is a reasonable way to scale annotations. The three-stage training recipe (audio alignment, face alignment, then joint instruction tuning) is clean, and the ablations in Table 1 show each modality adds value.\n\nSoft spots: the main one is the EMER reasoning evaluation. The judge is ChatGPT, the training data was made by Qwen2-VL + GPT-3.5, so the model is rewarded for imitating GPT-style verbose rationales. The margin over Emotion-LLaMA (8.22 vs 7.83) is small enough that a style confound could explain it. The paper provides no judge prompt, no variance, no correlation with human ratings. Table 3 also selects the fusion module on the same EMER benchmark, which inflates the reported score. The recognition results are less affected, but even those lack error bars and independent reproduction. No code or data are released, which limits verification.\n\nWho this is for: affective computing researchers and people building emotion-aware interactive systems. The architecture recipe and dataset construction approach are valuable despite the reasoning-eval caveat.\n\nRecommendation: send it to peer review, but require the authors to validate the ChatGPT overlap scores against human judgments (or at least release the judge prompt and a sample of rescored outputs), and release code/data. If the reasoning claim is softened to 'recognition + open-vocab', the core is publishable.","headline":"Solid face+audio MLLM for emotion recognition, but the reasoning SOTA rests on a ChatGPT overlap score confounded by GPT-generated training data.","tokens_in":20251,"tokens_out":2175,"would_cite":true,"duration_ms":19327,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Face and audio encoders push video emotion AI to state-of-the-art scores","keywords":["multimodal emotion analysis","video MLLM","facial expression modeling","audio emotion recognition","instruction tuning","emotion reasoning","open-vocabulary emotion","dataset curation"],"falsifier":"A concrete check would be to have human raters, blind to model identity, rank Omni-Emotion's EMER reasoning outputs against Emotion-LLaMA's; if human rankings do not correlate with ChatGPT's overlap scores, or if a simpler text-only model trained on SRE/HRE annotations matches the 8.22 clue overlap without any video input, the reasoning claim would be undercut.","tokens_in":19236,"feed_emoji":"🎭","tokens_out":5516,"duration_ms":49869,"temperature":0.7,"pith_summary":"The paper claims that video emotion understanding is held back by generic vision encoders that miss subtle facial micro-expressions and by the absence of audio modeling, and that both gaps can be closed by bolting a dedicated facial encoder and an audio encoder onto a video multimodal large language model. The authors attach FaceXFormer's face encoder and Whisper-large-v3's audio encoder to LLaVA-OneVision-7B, align their features to the language model's embedding space, and instruction-tune the whole system on two new datasets: 24,137 self-reviewed samples and 3,500 human-reviewed samples with detailed multimodal emotion annotations. They report state-of-the-art unweighted average recall on DFEW and MAFW, and the highest clue-overlap score on the EMER emotion-reasoning benchmark. A sympathetic reader would care because the recipe suggests a scalable path from coarse categorical emotion labels to reasoning about why a person feels a given emotion.","feed_headline":"Face and audio encoders push video emotion AI to state-of-the-art scores","feed_subtitle":"A 7B model reads micro-expressions and tone of voice, beating prior emotion models on three benchmarks.","key_machinery":"The mechanism is a three-encoder MLLM in which a general vision encoder, FaceXFormer's facial encoder, and Whisper-large-v3's audio encoder each feed through a two-layer linear projector into the LLaVA-OneVision embedding space. A visual feature fusion module concatenates whole-clip facial features with whole-clip general visual features, and modality-specific special tokens tell the decoder which embedding came from which channel. Training proceeds in three phases: audio-projector alignment on audio-text data, facial-projector alignment on emotion classification data, and joint instruction tuning of all projectors plus the LLM on the SRE and HRE datasets, whose annotations were produced by a pipeline using Qwen2-VL and GPT-3.5.","core_discovery":"The central claim is that explicitly integrating a facial encoder (FaceXFormer) and an audio encoder (Whisper-large-v3) into LLaVA-OneVision, together with instruction tuning on the proposed SRE and HRE datasets, yields state-of-the-art multimodal emotion recognition and reasoning. Concretely, the model reports a UAR of 68.80 on DFEW and 53.81 on MAFW, an average open-vocabulary score of 65.9 on EMER-OV, and a clue overlap of 8.22 on EMER, each above the prior state of the art. The paper further argues that the gain comes from aligning fine-grained facial cues and audio tone in a unified embedding space, not from a larger model.","pith_inferences":["A testable extension is to ablate the SRE/HRE training data itself: fine-tune Omni-Emotion on the raw categorical labels without the GPT-generated detailed annotations, and measure how much of the EMER gain actually comes from the annotation pipeline rather than the added encoders.","The paper's evaluation rewards outputs that resemble GPT-style verbose reasoning, so a natural next experiment is to replace ChatGPT scoring with human preference judgments or with a metric based on matched emotion vocabulary.","Given the paper's limitation that training videos are mostly third-person, the model would likely need additional first-person camera data for front-facing deployments."],"forward_implications":["Emotion-aware MLLMs can be built from existing video MLLMs by adding specialized face and audio encoders, without redesigning the base model.","The SRE and HRE datasets demonstrate a scalable pipeline that converts coarse categorical emotion labels into detailed multimodal reasoning annotations.","Open-vocabulary emotion recognition becomes practical, allowing models to output emotion descriptors beyond a fixed category set.","The EMER clue-overlap metric can be used to quantify whether a model genuinely combines facial, vocal, and contextual evidence.","The three-phase training recipe (per-modality alignment, then joint instruction tuning) should transfer to other fine-grained perception tasks requiring facial and audio cues."],"supporting_citations":[{"why":"Base video MLLM whose general vision encoder and LLM are extended with face and audio encoders.","marker":"[25]"},{"why":"Supplies the facial encoder that captures fine-grained face representations.","marker":"[39]"},{"why":"Supplies the audio encoder used to extract auditory emotion cues from raw waveform.","marker":"[46]"},{"why":"EMER dataset provides the open-vocabulary and reasoning benchmarks on which the model is evaluated.","marker":"[30]"},{"why":"Emotion-LLaMA supplies the comparison baseline for EMER reasoning and the MERR dataset precursor.","marker":"[10]"},{"why":"AffectGPT supplies the EMER-OV evaluation protocol and a baseline for open-vocabulary recognition.","marker":"[32]"},{"why":"DFEW is one of the two in-the-wild emotion recognition benchmarks used for UAR/WAR evaluation.","marker":"[19]"},{"why":"MAFW is the other in-the-wild multimodal emotion benchmark, also used as a source of training samples.","marker":"[34]"},{"why":"Qwen2-VL generates the global and facial descriptions used to build the SRE and HRE annotations.","marker":"[53]"},{"why":"GPT-3.5 performs consistency checking, self-review scoring, and dataset filtering.","marker":"[41]"}],"fun_headline_variants":["Face and audio fusion pushes emotion AI to state-of-the-art","Omni-Emotion: micro-expressions plus tone of voice beat prior models","Video MLLM integrates face and audio for sharper emotion reading","Fusing face and audio into video models sets new emotion AI record"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that ChatGPT's clue-overlap and label-overlap scores on the EMER benchmark actually measure emotion-reasoning quality, rather than stylistic similarity to GPT-generated descriptions; if that premise fails, the model's top reasoning scores do not establish superior emotion understanding.","fun_headline_variants_meta":{"raw":{"variants":["Face and audio fusion pushes emotion AI to state-of-the-art","Omni-Emotion: micro-expressions plus tone of voice beat prior models","Video MLLM integrates face and audio for sharper emotion reading","Fusing face and audio into video models sets new emotion AI record"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000315,"raw_usage":{"total_tokens":1774,"prompt_tokens":920,"completion_tokens":854,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":778}},"tokens_in":536,"tokens_out":854,"duration_ms":7872,"temperature":1.0,"reasoning_tokens":778,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:57:28.099267+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check would be to have human raters, blind to model identity, rank Omni-Emotion's EMER reasoning outputs against Emotion-LLaMA's; if human rankings do not correlate with ChatGPT's overlap scores, or if a simpler text-only model trained on SRE/HRE annotations matches the 8.22 clue overlap without any video input, the reasoning claim would be undercut.","supporting_citations":[{"cited_title":"Robust speech recognition via large-scale weak supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the audio encoder used to extract auditory emotion cues from raw waveform."},{"cited_title":"Dfew: A large-scale database for recognizing dynamic facial expres- sions in the wild","cited_arxiv_id":null,"evidence_quote":"DFEW is one of the two in-the-wild emotion recognition benchmarks used for UAR/WAR evaluation."},{"cited_title":"Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild","cited_arxiv_id":null,"evidence_quote":"MAFW is the other in-the-wild multimodal emotion benchmark, also used as a source of training samples."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-3.5 performs consistency checking, self-review scoring, and dataset filtering."}],"review_version":1}