{"id":"2ae16d9b-094a-40d1-a453-527be80dda75","arxiv_id":"2508.02429","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"Benchmarks seven open-source audio-video-text MLLMs on six affective datasets and shows a generative-knowledge prompting step improves fine-tuned emotion recognition.","lead":"This paper benchmarks seven open-source multimodal large language models on six emotion recognition datasets and proposes a hybrid strategy that combines generated descriptive prompts with supervised fine-tuning to improve performance. The interest for a generalist is a concrete recipe for boosting affective computing with off-the-shelf MLLMs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is not checkable from this version: no result tables appear, and the proposed prompting benefit is not isolated from a length-matched text control.","rationale":"Reading in good faith, the paper is a straightforward benchmark-plus-prompting study; the idea is plausible and the code release is a concrete artifact. The central claim is not obviously false, and I see no internal contradiction in the method. But the single load-bearing requirement for the claim—that the integrated strategy significantly helps—is that the experiments are reported and that the comparison isolates the knowledge content of the generated descriptions. Both are unmet in this version. The reader's weakest assumption about description accuracy is related but not the whole story: even accurate descriptions could produce gains through extra text, and inaccurate ones could still help through label-like leakage. My concrete test would settle the attribution question. If the authors supply the missing tables and the length-matched generic-text control shows no advantage for informative descriptions, the claim should be weakened. I leave the reader's conditional verdict unchanged, with the additional condition noted.","tokens_in":12848,"tokens_out":9015,"duration_ms":114957,"concrete_test":"On CMU-MOSI with Qwen2.5Omni, run three SFT conditions with identical LoRA rank, epochs, and prompt template: (a) no generated description; (b) the proposed generated audio/video descriptions; (c) length-matched non-emotional filler text. If condition (b) does not beat condition (c) beyond run-to-run variance on Acc7/MAE, the central claim that affective knowledge in the descriptions drives the improvement is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and conclusion claim that generative knowledge prompting plus SFT 'significantly improves performance across various MAC tasks.' In the submitted manuscript this claim is not checkable: no numerical result tables appear anywhere; Section IV refers to the Appendix for details, but the Appendix only defines metrics and describes models, and the hyperparameter sentence in Section IV.B is cut off. Without effect sizes, error bars, or per-dataset numbers, 'significantly' cannot be verified. The causal attribution is also confounded. The proposed SFT+knowledge condition differs from the SFT baseline both in the presence of generated descriptions and in the amount of prompt text. Because the MLLM itself generates the descriptions and they can contain emotion words (e.g., 'fear', 'pain', 'sense of urgency' in the Figure 2 case study), the gain could come from label-like leakage or from extra context rather than from accurate affective knowledge. No control condition with length-matched generic text is reported, and generated-description quality is not evaluated anywhere in Sections III.C.2 or V. Thus the central claim lacks both the evidence and the experimental isolation needed to support it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript benchmarks seven open-source multimodal large language models (MLLMs) that jointly process text, audio, and video on six established affective computing datasets (CMU-MOSI, CMU-MOSEI, CH-SIMS, CH-SIMS v2, MELD, UR-FUNNY v2). It compares these models with traditional multimodal learning baselines and analyzes how model architecture and dataset properties affect performance. To improve MLLM performance, the paper proposes a hybrid strategy that first uses zero-shot MLLM generation to produce textual descriptions of audio and video emotional cues, then incorporates these descriptions into the prompt and applies supervised fine-tuning with LoRA. The abstract and conclusion claim that this integrated approach significantly improves performance across tasks. The manuscript also releases code at a public repository.","tokens_in":13068,"tokens_out":2031,"duration_ms":26496,"significance":"If the claimed results are substantiated, the benchmark would be a useful contribution: it is among the first systematic evaluations of trimodal open-source MLLMs on standard affective computing benchmarks, and the proposed generative-knowledge-prompting-plus-SFT strategy is simple and potentially transferable. The analysis of audio-encoder pretraining alignment with downstream tasks also offers actionable guidance. However, the current manuscript does not contain the quantitative results needed to assess these claims, so the significance cannot yet be evaluated. The paper's stated strengths—open code, multiple models, multiple datasets—are conditional on the actual evidence being present and correct.","major_comments":[{"comment":"The central claim of significant performance improvement is not supported by any numerical results in the manuscript. No accuracy, F1, MAE, correlation, or error-bar values appear for any model or dataset. Section IV says that detailed evaluation information is in the Appendix, but the Appendix only defines metrics and describes model architectures; it contains no result tables. The abstract's 'significantly improves performance' and the conclusion's 'significantly enhanced' therefore cannot be checked. This is load-bearing: the paper is an empirical benchmark and boosting study, and without the actual numbers the entire contribution is unverifiable. Please add complete results tables with per-model, per-dataset scores for all baselines and for the proposed method, together with statistical significance or error bars where applicable.","section":"Abstract, Section IV, Appendix"},{"comment":"The proposed method is not experimentally isolated from confounds. The SFT+knowledge condition differs from the SFT baseline both in the addition of generated descriptions and in the length and content of the prompt text. The example in Figure 2 and Section V.C shows that the generated descriptions contain emotion words such as 'fear', 'pain', 'alarm', and 'sense of urgency.' These words may directly leak the target label or provide extra contextual cues that are unrelated to faithful affective knowledge extraction. Without a control condition using length-matched generic or emotion-free text, and without any evaluation of the quality of the generated descriptions (e.g., human ratings or agreement with human annotations), the observed gains cannot be attributed to the proposed knowledge-prompting mechanism. Please add such a control and a description-quality analysis.","section":"Section III.C.2, Figure 2, Section V.C"},{"comment":"The hyperparameter reporting is incomplete and one sentence is truncated. The text reads 'the training epoch was selected from' and then breaks off; no epoch values, learning rates, LoRA ranks, or optimization details are reported for the main experiments. Without these details, the experiments are not reproducible, and the claimed results cannot be independently verified. Please provide the complete experimental setup for every model and dataset, including prompt templates and split information.","section":"Section IV.B"},{"comment":"The analysis of modality contributions relies on Figures 3 and 4, but these figures are not present in the manuscript. The claims about audio-encoder pretraining alignment and visual encoder advantages rest entirely on these missing figures. Please include the figures with clear axes, legends, and captions, and ensure the quantitative values they display are consistent with the results tables.","section":"Section V.D"}],"minor_comments":[{"comment":"There is a typo in the second paragraph: 'or the CMU-MOSI and CMU-MOSEI datasets' should be 'for the CMU-MOSI and CMU-MOSEI datasets.'","section":"Section IV.A"},{"comment":"Several citations in the reference list contain formatting artifacts, such as 'V olume' in [43] and the missing space in 'arXiv e-prints , pp. arXiv–2502' in [39]. Please proofread the bibliography.","section":"General"},{"comment":"The dataset statistics mentioned at the start of Section III.A are not provided in the main text or appendix; a small table of dataset sizes, modalities, languages, and label distributions would help readers assess the benchmark scope.","section":"Section III.A"},{"comment":"The text states that 'all MLLMs achieve superior performance' on text modality, but without the underlying numbers this statement is too vague. Please tie this qualitative claim to specific metrics from the results tables.","section":"Section V.D"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is not ready for publication in its current form because the empirical evidence is absent. The missing results tables and the lack of a control condition for the prompting method are fixable within the scope of a revision, so I do not recommend reject at this stage. I would also ask the editor to verify that the repository contains the exact code and configuration files matching the eventual reported results, since the text alone currently provides no way to verify any of the quantitative claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful benchmark paper with a plausible but under-tested prompting trick, and the arXiv version does not include any of the numbers that back the main claim. If the authors supply the full results, it gets a fair shot; as it stands the central assertion is not checkable.\n\nThe genuinely new piece is the benchmark itself: seven open-source trimodal MLLMs (Qwen2.5-Omni, HumanOmni, Ola, VideoLLaMA2-AV, MiniCPM-o 2.6, PandaGPT, Emotion-LLaMA) evaluated across six established MAC datasets (MOSI, MOSEI, CH-SIMS, CH-SIMS v2, MELD, UR-FUNNY v2). As far as I know, and from reading the cited prior benchmarks, this is the first systematic comparison of audio+visual+text models on these tasks. That is a real service to the community. The authors also make a sensible observation about audio encoders: models with Whisper handle audio-only input far better than BEATs-based ones, which tracks the pretraining-task alignment. That kind of insight is actionable.\n\nThe proposed method—generate emotional descriptions from audio/video, concatenate them with the raw inputs, then LoRA fine-tune—is simple and reasonable. The benchmark itself is not circular: it uses external labeled data and published baselines. Credit where due.\n\nNow the soft spots. First, there are no numerical results anywhere in the manuscript. The abstract and conclusion say the hybrid strategy significantly improves performance, but the experiments section points to an appendix that only defines metrics and describes models. The hyperparameter sentence in IV.B is cut off mid-list. So the headline claim is unverifiable from this copy. Second, the prompting benefit is confounded. The SFT+knowledge condition adds both generated descriptions and more prompt text relative to the SFT baseline. The case study shows the model emitting words like 'fear', 'pain', 'sense of urgency'—those are label-like words. Without a length-matched generic-text control, you cannot attribute gains to accurate affective knowledge rather than to longer context or label leakage. The paper also does not evaluate the quality of the generated descriptions at all.\n\nIn proportion: these are not fatal errors in the benchmark idea, but they are load-bearing gaps in the reported evidence. The reader's conditional verdict is right, and the stress-test concern about the missing length-matched control is the sharper of the two.\n\nWho this is for: anyone working on MLLMs for sentiment/emotion tasks, and anyone building multimodal benchmarks. The paper deserves a serious referee only if the authors supply full result tables, error bars, and the missing hyperparameters; the reviewers should also demand the generic-text control and a description-quality check. I would not cite this version yet.","headline":"A plausibly first trimodal MLLM benchmark for affective computing, but the arXiv version omits all quantitative results and the prompting gain is not isolated from a length/text control.","tokens_in":13574,"tokens_out":2780,"would_cite":false,"duration_ms":31448,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Asking a multimodal language model to first describe emotion cues in audio and video, then fine-tuning it on those descriptions, improves affective computing across the benchmarked sentiment, emotion, and humor tasks.","keywords":["multimodal affective computing","multimodal large language models","generative knowledge prompting","supervised fine-tuning","low-rank adaptation","sentiment analysis","emotion recognition","humor detection"],"falsifier":"Run the proposed pipeline on one benchmark dataset, then re-run it with the same model and fine-tuning but with the model-generated audio and video descriptions replaced by descriptions taken from other clips; if accuracy does not drop, the content of the descriptions is not the load-bearing ingredient. A second check is to regenerate the descriptions with a differently trained MLLM and see whether the gains survive the switch.","tokens_in":12665,"feed_emoji":"🎭","tokens_out":9544,"duration_ms":108663,"temperature":0.7,"pith_summary":"Multimodal affective computing tries to infer emotion from text, audio, and video together, and this paper asks whether open-source multimodal large language models (MLLMs) can do that end-to-end on established benchmarks. The paper benchmarks seven such models on six datasets spanning sentiment analysis, emotion recognition, and humor detection, and argues that model architecture and dataset properties explain much of the performance spread. To improve accuracy, it proposes a hybrid strategy: first let the model, without training, generate written descriptions of emotion-relevant cues in the raw audio and video, then feed the original data together with those descriptions into a knowledge-augmented prompt and fine-tune the model with low-rank adapters. The central claim is that this integrated approach beats supervised fine-tuning alone across the benchmarked tasks, which would give practitioners a cheap, prompt-based route to better affective computing without new architectures or new datasets.","feed_headline":"Emotion AI improves when it first describes what it sees and hears","feed_subtitle":"Adding model-written audio and video descriptions before fine-tuning lifts scores across six emotion benchmarks.","key_machinery":"The mechanism is a two-stage generate-then-fine-tune setup. In stage one, the untuned MLLM is prompted zero-shot to produce explicit descriptions of emotional cues from the raw audio and video streams, such as a changing facial expression, a shouting voice, or background noise. In stage two, those descriptions are inserted, together with the original audio, video, and conversational text, into a knowledge-augmented prompt; supervised fine-tuning with low-rank adapter matrices then adjusts only a small fraction of the model's weights. The benchmark part of the paper treats the model zoo itself as an instrument: comparing MLLMs with different visual encoders, audio encoders, and fusion mechanisms, and correlating performance with dataset properties such as modality dominance.","core_discovery":"On its own terms, the paper makes two claims. The first is descriptive: on trimodal input, current MLLMs vary widely, and the variation tracks the fit between each model's pretraining choices, especially the audio encoder, and the downstream task, as well as the modality that dominates each dataset. The second is constructive: the proposed generate-then-fine-tune pipeline improves performance. The model first uses its zero-shot ability to produce explicit descriptions of emotional cues in the audio and video streams; these descriptions are then aggregated with the original audiovisual inputs and the dialogue text into one prompt; and supervised fine-tuning with low-rank adapters aligns the model to the task. The paper reports that this pipeline outperforms fine-tuning without generated descriptions on several datasets, with the gains concentrated where the audio or visual modality carries the discriminative signal, and with little or no gain on text-dominated datasets where the extra descriptions are redundant.","pith_inferences":["Because the descriptions come from the model itself, nothing in the method prevents using a stronger model to generate descriptions offline and feeding them to a weaker model at inference time; that would test whether the knowledge, rather than the model's own parameters, is doing the work.","A pure-prompting ablation, generative knowledge prompting without any fine-tuning, would separate the contribution of the prompt pattern from the contribution of the parameter update, and the paper's design makes that a natural next experiment.","The audio-encoder finding suggests a cheap selection heuristic for practical systems: among models of similar size, prefer one whose audio encoder was trained for speech-to-text, because that pretraining task appears to transfer well to affective judgments; this is testable by swapping audio encoders within one architecture.","The generated descriptions could also serve as external supervision: a large MLLM's descriptions of a dataset could be used to train a smaller, cheaper model that lacks multimodal input, effectively distilling the listening-and-watching step into text."],"forward_implications":["Practitioners can apply the recipe directly: feed raw audio and video to an MLLM, collect its written emotion-cue descriptions, add them to the prompt, and fine-tune only low-rank adapters.","Model selection for affective computing can be guided by the match between an encoder's pretraining objective and the target task; audio towers trained for speech recognition transfer better to sentiment tasks.","The size of the prompting gain is dataset-dependent, so gains should be expected mainly where audio or visual information, not text, carries the emotion signal.","The prompting boost is additive on top of supervised fine-tuning for several of the benchmarked sentiment, emotion, and humor detection tasks, rather than an either-or alternative."],"supporting_citations":[{"why":"Supplies the generative knowledge prompting method that the paper adapts to audio and video descriptions.","marker":"[21]"},{"why":"Supplies the low-rank adaptation technique used to make supervised fine-tuning of the MLLMs computationally feasible.","marker":"[47]"},{"why":"Provides the English sentiment-intensity dataset with seven-point labels used in the benchmark.","marker":"[16]"},{"why":"Provides the large-scale in-the-wild sentiment dataset used in the benchmark.","marker":"[43]"},{"why":"Provides the Chinese sentiment dataset with fine-grained modality annotations used in the benchmark.","marker":"[17]"},{"why":"Provides the extended Chinese sentiment dataset emphasizing non-verbal audio-visual cues.","marker":"[18]"},{"why":"Provides the conversational emotion recognition dataset used in the benchmark.","marker":"[19]"},{"why":"Provides the humor detection dataset used in the benchmark.","marker":"[20]"}],"fun_headline_variants":["Generate-then-fine-tune boosts multimodal emotion AI","Describing audio and video before tuning lifts emotion scores","Zero-shot descriptions then fine-tuning sharpens emotion AI","Emotion AI gets a boost from first describing its inputs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method only works if the descriptions the model writes about the audio and video are accurate and emotionally relevant; if those descriptions are noisy or biased, the measured gains could come from the prompting format rather than from real knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Generate-then-fine-tune boosts multimodal emotion AI","Describing audio and video before tuning lifts emotion scores","Zero-shot descriptions then fine-tuning sharpens emotion AI","Emotion AI gets a boost from first describing its inputs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1533,"prompt_tokens":941,"completion_tokens":592,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":527}},"tokens_in":557,"tokens_out":592,"duration_ms":6996,"temperature":1.0,"reasoning_tokens":527,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T04:57:11.012664+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the proposed pipeline on one benchmark dataset, then re-run it with the same model and fine-tuning but with the model-generated audio and video descriptions replaced by descriptions taken from other clips; if accuracy does not drop, the content of the descriptions is not the load-bearing ingredient. A second check is to regenerate the descriptions with a differently trained MLLM and see whether the gains survive the switch.","supporting_citations":[{"cited_title":"Lora: Low-rank adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"Supplies the low-rank adaptation technique used to make supervised fine-tuning of the MLLMs computationally feasible."},{"cited_title":"Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph,","cited_arxiv_id":null,"evidence_quote":"Provides the large-scale in-the-wild sentiment dataset used in the benchmark."},{"cited_title":"Ch- sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality,","cited_arxiv_id":null,"evidence_quote":"Provides the Chinese sentiment dataset with fine-grained modality annotations used in the benchmark."},{"cited_title":"Make acoustic and visual cues matter: Ch-sims v2. 0 dataset and av-mixup consistent module,","cited_arxiv_id":null,"evidence_quote":"Provides the extended Chinese sentiment dataset emphasizing non-verbal audio-visual cues."}],"review_version":1}