Pith. sign in

REVIEW 4 major objections 6 minor 45 references

The paper claims that a router agent picking a specialist expert before reasoning, trained with token-level reinforcement-learning credit assignment, lifts multimodal emotion understanding to a mean of 77.85% on a unified benchmark.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 08:41 UTC pith:GHY7YA2H

load-bearing objection A plausible RL-plus-agentic recipe for affective computing, but the headline SOTA claim currently rests on an unmeasured data-filtering confound and a deferred theoretical analysis. the 4 major comments →

arxiv 2607.21013 v2 pith:GHY7YA2H submitted 2026-07-23 cs.AI cs.CV

EmoAgent-R1: Towards Multimodal Emotion Understanding with Reinforcement Learning-based Dynamic Agent Specialization

Lihuang Fang , Yuchen Zou , kebing Jin , Jinghui Qin This is my paper
classification cs.AI cs.CV
keywords multimodal emotion understandingdynamic agent specializationagentic workflowP-GRPOreinforcement learningcredit assignmentchain-of-thoughtvideo understanding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multimodal emotion recognition from video is hard because the decisive evidence is usually not spread evenly across face, voice, and text: it is a fleeting glance, a tonal shift, or a sarcastic contradiction. The paper argues that the standard practice of giving a multimodal large language model a single fixed prompt biases it toward treating all cues uniformly, and that this "uniformity bias" can be broken by a two-stage agentic workflow in which a router agent first selects which specialized reasoning perspective fits the input and a specialist agent then produces focused chain-of-thought before answering. To train that workflow, the paper introduces P-GRPO (progressive group-relative policy optimization), a reinforcement-learning objective that keeps the group-relative advantage of the standard GRPO algorithm but redistributes it across tokens through a pointwise-mutual-information-inspired progressive modulation, converting a sparse terminal reward into fine-grained per-token learning signals. If the claim holds, emotion understanding in video improves without scaling model size, and the same routing-plus-credit-assignment recipe may transfer to other sparse-signal multimodal tasks. The paper reports a mean 77.85% on MER-UniBench, a unified benchmark covering sentiment, basic emotion, and fine-grained emotion tasks, ahead of the previous best specialist method at 75.95%.

Core claim

On its own terms, EmoAgent-R1's discovery is that the uniformity bias of fixed-prompt multimodal models is a measurable bottleneck and that two interventions fix it. First, dynamic agent specialization: the same backbone is prompted first as a router that chooses one of several specialist perspectives (for example, a facial-expression agent, a sarcasm-detection agent, or a text-centric agent), then as that specialist, producing a focused reasoning trace before the emotion label. Second, P-GRPO: instead of broadcasting the same group-relative advantage to every token of a sampled trajectory, P-GRPO multiplies each token's advantage by a modulation factor built from a proxy success probability

What carries the argument

The load-bearing machinery is P-GRPO combined with the two-stage agentic architecture. In P-GRPO, the sequence-level group advantage A^(g) = r^(g) − mean_j r^(j) is multiplied token-by-token by M_t^(g), a normalized modulation weight. The modulation is constructed from a proxy success probability p_final = σ(κ z) on the normalized group reward, a prefix-confidence schedule p_t = clip(p_final^(t/L)^β, p_min, p_max) that assumes confidence rises monotonically along a good trajectory, and a weight w_t = exp(α clip(log(p_t/p_{t−1}), −c, c)); M_t ≡ 1 recovers GRPO. The architecture side consists of a router agent that selects among K specialist prompt strategies and a specialist agent that reason

Load-bearing premise

The load-bearing premise is that training samples discarded because a teacher model could not produce consistent, high-quality reasoning are noise rather than genuinely hard emotion cases; if the discarded samples are concentrated in rare labels, sarcasm, or subtle expressions, the reported lead could come from training on an easier distribution rather than from the new credit-assignment method.

What would settle it

Hold out all training instances for which the teacher model failed to produce consistent reasoning, and evaluate the full model and the baseline GRPO variant on that hard subset. If the reported advantage over the previous best specialist shrinks or reverses there, the data-filtering premise is the explanation; if token-level modulation no longer beats baseline GRPO on that subset, the claimed credit-assignment benefit is not doing the work.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • On MER-UniBench, EmoAgent-R1 reports a mean 77.85%, ahead of the previous best specialist method's 75.95%, and improves on all but one of the nine sub-tasks, covering sentiment, basic emotion, and fine-grained open-vocabulary emotion understanding.
  • In ablations, P-GRPO beats vanilla GRPO on every reported emotion metric (e.g., MOSEI sentiment 82.66 vs. 81.11; fine-grained OV-MERD+ 64.29 vs. 63.35), consistent with the claim that token-level credit assignment adds a real gain beyond group-relative advantage.
  • Removing the dynamic router/specialist structure drops fine-grained OV-MERD+ from 64.29 to 60.53, which the paper reads as evidence that specialization itself, not just the RL objective, is responsible for the improvement.
  • A 3-billion-parameter version of EmoAgent-R1 beats a 7-billion-parameter instruction-tuned baseline on the reported emotion tasks (74.87 vs. 61.70 on the four-column subset), suggesting the approach can substitute for raw parameter scaling in this domain.
  • On general video benchmarks, EmoAgent-R1 roughly matches or slightly exceeds its 7-billion-parameter backbone (e.g., VSI-Bench 33.1 vs. 30.1, MMVU 61.9 vs. 60.0), indicating that emotion-focused RL with dynamic specialization does not degrade general video understanding.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The RL-phase data filtering discards samples for which the teacher could not produce consistent reasoning; a stress test would evaluate on exactly those discarded instances. If the model's advantage shrinks there, the headline lead may owe more to distribution trimming than to token-level credit assignment.
  • P-GRPO's prefix-confidence schedule assumes success probability rises monotonically along a correct trajectory. Emotional cues can arrive late (a punchline, a sarcastic reversal, a terminal micro-expression), so a non-monotonic or position-aware modulation variant is a natural testable extension for this domain.
  • The specialist library is implemented as fixed prompt templates. Learning the specialist embeddings, or allowing soft mixtures of specialists, could reduce hard routing errors and may transfer better to emotion taxonomies the router has not seen.
  • The evaluation is on one unified benchmark family. Comparing the router's chosen specialist against human judgments of which modality carries the emotion for the same videos would independently test whether the specialization decisions are semantically meaningful rather than a latent shortcut.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes EmoAgent-R1, a two-phase framework for multimodal emotion recognition (MER). Phase I is a cold-start SFT stage that trains a Qwen2.5-VL-7B backbone on synthetic answer-conditioned chain-of-thought data for specialized emotion-reasoning agents and on hindsight-relabeled routing data for a router agent. Phase II applies a new RL algorithm, P-GRPO, which reweights standard GRPO group-relative advantages by a token-level modulation factor derived from a monotone confidence schedule (Eqs. 5--6), with the stated aim of fixing coarse-grained credit assignment. The paper reports a new state-of-the-art mean score of 77.85% on MER-UniBench, ablation gains for the cold-start components, P-GRPO over GRPO, and the dynamic agent specialization architecture, plus generalization results on general video benchmarks.

Significance. If the empirical claims are validated, the paper addresses a real limitation of monolithic MLLM prompting for MER: emotional cues are often localized and modality-specific, and a dynamic router-plus-specialist decomposition is a plausible remedy. The P-GRPO idea—replacing uniform token-level advantage with a progressive, confidence-derived modulation—is also relevant beyond emotion recognition and could be useful to the broader RLVR community. The paper is broad in evaluation coverage (sentiment, basic emotion, fine-grained emotion, general video benchmarks) and clearly specifies the training pipeline. However, the central SOTA and ablation claims currently rest on an unmeasured data-filtering confound, the core algorithmic contribution lacks the promised theoretical analysis (which is deferred to a missing supplementary), and none of the experimental tables report variance or significance. These issues prevent the current manuscript from supporting its headline conclusions.

major comments (4)
  1. [§4.1, Table 1] The RL training set is filtered by discarding 'instances where the teacher model fails to generate consistent, high-quality reasoning paths.' No discard rate, no comparison of retained vs. discarded instances by task/label, and no evaluation on the discarded subset are reported. If the discarded instances are disproportionately hard—rare labels, sarcasm, micro-expressions, cross-modal contradictions—then the 1.9-point SOTA lead over AffectGPT-R1 (77.85 vs. 75.95) and the P-GRPO-vs-GRPO ablation gains could partly reflect an easier training distribution rather than dynamic agent specialization or token-level credit assignment. The paper must report filtering statistics, evaluate on the discarded subset, and/or include an unfiltered RL baseline to rule out this confound.
  2. [§3.4.2, Eqs. (5)–(6); §3.4.3] P-GRPO is the central algorithmic contribution, but its theoretical analysis is explicitly deferred to 'the Supplementary Materials,' which are not included in the manuscript. The modulation schedule is asserted with hand-picked constants (κ=2, β=1/2, pmin=0.1, pmax=0.99, α=0.5, c=2, ω0) and the key assumption that prefix confidence increases monotonically along a correct trajectory is not justified. As written, Eqs. (5)–(6) define an ad hoc reweighting of group-relative advantages; without a proof that this estimator avoids bias, preserves the GRPO fixed point, or retains convergence guarantees, the claim that P-GRPO 'mitigates the coarse-grained uniform credit assignment issue' is not supported. Please include the missing theoretical analysis and a sensitivity study over the free constants.
  3. [§4.2, Tables 2 and 4] No error bars, confidence intervals, or significance tests are reported anywhere. Several key differences are small: the SOTA margin is 1.9 points (77.85 vs. 75.95), the P-GRPO vs. GRPO OV-MERD+ gain is 0.94 points (64.29 vs. 63.35), and some general-benchmark gains are under 1 point. Without multiple seeds or paired significance testing, the claims that EmoAgent-R1 'consistently outperforms' baselines and that removing DAS causes a 'sharp decline' are not statistically established. At minimum, report mean±std over at least three runs for the main tables and a significance test for the SOTA and ablation comparisons.
  4. [§3.3.3] The router supervision is constructed by hindsight relabeling: for each training instance, all specialized agents are executed, and the routing targets are the agents that happen to predict the ground-truth label. This is self-referential because the router is trained to imitate an oracle that saw the answer. The manuscript does not report how often the success subset Z* is empty (so no routing sample is generated), how many routing samples were used, or the router's accuracy on held-out data. These statistics are needed to assess whether the Empirical Oracle provides clean supervision or merely encodes the model's own biases and spurious successes.
minor comments (6)
  1. [Eq. (5)] The PMI term PMI_t = log(p_t / p_{t-1}) requires p_0, but p_0 is never defined. Please specify the initial value (e.g., p_0 = pmin) or otherwise resolve the edge case at t=1.
  2. [Eqs. (5) and (7)] The symbol β is overloaded: it is the exponent in the confidence schedule in Eq. (5) and the KL regularization coefficient in Eq. (7). Use distinct notation to avoid ambiguity.
  3. [Table 5] The 'Mean' column in Table 5 appears to be computed over only four tasks (MOSEI, SIMS v2, MER23, OV-MERD+), giving 78.32 for EmoAgent-R1 (7B), whereas Table 2 reports a mean of 77.85 over nine tasks. Please state the task subset used for the mean and reconcile the two numbers.
  4. [Table 4] The row labels 'w/ GRPO', 'w/ P-GRPO', 'w/o DAS', and 'w/ DAS' do not specify which components are active in each row. In particular, it should be clear whether 'w/o DAS' uses P-GRPO and whether 'w/ GRPO' includes the dynamic agent specialization. Please clarify the factor combinations.
  5. [§5, Table 3] The generalization claims are overstated relative to the reported numbers: e.g., TempCompass is identical to the backbone (72.6 vs. 72.6) and VideoMME differs by only 0.5 points (57.1 vs. 56.6). With no error bars, these should be described as 'no clear degradation' rather than 'steady improvements.'
  6. [Throughout] Several implementation details are deferred to a supplementary that is not included: agent prompt designs, baseline descriptions, P-GRPO theory, qualitative analyses, and dynamics of agent selection. This limits reproducibility; please either include the supplementary or summarize the essential details in the main text.

Circularity Check

0 steps flagged

No significant circularity: benchmark comparisons are external; P-GRPO is an empirically evaluated reweighting, and the only self-referential component (hindsight-relabeled router targets) is auxiliary and disclosed.

full rationale

The central empirical claims—SOTA mean 77.85% on MER-UniBench and P-GRPO's gains over GRPO—rest on external benchmark evaluation and ablations, not on equations that reduce to the claim. The RL reward (Eq. 9) uses the same EW metric as the benchmark, but this is a standard, disclosed training/eval alignment; it is not a hidden renaming or fitted-parameter prediction because the test numbers come from MER-UniBench against external baselines. Eqs. (3)-(6) define P-GRPO as a per-token reweighting of GRPO's group advantage, with the paper explicitly noting 'When M_t^g ≡ 1, P-GRPO reduces exactly to GRPO'; this is a generalization, and its claimed benefit is tested by ablation (Table 4), not derived from the benchmark. Cold-start CoT synthesis (§3.3.2) uses teacher generation filtered to correct labels—rejection sampling, not a circular input. The hindsight-relabeled router supervision (§3.3.3) is self-referential in that routing targets are built from the model's own successes, but it is an auxiliary bootstrap signal: the final emotion prediction is still scored against ground truth, and the routing targets do not by construction determine the reported scores. The RL data filter in §4.1 ('instances where the teacher model fails to generate consistent, high-quality reasoning paths are discarded as noise') is a real distribution-shift/correctness risk for the SOTA comparison—no discard rate or discarded-subset evaluation is reported—but it is a confound, not a circularity, because no specific reduction from the filter to the headline number is exhibited. There are no load-bearing self-citations (the AffectGPT/AffectGPT-R1 references are external baselines/benchmarks) and no imported uniqueness theorem. The deferred 'theoretical analysis' of P-GRPO in supplementary material is an omitted proof, but omission is not circularity. Under the hard-rule standard that circularity requires a quotable identity or a fitted input renamed as prediction, none is present.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 2 invented entities

The central claim rides on the cold-start data, the hindsight-relabeled routing targets, and the P-GRPO per-token confidence schedule; all are in-paper or teacher-model assumptions rather than externally verified facts.

free parameters (2)
  • P-GRPO modulation constants κ, β, p_min, p_max, α, c, τ = κ=2; β=1/2; p_min=0.1; p_max=0.99; α=0.5; c=2; τ not reported
    Chosen by hand in §3.4.2 to shape token-level advantage; no sensitivity analysis or fitting procedure is given.
  • KL regularization coefficient in Eq. (7) = not reported
    The PPO-style KL weight β_KL is not specified in the implementation details, making exact replication difficult.
axioms (5)
  • domain assumption A single MLLM can realize qualitatively distinct reasoning strategies by switching prompt templates in the Specialized Agent Library.
    Section 3.2: agents are 'prompt templates' within one backbone; if prompt changes do not activate distinct knowledge, the routing gain is just prompt or length variation.
  • ad hoc to paper The monotone confidence schedule p_t^g = p_final^(t/L)^β approximates the true per-token causal contribution to success.
    Section 3.4.2, Eq. (5); no derivation or empirical validation of this surrogate PMI schedule is provided.
  • ad hoc to paper The Empirical Oracle built by hindsight relabeling (agents that solved the training instance define valid routing targets) provides clean router supervision.
    Section 3.3.3; correct-by-luck agents can produce noisy routing targets, and no analysis of this noise is given.
  • domain assumption Teacher Qwen2.5-VL-72B rationales filtered by GPT-4o are faithful and deduction-consistent.
    Section 3.3.2; the whole cold-start corpus inherits any teacher or verifier bias.
  • standard math GRPO's estimator and the PPO clipped objective with KL regularization are valid optimization targets for autoregressive policies.
    Sections 3.4.1–3.4.2 inherit these from GRPO [30] and PPO [29].
invented entities (2)
  • Router Agent role no independent evidence
    purpose: Selects one of K expert prompts before emotion reasoning begins.
    Implemented inside the same Qwen backbone as a prompt; no external falsifiable handle, only in-paper ablations.
  • Specialized Agent Library (facial expert, sarcasm expert, etc.) no independent evidence
    purpose: Focused reasoning under different prompt templates targeting different emotional evidence types.
    Presented as distinct reasoning strategies but realized only by prompt text within one weight set.

pith-pipeline@v1.3.0-alltime-deepseek · 14885 in / 13097 out tokens · 137013 ms · 2026-08-01T08:41:22.141961+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of EmoAgent-R1: Towards Multimodal Emotion Understanding with Reinforcement Learning-based Dynamic Agent Specialization." pith.science (2026). https://pith.science/paper/GHY7YA2H

@misc{pith2026260721013,
  author       = {Pith},
  title        = {Pith review of: EmoAgent-R1: Towards Multimodal Emotion Understanding with Reinforcement Learning-based Dynamic Agent Specialization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GHY7YA2H}},
  note         = {Machine review of arXiv:2607.21013}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Multimodal large language models (MLLMs) have achieved impressive performance in multimodal emotion recognition (MER) tasks and lifted MER to a new level that is complex emotion understanding with advanced video understanding abilities and natural language description. However, existing MLLM-based methods often use a fixed prompt to perceive the emotions, ignoring the dynamicity and complexity of the emotion source in the multimodal inputs. To address these issues, we propose a novel Reinforcement Learning-based Dynamic Agent Specialization framework (\textbf{EmoAgent-R1}) to optimize the emotion recognition, reasoning, and generalization abilities of an MLLM with dynamic agent specialization based on reinforcement learning. Specifically, we first adopt a cold start strategy to endow an MLLM with preliminary emotion recognition, reasoning, and agent routing ability by training with synthetic answer-conditioned chain-of-thought data and agent routing data. Then, we further train the MLLM with reinforcement learning to perceive emotions in a two-step agentic workflow with agent selection and agent specialization. To effectively train EmoAgent-R1, we propose a novel Progressive Group-Relative Policy Optimization (P-GRPO) to combine group-based relative advantages with a PMI-inspired progressive token-level modulation to transform sparse rewards into fine-grained learning signals, mitigating the coarse-grained uniform credit assignment issue in GRPO. Extensive experiments on MER benchmarks demonstrate the superiority of our EmoAgent-R1 in stronger emotion reasoning performance and improved optimization stability.

Figures

Figures reproduced from arXiv: 2607.21013 by Jinghui Qin, Kebing Jin, Lihuang Fang, Yuchen Zou.

Figure 1
Figure 1. Figure 1: Comparison between a conventional monolithic agent (top part) and the proposed EmoAgent-R1 framework (bottom [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The framework of EmoAgent-R1. The pipeline consists of two phases: (1) Cold-Start Training, where the router and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 18 linked inside Pith

  1. [1]

    AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis- Philippe Morency. 2018. Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Iryna Gurevych and Yusuke Miyao (Eds.). ...

    doi:10.18653/v1/p18-1208 2018

  2. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Rep...

    Pith/arXiv arXiv 2025

  3. [4]

    Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. 2024. Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning.Advances in Neural Information Processing Systems37 (2024), 110805–110853

    2024

  4. [5]

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. 2024. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476(2024)

    Pith/arXiv arXiv 2024

  5. [6]

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. 2025. Video- R1: Reinforcing Video Reasoning in MLLMs. InThe Thirty-ninth Annual Confer- ence on Neural Information Processing Systems. https://openreview.net/forum? id=a2JTVVvcEl

    2025

  6. [7]

    Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. 2025. Group-in-group policy optimization for llm agent training.arXiv preprint arXiv:2505.10978(2025)

    Pith/arXiv arXiv 2025

  7. [8]

    Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. 2019. DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-...

    2019

  8. [9]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948(2025)

    Pith/arXiv arXiv 2025

  9. [10]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024)

    Pith/arXiv arXiv 2024

  10. [11]

    Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. 2023. Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding.arXiv preprint arXiv:2311.08046(2023)

    Pith/arXiv arXiv 2023

  11. [12]

    Ao Li, Longwei Xu, Chen Ling, Jinghui Zhang, and Pengwei Wang. 2025. Emo- Verse: Enhancing Multimodal Large Language Models for Affective Computing via Multitask Learning.Neurocomputing650 (2025), 130810

    2025

  12. [13]

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Joshua Adrian Cahyono, Jingkang Yang, Chunyuan Li, and Ziwei Liu. 2025. Otter: A Multi-Modal Model With In-Context Instruction Tuning.IEEE Transactions on Pattern Analysis and Machine Intelligence47, 9 (2025), 7543–7557. doi:10.1109/TPAMI.2025.3571946

    arXiv 2025

  13. [14]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024. LLaVA- OneVision: Easy Visual Task Transfer.Transactions on Machine Learning Research (2024)

    2024

  14. [15]

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2024. VideoChat: Chat-Centric Video Understanding. arXiv:2305.06355 [cs.CV] https://arxiv.org/abs/2305.06355

    Pith/arXiv arXiv 2024

  15. [16]

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. 2024. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 22195–22206

    2024

  16. [17]

    Yanwei Li, Chengyao Wang, and Jiaya Jia. 2024. LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models.European Conference on Computer Vision

    2024

  17. [18]

    Zaijing Li, Ting-En Lin, Yuchuan Wu, Meng Liu, Fengxiao Tang, Ming Zhao, and Yongbin Li. 2023. Unisa: Unified generative framework for sentiment analysis. In Proceedings of the 31st ACM international conference on multimedia. 6132–6142

    2023

  18. [19]

    Zihao Li, Zhuoran Yang, and Mengdi Wang. 2023. Reinforcement learning with human feedback: Learning dynamic choices via pessimism.arXiv preprint arXiv:2305.18438(2023)

    Pith/arXiv arXiv 2023

  19. [20]

    Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Ze- bang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, et al . 2025. AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models. InForty-second International Conference on Machine Learning

    2025

  20. [21]

    Zheng Lian, Haiyang Sun, Licai Sun, Haoyu Chen, Lan Chen, Hao Gu, Zhuofan Wen, Shun Chen, Zhang Siyuan, Hailiang Yao, et al. [n. d.]. OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition. InForty-second International Conference on Machine Learning

  21. [22]

    Zheng Lian, Fan Zhang, Yazhou Zhang, Jianhua Tao, Rui Liu, Haoyu Chen, and Xiaobai Li. 2025. AffectGPT-R1: Leveraging Reinforcement Learning for Open-Vocabulary Multimodal Emotion Recognition. arXiv:2508.01318 [cs.HC] https://arxiv.org/abs/2508.01318

    arXiv 2025

  22. [23]

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024. Video-llava: Learning united visual representation by alignment before projection. InProceedings of the 2024 conference on empirical methods in natural language processing. 5971–5984

    2024

  23. [24]

    Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. 2024. Kangaroo: A Powerful Video- Language Model Supporting Long-context Video Input. arXiv:2408.15542 [cs.CV] https://arxiv.org/abs/2408.15542

    Pith/arXiv arXiv 2024

  24. [25]

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783(2025)

    Pith/arXiv arXiv 2025

  25. [26]

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, Xiuyu Li, Yunhao Fang, Yukang Chen, Cheng-Yu Hsieh, De-An Huang, An-Chieh Cheng, Vishwesh Nath, Jinyi Hu, Sifei Liu, Ranjay Krishna, Daguang Xu, Xiaolong Wang, Pavlo Molchanov, Jan Kautz, Hongxu Yin, Song Han, and Yao Lu. 2024. NVIL...

    Pith/arXiv arXiv 2024

  26. [27]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan

  27. [28]

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Mar- keeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. 2023. Perception test: A diagnostic benchmark for multimodal video models.Advances in Neural Information Processing Systems36 (2023), 42748–42761

    2023

  28. [29]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  29. [30]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeek- Math: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300 [cs.CL] https://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv 2024

  30. [31]

    Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. 2023. PandaGPT: One Model To Instruction-Follow Them All. InProceedings of the 1st Workshop on Taming Large Language Models: Controllability in the era of Interactive Assistants!, Devamanyu Hazarika, Xiangru Robert Tang, and Di Jin (Eds.). Association for Computational Linguistics, Prague,...

    2023

  31. [32]

    Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan

  32. [33]

    Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, Junrui Shen, Wenxiang Chen, Jiaqi Liu, Jixuan Huang, Zhihao Zhang, Honglin Guo, et al. 2025. Bapo: Stabilizing off-policy reinforcement learning for llms via balanced policy optimization with adaptive clipping.arXiv preprint arXiv:2510.18927(2025)

    arXiv 2025

  33. [34]

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 9777–9786

    2021

  34. [35]

    Yang Yang, Xunde Dong, Yupeng Qiang, Xunde Dong, and Yupeng Qiang. 2025. MSE-Adapter: A Lightweight Plugin Endowing LLMs with the Capability to Perform Multimodal Sentiment Analysis and Emotion Recognition. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 25642–25650

    2025

  35. [36]

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Jun- yang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou

  36. [37]

    Tenenbaum

    Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Tor- ralba, and Joshua B. Tenenbaum. 2020. CLEVRER: Collision Events for Video Representation and Reasoning. InICLR. Conference’17, July 2017, Washington, DC, USA Fang et al

    2020

  37. [38]

    En Yu, Kangheng Lin, Liang Zhao, Yana Wei, Zining Zhu, Haoran Wei, Jian- jian Sun, Zheng Ge, Xiangyu Zhang, Jingyu Wang, and Wenbing Tao. 2025. Unhackable Temporal Reward for Scalable Video MLLMs. InThe Thirteenth Inter- national Conference on Learning Representations. https://openreview.net/forum? id=Gf1uBeuUJW

    2025

  38. [39]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. 2025. Dapo: An open- source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476 (2025)

    Pith/arXiv arXiv 2025

  39. [40]

    arXiv:2304.14178 [cs.CL] https://arxiv.org/abs/2304.14178

    mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality. arXiv:2304.14178 [cs.CL] https://arxiv.org/abs/2304.14178

    Pith/arXiv arXiv

  40. [41]

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chun- yuan Li. 2024. Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713(2024)

    Pith/arXiv arXiv 2024

  41. [42]

    Jiaxing Zhao, Xihan Wei, and Liefeng Bo. 2025. R1-omni: Explainable omni- multimodal emotion recognition with reinforcement learning.arXiv preprint arXiv:2503.05379(2025)

    Pith/arXiv arXiv 2025

  42. [44]

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. 2024. Long Context Transfer from Language to Vision.arXiv preprint arXiv:2406.16852 (2024). https://arxiv.org/abs/2406.16852

    Pith/arXiv arXiv 2024

  43. [2017]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347 (2017)

    Pith/arXiv arXiv 2017

  44. [2021]

    In Annual Conference on Neural Information Processing Systems

    STAR: A Benchmark for Situated Reasoning in Real-World Videos. In Annual Conference on Neural Information Processing Systems

  45. [2024]

    InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024)

    Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024)

    2024