Pith. sign in

REVIEW 3 major objections 6 minor 57 references

Video emotion should be modeled as event-driven affect transitions, not one static clip label.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 13:18 UTC pith:42V6TWQT

load-bearing objection Solid new benchmark package for event-aligned viewer affect; main soft spot is MLLM-mediated labels, not a broken method. the 3 major comments →

arxiv 2607.10238 v1 pith:42V6TWQT submitted 2026-07-11 cs.CV

Benchmarking Dynamic Affective Reasoning: A Viewer-Centric Video Emotion Dataset

Zhiyan Zhang , Peipei Song , Jinpeng Hu , Jingyang Jia , Xun Yang , Xiaojun Chang This is my paper
classification cs.CV
keywords dynamic affective reasoningviewer-centric video emotionaffective segmentationcausal emotion reasoningmultimodal large language modelsreinforcement learningAffective Events Theoryvideo understanding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that standard video emotion analysis is too static: it labels whole clips and largely ignores how a viewer’s feeling changes when consecutive events accumulate. Drawing on Affective Events Theory, the authors reframe the problem as Dynamic Affective Reasoning—finding when emotion shifts, naming the fine-grained viewer emotion, and explaining why the visual event caused it. They release DAR, a large viewer-centric dataset of 15,087 videos and 36,908 event-aligned segments with 27 emotion categories and dense causal reasoning chains. They also define three linked tasks (affective segmentation, fine-grained classification, and causal reasoning) and train DAR-R1, a two-stage model that first learns the required output structure and then uses reinforcement learning with rewards for boundaries, emotion accuracy, and reasoning quality. Across more than ten multimodal models, DAR-R1 leads on temporal localization and affective reasoning, supporting the claim that emotion understanding needs event-level, viewer-centric dynamics.

Core claim

The paper claims that viewer emotion in video is better treated as a sequence of event-aligned affect phases with explicit causal explanations than as a single clip-level label, and that a large dataset built this way (DAR) plus a two-stage SFT-then-GRPO model (DAR-R1) yields new state-of-the-art results on emotional localization and causal reasoning.

What carries the argument

DAR-R1: cold-start supervised fine-tuning for structured multi-segment outputs, followed by Group Relative Policy Optimization with rewards for schema validity, segment count, temporal IoU/boundary precision, emotion match under IoU>0.5, and non-repetitive reasoning quality.

Load-bearing premise

The load-bearing premise is that the largely model-generated, model-verified emotion labels and causal rationales truly track human viewers’ affect transitions rather than a machine consensus story about emotion.

What would settle it

Have independent human viewers mark emotion-shift times, choose from the 27 categories, and write short causal reasons on the same test videos; if human boundaries, labels, and rationales systematically disagree with DAR annotations or reverse DAR-R1’s ranking against strong baselines, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Emotion-aware video systems should output multi-phase timelines with reasons, not one global tag.
  • Training and evaluation should reward temporal boundary precision and causal grounding, not only category accuracy.
  • Viewer-centric 27-way taxonomies become a practical target for fine-grained affect modeling beyond basic facial-expression labels.
  • Affective agents can be scored on when emotion changes, what it becomes, and why the preceding visual event caused the shift.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If model-written labels dominate the ground truth, future gains may overfit annotation style unless human re-annotation or live viewer studies become the primary yardstick.
  • The same event-aligned formulation could transfer to domains where stimuli accumulate over time, such as interactive counseling videos or continuous user-response monitoring.
  • Reward design that separately pressures boundaries, counts, and rationales may be reusable for other multi-segment video reasoning tasks beyond emotion.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper reframes video emotion analysis as Dynamic Affective Reasoning (DAR): temporally grounded segmentation of viewer affect shifts, fine-grained classification over 27 categories, and causal explanation of those shifts. It releases a large benchmark (15,087 videos, 36,908 event-aligned segments) built from VCE via a coarse-to-fine pipeline (semantic boundaries + PySceneDetect snapping, incremental differential captioning, stream-of-affect reasoning under Affective Events Theory), with dual-MLLM consistency checks and a post-hoc expert audit (Table 4). It also proposes DAR-R1 (SFT cold-start + GRPO with structural, count, temporal IoU, emotion, and reasoning rewards) and reports gains over 10+ MLLMs on segment-count accuracy, mIoU, emotion accuracy, and GPT-judged reasoning (Table 2), with supporting human ratings (Table 3), fine-tuned emotion baselines (Table 5), reward ablations (Table 7), and a small OOD localization check on TSL (Table 6).

Significance. If the annotations are accepted as a useful proxy for viewer-centric affect dynamics, the work is a clear contribution: it defines a hierarchical task suite that existing static clip-label and character-centric benchmarks do not cover, ships a large event-aligned resource with dense causal chains, and shows that task-specific GRPO rewards improve both boundary localization and reasoning quality over strong open MLLMs. The construction pipeline, multi-dimensional reward design, ablations, fine-tuned emotion baselines, and expert preference study are concrete engineering assets for affective computing and video MLLMs. The main significance risk is not novelty of the task idea but whether SOTA on DAR generalizes beyond model-consensus affect narratives to genuine human viewer transitions.

major comments (3)
  1. §3.3–3.4 and Table 4: The central claim that DAR is a large-scale viewer-centric ground-truth resource rests on MLLM generation (Gemini boundary proposals; Qwen3-VL captions and four emotion–reason pairs from VCE Top-3) filtered by other MLLMs (InternVL3.5, Qwen3-omni) on the same axes later used for evaluation (visual grounding, causal logic, viewer centricity, temporal consistency, answer consistency). Table 4 reports low expert error rates (~2–3%) on the full test split and a training subset, but this is post-hoc agreement with already-filtered model outputs, not independent multi-annotator human labeling of the same clips with inter-annotator agreement. Because generators and judges share similar pretraining priors, dual-consistency can certify coherent model-consensus stories rather than human affect transitions. This underwrites every supervised target and the R_emo / R_reason / GP
  2. §5.1 Evaluation Metrics and Table 2 GPT-Score: Reasoning quality is scored by GPT-4o on the same five dimensions used in dual-consistency verification during construction. Combined with MLLM-generated rationales as training targets, this creates a closed evaluation loop that can reward stylistic and prior alignment rather than human-validated causal correctness. Table 3’s expert ratings on 100 videos are a useful check and show agreement with GPT preferences, but N=100 and five experts are thin relative to 1,441 test videos. Strengthen the paper by expanding human evaluation (or releasing full expert scores), reporting correlation/disagreement between GPT-4o and experts per dimension, and reporting automatic text metrics (e.g., the ROUGE-L/SBERT already used in Tables 5 and 7) alongside GPT scores in the main comparison table.
  3. Table 2 Emo-Acc (28.6% for DAR-R1) and §5.2: Absolute emotion accuracy remains low even after SFT+GRPO, while SC-Acc and mIoU are only moderately improved. The manuscript attributes this to task difficulty, but under the annotation pipeline of §3.3 the same numbers are also consistent with residual label noise or over-constrained Top-3 candidate conditioning. Please analyze error modes (confusion among related emotions, boundary-induced label mismatch, sustained vs. transition segments) and, if possible, estimate an upper bound via human agreement on emotion labels for matched segments. Without that, it is hard to interpret whether DAR-R1 advances genuine affective recognition or mainly format/localization compliance on synthetic targets.
minor comments (6)
  1. Table 1: “Dynamic Emotion” is marked only for DAR; briefly define the criterion in the caption so the comparison is falsifiable against MELD timestamps or continuous-affect datasets discussed in Appendix A.1.
  2. §4.2 Eqs. (4)–(9): Reward weights (w1…w5), λ’s, and temperatures are stated in text/implementation details but not jointly tabulated; a single hyperparameter table would aid reproducibility.
  3. Figure 3(c) transition matrix: report how transitions are normalized (row-stochastic?) and whether rare emotions are pooled; the two highlighted transitions alone do not establish “structured non-random” dynamics without a null baseline.
  4. §5.1: Vision encoder is frozen for Qwen2.5-VL-3B; note whether this choice limits fine temporal boundary learning relative to unfreezing the aligner only, or provide a short ablation if available.
  5. Appendix E Table 5: Fine-tuned emotion baselines still lag badly on Emo-Acc; clarify training budget parity (epochs, data mixture, output schema adaptation) so the comparison is fair.
  6. Typos/consistency: “viewer-Centric” capitalization in §2.2; occasional missing spaces in compound words in the abstract/intro (“changeasaresultof”, “large-scalebenchmark”); arXiv id in header is fine but ensure camera-ready fixes.

Circularity Check

1 steps flagged

Mild closed-loop circularity on reasoning-quality scoring (same five axes used to filter synthetic rationales and to GPT-judge them); segmentation/emotion metrics are ordinary train–test, not definitional.

specific steps
  1. other [§3.4 Dual-Consistency Verification; §5.1 Evaluation Metrics (Reasoning Quality); App. I LLM-as-a-Judge prompt]
    "the generated emotion and reason pairs are evaluated across five distinct dimensions: 1. Visual Grounding. ... 2. Causal Logic. ... 3. Viewer Centricity. ... 4. Temporal Consistency. ... 5. Answer Consistency. ... To evaluate rationale quality, we employ GPT-4o as a judge. The judge rates each generated rationale on a 0–5 scale, focusing on visual grounding, causal logic, viewer centricity, temporal consistency, and answer consistency."

    Training rationales are generated by MLLMs and admitted only after dual-MLLM judges score them on VG/CL/VC/TC/AC (with rewrite-on-fail). The same five axes are then used by GPT-4o to score model outputs as 'reasoning quality' (Table 2 GPT-Score). Optimizing SFT/GRPO on that filtered distribution therefore improves the reported GPT-Score partly by construction—matching the filter criteria—rather than by an independent human-affect criterion. This does not collapse SC-Acc/mIoU/Emo-Acc, which use fixed labels and temporal IoU, but it does make the reasoning-quality SOTA partially self-referential.

full rationale

This is a dataset-and-benchmark paper, not a first-principles derivation. Emotion candidates are constrained by external VCE Top-3 labels; boundaries are snapped to PySceneDetect cuts; SC-Acc/mIoU/Emo-Acc are scored against a fixed held-out split with standard IoU matching—none of these reduce by construction to the training objective in a definitional sense. Human audit (Table 4) and expert ratings (Table 3) plus OOD TSL localization further supply independent content. The only clear circular step is the reasoning-quality loop: dual-MLLM verification in construction and GPT-4o evaluation both score the same five axes (visual grounding, causal logic, viewer centricity, temporal consistency, answer consistency), so high GPT-Score partly reflects training on text already selected/rewritten for those axes. That is partial methodological circularity on one reported metric, not a self-definitional derivation of the main SOTA claim. No load-bearing self-citation uniqueness theorem, no fitted physical parameter renamed as prediction, and no ansatz smuggled as a theorem. Score 3 is proportionate.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

The central scientific package rests on Affective Events Theory as a modeling prior, the VCE 27-way taxonomy and source videos, heavy use of proprietary/open MLLMs as annotators and judges, and a large set of hand-set reward and pipeline thresholds. The invented objects are the DAR task suite, the event-aligned annotation schema, and the DAR-R1 training recipe—not new physical entities. Free parameters are mostly engineering knobs in segmentation and GRPO rewards that shape the reported SOTA.

free parameters (5)
  • GRPO reward weights (w1..w5)
    Structural, count, segmentation, emotion, and reasoning rewards are weighted 0.10/0.25/0.25/0.25/0.15; these hand-chosen weights directly drive the RL policy that claims SOTA.
  • Structural reward lambdas and length limits
    λ1=0.5, λ2=0.2, λ3=0.3; τmin=0.4s; Lsoft=1200, Lhard=2400, τlen=400 control format, fragmentation, and length penalties.
  • Segment-count and temporal reward temperatures
    τcnt=1.5, α=0.5, τtemp=0.5 set how harshly count mismatch and boundary distance are penalized in Rcount/Rseg.
  • Reasoning length/uniqueness targets
    µlen=120, σlen=40, τdup=0.75 define the preferred rationale length and anti-repetition threshold in Rreason.
  • Boundary snap window and quality threshold T
    Semantic boundaries snap within ±0.5s of visual cuts; dual-consistency score threshold T gates inclusion/rewrite—both chosen in the construction pipeline and shape the gold labels.
axioms (4)
  • domain assumption Affective Events Theory: emotions are dynamic reactions to consecutive precipitating events and accumulate over time.
    §3.1 uses AET to justify event-aligned segmentation and stream-of-affect generation rather than fixed windows or static clip labels.
  • domain assumption The VCE 27-category taxonomy adequately covers fine-grained viewer reactions for this benchmark.
    §3.2 and Emotion Taxonomy section adopt VCE categories as the label space for all classification and reward terms.
  • ad hoc to paper Committee MLLM judges (InternVL3.5, Qwen3-omni, later GPT-4o) can score visual grounding, causal logic, viewer-centricity, temporal consistency, and answer consistency reliably enough to define gold data and metrics.
    §3.4 Dual-Consistency Verification and §5.1 Evaluation Metrics make model judges load-bearing for both dataset quality and reported reasoning scores.
  • standard math Standard MLLM fine-tuning and GRPO math (group-relative advantages, KL penalty) are valid optimization tools for structured temporal outputs.
    §4.2 imports GRPO equations from DeepSeek-R1-style RL without claiming a new optimizer; validity is assumed from prior RLHF literature.
invented entities (2)
  • DAR benchmark / Dynamic Affective Reasoning task suite no independent evidence
    purpose: Define and evaluate affective segmentation, 27-way viewer emotion classification, and causal affective reasoning over consecutive video events.
    The paper's primary contribution object; independent evidence is limited to internal human audits and one OOD TSL localization transfer, not external multi-lab replications.
  • DAR-R1 two-stage SFT+GRPO training recipe with multi-dimensional affect rewards no independent evidence
    purpose: Produce a strong baseline that improves boundary precision and reasoning quality on DAR.
    A training procedure, not a physical entity; its value is empirical on DAR metrics and depends on the reward design and synthetic labels.

pith-pipeline@v1.1.0-grok45 · 24108 in / 4038 out tokens · 45771 ms · 2026-07-14T13:18:14.398195+00:00 · methodology

0 comments
read the original abstract

Video emotion analysis is typically framed as a static classification problem, treating each clip as an independent labeled unit. However, such a formulation overlooks a key psychological fact: emotions change as a result of cumulative reactions to consecutive causal events. To bridge this gap, we introduce Dynamic Affective Reasoning, the first large-scale benchmark for viewer-centric affect transitions and causal reasoning over consecutive video events. DAR contains 15,087 videos and 36,908 event-aligned affective segments annotated with 27 emotion categories. Unlike existing video-based emotion datasets, DAR presents a new viewer-centric perspective on fine-grained emotional expressions and transitions, and provides dense, temporally grounded, and causally explicit reasoning chains. Based on DAR, we formally define three challenging tasks: affective segmentation, fine-grained emotion classification, and affective reasoning. Complementing this benchmark, we propose DAR-R1, a two-stage framework that combines supervised fine-tuning with Group Relative Policy Optimization. Experiments across 10+ MLLMs show that DAR-R1 sets a new state-of-the-art for dynamic affective reasoning, in terms of both emotional localization and affective reasoning. Project page: https://github.com/Zhang-Zhiyan/DAR.

Figures

Figures reproduced from arXiv: 2607.10238 by Jingyang Jia, Jinpeng Hu, Peipei Song, Xiaojun Chang, Xun Yang, Zhiyan Zhang.

Figure 1
Figure 1. Figure 1: Overview of the DAR Benchmark Tasks. We formulate three hierarchical tasks: (1) Affective Segmentation for locating temporal boundaries of emotion shifts; (2) Fine-grained Emotion Classification utilizing a 27-category viewer-centric taxon￾omy; and (3) Affective Reasoning for generating causal explanations grounded in visual evidence. 9, 46, 47] have further advanced video-based emotion understanding and r… view at source ↗
Figure 2
Figure 2. Figure 2: Data Construction Pipeline of DAR. The process consists of three stages: (1) Coarse-to-fine event segmentation combining semantic proposals with visual cuts; (2) Incremental differential captioning to reduce redundancy; and (3) Stream-of-affect reasoning generation grounded in AET. A dual-consistency verification protocol ensures annotation quality. affective reasoning. In this section, we first outline th… view at source ↗
Figure 3
Figure 3. Figure 3: Statistical Analysis of the DAR Dataset. (a) Video and segment duration distributions. (b) Segment counts by video duration. (c) Emotion transition matrix. (d) Distribution of 27 emotion categories. (e) Reasoning-text lengths. (f) High-frequency words in reasoning annotations. of the occurrences. This indicates a balanced emotional coverage within the dataset, where emotions such as Amusement, Interest, an… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative Comparison on DAR. We visualize the temporal alignment, predicted emotion labels, and reasoning text. Green highlights indicate correct rea￾soning, while red indicates hallucinations or errors. DAR-R1 demonstrates superior grounding capabilities compared to baselines. our proposed methods demonstrate superior performance. The high scores in Causal Logic and Viewer Centricity particularly verify… view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative inference example of DAR-R1. We provide a qualitative inference example of DAR-R1 in [PITH_FULL_IMAGE:figures/full_fig_p022_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Example of the DAR data construction process [PITH_FULL_IMAGE:figures/full_fig_p025_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 12 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    In: CVPR

    Achlioptas, P., Ovsjanikov, M., Guibas, L.J., Tulyakov, S.: Affection: Learning affective explanations for real-world visual data. In: CVPR. pp. 6641–6651 (2023)

    2023

  3. [3]

    In: CVPR

    Achlioptas,P.,Ovsjanikov,M.,Haydarov,K.,Elhoseiny,M.,Guibas,L.J.:Artemis: Affective language for visual art. In: CVPR. pp. 11569–11579 (2021)

    2021

  4. [4]

    arXiv preprint arXiv:2511.21631 (2025)

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)

    Pith/arXiv arXiv 2025

  5. [5]

    arXiv preprint arXiv:2502.13923 (2025)

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    Pith/arXiv arXiv 2025

  6. [6]

    arXiv preprint arXiv:1803.05434 (2018)

    Barros, P., Churamani, N., Lakomkin, E., Siqueira, H., Sutherland, A., Wermter, S.: The omg-emotion behavior dataset. arXiv preprint arXiv:1803.05434 (2018)

    Pith/arXiv arXiv 2018

  7. [7]

    IEEE transactions on affective computing5(4), 377–390 (2014)

    Cao, H., Cooper, D.G., Keutmann, M.K., Gur, R.C., Nenkova, A., Verma, R.: Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing5(4), 377–390 (2014)

    2014

  8. [8]

    Castellano, B.: PySceneDetect: Video scene cut and transition detection.https: //github.com/Breakthrough/PySceneDetect(2025)

    2025

  9. [9]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 24185–24198 (2024)

    2024

  10. [10]

    Advances in Neural Information Processing Systems37, 110805– 110853 (2024)

    Cheng, Z., Cheng, Z.Q., He, J.Y., Wang, K., Lin, Y., Lian, Z., Peng, X., Haupt- mann, A.: Emotion-llama: Multimodal emotion recognition and reasoning with in- struction tuning. Advances in Neural Information Processing Systems37, 110805– 110853 (2024)

    2024

  11. [11]

    arXiv preprint arXiv:2507.06261 (2025)

    Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al.: Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025)

    Pith/arXiv arXiv 2025

  12. [12]

    Cognition & emotion6(3-4), 169–200 (1992)

    Ekman, P.: An argument for basic emotions. Cognition & emotion6(3-4), 169–200 (1992)

    1992

  13. [13]

    Applied Sciences10(8), 2924 (2020)

    Filippini, C., Perpetuini, D., Cardone, D., Chiarelli, A.M., Merla, A.: Thermal infrared imaging-based affective computing and its application to facilitate human robot interaction: A review. Applied Sciences10(8), 2924 (2020)

    2020

  14. [14]

    arXiv preprint arXiv:2501.12948 (2025)

    Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al.: Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    Pith/arXiv arXiv 2025

  15. [15]

    International Journal of Computer Vision133(9), 6456–6472 (2025)

    Guo, Y., Siddiqui, F., Zhao, Y., Chellappa, R., Lo, S.Y.: Stimuvar: Spatiotemporal stimuli-aware video affective reasoning with multimodal large language models. International Journal of Computer Vision133(9), 6456–6472 (2025)

    2025

  16. [16]

    In: Proceedings of the 33rd ACM International Conference on Multimedia

    Han, Z., Zhu, B., Xu, Y., Song, P., Yang, X.: Benchmarking and bridging emo- tion conflicts for multimodal emotion reasoning. In: Proceedings of the 33rd ACM International Conference on Multimedia. pp. 5528–5537 (2025)

    2025

  17. [17]

    In: Dynamic Affective Reasoning 17 Proceedings of the 33rd ACM International Conference on Multimedia

    Hu, J., Shi, H., Dai, C., Li, Z., Song, P., Wang, M.: Beyond emotion recognition: A multi-turn multimodal emotion understanding and reasoning benchmark. In: Dynamic Affective Reasoning 17 Proceedings of the 33rd ACM International Conference on Multimedia. pp. 5814– 5823 (2025)

    2025

  18. [18]

    arXiv preprint arXiv:2410.21276 (2024)

    Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Os- trow, A., Welihinda, A., Hayes, A., Radford, A., et al.: Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024)

    Pith/arXiv arXiv 2024

  19. [19]

    In: Pro- ceedings of the 28th ACM international conference on multimedia

    Jiang, X., Zong, Y., Zheng, W., Tang, C., Xia, W., Lu, C., Liu, J.: Dfew: A large- scale database for recognizing dynamic facial expressions in the wild. In: Pro- ceedings of the 28th ACM international conference on multimedia. pp. 2881–2889 (2020)

    2020

  20. [20]

    arXiv preprint arXiv:1811.07770 (2018)

    Kollias,D.,Zafeiriou,S.:Aff-wild2:Extendingtheaff-wilddatabaseforaffectrecog- nition. arXiv preprint arXiv:1811.07770 (2018)

    Pith/arXiv arXiv 2018

  21. [21]

    Image and Vision Computing65, 23– 36 (2017)

    Kossaifi, J., Tzimiropoulos, G., Todorovic, S., Pantic, M.: Afew-va database for valence and arousal estimation in-the-wild. Image and Vision Computing65, 23– 36 (2017)

    2017

  22. [22]

    IEEE transactions on pattern analysis and machine intelligence43(3), 1022–1040 (2019)

    Kossaifi, J., Walecki, R., Panagakis, Y., Shen, J., Schmitt, M., Ringeval, F., Han, J., Pandit, V., Toisoul, A., Schuller, B., et al.: Sewa db: A rich database for audio- visual emotion and sentiment research in the wild. IEEE transactions on pattern analysis and machine intelligence43(3), 1022–1040 (2019)

    2019

  23. [23]

    International Conference on Machine Learning (2025)

    Lian, Z., Chen, H., Chen, L., Sun, H., Sun, L., Ren, Y., Cheng, Z., Liu, B., Liu, R., Peng, X., et al.: Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models. International Conference on Machine Learning (2025)

    2025

  24. [24]

    In: Proceedings of the 31st ACM International Conference on Multimedia

    Lian, Z., Sun, H., Sun, L., Chen, K., Xu, M., Wang, K., Xu, K., He, Y., Li, Y., Zhao, J., et al.: Mer 2023: Multi-label learning, modality robustness, and semi- supervised learning. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 9610–9614 (2023)

    2023

  25. [25]

    In: Proceedings of the 2024 conference on empirical methods in natural language processing

    Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., Yuan, L.: Video-llava: Learning united visual representation by alignment before projection. In: Proceedings of the 2024 conference on empirical methods in natural language processing. pp. 5971– 5984 (2024)

    2024

  26. [26]

    In: European conference on computer vision

    Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In: European conference on computer vision. pp. 38–55. Springer (2024)

    2024

  27. [27]

    Liu, S., Zheng, C., Demasi, O., Sabour, S., Li, Y., Yu, Z., Jiang, Y., Huang, M.: Towards emotional support dialog systems. In: Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: Long papers). pp. 3469– 3483 (2021)

    2021

  28. [28]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Maaz, M., Rasheed, H., Khan, S., Khan, F.: Video-chatgpt: Towards detailed video understanding via large vision and language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 12585–12602 (2024)

    2024

  29. [29]

    Advances in Neural Information Processing Systems35, 18571– 18585 (2022)

    Mazeika, M., Tang, E., Zou, A., Basart, S., Chan, J.S., Song, D., Forsyth, D., Stein- hardt, J., Hendrycks, D.: How would the viewer feel? estimating wellbeing from video scenarios. Advances in Neural Information Processing Systems35, 18571– 18585 (2022)

    2022

  30. [30]

    In: CVPR

    Mohamed, Y., Khan, F.F., Haydarov, K., Elhoseiny, M.: It is okay to not be okay: Overcoming emotional bias in affective image captioning by contrastive data col- lection. In: CVPR. pp. 21231–21240 (2022) 18 Z. Zhang et al

    2022

  31. [31]

    In: Proceedings of the 13th annual ACM international conference on Multimedia

    Pantic,M.,Sebe,N.,Cohn,J.F.,Huang,T.:Affectivemultimodalhuman-computer interaction. In: Proceedings of the 13th annual ACM international conference on Multimedia. pp. 669–676 (2005)

    2005

  32. [32]

    In: Proceedings of the 57th Conference of the Association for Computational Linguis- tics

    Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., Mihalcea, R.: Meld: A multimodal multi-party dataset for emotion recognition in conversations. In: Proceedings of the 57th Conference of the Association for Computational Linguis- tics. pp. 527–536 (2019)

    2019

  33. [33]

    In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG)

    Ringeval, F., Sonderegger, A., Sauer, J., Lalanne, D.: Introducing the recola multi- modal corpus of remote collaborative and affective interactions. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG). pp. 1–8. IEEE (2013)

    2013

  34. [34]

    arXiv preprint arXiv:1707.06347 (2017)

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

    Pith/arXiv arXiv 2017

  35. [35]

    IEEE Transactions on Image Processing33, 1122–1135 (2024)

    Song, P., Guo, D., Yang, X., Tang, S., Wang, M.: Emotional video captioning with vision-based emotion interpretation network. IEEE Transactions on Image Processing33, 1122–1135 (2024)

    2024

  36. [36]

    In: Proceedings of the 1st Workshop on Taming Large Language Models: Controllability in the era of Interactive Assistants! pp

    Su, Y., Lan, T., Li, H., Xu, J., Wang, Y., Cai, D.: Pandagpt: One model to instruction-follow them all. In: Proceedings of the 1st Workshop on Taming Large Language Models: Controllability in the era of Interactive Assistants! pp. 11–23 (2023)

    2023

  37. [37]

    IEEE Transactions on Knowledge and Data Engineering (2025)

    Wang, S., Yuan, F., Wang, K., Yang, X., Zhang, X., Wang, M.: Dual-state per- sonalized knowledge tracing with emotional incorporation. IEEE Transactions on Knowledge and Data Engineering (2025)

    2025

  38. [38]

    5: Advancing open-source multimodal models in versatility, reasoning, and efficiency

    Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., et al.: Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265 (2025)

    Pith/arXiv arXiv 2025

  39. [39]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Wang, Y., Sun, Y., Huang, Y., Liu, Z., Gao, S., Zhang, W., Ge, W., Zhang, W.: Ferv39k: A large-scale multi-scene dataset for facial expression recognition in videos. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 20922–20931 (2022)

    2022

  40. [40]

    Research in organizational behavior18(1), 1–74 (1996)

    Weiss, H.M., Cropanzano, R.: Affective events theory. Research in organizational behavior18(1), 1–74 (1996)

    1996

  41. [41]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Xie, H., Peng, C.J., Tseng, Y.W., Chen, H.J., Hsu, C.F., Shuai, H.H., Cheng, W.H.: Emovit: Revolutionizing emotion insights with visual instruction tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 26596–26605 (2024)

    2024

  42. [42]

    arXiv preprint arXiv:2509.17765 (2025)

    Xu, J., Guo, Z., Hu, H., Chu, Y., Wang, X., He, J., Wang, Y., Shi, X., He, T., Zhu, X., et al.: Qwen3-omni technical report. arXiv preprint arXiv:2509.17765 (2025)

    Pith/arXiv arXiv 2025

  43. [43]

    In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing

    Xu, Y., Hu, J., Zhao, Z., Duan, Z., Sun, X., Yang, X.: Multiagentesc: A llm- based multi-agent collaboration framework for emotional support conversation. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. pp. 4665–4681 (2025)

    2025

  44. [44]

    IEEE Transactions on Computational Social Systems (2025)

    Xu, Y., Zhao, Z., Sun, X., Yang, X.: Prompt learning with multiperspective cues for emotional support conversation systems. IEEE Transactions on Computational Social Systems (2025)

    2025

  45. [45]

    Com- puters & education142, 103649 (2019)

    Yadegaridehkordi, E., Noor, N.F.B.M., Ayub, M.N.B., Affal, H.B., Hussin, N.B.: Affective computing in education: A systematic review and future research. Com- puters & education142, 103649 (2019)

    2019

  46. [46]

    arXiv preprint arXiv:2505.09388 (2025) Dynamic Affective Reasoning 19

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025) Dynamic Affective Reasoning 19

    Pith/arXiv arXiv 2025

  47. [47]

    IEEE Transactions on Pattern Anal- ysis and Machine Intelligence (2026)

    Yang, X., Pan, H., Wang, X., Cao, Y., Li, J., Wang, M.: Fine: Fine-grained neuron- level model editing for reliable and safe llms. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence (2026)

    2026

  48. [48]

    Yannakakis, G.N.: Enhancing health care via affective computing (2018)

    2018

  49. [49]

    In: Pro- ceedings of the 33rd ACM International Conference on Multimedia

    Ye, C., Chen, W., Song, P., Liu, X., Zhang, L., Mao, Z.: Multi-round mutual emotion-cause pair extraction for emotion-attributed video captioning. In: Pro- ceedings of the 33rd ACM International Conference on Multimedia. pp. 3320–3329 (2025)

    2025

  50. [50]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zadeh, A., Chan, M., Liang, P.P., Tong, E., Morency, L.P.: Social-iq: A ques- tion answering benchmark for artificial social intelligence. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8807– 8817 (2019)

    2019

  51. [51]

    International Confer- ence on Learning Representations (2025)

    Zhang, F., Cheng, Z., Deng, C., Li, H., Lian, Z., Chen, Q., Liu, H., Wang, W., Zhang, Y.F., Zhang, R., et al.: Mme-emotion: A holistic evaluation benchmark for emotional intelligence in multimodal large language models. International Confer- ence on Learning Representations (2025)

    2025

  52. [52]

    Advances in Neural Information Processing Systems (2025)

    Zhang, Z., Wang, W., Zhu, Y., Qin, W., Wan, P., Zhang, D., Yang, J.: Videmo: Affective-tree reasoning for emotion-centric video foundation models. Advances in Neural Information Processing Systems (2025)

    2025

  53. [53]

    In: ACM MM (2022)

    Zhang,Z.,Yang,J.:Temporalsentimentlocalization:Listenandlookinuntrimmed videos. In: ACM MM (2022)

    2022

  54. [54]

    start_time

    Zhang, Z., Song, P., Hu, J., Chen, W., Ni, L., Yang, X.: Stimuli-aware emotion adaptor for enhancing llm in affective explanation captioning. In: ICASSP 2026- 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 10662–10666. IEEE (2026) 20 Z. Zhang et al. A Additional Related Work A.1 Dynamic Emotion Understanding Dy...

    2026

  55. [55]

    A target temporal segment: [START_TIME, END_TIME]

  56. [56]

    What emotion would a viewer feel in this segment, and why?

    A question: "What emotion would a viewer feel in this segment, and why?"

  57. [57]

    score": <integer from 0 to 5>, 24 Z. Zhang et al

    A model prediction containing: predicted emotion and reason. Important: - The emotion refers to the VIEWER’s induced emotion, not the emotion expressed by people in the video. - The reason should be grounded in visible evidence. - The answer should focus on the target temporal segment. - If the segment is not the first segment, consider whether the predic...