pith. sign in

arxiv: 2503.09158 · v6 · submitted 2025-03-12 · 💻 cs.CV

FaVChat: Hierarchical Prompt-Query Guided Facial Video Understanding with Data-Efficient GRPO

Fufangchen Zhao , Songbai Tan , Xuerui Qiu , Linrui Xun , Wenhao Jiang , Jinkai Zheng , Hehe Fan , Jian Gao
show 2 more authors
Danfeng Yan Ming Li
This is my paper

Pith reviewed 2026-05-23 00:18 UTC · model grok-4.3

classification 💻 cs.CV
keywords facial video understandingvideo large language modelsprompt-guided feature extractionhierarchical visual featuresdata-efficient reinforcement learningfacial cues reasoningzero-shot evaluation
0
0 comments X

The pith

FaVChat extracts question-relevant facial features at three levels and uses efficient reinforcement learning to improve video large language models on subtle facial reasoning tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing video large language models extract visual features without reference to the user's question, which discards details needed for facial understanding. FaVChat counters this by building a hierarchical extraction process that processes the visual input at three complementary levels, each conditioned on the prompt, then fuses the results dynamically before passing them to the language model. The system also introduces Data-Efficient GRPO, which estimates the utility of each training example and focuses learning on the most informative ones under limited data. The authors release a 170K-question benchmark of facial videos and report stronger zero-shot results than prior models on four facial understanding tasks.

Core claim

The paper claims that a hierarchical prompt-guided visual feature extractor operating at three levels, combined with dynamic fusion and Data-Efficient GRPO, produces more accurate reasoning about fine-grained and dynamic facial cues than prompt-agnostic encoders in existing video large language models.

What carries the argument

The hierarchical prompt-guided visual feature extraction framework that processes input at three complementary levels and dynamically fuses the resulting multi-level features for injection into the LLM.

If this is right

  • Video large language models can be made sensitive to fine-grained facial dynamics without requiring task-specific retraining of the visual encoder.
  • Reinforcement learning under data scarcity becomes more sample-efficient when utility is estimated per instance rather than uniformly.
  • A single model architecture can handle multiple facial understanding tasks by conditioning feature extraction on the query at inference time.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same hierarchical conditioning approach could be tested on other domains that require attention to subtle visual changes, such as medical imaging or industrial inspection videos.
  • If the utility estimation in Data-Efficient GRPO proves stable, it offers a general route to reduce annotation budgets when adapting large models to narrow visual domains.
  • The released 170K benchmark may serve as a testbed for measuring whether future models lose facial detail when scaled to longer videos or more open-ended questions.

Load-bearing premise

The multi-level prompt-guided features and fusion step reliably surface task-critical facial cues without discarding useful information or adding new biases.

What would settle it

An ablation that removes the prompt conditioning at one or more of the three feature levels and measures whether zero-shot accuracy on facial tasks falls compared with the full model.

Figures

Figures reproduced from arXiv: 2503.09158 by Danfeng Yan, Fufangchen Zhao, Hehe Fan, Jian Gao, Jinkai Zheng, Linrui Xun, Ming Li, Songbai Tan, Wenhao Jiang, Xuerui Qiu.

Figure 1
Figure 1. Figure 1: (a) The illustration of the proposed FaVChat for fine-grained facial video understanding. For input videos centered on human faces, FaVChat analyzes their fine-grained features based on the given prompts and provides fine-grained responses by integrating the analysis results with the posed questions. However, in the end-to-end user experience, the analysis results on the left side are not visible. (b) The … view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the proposed FaVChat framework. FaVChat augments the original visual encoder with an additional facial encoder (Narayan et al., 2024) and incorporates a multi-level prompt-guided feature extraction mechanism, comprising: (i) low-level prompt-query learning for progressive integration of Transformer features, (ii) mid-level prompt-query learning to support learnable queries, and (iii) high-level… view at source ↗
Figure 3
Figure 3. Figure 3: Concept diagrams of GRPO and DE-GRPO Facial-Specific Fine-Grained Reward. Instead of relying solely on relative preference reward, we design a structured reward that explicitly evaluates fine-grained facial semantics, providing precise optimization signals than preference-only supervision. Given a generated response yi , the reward is defined as R(yi) = X j∈{attr,emo,act} αi · Sim yi, y ∗ i,j )  , (7) whe… view at source ↗
Figure 4
Figure 4. Figure 4: An illustration of our proposed pre-training paradigm. 2022), HMDB51 (Kuehne et al., 2011), and Youtube Faces (Wolf et al., 2011a), covering diverse identities, expressions, and actions. CelebV-HQ contains 35,666 clips across 15,653 subjects with 83 manually labeled facial attributes, while 25,341 additional videos from the other datasets provide varied facial motions. To compensate for missing textual ann… view at source ↗
Figure 5
Figure 5. Figure 5: Comparison results between FaVChat-170K and other mainstream face video datasets, our FaVChat contains the largest number of videos and the richest set of attribute categories. To the best of our knowledge, it is also the first face video dataset designed for VQA training [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative comparison of facial video question-answering capabilities among FaVChat and other high-performance VLLMs. Given the same user prompt, FaVChat provides the most detailed and fine-grained description, capturing intricate facial attributes, expressions, and dynamic movements, while other models focus more on general appearance and background details. This highlights FaVChat’s superior ability in … view at source ↗
Figure 8
Figure 8. Figure 8: Influene of different inference frames length. mechanism, the reward exhibits little to no improvement after the first round. These findings demonstrate the sub￾stantial contribution of our proposed DE-GRPO framework, which incorporates the data recurrent mechanism. Influence of Inference Frame Length. We evaluate the effect of varying input frame counts during inference on both FaVChat and Qwen2.5-VL-7B (… view at source ↗
Figure 9
Figure 9. Figure 9: Training Data Creation Process. et al., 2019; 2016). However, common multi-task models are only capable of performing tasks that are highly correlated. This dilemma has been alleviated with the development of transformers. Face perception models based on transformers, such as FaceXFormer (Narayan et al., 2024), Q-Face (Sun et al., 2024b), Faceptor (Qin et al., 2024), and SwinFace (Qin et al., 2023), have b… view at source ↗
Figure 10
Figure 10. Figure 10: Distribution of the FaVChat dataset. For clarity, low-frequency attributes are grouped into an ”Other” category [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
read the original abstract

Existing video large language models (VLLMs) primarily leverage prompt agnostic visual encoders, which extract untargeted facial representations without awareness of the queried information, leading to the loss of task critical cues. To address this challenge, we propose FaVChat, the first VLLM designed for reasoning over subtle visual and dynamic facial cues. FaVChat introduces a hierarchical, prompt guided visual feature extraction framework that emphasizes question relevant information at three complementary levels. These multi level features are dynamically fused and injected into the LLM, enabling more accurate facial details reasoning To further improve learning efficiency under data scarcity, we propose Data Efficient GRPO, a reinforcement learning strategy that iteratively identifies high utility samples and maximizes the contribution of each instance via per instance utility estimation, substantially enhancing performance gains under limited supervision. We construct a large scale benchmark dataset FaVChat 170K, comprising approximately 60K high quality facial videos and 170K question answer pairs focusing on fine grained facial details. Extensive experiments, including zero shot evaluations on four facial understanding tasks, demonstrate that FaVChat consistently outperforms existing VLLMs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces FaVChat, a VLLM for facial video understanding that replaces prompt-agnostic visual encoders with a hierarchical three-level prompt-guided feature extraction framework whose outputs are dynamically fused before injection into the LLM. It further proposes Data-Efficient GRPO, an RL strategy that estimates per-instance utility to focus training on high-value samples under limited supervision. The authors release the FaVChat-170K dataset (≈60K videos, 170K QA pairs) and claim that the resulting model outperforms prior VLLMs in zero-shot evaluation on four facial-understanding tasks.

Significance. If the performance gains are shown to arise specifically from the hierarchical conditioning and utility-aware RL rather than from the new dataset or longer training, the work would supply a concrete mechanism for task-adaptive visual feature selection in VLLMs and a reusable benchmark for fine-grained facial reasoning. The dataset itself constitutes a clear positive contribution.

major comments (2)
  1. [Experiments] The experimental section supplies only aggregate end-to-end claims of outperformance; it reports neither quantitative metrics, baseline tables, nor statistical details for the four tasks. This absence is load-bearing because the central claim is that FaVChat “consistently outperforms existing VLLMs.”
  2. [Method / Experiments] No ablation is presented that removes the three-level prompt-guided encoder and dynamic fusion (or the per-instance utility estimation inside GRPO) while holding the base VLLM, FaVChat-170K data, and training budget fixed. Without these controls it is impossible to attribute gains to the claimed mechanisms rather than to the new QA pairs or implementation details.
minor comments (1)
  1. [Method] Notation for the three prompt levels and the dynamic fusion operator is introduced without an accompanying diagram or explicit equations, making the architecture difficult to reproduce from the text alone.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the major comments point by point below and will revise the manuscript to provide more detailed experimental evidence.

read point-by-point responses

  1. Referee: [Experiments] The experimental section supplies only aggregate end-to-end claims of outperformance; it reports neither quantitative metrics, baseline tables, nor statistical details for the four tasks. This absence is load-bearing because the central claim is that FaVChat “consistently outperforms existing VLLMs.”

    Authors: We acknowledge that the current presentation relies on aggregate claims and agree that detailed per-task quantitative metrics, full baseline tables, and statistical details are needed to support the outperformance assertions. In the revised manuscript we will expand the experimental section with comprehensive tables reporting accuracy, F1, and other metrics for each of the four tasks, including comparisons against prior VLLMs and any available significance testing. revision: yes

  2. Referee: [Method / Experiments] No ablation is presented that removes the three-level prompt-guided encoder and dynamic fusion (or the per-instance utility estimation inside GRPO) while holding the base VLLM, FaVChat-170K data, and training budget fixed. Without these controls it is impossible to attribute gains to the claimed mechanisms rather than to the new QA pairs or implementation details.

    Authors: We agree that controlled ablations are required to isolate the contributions of the hierarchical prompt-guided encoder with dynamic fusion and the per-instance utility estimation in Data-Efficient GRPO. We will add these ablations in the revision, comparing full FaVChat against variants that disable each component while keeping the base VLLM, FaVChat-170K dataset, and training budget identical. revision: yes

Circularity Check

0 steps flagged

No circularity; empirical proposal evaluated on new dataset and benchmarks

full rationale

The paper proposes an architectural framework (hierarchical prompt-guided visual encoder with dynamic fusion) and a training strategy (Data-Efficient GRPO) for facial video VLLMs, then reports zero-shot performance gains on four tasks using a newly constructed FaVChat-170K dataset. No derivation chain, equations, or first-principles results are presented that reduce to fitted parameters, self-referential quantities, or self-citation load-bearing premises. All claims rest on end-to-end empirical comparisons rather than any mathematical reduction or ansatz smuggled via prior self-work. This is the standard non-circular pattern for an applied systems paper introducing a new model and benchmark.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The work is empirical ML research relying on standard deep learning training assumptions and empirical validation; no explicit free parameters, mathematical axioms, or invented entities are described in the abstract.

pith-pipeline@v0.9.0 · 5758 in / 1113 out tokens · 82797 ms · 2026-05-23T00:18:11.604220+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Mind Your Margin and Boundary: Are Your Distilled Datasets Truly Robust?

    cs.CV 2026-05 unverdicted novelty 7.0

    C²R improves robust accuracy in distilled datasets by 2.8% on average by coupling an attack-aware margin-based curriculum with a class-balanced contrastive robustness objective.

Reference graph

Works this paper leans on

43 extracted references · 43 canonical work pages · cited by 1 Pith paper · 15 internal anchors

  1. [1]

    Qwen3-VL Technical Report

    URL https://www. anthropic.com/news/claude-4. Bai, S., Cai, Y ., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y ., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Men, R., Meng,...

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Facial Expression Recognition Based on Complexity Perception Classification Algorithm

    Chang, T., Wen, G., Hu, Y ., and Ma, J. Facial expression recognition based on complexity perception classification algorithm.arXiv preprint arXiv:1803.00185,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs

    Chen, H., Huang, H., Dong, J., Zheng, M., and Shao, D. Finecliper: Multi-modal fine-grained clip for dynamic fa- cial expression recognition with adapters. InProceedings of the 32nd ACM International Conference on Multime- dia, pp. 2301–2310, 2024a. Chen, L., Wei, X., Li, J., Dong, X., Zhang, P., Zang, Y ., Chen, Z., Duan, H., Tang, Z., Yuan, L., et al. S...

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    doi: 10.1016/j.patcog.2024. 110263. URL http://dx.doi.org/10.1016/j. patcog.2024.110263. Deng, H., Zou, D., Ma, R., Luo, H., Cao, Y ., and Kang, Y . Boosting the generalization and reasoning of vision language models with curriculum reinforcement learning. arXiv preprint arXiv:2503.07065, 2025a. Deng, Y ., Bansal, H., Yin, F., Peng, N., Wang, W., and Chan...

    work page doi:10.1016/j.patcog.2024 2024

  5. [5]

    Video-R1: Reinforcing Video Reasoning in MLLMs

    9 Preprint Feng, K., Gong, K., Li, B., Guo, Z., Wang, Y ., Peng, T., Wu, J., Zhang, X., Wang, B., and Yue, X. Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    GPT-4o System Card

    Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    LLaVA-OneVision: Easy Visual Task Transfer

    Li, B., Zhang, Y ., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y ., Liu, Z., et al. Llava- onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024a. Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational confe...

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Video-LLaVA: Learning United Visual Representation by Alignment Before Projection

    Lin, B., Ye, Y ., Zhu, B., Cui, J., Ning, M., Jin, P., and Yuan, L. Video-llava: Learning united visual represen- tation by alignment before projection.arXiv preprint arXiv:2311.10122,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Understanding R1-Zero-Like Training: A Critical Perspective

    Liu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S., and Lin, M. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025a. Liu, Z., Sun, Z., Zang, Y ., Dong, X., Cao, Y ., Duan, H., Lin, D., and Wang, J. Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025b. 10 Preprint Lo...

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Valley: Video assistant with large language model enhanced ability.arXiv preprint arXiv:2306.07207,

    Luo, R., Zhao, Z., Yang, M., Dong, J., Li, D., Lu, P., Wang, T., Hu, L., Qiu, M., and Wei, Z. Valley: Video assis- tant with large language model enhanced ability.arXiv preprint arXiv:2306.07207,

    work page arXiv

  12. [12]

    Vista-llama: Reliable video narrator via equal distance to visual tokens

    Ma, F., Jin, X., Wang, H., Xian, Y ., Feng, J., and Yang, Y . Vista-llama: Reliable video narrator via equal distance to visual tokens.arXiv preprint arXiv:2312.08870,

    work page arXiv

  13. [13]

    Maaz, M., Rasheed, H., Khan, S., and Khan, F. S. Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Narayan, K., VS, V ., Chellappa, R., and Patel, V . M. Facex- former: A unified transformer for facial analysis.arXiv preprint arXiv:2403.12960,

    work page arXiv

  15. [15]

    URL http://dx.doi

    doi: 10.1109/tpami.2017.2781233. URL http://dx.doi. org/10.1109/tpami.2017.2781233. Ren, S., Yao, L., Li, S., Sun, X., and Hou, L. Timechat: A time-sensitive multimodal large language model for long video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14313–14323,

    work page doi:10.1109/tpami.2017.2781233 2017

  16. [16]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    Learning spatial-semantic relationship for fa- cial attribute recognition with limited labeled data

    Shu, Y ., Yan, Y ., Chen, S., Xue, J.-H., Shen, C., and Wang, H. Learning spatial-semantic relationship for fa- cial attribute recognition with limited labeled data. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun

    work page 2021

  18. [18]

    Derf: Decomposed radiance fields,

    doi: 10.1109/ cvpr46437.2021.01174. URL http://dx.doi.org/ 10.1109/cvpr46437.2021.01174. Shumailov, I., Shumaylov, Z., Zhao, Y ., Papernot, N., Ander- son, R., and Gal, Y . Ai models collapse when trained on recursively generated data.Nature, 631(8022):755–759,

    work page doi:10.1109/cvpr46437.2021.01174 2021

  19. [19]

    PandaGPT: One Model To Instruction-Follow Them All

    Su, Y ., Lan, T., Li, H., Xu, J., Wang, Y ., and Cai, D. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    Face- mllm: A large face perception model.arXiv preprint arXiv:2410.20717, 2024a

    Sun, H., He, M., Lian, T., Han, H., and Shan, S. Face- mllm: A large face perception model.arXiv preprint arXiv:2410.20717, 2024a. Sun, H., He, M., Shan, S., Han, H., and Chen, X. Task- adaptive q-face.arXiv preprint arXiv:2405.09059, 2024b. Sun, L., Lian, Z., Liu, B., and Tao, J. Mae-dfer: Efficient masked autoencoder for self-supervised dynamic facial e...

    work page arXiv 2020

  21. [21]

    doi: 10.1109/tpami.2020. 3046323. URL http://dx.doi.org/10.1109/ tpami.2020.3046323. Wang, J., Yuan, L., Zhang, Y ., and Sun, H. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00634, 2024a. Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: E...

    work page doi:10.1109/tpami.2020 2020

  22. [22]

    S., Guo, A., Oler, E., Wang, F., Anjum, A., Peters, H., Dizon, R., Sayeeda, Z., Tian, S., Lee, B

    Wishart, D. S., Guo, A., Oler, E., Wang, F., Anjum, A., Peters, H., Dizon, R., Sayeeda, Z., Tian, S., Lee, B. L., et al. Hmdb 5.0: the human metabolome database for 2022.Nucleic acids research, 50(D1):D622–D631,

    work page 2022

  23. [23]

    Face recognition in unconstrained videos with matched background similar- ity

    Wolf, L., Hassner, T., and Maoz, I. Face recognition in unconstrained videos with matched background similar- ity. InThe 24th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011, Colorado Springs, CO, USA, 20-25 June 2011, pp. 529–534. IEEE Com- puter Society, 2011a. doi: 10.1109/CVPR.2011.5995566. URL https://doi.org/10.1109/CVPR.2011. ...

    work page doi:10.1109/cvpr.2011.5995566 2011

  24. [24]

    Qwen2 Technical Report

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai...

    work page internal anchor Pith review Pith/arXiv arXiv

  25. [25]

    Qwen3 Technical Report

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025a. Yang, Q., Bai, D., Peng, Y .-X., and Wei, X. Omni-emotion: Extending video mllm with detailed face and audio mod- eling for multimodal emotion analysis.arXiv preprint arXiv:2501.09502, 2025b. ...

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [26]

    Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

    Zhang, H., Li, X., and Bing, L. Video-llama: An instruction- tuned audio-visual language model for video understand- ing.arXiv preprint arXiv:2306.02858, 2023a. Zhang, J., Huang, J., Yao, H., Liu, S., Zhang, X., Lu, S., and Tao, D. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization.arXiv prepri...

    work page internal anchor Pith review Pith/arXiv arXiv

  27. [27]

    doi: 10.1109/lsp.2016. 2603342. URL http://dx.doi.org/10.1109/ lsp.2016.2603342. Zhang, X., Li, M., Lin, S., Xu, H., and Xiao, G. Transformer- based multimodal emotional perception for dynamic fa- cial expression recognition in the wild.IEEE Transac- tions on Circuits and Systems for Video Technology, 34 (5):3192–3203, 2023b. 12 Preprint Zhang, Y ., Li, B...

    work page doi:10.1109/lsp.2016 2016

  28. [28]

    Fa- cial dynamics in video: Instruction tuning for improved fa- cial expression perception and contextual awareness

    Zhao, J., Sun, B., Chen, X., and Wei, X. Facial dynamics in video: Instruction tuning for improved facial expres- sion perception and contextual awareness.arXiv preprint arXiv:2501.07978, 2025a. Zhao, J., Wei, X., and Bo, L. R1-omni: Explainable omni-multimodal emotion recognition with reinforce- ment learning.arXiv preprint arXiv:2503.05379, 2025b. Zhao,...

    work page arXiv

  29. [29]

    General facial representation learning in a visual-linguistic manner

    Zheng, Y ., Yang, H., Zhang, T., Bao, J., Chen, D., Huang, Y ., Yuan, L., Chen, D., Zeng, M., and Wen, F. General facial representation learning in a visual-linguistic manner. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun

    work page 2022

  30. [30]

    A ConvNet for the 2020s

    doi: 10.1109/ cvpr52688.2022.01814. URL http://dx.doi.org/ 10.1109/cvpr52688.2022.01814. Zhu, B., Lin, B., Ning, M., Yan, Y ., Cui, J., Wang, H., Pang, Y ., Jiang, W., Zhang, J., Li, Z., et al. Language- bind: Extending video-language pretraining to n-modality by language-based semantic alignment.arXiv preprint arXiv:2310.01852, 2023a. Zhu, D., Chen, J., ...

    work page doi:10.1109/cvpr52688.2022.01814 2022

  31. [31]

    Related Works The rapid advancement of multimodal large language models (MLLMs) has spurred the emergence of Video-MLLMs

    13 Preprint A. Related Works The rapid advancement of multimodal large language models (MLLMs) has spurred the emergence of Video-MLLMs. To mitigate the dramatic increase in visual token count caused by the temporal dimension, existing approaches typically either insert projectors such as MLPs and Q-Formers (Li et al., 2023a) after the visual encoder (Maa...

    work page 2023

  32. [32]

    However, these strategies often sacrifice fine-grained features particularly in domains with rich and intricate details such as human faces

    or train a dedicated video encoder from scratch (Maaz et al., 2023; Luo et al., 2023; Ren et al., 2024; Wang et al., 2024c). However, these strategies often sacrifice fine-grained features particularly in domains with rich and intricate details such as human faces. While significant progress has been achieved in fine-grained facial understanding for stati...

    work page 2023

  33. [33]

    Below is a more comprehensive discussion of related work

    offers a promising avenue to enhance such understanding; however, current label-free preference-based RL methods struggle to supervise fine-grained descriptions (Wang et al., 2025; Zhao et al., 2025b; Feng et al., 2025), while obtaining fine-grained annotations is prohibitively labor-intensive, which is a dual challenge that FaVChat seeks to overcome. Bel...

    work page 2025

  34. [34]

    directly applies a pre-trained Q-Former to video frames, resulting in more compact representations. On the other hand, since videos can be decomposed into sequences of images, the majority of current research still employs image encoders to extract features from video frames as video representations. For instance, VideoChat (Li et al., 2023b), Video-ChatG...

    work page 2023

  35. [35]

    However, treating videos merely as sequences of images overlooks the temporal characteristics of videos

    to process videos, while LLaMA-VID (Li et al., 2024c), TimeChat (Ren et al., 2024), and Emu3 (Wang et al., 2024c) opt for Eva-clip ViT (Sun et al., 2023b) as their visual encoder. However, treating videos merely as sequences of images overlooks the temporal characteristics of videos. Consequently, subsequent researchers advocate employing pre-trained vide...

    work page 2024

  36. [36]

    Hair": "red, curly

    have been extensively studied by the academic community. In addition, subsequent researchers have attempted to build multi-task models to handle multiple tasks using a single general model (Zhang et al., 2016; Ranjan 14 Preprint & Facial Proportion 60% others CelebV-HQ HMDB51 FERV39K YouTube Faces Stage0 Fiacial Video Filtering Sampling (Fine-grained) Fea...

    work page 2016

  37. [37]

    as an additional visual encoder, focusing on extracting visual facial features. However, despite the significant progress made in the image domain, there is still a lack of research on high-performance fine-grained video face understanding frameworks when extending the task of fine-grained face understanding to the video domain. This is the focus of the c...

    work page 2025

  38. [38]

    face-centric

    enhanced spatial reasoning in videos by integrating spatiotemporal sequential structure into GRPO. Building upon this, STAR-R1 (Qi et al.) introduced a tailored spatiotemporal reward mechanism to further improve the model’s reasoning capabilities in dynamic, long-duration scenarios. In this work, we propose DE-GRPO, a novel reinforcement learning framewor...

    work page 2001

  39. [39]

    Detailed descriptions are given in the following subsections. B.1. Filtering Facial Videos from Existing datasets Our video raw data consists of four parts: CelebV-HQ (Zhu et al., 2022), HMDB51 (Kuehne et al., 2011), FERV39K (Wang et al.,

    work page 2022

  40. [40]

    Specifically, given the detailed video attributes of CelebV-HQ, we have incorporated all 35,666 video from CelebV-HQ

    and YouTube Faces (Wolf et al., 2011a). Specifically, given the detailed video attributes of CelebV-HQ, we have incorporated all 35,666 video from CelebV-HQ. For other videos, we first conducted face detaction utilizing AntelopeV2 (Ren et al., 2023). If the minimum proportion of human faces in the video excceded 60%, the video was considered to meet 16 Pr...

    work page 2023

  41. [41]

    The raw video frames are down sample by 16 to obtain a shorter frame sequence

    The video textual generation process is further divided into four sub-steps: •Frame Sequence Generation. The raw video frames are down sample by 16 to obtain a shorter frame sequence. • Fine-grained Description Generation. We feed the frame sequence from the preceding step to the trained feature extractors. And the output is subsequently converted into a ...

    work page 2024

  42. [42]

    18 Preprint Table 7.Performance comparison on the DFEC (Zhao et al., 2025a) dataset for textual emotion analysis

    The proposed FaVChat model achieves superior performance across all existing VLLMs. 18 Preprint Table 7.Performance comparison on the DFEC (Zhao et al., 2025a) dataset for textual emotion analysis. The ∗ columns show VideoChatGPT Scores (0–10) evaluated on our internally curated test dataset. ♣ denotes a model trained to process both visual and speech mod...

    work page 2000

  43. [43]

    The YouTube Faces dataset is well suited for this setting, as it contains multiple videos per person under diverse conditions

    - 71.29 93.61 Claude4-Sonnet (Anthropic, 2025)- 73.4196.14 VideoChat (Li et al., 2023b) 7B 54.74 73.61 VideoChat2 (Li et al., 2024b) 7B 57.30 81.32 VideoLLaMa2 (Cheng et al., 2024)7B 51.03 75.05 Qwen2.5-VL-7B-Face (Bai et al., 2025b)7B 64.28 85.58 Qwen2.5-VL-72B (Bai et al., 2025b)72B 66.84 85.87 FaVChat 7B88.38- Face Recognition.Face recognition can be v...

    work page 2025