Pith. sign in

REVIEW 4 major objections 7 minor 65 references

From Coarse to Nuanced: Cross-Modal Alignment of Fine-Grained Linguistic Cues and Visual Salient Regions for Dynamic Emotion Recognition

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GRACE aligns emotion words with facial motion patches via optimal transport and reports state-of-the-art results on three in-the-wild benchmarks.

desk verdict Plausible engineering gains with an under-specified label-guided captioning pipeline; the central SOTA claim is credible but needs protocol and artifact disclosure. read the letter →

arxiv 2507.11892 v1 pith:KO4XAOSK submitted 2025-07-16 cs.CV cs.AIcs.HC

Yu Liu , Leyuan Qu , Hanlei Shi , Di Gao , Yuhua Zheng , Taihao Li This is my paper
classification cs.CVcs.AIcs.HC
keywords dynamicfacialexpressionrecognitioncross-modalalignmentoptimaltransportvision-languagemodelspatiotemporallocalizationemotionmotion-differenceweightingin-the-wildvideo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GRACE is a framework for dynamic facial expression recognition that tries to show that keeping fine-grained structure on both sides of a vision-language alignment—text tokens instead of one pooled caption, and motion-weighted video patches instead of pooled frames—lets a model locate emotionally salient facial dynamics and ignore irrelevant ones. It combines a coarse-to-fine text-enhancement stage, a motion-difference weighting mechanism, and token-level alignment through a mathematical matching procedure called entropy-regularized optimal transport. On DFEW, FERV39k and MAFW, the paper reports state-of-the-art unweighted and weighted average recall, with the largest gains on ambiguous and long-tail emotion classes. The reason to care is that existing vision-language DFER methods flatten captions and video clips into single embeddings, losing the word-to-region correspondences that could ground an emotion prediction.

What carries the argument

The load-bearing mechanism is the entropy-regularized optimal transport alignment between text tokens and spatio-temporal video patches. The model builds a cosine-distance cost matrix between L text-token embeddings and N motion-weighted visual patch embeddings, then solves for a soft transport plan under uniform marginals with an entropy regularization term, using an iterative entropy-regularized solver. The transport plan serves as a soft correspondence map: emotionally descriptive words are matched to the video segments where those actions occur, and patches without low-cost matches receive little weight in the final classification. Two supporting modules feed this alignment: a coarse-to-fine text enhancement that injects emotion-category phrases and classifier-guided top-k labels into the caption while preserving its token structure, and a motion-difference weighting that multiplies visual features by normalized inter-frame difference scores so expressive regions dominate. The paper's ablation tables treat the optimal transport alignment as the foundational component, with the text enhancement and motion weighting adding to it.

What would settle it

Train GRACE on a held-out fold with the original raw captions instead of the refined ones, or with the top-3 labels randomly shuffled before refinement; if UAR does not drop meaningfully, the claimed contribution of the coarse-to-fine text enhancement is not doing the work attributed to it.

Watch

Extended reading notes

Core claim

The central claim is that the flatten-then-align paradigm used by prior vision-language DFER methods is the bottleneck: compressing an emotion description into one embedding and pooling all frames lets emotion-irrelevant motion such as blinks and head turns survive, while discarding fine-grained semantic cues. GRACE instead keeps the caption as a token sequence, enhances it with emotion-descriptor phrases and classifier-provided top-k labels, reweights visual features by inter-frame difference magnitude, and aligns the two sides with entropy-regularized optimal transport, in which the transport plan both fuses modalities and down-weights visual patches that have no cheap textual match. The paper reports 68.94% UAR and 76.25% WAR on DFEW, 49.12% UAR and 54.63% WAR on FERV39k, and 45.09% UAR and 58.25% WAR on MAFW, exceeding prior methods compiled in its comparison table on all three benchmarks. Ablations support that both token granularity and the transport objective matter: switching from sentence-level to fine-grained tokens under optimal transport adds 2.21% UAR on DFEW fold 1, and switching from cosine matching to optimal transport with fine-grained tokens adds 7.32% UAR.

Load-bearing premise

The load-bearing premise is that the generated captions are faithful, emotion-relevant descriptions of the facial dynamics in every video, and that the refinement step guided by the auxiliary classifier's top-3 predictions improves rather than distorts them; the paper reports no caption-quality evaluation, no error analysis, and no pinned version of the refinement model.

Editorial extensions

If this is right

  • Token-level optimal transport can replace sentence-level pooling in vision-language DFER, since the paper reports consistent gains when alignment operates on fine-grained tokens rather than sentence embeddings.
  • Motion-difference weighting is presented as a way to suppress emotion-irrelevant dynamics without handcrafted optical flow or landmark preprocessing, so if correct it can be appended to existing video encoders cheaply.
  • The learned transport plan gives interpretable phrase-to-frame correspondences, meaning emotion predictions can be traced to specific words and video segments rather than to a black-box pooled feature.
  • The reported gains on minority and ambiguous classes imply that granularity-preserving text and alignment can partially compensate for long-tail imbalance in affective datasets.
  • The full-model comparisons on DFEW, FERV39k, and MAFW suggest the framework transfers across seven-class and eleven-class in-the-wild benchmarks and across balanced and imbalanced label distributions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the caption pipeline is the real driver, the method's gains could partly inherit from the auxiliary classifier's top-k labels rather than from genuine visual-to-text grounding; replacing those labels with shuffled or withheld labels would isolate that contribution.
  • Raw inter-frame difference weighting may amplify blinks and head turns as much as expressions, contrary to the paper's stated suppression; measuring motion scores on deliberately neutral videos with frequent blinks would show whether the filter is actually emotion-selective.
  • The same token-level transport machinery could transfer to other fine-grained video-language tasks with transient localized cues, such as micro-expression spotting or action-unit detection, where global pooling is known to wash out short events.
  • The paper does not report caption-quality metrics or prompt details, so the gains may be partly tied to the specific commercial language model used for refinement; pinning the model version and evaluating caption fidelity would make the comparison reproducible.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes GRACE, a dynamic facial expression recognition (DFER) framework that combines three components: a motion-difference visual weighting module, a Coarse-to-fine Affective Text Enhancement (CATE) module that refines VLM-generated captions using an auxiliary classifier's top-3 emotion predictions and GPT-4o mini, and a token-level cross-modal alignment module based on entropy-regularized optimal transport. The authors report new state-of-the-art results on DFEW (68.94% UAR, 76.25% WAR), FERV39k (49.12% UAR, 54.63% WAR), and MAFW (45.09% UAR, 58.25% WAR), with ablations on DFEW fold 1 showing that each module contributes positively. The central claim is that preserving token-level granularity in language and selectively weighting motion-difference features enables interpretable, emotion-grounded matching between text tokens and spatiotemporal patches.

Significance. If the reported results hold, GRACE would be a meaningful advance for DFER: it directly targets two acknowledged weaknesses of prior vision-language DFER methods, namely text-granularity loss and emotion-irrelevant motion, and it offers an interpretable OT-based alignment mechanism. The paper has notable strengths: evaluation on three external in-the-wild benchmarks, a controlled fold-1 ablation design (Tables II-V), and an ablation that disentangles token granularity from alignment strategy (Table III). However, the headline SOTA claim currently rests on a caption-generation pipeline that is underspecified and that injects test-time classifier predictions into the text branch; this raises a fairness concern relative to baselines that do not receive such a signal. The empirical claims also lack error bars or significance tests, and the main-table DFEW numbers are not reconciled with the fold-1 ablation numbers. These issues are load-bearing for the central claim.

major comments (4)
  1. [Section III-B, Table IV] The CATE pipeline uses a baseline emotion recognition model to obtain top-3 predicted categories, converts them to phrases of the form 'an emotion of [class]', and feeds them to GPT-4o mini at both training and test time. Table IV shows that removing this Top-3 label guidance costs 2.15 UAR on DFEW fold 1 (68.42 to 66.27), the largest textual ablation. The manuscript does not identify this auxiliary classifier, state whether it was trained only on each dataset's training folds, provide the prompts used, or report any caption-quality evaluation. Because the refined caption explicitly contains class-name phrases, a strong auxiliary classifier could allow the final model to read the emotion category from the text rather than learn the claimed emotion-grounded visual-textual alignment. The comparison in Table I is therefore not apples-to-apples unless the same test-time label signal is available to the baseline methods. Please identify the auxiliary classifier, verify train/test split purity, and add a controlled experiment that evaluates GRACE without any test-time label-derived phrases, or otherwise address the circularity concern.
  2. [Section V, Tables I-II] Table II reports the full GRACE model on DFEW fold 1 as 68.42% UAR and 76.29% WAR, while Table I reports GRACE*†‡ as 68.94% UAR and 76.25% WAR on DFEW. The text never explains whether Table I reports a 5-fold average or a different protocol, nor does it provide fold-wise results or error bars. Since the paper's central claim is 'new SOTA results on three in-the-wild benchmarks,' the DFEW number must be reproducible and clearly reconciled with the fold-1 ablations. Please state the evaluation protocol for Table I explicitly, report per-fold results, and include variance or significance measures where possible.
  3. [Section III-C, Eq. (2)-(3), Table V] The motion-difference module reweights features by the L2 norm of inter-frame feature differences. The paper claims this 'amplifies emotion-bearing dynamics while suppressing irrelevant motions' and explicitly names eye blinks and head tilts as suppressed noise. However, raw inter-frame differences amplify all motion, including blinks, head turns, and background changes, with no mechanism that distinguishes expressive motion from non-expressive motion. Additionally, Table V reports results for 'Spatial Difference Only' and 'Temporal Difference Only,' but Section III-C defines only temporal inter-frame differencing in Eq. (2); the spatial differencing used in the ablation is never defined. Please provide a formal definition of the spatial and temporal scoring, and add quantitative or qualitative evidence that the weighting actually suppresses the claimed irrelevant dynamics rather than merely amplifying overall motion.
  4. [Section III-B, Section VII] The entire OT alignment and emotion-grounded localization claim depends on the faithfulness of the caption pipeline: VideoLLaVA must describe actual facial dynamics, and GPT-4o mini refinement must improve rather than distort those descriptions. The paper reports no caption-quality evaluation, no error analysis, no prompts, and no pinned GPT-4o mini version. The Limitations section acknowledges dependence on the quality of visual and textual representations but does not assess caption fidelity directly. Please add a caption-quality study (e.g., human evaluation, caption-to-AU agreement, or failure-mode analysis) and state the exact model versions and prompts used, since the method's reproducibility and its interpretability claims rest on this pipeline.
minor comments (7)
  1. [Abstract] The sentence beginning 'We propose GRACE' should be capitalized consistently as 'we propose'.
  2. [Section III-B.b] The phrase 'emotion-descriptor phrase of the form' should read 'emotion-descriptor phrases of the form' for grammatical consistency.
  3. [Section V] In the paragraph before Table III, 'we conduct a ablation' should be 'we conduct an ablation'.
  4. [Eq. (7)] The denominator of the supervised contrastive loss uses the notation '[ [j≠i] ]', which should be written as the indicator 1_{j≠i}; the index range 2N is also unclear given that the numerator sums over N samples.
  5. [Table I] The footnote defining *, †, and ‡ should be placed where the table is first referenced, and the row label 'GRACE*†‡' should be explained in the main text as the full GRACE model to avoid confusion with the full-model row in Table II.
  6. [Section VI] The text refers to 'Fig. V' but the figure is numbered Fig. 5; please correct the reference.
  7. [Fig. 2] The figure caption mentions 'entropy-controlled prompt buckets,' but this mechanism is not defined in the body of the paper; please either define it or remove the term from the caption.

Circularity Check

1 steps flagged · score 6.0 of 10

CATE injects an auxiliary classifier's top-3 emotion predictions into test-time text, so part of the SOTA gain is a pseudo-label readout rather than an independent test of token-level OT alignment.

  1. fitted input called prediction [Section III-B (Coarse-to-fine Affective Text Enhancement), Eq. (1), Table IV]
    "we first apply a baseline emotion recognition model to extract the top-3 predicted emotion categories from sampled frames, offering a coarse but semantically aligned emotional context... These predicted categories are converted into emotion-descriptor phrase of the form 'an emotion of [class]', These emotion-informed prompts are concatenated with the initial caption and passed through a lightweight LLM-based rewriter [54], resulting in a refined, emotion-aware description Dfinal."

    Dfinal is encoded by Et and used in Eq. (1) as y = F(A(M(Ev(V)), P(Et(D)))). Since Dfinal literally contains 'an emotion of [class]' where [class] is produced by a baseline emotion classifier on the same video, the final classifier can recover an emotion prediction by reading the injected class token, without relying on the claimed motion-difference filtering or OT token-patch alignment. Table IV confirms this is load-bearing: removing 'Top-3 labels guidance' costs 2.15 UAR on DFEW fold 1 (68.42 to 66.27), the largest single textual ablation. Thus part of the reported SOTA improvement is an auxiliary emotion predictor's output renamed as 'semantic guidance', and comparisons to baselines without this test-time signal are not apples-to-apples.

full rationale

The central SOTA claim is evaluated on held-out DFEW/FERV39k/MAFW folds, so there is genuine external grounding and no evidence of ground-truth leakage into training. I found no load-bearing self-citation chain: the same-author citations (e.g., DK-CLIP [25]) appear only in related work and do not justify a core design choice. The substantial circular element is the CATE pipeline: a baseline classifier's top-3 predictions are inserted verbatim into the test-time text, and the final classifier can exploit those class-name tokens. This makes part of the 'prediction' a function of another emotion predictor's output rather than of the learned visual-textual alignment; the paper's own ablation shows that this injected signal is the largest textual contributor. The motion-difference mechanism's claim to suppress blinks and head turns is a correctness concern (raw L2 temporal differencing does not distinguish expressive from non-expressive motion), not a circularity, so it is not scored here. Overall: partial circularity in the test-time text-construction step, but the remaining components and held-out benchmarks give the paper some independent content.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

GRACE introduces no physical or ontological entities; it composes existing techniques (VideoMAE, VideoCLIP, VideoLLaVA captions, GPT-4o mini refinement, Sinkhorn optimal transport, inter-frame differencing). The main costs are concentrated in tuned-but-unreported hyperparameters and in two domain assumptions: that the generated captions are faithful for every video, and that motion magnitude tracks emotional expressiveness. The auxiliary top-3 label guidance acts as a soft supervisory channel that blurs attribution of the reported gains, since removing it costs 2.15 UAR points by the paper's own ablation.

free parameters (6)
  • loss balancing weights lambda_1, lambda_2, lambda_3 = not reported
    Section III-E states the total loss is a weighted combination and that the lambdas are 'carefully tuned', but their values are never given, so the reported results cannot be reproduced or checked for sensitivity.
  • focal loss parameters (gamma, alpha_t, class weights w_c) = not reported
    Equation (6) defines the primary loss with gamma and alpha_t, and class-specific weights are said to address imbalance, but no values are reported.
  • entropy regularization coefficient for Sinkhorn OT = not reported
    Equation (5) includes the entropy regularization term lambda*H(T); the value of lambda is never stated, and it directly controls the softness of the token-to-patch alignment.
  • contrastive temperature tau = not reported
    Equation (7) uses a temperature tau in the supervised contrastive loss; the value is not reported.
  • fixed proportion of top-ranked tokens retained for alignment = not reported
    Section VII admits the method 'selects salient features for alignment by retaining a fixed proportion of top-ranked tokens based on saliency scores', but the proportion is never given.
  • mixup augmentation strength for supervised contrastive loss = not reported
    Section III-E says contrastive learning uses mixup augmentation, but the mixup parameter is not reported.
assumptions (5)
  • ad hoc to paper Inter-frame feature difference magnitude correlates with emotional expressiveness, so motion weighting amplifies expression and suppresses irrelevant motion
    Section III-C asserts 'regions with strong inter-frame variation are more likely to correspond to expression-related facial movements'. Frame differencing amplifies all motion, including blinks and head turns, so this assumption is doing real work and is not validated independently.
  • domain assumption The VLM-generated captions and GPT-4o mini refinements are semantically faithful for every video
    Section III-B relies on VideoLLaVA descriptions refined by GPT-4o mini with top-3 label guidance; no caption-quality study or error analysis is provided, and the proprietary model version and prompts are undisclosed.
  • domain assumption Top-3 predictions from an auxiliary classifier provide useful emotion context without leaking the test label
    Section III-B uses a baseline classifier's top-3 emotion predictions to condition the text at both train and test time; the framework's gain partly inherits this auxiliary signal, and the classifier's identity is never specified.
  • domain assumption Pretrained VideoMAE (VoxCeleb2) and VideoCLIP (HowTo100M) features are sufficient backbones for cross-modal alignment
    Section IV-A3 adopts these pretrained components; the paper's own Limitations section notes performance is bounded by backbone representational capacity.
  • standard math Sinkhorn's algorithm converges to a valid approximate optimal transport plan for the reported cost matrices
    Section III-D invokes Cuturi's Sinkhorn algorithm [39]; this is standard numerical machinery, assumed reliable at the reported matrix sizes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Coarse to Nuanced: Cross-Modal Alignment of Fine-Grained Linguistic Cues and Visual Salient Regions for Dynamic Emotion Recognition." pith.science (2026). https://pith.science/paper/KO4XAOSK

@misc{pith2026250711892,
  author       = {Pith},
  title        = {Pith review of: From Coarse to Nuanced: Cross-Modal Alignment of Fine-Grained Linguistic Cues and Visual Salient Regions for Dynamic Emotion Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KO4XAOSK}},
  note         = {Machine review of arXiv:2507.11892}
}
read the original abstract

Dynamic Facial Expression Recognition (DFER) aims to identify human emotions from temporally evolving facial movements and plays a critical role in affective computing. While recent vision-language approaches have introduced semantic textual descriptions to guide expression recognition, existing methods still face two key limitations: they often underutilize the subtle emotional cues embedded in generated text, and they have yet to incorporate sufficiently effective mechanisms for filtering out facial dynamics that are irrelevant to emotional expression. To address these gaps, We propose GRACE, Granular Representation Alignment for Cross-modal Emotion recognition that integrates dynamic motion modeling, semantic text refinement, and token-level cross-modal alignment to facilitate the precise localization of emotionally salient spatiotemporal features. Our method constructs emotion-aware textual descriptions via a Coarse-to-fine Affective Text Enhancement (CATE) module and highlights expression-relevant facial motion through a motion-difference weighting mechanism. These refined semantic and visual signals are aligned at the token level using entropy-regularized optimal transport. Experiments on three benchmark datasets demonstrate that our method significantly improves recognition performance, particularly in challenging settings with ambiguous or imbalanced emotion classes, establishing new state-of-the-art (SOTA) results in terms of both UAR and WAR.

Figures

Figures reproduced from arXiv: 2507.11892 by the authors.

Figure 1
Figure 1. Comparison of existing methods and our proposed GRACE. (a) [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Detailed architecture of the proposed GRACE framework. (a) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the CATE. This enhancement mechanism serves three key purposes: (1) it ensures semantic consistency with emotion-specific patterns, [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of textual representations across three stages of our textual [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: t-SNE visualization of learned emotion representations for the Baseline [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Visualization of the GRACE alignment process. Each color-coded phrase in the expression description is aligned with the most relevant video frame [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

65 extracted references · 41 canonical work pages

  1. [1]

    The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression,

    P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, “The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression,” in 2010 ieee computer soci- ety conference on computer vision and pattern recognition-workshops . IEEE, 2010, pp. 94–101

    work page 2010

  2. [2]

    Induced disgust, happiness and surprise: an addition to the mmi facial expression database,

    M. Valstar, M. Pantic et al. , “Induced disgust, happiness and surprise: an addition to the mmi facial expression database,” in Proc. 3rd Intern. Workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect , vol. 10. Paris, France., 2010, pp. 65–70

    work page

  3. [3]

    Facial expression recognition from near-infrared videos,

    G. Zhao, X. Huang, M. Taini, S. Z. Li, and M. Pietik ¨aInen, “Facial expression recognition from near-infrared videos,” Image and vision computing, vol. 29, no. 9, pp. 607–619, 2011

    work page 2011

  4. [4]

    Dfew: A large-scale database for recognizing dynamic facial expressions in the wild,

    X. Jiang, Y . Zong, W. Zheng, C. Tang, W. Xia, C. Lu, and J. Liu, “Dfew: A large-scale database for recognizing dynamic facial expressions in the wild,” in Proceedings of the 28th ACM international conference on multimedia, 2020, pp. 2881–2889

    work page 2020

  5. [5]

    Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild,

    Y . Liu, W. Dai, C. Feng, W. Wang, G. Yin, J. Zeng, and S. Shan, “Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild,” in Proceedings of the 30th ACM international conference on multimedia , 2022, pp. 24–32

    work page 2022

  6. [6]

    Ferv39k: A large-scale multi-scene dataset for facial expres- sion recognition in videos,

    Y . Wang, Y . Sun, Y . Huang, Z. Liu, S. Gao, W. Zhang, W. Ge, and W. Zhang, “Ferv39k: A large-scale multi-scene dataset for facial expres- sion recognition in videos,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 20 922–20 931

    work page 2022

  7. [7]

    Dep-fer: Facial expression recognition in depressed patients based on voluntary facial expression mimicry,

    J. Ye, Y . Yu, Y . Zheng, Y . Liu, and Q. Wang, “Dep-fer: Facial expression recognition in depressed patients based on voluntary facial expression mimicry,” IEEE Transactions on Affective Computing , 2024

    work page 2024

  8. [8]

    Efficient facial expression recognition with representation reinforcement network and transfer self-training for human–machine interaction,

    C.-S. Jiang, Z.-T. Liu, M. Wu, J. She, and W.-H. Cao, “Efficient facial expression recognition with representation reinforcement network and transfer self-training for human–machine interaction,” IEEE Transac- tions on Industrial Informatics , vol. 19, no. 9, pp. 9943–9952, 2023

    work page 2023

Show all 65 references
  1. [9]

    Predicting personal- ized image emotion perceptions in social networks,

    S. Zhao, H. Yao, Y . Gao, G. Ding, and T.-S. Chua, “Predicting personal- ized image emotion perceptions in social networks,” IEEE transactions on affective computing , vol. 9, no. 4, pp. 526–540, 2016

    2016

  2. [10]

    Spatio-temporal convolutional features with nested lstm for facial expression recognition,

    Z. Yu, G. Liu, Q. Liu, and J. Deng, “Spatio-temporal convolutional features with nested lstm for facial expression recognition,” Neurocom- puting, vol. 317, pp. 50–57, 2018

    2018

  3. [11]

    Saanet: Siamese action-units attention network for improving dynamic facial expression recognition,

    D. Liu, X. Ouyang, S. Xu, P. Zhou, K. He, and S. Wen, “Saanet: Siamese action-units attention network for improving dynamic facial expression recognition,” Neurocomputing, vol. 413, pp. 145–157, 2020

    2020

  4. [12]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778

    2016

  5. [13]

    Former-dfer: Dynamic facial expression recog- nition transformer,

    Z. Zhao and Q. Liu, “Former-dfer: Dynamic facial expression recog- nition transformer,” in Proceedings of the 29th ACM international conference on multimedia , 2021, pp. 1553–1561

    2021

  6. [14]

    Ex- pression snippet transformer for robust video-based facial expression recognition,

    Y . Liu, W. Wang, C. Feng, H. Zhang, Z. Chen, and Y . Zhan, “Ex- pression snippet transformer for robust video-based facial expression recognition,” Pattern Recognition, vol. 138, p. 109368, 2023

    2023

  7. [16]

    Freq-hd: An interpretable frequency-based high-dynamics affective clip selection method for in-the-wild facial expression recognition in videos,

    Z. Tao, Y . Wang, Z. Chen, B. Wang, S. Yan, K. Jiang, S. Gao, and W. Zhang, “Freq-hd: An interpretable frequency-based high-dynamics affective clip selection method for in-the-wild facial expression recognition in videos,” in Proceedings of the 31st ACM International Conferenc...

    2023

  8. [17]

    Facial expression recognition with adaptive frame rate based on multiple testing correction,

    A. Savchenko, “Facial expression recognition with adaptive frame rate based on multiple testing correction,” in International Conference on Machine Learning. PMLR, 2023, pp. 30 119–30 129

    2023

  9. [18]

    Video-based facial micro-expression analysis: A survey of datasets, features and algorithms,

    X. Ben, Y . Ren, J. Zhang, S.-J. Wang, K. Kpalma, W. Meng, and Y .-J. Liu, “Video-based facial micro-expression analysis: A survey of datasets, features and algorithms,” IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 9, pp. 5826–5846, 2021. 13

    2021

  10. [19]

    Dynamic facial expression recognition under partial occlusion with optical flow reconstruction,

    D. Poux, B. Allaert, N. Ihaddadene, I. M. Bilasco, C. Djeraba, and M. Bennamoun, “Dynamic facial expression recognition under partial occlusion with optical flow reconstruction,” IEEE Transactions on Image Processing, vol. 31, pp. 446–457, 2021

    2021

  11. [21]

    From static to dynamic: Adapting landmark-aware image models for facial expression recognition in videos,

    Y . Chen, J. Li, S. Shan, M. Wang, and R. Hong, “From static to dynamic: Adapting landmark-aware image models for facial expression recognition in videos,” IEEE Transactions on Affective Computing , pp. 1–15, 2024

    2024

  12. [22]

    A survey on facial expression recognition of static and dynamic emotions,

    Y . Wang, S. Yan, Y . Liu, W. Song, J. Liu, Y . Chang, X. Mai, X. Hu, W. Zhang, and Z. Gan, “A survey on facial expression recognition of static and dynamic emotions,” arXiv preprint arXiv:2408.15777 , 2024

    2024 arXiv

  13. [23]

    Prompting visual-language models for dynamic facial expression recognition,

    Z. Zhao and I. Patras, “Prompting visual-language models for dynamic facial expression recognition,” 2024. [Online]. Available: https://arxiv.org/abs/2308.13382

    2024 arXiv

  14. [24]

    Emoclip: A vision-language method for zero-shot video facial expression recognition,

    N. M. Foteinopoulou and I. Patras, “Emoclip: A vision-language method for zero-shot video facial expression recognition,” in 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG). IEEE, 2024, pp. 1–10

    2024

  15. [25]

    Domain knowledge enhanced vision-language pretrained model for dynamic facial expression recognition,

    L. Li, Y . Zheng, S. Liu, X. Xu, and T. Li, “Domain knowledge enhanced vision-language pretrained model for dynamic facial expression recognition,” in Proceedings of the 32nd ACM International Conference on Multimedia , ser. MM ’24. New York, NY , USA: Association for Computin...

    2024

  16. [26]

    Enhancing zero-shot facial expression recognition by llm knowledge transfer,

    Z. Zhao, Y . Cao, S. Gong, and I. Patras, “Enhancing zero-shot facial expression recognition by llm knowledge transfer,” in 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2025, pp. 815–824

    2025

  17. [28]

    Finecliper: Multi-modal fine-grained clip for dynamic facial expression recognition with adapters,

    H. Chen, H. Huang, J. Dong, M. Zheng, and D. Shao, “Finecliper: Multi-modal fine-grained clip for dynamic facial expression recognition with adapters,” in Proceedings of the 32nd ACM International Conference on Multimedia , ser. MM ’24. New York, NY , USA: Association for Comp...

    2024

  18. [30]

    Describe your facial expressions by linking image encoders and large language models

    Y . Yuan, J. Zeng, and S. Shan, “Describe your facial expressions by linking image encoders and large language models.” in BMVC, 2023, p. 377

    2023

  19. [31]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Pro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo...

    2019

  20. [32]

    Hierarchical transformers for multi-document summarization,

    Y . Liu and M. Lapata, “Hierarchical transformers for multi-document summarization,” arXiv preprint arXiv:1905.13164 , 2019

    1905 arXiv

  21. [33]

    Hdt: Hierarchical document transformer,

    H. He, M. Flicke, J. Buchmann, I. Gurevych, and A. Geiger, “Hdt: Hierarchical document transformer,” arXiv preprint arXiv:2407.08330 , 2024

    2024 arXiv

  22. [34]

    Multi-task learning of hierarchical vision-language representation,

    D.-K. Nguyen and T. Okatani, “Multi-task learning of hierarchical vision-language representation,” in Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2019, pp. 10 492– 10 501

    2019

  23. [35]

    Hero: Hier- archical encoder for video+ language omni-representation pre-training,

    L. Li, Y .-C. Chen, Y . Cheng, Z. Gan, L. Yu, and J. Liu, “Hero: Hier- archical encoder for video+ language omni-representation pre-training,” arXiv preprint arXiv:2005.00200 , 2020

    2005 arXiv

  24. [36]

    Hierarchical modular network for video captioning,

    H. Ye, G. Li, Y . Qi, S. Wang, Q. Huang, and M.-H. Yang, “Hierarchical modular network for video captioning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 17 939–17 948

    2022

  25. [37]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PmLR, 2021, pp. 8748–8763

    2021

  26. [38]

    Ceprompt: Cross-modal emotion-aware prompting for facial expression recognition,

    H. Zhou, S. Huang, F. Zhang, and C. Xu, “Ceprompt: Cross-modal emotion-aware prompting for facial expression recognition,” IEEE Transactions on Circuits and Systems for Video Technology , 2024

    2024

  27. [39]

    Sinkhorn distances: Lightspeed computation of optimal transport,

    M. Cuturi, “Sinkhorn distances: Lightspeed computation of optimal transport,” Advances in neural information processing systems , vol. 26, 2013

    2013

  28. [40]

    Recent advances in optimal transport for machine learning,

    E. F. Montesuma, F. M. N. Mboula, and A. Souloumiac, “Recent advances in optimal transport for machine learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024

    2024

  29. [41]

    Reliable weighted optimal transport for unsupervised domain adaptation,

    R. Xu, P. Liu, L. Wang, C. Chen, and J. Wang, “Reliable weighted optimal transport for unsupervised domain adaptation,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2020. [Online]. Available: http://dx.doi.org/10.1109/ cvpr42600.2020.00445

    2020

  30. [42]

    Unsupervised learning of visual features by contrasting cluster as- signments,

    M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster as- signments,” Le Centre pour la Communication Scientifique Directe - HAL - Universit ´e Paris Descartes,Le Centre pour la Communication Sci...

    2020

  31. [43]

    Optimal partial transport based sentence selection for long-form document matching,

    W. Yu, L. Pang, J. Xu, B. Su, Z. Dong, and J.-R. Wen, “Optimal partial transport based sentence selection for long-form document matching,” in Proceedings of the 29th international conference on computational linguistics, 2022, pp. 2363–2373

    2022

  32. [44]

    Learning to align sequential actions in the wild,

    W. Liu, B. Tekin, H. Coskun, V . Vineet, P. Fua, and M. Pollefeys, “Learning to align sequential actions in the wild,” Cornell University - arXiv,Cornell University - arXiv , Nov 2021

    2021

  33. [45]

    What when and where? self-supervised spatio-temporal grounding in untrimmed multi-action videos from narrated instructions,

    B. Chen, N. Shvetsova, A. Rouditchenko, D. Kondermann, S. Thomas, S.-F. Chang, R. Feris, J. Glass, and H. Kuehne, “What when and where? self-supervised spatio-temporal grounding in untrimmed multi-action videos from narrated instructions,” in Proceedings of the IEEE/CVF confer...

    2024

  34. [46]

    Multi- granularity correspondence learning from long-term noisy videos,

    Y . Lin, J. Zhang, Z. Huang, J. Liu, Z. Wen, and X. Peng, “Multi- granularity correspondence learning from long-term noisy videos,” arXiv preprint arXiv:2401.16702, 2024

    2024 arXiv

  35. [47]

    Spatial- temporal graphs plus transformers for geometry-guided facial expression recognition,

    R. Zhao, T. Liu, Z. Huang, D. P. Lun, and K.-M. Lam, “Spatial- temporal graphs plus transformers for geometry-guided facial expression recognition,” IEEE Transactions on Affective Computing , vol. 14, no. 4, pp. 2751–2767, 2022

    2022

  36. [49]

    Multimodal transformer for unaligned multimodal language sequences,

    Y .-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov, “Multimodal transformer for unaligned multimodal language sequences,” in Proceedings of the conference. Association for computational linguistics. Meeting , vol. 2019, 2019, p. 6558

    2019

  37. [50]

    Vilt: Vision-and-language transformer without convolution or region supervision,

    W. Kim, B. Son, and I. Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” in International conference on machine learning . PMLR, 2021, pp. 5583–5594

    2021

  38. [51]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in International conference on machine learning . PMLR, 2022, pp. 12 888–12 900

    2022

  39. [52]

    Cliper: A unified vision-language framework for in-the-wild facial expression recognition,

    H. Li, H. Niu, Z. Zhu, and F. Zhao, “Cliper: A unified vision-language framework for in-the-wild facial expression recognition,” in 2024 IEEE International Conference on Multimedia and Expo (ICME) . IEEE, 2024, pp. 1–6

    2024

  40. [53]

    Video-llava: Learning united visual representation by alignment before projection,

    B. Lin, Y . Ye, B. Zhu, J. Cui, M. Ning, P. Jin, and L. Yuan, “Video-llava: Learning united visual representation by alignment before projection,” arXiv preprint arXiv:2311.10122 , 2023

    2023 arXiv

  41. [54]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  42. [55]

    Videoclip: Contrastive pre-training for zero-shot video-text understanding,

    H. Xu, G. Ghosh, P.-Y . Huang, D. Okhonko, A. Aghajanyan, F. Metze, L. Zettlemoyer, and C. Feichtenhofer, “Videoclip: Contrastive pre-training for zero-shot video-text understanding,” arXiv preprint arXiv:2109.14084, 2021

    2021 arXiv

  43. [56]

    Videomae: Masked autoen- coders are data-efficient learners for self-supervised video pre-training,

    Z. Tong, Y . Song, J. Wang, and L. Wang, “Videomae: Masked autoen- coders are data-efficient learners for self-supervised video pre-training,” Advances in neural information processing systems, vol. 35, pp. 10 078– 10 093, 2022

    2022

  44. [57]

    Mae-dfer: Efficient masked autoencoder for self-supervised dynamic facial expression recognition,

    L. Sun, Z. Lian, B. Liu, and J. Tao, “Mae-dfer: Efficient masked autoencoder for self-supervised dynamic facial expression recognition,” in Proceedings of the 31st ACM International Conference on Multimedia, ser. MM ’23. New York, NY , USA: Association for Computing Machinery,...

    2023

  45. [58]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image cap- 14 tioning,

    P. Sharma, N. Ding, S. Goodman, and R. Soricut, “Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image cap- 14 tioning,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, p...

    2018

  46. [59]

    Frozen in time: A joint video and image encoder for end-to-end retrieval,

    M. Bain, A. Nagrani, G. Varol, and A. Zisserman, “Frozen in time: A joint video and image encoder for end-to-end retrieval,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 1728–1738

    2021

  47. [60]

    Emotion recognition using imperfect speech recognition,

    F. Metze, A. Batliner, F. Eyben, T. Polzehl, B. Schuller, and S. Steidl, “Emotion recognition using imperfect speech recognition,” 2010

    2010

  48. [61]

    Posterior calibration for multi- class paralinguistic classification,

    G. Gosztolya and R. Busa-Fekete, “Posterior calibration for multi- class paralinguistic classification,” in 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2018, pp. 119–125

    2018

  49. [62]

    Rethinking the learning paradigm for dynamic facial expression recog- nition,

    H. Wang, B. Li, S. Wu, S. Shen, F. Liu, S. Ding, and A. Zhou, “Rethinking the learning paradigm for dynamic facial expression recog- nition,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 17 958–17 968

    2023

  50. [63]

    A ˆ3 lign-dfer: Pioneering comprehensive dynamic affective alignment for dynamic facial expression recognition with clip,

    Z. Tao, Y . Wang, J. Lin, H. Wang, X. Mai, J. Yu, X. Tong, Z. Zhou, S. Yan, Q. Zhao et al. , “A ˆ3 lign-dfer: Pioneering comprehensive dynamic affective alignment for dynamic facial expression recognition with clip,” arXiv preprint arXiv:2403.04294 , 2024

    2024 arXiv

  51. [64]

    Clip-aware expressive feature learning for video-based facial expression recognition,

    Y . Liu, C. Feng, X. Yuan, L. Zhou, W. Wang, J. Qin, and Z. Luo, “Clip-aware expressive feature learning for video-based facial expression recognition,” Information Sciences, vol. 598, pp. 182–195, 2022

    2022

  52. [65]

    Nr-dfernet: Noise-robust network for dy- namic facial expression recognition,

    H. Li, M. Sui, Z. Zhu et al., “Nr-dfernet: Noise-robust network for dy- namic facial expression recognition,” arXiv preprint arXiv:2206.04975 , 2022

    2022 arXiv

  53. [66]

    Logo-former: Local-global spatio-temporal transformer for dynamic facial expression recognition,

    F. Ma, B. Sun, and S. Li, “Logo-former: Local-global spatio-temporal transformer for dynamic facial expression recognition,” inICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  54. [67]

    Intensity-aware loss for dynamic facial expression recognition in the wild,

    H. Li, H. Niu, Z. Zhu, and F. Zhao, “Intensity-aware loss for dynamic facial expression recognition in the wild,” in Proceedings of the AAAI conference on artificial intelligence , vol. 37, no. 1, 2023, pp. 67–75

    2023

  55. [68]

    Transformer-based multimodal emotional perception for dynamic facial expression recogni- tion in the wild,

    X. Zhang, M. Li, S. Lin, H. Xu, and G. Xiao, “Transformer-based multimodal emotional perception for dynamic facial expression recogni- tion in the wild,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 5, pp. 3192–3203, 2023

    2023

  56. [69]

    Svfap: Self-supervised video facial affect perceiver,

    L. Sun, Z. Lian, K. Wang, Y . He, M. Xu, H. Sun, B. Liu, and J. Tao, “Svfap: Self-supervised video facial affect perceiver,”IEEE Transactions on Affective Computing , 2024

    2024

  57. [70]

    Hicmae: Hierarchical contrastive masked autoencoder for self-supervised audio-visual emotion recogni- tion,

    L. Sun, Z. Lian, B. Liu, and J. Tao, “Hicmae: Hierarchical contrastive masked autoencoder for self-supervised audio-visual emotion recogni- tion,” Information Fusion, vol. 108, p. 102382, 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.