Pith. sign in

REVIEW 3 major objections 5 minor 52 references

TAGF: Time-aware Gated Fusion for Multimodal Valence-Arousal Estimation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A BiLSTM gate over recursive attention steps makes audio-visual fusion time-aware and robust to misalignment.

desk verdict Real but modest extension; central robustness claim unsupported by architecture or results. read the letter →

arxiv 2507.02080 v1 pith:KRJDASAQ submitted 2025-07-02 cs.MM cs.SD

Yubeen Lee , Sangeun Lee , Chaewon Park , Junyeop Cha , Eunil Park This is my paper
classification cs.MMcs.SD
keywords multimodalemotionrecognitionvalence-arousalestimationtemporalgatingrecursivecross-attentionBiLSTMaudio-visualfusionAff-Wild2robustnesstomisalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the reason multimodal valence-arousal estimation fails under noisy or misaligned audio-visual streams is that existing recursive-attention fusion gates every time step the same way. It proposes TAGF, which treats the $T$ outputs of recursive joint cross-attention as a time-ordered sequence, encodes them with a bidirectional LSTM, and uses the resulting gating vectors to weight a sum of those outputs. On Aff-Wild2, TAGF reports a valence CCC of 0.427 and an arousal CCC of 0.676 on the validation set, and 0.512 and 0.568 on the test set, and the paper argues that the time-aware weights let the model suppress noisy or misaligned frames without frame-level supervision. If correct, the claim matters because it offers a simple, label-free way to make multimodal fusion robust in real-world emotion recognition.

What carries the argument

The central object is a BiLSTM-based temporal gating mechanism placed on top of recursive joint cross-attention. The recursion produces one representation $H^{(t)}$ per joint attention step; the encoder $f_{\mathrm{temp}}$ (a bidirectional LSTM) reads the ordered sequence $\{H^{(1)}, \dots, H^{(T)}\}$ and emits gating vectors $G = \{g_1, \dots, g_T\}$, and a learned parameter vector $w$ turns each $g_t$ into a softmax scalar $\alpha_t$ that weights $H^{(t)}$ in the final sum. This machinery converts a static pool-over-recursion-steps into temporal attention pooling that is end-to-end trainable and requires no frame-level labels.

What would settle it

Shuffle the order of the recursive outputs before the BiLSTM and retrain; if the CCC stays the same, the gate is not using temporal order. A second check: replace the recursion-axis BiLSTM with a gate applied along the actual frame axis; if that version outperforms TAGF on offset audio-visual inputs, the claimed cross-modal robustness is not explained by the recursion-axis gating.

Watch

Extended reading notes

Core claim

The central claim is that temporal awareness belongs inside the fusion step, not only in feature extraction: the outputs $\{H^{(1)}, \dots, H^{(T)}\}$ of recursive joint cross-attention form a sequence whose ordering encodes how the modalities refine each other, and a bidirectional LSTM (Eq. 6) can read that sequence to assign each step a context-dependent gate. Aggregation is then $F = \sum_{t=1}^{T} \alpha_t H^{(t)}$ with $\alpha_t = \operatorname{softmax}(w^\top g_t)$ (Eqs. 7-8), so the model can upweight informative recursive steps and downweight noisy or temporally misaligned ones. TAGF reports valence CCC 0.427 and arousal CCC 0.676 on the Aff-Wild2 validation split, and 0.512 and 0.568 on the official test split, and attributes its robustness to this time-aware gating.

Load-bearing premise

The load-bearing premise is that the order of the recursive attention steps carries real video-time meaning, so a bidirectional LSTM over those few steps can model moment-to-moment emotional changes; if the order is just an implementation detail, the gate cannot actually suppress individual noisy or shifted frames.

Editorial extensions

If this is right

  • A fusion model can be made time-aware by gating its recursive attention outputs, so the approach transfers to any recursive cross-attention architecture without changing the underlying attention layers.
  • The time-aware gate suppresses noisy or misaligned steps such as occluded faces or off-screen speech without masking heuristics or frame-level supervision.
  • Training only on Aff-Wild2 with a single modality pair, TAGF reaches an arousal CCC of 0.676 on validation, second among the compared methods, which suggests the gating helps most where temporal dynamics dominate.
  • Because the gate weights are computed per video from the recursive outputs, the fusion strategy adapts to each sequence rather than applying one fixed gating pattern across all time steps.
  • On the official test set, the model's average CCC of 0.540 stays competitive with larger pipelines, supporting the paper's claim that the mechanism generalizes beyond the validation split.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that the recursion axis is not the video-time axis: if only a few recursive steps are used, a BiLSTM over whole-sequence outputs cannot literally perform per-frame suppression, so the reported robustness likely also depends on the frame-level temporal context learned by the backbone and temporal convolutional layers.
  • Nothing in the gating equations is emotion-specific, so the same construction could be tested on other continuous-output multimodal regression tasks with offset or partially dropped streams, such as pain or engagement estimation.
  • A natural testable extension would run the same gate along the actual frame axis instead of the recursion axis; the paper does not compare these two placements, and that comparison would separate temporal modeling of emotions from temporal modeling of the refinement process.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes TAGF, a Time-aware Gated Fusion framework for audio-visual valence-arousal estimation. The claimed contribution is a BiLSTM-based gate that reweights the outputs of recursive joint cross-attention steps (Eqs. 6–8) so that the fusion becomes time-aware, captures the sequential evolution of emotion, and is robust to cross-modal misalignment and noise. The model is evaluated on the Aff-Wild2 dataset with official validation and test splits, reporting mean CCC values of 0.552 (validation) and 0.540 (test). The central claim is that embedding temporal awareness into recursive fusion improves robustness to noisy or misaligned audio-visual streams without frame-level supervision.

Significance. If the claimed mechanism worked as described, it would be a lightweight, useful extension to recursive joint cross-attention models for dimensional emotion recognition. The paper provides a clean formulation of the gating equations, uses an established external benchmark (Aff-Wild2), and reports results on the official test server, which are all strengths. However, the core architectural claim is not supported by the equations as written: the gating weights are computed over recursive refinement steps, not over video frames, so the mechanism cannot implement the claimed frame-level suppression of misaligned or noisy inputs. In addition, the experimental section contains no misalignment or noise ablation, no gating ablation, and no error bars, and the reported average CCC is below the GRJCA/HGRJCA baselines to which the paper most directly relates. The paper is therefore better characterized as an unsupported incremental modification than as a validated robustness mechanism.

major comments (3)
  1. [Section 3.3.1, Eqs. (6)–(8)] The temporal gate runs over recursive-attention steps, not over video frames, so it cannot deliver the claimed frame-level robustness. Equation (6) defines each H(t) as a vector in R^d and G as an element of R^{T×d}; Eq. (7) produces one scalar α_t per recursive step; and Eq. (8) applies that same scalar to every frame. There is no frame index l in α_t. The BiLSTM therefore models the T recursive-refinement steps, not the L video frames of Eq. (1). This contradicts the statement in Section 3.2 that the module produces "attention weights for each frame and step" and the claim in Section 3.3.2 that the model can suppress "outputs that may be noisy or temporally misaligned, such as those corresponding to occluded faces or off-screen speech frames." The actual frame-level temporal modeling in the pipeline is performed by the TCNs described in Section 4.2.1, not by the proposed gate. This mismatch directly affects the central claim in the Abstract and the Conclusion.
  2. [Section 4.4, Tables 1–2] There is no experiment that tests cross-modal misalignment or noise, despite the Abstract and Conclusion claiming "strong robustness" to these conditions. The evaluation reports only clean CCC values on the official splits. Moreover, there is no ablation that removes or replaces the gating module; because Tables 1–2 compare against methods with different backbones, training schedules, and auxiliary components, any observed difference cannot be attributed to the BiLSTM gate. The average CCC of TAGF (0.552 validation, 0.540 test) is below GRJCA (0.556/0.581) and HGRJCA (0.562/0.591), so even the "competitive performance" claim requires more careful qualification, and the robustness claim has no direct empirical support.
  3. [Section 4.2.2 and Section 4.4.1] The paper reports no error bars, no multiple-seed statistics, and no sensitivity analysis over the number of recursive steps T or the BiLSTM hidden size. The statement in Section 4.4.1 that "consistent results observed across cross-validation folds beyond the official split" is not accompanied by any table, figure, or numerical summary. Given that the reported differences between TAGF and the relevant baselines are small (e.g., 0.552 versus 0.556 on the validation set), single-checkpoint CCC values cannot support the claim of stable or robust improvement.
minor comments (5)
  1. [Tables 1–2] The GRJCA and HGRJCA baselines are cited as reference [29], but [29] is Mennin et al., a psychotherapy article in Behavior Therapy, not an audio-visual emotion recognition paper. The correct references appear to be [34] and [37].
  2. [Section 4.2.1] The claim in Section 4.4.1 that the model is trained "without relying on multiple pretrained backbones or external resources" is contradicted by Section 4.2.1, which uses a ResNet-50 pretrained on MS-Celeb-1M and a VGGish model pretrained on AudioSet.
  3. [Section 4.2.2] Key implementation details are missing: the number of recursive cross-attention steps T, the BiLSTM hidden size, the TCN layer configuration, and whether the gate is applied to visual and audio features jointly or separately. No code or repository is provided.
  4. [General presentation] There are several typos and formatting issues: "seperated" in Section 4.2.1, "an Multi-Layer Perceptron" in the Figure 2 caption, and the column header "CCC A vg" in Tables 1 and 2, which should be "CCC Avg."
  5. [Table 2 caption] The test set is evaluated through the official challenge server, so describing it as "fold-0" is misleading; the test split is not a cross-validation fold.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation chain: TAGF's gate is learned over recursive-attention steps and evaluated on an external benchmark.

full rationale

I walked the derivation from Eq. (2) through Eqs. (3)-(8). The temporal gate computes a softmax distribution over T recursive cross-attention outputs and a weighted sum; the weight vector w and BiLSTM parameters are learned from Aff-Wild2 training data. Nothing in the paper fits a parameter to the validation/test CCC and then repackages that fit as a prediction. The benchmark numbers in Tables 1 and 2 come from an official external server/split and published baselines, so the empirical claim is not circular. The paper contains no load-bearing self-citation: the cited recursive-attention prior work (JCA/RJCA/GRJCA, refs. [34,37,29]) is used as background and baseline, not as an authority that forces the TAGF design. There are legitimate concerns that are not circularity: (i) the BiLSTM gate operates over T recursion steps (small) and alpha_t is step-level, not frame-level, so the stated mechanism cannot per-frame suppress 'occluded faces or off-screen speech frames' as claimed; (ii) no misalignment/noise ablation supports the robustness claim; and (iii) ref. [29] is a psychotherapy paper rather than the GRJCA source. These are overclaiming/citation-accuracy issues, not reductions of the result to its own inputs. Since the central derivation is self-contained against external data and no equation is definitionally identical to the target result, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The model itself uses standard learned weights; the entries above highlight the hyperparameters and background assumptions that the paper's central claims depend on but that are not independently justified. The key ad hoc assumption is treating the recursion-step index as a temporal axis.

free parameters (3)
  • T (number of recursive cross-attention steps) = Not reported in the paper
    Determines the sequence length fed to the BiLSTM; if T is small (2-3), the 'temporal' encoder sees almost no temporal sequence. No ablation over T is provided, and the value is essential to the time-aware claim.
  • Sequence length and stride (300 frames, stride 200) = 300 frames with 200-frame stride
    This windowing choice affects how much temporal context the model sees and how emotion transitions are segmented; no alternative settings or ablations are given.
  • BiLSTM hidden size and TCN parameters = Not reported
    These hyperparameters are central to the gating module, which is the paper's main contribution; without them the architecture cannot be faithfully reproduced.
assumptions (4)
  • domain assumption Aff-Wild2 annotations and the official fold-0 split provide a reliable, noise-free measure of valence-arousal quality.
    All conclusions rest on CCC computed on this single dataset and split (Sections 4.3-4.4), with no cross-dataset validation.
  • ad hoc to paper The sequence of recursive cross-attention outputs H(1)...H(T) can be treated as a temporally ordered sequence whose interdependencies correspond to frame-level emotional dynamics.
    Invoked in Section 3.3.1 Eq. (6); the recursion index is not the video time axis, yet the paper calls the gate 'time-aware' and claims frame-level robustness. This is the load-bearing assumption behind the main contribution.
  • domain assumption Cross-paper comparison of CCC scores on Aff-Wild2 is valid without re-running baselines under identical training conditions.
    Tables 1-2 compare against scores reported by other teams without matching backbones, TCN usage, training schedules, or folds; no re-implementation is provided.
  • standard math Standard definitions of multi-head cross-attention and BiLSTM.
    Used in Eqs. (3)-(6); unproved background assumed from the literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TAGF: Time-aware Gated Fusion for Multimodal Valence-Arousal Estimation." pith.science (2026). https://pith.science/paper/KRJDASAQ

@misc{pith2026250702080,
  author       = {Pith},
  title        = {Pith review of: TAGF: Time-aware Gated Fusion for Multimodal Valence-Arousal Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KRJDASAQ}},
  note         = {Machine review of arXiv:2507.02080}
}
read the original abstract

Multimodal emotion recognition often suffers from performance degradation in valence-arousal estimation due to noise and misalignment between audio and visual modalities. To address this challenge, we introduce TAGF, a Time-aware Gated Fusion framework for multimodal emotion recognition. The TAGF adaptively modulates the contribution of recursive attention outputs based on temporal dynamics. Specifically, the TAGF incorporates a BiLSTM-based temporal gating mechanism to learn the relative importance of each recursive step and effectively integrates multistep cross-modal features. By embedding temporal awareness into the recursive fusion process, the TAGF effectively captures the sequential evolution of emotional expressions and the complex interplay between modalities. Experimental results on the Aff-Wild2 dataset demonstrate that TAGF achieves competitive performance compared with existing recursive attention-based models. Furthermore, TAGF exhibits strong robustness to cross-modal misalignment and reliably models dynamic emotional transitions in real-world conditions.

Figures

Figures reproduced from arXiv: 2507.02080 by the authors.

Figure 1
Figure 1. A two-dimensional Valence-Arousal space illustrating [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall architecture of the proposed TAGF framework. Visual and audio inputs are first encoded via modality-specific back [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 43 canonical work pages

  1. [29]

    United we stand: Emphasizing commonali- ties across cognitive-behavioral therapies

    Douglas S Mennin, Kristen K Ellard, David M Fresco, and James J Gross. United we stand: Emphasizing commonali- ties across cognitive-behavioral therapies. Behavior therapy, 44(2):234–248, 2013. 6

    work page 2013

  2. [1]

    Maven: Multi-modal attention for valence-arousal emotion network

    Vrushank Ahire, Kunal Shah, Mudasir Khan, Nikhil Pakhale, Lownish Sookha, Mudasir Ganaie, and Abhinav Dhall. Maven: Multi-modal attention for valence-arousal emotion network. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 5789–5799, 2025. 2

    2025

  3. [2]

    Gated multimodal networks

    John Arevalo, Thamar Solorio, Manuel Montes-y Gomez, and Fabio A Gonz´alez. Gated multimodal networks. Neural Computing and Applications, 32:10209–10228, 2020. 2

    work page 2020

  4. [3]

    Valence focus and arousal focus: Individual differences in the structure of affective experience

    Lisa A Feldman. Valence focus and arousal focus: Individual differences in the structure of affective experience. Journal of personality and social psychology, 69(1):153, 1995. 1

    work page 1995

  5. [4]

    Joint modelling of audio-visual cues using attention mechanisms for emotion recognition

    Esam Ghaleb, Jan Niehues, and Stylianos Asteriadis. Joint modelling of audio-visual cues using attention mechanisms for emotion recognition. Multimedia Tools and Applications, 82(8):11239–11264, 2023. 2

    work page 2023

  6. [5]

    Ms-celeb-1m: A dataset and benchmark for large-scale face recognition

    Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Nether- lands, October 11-14, 2016, Proceedings, Part III 14, pages 87–102. Springer, 2016. 5

    work page 2016

  7. [6]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 5

    2016

  8. [7]

    Deep learning-based approach for con- tinuous affect prediction from facial expression images in valence-arousal space

    Stephen Khor Wen Hwooi, Alice Othmani, and Aznul Qalid Md Sabri. Deep learning-based approach for con- tinuous affect prediction from facial expression images in valence-arousal space. IEEE Access , 10:96053–96065,

    work page

Show all 52 references
  1. [8]

    Continuous- time audiovisual fusion with recurrence vs

    Vincent Karas, Mani Kumar Tellamekala, Adria Mallol- Ragolta, Michel Valstar, and Bj¨orn W Schuller. Continuous- time audiovisual fusion with recurrence vs. attention for in-the-wild affect recognition. arXiv preprint arXiv:2203.13285, 2022. 2

    2022 arXiv

  2. [9]

    Abaw: Valence-arousal estimation, ex- pression recognition, action unit detection & multi-task learning challenges

    Dimitrios Kollias. Abaw: Valence-arousal estimation, ex- pression recognition, action unit detection & multi-task learning challenges. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 2328–2336, 2022. 1, 5

    2022

  3. [10]

    Abaw: Learning from synthetic data & multi-task learning challenges

    Dimitrios Kollias. Abaw: Learning from synthetic data & multi-task learning challenges. In European Conference on Computer Vision, pages 157–172. Springer, 2023. 1

    2023

  4. [11]

    Multi-label compound expression recog- nition: C-expr database & network

    Dimitrios Kollias. Multi-label compound expression recog- nition: C-expr database & network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5589–5598, 2023. 1

    2023

  5. [12]

    Expression, affect, action unit recognition: Aff-wild2, multi-task learning and arcface

    Dimitrios Kollias and Stefanos Zafeiriou. Expression, affect, action unit recognition: Aff-wild2, multi-task learning and arcface. arXiv preprint arXiv:1910.04855, 2019

    1910 arXiv

  6. [13]

    Affect analysis in-the-wild: Valence-arousal, expressions, action units and a unified framework

    Dimitrios Kollias and Stefanos Zafeiriou. Affect analysis in-the-wild: Valence-arousal, expressions, action units and a unified framework. arXiv preprint arXiv:2103.15792, 2021. 1

    2021 arXiv

  7. [14]

    Analysing affec- tive behavior in the second abaw2 competition

    Dimitrios Kollias and Stefanos Zafeiriou. Analysing affec- tive behavior in the second abaw2 competition. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 3652–3660, 2021. 1

    2021

  8. [15]

    Analysing affective behavior in the first abaw 2020 competition

    D Kollias, A Schulc, E Hajiyev, and S Zafeiriou. Analysing affective behavior in the first abaw 2020 competition. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)(FG) , pages 794–

    2020

  9. [16]

    Face behavior a la carte: Expressions, af- fect and action units in a single network

    Dimitrios Kollias, Viktoriia Sharmanska, and Stefanos Zafeiriou. Face behavior a la carte: Expressions, af- fect and action units in a single network. arXiv preprint arXiv:1910.11111, 2019

    1910 arXiv

  10. [17]

    Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architec- tures, and beyond

    Dimitrios Kollias, Panagiotis Tzirakis, Mihalis A Nicolaou, Athanasios Papaioannou, Guoying Zhao, Bj ¨orn Schuller, Irene Kotsia, and Stefanos Zafeiriou. Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architec- tures, and beyond. International Journa...

    2019

  11. [18]

    Distribution matching for heterogeneous multi- task learning: a large-scale face study

    Dimitrios Kollias, Viktoriia Sharmanska, and Stefanos Zafeiriou. Distribution matching for heterogeneous multi- task learning: a large-scale face study. arXiv preprint arXiv:2105.03790, 2021

    2021 arXiv

  12. [19]

    Abaw: Valence-arousal esti- mation, expression recognition, action unit detection & emo- tional reaction intensity estimation challenges

    Dimitrios Kollias, Panagiotis Tzirakis, Alice Baird, Alan Cowen, and Stefanos Zafeiriou. Abaw: Valence-arousal esti- mation, expression recognition, action unit detection & emo- tional reaction intensity estimation challenges. In Proceed- ings of the IEEE/CVF Conference on Com...

    2023

  13. [20]

    Distribution matching for multi-task learning of classification tasks: a large-scale study on faces & beyond

    Dimitrios Kollias, Viktoriia Sharmanska, and Stefanos Zafeiriou. Distribution matching for multi-task learning of classification tasks: a large-scale study on faces & beyond. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 2813–2821, 2024. 7

    2024

  14. [21]

    The 6th affective behav- ior analysis in-the-wild (abaw) competition

    Dimitrios Kollias, Panagiotis Tzirakis, Alan Cowen, Ste- fanos Zafeiriou, Irene Kotsia, Alice Baird, Chris Gagne, Chunchang Shao, and Guanyu Hu. The 6th affective behav- ior analysis in-the-wild (abaw) competition. In Proceedings of the IEEE/CVF Conference on Computer Vision a...

    2024

  15. [22]

    7th abaw competition: Multi-task learning and compound expression recognition

    Dimitrios Kollias, Stefanos Zafeiriou, Irene Kotsia, Abhinav Dhall, Shreya Ghosh, Chunchang Shao, and Guanyu Hu. 7th abaw competition: Multi-task learning and compound expression recognition. arXiv preprint arXiv:2407.03835 ,

    arXiv

  16. [23]

    Dvd: A comprehensive dataset for advanc- ing violence detection in real-world scenarios.arXiv preprint arXiv:2506.05372, 2025

    Dimitrios Kollias, Damith C Senadeera, Jianian Zheng, Kaushal KK Yadav, Greg Slabaugh, Muhammad Awais, and Xiaoyun Yang. Dvd: A comprehensive dataset for advanc- ing violence detection in real-world scenarios.arXiv preprint arXiv:2506.05372, 2025. 1

    2025 arXiv

  17. [24]

    Advancements in affective and behavior analysis: The 8th abaw workshop and competition

    Dimitrios Kollias, Panagiotis Tzirakis, Alan Cowen, Irene Kotsia, UK Cogitat, Eric Granger, Marco Pedersoli, Simon Bacon, Alice Baird, Chunchang Shao, et al. Advancements in affective and behavior analysis: The 8th abaw workshop and competition. 2025. 1, 5

    2025

  18. [25]

    Gated mechanism for at- tention based multi modal sentiment analysis

    Ayush Kumar and Jithendra Vepa. Gated mechanism for at- tention based multi modal sentiment analysis. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4477–4481. IEEE, 2020. 3

    2020

  19. [26]

    Emotion, artificial intelligence, and ethics

    Kevin LaGrandeur. Emotion, artificial intelligence, and ethics. Beyond artificial intelligence: The disappearing human-machine divide, pages 97–109, 2015. 1

    2015

  20. [27]

    Multi-modal fusion network with complementarity and importance for emotion recognition

    Shuai Liu, Peng Gao, Yating Li, Weina Fu, and Weiping Ding. Multi-modal fusion network with complementarity and importance for emotion recognition. Information Sci- ences, 619:679–694, 2023. 2

    2023

  21. [28]

    Valence and arousal estimation based on multimodal temporal-aware features for videos in the wild

    Liyu Meng, Yuchen Liu, Xiaolong Liu, Zhaopei Huang, Wenqiang Jiang, Tenggan Zhang, Chuanhe Liu, and Qin Jin. Valence and arousal estimation based on multimodal temporal-aware features for videos in the wild. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pa...

    2022

  22. [30]

    Cnn-lstm facial ex- pression recognition method fused with two-layer attention mechanism

    Ye Ming, Hu Qian, and Liu Guangyuan. Cnn-lstm facial ex- pression recognition method fused with two-layer attention mechanism. Computational Intelligence and Neuroscience, 2022(1):7450637, 2022. 2

    2022

  23. [31]

    Mul- timodal emotion recognition using cross modal audio-video fusion with attention and deep metric learning

    Bogdan Mocanu, Ruxandra Tapu, and Titus Zaharia. Mul- timodal emotion recognition using cross modal audio-video fusion with attention and deep metric learning. Image and Vision Computing, 133:104676, 2023. 2

    2023

  24. [32]

    Cross-attention is not always needed: Dynamic cross-attention for audio-visual di- mensional emotion recognition

    R Gnana Praveen and Jahangir Alam. Cross-attention is not always needed: Dynamic cross-attention for audio-visual di- mensional emotion recognition. In 2024 IEEE International Conference on Multimedia and Expo (ICME) , pages 1–6. IEEE, 2024. 6

    2024

  25. [33]

    Incongruity-aware cross-modal attention for audio-visual fusion in dimensional emotion recognition

    R Gnana Praveen and Jahangir Alam. Incongruity-aware cross-modal attention for audio-visual fusion in dimensional emotion recognition. IEEE Journal of Selected Topics in Sig- nal Processing, 2024. 3

    2024

  26. [34]

    Recursive joint cross- modal attention for multimodal fusion in dimensional emo- tion recognition

    R Gnana Praveen and Jahangir Alam. Recursive joint cross- modal attention for multimodal fusion in dimensional emo- tion recognition. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 4803–4813, 2024. 2, 3

    2024

  27. [35]

    A joint cross-attention model for audio-visual fusion in dimensional emotion recognition

    R Gnana Praveen, Wheidima Carneiro de Melo, Nasib Ul- lah, Haseeb Aslam, Osama Zeeshan, Th ´eo Denorme, Marco Pedersoli, Alessandro L Koerich, Simon Bacon, Patrick Car- dinal, et al. A joint cross-attention model for audio-visual fusion in dimensional emotion recognition. In P...

    2022

  28. [36]

    Audio–visual fusion for emotion recognition in the valence– arousal space using joint cross-attention

    R Gnana Praveen, Patrick Cardinal, and Eric Granger. Audio–visual fusion for emotion recognition in the valence– arousal space using joint cross-attention. IEEE Transactions on Biometrics, Behavior, and Identity Science, 5(3):360–373,

  29. [37]

    Re- cursive joint attention for audio-visual fusion in regression based emotion recognition

    R Gnana Praveen, Eric Granger, and Patrick Cardinal. Re- cursive joint attention for audio-visual fusion in regression based emotion recognition. In ICASSP 2023-2023 IEEE In- ternational Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pages 1–5. IEEE, 2023. 2, 3, 6

    2023

  30. [38]

    A circumplex model of affect

    James A Russell. A circumplex model of affect. Journal of personality and social psychology, 39(6):1161, 1980. 2

    1980

  31. [39]

    Audiovisual three-level fusion for continuous esti- mation of russell’s emotion circumplex

    Enrique S ´anchez-Lozano, Paula Lopez-Otero, Laura Docio- Fernandez, Enrique Argones-R ´ua, and Jos ´e Luis Alba- Castro. Audiovisual three-level fusion for continuous esti- mation of russell’s emotion circumplex. In Proceedings of the 3rd ACM international workshop on Audio/v...

    2013

  32. [40]

    A novel spatio-temporal convolu- tional neural framework for multimodal emotion recogni- tion

    Masoumeh Sharafi, Mohammadreza Yazdchi, Reza Rasti, and Fahimeh Nasimi. A novel spatio-temporal convolu- tional neural framework for multimodal emotion recogni- tion. Biomedical Signal Processing and Control, 78:103970,

  33. [41]

    Very deep convo- lutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 5

    2014 arXiv

  34. [42]

    Using deep and convo- lutional neural networks for accurate emotion classification on deap data

    Samarth Tripathi, Shrinivas Acharya, Ranti Sharma, Sudhan- shi Mittal, and Samit Bhattacharya. Using deep and convo- lutional neural networks for accurate emotion classification on deap data. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4746–4752, 2017. 1

    2017

  35. [43]

    A multimodal fusion emotion recognition method based on multitask learn- ing and attention mechanism

    Jinbao Xie, Jiyu Wang, Qingyan Wang, Dali Yang, Jinming Gu, Yongqiang Tang, and Yury I Varatnitski. A multimodal fusion emotion recognition method based on multitask learn- ing and attention mechanism. Neurocomputing, 556:126649,

  36. [44]

    Speech emotion classi- fication using attention-based lstm

    Yue Xie, Ruiyu Liang, Zhenlin Liang, Chengwei Huang, Cairong Zou, and Bj ¨orn Schuller. Speech emotion classi- fication using attention-based lstm. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 27(11):1675– 1685, 2019. 1

    2019

  37. [45]

    Aff-wild: Valence and arousal ‘in-the-wild’challenge

    Stefanos Zafeiriou, Dimitrios Kollias, Mihalis A Nicolaou, Athanasios Papaioannou, Guoying Zhao, and Irene Kot- sia. Aff-wild: Valence and arousal ‘in-the-wild’challenge. 8 In Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on , pages 1980–1987....

    2017

  38. [46]

    Contin- uous emotion recognition with audio-visual leader-follower attentive fusion

    Su Zhang, Yi Ding, Ziquan Wei, and Cuntai Guan. Contin- uous emotion recognition with audio-visual leader-follower attentive fusion. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 3567–3574,

  39. [47]

    Deep learning-based multimodal emotion recognition from audio, visual, and text modalities: A systematic review of recent advancements and future prospects

    Shiqing Zhang, Yijiao Yang, Chen Chen, Xingnan Zhang, Qingming Leng, and Xiaoming Zhao. Deep learning-based multimodal emotion recognition from audio, visual, and text modalities: A systematic review of recent advancements and future prospects. Expert Systems with Applications...

    2024

  40. [48]

    A multimodal semantic fusion network with cross-modal alignment for multimodal senti- ment analysis

    Shunxiang Zhang, Jiajia Liu, Yixuan Jiao, Yulei Zhang, Lei Chen, and Kuanching Li. A multimodal semantic fusion network with cross-modal alignment for multimodal senti- ment analysis. ACM Transactions on Multimedia Comput- ing, Communications and Applications, 2025. 2

    2025

  41. [49]

    Multi- modal facial affective analysis based on masked autoencoder

    Wei Zhang, Bowen Ma, Feng Qiu, and Yu Ding. Multi- modal facial affective analysis based on masked autoencoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5793–5802, 2023. 6

    2023

  42. [50]

    M 3 f: Multi-modal continuous valence-arousal esti- mation in the wild

    Yuan-Hang Zhang, Rulin Huang, Jiabei Zeng, and Shiguang Shan. M 3 f: Multi-modal continuous valence-arousal esti- mation in the wild. In 2020 15th IEEE International Confer- ence on Automatic Face and Gesture Recognition (FG 2020), pages 632–636. IEEE, 2020. 2

    2020

  43. [51]

    Abaw5 challenge: A facial affect recognition ap- proach utilizing transformer encoder and audiovisual fusion

    Ziyang Zhang, Liuwei An, Zishun Cui, Ao Xu, Tengteng Dong, Yueqi Jiang, Jingyi Shi, Xin Liu, Xiao Sun, and Meng Wang. Abaw5 challenge: A facial affect recognition ap- proach utilizing transformer encoder and audiovisual fusion. In Proceedings of the IEEE/CVF Conference on Comp...

    2023

  44. [52]

    Leveraging tcn and transformer for effective visual- audio fusion in continuous emotion recognition

    Weiwei Zhou, Jiada Lu, Zhaolong Xiong, and Weifeng Wang. Leveraging tcn and transformer for effective visual- audio fusion in continuous emotion recognition. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5756–5763, 2023. 6 9

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.