Pith. sign in

REVIEW 4 major objections 5 minor 44 references

Distance-aware Soft Prompt Guidance for Multimodal Valence-Arousal Estimation

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that re-framing valence-arousal regression as soft-label matching over a 3×3 emotion grid lets pre-trained vision-language knowledge improve in-the-wild emotion estimation.

desk verdict A plausible soft-labeling idea undermined by a central mechanism that is never actually specified, so the reported gains are unattributable. read the letter →

arxiv 2603.13415 v2 pith:XVBQ3GRA submitted 2026-03-12 cs.CV

Byeongjin Jung , Chanyeong Park , Sejoon Lim This is my paper
classification cs.CV
keywords valence-arousalestimationCLIPsoftpromptlearningmultimodalfusionGaussianlabelsAff-Wild2cross-modalattentionemotionrecognition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes to bridge CLIP's discrete text-prompt semantics and the continuous valence-arousal (VA) plane by partitioning the VA space into a 3×3 grid of nine emotion regions, each described by short text templates. Instead of hard region labels, it assigns a Gaussian soft target to each ground-truth VA coordinate based on Euclidean distance to the region centers. The model, built from a CLIP image encoder, an Audio Spectrogram Transformer, GRU temporal modeling, and hierarchical cross-modal attention plus gated fusion, is trained with a CCC regression loss plus a KL-divergence term meant to align its predicted distribution with those soft targets. On Aff-Wild2 the framework reaches 0.5361 mean CCC, beating the official baseline and its own ablations. If the mechanism works as described, it offers a cheap way to inject categorical semantic constraints into continuous emotion regression.

What carries the argument

The key machinery is the Gaussian soft-labeling over a 3×3 VA grid combined with CLIP text prototypes. Region centers ci ∈ {−0.66, 0, 0.66} and a kernel width σ = 0.45 map each ground-truth VA coordinate y to a soft probability wi over nine regions, and a KL-divergence loss (λ = 0.2) is added to the CCC loss to encourage the model's predictions to match that distribution. The paper also builds a hierarchical fusion stack — cross-modal attention from audio to visual features followed by a learned gate that blends the attended signal with the original visual stream — plus bidirectional GRUs for temporal modeling.

What would settle it

Train the same architecture with the KL loss removed (λ = 0) and with the soft labels replaced by hard one-hot region labels; if mean CCC does not drop or if the model with hard labels performs equally, the distance-aware soft-prompt mechanism is not what drives the reported improvement. Also, compute the cosine similarity between the nine text prototypes and the VA ground-truth labels to see whether the semantic distances actually correlate with coordinate distances.

Watch

Extended reading notes

Core claim

The central claim is that 'Distance-aware Soft Prompt Learning' closes the gap between semantic space and continuous dimensions: by treating each VA coordinate as a point whose distance to nine emotion-region centers defines a soft distribution, the model can use CLIP's text-embedding space as a semantic regularizer. The authors report that this semantic-guided approach, combined with CLIP ViT-B/16 and AST backbones and hierarchical fusion, significantly enhances VA estimation accuracy, achieving a mean CCC of 0.5361 on Aff-Wild2. The claim is that the gain comes from the soft-prompt semantic guidance rather than from the backbone or fusion choices alone.

Load-bearing premise

The load-bearing premise is that the Euclidean distance between valence-arousal coordinates corresponds to semantic similarity in CLIP's text-embedding space — if that mapping is wrong, the Gaussian soft targets and the KL loss may be injecting noise into the regression, not semantic guidance.

Editorial extensions

If this is right

  • If the semantic-guidance mechanism is the source of the gains, any VA regression model could be upgraded by adding a soft emotion-grid head without changing its backbone.
  • The localized grid centers (±0.66, σ = 0.45) suggest that matching the semantic regions to the label distribution of the target dataset matters more than using a generic grid.
  • The reported CCC gains indicate that both feature quality and semantic target design contribute, and that the two interact.
  • The method provides a template for applying CLIP-style discrete prompts to other continuous regression tasks where the output space has a meaningful geometry, such as action unit intensity or physiological signals.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never defines the predicted distribution q used in the KL loss, and the text prototypes p_i, though described, do not appear in any equation; if the semantic head is not actually conditioning on CLIP text embeddings, the reported gains may come from the soft-label target alone rather than from 'prompt guidance.'
  • The core assumption that Euclidean distance in VA space mirrors semantic similarity in CLIP's embedding space is untested; a direct check — correlating VA distances with CLIP text-embedding cosine distances for emotion words — would either ground or undermine the bridge.
  • The 0.5361 mean CCC comes from a single split of Aff-Wild2 with no reported variance or statistical significance; a fair comparison would require multiple runs and confidence intervals.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a multimodal valence-arousal (VA) estimation framework for Aff-Wild2. A CLIP ViT-B/16 image encoder and an Audio Spectrogram Transformer extract features, bidirectional GRUs model temporal dynamics, and a hierarchical fusion module combines cross-modal attention with gated fusion. The claimed novelty is a 'Distance-aware Soft Prompt Learning' mechanism: the VA plane is partitioned into a 3x3 grid, each region is assigned text prototypes from CLIP, and Gaussian soft targets are computed from the Euclidean distance between the ground-truth VA coordinates and tuned grid centers. A KL-divergence term is added to the CCC loss to provide 'semantic regularization.' The best configuration reports CCC_mean 0.5361. However, the semantic-guidance mechanism is not actually specified: the text prototypes are defined but never used, and the KL loss is introduced without defining the predicted distribution it compares against. The experimental section therefore cannot support the paper's central claim.

Significance. If fully specified and validated, the idea of connecting continuous VA regression to discrete semantic prototypes through Gaussian soft labels and a KL regularizer would be of interest to the affective computing community. The paper is clearly organized, and Tables 1-3 provide systematic comparisons of backbones, temporal encoders, fusion strategies, and grid configurations. These architecture-level comparisons are a useful starting point. Nevertheless, the core contribution—semantic soft-prompt guidance—is never defined in a computable form, and no experiment isolates its effect. As written, the significance of the claimed contribution cannot be assessed, and the reported improvements cannot be attributed to the proposed semantic mechanism.

major comments (4)
  1. [§3.3, §3.5, Eq. (2), Eq. (6)] The text prototypes p_i introduced in §3.3 are never used in any subsequent equation or model description. The soft targets w_i in Eq. (2) depend only on the Euclidean distance between the ground-truth coordinate y and the region centers c_i; they involve no semantic content from p_i. In §3.5, the loss L_KL in Eq. (6) is never defined: the predicted distribution q over the nine regions is not specified, and no semantic head producing q is described. As written, L_KL cannot be computed from the manuscript. This is not a minor exposition gap: the paper's central claim that distance-aware soft prompting bridges semantic and continuous VA space rests on a term that is only a placeholder.
  2. [§4.2-§4.4, Tables 1-3] No ablation removes the KL loss or disables the semantic head. Tables 1-3 vary backbone combinations, temporal encoders, fusion strategies, and grid centers/σ, but every reported 'Ours' configuration includes the unspecified L_KL. The performance gains in Table 3 (CCC_mean 0.4795 to 0.5361) could therefore come from the stronger backbones, the hierarchical fusion, or the choice of grid centers and σ, rather than from semantic prompting. The central attribution is unsupported.
  3. [§4.4, §5.2, Table 3] The grid centers c ∈ {-0.66, 0, 0.66} and σ = 0.45 are explicitly tuned to the Aff-Wild2 label distribution, as stated in §5.2 ('the majority of in-the-wild emotional states are clustered toward the center'). The paper then presents this tuned configuration as 'the proposed method.' There is no held-out validation of these hyperparameters, no sensitivity analysis beyond the two settings in Table 3, and no evaluation on another dataset. Since the final configuration is selected using properties of the test distribution, the paper should either validate the choice on a separate split or reframe it as dataset-specific engineering rather than a general principle.
  4. [§3.3] The method assumes that Euclidean distance in VA space corresponds to semantic similarity in CLIP's text-embedding space, so that the nine hand-written descriptions and Gaussian soft targets provide meaningful semantic regularization. This premise is never tested. Even if the KL term were properly defined, the authors should provide evidence for this mapping, e.g., by measuring agreement between VA pairwise distances and CLIP text-embedding similarities, or by showing that removing the p_i-based semantic component degrades performance. Without such validation, the 'semantic' interpretation of the regularization is unjustified.
minor comments (5)
  1. [Title / Abstract] The title and Section 3 use 'Distance-aware Soft Prompt Learning,' while the abstract uses 'Distance-aware Soft Prompt Guidance.' Please unify terminology.
  2. [Eq. (2)] The region centers are written as c_i ∈ {-0.66, 0, 0.66}, but since y = (v, a), each c_i should be an element of the Cartesian product {-0.66, 0, 0.66}^2. Please clarify the notation.
  3. [§5.3] The qualitative claim that the model produces 'smoother and more natural transitions' is not supported by quantitative trajectory analysis. Please include plots or metrics (e.g., temporal smoothness, comparison with baselines).
  4. [§4.1] The evaluation protocol for Aff-Wild2 is not fully described: the train/validation split, fold assignment, and how the reported CCC is computed over sequences should be stated for reproducibility.
  5. [Table 2] The row labeled 'Ours (GRU) — Attn. + Gated' duplicates the architecture described as 'GRU — Attn. + Gated.' Please label rows consistently and avoid using 'Ours' when the configuration is already listed.

Circularity Check

1 steps flagged · score 6.0 of 10

Grid centers and sigma are tuned to the Aff-Wild2 label distribution and then presented as the proposed method; the semantic KL term is never defined, so the reported gain is a post-hoc fit rather than an independent validation.

  1. fitted input called prediction [Section 4.4, Table 3; discussion in Section 5.2]
    "We compare the standard setting (c ∈ {−1.0,0,1.0}, σ = 0.6) against a localized setting (c ∈ {−0.66,0,0.66}, σ = 0.45). Table 3 shows that the localized configuration provides superior performance. This is because the emotional intensities in the Aff-Wild2 dataset are more frequently distributed within the ±0.66 range, allowing the soft-prompting mechanism to provide more precise semantic guidance."

    The grid centers c_i and smoothing σ are not derived from an independent principle or held-out validation; they are selected because the Aff-Wild2 labels cluster within ±0.66. The same label distribution is then invoked to explain why the selected 'Ours' configuration wins. The reported CCC improvement of the proposed configuration is therefore the outcome of fitting the grid to the evaluation data, not an independent confirmation of the semantic-prompting mechanism. No ablation removes the KL term, so the gain cannot be traced to the semantic loss; the localized grid is a post-hoc fit presented as the proposed method.

full rationale

The concretely exhibitable circularity is the grid-configuration selection. Section 4.4 chooses c ∈ {−0.66, 0, 0.66}, σ = 0.45 over c ∈ {−1,0,1}, σ = 0.6, and Section 5.2 explains the superiority by the Aff-Wild2 label distribution—the same distribution that motivated the choice. Thus the 'Ours' configuration is a fitted quantity and its advantage is a selection artifact rather than a derived semantic property. Separately, the semantic-guidance mechanism is internally underspecified: text prototypes p_i are defined in Section 3.3 but never appear in any equation; L_KL in Eq. (6) is never defined and no predicted distribution q over the nine regions is ever introduced. This is an internal-consistency gap rather than a strict circular reduction, but it prevents attributing the reported CCC gains to CLIP prompt semantics. No self-citation chain is load-bearing, and the architecture ablations (Table 2) are not circular. Overall, the central claim that the proposed soft prompting drives the improvement is partially circular because its only direct evidence comes from a configuration fit to the evaluation data, with the semantic term itself left unspecified.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central method adds three hand-set hyperparameters (grid centers, σ, λ) and one unvalidated conceptual bridge (VA Euclidean geometry ≈ CLIP semantic similarity). No new physical or mathematical entities are introduced. The main unstated load is the semantic alignment assumption and the missing definition of the predicted distribution used in the KL term.

free parameters (3)
  • Grid centers c_i = {-0.66, 0.0, 0.66}
    Chosen by hand to match the Aff-Wild2 label distribution (Section 4.4, 5.2); tuned against the alternative {−1.0,0,1.0} using the same evaluation data.
  • Gaussian smoothing sigma = 0.45
    Set by hand and compared to 0.6 in Table 3; selected based on test/validation performance.
  • KL loss weight lambda = 0.2
    Hyperparameter chosen without sensitivity analysis (Section 3.5, Section 4.1).
assumptions (4)
  • domain assumption Valence-arousal space is a 2D Euclidean plane where Euclidean distance corresponds to semantic emotion dissimilarity
    Used to define the 3×3 grid and Gaussian soft labels (Eq. 2). The paper gives no empirical or theoretical justification that linear distance in VA coordinates matches semantic closeness in CLIP space.
  • ad hoc to paper CLIP text embeddings of the nine hand-written emotion descriptions are semantically informative for VA regression
    The method relies on the averaged text prototypes p_i being meaningful, but p_i never appear in any loss or equation, so even the mechanism of semantic guidance is unstated.
  • domain assumption The Aff-Wild2 ground-truth annotations are reliable and the 10s window / 20-frame sampling captures the emotion signal
    All results depend on this standard dataset assumption; no label-noise or preprocessing validation is reported.
  • ad hoc to paper The KL divergence between an unspecified predicted distribution and the Gaussian soft targets acts as an effective regularizer
    The predicted distribution q is never defined in Section 3.5, making the loss term unverifiable and the assumption that it helps unfounded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distance-aware Soft Prompt Guidance for Multimodal Valence-Arousal Estimation." pith.science (2026). https://pith.science/paper/XVBQ3GRA

@misc{pith2026260313415,
  author       = {Pith},
  title        = {Pith review of: Distance-aware Soft Prompt Guidance for Multimodal Valence-Arousal Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XVBQ3GRA}},
  note         = {Machine review of arXiv:2603.13415}
}
read the original abstract

Valence-arousal (VA) estimation is crucial for capturing the nuanced nature of human emotions in naturalistic environments. While pre-trained vision-language models such as CLIP have demonstrated remarkable semantic alignment capabilities, their application to continuous regression tasks is often limited by the discrete nature of text prompts. In this paper, we propose a novel multimodal framework for VA estimation that introduces Distance-aware Soft Prompt Guidance to bridge the gap between semantic representations and continuous affective dimensions. Specifically, we partition the VA space into multiple discrete regions, each associated with distinct textual descriptions. Rather than relying on hard categorization, we employ a Gaussian kernel to compute soft labels based on the Euclidean distance between the ground-truth coordinates and the region centers, allowing the model to learn fine-grained emotional transitions. For multimodal integration, our architecture utilizes a CLIP image encoder and an Audio Spectrogram Transformer to extract robust visual and acoustic features. These features are temporally modeled using Gated Recurrent Units and integrated through a hierarchical fusion scheme that sequentially combines cross-modal attention for alignment and gated fusion for adaptive refinement. Experimental results on the Aff-Wild2 dataset show that the proposed semantic-guided approach outperforms the official baseline and demonstrates robust performance on in-the-wild data.

Figures

Figures reproduced from arXiv: 2603.13415 by the authors.

Figure 1
Figure 1. Framework Overview: An overview of the proposed multimodal valence-arousal (VA) estimation framework. The architecture consists of (a) dual-branch feature extraction using CLIP ViT-B/16 for visual frames and AST for spectrograms, (b) temporal modeling via bidirectional GRUs, (c) a hierarchical fusion module combining cross-modal attention and gated fusion, and (d) a semantic-guided head utilizing Distance-aware Soft… view at source ↗

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 10 linked inside Pith

  1. [1]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 1

    2016

  2. [2]

    Cnn archi- tectures for large-scale audio classification

    Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn archi- tectures for large-scale audio classification. In2017 ieee in- ternational conference on acoustics, speech and signal pro- cessing (icassp), pages 131–135. IEEE, 2017. 1

    2017

  3. [3]

    Davis and P

    S. Davis and P. Mermelstein. Comparison of parametric rep- resentations for monosyllabic word recognition in continu- ously spoken sentences.IEEE Transactions on Acoustics, Speech, and Signal Processing, 28(4):357–366, 1980. 1, 2 6

    1980

  4. [4]

    An empirical evaluation of generic convolutional and recur- rent networks for sequence modeling.arXiv preprint arXiv:1803.01271, 2018

    Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recur- rent networks for sequence modeling.arXiv preprint arXiv:1803.01271, 2018. 1, 3

    arXiv 2018

  5. [5]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021. 1, 4

    2021

  6. [6]

    Ast: Audio spectrogram transformer.arXiv preprint arXiv:2104.01778,

    Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer.arXiv preprint arXiv:2104.01778,

    arXiv

  7. [7]

    Expression, af- fect, action unit recognition: Aff-wild2, multi-task learning and arcface.arXiv preprint arXiv:1910.04855, 2019

    Dimitrios Kollias and Stefanos Zafeiriou. Expression, af- fect, action unit recognition: Aff-wild2, multi-task learning and arcface.arXiv preprint arXiv:1910.04855, 2019. 2, 3

    arXiv 1910

  8. [8]

    A circumplex model of affect.Journal of personality and social psychology, 39(6):1161, 1980

    James A Russell. A circumplex model of affect.Journal of personality and social psychology, 39(6):1161, 1980. 2

    1980

Show all 44 references
  1. [9]

    A concordance correlation coef- ficient to evaluate reproducibility.Biometrics, pages 255– 268, 1989

    I Lawrence and Kuei Lin. A concordance correlation coef- ficient to evaluate reproducibility.Biometrics, pages 255– 268, 1989. 2, 3, 5

    1989

  2. [10]

    Aff-wild: Valence and arousal ‘in-the-wild’challenge

    Stefanos Zafeiriou, Dimitrios Kollias, Mihalis A Nicolaou, Athanasios Papaioannou, Guoying Zhao, and Irene Kot- sia. Aff-wild: Valence and arousal ‘in-the-wild’challenge. InComputer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on, pages 1980–1987. IEE...

    2017

  3. [11]

    Analysing affective behavior in the first abaw 2020 competition

    D Kollias, A Schulc, E Hajiyev, and S Zafeiriou. Analysing affective behavior in the first abaw 2020 competition. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)(FG), pages 794– 800, 2020. 2, 3

    2020

  4. [12]

    Analysing affec- tive behavior in the second abaw2 competition

    Dimitrios Kollias and Stefanos Zafeiriou. Analysing affec- tive behavior in the second abaw2 competition. InProceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 3652–3660, 2021. 2, 3

    2021

  5. [13]

    Abaw: Valence-arousal estimation, ex- pression recognition, action unit detection & multi-task learning challenges

    Dimitrios Kollias. Abaw: Valence-arousal estimation, ex- pression recognition, action unit detection & multi-task learning challenges. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 2328–2336, 2022. 2, 3

    2022

  6. [14]

    Abaw: learning from synthetic data & multi-task learning challenges

    Dimitrios Kollias. Abaw: learning from synthetic data & multi-task learning challenges. InEuropean Conference on Computer Vision, pages 157–172. Springer, 2023. 2, 3

    2023

  7. [15]

    Abaw: Valence-arousal estimation, expression recognition, action unit detection & emotional reaction intensity estimation challenges

    Dimitrios Kollias, Panagiotis Tzirakis, Alice Baird, Alan Cowen, and Stefanos Zafeiriou. Abaw: Valence-arousal estimation, expression recognition, action unit detection & emotional reaction intensity estimation challenges. InPro- ceedings of the IEEE/CVF Conference on Computer...

    2023

  8. [16]

    The 6th affective behav- ior analysis in-the-wild (abaw) competition

    Dimitrios Kollias, Panagiotis Tzirakis, Alan Cowen, Ste- fanos Zafeiriou, Irene Kotsia, Alice Baird, Chris Gagne, Chunchang Shao, and Guanyu Hu. The 6th affective behav- ior analysis in-the-wild (abaw) competition. InProceedings of the IEEE/CVF Conference on Computer Vision an...

    2024

  9. [17]

    7th abaw competition: Multi-task learning and com- pound expression recognition

    Dimitrios Kollias, Stefanos Zafeiriou, Irene Kotsia, Abhi- nav Dhall, Shreya Ghosh, Chunchang Shao, and Guanyu Hu. 7th abaw competition: Multi-task learning and com- pound expression recognition. InEuropean Conference on Computer Vision, pages 31–45. Springer, 2024. 2, 3

    2024

  10. [18]

    Advancements in affective and behavior analysis: The 8th abaw workshop and competition

    Dimitrios Kollias, Panagiotis Tzirakis, Alan Cowen, Ste- fanos Zafeiriou, Irene Kotsia, Eric Granger, Marco Ped- ersoli, Simon Bacon, Alice Baird, Chris Gagne, et al. Advancements in affective and behavior analysis: The 8th abaw workshop and competition. InProceedings of the C...

    2025

  11. [19]

    From emotions to violence: Multimodal fine- grained behavior analysis at the 9th abaw

    Dimitrios Kollias, Stefanos Zafeiriou, Irene Kotsia, Greg Slabaugh, Damith Chamalke Senadeera, Jianian Zheng, Kaushal Kumar Keshlal Yadav, Chunchang Shao, and Guanyu Hu. From emotions to violence: Multimodal fine- grained behavior analysis at the 9th abaw. InProceedings of the...

    2025

  12. [20]

    Multi-label compound expression recog- nition: C-expr database & network

    Dimitrios Kollias. Multi-label compound expression recog- nition: C-expr database & network. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5589–5598, 2023. 2

    2023

  13. [21]

    Affectnet: A database for facial expression, valence, and arousal computing in the wild.IEEE transactions on affective computing, 10(1):18–31, 2017

    Ali Mollahosseini, Behzad Hasani, and Mohammad H Ma- hoor. Affectnet: A database for facial expression, valence, and arousal computing in the wild.IEEE transactions on affective computing, 10(1):18–31, 2017. 2

    2017

  14. [22]

    Introducing the recola multimodal corpus of remote collaborative and affective interactions

    Fabien Ringeval, Andreas Sonderegger, Juergen Sauer, and Denis Lalanne. Introducing the recola multimodal corpus of remote collaborative and affective interactions. In2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), pages 1–...

    2013

  15. [23]

    Afew-va database for valence and arousal estimation in-the-wild.Image and Vision Computing, 65:23–36, 2017

    Jean Kossaifi, Georgios Tzimiropoulos, Sinisa Todorovic, and Maja Pantic. Afew-va database for valence and arousal estimation in-the-wild.Image and Vision Computing, 65:23–36, 2017. 2

    2017

  16. [24]

    Sewa db: A rich database for audio-visual emotion and sentiment re- search in the wild.IEEE transactions on pattern analysis and machine intelligence, 43(3):1022–1040, 2019

    Jean Kossaifi, Robert Walecki, Yannis Panagakis, Jie Shen, Maximilian Schmitt, Fabien Ringeval, Jing Han, Vedhas Pandit, Antoine Toisoul, Bj ¨orn Schuller, et al. Sewa db: A rich database for audio-visual emotion and sentiment re- search in the wild.IEEE transactions on patter...

    2019

  17. [25]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia 7 Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017. 2, 3, 4, 5

    2017

  18. [26]

    Tensor fusion network for multimodal sentiment analysis

    Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. InProceedings of the 2017 conference on empirical methods in natural language pro- cessing, pages 1103–1114, 2017. 2

    2017

  19. [27]

    Efficient low-rank multimodal fu- sion with modality-specific factors

    Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshmi- narasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. Efficient low-rank multimodal fu- sion with modality-specific factors. InProceedings of the 56th Annual Meeting of the Association for Computational Ling...

  20. [28]

    Gated multimodal units for infor- mation fusion.arXiv preprint arXiv:1702.01992, 2017

    John Arevalo, Thamar Solorio, Manuel Montes-y G ´omez, and Fabio A Gonz ´alez. Gated multimodal units for infor- mation fusion.arXiv preprint arXiv:1702.01992, 2017. 2, 4

    2017 arXiv

  21. [29]

    Multimodal transformer for unaligned multimodal language sequences

    Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. InProceedings of the 57th annual meeting of the association for computational linguistics, pag...

    2019

  22. [30]

    Incongruity-aware cross-modal attention for audio-visual fusion in dimen- sional emotion recognition.IEEE Journal of Selected Topics in Signal Processing, 18(3):444–458, 2024

    R Gnana Praveen and Jahangir Alam. Incongruity-aware cross-modal attention for audio-visual fusion in dimen- sional emotion recognition.IEEE Journal of Selected Topics in Signal Processing, 18(3):444–458, 2024. 3

    2024

  23. [31]

    Interactive multimodal framework with tempo- ral modeling for emotion recognition

    Jun Yu, Yongqi Wang, Lei Wang, Yang Zheng, and Sheng- fan Xu. Interactive multimodal framework with tempo- ral modeling for emotion recognition. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 5699–5706, 2025. 3

    2025

  24. [32]

    Face behavior a la carte: Expressions, af- fect and action units in a single network.arXiv preprint arXiv:1910.11111, 2019

    Dimitrios Kollias, Viktoriia Sharmanska, and Stefanos Zafeiriou. Face behavior a la carte: Expressions, af- fect and action units in a single network.arXiv preprint arXiv:1910.11111, 2019. 3

    1910 arXiv

  25. [33]

    Deep affect predic- tion in-the-wild: Aff-wild database and challenge, deep ar- chitectures, and beyond.International Journal of Computer Vision, pages 1–23, 2019

    Dimitrios Kollias, Panagiotis Tzirakis, Mihalis A Nicolaou, Athanasios Papaioannou, Guoying Zhao, Bj ¨orn Schuller, Irene Kotsia, and Stefanos Zafeiriou. Deep affect predic- tion in-the-wild: Aff-wild database and challenge, deep ar- chitectures, and beyond.International Journ...

    2019

  26. [34]

    Distribution matching for heterogeneous multi- task learning: a large-scale face study.arXiv preprint arXiv:2105.03790, 2021

    Dimitrios Kollias, Viktoriia Sharmanska, and Stefanos Zafeiriou. Distribution matching for heterogeneous multi- task learning: a large-scale face study.arXiv preprint arXiv:2105.03790, 2021. 3

    2021 arXiv

  27. [35]

    Affect analysis in- the-wild: Valence-arousal, expressions, action units and a unified framework.arXiv preprint arXiv:2103.15792, 2021

    Dimitrios Kollias and Stefanos Zafeiriou. Affect analysis in- the-wild: Valence-arousal, expressions, action units and a unified framework.arXiv preprint arXiv:2103.15792, 2021. 3

    2021 arXiv

  28. [36]

    Distribution matching for multi-task learning of classification tasks: a large-scale study on faces & beyond

    Dimitrios Kollias, Viktoriia Sharmanska, and Stefanos Zafeiriou. Distribution matching for multi-task learning of classification tasks: a large-scale study on faces & beyond. InProceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pages 2813–2821, 2024. 3

    2024

  29. [37]

    Behaviour4all: in-the-wild facial be- haviour analysis toolkit.arXiv preprint arXiv:2409.17717,

    Dimitrios Kollias, Chunchang Shao, Odysseus Kaloidas, and Ioannis Patras. Behaviour4all: in-the-wild facial be- haviour analysis toolkit.arXiv preprint arXiv:2409.17717,

    arXiv

  30. [38]

    Multimodal machine learning: A survey and tax- onomy.IEEE transactions on pattern analysis and machine intelligence, 41(2):423–443, 2018

    Tadas Baltru ˇsaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and tax- onomy.IEEE transactions on pattern analysis and machine intelligence, 41(2):423–443, 2018. 3, 4

    2018

  31. [39]

    Dvd: A comprehensive dataset for ad- vancing violence detection in real-world scenarios.arXiv preprint arXiv:2506.05372, 2025

    Dimitrios Kollias, Damith C Senadeera, Jianian Zheng, Kaushal KK Yadav, Greg Slabaugh, Muhammad Awais, and Xiaoyun Yang. Dvd: A comprehensive dataset for ad- vancing violence detection in real-world scenarios.arXiv preprint arXiv:2506.05372, 2025. 3

    2025 arXiv

  32. [40]

    On informa- tion and sufficiency.The annals of mathematical statistics, 22(1):79–86, 1951

    Solomon Kullback and Richard A Leibler. On informa- tion and sufficiency.The annals of mathematical statistics, 22(1):79–86, 1951. 3, 5

    1951

  33. [41]

    Recurrent neural network based language model

    Tomas Mikolov, Martin Karafi ´at, Lukas Burget, Jan Cer- nock`y, and Sanjeev Khudanpur. Recurrent neural network based language model. InInterspeech, volume 2, pages 1045–1048. Makuhari, 2010. 3

    2010

  34. [42]

    Long short-term memory.Neural computation, 9(8):1735–1780, 1997

    Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory.Neural computation, 9(8):1735–1780, 1997. 3

    1997

  35. [43]

    Empirical evaluation of gated recurrent neural networks on sequence modeling.arXiv preprint arXiv:1412.3555, 2014

    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling.arXiv preprint arXiv:1412.3555, 2014. 3, 4

    2014 arXiv

  36. [44]

    Temporal modeling matters: A novel temporal emotional modeling approach for speech emotion recognition

    Jiaxin Ye, Xin-Cheng Wen, Yujie Wei, Yong Xu, Kunhong Liu, and Hongming Shan. Temporal modeling matters: A novel temporal emotional modeling approach for speech emotion recognition. InICASSP 2023-2023 IEEE interna- tional conference on acoustics, speech and signal process- ing...

    2023

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.