Pith. sign in

REVIEW 4 major objections 4 minor 42 references

GIA-MIC: Multimodal Emotion Recognition with Gated Interactive Attention and Modality-Invariant Learning Constraints

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GIA-MIC combines gated interactive cross-modal attention with a modality-invariant generator and symmetric-KL alignment constraints, reaching 80.7% weighted accuracy on IEMOCAP.

desk verdict Plausible architecture with a coherent design, but the key regularizer is mathematically undefined as written and the SOTA claim rests on a single run without code or error bars—send back for major revision, not desk reject. read the letter →

arxiv 2506.00865 v1 pith:3VQZKJTA submitted 2025-06-01 cs.AI cs.LG

Jiajun He , Jinyi Mi , Tomoki Toda This is my paper
classification cs.AIcs.LG
keywords multimodalemotionrecognitiongatedinteractiveattentionmodality-invariantrepresentationcross-modalfusionIEMOCAPsymmetricKLdivergencepretrainedencodersclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal emotion recognition fuses video, speech, and text to infer emotion, but modalities are heterogeneous and hard to align. This paper proposes a single framework that learns two complementary things at once: modality-specific features through a gated interactive attention (GIA) block that weighs each cross-modal interaction, and modality-invariant features through a shared-attention generator (MIG) constrained by a symmetric Kullback-Leibler divergence between modalities. On the IEMOCAP benchmark with ground-truth transcripts the method reports a weighted accuracy of 80.7% and an unweighted accuracy of 81.3%, above the previous best of 79.5%/80.3%. The paper argues that separating what is unique to each modality from what is common across modalities is the reason for the gain, and shows ablations where removing any of the three components lowers accuracy.

What carries the argument

The load-bearing objects are the gated interactive attention (GIA) block and the modality-invariant generator (MIG) with the modality-invariant constraint (MIC). The GIA block computes cross-attention from modality A to modality B, then applies a sigmoid gate G to the attended output: $H_A^{(\mathrm{GIA})_B} = G \odot H_{A\to B} + (1-G) \odot H_A$, so the representation can stay close to the original or absorb the other modality. The MIG block concatenates all three preliminary embeddings into one query $H_{VST}$, uses each modality's MSR as key and value, and filters the attended result through a 1$\times$1 convolution mask with a residual connection. The MIC loss is the symmetric KL divergence $D_{SKL}(H_M^{(\mathrm{MIR})}, H_N^{(\mathrm{MIR})})$ summed over the three modality pairs, and is added to the classification loss as $L = L_{ER} + \gamma L_{MIR}$. These components are what the paper claims carry the performance gain.

What would settle it

Take a trained or untrained GIA-MIC model, extract the MIR matrices $H_V^{(\mathrm{MIR})}, H_S^{(\mathrm{MIR})}, H_T^{(\mathrm{MIR})}$ from any batch, and evaluate Eq. (14) exactly as written. If any entry of $P/Q$ is negative or the sums are not 1, the logarithm is undefined or the value is not a divergence; the result would be NaN or a number that changes with arbitrary scaling. Alternatively, replace the SKL term with a properly normalized divergence (softmax then KL, or a cosine-based distance) and retrain; if accuracy does not drop, the claimed role of the SKL constraint is not substantiated.

Watch

Extended reading notes

Core claim

The central claim is that emotion recognition improves when the model explicitly separates modality-specific from modality-invariant information, and then aligns the invariant streams. Modality-specific representations come from gated cross-attention: for each pair of modalities, one modality attends to the other, and a learned sigmoid gate interpolates between the cross-modal output and the original modality features, so the model can decide how much each modality should borrow from each other. Modality-invariant representations come from a generator that uses the concatenation of all three modalities as query and each modality's specific representation as key/value, then applies a convolutional mask and a residual connection. A symmetric KL-divergence loss between every pair of modality-invariant outputs pushes the three invariant streams toward the same distribution. The paper demonstrates on IEMOCAP that this two-stream design with the alignment loss reaches 80.7% WA and 81.3% UA with ground-truth transcripts, and 79.6%/80.3% with ASR transcripts, outperforming prior published methods.

Load-bearing premise

The symmetric KL divergence in Eqs. (14)-(15) is applied directly to the real-valued representation matrices $H_M^{(\mathrm{MIR})}$, but KL divergence is only defined for probability distributions; the paper does not specify a normalization such as softmax, so if those matrices contain negative entries or do not sum to one, the MIC loss is not mathematically well-defined and the reported benefit of that constraint is unsupported.

Editorial extensions

If this is right

  • If the GIA-MIC numbers hold, the new state of the art on IEMOCAP is 80.7% WA and 81.3% UA with ground-truth transcripts, and 79.6%/80.3% with ASR transcripts, improving on the previous best of 79.5%/80.3%.
  • The gating mechanism suggests a general recipe: each modality decides how much to borrow from each other modality rather than fixed cross-attention, which can transfer to other multimodal tasks beyond emotion.
  • The ablations show that modality-specific information matters most, since removing MSR costs 1.1% WA, implying that future fusion methods should preserve per-modality identity alongside alignment.
  • Because the model works with ASR transcripts at 79.6% WA, the approach is usable without manual transcription, though with a gap of about 1.1 points compared to ground-truth text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An editor's check: the MIC loss is written as KL divergence on real-valued MIR matrices, which is undefined without a normalization like softmax; if the intended object is softmax-normalized probabilities, the published equations omit that step, and the alignment benefit could be reproduced with any well-defined distributional divergence.
  • The gated interpolation in Eq. (4) could let the model behave almost unimodally when a modality is uninformative, which is an implicit robustness property not tested in this paper.
  • A testable extension is to apply the same MSR/MIR split and SKL constraint to other multimodal benchmarks such as sentiment or humor detection; if the gains transfer, the separation of modality-specific and modality-invariant features is a general principle, and if not, the improvement may be specific to IEMOCAP's four-emotion setup.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript proposes GIA-MIC, a multimodal emotion recognition framework for IEMOCAP. It combines a gated interactive attention (GIA) mechanism for pairwise cross-modal fusion, a modality-invariant generator (MIG), and a modality-invariant constraint (MIC) based on symmetric KL divergence. The final representation is formed from modality-specific and modality-invariant parts and passed to a softmax classifier. The paper reports state-of-the-art results with ground-truth transcripts (WA 80.7%, UA 81.3% in Table 1), ablation studies, and a t-SNE visualization.

Significance. If fully specified and reproducible, the method would offer a modest but useful contribution: a gated fusion mechanism that adaptively weights cross-modal interactions and an invariant constraint to align representations. The experimental protocol (5-fold leave-one-session-out on IEMOCAP, comparison with recent SOTA) is appropriate and the paper is clearly motivated. However, the current manuscript contains several load-bearing specification gaps: the MIC loss is mathematically undefined as written, the fusion equation has incompatible dimensions, and the reported accuracy gains are not accompanied by variance estimates or significance tests. These issues currently prevent the empirical claim from being verified. No code or checkpoints are provided.

major comments (4)
  1. [Section 2.3.2, Eqs. (14)-(16)] The symmetric KL divergence is applied to H^(MIR)_M and H^(MIR)_N, which are real-valued feature matrices produced by layer normalization and 1x1 convolutions in Eqs. (10)-(12). D_KL is defined only for probability distributions over a common index set, and no flattening, softmax, or other normalization is specified. Thus L_MIR cannot be evaluated as written: the logarithm of a negative or zero ratio is undefined, and even for positive entries the quantity is not a KL divergence between distributions. Since L_MIR is part of the joint objective in Eq. (19), the training procedure cannot be executed from the manuscript alone, and the ablation evidence in Table 2 and the t-SNE visualization in Fig. 2 do not provide support for the MIC contribution as described. Please specify the exact distributional form (e.g., softmax over flattened vectors per sample or per batch), or replace the loss with a well-defined alignment measure.
  2. [Sections 2.3.1-2.3.2, Eqs. (7), (12), (13)] Equation (13) states H^(fus)_VST = H^(MSR) + H^(MIR) ∈ R^{4(k+m+n)×d}, but H^(MSR) is defined in Eq. (7) as an element of R^{(k+m+n)×d}, while H^(MIR) is defined in Eq. (12) as an element of R^{3(k+m+n)×d}. These shapes cannot be added. If concatenation is intended, the operation and the resulting dimension should be stated consistently; if addition is intended, the individual shapes must match. As written, the forward pass into the classifier (Eq. (17)) is undefined.
  3. [Section 3.2, Tables 1 and 2] All results are reported as point estimates. The 5-fold leave-one-session-out protocol has known session-dependent variance, and the claimed SOTA margins over CAT-BC (about 1.2 WA and 1.0 UA) as well as the MIC ablation gain (0.5 WA) are small. Without per-fold numbers, standard deviations across runs, or significance tests, the 'outperforms SOTA' claim is not statistically supported. Please report run-level variability and, if possible, significance tests for the main comparisons and ablations.
  4. [Section 2.3.2, Eq. (10)] Equation (10) concatenates H^(MSR)_M, whose sequence length is t_M (e.g., k for video), with HVST, whose sequence length is k+m+n, using the notation [H^(MSR)_M, HVST]. The concatenation axis and any required alignment, padding, or upsampling are unspecified, so the input shape to Conv1d is ambiguous. Please clarify the operation or provide the exact tensor shapes before and after this step.
minor comments (4)
  1. [Section 3.2] The body text reports 'UA of 81.1%' while Table 1 and the abstract report 81.3%; please reconcile these numbers.
  2. [References] Reference [6] and reference [28] are the same paper (Wu et al., ICASSP 2024); please deduplicate.
  3. [Figure 2] The t-SNE visualization should state which representation layer is plotted, how many points are used, and the t-SNE hyperparameters; t-SNE can create visual separation regardless of true alignment, so a quantitative alignment metric (e.g., cosine similarity or MMD) would strengthen the claim.
  4. [Reproducibility] No code, checkpoints, or detailed training hyperparameters (e.g., learning rate schedule, number of epochs, random seeds) are provided; please include these to support reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported accuracies are direct held-out empirical results, and no load-bearing claim reduces by construction to its own inputs.

full rationale

The paper's central claims are empirical: GIA-MIC achieves WA 80.7% and UA 81.3% on IEMOCAP under 5-fold leave-one-session-out cross-validation (Table 1), and ablations (Table 2) show performance drops when components are removed. These are direct test-set measurements, not quantities derived from the model's own definitions. The modality-invariant learning constraint (MIC) in Eqs. (14)-(16) is a regularizer added to the training objective; it does not by construction force the reported accuracy values, and the hyperparameter gamma is a fixed scalar (0.1), not a fitted parameter tuned to reproduce the test numbers. The paper cites several of its own prior works (references 1, 5, 8, 10, 26, 27), but these are used as related work or baselines and are not load-bearing for the claimed SOTA result; there is no invocation of a 'uniqueness theorem' or other self-citation chain that determines the architecture or the outcome. The mathematical ill-posedness of applying KL divergence to real-valued feature matrices noted in the review is a correctness/reproducibility concern about the loss definition, not a circularity: it does not make the reported accuracy equivalent to an input assumption. No step in the paper fits the enumerated circularity patterns, so the appropriate finding is no significant circularity with score 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim depends on three hyperparameters, a mathematically questionable loss definition, and the assumption that pretrained features and public benchmark numbers are reliable. No new physical entities are introduced.

free parameters (3)
  • gamma (loss balance weight) = 0.1
    Hand-chosen in Eq. (19) to balance L_ER and L_MIR; affects the strength of the modality-invariant constraint.
  • learning rate = 1e-5
    Fixed Adam learning rate reported in Section 3.1; standard choice but not justified by tuning.
  • batch size = 32
    Reported in Section 3.1; affects optimization dynamics.
assumptions (3)
  • ad hoc to paper The KL divergence in Eq. (14)-(15) is a valid similarity measure for the representation vectors H_M^(MIR)
    The vectors are outputs of linear and convolutional layers and may contain negative values; no softmax or other normalization to probability space is specified, so the loss is not well-defined as written.
  • domain assumption Pretrained CLIP, WavLM, and RoBERTa features provide useful emotion cues
    The whole pipeline rests on these encoders; if these features do not carry emotion information, the downstream fusion cannot recover it.
  • domain assumption Leave-one-session-out cross-validation produces results directly comparable to published baselines
    The SOTA claim depends on comparing absolute numbers across papers without re-running baselines, which assumes the protocol and splits are equivalent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GIA-MIC: Multimodal Emotion Recognition with Gated Interactive Attention and Modality-Invariant Learning Constraints." pith.science (2026). https://pith.science/paper/3VQZKJTA

@misc{pith2026250600865,
  author       = {Pith},
  title        = {Pith review of: GIA-MIC: Multimodal Emotion Recognition with Gated Interactive Attention and Modality-Invariant Learning Constraints},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3VQZKJTA}},
  note         = {Machine review of arXiv:2506.00865}
}
read the original abstract

Multimodal emotion recognition (MER) extracts emotions from multimodal data, including visual, speech, and text inputs, playing a key role in human-computer interaction. Attention-based fusion methods dominate MER research, achieving strong classification performance. However, two key challenges remain: effectively extracting modality-specific features and capturing cross-modal similarities despite distribution differences caused by modality heterogeneity. To address these, we propose a gated interactive attention mechanism to adaptively extract modality-specific features while enhancing emotional information through pairwise interactions. Additionally, we introduce a modality-invariant generator to learn modality-invariant representations and constrain domain shifts by aligning cross-modal similarities. Experiments on IEMOCAP demonstrate that our method outperforms state-of-the-art MER approaches, achieving WA 80.7% and UA 81.3%.

Figures

Figures reproduced from arXiv: 2506.00865 by the authors.

Figure 1
Figure 1. The overall architecture of our proposed method. 2.2. Embedding Module We utilize pretrained encoders CLIP, WavLM, and RoBERTa to extract features from the visual, speech, and text modalities, re￾spectively. To ensure consistency in feature dimensions across modalities, we apply a linear transformation layer to the speech features. These extracted features are then passed through a Feature Extractor, which consists … view at source ↗
Figure 2
Figure 2. t-SNE visualizations of the distribution of modality￾invariant representations before and after introducing the modality-invariant learning constraints. a 0.6% WA drop when removed, suggesting its role in cross￾modal alignment. Although MIC has the smallest effect, remov￾ing it leads to a 0.5% WA decrease, indicating its contribution to refining learned representations through consistency across modalities. Overall,… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 35 canonical work pages

  1. [1]

    MER has broad applications in fields such as human- computer interaction and intelligent customer service [1]

    Introduction Multimodal emotion recognition (MER) aims to leverage in- formation from multiple perceptual modalities, such as visual, acoustic, and textual expressions, to accurately identify human emotions. MER has broad applications in fields such as human- computer interaction and intelligent customer service [1]. Although MER has demonstrated signific...

    work page

  2. [2]

    σ(·)” is the sigmoid activation function. The final MSR of modality A is computed as: H (GIA)B A =G⊙H A→B + (1−G)⊙H A ∈R tA×d,(4) where “⊙

    Proposed Method 2.1. Problem Formulation The MER task is defined asf(V, S, T) =L, whereV,S, and Trepresent the video, speech, and text modalities, respectively. The goal is to fuse these modalities for emotion classification, yieldingL∈ {l1, l2, . . . , le}, whereeis the number of emotion categories. arXiv:2506.00865v1 [cs.AI] 1 Jun 2025 Embedding WavLM R...

    arXiv 2025

  3. [3]

    neu- tral

    Experiments and Results 3.1. Experimental Conditions Experiment Settings.Our method was implemented with Python 3.10.0 and Pytorch 1.11.0 and was trained on a sys- tem with an Intel Xeon Gold 6248 CPU, 32GB RAM, and an NVIDIA Tesla V100 GPU. The visual, speech, and text en- coders were initialized using CLIP1, WavLM2, and RoBERTa3, 1https://huggingface.co...

    work page

  4. [4]

    In contrast, with temporal modality-invariant constraints (γ̸= 0), the three modalities exhibit greater overlap, indicating increased shared information

    Whenγ= 0, meaning no modality-invariant constraints are applied, the representations of different modalities remain dis- tinct. In contrast, with temporal modality-invariant constraints (γ̸= 0), the three modalities exhibit greater overlap, indicating increased shared information. This suggests that the constraints effectively enhance modality alignment, ...

    work page

  5. [5]

    Conclusion In this paper, we propose GIA-MIC, a novel multimodal emo- tion recognition framework that effectively integrates modality- invariant and modality-specific representations while enforc- ing cross-modal consistency through modality-invariant con- straints. Our approach addresses the challenges of modality heterogeneity and misalignment by learni...

    work page

  6. [6]

    Acknowledgements This work was partly supported by JST AIP Acceleration Research JPMJCR25U5 and JSPS KAKENHI Grant Number 21H05054, Japan

    work page

  7. [7]

    Semi-supervised multimodal emotion recognition with consensus decision-making and label correction,

    J. Tian, D. Hu, X. Shi, J. He, X. Li, Y . Gao, T. Toda, X. Xu, and X. Hu, “Semi-supervised multimodal emotion recognition with consensus decision-making and label correction,” inProc. MRAC, 2023, pp. 67–73

    work page 2023

  8. [8]

    Group gated fusion on attention- based bidirectional alignment for multimodal emotion recogni- tion,

    P. Liu, K. Li, and H. Meng, “Group gated fusion on attention- based bidirectional alignment for multimodal emotion recogni- tion,” inProc. Interspeech, 2020, pp. 379–383

    work page 2020

Show all 42 references
  1. [9]

    Speech emotion recognition using recurrent neural networks with directional self- attention,

    D. Li, J. Liu, Z. Yang, L. Sun, and Z. Wang, “Speech emotion recognition using recurrent neural networks with directional self- attention,”Expert Systems with Applications, vol. 173, p. 114683, 2021

    2021

  2. [10]

    Rep- resentation learning with spectro-temporal-channel attention for speech emotion recognition,

    L. Guo, L. Wang, C. Xu, J. Dang, E. S. Chng, and H. Li, “Rep- resentation learning with spectro-temporal-channel attention for speech emotion recognition,” inProc. ICASSP, 2021, pp. 6304– 6308

    2021

  3. [11]

    Two- stage framework for robust speech emotion recognition using tar- get speaker extraction in human speech noise conditions,

    J. Mi, X. Shi, D. Ma, J. He, T. Fujimura, and T. Toda, “Two- stage framework for robust speech emotion recognition using tar- get speaker extraction in human speech noise conditions,” inProc. APSIPA ASC, 2024

    2024

  4. [13]

    Joint multi- modal transformer for emotion recognition in the wild,

    P. Waligora, M. H. Aslam, M. O. Zeeshan, S. Belharbi, A. L. Koerich, M. Pedersoli, S. Bacon, and E. Granger, “Joint multi- modal transformer for emotion recognition in the wild,” inProc. CVPRW, 2024, pp. 4625–4635

    2024

  5. [14]

    Multi-modal video summarization based on two-stage fusion of audio, visual, and recognized text information,

    Z. Yang, J. He, and T. Toda, “Multi-modal video summarization based on two-stage fusion of audio, visual, and recognized text information,” inProc APSIPA ASC, 2024, pp. 1–6

    2024

  6. [15]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agar- wal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” in Proc. ICML, 2021, pp. 8748–8763

    2021

  7. [16]

    2DP-2MRC: 2-dimensional pointer-based ma- chine reading comprehension method for multimodal moment re- trieval,

    J. He and T. Toda, “2DP-2MRC: 2-dimensional pointer-based ma- chine reading comprehension method for multimodal moment re- trieval,” inProc. Interspeech, 2024, pp. 5073–5077

    2024

  8. [17]

    Vision trans- formers need registers,

    T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski, “Vision trans- formers need registers,” inProc. ICLR, 2024

    2024

  9. [18]

    HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhut- dinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 29, pp. 3451–3460, 2021

    2021

  10. [19]

    WavLM: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y . Qian, Y . Qian, M. Zeng, and F. Wei, “WavLM: Large-scale self- supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Si...

    2021

  11. [20]

    DeBERTa: Decoding- enhanced bert with disentangled attention,

    P. He, X. Liu, J. Gao, and W. Chen, “DeBERTa: Decoding- enhanced bert with disentangled attention,” inProc. ICLR, 2021

    2021

  12. [21]

    RoBERTa: a ro- bustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “RoBERTa: a ro- bustly optimized bert pretraining approach,” inProc. ICLR, 2020

    2020

  13. [22]

    Multi-modal speech emotion recognition using self-attention mechanism and multi-scale fusion framework,

    Y . Liu, H. Sun, W. Guan, Y . Xia, and Z. Zhao, “Multi-modal speech emotion recognition using self-attention mechanism and multi-scale fusion framework,”Speech Communication, vol. 139, pp. 1–9, 2022

    2022

  14. [23]

    Recursive joint cross-modal atten- tion for multimodal fusion in dimensional emotion recognition,

    R. G. Praveen and J. Alam, “Recursive joint cross-modal atten- tion for multimodal fusion in dimensional emotion recognition,” inProc. CVPRW, 2024, pp. 4803–4813

    2024

  15. [24]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,

    K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” inProc. ICCV, 2015, pp. 1026–1034

    2015

  16. [25]

    Layer normalization,

    J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv:1607.06450, 2016

    2016 arXiv

  17. [26]

    Adam: A method for stochastic opti- mization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic opti- mization,” inProc. ICLR, 2015, pp. 7–9

    2015

  18. [27]

    IEMOCAP: Interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “IEMOCAP: Interactive emotional dyadic motion capture database,”Language resources and evaluation, vol. 42, pp. 335–359, 2008

    2008

  19. [28]

    Robust speech recognition via large-scale weak su- pervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” inProc. ICML, 2023, pp. 28 492–28 518

    2023

  20. [29]

    Robust multi-modal speech emotion recog- nition with ASR error adaptation,

    B. Lin and L. Wang, “Robust multi-modal speech emotion recog- nition with ASR error adaptation,” inProc. ICASSP, 2023, pp. 1–5

    2023

  21. [30]

    MGAT: Multi-granularity attention based transformers for multi-modal emotion recogni- tion,

    W. Fan, X. Xing, B. Cai, and X. Xu, “MGAT: Multi-granularity attention based transformers for multi-modal emotion recogni- tion,” inProc. ICASSP, 2023, pp. 1–5

    2023

  22. [31]

    Inter-modality and intra- sample alignment for multi-modal emotion recognition,

    Y . Wang, D. Li, and J. Shen, “Inter-modality and intra- sample alignment for multi-modal emotion recognition,” inProc. ICASSP, 2024, pp. 8301–8305

    2024

  23. [32]

    A study on multimodal fusion and layer adapter in emotion recognition,

    X. Shi, Y . Gao, J. He, J. Mi, X. Li, and T. Toda, “A study on multimodal fusion and layer adapter in emotion recognition,” in Proc. APSIPA ASC, 2024, pp. 1–6

    2024

  24. [33]

    MF-AED-AEC: Speech emo- tion recognition by leveraging multimodal fusion, ASR error de- tection, and ASR error correction,

    J. He, X. Shi, X. Li, and T. Toda, “MF-AED-AEC: Speech emo- tion recognition by leveraging multimodal fusion, ASR error de- tection, and ASR error correction,” inProc. ICASSP, 2024, pp. 11 066–11 070

    2024

  25. [34]

    Multi-modal emotion recognition using multiple acoustic features and dual cross-modal transformer,

    Y . Wu, P. Yue, L. Qu, T. Li, and Y .-P. Ruan, “Multi-modal emotion recognition using multiple acoustic features and dual cross-modal transformer,” inProc. ICASSP, 2024, pp. 10 496–10 500

    2024

  26. [35]

    Fine- grained disentangled representation learning for multimodal emo- tion recognition,

    H. Sun, S. Zhao, X. Wang, W. Zeng, Y . Chen, and Y . Qin, “Fine- grained disentangled representation learning for multimodal emo- tion recognition,” inProc. ICASSP, 2024, pp. 11 051–11 055

    2024

  27. [36]

    Coordination attention based transformers with bidirectional contrastive loss for multimodal speech emotion recognition,

    W. Fan, X. Xu, G. Zhou, X. Deng, and X. Xing, “Coordination attention based transformers with bidirectional contrastive loss for multimodal speech emotion recognition,”Speech Communi- cation, pp. 103 198–103 207, 2025

    2025

  28. [37]

    Multi- channel weight-sharing autoencoder based on cascade multi-head attention for multimodal emotion recognition,

    J. Zheng, S. Zhang, Z. Wang, X. Wang, and Z. Zeng, “Multi- channel weight-sharing autoencoder based on cascade multi-head attention for multimodal emotion recognition,”IEEE Transac- tions on Multimedia, vol. 25, pp. 2213–2225, 2022

    2022

  29. [38]

    GCNet: Graph completion network for incomplete multimodal learning in con- versation,

    Z. Lian, L. Chen, L. Sun, B. Liu, and J. Tao, “GCNet: Graph completion network for incomplete multimodal learning in con- versation,”IEEE Transactions on pattern analysis and machine intelligence, vol. 45, no. 7, pp. 8419–8432, 2023

    2023

  30. [39]

    Enhancing modal fusion by alignment and label matching for multimodal emotion recognition,

    Q. Li, Y . Gao, Y . Wen, C. Wang, and Y . Li, “Enhancing modal fusion by alignment and label matching for multimodal emotion recognition,” inProc. Interspeech, 2024, pp. 4663–4667

    2024

  31. [40]

    Semi-supervised multi-modal emo- tion recognition with cross-modal distribution matching,

    J. Liang, R. Li, and Q. Jin, “Semi-supervised multi-modal emo- tion recognition with cross-modal distribution matching,” inProc. ACM-MM, 2020, pp. 2852–2861

    2020

  32. [41]

    Adaptive multimodal emotion detection architecture for social robots,

    J. Heredia, E. Lopes-Silva, Y . Cardinale, J. Diaz-Amado, I. Dongo, W. Graterol, and A. Aguilera, “Adaptive multimodal emotion detection architecture for social robots,”IEEE Access, vol. 10, pp. 20 727–20 744, 2022

    2022

  33. [42]

    A framework to evaluate fusion methods for multimodal emotion recognition,

    D. Pe ˜na, A. Aguilera, I. Dongo, J. Heredia, and Y . Cardinale, “A framework to evaluate fusion methods for multimodal emotion recognition,”IEEE Access, vol. 11, pp. 10 218–10 237, 2023

    2023

  34. [43]

    Visualizing data using t-sne,

    L. van der Maaten and G. Hinton, “Visualizing data using t-sne,” Journal of Machine Learning Research, vol. 9, no. 86, pp. 2579– 2605, 2008

    2008

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.