Semi-IIN: Semi-supervised Intra-inter modal Interaction Learning Network for Multimodal Sentiment Analysis

Jinhao Lin; Qi Liu; Yanwu Xu; Yifei Wang

REVIEW 5 major objections 5 minor 35 references

Semi-IIN: Semi-supervised Intra-inter modal Interaction Learning Network for Multimodal Sentiment Analysis

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 1.6M-parameter network tops prior multimodal sentiment models.

desk verdict A clean two-branch masked attention architecture with a learned gate, but the self-training stage's confidence filter is vacuous on MOSEI and the SOTA claim rests on an unfair comparison. read the letter →

arxiv 2412.09784 v1 pith:S34FPURO submitted 2024-12-13 cs.CL cs.AI

Jinhao Lin , Yifei Wang , Yanwu Xu , Qi Liu This is my paper · ORCID

classification cs.CLcs.AI

keywords multimodalsentimentanalysissemi-supervisedlearningmaskedattentionintra-modalinteractioninter-modalself-trainingCMU-MOSICMU-MOSEI

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that multimodal sentiment analysis can be improved by explicitly separating intra-modal (within-modality) and inter-modal (across-modality) attention, then letting a learned gate decide per sample how much of each to trust. It adds a self-training stage that uses high-confidence pseudo-labels from an unlabeled meeting corpus to reduce dependence on human annotation. On CMU-MOSI and CMU-MOSEI, the combination beats previous state-of-the-art systems, including a RoBERTa-large baseline, while using far fewer trainable parameters. The broader point is that different utterances need different interaction patterns, so a fixed fusion recipe is suboptimal.

What carries the argument

The load-bearing object is a pair of attention masks added to the standard attention score $QK^T/\sqrt{d_k}$ before the softmax: $\mathrm{IntraMASK}_{ij}=0$ when tokens $i$ and $j$ belong to the same modality and $-\infty$ otherwise, while $\mathrm{InterMASK}_{ij}=0$ when they belong to different modalities and $-\infty$ otherwise. This forces the network to compute, in separate units (IntraMAU and InterMAU), a within-modality context and a cross-modality context. A sigmoid gate with per-sample inputs then mixes the two fused representations, dynamically choosing how much modality-specific versus modality-complementary knowledge to use.

What would settle it

Run the same pipeline with pseudo-labels replaced by labels from a completely wrong task (or shuffled labels) and compare final accuracy; if the gain over the no-self-training baseline persists, the claimed benefit is not coming from meaningful pseudo-labels. Alternatively, sweep the confidence threshold k smoothly and show that tiny perturbations erase the improvement, which would indicate the result hinges on a fragile choice rather than a robust mechanism.

Watch

Extended reading notes

Core claim

On its own terms, Semi-IIN establishes that a transformer-style architecture with two separate masked attention paths—one attending only within each modality (IntraMA) and one attending only across modalities (InterMA)—captures cleaner emotional cues than unrestricted global attention. A sigmoid gate blends the two paths' fused outputs, and the gate is learned per sample. Combined with self-training that keeps only top-k confident pseudo-labels (k=40 for MOSI, all unlabeled instances for MOSEI), the model reaches state-of-the-art numbers: MAE 0.679, Corr 0.822, Acc-7 46.50 on MOSI, and MAE 0.497, Corr 0.804, Acc-7 55.89 on MOSEI, surpassing the RoBERTa-large baseline TCDN on both datasets with only 1.6M trainable parameters.

Load-bearing premise

The self-training stage assumes that the top-k most confident predictions generated on the AMI meeting corpus are accurate enough to improve a sentiment model trained on YouTube review videos, despite AMI having no sentiment labels and a different domain.

Editorial extensions

If this is right

Explicitly separating intra- and inter-modal attention and gating their combination improves multimodal sentiment analysis over a single global attention baseline.
The self-training strategy, with top-k confidence pseudo-labels from AMI, improves performance even when a small fraction of labeled data is available.
The best configuration adds only 0.3M parameters over the baseline and still outperforms it, showing that targeted masking, not model size, drives the gain.
Strong results with only 1.6M parameters and short training times (90 seconds on MOSI, 20 minutes on MOSEI) suggest the design is practical for larger-scale multimodal tasks.
The gated dual-branch design offers a new point in the design space between full cross-modal fusion and strict unimodal processing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The learned gate could be studied as a per-sample router; one testable hypothesis is that gate values correlate with annotation ambiguity or with which modality carries the dominant emotional signal.
The top-k confidence self-training on AMI is sensitive to domain shift, so ablation with alternative unlabeled corpora or confidence thresholds could isolate whether the gain comes from pseudo-label content or simply from retraining on more data.
The masked-attention-plus-gate pattern may transfer to other multimodal tasks where some inputs are noisy, such as emotion recognition in the wild, where the model can learn to ignore a corrupted modality.
The reported performance drop at 50% and 75% labeled-data ratios suggests that pseudo-label balance matters; a confidence-threshold curriculum might stabilize and extend the self-training gains.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea is a clean extension: two masked attention branches (intra- and inter-modal) with a learned gate, plus self-training on unlabeled data. That's a legitimate contribution, and the paper does it honestly in the sense that the ablations are thorough. Table 2 shows both the masked attention and the semi-supervised stage contribute, and Table 3 shows the gate beats other fusion modes. The authors also release code, which is good practice.

The problems are empirical and concentrated in the self-training stage. The most serious is the MOSEI setting: the paper says k is set to the total number of unlabeled instances. That's not a top-k filter; it's a no-op. Algorithm 1's line 7 promises "top-k highest confidence in each category," but for MOSEI every pseudo-label is used. The rationale given—that MOSEI predictions are more confident—doesn't justify abandoning the filter. And there's no evidence that AMI meeting data yields reliable sentiment pseudo-labels for YouTube review videos. The domain shift is real, and no validation of pseudo-label quality is provided.

Second, the comparison against TCDN is apples-to-oranges. Semi-IIN uses extra unlabeled AMI data; TCDN is supervised. The gains over TCDN are a few hundredths in MAE and ~0.5% accuracy, exactly where the self-training stage matters. To support a SOTA claim, you'd need either a supervised baseline trained with the same self-training setup or an analysis of how much of the gain is from the unlabeled data alone.

Third, Figure 4 shows performance dropping when labeled data ratio increases to 50-75%. That's unexplained and worrying. The paper's hand-wave about "lack of balance in fake samples" doesn't cut it. It suggests the semi-supervised benefit is not robust.

No error bars or significance tests are reported, which is common but still a soft spot for a "new SOTA" claim.

Who is this for? Researchers working on multimodal fusion or semi-supervised sentiment analysis who want a baseline to compare against. The architecture is worth knowing about, but the empirical evidence is not yet convincing enough to trust the headline result.

My recommendation: send it to peer review, but expect major revision. The authors need to justify or fix the k=total choice, provide pseudo-label quality analysis, and run a fairer comparison. As is, the SOTA claim should be softened.

Referee Report

5 major / 5 minor

Summary. The paper proposes Semi-IIN, a semi-supervised multimodal sentiment analysis network. The architecture combines two masked attention mechanisms (IntraMA and InterMA) to capture intra-modal and inter-modal token interactions separately, a gating mechanism to fuse the two branches, and a self-training stage in which a model trained on labeled MOSI/MOSEI data generates pseudo-labels for the unlabeled AMI meeting corpus with a top-k confidence selection. Experiments on CMU-MOSI and CMU-MOSEI report improved metrics over existing methods, including the RoBERTa-large baseline TCDN, and the conclusion states that Semi-IIN sets a new state of the art for MSA. Code is provided and supplementary comparisons with MMML are included.

Significance. The masked-attention design is a clean and lightweight contribution: explicitly separating intra- and inter-modal attention masks with a learned gate is a reasonable way to model token-level interactions, and the paper offers ablations, case studies, and attention visualizations. The reported training efficiency and public code are also strengths. If the performance claims survive a controlled comparison, the method would be a useful addition to multimodal sentiment analysis. However, the central empirical claim is not currently established because the main comparison is uncontrolled with respect to the use of additional unlabeled data, the pseudo-label confidence filter is vacuous for MOSEI, and the claimed SOTA is contradicted by the paper's own supplementary MMML results on some metrics.

major comments (5)

[Experiment, Table 1] The headline comparison in Table 1 is not controlled for the use of additional unlabeled data. Semi-IIN is trained with a self-training stage on the AMI corpus, whereas TCDN and the other baselines are fully supervised. The reported improvements over TCDN (0.016 MAE and 0.017 Corr on MOSI; 0.024 MAE and 0.45% Acc-2 on MOSEI) could therefore come from the extra unlabeled corpus rather than from the proposed architecture. The paper should include baselines trained under the same semi-supervised protocol, or reframe the contribution as a semi-supervised method and compare against semi-supervised baselines only.
[Self-training, Algorithm 1] For MOSEI, k is set to the total number of unlabeled instances, so the top-k confidence filter in Algorithm 1 (line 7) selects all pseudo-labeled samples and performs no confidence-based filtering. This makes the claimed 'dependable pseudo-labels' mechanism vacuous for the larger dataset, and the MOSEI results in Table 1 depend entirely on unvalidated pseudo-labels from AMI, a meeting corpus with a different domain and no sentiment annotations. Please report pseudo-label accuracy on a held-out labeled subset and perform a k-sweep (for example k=20, 40, 60, and all) with the corresponding test metrics.
[Dataset and Self-training sections] The use of AMI as the unlabeled corpus is underspecified. The paper does not describe how the 100 hours of meeting recordings are segmented into samples, whether the same feature extractors (RoBERTa, Fabnet, HuBERT) are applied to AMI data, or how many pseudo-labeled samples are added for each class. Without these details and without evidence that the pseudo-labels are reliable under the domain shift from YouTube review videos to meeting recordings, the self-training contribution cannot be evaluated. Figure 4 compounds this concern: performance drops when the labeled-data ratio is 50% or 75%, so the semi-supervised gain is not robust across labeled-data proportions.
[Supplementary Tables 4-5 and Conclusion] The conclusion states that Semi-IIN 'sets the new SOTA result for MSA', but the paper's own supplementary comparison shows MMML achieving higher Corr (0.846 vs 0.822) and Acc-7 (47.48 vs 46.50) on MOSI. The SOTA claim should be restricted to the metrics and settings where Semi-IIN actually leads, and the comparison with MMML should be discussed in the main text, including the trade-offs in parameters and training time.
[Experiment, Comparison to State-of-the-art Methods] No error bars, standard deviations, or significance tests are reported for any of the metrics in Table 1. Several improvements over the strongest baseline are small (e.g., 0.016 MAE, 0.45% accuracy), and without repeated runs or a paired significance test it is not possible to tell whether the differences are meaningful. Please report the mean and standard deviation over at least 3-5 seeds for the proposed method and the closest baselines, or provide statistical significance tests.

minor comments (5)

[References] The reference list appears twice: once after the Conclusion and once after the Supplementary Materials; please remove the duplicate.
[Algorithm 1] Algorithm 1 contains the typos 'caculate' in lines 3 and 11, and the variable n in the Feature fusion section (l ∈ [1, n]) is not defined before use.
[Equation (16)] Equation (16) uses the notation 'all ∈ Rdh×dh' and 'b1 and b'1 ∈ Rdh' without proper superscripts; the dimensionalities should be written as R^{dh×dh} and R^{dh}.
[Table 1] Table 1 abbreviates 'MulT' as 'Mult' for one baseline, and the caption does not explain the practical difference between the aligned and unaligned settings; please make the table self-contained.
[Figure 3] Figure 3 and the surrounding text refer to 'masked tokens' and 'unmasked tokens' without explicitly connecting them to the IntraMASK and InterMASK matrices defined in equations (11)-(12); please clarify this relationship.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Semi-IIN's reported improvements are empirical comparisons on held-out benchmarks, not derivations that reduce to the paper's inputs.

full rationale

Semi-IIN's central claims are empirical. The architecture is specified in Eqs. (1)-(18), trained with the loss functions in Eqs. (19)-(23), and evaluated on the held-out test splits of CMU-MOSI and CMU-MOSEI against published baselines in Table 1. No predicted quantity is defined in terms of the target metric, and no parameter is fitted to the test labels. The self-training component (Algorithm 1) generates pseudo-labels on the unlabeled AMI corpus and retrains the model, which is a standard semi-supervised procedure rather than a derivation that reduces to its own inputs; final metrics are measured on labeled test data, not on the pseudo-labels. The paper itself notes in Figure 4 that performance decreases at 50% and 75% labeled ratios, and the MOSEI setting of k equal to the total number of unlabeled instances makes the described top-k confidence filter vacuous for that dataset; these are legitimate validity and robustness concerns about pseudo-label quality and hyperparameter selection, but they are not circularity. There is no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled in via citation. Therefore no specific circular step can be exhibited, and the honest finding is no circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard supervised learning assumptions plus manually chosen hyperparameters. No invented entities are introduced. The most consequential assumption is that pseudo-labels from a domain-shifted unlabeled corpus are reliable enough for retraining, which is only weakly supported by the paper's ablations.

free parameters (6)

lambda1 regression loss weight = 0.6
Set manually in Implementation Details; no sweep or validation curve is shown.
lambda2 pseudo-label classification loss weight = 0.2
Set manually in Implementation Details; no sensitivity analysis is provided.
top-k threshold for MOSI pseudo-labels = 40
Chosen based on the authors' observation of model confidence; equivalent to a hyperparameter tuned to the setting.
top-k threshold for MOSEI pseudo-labels = total number of unlabeled instances
This means no filtering is applied, contradicting the top-k selection idea; the choice is justified only by observed confidence.
Conv1D kernel sizes kt, kv, ka1, ka2 = not reported
These sizes appear in equations (4)-(6) but are never specified in the text.
number of masked attention units n = not reported
The depth of the IntraMAU and InterMAU stacks is not stated.

assumptions (4)

domain assumption Pre-trained feature extractors RoBERTa, Fabnet, and HuBERT provide adequate token-level representations for text, vision, and audio.
Used in Feature encoding, equations (1)-(3); the method inherits the biases and limitations of these extractors.
domain assumption AMI meeting recordings are a suitable unlabeled source for sentiment learning on MOSI and MOSEI.
Used in Dataset and Self-training; the domain shift from meeting recordings to YouTube review videos is not quantified.
ad hoc to paper The pre-defined IntraMASK and InterMASK matrices correctly separate relevant intra-modal and inter-modal token relations.
Equations (11)-(12); the masks are manually designed and their correctness is not established independently of the reported results.
domain assumption Top-k confidence pseudo-labels are reliable enough for retraining.
Used in the Self-training algorithm; no confidence calibration or noise analysis is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Semi-IIN: Semi-supervised Intra-inter modal Interaction Learning Network for Multimodal Sentiment Analysis." pith.science (2026). https://pith.science/paper/S34FPURO

@misc{pith2026241209784,
  author       = {Pith},
  title        = {Pith review of: Semi-IIN: Semi-supervised Intra-inter modal Interaction Learning Network for Multimodal Sentiment Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S34FPURO}},
  note         = {Machine review of arXiv:2412.09784}
}

read the original abstract

Despite multimodal sentiment analysis being a fertile research ground that merits further investigation, current approaches take up high annotation cost and suffer from label ambiguity, non-amicable to high-quality labeled data acquisition. Furthermore, choosing the right interactions is essential because the significance of intra- or inter-modal interactions can differ among various samples. To this end, we propose Semi-IIN, a Semi-supervised Intra-inter modal Interaction learning Network for multimodal sentiment analysis. Semi-IIN integrates masked attention and gating mechanisms, enabling effective dynamic selection after independently capturing intra- and inter-modal interactive information. Combined with the self-training approach, Semi-IIN fully utilizes the knowledge learned from unlabeled data. Experimental results on two public datasets, MOSI and MOSEI, demonstrate the effectiveness of Semi-IIN, establishing a new state-of-the-art on several metrics. Code is available at https://github.com/flow-ljh/Semi-IIN.

Figures

Figures reproduced from arXiv: 2412.09784 by the authors.

**Figure 2.** The overall architecture of Semi-IIN. Notably, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Implementation of InterMA(top) and IntraMA(bottom). hvi = F abnet(Xvi; θ F abnet v ) ∈ R lv×dv (2) hai = HuBERT(Xai; θ HuBERT a ) ∈ R la×da (3) where hti, hvi, and hai represent features corresponding to the lexical, visual, and acoustic modalities for the i-th sample, respectively. Next, a module for local feature extraction is used, which includes an 1D convolutional neural network (Conv1D) with various recept… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Results under different proportions of labeled sam [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Comparison between different embedding on two [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Case study for the Semi-IIN. The “Only Intra” and the “Only Inter” refer to the stacked IntraMAU and InterMAU [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Visualization of IntraMA and InterMA mecha [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 20 canonical work pages

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Carletta, J.; Ashby, S.; Bourban, S.; Flynn, M.; Guillemot, M.; Hain, T.; Kadlec, J.; Karaiskos, V.; Kraaij, W.; Kronenthal, M.; et al. 2005. The AMI meeting corpus: A pre-announcement. In International workshop on machine learning for multimodal interaction, 28--39. Springer

work page 2005
[4]

Chen, C.; Hong, H.; Guo, J.; and Song, B. 2023 a . Inter-intra modal representation augmentation with trimodal collaborative disentanglement network for multimodal sentiment analysis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31: 1476--1488

work page 2023
[5]

Chen, H.; Guo, C.; Li, Y.; Zhang, P.; and Jiang, D. 2023 b . Semi-Supervised Multimodal Emotion Recognition with Class-Balanced Pseudo-labeling. In Proceedings of the 31st ACM International Conference on Multimedia, 9556--9560

work page 2023
[6]

Cheng, Z.; Lin, Y.; Chen, Z.; Li, X.; Mao, S.; Zhang, F.; Ding, D.; Zhang, B.; and Peng, X. 2023. Semi-supervised multimodal emotion recognition with expression mae. In Proceedings of the 31st ACM International Conference on Multimedia, 9436--9440

work page 2023
[7]

Han, W.; Chen, H.; and Poria, S. 2021. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. arXiv preprint arXiv:2109.00412

arXiv 2021
[8]

Hazarika, D.; Zimmermann, R.; and Poria, S. 2020. Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM international conference on multimedia, 1122--1131

2020

Show all 35 references

[9]

H.; Lakhotia, K.; Salakhutdinov, R.; and Mohamed, A

Hsu, W.-N.; Bolte, B.; Tsai, Y.-H. H.; Lakhotia, K.; Salakhutdinov, R.; and Mohamed, A. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29: 3451--3460

2021
[10]

Hu, G.; Lu, G.; and Zhao, Y. 2021. Bidirectional hierarchical attention networks based on document-level context for emotion cause extraction. In Findings of the Association for Computational Linguistics: EMNLP 2021, 558--568

2021
[11]

Hu, P.; Zhu, H.; Peng, X.; and Lin, J. 2020. Semi-supervised multi-modal learning with balanced spectral decomposition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 99--106

2020
[12]

Lian, Z.; Liu, B.; and Tao, J. 2022. Smin: Semi-supervised multi-modal interaction network for conversational emotion recognition. IEEE Transactions on Affective Computing

2022
[13]

Lian, Z.; Sun, H.; Sun, L.; Chen, K.; Xu, M.; Wang, K.; Xu, K.; He, Y.; Li, Y.; Zhao, J.; et al. 2023. Mer 2023: Multi-label learning, modality robustness, and semi-supervised learning. In Proceedings of the 31st ACM International Conference on Multimedia, 9610--9614

2023
[14]

Liang, J.; Li, R.; and Jin, Q. 2020. Semi-supervised multi-modal emotion recognition with cross-modal distribution matching. In Proceedings of the 28th ACM international conference on multimedia, 2852--2861

2020
[15]

Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

2019 arXiv
[16]

B.; Liang, P

Liu, Z.; Shen, Y.; Lakshminarasimhan, V. B.; Liang, P. P.; Zadeh, A.; and Morency, L.-P. 2018. Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv:1806.00064

2018 arXiv
[17]

Lv, F.; Chen, X.; Huang, Y.; Duan, L.; and Lin, G. 2021. Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2554--2562

2021
[18]

Parthasarathy, S.; and Busso, C. 2020. Semi-supervised speech emotion recognition with ladder networks. IEEE/ACM transactions on audio, speech, and language processing, 28: 2697--2709

2020
[19]

Poria, S.; Hazarika, D.; Majumder, N.; and Mihalcea, R. 2020. Beneath the tip of the iceberg: Current challenges and new directions in sentiment analysis research. IEEE Transactions on Affective Computing

2020
[20]

K.; Lee, S.; Zadeh, A.; Mao, C.; Morency, L.-P.; and Hoque, E

Rahman, W.; Hasan, M. K.; Lee, S.; Zadeh, A.; Mao, C.; Morency, L.-P.; and Hoque, E. 2020. Integrating multimodal information in large pretrained transformers. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2020, 2359. NIH Public Access

2020
[21]

Sun, Z.; Sarma, P.; Sethares, W.; and Liang, Y. 2020. Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 8992--8999

2020
[22]

H.; Bai, S.; Liang, P

Tsai, Y.-H. H.; Bai, S.; Liang, P. P.; Kolter, J. Z.; Morency, L.-P.; and Salakhutdinov, R. 2019. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2019, 6558. NIH Pu...

2019
[23]

H.; Liang, P

Tsai, Y.-H. H.; Liang, P. P.; Zadeh, A.; Morency, L.-P.; and Salakhutdinov, R. 2018. Learning factorized multimodal representations. arXiv preprint arXiv:1806.06176

2018 arXiv
[24]

N.; Kaiser, .; and Polosukhin, I

Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30

2017
[25]

Wang, K.; Lian, Z.; Sun, L.; Liu, B.; Tao, J.; and Fan, Y. 2022. Emotional reaction analysis based on multi-label graph convolutional networks and dynamic facial expression recognition transformer. In Proceedings of the 3rd International on Multimodal Sentiment Analysis Worksh...

2022
[26]

Wiles, O.; Koepke, A.; and Zisserman, A. 2018. Self-supervised learning of a facial attribute embedding from video. arXiv preprint arXiv:1808.06882

2018 arXiv
[27]

Yang, J.; Yu, Y.; Niu, D.; Guo, W.; and Xu, Y. 2023. ConFEDE: Contrastive Feature Decomposition for Multimodal Sentiment Analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 7617--7630

2023
[28]

Yu, T.; Gao, H.; Lin, T.-E.; Yang, M.; Wu, Y.; Ma, W.; Wang, C.; Huang, F.; and Li, Y. 2023. Speech-Text Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics...

2023
[29]

Yu, W.; Xu, H.; Yuan, Z.; and Wu, J. 2021. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. In Proceedings of the AAAI conference on artificial intelligence, volume 35, 10790--10797

2021
[30]

Yuan, Z.; Fang, J.; Xu, H.; and Gao, K. 2024. Multimodal Consistency-Based Teacher for Semi-Supervised Multimodal Sentiment Analysis. IEEE/ACM Transactions on Audio, Speech, and Language Processing

2024
[31]

Zadeh, A.; Chen, M.; Poria, S.; Cambria, E.; and Morency, L.-P. 2017. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250

2017 arXiv
[32]

Zadeh, A.; Zellers, R.; Pincus, E.; and Morency, L.-P. 2016. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259

2016 arXiv
[33]

B.; Liang, P

Zadeh, A. B.; Liang, P. P.; Poria, S.; Cambria, E.; and Morency, L.-P. 2018. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1...

2018
[34]

Zhang, D.; Li, S.; Zhu, Q.; and Zhou, G. 2020. Multi-modal sentiment classification with independent and interactive knowledge via semi-supervised learning. IEEE Access, 8: 22945--22954

2020
[35]

Zhang, H.; Xu, H.; and Lin, T.-E. 2021. Deep open intent classification with adaptive decision boundary. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 14374--14382

2021

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Carletta, J.; Ashby, S.; Bourban, S.; Flynn, M.; Guillemot, M.; Hain, T.; Kadlec, J.; Karaiskos, V.; Kraaij, W.; Kronenthal, M.; et al. 2005. The AMI meeting corpus: A pre-announcement. In International workshop on machine learning for multimodal interaction, 28--39. Springer

work page 2005

[4] [4]

Chen, C.; Hong, H.; Guo, J.; and Song, B. 2023 a . Inter-intra modal representation augmentation with trimodal collaborative disentanglement network for multimodal sentiment analysis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31: 1476--1488

work page 2023

[5] [5]

Chen, H.; Guo, C.; Li, Y.; Zhang, P.; and Jiang, D. 2023 b . Semi-Supervised Multimodal Emotion Recognition with Class-Balanced Pseudo-labeling. In Proceedings of the 31st ACM International Conference on Multimedia, 9556--9560

work page 2023

[6] [6]

Cheng, Z.; Lin, Y.; Chen, Z.; Li, X.; Mao, S.; Zhang, F.; Ding, D.; Zhang, B.; and Peng, X. 2023. Semi-supervised multimodal emotion recognition with expression mae. In Proceedings of the 31st ACM International Conference on Multimedia, 9436--9440

work page 2023

[7] [7]

Han, W.; Chen, H.; and Poria, S. 2021. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. arXiv preprint arXiv:2109.00412

arXiv 2021

[8] [8]

Hazarika, D.; Zimmermann, R.; and Poria, S. 2020. Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM international conference on multimedia, 1122--1131

2020