Pith. sign in

REVIEW 4 major objections 5 minor 44 references

SentiLLM claims that a frozen 1.5-billion-parameter language model can outperform prior multimodal sentiment systems on four benchmarks when video and audio are first distilled into compact, text-aligned sentiment tokens by a roughly 5-mill

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 01:20 UTC pith:5ZRF5X72

load-bearing objection A legitimate plug-in architecture for LLM-based MSA, with genuinely careful ablations—but thin statistics, an overclaiming abstract, and a semantic-alignment claim that is asserted rather than measured. the 4 major comments →

arxiv 2607.27790 v1 pith:5ZRF5X72 submitted 2026-07-30 cs.CL cs.AI

Semantic-Aligned Structural Abstraction for Multimodal Sentiment Analysis

Wei Chen , Junkai Li , Tongguan Wang , Hui Liu , Feiyue Xue , Chuanxiang Ma , Ying Sha This is my paper
classification cs.CL cs.AI
keywords multimodal sentiment analysislarge language modelsstructural abstractionsemantic alignmentaudio-visual fusionparameter-efficient adaptationsalience-context calibration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models reason over text, but raw video and audio are long, noisy, continuous signals that do not obviously live in an LLM's semantic space. The paper argues that non-verbal modalities are structurally like language—temporal sequences whose meaning comes from order and context—and therefore can be translated into compact, text-aligned 'sentiment tokens' instead of being linearly projected into the model. SentiLLM does this translation with a Dual-Stream Salience-Context Calibration Mechanism: it softly splits each audio/video sequence into a focus stream of salient sentiment shifts and an ambient stream of background state, uses text descriptions as attention queries to extract the shifts, then lets the background calibrate their confidence before a frozen LLM reads the tokens. The paper reports accuracy that matches or beats prior systems on MOSI, MOSEI, CH-SIMS, and CH-SIMS v2 while training only about five million parameters. The paper's claim, read sympathetically, is that a small plug-in which abstracts structure—not a bigger or fine-tuned LLM—is what enables multimodal reasoning.

Core claim

The central claim is that sentiment is not a static time-step mapping but the structural unity of salient sentiment shifts and contextual states, and that an ideal non-verbal representation must be both structural and calibrated. SentiLLM instantiates this with a Dual-Stream Salience-Context Calibration Mechanism: a learned salience score softly separates each time step into focus and ambient streams; text-anchor queries (e.g., 'a face showing clear positive emotion') retrieve sentiment shifts from the focus stream while learnable queries summarize the ambient stream; a confidence gate then amplifies or suppresses focus features according to background. The resulting sentiment tokens replace

What carries the argument

The load-bearing object is the Dual-Stream Salience-Context Calibration Mechanism—a soft, learned disentanglement of a non-verbal sequence into a focus stream of salient sentiment shifts and an ambient stream of background states, followed by dual cross-attention abstraction and a confidence-modulated calibration. A target-ratio sparsity loss keeps salience scores away from trivial all-keep or all-drop solutions. This mechanism performs the work of converting continuous raw video and audio features into a few compact sentiment tokens whose representation space is the frozen LLM's embedding space.

Load-bearing premise

The method assumes that hand-written text descriptions embedded by the frozen LLM (like 'a face showing clear positive emotion') genuinely sit close to the relevant visual and audio features in that embedding space, so the cross-attention queries really do pull out sentiment-aligned structure; the paper measures downstream accuracy but never directly measures this alignment.

What would settle it

Run the model on MOSI with the semantic anchors replaced by unrelated sentences (e.g., 'a table by the window') while keeping everything else fixed. If Acc-7 stays well above the 43.44% reported for random vectors, then anchor semantics are not the active ingredient and the 'semantic-aligned' explanation collapses; if it drops toward that baseline, the alignment story is supported.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • On MOSI, MOSEI, CH-SIMS, and CH-SIMS v2, the method reports accuracy matching or exceeding larger LLM-based systems while using a frozen 1.5B backbone and about 5M trainable parameters.
  • Because the LLM is frozen, the abstraction module can be attached to different LLMs; the paper shows consistent gains on MOSI with four different backbones.
  • The same dual-stream structure works on both English and Chinese benchmarks, suggesting the abstraction captures sentiment shifts that are not tied to a specific language.
  • The ablation study ties most of the gain to dual-stream calibration and to semantic text anchors, indicating that structural abstraction, not raw model scale, drives the reported performance.
  • The measured overhead over the vanilla baseline is small—roughly 12% extra FLOPs and 71 MB extra memory per batch—so the abstraction layer is cheap enough to deploy as a plug-in.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension not run in the paper: do the learned salience scores align with human-annotated event boundaries or sentiment-shift moments? The focus/ambient split implies such a correspondence, but the paper does not evaluate it.
  • The recipe could in principle transfer to other continuous temporal signals—physiological streams, sensor logs, financial time series—where brief salient events need context calibration; the paper studies only sentiment.
  • Because the text anchors are hand-written, the approach's ceiling may depend on the LLM's semantic coverage; a natural next step would be to generate anchor sets automatically by prompting the LLM itself, which the paper does not explore.
  • The paper's own stated limitations—no generated rationales for its predictions and no handling of missing audio or video—bound the claim to classification under complete modality availability.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SentiLLM, a plug-in module for LLM-based multimodal sentiment analysis. It converts video/audio feature sequences into compact 'sentiment tokens' through a Dual-Stream Salience-Context Calibration Mechanism: a salience scorer splits features into focus and ambient streams (Eqs. 2–3), cross-attention with LLM-embedded text anchors extracts focus tokens while learnable queries extract ambient tokens (Eqs. 4–5), a confidence modulation calibrates focus using ambient context (Eq. 6), and the resulting tokens are concatenated with text for frozen-LLM reasoning (Eq. 7). Training combines an MAE task loss with a sparsity regularizer (Eqs. 8–10). Experiments on MOSI, MOSEI, CH-SIMS, and CH-SIMS v2 report SOTA or competitive results with only ~5M trainable parameters.

Significance. If the empirical claims are robust, the paper is a useful and parameter-efficient contribution: it gives a concrete tokenization scheme for non-verbal modalities into an LLM's input space, with component-level ablations (Tables 3–5, Appendix B), four datasets, and released code. Equations 1–10 are internally consistent, and the ablation structure isolates the main design choices. However, the significance is currently conditional on two unresolved points: the reported performance margins are not backed by variance estimates, and the central 'semantic-aligned' mechanism is asserted rather than directly measured. The paper's own Appendix C.2 also shows that the focus stream is not well separated on two of the four datasets.

major comments (4)
  1. [§4.4, Tables 1–2] No standard deviations, number of seeds, or significance tests are reported. The headline margins over the strongest baselines are small (MOSI Acc-2 +0.97 points, MOSEI Acc-2 +0.47, CH-SIMS Acc-2 +0.65), and on CH-SIMS v2 SentiLLM trails MSE-ChatGLM-6B by 2.03 points. Without multiple-seed runs or a paired significance test, the SOTA claim is not established; the reported differences are within plausible run-to-run variation for fine-tuned LLM adapters. Please report means and stds over at least 3–5 seeds, plus a significance test or confidence intervals for the principal comparisons.
  2. [Eq. (4), App. C.1, Table 4] The paper's core claim is that text-anchor queries align non-verbal features with the LLM semantic space, but alignment is never measured. Table 4 only shows that semantic sentences outperform random vectors and simplified keywords on downstream accuracy; this does not rule out a generic initialization benefit from well-scaled LLM embeddings. I request a direct alignment check, e.g., cosine similarity or a linear probe between Q_foc and H_foc, or between the produced Z_v/Z_a and the anchor embeddings, compared against the random-vector counterpart. This is load-bearing because the abstract and Section 1 attribute the gains specifically to semantic alignment.
  3. [App. C.2, Figure 7] The paper's own diagnostic weakens the disentanglement/alignment claim on MOSEI and CH-SIMS v2: 'the focus stream shows significant distributional overlap between sentiment polarities.' The main-text visualization (Figure 5) shows only MOSI, where separation is favorable. Please provide quantitative separability metrics (e.g., linear-probe accuracy, silhouette score) per dataset for both streams, and discuss why the focus stream is not semantically separated where the method fails to reach SOTA. As it stands, the evidence for the 'structural abstraction' mechanism is cherry-picked to the dataset where it works best.
  4. [§1, §3.3] The Focus-Ambient calibration hypothesis is asserted in the introduction, built into the architecture, and then supported by the architecture's benchmark success. This is not a formal circularity in the equations, but the cognitive motivation is not independently tested. A parameter-matched control that replaces the mechanism-specific parts with a generic learned gate (same parameter count, same losses) would be needed to attribute gains specifically to the focus/ambient design rather than to the extra capacity or disentanglement-style losses. The paper should at least soften the causal language from the cognitive hypothesis to the implemented mechanism.
minor comments (5)
  1. [Abstract] The abstract says 'achieves superior performance on four datasets', but Table 2 shows that on CH-SIMS v2 SentiLLM is not state-of-the-art (Acc-2 79.88 vs. MSE-ChatGLM-6B 81.91). Recommend 'superior on three datasets and competitive on a fourth' or similarly precise wording.
  2. [§4.4] The text refers to 'MSE-ChatGLM3-6B', while the tables list 'MSE-ChatGLM-6B'. Please make the baseline name consistent.
  3. [Figure 1] There are typos in the figure/caption: 'an Contextual Temporal Encoding' and 'Laryer Norm'. Also, the 'sp' superscript notation for sparsity is not defined in the caption.
  4. [§4.5, Figure 2] The phrase 'Low sparsity causes severe information loss by discarding salient moments' is ambiguous. If the horizontal axis is the target activation ratio k, then low k is high sparsity (discarding 90%). Please rephrase to avoid confusion between k and 'sparsity'.
  5. [Appendix D] The limitations section lists interpretability and missing-modality scenarios, but does not mention the absence of multiple-seed variance reporting or the unverified semantic-alignment assumption. Adding these would give reviewers and readers a more complete picture of the current empirical support.

Circularity Check

0 steps flagged

No significant circularity: benchmark claims are externally grounded; the semantic-alignment explanation is under-tested but not definitionally forced.

full rationale

This is an empirical architecture paper, not a formal derivation. The proposed module is fully specified by Eqs. (1)-(10), and the central SOTA claims are comparisons on held-out test splits of MOSI, MOSEI, CH-SIMS, and CH-SIMS v2 (Tables 1-2), so the main result is externally grounded rather than equivalent to the method's inputs. The text-anchor cross-attention of Eq. (4) (Appendix C.1) is a real architectural component, and Table 4's comparison of semantic sentences versus random vectors is a controlled initialization ablation, not a fitted value renamed as a prediction. The Focus-Ambient hypothesis motivates the architecture, and Table 3's ablations test whether removing the focus/ambient streams hurts performance; this is ordinary empirical support, not definitional circularity. The only self-citation, [30] (RCLMuFN, sharing three authors), appears in a general list of existing fusion methods in Section 1 and is not load-bearing. Appendix C.2 candidly reports that on MOSEI and CH-SIMS v2 the focus stream shows 'significant distributional overlap between sentiment polarities,' which weakens the paper's semantic-alignment interpretation and identifies where the method does not reach SOTA; this is a limitation of the explanatory claim, not evidence that the derivation reduces to its inputs. The reviewer's concern that alignment is asserted rather than directly measured is a correctness/evidence issue, not a circularity issue under the stated criteria.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central empirical claim rests mainly on benchmark performance. The conceptual framing adds three unverified assumptions (isomorphism, Focus-Ambient cognition, LLM embedding alignment), and the method depends on two tuned hyperparameters plus hand-designed prompts. There is no formal derivation, so no hidden mathematical free parameters appear in an equation-level sense.

free parameters (3)
  • Sparsity ratio k = MOSI: 0.6, MOSEI: 0.4, CH-SIMS: 0.6, CH-SIMS v2: 0.6
    Controls the target fraction of time steps assigned to the focus stream in Eq. 9; tuned per dataset (Table 6, Figure 2) and directly shapes the disentanglement behavior.
  • Sparsity loss weight lambda = not reported
    Scales the sparsity loss in Eq. 10; no value or tuning procedure is given in the text, so exact reproduction of the joint objective is incomplete.
  • Hand-crafted text-anchor prompts = 8 natural-language sentences per modality per language (Appendix C.1)
    Used to initialize the focus-stream queries in Eq. 4; Table 4 shows a 5.3-point Acc-7 swing between these prompts and random vectors, so the central mechanism depends on this hand-designed choice.
axioms (4)
  • domain assumption Non-verbal modalities and natural language are structurally isomorphic temporal sequences.
    Stated in Section 1 and used to justify treating video/audio as 'temporal sentences'; cites VideoBERT/HuBERT but does not prove or operationalize the isomorphism.
  • ad hoc to paper Human sentiment judgment follows a Focus-Ambient calibration mechanism.
    Section 1: 'human first attend to salient sentiment shifts... and subsequently verify reliability through contextual cues.' No external cognitive-science citation or independent test is provided; the mechanism is built on this asserted theory.
  • domain assumption Frozen LLM text embeddings are a valid semantic anchor space for non-verbal sentiment queries.
    Used in Eq. 4 with text-anchor queries; only indirect accuracy evidence in Table 4, no direct alignment measurement.
  • domain assumption The [CLS] token from the frozen LLM after concatenated tokens is a complete sentiment representation.
    Eq. 7 and Section 3.3.4 rely on standard CLS-pooling behavior for prediction; no analysis of pooling faithfulness is given.

pith-pipeline@v1.3.0-daily-deepseek · 24076 in / 14923 out tokens · 139925 ms · 2026-08-01T01:20:07.196451+00:00 · methodology

0 comments
read the original abstract

Multimodal Sentiment Analysis (MSA) aims to interpret complex human emotions by integrating natural language with non-verbal modalities. Non-verbal modalities share a structural isomorphism with natural language, as both can be viewed as feature sequences evolving over time. This isomorphism enables the transformation of non-verbal modalities into text-like tokens for unified semantic reasoning. Large Language Models (LLMs), designed to understand and generate sequential data, can thus be utilized to interpret complex affective sequences. However, existing LLM-based methods primarily capture low-level superficial features, failing to model affective semantics arising from structural variations and contextual interactions. To address this limitation, we propose \textbf{SentiLLM}, a unified framework that leverages \textit{Semantic-Aligned Structural Abstraction} to distill continuous raw signals into compact, semantically meaningful tokens. Specifically, we introduce a \textit{Dual-Stream Salience-Context Calibration Mechanism}, which disentangles non-verbal feature sequences into a focus stream and an ambient stream. The focus stream captures salient sentiment shifts (e.g., facial expressions) guided by textual priors, while the ambient stream characterizes stable background states. Through calibrating these dynamic sentiment shifts against background states, SentiLLM effectively projects non-verbal modalities into a unified semantic space, making them naturally understandable for LLMs. Serving as a plug-and-play module, SentiLLM significantly improves discriminative performance with only a small number of trainable parameters. Our method achieves superior performance on four datasets, MOSI, MOSEI, CH-SIMS, and CH-SIMS v2, demonstrating the effectiveness of the structural abstraction paradigm in MSA. Our code is available at: \href{https://github.com/especiallyW/SentiLLM}.

Figures

Figures reproduced from arXiv: 2607.27790 by Chuanxiang Ma, Feiyue Xue, Hui Liu, Junkai Li, Tongguan Wang, Wei Chen, Ying Sha.

Figure 1
Figure 1. Figure 1: Overall architecture of SentiLLM. The model consists of two core modules: an Contextual Temporal Encoding Module, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Ablation study on the impact of sparsity ratio [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of learned representation spaces. Rows indicate the methods (Baseline vs. Ours) and columns represent [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Confusion matrix and per-class accuracy for 7-class [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of feature disentanglement on the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Visualization of feature disentanglement on the four datasets. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Confusion matrix and per-class accuracy for 7-class sentiment prediction on the four datasets. HN: Highly Negative; [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 9 linked inside Pith

  1. [1]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al

  2. [2]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normaliza- tion.arXiv preprint arXiv:1607.06450(2016)

    Pith/arXiv arXiv 2016

  3. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

    2020

  4. [4]

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Internlm2 technical report. arXiv preprint arXiv:2403.17297(2024)

    Pith/arXiv arXiv 2024

  5. [5]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 4171–4186

    2019

  6. [6]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.arXiv e-prints(2024), arXiv–2407

    2024

  7. [7]

    Wei Han, Hui Chen, and Soujanya Poria. 2021. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis.arXiv preprint arXiv:2109.00412(2021)

    Pith/arXiv arXiv 2021

  8. [8]

    Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020. MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Anal- ysis.Proceedings of the 28th ACM International Conference on Multimedia(2020). https://api.semanticscholar.org/CorpusID:218538102

    2020

  9. [9]

    Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. HuBERT: How much can a bad teacher benefit ASR pre-training?. InICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 6533–6537

    2021

  10. [10]

    Guimin Hu, Ting-En Lin, Yi Zhao, Guangming Lu, Yuchuan Wu, and Yongbin Li. 2022. UniMSE: Towards unified multimodal sentiment analysis and emotion recognition.arXiv preprint arXiv:2211.11256(2022)

    Pith/arXiv arXiv 2022

  11. [11]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning. PMLR, 19730–19742

    2023

  12. [12]

    Yong Li, Yuanzhi Wang, and Zhen Cui. 2023. Decoupled multimodal distilling for emotion recognition. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 6631–6640

    2023

  13. [13]

    Zaijing Li, Ting-En Lin, Yuchuan Wu, Meng Liu, Fengxiao Tang, Ming Zhao, and Yongbin Li. 2023. Unisa: Unified generative framework for sentiment analysis. In Proceedings of the 31st ACM international conference on multimedia. 6132–6142

    2023

  14. [14]

    George C Linderman, Manas Rachh, Jeremy G Hoskins, Stefan Steinerberger, and Yuval Kluger. 2017. Efficient algorithms for t-distributed stochastic neighborhood embedding.arXiv preprint arXiv:1712.09005(2017)

    Pith/arXiv arXiv 2017

  15. [15]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 26296–26306

    2024

  16. [16]

    Yihe Liu, Ziqi Yuan, Huisheng Mao, Zhiyun Liang, Wanqiuyue Yang, Yuanzhe Qiu, Tie Cheng, Xiaoteng Li, Hua Xu, and Kai Gao. 2022. Make acoustic and visual cues matter: Ch-sims v2. 0 dataset and av-mixup consistent module. InProceedings of the 2022 international conference on multimodal interaction. 247–258

    2022

  17. [17]

    Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. 2018. Efficient low-rank multimodal fusion with modality-specific factors. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2247–2256

    2018

  18. [18]

    Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y Hou, and Max Tegmark. 2024. Kan: Kolmogorov-arnold networks.arXiv preprint arXiv:2404.19756(2024)

    Pith/arXiv arXiv 2024

  19. [19]

    Miaosen Luo, Yuncheng Jiang, and Sijie Mai. 2025. Towards explainable fusion and balanced learning in multimodal sentiment analysis. InProceedings of the 33rd ACM International Conference on Multimedia. 1997–2006

    2025

  20. [20]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners.OpenAI blog 1, 8 (2019), 9

    2019

  21. [21]

    Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Louis-Philippe Morency, and Ehsan Hoque. 2020. Integrat- ing multimodal information in large pretrained transformers. InProceedings of the 58th annual meeting of the association for computational linguistics. 2359–2369

    2020

  22. [22]

    Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model.IEEE transactions on neural networks20, 1 (2008), 61–80

    2008

  23. [23]

    Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid

  24. [24]

    Hao Sun, Hongyi Wang, Jiaqing Liu, Yen-Wei Chen, and Lanfen Lin. 2022. Cube- MLP: An MLP-based model for multimodal sentiment analysis and depression estimation. InProceedings of the 30th ACM international conference on multimedia. 3722–3729

    2022

  25. [25]

    Shuangyang Sun, Guoyan Xu, and Sijun Lu. 2024. MFM: Multimodal Sentiment Analysis Based on Modal Focusing Model. In2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 1524–1529

    2024

  26. [26]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118(2024)

    Pith/arXiv arXiv 2024

  27. [27]

    Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. InProceedings of the conference. Association for computational linguistics. Meeting, Vol. 2019. 6558

    2019

  28. [28]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Advances in neural information processing systems30 (2017)

    2017

  29. [29]

    Pan Wang, Qiang Zhou, Yawen Wu, Tianlong Chen, and Jingtong Hu. 2025. DLF: Disentangled-language-focused multimodal sentiment analysis. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 21180–21188

    2025

  30. [30]

    Tongguan Wang, Junkai Li, Guixin Su, Yongcheng Zhang, Dongyu Su, Yuxue Hu, and Ying Sha. 2025. RCLMuFN: Relational context learning and multiplex fusion network for multimodal sarcasm detection.Knowledge-Based Systems319 (2025), 113614

    2025

  31. [31]

    Jennifer Williams, Steven Kleinegesse, Ramona Comanescu, and Oana Radu

  32. [32]

    Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin,...

    Pith/arXiv arXiv 2024

  33. [33]

    Yang Yang, Xunde Dong, and Yupeng Qiang. 2025. MSE-Adapter: A Lightweight Plugin Endowing LLMs with the Capability to Perform Multimodal Sentiment Analysis and Emotion Recognition. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 25642–25650

    2025

  34. [34]

    Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. 2020. Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. InProceedings of the 58th annual meeting of the association for computational linguistics. 3718–3727

    2020

  35. [35]

    Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. 2021. Learning modality-specific representations with self-supervised multi-task learning for multimodal senti- ment analysis. InProceedings of the AAAI conference on artificial intelligence, Vol. 35. 10790–10797

    2021

  36. [36]

    Cambria, and Louis philippe Morency

    Amir Zadeh, Minghai Chen, Soujanya Poria, E. Cambria, and Louis philippe Morency. 2017. Tensor Fusion Network for Multimodal Sentiment Analysis. InConference on Empirical Methods in Natural Language Processing. https: //api.semanticscholar.org/CorpusID:950292

    2017

  37. [37]

    Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Memory fusion network for multi-view sequential learning. InProceedings of the AAAI conference on artificial intelligence, Vol. 32

    2018

  38. [38]

    Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis philippe Morency. 2016. MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos.ArXivabs/1606.06259 (2016). https://api.semanticscholar.org/ CorpusID:13978043

    Pith/arXiv arXiv 2016

  39. [39]

    AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis- Philippe Morency. 2018. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2236–2246

    2018

  40. [40]

    Heming Zhang. 2024. A comprehensive survey on multimodal sentiment analysis: Techniques, models, and applications.Advances in Engineering Innovation(2024). https://api.semanticscholar.org/CorpusID:282217843 MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Wei Chen et al

    2024

  41. [41]

    text only

    Xiangmin Zhang, Wei Wei, and Shihao Zou. 2025. Modal feature optimization network with prompt for multimodal sentiment analysis. InProceedings of the 31st International Conference on Computational Linguistics. 4611–4621. SentiLLM MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Appendix A Experimental Details A.1 Datasets In this section, we briefly i...

    2025

  42. [2018]

    In Grand Challenge and Workshop on Human Multimodal Language

    Recognizing emotions in video using multimodal DNN feature fusion. In Grand Challenge and Workshop on Human Multimodal Language. Association for Computational Linguistics, 11–19

  43. [2019]

    InProceedings of the IEEE/CVF international conference on computer vision

    Videobert: A joint model for video and language representation learning. InProceedings of the IEEE/CVF international conference on computer vision. 7464– 7473

  44. [2022]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems35 (2022), 23716–23736

    2022