Pith. sign in

REVIEW 4 major objections 4 minor 57 references

Towards Multimodal Sentiment Analysis via Contrastive Cross-modal Retrieval Augmentation and Hierachical Prompts

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Retrieval-augmented prompts bring cross-sample context to multimodal sentiment analysis.

desk verdict Plausible retrieval-augmented sentiment model, but the full text is corrupt and the abstract gives no numbers; desk reject pending a clean, complete resubmission. read the letter →

arxiv 2508.07666 v1 pith:APV5NYF4 submitted 2025-08-11 cs.MM

Xianbing Zhao , Shengzun Yang , Buzhou Tang , Ronghuan Jiang This is my paper · ORCID
classification cs.MM
keywords multimodalsentimentanalysisretrieval-augmentedcontrastivecross-modalretrievalpromptlearningmodality-levelsample-levelreferencecontextaffectivecomputing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal sentiment analysis usually fuses modalities within a single clip, drawing on the clip's own text, audio, and video. This paper argues that such within-sample fusion is not enough: clips that are semantically similar to the target clip carry reference context that should also shape the prediction. The proposed model retrieves similar samples with a contrastive cross-modal retrieval module, then uses two kinds of prompts (modality-level and sample-level) to build reference contexts that enhance the target modality before classification. The central claim is that combining both kinds of reference context improves sentiment prediction beyond current cross-modal interaction methods, and the authors report that this holds on two public benchmark datasets.

What carries the argument

The central mechanism is a contrastive cross-modal retrieval module paired with two prompt generators. The retrieval module learns an embedding space in which clips from different modalities that share sentiment content are mapped close together, allowing a target clip to pull in its nearest neighbours; the modality-level prompt and sample-level prompt convert those neighbours into reference-context tokens. The cross-modal retrieval-augmented encoder then fuses these prompt tokens with the target clip's own modality features. In short, the machinery is retrieve by cross-modal similarity, prompt with both within-sample and cross-sample context, and fuse before classification.

What would settle it

Take the trained model and rerun evaluation with the retrieval pool explicitly restricted to the training split, then run it again with the target clip's own data added back into the pool; if accuracy does not drop when the target clip is removed from the neighbor set, the sample-level reference context is doing real work, but if it drops, the gain is an artifact of test-time leakage. A second check: replace the contrastively retrieved neighbors with randomly sampled training clips; if accuracy on the benchmarks is unchanged, retrieval quality is not the source of the improvement.

Watch

Extended reading notes

Core claim

The paper's central discovery claim is that sorting the training set by cross-modal similarity and feeding the top neighbours into prompt-based encoders gives a sentiment classifier a form of context that neither unimodal nor within-sample cross-modal fusion can supply. Concretely, a contrastive cross-modal retrieval module embeds clips so that text, audio, and video from one sample can retrieve semantically related clips; the retrieved clips are then converted into modality-level prompts (which encode how modalities align inside a clip) and sample-level prompts (which encode what a similar clip's content means), and a cross-modal retrieval-augmented encoder injects both into the target clip

Load-bearing premise

The reported gains depend on the retrieval module consulting only training-set samples at evaluation time, so that the sample-level reference context cannot leak the test clip's identity or label, and on the contrastively learned similarity being good enough that the neighbors it finds are genuinely informative rather than noise.

Editorial extensions

If this is right

  • If the central claim is right, retrieval-augmented prompt construction should outperform the same encoder without cross-sample reference context on the two benchmark datasets.
  • The two prompt types should be complementary: modality-level prompts supply within-sample cross-modal grounding, and sample-level prompts supply cross-sample contextualization.
  • The contrastive retrieval module should be essential: replacing it with random neighbor selection should degrade performance on the reported benchmarks.
  • The framework offers a template for other multimodal recognition tasks where cross-sample relationships can serve as reference context beyond sentiment.
  • The reported gains imply that sample-level reference context carries predictive signal not present in the target clip alone, which is a substantive claim about the structure of multimodal sentiment data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension is to apply the same retrieval-prompt mechanism to other multimodal tasks such as emotion recognition, humor detection, or multimodal hate-speech detection, where cross-sample context should help in a similar way.
  • The modality-level versus sample-level distinction could be operationalized as two separate prompt slots, enabling future ablations that test whether the optimal fusion weight differs across datasets or modalities.
  • If the retrieval pool is strictly the training set, the approach effectively becomes a nonparametric memory for sentiment: model performance should improve as the training corpus grows, which is a testable scaling prediction.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a multimodal sentiment analysis framework that augments cross-modal interaction by retrieving semantically similar samples through a contrastive cross-modal retrieval module, then uses two types of prompts (modality-level and sample-level) to inject reference context into a retrieval-augmented encoder. The abstract claims superior performance on two public datasets. However, the supplied full text is a corrupted mojibake dump, and it interleaves an unrelated arXiv identifier; consequently the architecture, equations, training objective, datasets, baselines, and numerical results cannot be inspected. The evaluation is therefore unverifiable from the submitted manuscript.

Significance. If the claimed gains hold, the idea of adding cross-sample reference context through retrieval is of interest to affective computing and could be a solid empirical contribution. The contribution is potentially useful because retrieval-augmented methods are underexplored in multimodal sentiment analysis. But at present the manuscript provides no auditable evidence: no legible methods, no error bars, no readable tables, and no reproducible derivation. The conceptual design is plausible, but the paper as submitted cannot support its claims. The clarity of the abstract is a strength; the corruption of the body is a blocking defect.

major comments (4)
  1. [Full text (all pages after abstract)] The body is undecodable mojibake, and it repeatedly contains the string 'arXiv:2508.07663v2 [physics.flu-dyn] 31 Oct 2025'. This is not a minor OCR artifact: none of the proposed modules, training losses, equations, or algorithmic descriptions can be read. The central claim cannot be checked. A clean, correctly encoded manuscript must be supplied before substantive review can occur.
  2. [Abstract, final sentence; experimental section (unreadable)] The abstract claims 'extensive experiments demonstrate the effectiveness and superiority of our model', but no accuracy numbers, standard deviations, significance tests, dataset statistics, baseline list, or hyperparameter settings are legible anywhere in the manuscript. Without reported results and an error-bar analysis, the superiority claim is unsupported even if the framework itself is well defined.
  3. [Abstract (contrastive cross-modal retrieval module) and full text (method, unreadable)] The load-bearing premise is that retrieval of sample-level reference contexts improves accuracy. The evaluation must specify whether the retrieval index is restricted to the training split, whether the query clip is excluded, and whether near-duplicate or same-video segments are removed. None of these boundary conditions can be verified in the current text. A train-only retrieval index, an ablation without retrieval, and explicit leakage checks (e.g., excluding same-session neighbors) are needed; otherwise the reported gains could be inflated by label leakage.
  4. [Full text (results/tables)] No tables, figures, or quantitative comparisons are legible. The paper presumably contains an experimental section, but it cannot be read or cited. The authors need to reproduce all results in readable form, including dataset splits, evaluation metrics, and comparisons to established baselines.
minor comments (4)
  1. [Title] Typo: 'Hierachical' should be 'Hierarchical'.
  2. [Full text, opening pages] The unrelated arXiv ID '2508.07663v2 [physics.flu-dyn]' is interleaved in the body. This indicates a corrupted PDF build; the authors should ensure the source matches this paper.
  3. [Full text, final portions] Several blocks of text are repeated verbatim near the end of the document, suggesting duplicate content that should be cleaned up.
  4. [Abstract] The phrase 'inter-sample modality-level reference context' is confusing; 'within-sample modality-level reference context' would better match the later contrast with 'cross-sample sample-level reference context'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity detectable from the readable material; the derivation chain is not available due to text corruption, and the abstract describes standard supervised retrieval augmentation.

full rationale

The only legible portion of the manuscript is the abstract; the body is mojibake and contains an interleaved arXiv identifier from an unrelated physics paper, so no equation-level derivation chain can be audited. Based on the abstract, the proposed framework trains a contrastive cross-modal retrieval module and prompt encoders on labeled training data and then retrieves 'semantic similar samples' to produce modality-level and sample-level reference contexts. This is ordinary supervised learning plus a nearest-neighbor-style retrieval augmentation, not a fitted constant renamed as a prediction or a quantity defined in terms of itself. No self-citation is invoked as a load-bearing premise, and no uniqueness or ansatz is imported from the authors' prior work. The skeptical concern about retrieval-boundary leakage (whether the retrieval index includes test clips or near-duplicate session-level samples) is an experimental-validity risk that cannot be checked from the corrupted text, but it is a data-leakage boundary condition, not a demonstration that any result reduces to its inputs by construction. Under the hard rule that circularity must be exhibited by a specific quoted reduction, no circular step can be identified, so the score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

All entries are inferred from the abstract because the full text is corrupted. The central claim rests on a trained retrieval module and prompt conditioning, so the honest ledger is dominated by undisclosed hyperparameters and data-boundary assumptions rather than by free parameters fitted to the final result. No new physical or theoretical entities are introduced; the prompts and retrieval module are architectural components.

free parameters (3)
  • Number of retrieved samples K
    The retrieval module must select K similar samples per target; K is a hyperparameter likely tuned on a validation set. No value is given in the abstract.
  • Contrastive temperature tau
    Contrastive cross-modal retrieval typically uses a temperature-scaled similarity (InfoNCE type), with tau chosen by hand. Not disclosed in the abstract.
  • Loss weighting coefficients
    A model with a contrastive loss and a classification loss requires weighting coefficients; not disclosed in the abstract.
assumptions (3)
  • domain assumption The two public datasets used for evaluation are standard benchmarks whose labels are reliable and whose splits match those of the compared baselines.
    The abstract claims superiority on two public datasets without naming them or stating the split protocol; comparability of splits is assumed.
  • domain assumption Retrieval for the target sample is performed over the training split only, so no test label information enters through retrieved neighbors.
    Sample-level reference context is the paper's key mechanism; if retrieval crosses the train/test boundary, the reported gains would be leakage artifacts. The corrupted full text prevents verification of the retrieval boundary.
  • domain assumption Contrastive training of the retrieval module yields similarity scores that transfer to unseen test samples.
    The framework relies on the contrastively learned retriever ranking useful neighbors at inference; the abstract does not analyze retrieval quality or failure cases.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Multimodal Sentiment Analysis via Contrastive Cross-modal Retrieval Augmentation and Hierachical Prompts." pith.science (2026). https://pith.science/paper/APV5NYF4

@misc{pith2026250807666,
  author       = {Pith},
  title        = {Pith review of: Towards Multimodal Sentiment Analysis via Contrastive Cross-modal Retrieval Augmentation and Hierachical Prompts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/APV5NYF4}},
  note         = {Machine review of arXiv:2508.07666}
}
read the original abstract

Multimodal sentiment analysis is a fundamental problem in the field of affective computing. Although significant progress has been made in cross-modal interaction, it remains a challenge due to the insufficient reference context in cross-modal interactions. Current cross-modal approaches primarily focus on leveraging modality-level reference context within a individual sample for cross-modal feature enhancement, neglecting the potential cross-sample relationships that can serve as sample-level reference context to enhance the cross-modal features. To address this issue, we propose a novel multimodal retrieval-augmented framework to simultaneously incorporate inter-sample modality-level reference context and cross-sample sample-level reference context to enhance the multimodal features. In particular, we first design a contrastive cross-modal retrieval module to retrieve semantic similar samples and enhance target modality. To endow the model to capture both inter-sample and intra-sample information, we integrate two different types of prompts, modality-level prompts and sample-level prompts, to generate modality-level and sample-level reference contexts, respectively. Finally, we design a cross-modal retrieval-augmented encoder that simultaneously leverages modality-level and sample-level reference contexts to enhance the target modality. Extensive experiments demonstrate the effectiveness and superiority of our model on two publicly available datasets.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 48 canonical work pages

  1. [1]

    Y.-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov, ``Multimodal transformer for unaligned multimodal language sequences,'' in Proceedings of the conference. Association for computational linguistics. Meeting, vol. 2019, 2019, p. 6558

    work page 2019

  2. [2]

    Rahman, M

    W. Rahman, M. K. Hasan, S. Lee, A. Zadeh, C. Mao, L.-P. Morency, and E. Hoque, ``Integrating multimodal information in large pretrained transformers,'' in Proceedings of the conference. Association for computational linguistics. Meeting, vol. 2020, 2020, p. 2359

    work page 2020

  3. [3]

    J. Yang, Y. Wang, R. Yi, Y. Zhu, A. Rehman, A. Zadeh, S. Poria, and L.-P. Morency, ``Mtag: Modal-temporal attention graph for unaligned human multimodal language sequences,'' in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 1009--1021

    work page 2021

  4. [4]

    F. Lv, X. Chen, Y. Huang, L. Duan, and G. Lin, ``Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences,'' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 2554--2562

    work page 2021

  5. [5]

    W. Han, H. Chen, A. Gelbukh, A. Zadeh, L.-p. Morency, and S. Poria, ``Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis,'' in Proceedings of the 2021 international conference on multimodal interaction, 2021, pp. 6--15

    work page 2021

  6. [6]

    W. Yu, H. Xu, Z. Yuan, and J. Wu, ``Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis,'' in Proceedings of the AAAI conference on artificial intelligence, vol. 35, no. 12, 2021, pp. 10\,790--10\,797

    work page 2021

  7. [7]

    D. Yang, S. Huang, H. Kuang, Y. Du, and L. Zhang, ``Disentangled representation learning for multimodal emotion recognition,'' in Proceedings of the 30th ACM international conference on multimedia, 2022, pp. 1642--1651

    work page 2022

  8. [8]

    X. Zhao, Y. Chen, S. Liu, X. Zang, Y. Xiang, and B. Tang, ``Tmmda: A new token mixup multimodal data augmentation for multimodal sentiment analysis,'' in Proceedings of the ACM Web Conference 2023, 2023, pp. 1714--1722

    work page 2023

Show all 57 references
  1. [9]

    Y. Li, Y. Wang, and Z. Cui, ``Decoupled multimodal distilling for emotion recognition,'' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 6631--6640

    2023

  2. [10]

    J. Yang, Y. Yu, D. Niu, W. Guo, and Y. Xu, ``Confede: Contrastive feature decomposition for multimodal sentiment analysis,'' in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 7617--7630

    2023

  3. [11]

    Y. Zeng, W. Yan, S. Mai, and H. Hu, ``Disentanglement translation network for multimodal sentiment analysis,'' Information Fusion, vol. 102, p. 102031, 2024

    2024

  4. [12]

    Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, and H. Wang, ``Retrieval-augmented generation for large language models: A survey.''

  5. [13]

    A. Long, W. Yin, T. Ajanthan, V. Nguyen, P. Purkait, R. Garg, A. Blair, C. Shen, and A. van den Hengel, ``Retrieval augmented classification for long-tail visual recognition,'' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 6959--6969

    2022

  6. [14]

    O. Ram, Y. Levine, I. Dalmedigos, D. Muhlgay, A. Shashua, K. Leyton-Brown, and Y. Shoham, ``In-context retrieval-augmented language models,'' Transactions of the Association for Computational Linguistics, vol. 11, pp. 1316--1331, 2023

    2023

  7. [15]

    Blattmann, R

    A. Blattmann, R. Rombach, K. Oktay, J. M \"u ller, and B. Ommer, ``Retrieval-augmented diffusion models,'' in Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022, pp. 15\,309--15\,324

    2022

  8. [16]

    S. Gur, N. Neverova, C. Stauffer, S.-N. Lim, D. Kiela, and A. Reiter, ``Cross-modal retrieval augmentation for multi-modal classification,'' in Findings of the Association for Computational Linguistics: EMNLP 2021, 2021, pp. 111--123

    2021

  9. [17]

    W. Han, H. Chen, and S. Poria, ``Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis,'' in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 9180--9192

    2021

  10. [18]

    Y.-H. H. Tsai, P. P. Liang, A. Zadeh, L.-P. Morency, and R. Salakhutdinov, ``Learning factorized multimodal representations,'' in International Conference on Representation Learning, 2019

    2019

  11. [19]

    M. Chen, S. Wang, P. P. Liang, T. Baltru s aitis, A. Zadeh, and L.-P. Morency, ``Multimodal sentiment analysis with word-level fusion and reinforcement learning,'' in Proceedings of the 19th ACM international conference on multimodal interaction, 2017, pp. 163--171

    2017

  12. [20]

    D. Zong, C. Ding, B. Li, J. Li, K. Zheng, and Q. Zhou, ``Acformer: An aligned and compact transformer for multimodal sentiment analysis,'' in Proceedings of the 31st ACM international conference on multimedia, 2023, pp. 833--842

    2023

  13. [21]

    Zadeh, M

    A. Zadeh, M. Chen, S. Poria, E. Cambria, and L.-P. Morency, ``Tensor fusion network for multimodal sentiment analysis,'' in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017, pp. 1103--1114

    2017

  14. [22]

    Liu and Y

    Z. Liu and Y. Shen, ``Efficient low-rank multimodal fusion with modality-specific factors,'' in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Long Papers), 2018

    2018

  15. [23]

    S. Mai, Y. Sun, Y. Zeng, and H. Hu, ``Excavating multimodal correlation for representation learning,'' Information Fusion, vol. 91, pp. 542--555, 2023

    2023

  16. [24]

    Y. Yu, M. Zhao, S.-a. Qi, F. Sun, B. Wang, W. Guo, X. Wang, L. Yang, and D. Niu, ``Conki: Contrastive knowledge injection for multimodal sentiment analysis,'' in Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 13\,610--13\,624

    2023

  17. [25]

    Zhang, W

    X. Zhang, W. Wei, and S. Zou, ``Modal feature optimization network with prompt for multimodal sentiment analysis,'' in Proceedings of the 31st International Conference on Computational Linguistics, 2025, pp. 4611--4621

    2025

  18. [26]

    Li and L

    Z. Li and L. Li, ``t-hne: A text-guided hierarchical noise eliminator for multimodal sentiment analysis,'' in Proceedings of the 31st International Conference on Computational Linguistics, 2025, pp. 2834--2844

    2025

  19. [27]

    Liang, Y

    Z. Liang, Y. Xu, Y. Hong, P. Shang, Q. Wang, Q. Fu, and K. Liu, ``A survey of multimodel large language models,'' in Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, 2024, pp. 405--409

    2024

  20. [28]

    [Online]

    Mixtral-AI, ``Mixtral of experts,'' 2023.11. [Online]. Available: https:// mistral.ai/news/mixtral-of-experts/

    2023

  21. [29]

    [Online]

    Google, ``Gemma - google’s new open llm,'' 2024.02. [Online]. Available: https: //huggingface.co/blog/gemma

    2024

  22. [30]

    [Online]

    OpenAI, ``Chatgpt: Large-scale language model fine-tuned for conversational applications,'' 2023a. [Online]. Available: https://openai.com

  23. [31]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., ``Llama 2: Open foundation and fine-tuned chat models,'' arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  24. [32]

    [Online]

    Google, ``Gemini: our largest and most capable ai model,'' 2023. [Online]. Available: https://blog.google/technology/ ai/google-gemini-ai/#sundar-note

    2023

  25. [33]

    J. Li, D. Li, S. Savarese, and S. Hoi, ``Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,'' 2023. [Online]. Available: https://arxiv.org/abs/2301.12597

    2023 arXiv

  26. [34]

    D. Wang, Q. Wang, L. He, X. Gao, and Y. Tian, ``Joint and individual matrix factorization hashing for large-scale cross-modal retrieval,'' Pattern recognition, vol. 107, p. 107479, 2020

    2020

  27. [35]

    Jiang, F

    Z. Jiang, F. F. Xu, L. Gao, Z. Sun, Q. Liu, J. Dwivedi-Yu, Y. Yang, J. Callan, and G. Neubig, ``Active retrieval augmented generation,'' in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 7969--7992

    2023

  28. [36]

    R. Xu, Y. Yu, J. Ho, and C. Yang, ``Weakly-supervised scientific document classification via retrieval-augmented multi-stage training,'' in Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023, pp. 2501--2505

    2023

  29. [37]

    u ttler, M. Lewis, W.-t. Yih, T. Rockt \

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. K \"u ttler, M. Lewis, W.-t. Yih, T. Rockt \"a schel et al., ``Retrieval-augmented generation for knowledge-intensive nlp tasks,'' Advances in neural information processing systems, vol. 33, pp. 9459--9474, 2020

    2020

  30. [38]

    M. R. Rezaei and A. B. Dieng, ``Vendi-rag: Adaptively trading-off diversity and quality significantly improves retrieval augmented generation with llms,'' arXiv preprint arXiv:2502.11228, 2025

    2025 arXiv

  31. [39]

    S. Yang, D. Wang, H. Zheng, and R. Jin, ``Timerag: Boosting llm time series forecasting via retrieval-augmented generation,'' in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 1em plus 0.5em minus 0.4em IEEE, 2025, pp. 1--5

    2025

  32. [40]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, ``Exploring the limits of transfer learning with a unified text-to-text transformer,'' Journal of machine learning research, vol. 21, no. 140, pp. 1--67, 2020

    2020

  33. [41]

    Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, ``Vggface2: A dataset for recognising faces across pose and age,'' in 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018). 1em plus 0.5em minus 0.4em IEEE, 2018, pp. 67--74

    2018

  34. [42]

    W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``Hubert: Self-supervised speech representation learning by masked prediction of hidden units,'' IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451--3460, 2021

    2021

  35. [43]

    Hazarika, R

    D. Hazarika, R. Zimmermann, and S. Poria, ``Misa: Modality-invariant and-specific representations for multimodal sentiment analysis,'' in Proceedings of the 28th ACM international conference on multimedia, 2020, pp. 1122--1131

    2020

  36. [44]

    Z. Guo, T. Jin, and Z. Zhao, ``Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition,'' in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 1726--1736

    2024

  37. [45]

    K. He, X. Zhang, S. Ren, and J. Sun, ``Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,'' in Proceedings of the IEEE international conference on computer vision, 2015, pp. 1026--1034

    2015

  38. [46]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, . Kaiser, and I. Polosukhin, ``Attention is all you need,'' Advances in neural information processing systems, vol. 30, 2017

    2017

  39. [47]

    Zadeh, R

    A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, ``Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos,'' arXiv preprint arXiv:1606.06259, 2016

    2016 arXiv

  40. [48]

    A. B. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency, ``Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph,'' in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long...

    2018

  41. [49]

    Y. Sun, S. Mai, and H. Hu, ``Learning to learn better unimodal representations via adaptive multimodal meta-learning,'' IEEE Transactions on Affective Computing, vol. 14, no. 3, pp. 2209--2223, 2022

    2022

  42. [50]

    Z. Wu, Q. Zhang, D. Miao, K. Yi, W. Fan, and L. Hu, ``Hydiscgan: a hybrid distributed cgan for audio-visual privacy preservation in multimodal sentiment analysis,'' in Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, 2024, pp. 6550--6558

    2024

  43. [51]

    Y. Chen, Y. Wang, and Y. Liu, ``Mofn: Multi-offset-flow-based network for video restoration and enhancement,'' in 2022 IEEE International Conference on Multimedia and Expo Workshops (ICMEW). 1em plus 0.5em minus 0.4em IEEE, 2022, pp. 1--6

    2022

  44. [52]

    X. Yang, W. Wu, S. Feng, M. Wang, D. Wang, Y. Li, Q. Sun, Y. Zhang, X. Fu, and S. Poria, ``Mm-instructeval: Zero-shot evaluation of (multimodal) large language models on multimodal reasoning tasks,'' arXiv preprint arXiv:2405.07229, 2024

    2024 arXiv

  45. [53]

    H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma et al., ``Scaling instruction-finetuned language models,'' Journal of Machine Learning Research, vol. 25, no. 70, pp. 1--53, 2024

    2024

  46. [54]

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, ``Qwen-vl: A frontier large vision-language model with versatile abilities,'' arXiv preprint arXiv:2308.12966, vol. 1, no. 2, p. 3, 2023. [Online]. Available: https://arxiv.org/abs/ 2308.12966

    2023 arXiv

  47. [55]

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, ``Instructblip: Towards general-purpose vision-language models with instruction tuning,'' 2023. [Online]. Available: https://arxiv.org/abs/2305.06500

    2023 arXiv

  48. [56]

    [Online]

    OpenAI, ``Gpt-4v(ision) system card,'' 2023b. [Online]. Available: https://openai.com/research/ gpt-4v-system-card

  49. [57]

    [Online]

    Anthropic, ``Meet claude,'' 2024. [Online]. Available: https://www. anthropic.com/claude

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.