REVIEW 5 major objections 6 minor 43 references
Exploring Large Language Models for Multimodal Sentiment Analysis: Challenges, Benchmarks, and Future Directions
T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Off-the-shelf large language models underperform supervised methods on multimodal aspect-based sentiment analysis.
desk verdict A plausible first probe of LLMs on MABSA, but the missing output-parsing details and single-run numbers make the headline gap unverifiable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The LLM4SA framework, which converts a MABSA datapoint into a visual prefix plus text and a question prompt, with a few in-context examples taken from the training set. Visual features are extracted with a frozen ViT and projected through a linear layer into the LLM's embedding space. The model must then generate, for each aspect in the text, an aspect–sentiment pair; correct prediction requires the model to solve the three subtasks of detecting how many aspect-sentiment pairs exist, identifying each aspect span, and assigning the correct polarity.
What would settle it
Re-running the same LLM4SA prompts on Twitter-2015 and Twitter-2017 with, say, five different random draws of in-context examples and temperature 0.2, 0.7, and 1.0, and comparing the mean F1 against the reported values; if the best LLM's mean F1 reaches or exceeds the supervised baseline (71.9 on Twitter-2015, 70.6 on Twitter-2017), the claim of a consistent performance gap would be refuted.
Extended reading notes
Core claim
The central claim is that off-the-shelf LLMs, used with in-context learning, do not reach the accuracy of published supervised systems on MABSA. On Twitter-2015, the best LLM (LLaVA) achieves 55.62 F1 against 71.90 for the supervised DQPSA; on Twitter-2017, LLaVA achieves 61.74 against 70.60 for DQPSA. The paper attributes the shortfall to three factors: the MABSA task format is rare in instruction-tuning data, so the models are not 'well-acquainted' with it; the constrained number of randomly selected ICL examples limits learning; and the models' inference cost is orders of magnitude higher. As a result, the paper concludes that LLM-based methods still exhibit a performance gap compared to traditional methods, despite their demonstrated ability in general multimodal understanding.
Load-bearing premise
The central claim rests on the assumption that the single reported run for each model in Table 3 is representative: the paper reports no standard deviations across random in-context example draws, decoding temperatures, or random seeds, and only one number per model per dataset appears. If a re-run gave materially different F1 values, the size of the claimed gap would change accordingly.
Editorial extensions
If this is right
- If the claim holds, zero- and few-shot prompting of current LLMs is not a viable substitute for supervised training on MABSA, and reported SOTA numbers for the task remain with supervised models.
- The gap identifies a benchmark target: an LLM-based method that closes the F1 gap will need to handle joint aspect extraction and sentiment classification reliably.
- The large inference-time gap implies that practical deployment of LLMs for fine-grained sentiment tasks requires substantial compression or acceleration, not just accuracy improvements.
- The finding suggests that task-specific instruction tuning with MABSA-style data may be a necessary next step for LLMs, since general instruction tuning appears insufficient.
Reading between the lines
- The paper leaves implicit that the strict token-level matching metric may penalize LLM generation formats more than their actual understanding; a human-readable or span-normalized evaluation could change the size of the gap.
- The paper's random ICL selection suggests a testable extension: if ICL examples are chosen to be most similar to the query (e.g., by embedding distance), the LLM F1 might rise substantially, narrowing the reported gap.
- Another extension is to fine-tune a 7B LLM on the MABSA training set rather than only prompting it; this would test whether the gap is due to inherent capability or to the prompting protocol.
- The reported inference times mix API latency for ChatGPT with local generation for the open models, so a controlled local-generation comparison would better isolate model speed.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces LLM4SA, a zero/few-shot in-context learning framework for Multimodal Aspect-Based Sentiment Analysis (MABSA), and evaluates Llama2-7B, LLaVA, and GPT-3.5-turbo on Twitter-2015 and Twitter-2017 against supervised baselines. The central claim is that off-the-shelf LLMs underperform traditional supervised methods on MABSA accuracy and are substantially slower. The authors attribute this to task unfamiliarity, limited in-context learning effectiveness, and inference cost, and outline future directions for instruction tuning and efficient adaptation.
Significance. If the reported results are valid, the paper provides a useful negative result for practitioners and a starting benchmark for LLM-based MABSA. The use of public datasets, the explicit reproduction of the DTCA baseline, and the attempt to quantify inference time are strengths. However, the current evidence does not yet support the central claim, because the evaluation protocol for free-form LLM outputs is not specified, all LLM numbers come from single runs, and the visual input pathway for Llama2 and ChatGPT is unclear. With the evaluation harness, prompts, and variance estimates supplied, this could become a solid benchmark contribution; in its present form it reads as an early draft.
major comments (5)
- [Section 3.2.1, Section 4.1] The paper never specifies how the free-form text in the generated '[Answer]' field is converted into aspect-polarity pairs. Section 4.1 states that an aspect is correct only if 'the aspect term and polarity respectively match the ground truth,' but no matching rule is defined: exact string match, substring containment, normalization of surface forms, span alignment, or manual inspection. Because LLMs can produce paraphrases such as 'Taylor Swift' for ground-truth 'Taylor Alison Swift,' the F1 values in Table 3 may underestimate performance or reflect an artifact of a brittle parser. Since every downstream conclusion in Section 4.4 depends on these F1 values, the authors must provide the parsing script or a precise, deterministic matching specification and report scores under alternative matching rules.
- [Section 3, Table 3] All LLM results are reported as single numbers with no standard deviations or error bars. The ICL examples are randomly selected from the training set (Section 3), and both Llama2 and GPT-3.5-turbo have stochastic decoding, so the F1 scores in Table 3 are random variables. The authors should run multiple seeds with different random ICL selections and decoding temperatures and report mean plus/minus standard deviation, or at least the range across runs. Without this, the reported gaps of roughly 8-20 F1 points cannot be distinguished from run-to-run variation.
- [Section 3.1, Section 3.2.2] The visual feature pathway for Llama2 and ChatGPT is underspecified. The paper claims that visual tokens H_v are fed into the LLM, but Llama2-7B is a text-only language model and GPT-3.5-turbo does not accept custom visual token embeddings through its public API. The authors need to clarify whether images were actually used for these two models, how the projector W_l was trained or initialized, and which exact model variant and API were used. If Llama2 and ChatGPT received text only, then the comparison in Table 3 is not a multimodal comparison for those rows, and the 'multimodal understanding' claim in the abstract is not supported.
- [Section 4.2, Table 3] The inference-time comparison is not controlled. Section 4.2 states that all experiments are conducted on an RTX3090, while Table 3 reports times as 'run on single A100 GPU,' and ChatGPT timing includes API-call waiting time. The authors must specify the exact hardware and software versions for each row, whether local models were run with batching, and how API latency was measured. As written, the large time gap between LLMs and supervised methods may be partly a hardware and protocol artifact.
- [Table 3, Section 4.3] The comparison mixes reproduced and quoted results. Only DTCA is marked as reproduced; RoBERTa, AoM, and DQPSA numbers are presumably taken from their original papers. If those numbers used slightly different evaluation scripts, data splits, or preprocessing, then the comparison is not apples-to-apples. The authors should run all baselines with the same evaluation harness or, at minimum, provide the exact source of each number and release the evaluation script so that LLM outputs and baseline outputs are scored identically.
minor comments (6)
- [CCS Concepts and Keywords] The CCS Concepts and keywords fields are still template placeholders ('Do Not Use This Code...'); these should be filled in before submission.
- [Table 2] The descriptive text in Table 2 says 'neural aspects' where 'neutral aspects' is intended, and the caption lists '#Pos, #Neu, #Neg' but the text says 'positive aspects' twice; please correct these typos.
- [Section 4.4] The acronym 'SLM' is used without definition; if it stands for 'supervised learning method,' the term should be introduced, since 'SLM' conventionally denotes small language models.
- [References] References [36] and [37] are identical; the dataset citation should be distinguished from the original Adaptive Co-attention Network paper.
- [Figures] Figure 1 contains typos ('Postive' and 'Robert De Niro declare'), and Figure 2's text omits part of the Twitter example; please proofread the figures.
- [Section 4.4] The phrase 'additional response decay' is unclear; it should probably read 'response delay' or 'decoding time' for external APIs.
Circularity Check
No circularity: the central comparison uses external benchmarks and independent evaluation; the main risk is underspecified answer parsing, not circular reasoning.
full rationale
The paper's central claim is an empirical comparison: Table 3 reports F1 for Llama2, LLaVA, and ChatGPT against RoBERTa, DTCA, AoM, and DQPSA on Twitter-2015 and Twitter-2017, with annotations from Lu et al. The LLMs are evaluated zero/few-shot using ICL examples drawn from the training set, and the test-set labels are not used to fit any parameter that is then reported as a prediction. No equation in the paper defines the outcome in terms of its own inputs; the visual-feature pipeline adopted from LLaVA is an implementation detail, not a fitted constant or ansatz whose output forces the measured gap. There are no load-bearing self-citations: earlier sentiment findings are attributed to external work (e.g., Zhang Wenxuan et al. [31], Ma et al. [19]), and the author does not invoke a prior uniqueness theorem. The paper does overstate the novelty of 'constructing' a benchmark that reuses existing datasets, but that is a framing issue, not circularity. The one substantive weakness is methodological transparency: Section 3.2.1 states that evaluation is based on the generated '[Answer]' field, and Section 4.1 scores an aspect as correct only when the aspect term and polarity match the ground truth, yet the paper never specifies how free-form LLM text is parsed, normalized, or matched. A brittle exact-format matcher could undercount semantically correct LLM outputs and thereby exaggerate the gap, but this is an evaluation/reproducibility concern rather than a circular derivation: the LLM outputs are independent of the ground-truth labels, and no fitted parameter is renamed as a prediction. Given the external benchmark and independent comparisons, there is no significant circularity.
Assumptions & free parameters
free parameters (2)
- ICL example count lambda =
10 for Llama2/LLaVA, 5 for ChatGPT
- Parsing rule for model outputs =
Unspecified
assumptions (3)
- domain assumption The Twitter-2015 and Twitter-2017 annotations correctly represent the MABSA task.
- domain assumption The comparison numbers for RoBERTa and AoM are accurate as cited.
- domain assumption Experiments were run on a single A100 GPU as stated in Table 3.
Cite this review
Pith. "Pith review of Exploring Large Language Models for Multimodal Sentiment Analysis: Challenges, Benchmarks, and Future Directions." pith.science (2026). https://pith.science/paper/AYNVNMPV
@misc{pith2026241115408,
author = {Pith},
title = {Pith review of: Exploring Large Language Models for Multimodal Sentiment Analysis: Challenges, Benchmarks, and Future Directions},
year = {2026},
howpublished = {\url{https://pith.science/paper/AYNVNMPV}},
note = {Machine review of arXiv:2411.15408}
}
read the original abstract
Multimodal Aspect-Based Sentiment Analysis (MABSA) aims to extract aspect terms and their corresponding sentiment polarities from multimodal information, including text and images. While traditional supervised learning methods have shown effectiveness in this task, the adaptability of large language models (LLMs) to MABSA remains uncertain. Recent advances in LLMs, such as Llama2, LLaVA, and ChatGPT, demonstrate strong capabilities in general tasks, yet their performance in complex and fine-grained scenarios like MABSA is underexplored. In this study, we conduct a comprehensive investigation into the suitability of LLMs for MABSA. To this end, we construct a benchmark to evaluate the performance of LLMs on MABSA tasks and compare them with state-of-the-art supervised learning methods. Our experiments reveal that, while LLMs demonstrate potential in multimodal understanding, they face significant challenges in achieving satisfactory results for MABSA, particularly in terms of accuracy and inference time. Based on these findings, we discuss the limitations of current LLMs and outline directions for future research to enhance their capabilities in multimodal sentiment analysis.
Figures
Reference graph
Works this paper leans on
-
[1]
Omar Adjali, Romaric Besançon, Olivier Ferret, Hervé Le Borgne, and Brigitte Grau. 2020. Multimodal entity linking for tweets. In European Conference on Information Retrieval. Springer, 463–478
2020
-
[2]
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski,...
-
[3]
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision . 2425–2433
2015
-
[4]
UC Berkeley, Stanford Cmu, and UC San. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality. (2023)
work page 2023
-
[5]
Jingru Gan, Jinchang Luo, Haiwei Wang, Shuhui Wang, Wei He, and Qingming Huang. 2021. Multimodal entity linking: a new dataset and a baseline. In Pro- ceedings of the 29th ACM International Conference on Multimedia . 993–1001
work page 2021
-
[6]
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. Cornell University - arXiv,Cornell University - arXiv (Jun 2020)
2020
-
[7]
MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga
-
[8]
Ehsan Hosseini-Asl, Wenhao Liu, and Caiming Xiong. 2022. A Generative Lan- guage Model for Few-shot Aspect-Based Sentiment Analysis. (Apr 2022)
work page 2022
Show all 43 references
-
[9]
Shamim Hossain, Min Chen, Abdulhameed Alelaiwi, and Muneer Al-Hammadi
Yingying Jiang, Wei Li, M. Shamim Hossain, Min Chen, Abdulhameed Alelaiwi, and Muneer Al-Hammadi. 2020. A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition. Information Fusion (Jan 2020), 209–221. https://doi.org/10.1016/...
2020 doi
-
[10]
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021)
2021 arXiv
-
[11]
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.. InProceedings of the 58t...
2020 doi
-
[12]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. [n. d.]. Blip-2: Bootstrap- ping language-image pre-training with frozen image encoders and large language models. ([n. d.])
-
[13]
Qian Li, Hao Peng, Jianxin Li, Congying Xia, Renyu Yang, Lichao Sun, Philip S Yu, and Lifang He. 2020. A survey on text classification: From shallow to deep learning. arXiv preprint arXiv:2008.00364 (2020). Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Trovato et al
2020 arXiv
-
[14]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruc- tion Tuning. (2023)
2023
-
[15]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Michael Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. Cornell University - arXiv,Cornell University - arXiv (Jul 2019)
2019
-
[16]
Adam Lopez. 2008. Statistical machine translation. ACM Computing Surveys (CSUR) 40, 3 (2008), 1–49
2008
-
[17]
Ilya Loshchilov and Frank Hutter. 2017. Decoupled Weight Decay Regularization. Learning,Learning (Nov 2017)
2017
-
[18]
Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018. Visual Attention Model for Name Tagging in Multimodal Social Media. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). https://doi.org/10.18653...
2018 doi
-
[19]
Yubo Ma, Yixin Cao, YongChing Hong, and Aixin Sun. 2023. Large language model is not a good few-shot information extractor, but a good reranker for hard samples! arXiv preprint arXiv:2303.08559 (2023)
2023 arXiv
- [20]
-
[21]
Tianshuo Peng, Zuchao Li, Ping Wang, Lefei Zhang, and Hai Zhao. 2024. A Novel Energy Based Model Mechanism for Multi-Modal Aspect-Based Sentiment Analysis. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 38. 18869–18878
2024
-
[22]
Yongfeng Qian, Yin Zhang, Xiao Ma, Han Yu, and Limei Peng. 2019. EARS: Emotion-aware recommender system based on hybrid information fusion. Infor- mation Fusion 46 (Mar 2019), 141–146. https://doi.org/10.1016/j.inffus.2018.06.004
2019 doi
-
[23]
Senbao Shi, Zhenran Xu, Baotian Hu, and Min Zhang. 2023. Generative Multi- modal Entity Linking. arXiv preprint arXiv:2306.12725 (2023)
2023 arXiv
-
[24]
Mingkang Tang, Zhanyu Wang, Zhaoyang Zeng, Xiu Li, and Luping Zhou. 2023. Stay in Grid: Improving Video Captioning via Fully Grid-Level Representation. IEEE Transactions on Circuits and Systems for Video Technology 33, 7 (2023), 3319–
2023
-
[25]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth’ee Lacroix, Baptiste Rozi‘ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lam- ple. [n. d.]. LLaMA: Open and Efficient Founda...
-
[26]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
2023 arXiv
-
[27]
Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021. Automated Concatenation of Embeddings for Struc- tured Prediction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internatio...
2021 doi
-
[28]
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amir- reza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. a...
2022 arXiv
-
[29]
Yaxian Wang, Bifan Wei, Jun Liu, Qika Lin, Lingling Zhang, and Yaqiang Wu
-
[30]
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021)
2021 arXiv
-
[31]
Zhang Wenxuan, Deng Yue, Liu Bing, Pan SinnoJialin, and Bing Lidong. 2023. Sentiment Analysis in the Era of Large Language Models: A Reality Check. (2023)
2023
-
[33]
Binwei Yao, Ming Jiang, Diyi Yang, and Junjie Hu. 2023. Empowering LLM-based Machine Translation with Cultural Awareness. arXiv preprint arXiv:2305.14328 (2023)
2023 arXiv
-
[34]
Zhewen Yu and Jin Wang. 2022. Dual-Encoder Transformers with Cross-modal Alignment for Multimodal Aspect-based Sentiment Analysis. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computa- tional Linguistics and the 12th International Joi...
2022
-
[35]
Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy. 2023. Contextual Object Detection with Multimodal Large Language Models. arXiv preprint arXiv:2305.18279 (2023)
2023 arXiv
-
[37]
Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2022. Adaptive Co- attention Network for Named Entity Recognition in Tweets. Proceedings of the AAAI Conference on Artificial Intelligence (Nov 2022). https://doi.org/10.1609/ aaai.v32i1.11962
2022
-
[38]
You Zhang, Jin Wang, and Xuejie Zhang. 2021. Learning sentiment sentence representation with multiview attention model. Information Sciences 571 (Sep 2021), 459–474. https://doi.org/10.1016/j.ins.2021.05.044
2021 doi
-
[39]
Shan Zhao, Minghao Hu, Zhiping Cai, Zhanjun Zhang, Tongqing Zhou, and Fang Liu. 2021. Enhancing Chinese character representation with lattice-aligned attention. IEEE Transactions on Neural Networks and Learning Systems (2021)
2021
-
[40]
Shan Zhao, ChengYu Wang, Minghao Hu, Tianwei Yan, and Meng Wang. 2023. MCL: multi-granularity contrastive learning framework for Chinese NER. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 14011–14019
2023
-
[41]
Ru Zhou, Wenya Guo, Xumeng Liu, Shenglong Yu, Ying Zhang, and Xiaojie Yuan
-
[45]
arXiv preprint arXiv:2306.01004 (2023)
AoM: Detecting Aspect-oriented Information for Multimodal Aspect-Based Sentiment Analysis. arXiv preprint arXiv:2306.01004 (2023). Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009
2023 arXiv
-
[2019]
ACM Computing Surveys (CsUR) 51, 6 (2019), 1–36
A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CsUR) 51, 6 (2019), 1–36
2019
-
[2023]
IEEE Transactions on Circuits and Systems for Video Technology 33, 7 (2023), 3214–3228
Spatial-Semantic Collaborative Graph Network for Textbook Question Answering. IEEE Transactions on Circuits and Systems for Video Technology 33, 7 (2023), 3214–3228. https://doi.org/10.1109/TCSVT.2022.3231463
2023
-
[3332]
https://doi.org/10.1109/TCSVT.2022.3232634
2022
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.