Pith. sign in

REVIEW 3 major objections 4 minor 25 references

LLaVAC: Fine-tuning LLaVA as a Multimodal Sentiment Classifier

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning the LLaVA vision-language model with a structured prompt that asks for image, text, and joint sentiment labels produces a multimodal sentiment classifier that outperforms eight baselines on the MVSA-Single dataset.

desk verdict A clean, honest empirical study showing LLaVA can be fine-tuned into a competitive multimodal sentiment classifier, but the SOTA claim rests on uncontrolled baseline comparisons and no significance testing. read the letter →

arxiv 2502.02938 v1 pith:5SIH5Q46 submitted 2025-02-05 cs.CL

T. Chay-intr , Y. Chen , K. Viriyayudhakorn , T. Theeramunkong This is my paper · ORCID
classification cs.CL
keywords multimodalsentimentanalysisLLaVAfine-tuningpromptdesignMVSA-SingleLoRAvision-languagemodelsclassificationfusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces LLaVAC, a method that turns the LLaVA vision-language model into a multimodal sentiment classifier by fine-tuning it with a structured prompt that asks for three labels at once: the sentiment of the image, the sentiment of the text, and the sentiment of their combination. On the MVSA-Single dataset, the method is reported to outperform eight previous multimodal sentiment analysis baselines across three different data-splitting procedures, reaching 79.46% accuracy and 79.00 weighted F1 on a random 8:1:1 split and 82.85% and 82.03 on a split used by one of the baselines. The authors claim this demonstrates that a fine-tuned vision-language model, without complex feature engineering, is a strong foundation for multimodal sentiment classification. They also show through an ablation that including the unimodal image and text labels in the prompt improves accuracy by 1.33 points and weighted F1 by 2.95 points over fine-tuning on the multimodal label alone.

What carries the argument

The central object is a structured prompt-response template used for both fine-tuning and zero-shot inference. The prompt instructs LLaVA to consider the image and text, classify the sentiment of each modality separately, then jointly analyze both to produce a single multimodal label, with the response formatted as three short labels such as 'positive, positive, positive'. This design converts LLaVA's generative output into a constrained classifier, and the ablation indicates that the explicit unimodal intermediate labels are what lift performance beyond a version that only asks for the multimodal label.

What would settle it

Run LLaVAC on the exact MVSA-Single train, validation, and test splits used by the original baseline papers, applying the same hashtag and link removal that those baselines used, and check whether the accuracy and weighted F1 gaps in Table 2 persist; alternatively, re-run a top baseline such as CBAM on the paper's own splits with hashtags retained to see whether its scores move closer to LLaVAC's.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that fine-tuning LLaVA v1.5-7b for a single epoch with a classification-oriented prompt yields state-of-the-art performance on MVSA-Single, outperforming eight baseline methods across all three evaluation procedures. The prompt structure matters: the model is directed to label image, text, and multimodal sentiment sequentially, with the unimodal labels acting as context for the joint decision. The reported numbers are 76.48 accuracy and 75.84 F1 on Cheema-style splits, 79.46 and 79.00 on the random split, and 82.85 and 82.03 on the Zhang-style split, all beating the strongest cited baselines in each column. The ablation attributes part of the gain to the multi-label prompt design rather than to the base model alone.

Load-bearing premise

The load-bearing premise is that LLaVAC's results are directly comparable to the baseline scores quoted from earlier papers, even though the data splits and text preprocessing differ; if the splits or preprocessing are not matched, the performance gaps may reflect different test sets rather than method quality.

Editorial extensions

If this is right

  • If the reported results hold, fine-tuning a vision-language model with a multi-label prompt is a viable alternative to specialized fusion architectures for multimodal sentiment analysis.
  • The multi-label prompt design implies that explicit intermediate unimodal predictions can serve as auxiliary supervision for the joint task, consistent with the ablation results.
  • A single epoch of LoRA fine-tuning being sufficient suggests that strong multimodal sentiment classifiers can be built at lower training cost than training specialized fusion models from scratch.
  • LLaVAC establishes a new reference point that subsequent MVSA-Single evaluations would be measured against.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because LLaVAC retains hashtags and links while some baselines remove them, and because the random and 10-fold splits are generated by the paper rather than taken from the baseline papers, the exact margins in Table 2 may not transfer to a strictly controlled re-comparison; this is an editorial caution, not a claim made by the paper.
  • The same prompt template could be applied to other image-text classification settings with per-modality plus joint labels, such as product review polarity or rumor stance detection, though the paper does not test these tasks.
  • The improvement of the full prompt over the multimodal-only prompt in the ablation suggests that requiring the model to commit to unimodal labels first may act as a regularization or reasoning scaffold; future work could vary the prompt to test this mechanism directly.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes LLaVAC, a method that fine-tunes LLaVA-v1.5-7b with LoRA on the MVSA-Single dataset to predict image, text, and multimodal sentiment labels from a structured prompt. The prompt is designed so that the model first classifies image and text sentiments independently and then jointly classifies the multimodal sentiment; the authors argue that including unimodal labels as training signals helps the multimodal prediction. The method is evaluated on three data-processing procedures (a 10-fold split, a random 8:1:1 split with seed 42, and the split from Zhang et al.), and the paper reports that LLaVAC outperforms eight baselines in accuracy and weighted F1, including on the random split, where it reports 79.46 accuracy and 79.00 F1. An ablation removes the unimodal labels from the prompt and reports a drop of 1.33 accuracy points and 2.95 F1 points. The authors release both training code and a fine-tuned model checkpoint.

Significance. If the reported comparisons are valid, the contribution is a clean, reproducible demonstration that a general-purpose multimodal LLM can serve as a competitive sentiment classifier after only one epoch of LoRA fine-tuning, with no task-specific fusion architecture and no manual feature engineering. The prompt design is a simple and plausible mechanism for injecting unimodal supervision, and the ablation results support the value of that supervision. The public release of code and weights is a concrete strength that lowers the barrier for replication. However, the headline claim of state-of-the-art performance is load-bearing and currently depends on comparing internally generated splits and different text preprocessing against numbers reported in papers that used different protocols; this is not established by the manuscript as written.

major comments (3)
  1. [§4.1, §4.2, §4.4, Table 2] The claim that LLaVAC outperforms all eight baselines is not supported by a controlled comparison. Section 4.1 states that the 10-fold splits are divided by the authors and that the random split is generated using seed 42, while the baselines from Cheema et al., Wang et al., and Zhang et al. are cited with their reported scores; there is no evidence that the baselines were evaluated on the same test folds. Section 4.2 additionally states that LLaVAC retains hashtags and links, whereas Cheema et al. removed them, and that LLaVAC does not use a validation set. Consequently, the gaps in Table 2, such as 79.46 versus 77.11 on the random split and 76.48 versus 75.33 on the Cheema-style split, may be caused by different test sets or preprocessing rather than by the method itself. The authors should either run the baselines under their own protocol, obtain and use the exact original splits, or substantially soften the state-of-the-art claim and label the comparisons as reported numbers under different settings.
  2. [§4.3, §4.4, Table 2] No measure of uncertainty is reported. The 10-fold results are given only as averages without standard deviations, and the random split appears to be a single run. The gap of 1.15 accuracy points over CBAM on the random split and 1.15 points over Se-MLNN on the 10-fold split is within the range that split variability could plausibly explain, especially with no significance test. The authors should report per-fold standard deviations, run multiple seeds (the configuration includes random seed and LoRA rank), and, where possible, apply paired tests to support the superiority claim.
  3. [§4.5, Table 3] The ablation evidence for the unimodal labels is suggestive but not fully quantified. The comparison is based on a single random split, and the authors interpret the larger gain in weighted F1 (2.95 points) as indicating 'enhanced balance between precision and recall across sentiment classes.' Weighted F1 alone does not establish improved class balance; per-class precision, recall, and support, or a confusion matrix, are needed. Reporting multiple seeds and standard deviations would also strengthen this result.
minor comments (4)
  1. [§4.1] The phrase 'divided it into 10 splits, each containing training, validation, and test sets' is ambiguous; the authors should state explicitly that this is 10-fold cross-validation, and clarify what the validation split is used for given that Section 4.2 says no validation set is used by LLaVAC.
  2. [Table 2] For the Zhang et al. split, only one baseline (QMF) is listed and no F1 is reported for it, but Section 4.4 says LLaVAC outperforms all baselines across 'all evaluation metrics'; this should be qualified to the metrics for which baseline values exist.
  3. [§6] The Limitations section lists domain applicability, configuration optimization, and prompt sensitivity, but does not mention the uncontrolled baseline comparison or the lack of statistical significance testing; these are central to the paper's main claim and should be discussed.
  4. [Overall] The model is referred to inconsistently as 'LLaVAC' in the title and 'LLaV AC' in the body text; the authors should standardize the spelling.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LLaVAC is an empirical fine-tuning pipeline evaluated on held-out splits against external baselines.

full rationale

The paper's derivation chain is an empirical training-and-evaluation procedure, not a formal derivation. LLaVAC fine-tunes LLaVA on training splits of MVSA-Single and reports accuracy and weighted F1 on held-out test splits (Sections 4.2-4.4). The target metric is computed solely from the multimodal label, which is part of the dataset labels used as supervision, not a quantity fitted to the test set. The unimodal labels are auxiliary training signals in the prompt, and the ablation (Table 3) compares two prompt variants under the same random split, so the claimed benefit of unimodal labels is an empirical contrast, not a definitional identity. No parameter is fitted to the test set, no uniqueness theorem is imported, and the self-references are to the base LLaVA model and its standard hyperparameters, which are external resources. Concerns about differing splits and preprocessing (e.g., retaining hashtags and links versus baseline preprocessing) affect comparability and external validity, but they are not circularity: they do not make the reported result equal to its inputs by construction. Score 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The main listed free parameters are the fine-tuning hyperparameters, chosen from LLaVA defaults rather than tuned. The axioms are the assumptions about dataset quality, baseline comparability, base model suitability, and split equivalence.

free parameters (6)
  • Number of training epochs = 1
    Chosen to test minimal training; not tuned on a validation set.
  • Batch size = 32
    Standard LLaVA fine-tuning setting.
  • Initial learning rate = 2e-5
    Follows LLaVA default hyperparameters.
  • Random seed = 42
    Set for reproducibility of the random data split and training.
  • Temperature = 0.01
    Low temperature chosen for near-deterministic label outputs.
  • LoRA rank = 128
    LoRA rank for efficient fine-tuning.
assumptions (4)
  • domain assumption MVSA-Single sentiment labels are correct and representative for multimodal sentiment analysis.
    The entire evaluation relies on this dataset's labels.
  • domain assumption Baseline numbers cited from prior papers are accurately reported and directly comparable to LLaVAC results.
    Comparisons use published numbers, not reruns under identical conditions.
  • domain assumption LLaVA v1.5-7b provides a suitable pretrained representation for sentiment classification.
    The method's performance depends on the base model's quality.
  • domain assumption The 10-fold and random splits generated by the authors are equivalent to the splits used in the cited baseline papers.
    The paper divides the dataset itself in two of the three procedures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLaVAC: Fine-tuning LLaVA as a Multimodal Sentiment Classifier." pith.science (2026). https://pith.science/paper/5SIH5Q46

@misc{pith2026250202938,
  author       = {Pith},
  title        = {Pith review of: LLaVAC: Fine-tuning LLaVA as a Multimodal Sentiment Classifier},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5SIH5Q46}},
  note         = {Machine review of arXiv:2502.02938}
}
read the original abstract

We present LLaVAC, a method for constructing a classifier for multimodal sentiment analysis. This method leverages fine-tuning of the Large Language and Vision Assistant (LLaVA) to predict sentiment labels across both image and text modalities. Our approach involves designing a structured prompt that incorporates both unimodal and multimodal labels to fine-tune LLaVA, enabling it to perform sentiment classification effectively. Experiments on the MVSA-Single dataset demonstrate that LLaVAC outperforms existing methods in multimodal sentiment analysis across three data processing procedures. The implementation of LLaVAC is publicly available at https://github.com/tchayintr/llavac.

Figures

Figures reproduced from arXiv: 2502.02938 by the authors.

Figure 1
Figure 1. Our LLaVAC method that utilizes fine-tuned [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Example prompt with its response, including [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Example prompt with its response, including [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 8 canonical work pages

  1. [1]

    A Fair and Comprehensive Comparison of Multimodal Tweet Sentiment Analysis Methods

    Gullal S. Cheema, Sherzod Hakimov, Eric Müller-Budack, and Ralph Ewerth. 2021. https://arxiv.org/abs/2106.08829 A fair and comprehensive comparison of multimodal tweet sentiment analysis methods . Preprint, arXiv:2106.08829

    work page Pith review arXiv 2021

  2. [2]

    Junyu Chen, Jie An, Hanjia Lyu, Christopher Kanan, and Jiebo Luo. 2024. https://arxiv.org/abs/2211.12981 Holistic visual-textual sentiment analysis with prior models . Preprint, arXiv:2211.12981

    work page Pith review arXiv 2024

  3. [3]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    doi:10.18653/v1/n19-1423 2019

  4. [4]

    Ankita Gandhi, Kinjal Adhvaryu, Soujanya Poria, Erik Cambria, and Amir Hussain. 2023. https://doi.org/10.1016/j.inffus.2022.09.025 Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions . Information Fusion, 91:424--444

    doi:10.1016/j.inffus.2022.09.025 2023

  5. [5]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. https://arxiv.org/abs/1512.03385 Deep residual learning for image recognition . Preprint, arXiv:1512.03385

    arXiv 2015

  6. [6]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

    arXiv 2021

  7. [7]

    Tao Jiang, Jiahai Wang, Zhiyue Liu, and Yingbiao Ling. 2020. https://doi.org/10.1007/978-3-030-47436-2_59 Fusion-extraction network for multimodal sentiment analysis

    work page doi:10.1007/978-3-030-47436-2_59 2020

  8. [8]

    Zhen Li, Bing Xu, Conghui Zhu, and Tiejun Zhao. 2022. https://doi.org/10.18653/v1/2022.findings-naacl.175 CLMLF :a contrastive learning and multi-layer fusion method for multimodal sentiment detection . In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2282--2294, Seattle, United States. Association for Computational Linguistics

    work page doi:10.18653/v1/2022.findings-naacl.175 2022

Show all 25 references
  1. [9]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023 a . https://arxiv.org/abs/2310.03744 Improved baselines with visual instruction tuning . Preprint, arXiv:2310.03744

    2023 arXiv

  2. [10]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 b . https://arxiv.org/abs/2304.08485 Visual instruction tuning . Preprint, arXiv:2304.08485

    2023 arXiv

  3. [11]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692

    2019 arXiv

  4. [12]

    Alexandre, and Joao Cordeiro

    Vasco Lopes, Antonio Gaspar, Luis A. Alexandre, and Joao Cordeiro. 2021. https://doi.org/10.1109/ijcnn52387.2021.9533552 An automl-based approach to multimodal image sentiment analysis . In 2021 International Joint Conference on Neural Networks (IJCNN). IEEE

    2021

  5. [13]

    Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. 2023. https://arxiv.org/abs/2307.06435 A comprehensive overview of large language models . Preprint, arXiv:2307.06435

    2023 arXiv

  6. [14]

    Cheng Peng, Chunxia Zhang, Xiaojun Xue, Jiameng Gao, Hongjian Liang, and Zhengdong Niu. 2022. https://api.semanticscholar.org/CorpusID:245025098 Cross-modal complementary network with hierarchical fusion for multimodal sentiment classification . Tsinghua Science and Technology

    2022

  7. [15]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. https://arxiv.org/abs/2103.00020 Learning transferable visual models from natural lan...

    2021 arXiv

  8. [16]

    Danae Sanchez Villegas, Daniel Preotiuc-Pietro, and Nikolaos Aletras. 2024. https://aclanthology.org/2024.findings-eacl.76 Improving multimodal classification of social media posts by leveraging image-text auxiliary tasks . In Findings of the Association for Computational Ling...

    2024

  9. [17]

    Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023 a . https://doi.org/10.18653/v1/2023.findings-emnlp.603 Text classification via large language models . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages ...

    2023 doi

  10. [18]

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. 2023 b . https://arxiv.org/abs/2309.14525 Aligning large multimodal models with factually augmented rlhf . Pre...

    2023 arXiv

  11. [19]

    Huiru Wang, Xiuhong Li, Zenyu Ren, Dan Yang, and chunming Ma. 2023. https://arxiv.org/abs/2303.14708 Exploring multimodal sentiment analysis via cbam attention and double-layer bilstm architecture . Preprint, arXiv:2303.14708

    2023 arXiv

  12. [20]

    Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. 2018. https://arxiv.org/abs/1807.06521 Cbam: Convolutional block attention module . Preprint, arXiv:1807.06521

    2018 arXiv

  13. [21]

    Nan Xu and Wenji Mao. 2017. https://doi.org/10.1145/3132847.3133142 Multisentinet: A deep semantic network for multimodal sentiment analysis . In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM '17, page 2399–2402, New York, NY, USA. Ass...

    2017

  14. [22]

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023. https://arxiv.org/abs/2309.17421 The dawn of lmms: Preliminary explorations with gpt-4v(ision) . Preprint, arXiv:2309.17421

    2023 arXiv

  15. [23]

    Qingyang Zhang, Haitao Wu, Changqing Zhang, Qinghua Hu, Huazhu Fu, Joey Tianyi Zhou, and Xi Peng. 2023. https://arxiv.org/abs/2306.02050 Provable dynamic fusion for low-quality multimodal data . Preprint, arXiv:2306.02050

    2023 arXiv

  16. [24]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  17. [25]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.