{"id":"88fefcc6-beae-4aaa-b600-0483d1b9e9d8","arxiv_id":"2411.15408","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On two Twitter multimodal aspect-based sentiment benchmarks, zero-shot/few-shot LLMs like Llama2, LLaVA, and ChatGPT score 7 to 16 F1 points below supervised baselines and take orders of magnitude longer to run.","lead":"This paper tests how well large language models like Llama2, LLaVA, and ChatGPT can extract aspects and sentiments from text plus images. It finds these models underperform specialized supervised methods on two Twitter multimodal sentiment benchmarks while being much slower.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported LLM performance gap may be an evaluation artifact: the paper never specifies how free-form LLM answers are parsed and matched to ground-truth aspect-polarity pairs, so correct predictions could be scored as errors.","rationale":"The reader's weakest assumption focused on missing standard deviations, seeds, and code release. That is a valid reproducibility concern but not the most attackable point. My concern is more fundamental: the evaluation protocol for LLM outputs is unspecified. Even with perfect reproducibility, if the parser is too strict or undisclosed, the central comparison between LLMs and SLMs is not a fair test of capability. The paper's conclusion that LLMs underperform rests entirely on Table 3, and Table 3 depends on converting free-form generated text into aspect-sentiment tuples. No such procedure appears in the manuscript. This is not a disagreement with the broader community finding that LLMs struggle with fine-grained extraction; it is a correctness risk specific to this paper's evidence. The reader's conditional verdict is appropriate because the claim is plausible but not firmly established. Adding the parser concern makes the condition more precise rather than changing the verdict, so I recommend no change to the reader's CONDITIONAL outcome. A favorable outcome of the proposed re-scoring test would substantially strengthen the paper; an unfavorable outcome would require the authors to weaken or qualify their central claim.","tokens_in":9494,"tokens_out":2588,"duration_ms":27915,"concrete_test":"Obtain or re-run the authors' evaluation on the raw LLM generations, then re-score the same outputs with a documented tolerant parser that (a) extracts all aspect-sentiment pairs from the Answer field, (b) matches aspects using exact span, normalized span, and substring containment, and (c) includes a manual audit of 100 responses per model. Also rerun with at least three random ICL selections and decoding temperatures. If the LLM F1 gains remain below 3 points and no LLM crosses an SLM baseline, the reported gap is robust. If any LLM F1 rises by more than 5 points or crosses the SLM baseline, the paper's headline conclusion must be substantially revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2.1 states that 'evaluation of model performance is based on the content of the generated [Answer] field,' and Section 4.1 defines a correct prediction only when the aspect term and polarity respectively match the ground truth. The paper never describes the parsing or matching procedure used to convert free-form LLM text into aspect-polarity pairs. There is no mention of exact string matching, substring containment, normalization, span alignment, or manual inspection. LLMs frequently generate extra words, different surface forms, or reordered aspects; for example, the ground truth may annotate 'Taylor Alison Swift' while the LLM reasonably outputs 'Taylor Swift.' If a brittle exact-format matcher is applied, semantically correct predictions are counted as errors. Since all LLM F1 values in Table 3 rest on this unspecified evaluation, the central claim that LLM-based methods exhibit a performance gap compared to traditional methods could be an artifact of measurement rather than a genuine limitation. The appendix promises code release, but no code, prompts, or raw outputs are provided, making the evaluation impossible to audit. This is the most load-bearing weakness because every downstream conclusion—accuracy gap, task unfamiliarity, ICL ineffectiveness, and practical usability—depends on the integrity of these numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LLM4SA, a zero/few-shot in-context learning framework for Multimodal Aspect-Based Sentiment Analysis (MABSA), and evaluates Llama2-7B, LLaVA, and GPT-3.5-turbo on Twitter-2015 and Twitter-2017 against supervised baselines. The central claim is that off-the-shelf LLMs underperform traditional supervised methods on MABSA accuracy and are substantially slower. The authors attribute this to task unfamiliarity, limited in-context learning effectiveness, and inference cost, and outline future directions for instruction tuning and efficient adaptation.","tokens_in":9710,"tokens_out":7125,"duration_ms":63568,"significance":"If the reported results are valid, the paper provides a useful negative result for practitioners and a starting benchmark for LLM-based MABSA. The use of public datasets, the explicit reproduction of the DTCA baseline, and the attempt to quantify inference time are strengths. However, the current evidence does not yet support the central claim, because the evaluation protocol for free-form LLM outputs is not specified, all LLM numbers come from single runs, and the visual input pathway for Llama2 and ChatGPT is unclear. With the evaluation harness, prompts, and variance estimates supplied, this could become a solid benchmark contribution; in its present form it reads as an early draft.","major_comments":[{"comment":"The paper never specifies how the free-form text in the generated '[Answer]' field is converted into aspect-polarity pairs. Section 4.1 states that an aspect is correct only if 'the aspect term and polarity respectively match the ground truth,' but no matching rule is defined: exact string match, substring containment, normalization of surface forms, span alignment, or manual inspection. Because LLMs can produce paraphrases such as 'Taylor Swift' for ground-truth 'Taylor Alison Swift,' the F1 values in Table 3 may underestimate performance or reflect an artifact of a brittle parser. Since every downstream conclusion in Section 4.4 depends on these F1 values, the authors must provide the parsing script or a precise, deterministic matching specification and report scores under alternative matching rules.","section":"Section 3.2.1, Section 4.1"},{"comment":"All LLM results are reported as single numbers with no standard deviations or error bars. The ICL examples are randomly selected from the training set (Section 3), and both Llama2 and GPT-3.5-turbo have stochastic decoding, so the F1 scores in Table 3 are random variables. The authors should run multiple seeds with different random ICL selections and decoding temperatures and report mean plus/minus standard deviation, or at least the range across runs. Without this, the reported gaps of roughly 8-20 F1 points cannot be distinguished from run-to-run variation.","section":"Section 3, Table 3"},{"comment":"The visual feature pathway for Llama2 and ChatGPT is underspecified. The paper claims that visual tokens H_v are fed into the LLM, but Llama2-7B is a text-only language model and GPT-3.5-turbo does not accept custom visual token embeddings through its public API. The authors need to clarify whether images were actually used for these two models, how the projector W_l was trained or initialized, and which exact model variant and API were used. If Llama2 and ChatGPT received text only, then the comparison in Table 3 is not a multimodal comparison for those rows, and the 'multimodal understanding' claim in the abstract is not supported.","section":"Section 3.1, Section 3.2.2"},{"comment":"The inference-time comparison is not controlled. Section 4.2 states that all experiments are conducted on an RTX3090, while Table 3 reports times as 'run on single A100 GPU,' and ChatGPT timing includes API-call waiting time. The authors must specify the exact hardware and software versions for each row, whether local models were run with batching, and how API latency was measured. As written, the large time gap between LLMs and supervised methods may be partly a hardware and protocol artifact.","section":"Section 4.2, Table 3"},{"comment":"The comparison mixes reproduced and quoted results. Only DTCA is marked as reproduced; RoBERTa, AoM, and DQPSA numbers are presumably taken from their original papers. If those numbers used slightly different evaluation scripts, data splits, or preprocessing, then the comparison is not apples-to-apples. The authors should run all baselines with the same evaluation harness or, at minimum, provide the exact source of each number and release the evaluation script so that LLM outputs and baseline outputs are scored identically.","section":"Table 3, Section 4.3"}],"minor_comments":[{"comment":"The CCS Concepts and keywords fields are still template placeholders ('Do Not Use This Code...'); these should be filled in before submission.","section":"CCS Concepts and Keywords"},{"comment":"The descriptive text in Table 2 says 'neural aspects' where 'neutral aspects' is intended, and the caption lists '#Pos, #Neu, #Neg' but the text says 'positive aspects' twice; please correct these typos.","section":"Table 2"},{"comment":"The acronym 'SLM' is used without definition; if it stands for 'supervised learning method,' the term should be introduced, since 'SLM' conventionally denotes small language models.","section":"Section 4.4"},{"comment":"References [36] and [37] are identical; the dataset citation should be distinguished from the original Adaptive Co-attention Network paper.","section":"References"},{"comment":"Figure 1 contains typos ('Postive' and 'Robert De Niro declare'), and Figure 2's text omits part of the Twitter example; please proofread the figures.","section":"Figures"},{"comment":"The phrase 'additional response decay' is unclear; it should probably read 'response delay' or 'decoding time' for external APIs.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is not ready in its current form for a refereed venue; the placeholder template fields suggest an early draft. If the authors can supply the evaluation harness, prompts, seeds, and multi-run variance estimates, the negative result would be a publishable benchmark and analysis contribution. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First probe of off-the-shelf LLMs on MABSA, and the qualitative conclusion—LLMs lag supervised baselines by about 10-15 F1 and are orders of magnitude slower—is plausible and consistent with the text-only 'Reality Check' literature. Credit where due: the LLM4SA framework with ICL on Twitter-2015/2017 is new, the comparison table is new, and the authors are measured in their discussion, not overclaiming.\n\nBut the evaluation is not auditable. The paper never specifies how free-form LLM outputs were parsed into aspect-sentiment pairs. Section 4.1 only says a prediction is correct if the aspect and polarity match ground truth, with no exact-match, substring, normalization, span alignment, or manual inspection described. LLMs rarely reproduce the exact surface form of an annotated aspect; if a brittle matcher was used, correct answers would be scored as errors and the entire performance gap could be an artifact. The stress-test note lands squarely here—this is the load-bearing weakness because every downstream claim rests on those numbers.\n\nAlso missing: standard deviations across seeds, random ICL draws, or decoding temperatures. The reported F1 differences between Llama2, LLaVA, and ChatGPT are small enough that single runs could flip the ordering. The baseline numbers for RoBERTa, AoM, and DQPSA are lifted from prior papers without settings; only DTCA was reproduced. And the paper is clearly a draft—ACM placeholder text, inconsistent reference numbering, no code link despite promising release.\n\nThe central argument may well be correct, but the current evidence does not establish the size or the cause of the gap. This paper is for practitioners who want a sanity check on zero/few-shot LLM performance on MABSA, and for researchers building reliable evaluation protocols for generative sentiment outputs. It deserves a serious referee only after the authors specify and code up the parsing, add seeds and error bars, and release the prompts and raw outputs. In its current form, I'd desk-reject on completeness grounds, with an invitation to resubmit a cleaned version.","headline":"A plausible first probe of LLMs on MABSA, but the missing output-parsing details and single-run numbers make the headline gap unverifiable.","tokens_in":10250,"tokens_out":4177,"would_cite":false,"duration_ms":37277,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Off-the-shelf large language models underperform supervised methods on multimodal aspect-based sentiment analysis.","keywords":["multimodal aspect-based sentiment analysis","large language models","in-context learning","benchmark","sentiment analysis","visual-language models","LLM evaluation","performance gap"],"falsifier":"Re-running the same LLM4SA prompts on Twitter-2015 and Twitter-2017 with, say, five different random draws of in-context examples and temperature 0.2, 0.7, and 1.0, and comparing the mean F1 against the reported values; if the best LLM's mean F1 reaches or exceeds the supervised baseline (71.9 on Twitter-2015, 70.6 on Twitter-2017), the claim of a consistent performance gap would be refuted.","tokens_in":9263,"feed_emoji":"📉","tokens_out":5938,"duration_ms":45004,"temperature":0.7,"pith_summary":"This paper asks whether large language models, prompted with a few examples, can perform multimodal aspect-based sentiment analysis (MABSA) as well as or better than traditional supervised models. It constructs a benchmark on the Twitter-2015 and Twitter-2017 datasets and evaluates Llama2, LLaVA, and ChatGPT under the proposed LLM4SA prompting framework. Its central finding is that all three LLMs lag behind supervised baselines by a wide margin in F1 score, while taking hundreds or thousands of times longer to run. The paper argues this gap comes from the models' unfamiliarity with the fine-grained task format, limited and unrepresentative in-context examples, and high computational cost. If this result holds, it is a caution that general-purpose LLMs are not yet ready for fine-grained multimodal sentiment analysis without task-specific adaptation.","feed_headline":"LLMs fall short of supervised models on aspect-based sentiment.","feed_subtitle":"Zero-shot tests on Twitter show up-to-20-point F1 gaps and 100x slower inference.","key_machinery":"The LLM4SA framework, which converts a MABSA datapoint into a visual prefix plus text and a question prompt, with a few in-context examples taken from the training set. Visual features are extracted with a frozen ViT and projected through a linear layer into the LLM's embedding space. The model must then generate, for each aspect in the text, an aspect–sentiment pair; correct prediction requires the model to solve the three subtasks of detecting how many aspect-sentiment pairs exist, identifying each aspect span, and assigning the correct polarity.","core_discovery":"The central claim is that off-the-shelf LLMs, used with in-context learning, do not reach the accuracy of published supervised systems on MABSA. On Twitter-2015, the best LLM (LLaVA) achieves 55.62 F1 against 71.90 for the supervised DQPSA; on Twitter-2017, LLaVA achieves 61.74 against 70.60 for DQPSA. The paper attributes the shortfall to three factors: the MABSA task format is rare in instruction-tuning data, so the models are not 'well-acquainted' with it; the constrained number of randomly selected ICL examples limits learning; and the models' inference cost is orders of magnitude higher. As a result, the paper concludes that LLM-based methods still exhibit a performance gap compared to traditional methods, despite their demonstrated ability in general multimodal understanding.","pith_inferences":["The paper leaves implicit that the strict token-level matching metric may penalize LLM generation formats more than their actual understanding; a human-readable or span-normalized evaluation could change the size of the gap.","The paper's random ICL selection suggests a testable extension: if ICL examples are chosen to be most similar to the query (e.g., by embedding distance), the LLM F1 might rise substantially, narrowing the reported gap.","Another extension is to fine-tune a 7B LLM on the MABSA training set rather than only prompting it; this would test whether the gap is due to inherent capability or to the prompting protocol.","The reported inference times mix API latency for ChatGPT with local generation for the open models, so a controlled local-generation comparison would better isolate model speed."],"forward_implications":["If the claim holds, zero- and few-shot prompting of current LLMs is not a viable substitute for supervised training on MABSA, and reported SOTA numbers for the task remain with supervised models.","The gap identifies a benchmark target: an LLM-based method that closes the F1 gap will need to handle joint aspect extraction and sentiment classification reliably.","The large inference-time gap implies that practical deployment of LLMs for fine-grained sentiment tasks requires substantial compression or acceleration, not just accuracy improvements.","The finding suggests that task-specific instruction tuning with MABSA-style data may be a necessary next step for LLMs, since general instruction tuning appears insufficient."],"supporting_citations":[{"why":"Supplies the Twitter-2015 and Twitter-2017 datasets on which the benchmark and all comparisons are built.","marker":"[36]"},{"why":"Provides the aspect-level sentiment annotations used as ground truth for the two Twitter datasets.","marker":"[18]"},{"why":"Text-only RoBERTa baseline that defines the lower bound of supervised performance the LLMs must beat.","marker":"[15]"},{"why":"DTCA multimodal baseline; the author's replication result is the direct multimodal comparison point.","marker":"[34]"},{"why":"AoM baseline, a recent supervised multimodal method whose F1 is used to show the gap.","marker":"[41]"},{"why":"DQPSA baseline, the strongest supervised system, setting the target that LLMs fail to reach.","marker":"[21]"},{"why":"Llama2, one of the three LLMs evaluated; its F1 and inference time are a core data point.","marker":"[26]"},{"why":"LLaVA, both an evaluated LLM and the source of the visual feature extraction pipeline used by the framework.","marker":"[14]"},{"why":"ChatGPT, evaluated through API calls; its F1 and time contribute to the central comparison.","marker":"[20]"},{"why":"Cited to support the explanation that MABSA is scarce in instruction-tuning data, explaining the LLMs' poor task familiarity.","marker":"[19]"}],"fun_headline_variants":["LLMs trail supervised models by 20 F1 on multimodal sentiment","Zero-shot LLMs lag tuned systems on aspect-based sentiment","LLM sentiment accuracy lags, inference runs 100x slower","Multimodal aspect sentiment: LLMs underperform supervised baselines","LLM fine-grained sentiment: 20-point gap, 100x slower inference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the single reported run for each model in Table 3 is representative: the paper reports no standard deviations across random in-context example draws, decoding temperatures, or random seeds, and only one number per model per dataset appears. If a re-run gave materially different F1 values, the size of the claimed gap would change accordingly.","fun_headline_variants_meta":{"raw":{"variants":["LLMs trail supervised models by 20 F1 on multimodal sentiment","Zero-shot LLMs lag tuned systems on aspect-based sentiment","LLM sentiment accuracy lags, inference runs 100x slower","Multimodal aspect sentiment: LLMs underperform supervised baselines","LLM fine-grained sentiment: 20-point gap, 100x slower inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000299,"raw_usage":{"total_tokens":1718,"prompt_tokens":925,"completion_tokens":793,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":701}},"tokens_in":541,"tokens_out":793,"duration_ms":6946,"temperature":1.0,"reasoning_tokens":701,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:18:52.416245+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the same LLM4SA prompts on Twitter-2015 and Twitter-2017 with, say, five different random draws of in-context examples and temperature 0.2, 0.7, and 1.0, and comparing the mean F1 against the reported values; if the best LLM's mean F1 reaches or exceeds the supervised baseline (71.9 on Twitter-2015, 70.6 on Twitter-2017), the claim of a consistent performance gap would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Text-only RoBERTa baseline that defines the lower bound of supervised performance the LLMs must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DTCA multimodal baseline; the author's replication result is the direct multimodal comparison point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DQPSA baseline, the strongest supervised system, setting the target that LLMs fail to reach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LLaVA, both an evaluated LLM and the source of the visual feature extraction pipeline used by the framework."}],"review_version":1}