Evaluating VLMs on Multimodal Aristotelian Persuasion Tasks

Khondoker Ittehadul Islam

REVIEW 4 major objections 4 minor 14 references

Evaluating VLMs on Multimodal Aristotelian Persuasion Tasks

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper reports that zero-shot Qwen vision-language models beat the 2022 ImageArg supervised baselines on all three persuasion-mode tasks, with Qwen3 raising Ethos F1 from 0.272 to 0.365.

desk verdict Small but honest zero-shot VLM evaluation on ImageArg; the abstract overstates Qwen3's Pathos result, which is actually below baseline, but the paper is redeemable with corrections. read the letter →

arxiv 2608.01238 v1 pith:NO6NR2IM submitted 2026-08-02 cs.CL cs.MM

Khondoker Ittehadul Islam This is my paper · ORCID

classification cs.CLcs.MM

keywords vision-languagemodelspersuasionmodeslogospathosethosImageArgzero-shotevaluationargumentationmining

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether general vision-language models can classify the three Aristotelian persuasion modes—logos (reasoning), pathos (emotion), and ethos (credibility)—in tweet-plus-image arguments. Running Qwen2-VL-7B-Instruct and Qwen3-VL-8B-Instruct zero-shot on the ImageArg gun-control dataset, with the original annotation manual as the prompt, it reports that the best Qwen model beats every 2022 supervised baseline: logos $F_1$ rises from 0.575 to 0.709, pathos from 0.712 to 0.714, and ethos from 0.272 to 0.365. The largest relative gain is on ethos, the mode with the lowest human annotator agreement and the weakest old text baseline. The paper concludes that newer Qwen generations have improved at reading images and applying annotation rules without task-specific training, making a reusable step toward automatic multimodal persuasion analysis.

What carries the argument

The load-bearing mechanism is the ImageArg coding manual reused as a zero-shot prompt. The paper feeds the same annotation instructions that built the dataset directly to two Qwen models, sets the generation prompt to TRUE, and asks for a classification on each test tweet-image pair; the comparison point is the original corpus's published baselines (random bm and text-only T-M for Logos/Pathos and Ethos respectively). Aristotle's persuasion triangle—logos, pathos, ethos—is the classification scheme that gives the task its three overlapping labels, and the low annotator agreement around $\alpha=0.51$–$0.58$ is what makes the apparent VLM improvement nontrivial.

What would settle it

Annotate a fresh sample of about 259 gun-control tweet-plus-image pairs with the same ImageArg manual, run the same Qwen2/Qwen3 zero-shot prompts, and compare F1 against the paper's baselines ($0.575$ Logos, $0.712$ Pathos, $0.272$ Ethos); if the Qwen models do not beat those numbers on the fresh sample, the claimed zero-shot advantage is an artifact of test-set leakage.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that zero-shot generative VLMs can outperform supervised classifiers on ImageArg's multimodal persuasion-mode task. With no gradient updates, Qwen2-VL-7B-Instruct and Qwen3-VL-8B-Instruct achieve the best $F_1$ on each mode: both reach 0.709 on logos against the 0.575 baseline, Qwen2 reaches 0.714 on pathos against 0.712, and Qwen3 reaches 0.365 on ethos against 0.272. The paper attributes Qwen3's ethos jump to better image processing and numerical-image understanding, and Qwen2's pathos edge to slightly better recognition of hidden emotional appeal. The paper leaves the image-to-text contribution to improvement open, noting that an analysis of how

Load-bearing premise

The zero-shot comparison collapses if the Qwen models have already seen ImageArg or very similar examples during pretraining, because model training data are not public and the paper states it cannot verify this.

Editorial extensions

If this is right

The annotation-manual-as-prompt setup can serve as a reusable zero-shot benchmark for tracking VLM progress on persuasion detection, removing the need to retrain a classifier for each new model.
Qwen3's Ethos gain points to a specific capability—reading credibility cues from images, such as a person portrayed as trustworthy—that older text-only supervised models lacked, making Ethos a useful diagnostic for multimodal reasoning.
If the zero-shot result reproduces on fresh data, ImageArg becomes a cheap capability probe for open VLM families, complementing visual question answering and image captioning benchmarks.
Pathos being nearly flat suggests emotional persuasion is the mode where raw model scale alone does not help; progress there may require modeling audience-specific bias rather than better image-text fusion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct ablation the paper anticipates but does not run—text-only, image-only, and combined prompts—would tell whether Qwen3's Ethos advantage is genuinely multimodal; if image-only prompts match or exceed text-only prompts, the model is reading visual credibility cues rather than the tweet text.
Because the ImageArg coding manual is public, the manual-as-prompt protocol could transfer to other annotated multimodal argumentation tasks, such as fallacy detection or stance detection, giving future work a zero-shot baseline without new training.
A cross-topic replication on non-gun-control arguments would separate a general VLM capability for persuasion from topic-specific exposure to gun-control discourse during pretraining.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing you should know: this is a compact empirical note, not a breakthrough. It reports zero-shot F1 scores for two Qwen VLMs on the ImageArg persuasion-mode tasks (Logos, Pathos, Ethos). The underlying numbers are new and could be a useful data point for the argument-mining crowd. But the abstract and conclusion oversell the results—specifically, they say Qwen3 performs exceptionally well on Pathos, while Table 2 shows its F1 (0.696) is below both Qwen2 (0.714) and the baseline (0.712). That is a direct internal contradiction, and it needs fixing before anyone trusts the narrative.

What the paper does well: it is transparent about its setup, uses the dataset's own annotation manual as the prompt, reports precision/recall/F1, and releases code. The limitations section is unusually candid—it explicitly acknowledges that Qwen's training data is unknown and that the zero-shot comparison to supervised baselines may be contaminated. That kind of honesty is worth crediting.

The soft spots are the usual ones for a small zero-shot study, but they matter here because the claims are keyed to small differences. No error bars, no multiple seeds, no significance tests. The test set has only 259 instances, so a 0.002 F1 difference (Qwen2 vs. baseline on Pathos) is noise. The contamination issue is real and acknowledged, but it doesn't rescue the Pathos claim—even taking the numbers at face value, Qwen3 is not better than the baseline on Pathos. The comparison to supervised baselines is also confounded by unknown pretraining, which the authors admit.

Another minor point: the paper only tests one model family (Qwen), so we learn nothing about how other VLMs would do. That's fine for a workshop-style study, but it limits the generality of any positive conclusion.

Who is this for? Anyone working on multimodal argumentation or persuasion detection who wants a quick sense of how off-the-shelf VLMs compare to the 2022 ImageArg baselines. It is not a methodological advance, but it is a legitimate evaluation.

My recommendation: send it to peer review, but require the authors to correct the abstract and conclusion to match Table 2, add at least a basic variance estimate (seeds or bootstrapping), and reframe the Pathos claim. With those changes, it becomes a modest but honest contribution. As written, the central narrative overstates the evidence.

Referee Report

4 major / 4 minor

Summary. This manuscript evaluates two open vision-language models, Qwen2-VL-7B-Instruct and Qwen3-VL-8B-Instruct, in a zero-shot setting on the ImageArg persuasion-mode detection tasks (Logos, Pathos, Ethos). It compares their precision, recall, and F1 against the best supervised baselines reported by Liu et al. (2022) and claims that Qwen models achieve the highest F1 scores on all persuasion modes. The paper also releases code for reproducibility. The clearest positive result in Table 2 is Qwen3's Ethos score, but the abstract's claim that Qwen3 performs exceptionally well on Logos and Pathos is contradicted by the paper's own table: Qwen3's Pathos F1 is 0.696, below the 0.712 baseline. The evaluation is a single-run zero-shot comparison with no significance tests and no contamination checks.

Significance. If the Qwen3 Ethos improvement (0.365 vs. 0.272) replicates, it would be a useful data point for multimodal argumentation mining and for the capability of current VLMs on complex persuasion-mode tasks. The paper's strengths are its use of an established benchmark, direct comparison to published baselines, and its release of code; it does not introduce fitted parameters or circular derivations. The current contribution is limited by the absence of uncertainty quantification, the zero-shot-versus-supervised confound, and internal contradictions between the abstract/discussion and the reported numbers. The paper is a candidate for major revision rather than acceptance as-is.

major comments (4)

[Abstract; §3.2; Table 2] The abstract and conclusion claim that 'Qwen3 performs exceptionally well on Logos and Pathos tasks, while Qwen2 exhibits competitive performance on the more complex Ethos detection task.' Table 2 contradicts this. On Pathos, Qwen3's F1 is 0.696, below the 0.712 baseline and below Qwen2's 0.714. On Logos, Qwen3 ties Qwen2 at 0.709. The only clear Qwen3 advantage is Ethos (0.365 vs. 0.272). The narrative and the table cannot both stand; the text must be corrected to reflect the actual ranking, or the tables and experimental results must be rechecked.
[§3.2; Table 2] No uncertainty is reported and no significance testing is performed. With a test set of 259 instances, the Pathos difference between Qwen2 (0.714) and the baseline (0.712) is a 0.002 F1 gap, which could be within chance for this sample size; the Qwen2 Ethos gain over T-M (0.277 vs. 0.272) is similarly negligible. The claim that 'Qwen models achieved the highest F1-scores across all persuasion modes' requires confidence intervals, multiple seeds, or an instance-level significance test such as McNemar's test. Without this, the improvements are not established.
[§5.2; §3.1] The zero-shot comparison to supervised baselines is confounded by possible data contamination. Section 5.2 states that the authors 'assume that the training instances for these tasks were included in the models' training,' which, if true, would invalidate the comparison; if false, the suggested n-shot alternative is misdirected. In addition, using the ImageArg annotation coding manual as the prompt (§3.1) increases the risk that the prompt itself contains dataset-specific cues. Please add a contamination or memorization check, benchmark on a freshly collected sample, and discuss what the reported numbers mean under each assumption.
[§3.2; Table 2] The sentence 'Ethos is the only task where both models were very close to the baseline' is contradicted by Table 2. Qwen3's Ethos F1 of 0.365 is the largest relative improvement over any baseline in the table (baseline 0.272), while Qwen2's Ethos F1 of 0.277 is close; Logos shows a 0.134-point gain for both models. The discussion of Ethos as the 'difficult' task should be revised to account for Qwen3's strong Ethos result and the mixed Pathos outcome.

minor comments (4)

[§3.1] The experimental setup is underspecified: what exact prompt was used, how the model's free-form output was mapped to Logos/Ethos/Pathos labels, what decoding parameters (temperature, max tokens) were used, and whether one or more runs were performed. The phrase 'set generation prompt as TRUE' is unclear and should be explained.
[§1.1] The opening sentence and the phrase 'resembles a triangle shape, which highlights its inherent challenges related to personal biases' are vague. The connection between the persuasion triangle and personal biases is not established; please rewrite.
[Table 1] The column heading 'Alpha' should specify Krippendorff's α, and the values appear to be percentages (55.3, 51.0, 57.8) while the text says 'ranged from 50% to 58%'. Please make the units and notation consistent.
[§5.2] The first sentence of §5.2 says 'we assume that the training instances for these tasks were included in the models' training' but the next sentence says 'However, this may not necessarily be the case.' If the authors truly assume inclusion, the comparison is already invalid; if they do not, the assumption should be stated differently. Please clarify the intended threat model.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports zero-shot VLM benchmark numbers against external ImageArg baselines, with no fitted parameters, derived predictions, or self-citation chain carrying the central claim.

full rationale

The paper is an empirical evaluation study, not a derivation. It takes the ImageArg dataset and its published baselines (bm, T-M) as external inputs, runs two Qwen VLMs zero-shot, and reports Precision/Recall/F1. There are no fitted parameters, no equations, and no quantity is defined in terms of another quantity it is then said to predict. The claim that Qwen models 'achieved improved F1 scores' is a direct report of the numbers in Table 2, not a prediction derived from the data. The one self-citation (Zhou et al., 2025, which overlaps with the author) appears only as background motivation in Section 1.3 and is not load-bearing: the evaluation would stand unchanged without it. Using the ImageArg annotation manual as the prompt is an evaluation design choice that raises possible contamination concerns, but it is not a circularity step because the prompt is not derived from the model outputs and the results are still externally benchmarked against the original baselines. The abstract's statement that Qwen3 performs 'exceptionally well on the Logos and Pathos tasks' is contradicted by Table 2 for Pathos (Qwen3 F1=0.696 below the baseline 0.712), but that is an internal consistency/overstatement issue, not circularity. No circular step can be quoted or exhibited from the paper's derivations because no derivations are attempted.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This paper introduces no fitted parameters and no new theoretical entities. Its central comparison rests on the trustworthiness of the ImageArg labels and baselines, on the validity of zero-shot comparisons without contamination checks, on single-run stability, and on the prompt design.

assumptions (4)

domain assumption ImageArg annotations and baseline scores reported by Liu et al. (2022) are correct and comparable.
The paper relies on the published dataset, inter-annotator agreement, and baseline F1 numbers as ground truth without re-verifying them (used throughout Section 2 and Table 2).
domain assumption Zero-shot evaluation is meaningful even though Qwen models may have seen ImageArg or similar persuasion data during pretraining.
Authors explicitly assume this in Limitations Section 5.2: 'we assume that the training instances for these tasks were included in the models' training. However, this may not necessarily be the case.'
domain assumption A single test-set run without error bars is representative of model performance.
Methodology Section 3.1 describes one run per model on a single A100 GPU with default generation parameters and no variance reporting.
ad hoc to paper Using the ImageArg annotation coding manual as the prompt does not contaminate the task.
Section 3.1 says the prompt is the coding manual and instructions used to build the dataset. This is not in the original evaluation protocol and could artificially raise scores by leaking annotation criteria.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating VLMs on Multimodal Aristotelian Persuasion Tasks." pith.science (2026). https://pith.science/paper/NO6NR2IM

@misc{pith2026260801238,
  author       = {Pith},
  title        = {Pith review of: Evaluating VLMs on Multimodal Aristotelian Persuasion Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NO6NR2IM}},
  note         = {Machine review of arXiv:2608.01238}
}

read the original abstract

Vision Language Models (VLMs) have demonstrated exceptional performance across various tasks. However, they have not yet been thoroughly evaluated on more complex tasks. The Persuasion Model, conceived by Aristotle, resembles a triangle shape, which highlights its inherent challenges related to personal biases. To assess the progress of VLMs on these complex tasks, we use the ImageArg datasets, focusing on the Logos, Ethos, and Pathos detection tasks. Our findings indicate that models from the Qwen family achieve improved F1 scores, with Qwen3 performing exceptionally well on the Logos and Pathos tasks, while Qwen2 exhibits competitive performance on the more complex Ethos detection task. We release the code to foster research in this direction.

Figures

Figures reproduced from arXiv: 2608.01238 by the authors.

**Figure 1.** The persuasion triangle, illustrating how logos, pathos, and ethos overlap and jointly contribute to persuasive message. 1.2 Computational Persuasion Mode Mining In the field of computational mining, the goal is to automatically identify persuasion structures and relationships. Argumentation mining and fallacy detection both require an understanding of persuasion modes. Research by Higgins and Walker (2012); Carlile… view at source ↗

**Figure 2.** Example of logos in the ImageArg dataset. The image presents statistical evidence on gun ownership and gun deaths across countries, while the accompanying tweet text interprets the observed relationship to support an argument about gun violence [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Example of ethos in the ImageArg dataset. The image depicts Abraham Lincoln in an American-themed outfit, with his hand covering his face in grief. The accompanying tweet conveys a personal narrative from an independent individual asking for strength. 4 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Example of pathos in the ImageArg dataset. The image features a paper-cut headline from The New York Times alongside a paper-cut excerpt from the newspaper that provides facts supporting the headline. The accompanying tweets reiterate those facts. 2.2 Multi-modal Bench…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 12 canonical work pages

[1]

L., and Parikh, D

Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. (2015). Vqa: Visual question answering. InProceedings of the IEEE international conference on computer vision, pages 2425–2433

work page 2015
[2]

Y., Ajay, A., Li, A

Bordes, F., Pang, R. Y., Ajay, A., Li, A. C., Bardes, A., Petryk, S., Ma˜ nas, O., Lin, Z., Mahmoud, A., Jayaraman, B., et al. (2024). An introduction to vision-language modeling.arXiv preprint arXiv:2405.17247

arXiv 2024
[3]

Carlile, W., Gurrapadi, N., Ke, Z., and Ng, V. (2018). Give me more feedback: Annotating argument persuasiveness and related attributes in student essays. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 621–631

work page 2018
[4]

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 con- ference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186

work page 2019
[5]

Guo, M., Hwa, R., Lin, Y.-R., and Chung, W.-T. (2020). Inflating topic relevance with ideology: A case study of political ideology bias in social topic detection models. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4873–4885

work page 2020
[6]

He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recogni- tion. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778

work page 2016
[7]

and Walker, R

Higgins, C. and Walker, R. (2012). Ethos, logos, pathos: Strategies of persuasion in social/environmental reports. InAccounting forum, volume 36, pages 194–208. Elsevier

work page 2012
[8]

Jaini, P., Clark, K., and Geirhos, R. (2024). Intriguing properties of generative classifiers. InInternational Conference on Learning Representations, volume 2024, pages 13898– 13923

work page 2024

Show all 14 references

[9]

Krippendorff, K. (2011). Computing krippendorff’s alpha-reliability

2011
[10]

Liu, Z., Guo, M., Dai, Y., and Litman, D. (2022). Imagearg: A multi-modal tweet dataset for image persuasiveness mining. InProceedings of the 9th Workshop on Argument Mining, pages 1–18

2022
[11]

Mancini, E., Ruggeri, F., Villata, S., and Torroni, P. (2025). Overview of mm- argfallacy2025 on multimodal argumentative fallacy detection and classification in po- litical debates. InProceedings of the 12th Argument mining Workshop, pages 358–368. 10

2025
[12]

Askell, A., Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR

2021
[13]

Lv, C., et al. (2025). Qwen3 technical report.arXiv preprint arXiv:2505.09388

2025 arXiv
[14]

Zhou, H., Westerdijk, H., and Islam, K. I. (2025). Joint effects of argumentation theory, audio modality and data enrichment on llm-based fallacy classification.arXiv preprint arXiv:2509.11127. 11

2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

L., and Parikh, D

Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. (2015). Vqa: Visual question answering. InProceedings of the IEEE international conference on computer vision, pages 2425–2433

work page 2015

[2] [2]

Y., Ajay, A., Li, A

Bordes, F., Pang, R. Y., Ajay, A., Li, A. C., Bardes, A., Petryk, S., Ma˜ nas, O., Lin, Z., Mahmoud, A., Jayaraman, B., et al. (2024). An introduction to vision-language modeling.arXiv preprint arXiv:2405.17247

arXiv 2024

[3] [3]

Carlile, W., Gurrapadi, N., Ke, Z., and Ng, V. (2018). Give me more feedback: Annotating argument persuasiveness and related attributes in student essays. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 621–631

work page 2018

[4] [4]

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 con- ference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186

work page 2019

[5] [5]

Guo, M., Hwa, R., Lin, Y.-R., and Chung, W.-T. (2020). Inflating topic relevance with ideology: A case study of political ideology bias in social topic detection models. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4873–4885

work page 2020

[6] [6]

He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recogni- tion. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778

work page 2016

[7] [7]

and Walker, R

Higgins, C. and Walker, R. (2012). Ethos, logos, pathos: Strategies of persuasion in social/environmental reports. InAccounting forum, volume 36, pages 194–208. Elsevier

work page 2012

[8] [8]

Jaini, P., Clark, K., and Geirhos, R. (2024). Intriguing properties of generative classifiers. InInternational Conference on Learning Representations, volume 2024, pages 13898– 13923

work page 2024