{"id":"b4787269-5e65-4975-a5b5-e5b7358bf36c","arxiv_id":"2608.01238","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Qwen2-VL and Qwen3-VL beat 2022 ImageArg baselines in zero-shot Logos and Ethos F1, but Qwen3 falls below baseline on Pathos despite the abstract's claim.","lead":"Researchers tested two Qwen vision language models on detecting Aristotle's three persuasion styles, logic, credibility, and emotion, in tweets with images. The models beat older baselines on most tasks, but the paper's claim that Qwen3 excels on all three is undercut by its own Pathos score.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's claim that Qwen3 is exceptional on Pathos contradicts Table 2 (F1=0.696 vs baseline 0.712); central narrative overstates results.","rationale":"The reader's verdict is CONDITIONAL. I agree with the conditional outcome, but the most load-bearing concern is not (only) the contamination risk identified by the reader; it is that the paper's headline claim is internally contradicted by its own results. The abstract and conclusion state Qwen3 is exceptionally good at Pathos, while Table 2 shows the opposite. This is a factual error in the central narrative. A simple recomputation and bootstrap would settle whether the Pathos result is meaningful. The verdict remains CONDITIONAL because the underlying data may support a corrected claim (Qwen2 improves on all modes, Qwen3 on Logos/Ethos), but the paper must be revised. I partially agree with the reader's weakest assumption: contamination is a real threat, but the internal contradiction is more immediate and concrete.","tokens_in":5007,"tokens_out":11595,"duration_ms":104285,"concrete_test":"Bootstrap the 259 test instances (e.g., 10,000 resamples) to compute 95% CIs for each model's F1 on Logos, Pathos, and Ethos; if the CI for Qwen3 Pathos includes the baseline's 0.712, the claim of exceptional Pathos performance is unsupported. Independently recompute the F1 values from the reported precision/recall in Table 2 and correct the abstract to match the table if the contradiction is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the abstract and Section 4 is that Qwen3 performs exceptionally well on Logos and Pathos. Table 2 directly contradicts this: Qwen3's Pathos F1 is 0.696, below Qwen2 (0.714) and below the baseline (0.712). The only mode where Qwen3 is clearly best is Ethos (0.365 vs 0.272). Furthermore, Qwen2's Pathos gain over baseline is 0.002 F1, which is within chance for a 259-instance test set. The paper reports no error bars, no significance tests, and no multiple runs, so even the Logos/Ethos improvements may not be stable. The abstract thus misrepresents the paper's own results; this is an internal inconsistency, not just a matter of external validity. The comparison to supervised baselines also lacks a contamination check (Section 5.2), but the Pathos claim fails even before considering contamination.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript evaluates two open vision-language models, Qwen2-VL-7B-Instruct and Qwen3-VL-8B-Instruct, in a zero-shot setting on the ImageArg persuasion-mode detection tasks (Logos, Pathos, Ethos). It compares their precision, recall, and F1 against the best supervised baselines reported by Liu et al. (2022) and claims that Qwen models achieve the highest F1 scores on all persuasion modes. The paper also releases code for reproducibility. The clearest positive result in Table 2 is Qwen3's Ethos score, but the abstract's claim that Qwen3 performs exceptionally well on Logos and Pathos is contradicted by the paper's own table: Qwen3's Pathos F1 is 0.696, below the 0.712 baseline. The evaluation is a single-run zero-shot comparison with no significance tests and no contamination checks.","tokens_in":5235,"tokens_out":6280,"duration_ms":59997,"significance":"If the Qwen3 Ethos improvement (0.365 vs. 0.272) replicates, it would be a useful data point for multimodal argumentation mining and for the capability of current VLMs on complex persuasion-mode tasks. The paper's strengths are its use of an established benchmark, direct comparison to published baselines, and its release of code; it does not introduce fitted parameters or circular derivations. The current contribution is limited by the absence of uncertainty quantification, the zero-shot-versus-supervised confound, and internal contradictions between the abstract/discussion and the reported numbers. The paper is a candidate for major revision rather than acceptance as-is.","major_comments":[{"comment":"The abstract and conclusion claim that 'Qwen3 performs exceptionally well on Logos and Pathos tasks, while Qwen2 exhibits competitive performance on the more complex Ethos detection task.' Table 2 contradicts this. On Pathos, Qwen3's F1 is 0.696, below the 0.712 baseline and below Qwen2's 0.714. On Logos, Qwen3 ties Qwen2 at 0.709. The only clear Qwen3 advantage is Ethos (0.365 vs. 0.272). The narrative and the table cannot both stand; the text must be corrected to reflect the actual ranking, or the tables and experimental results must be rechecked.","section":"Abstract; §3.2; Table 2"},{"comment":"No uncertainty is reported and no significance testing is performed. With a test set of 259 instances, the Pathos difference between Qwen2 (0.714) and the baseline (0.712) is a 0.002 F1 gap, which could be within chance for this sample size; the Qwen2 Ethos gain over T-M (0.277 vs. 0.272) is similarly negligible. The claim that 'Qwen models achieved the highest F1-scores across all persuasion modes' requires confidence intervals, multiple seeds, or an instance-level significance test such as McNemar's test. Without this, the improvements are not established.","section":"§3.2; Table 2"},{"comment":"The zero-shot comparison to supervised baselines is confounded by possible data contamination. Section 5.2 states that the authors 'assume that the training instances for these tasks were included in the models' training,' which, if true, would invalidate the comparison; if false, the suggested n-shot alternative is misdirected. In addition, using the ImageArg annotation coding manual as the prompt (§3.1) increases the risk that the prompt itself contains dataset-specific cues. Please add a contamination or memorization check, benchmark on a freshly collected sample, and discuss what the reported numbers mean under each assumption.","section":"§5.2; §3.1"},{"comment":"The sentence 'Ethos is the only task where both models were very close to the baseline' is contradicted by Table 2. Qwen3's Ethos F1 of 0.365 is the largest relative improvement over any baseline in the table (baseline 0.272), while Qwen2's Ethos F1 of 0.277 is close; Logos shows a 0.134-point gain for both models. The discussion of Ethos as the 'difficult' task should be revised to account for Qwen3's strong Ethos result and the mixed Pathos outcome.","section":"§3.2; Table 2"}],"minor_comments":[{"comment":"The experimental setup is underspecified: what exact prompt was used, how the model's free-form output was mapped to Logos/Ethos/Pathos labels, what decoding parameters (temperature, max tokens) were used, and whether one or more runs were performed. The phrase 'set generation prompt as TRUE' is unclear and should be explained.","section":"§3.1"},{"comment":"The opening sentence and the phrase 'resembles a triangle shape, which highlights its inherent challenges related to personal biases' are vague. The connection between the persuasion triangle and personal biases is not established; please rewrite.","section":"§1.1"},{"comment":"The column heading 'Alpha' should specify Krippendorff's α, and the values appear to be percentages (55.3, 51.0, 57.8) while the text says 'ranged from 50% to 58%'. Please make the units and notation consistent.","section":"Table 1"},{"comment":"The first sentence of §5.2 says 'we assume that the training instances for these tasks were included in the models' training' but the next sentence says 'However, this may not necessarily be the case.' If the authors truly assume inclusion, the comparison is already invalid; if they do not, the assumption should be stated differently. Please clarify the intended threat model.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: I would flag that the manuscript reads more like a student seminar report than a full research paper. The contribution is narrow (two models, one benchmark, zero-shot only), and the abstract materially misstates the results. The most valuable next step is a rigorous re-analysis with significance tests, contamination checks, and corrected claims; without those, the paper is not ready for a research journal. Scope fit should also be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this is a compact empirical note, not a breakthrough. It reports zero-shot F1 scores for two Qwen VLMs on the ImageArg persuasion-mode tasks (Logos, Pathos, Ethos). The underlying numbers are new and could be a useful data point for the argument-mining crowd. But the abstract and conclusion oversell the results—specifically, they say Qwen3 performs exceptionally well on Pathos, while Table 2 shows its F1 (0.696) is below both Qwen2 (0.714) and the baseline (0.712). That is a direct internal contradiction, and it needs fixing before anyone trusts the narrative.\n\nWhat the paper does well: it is transparent about its setup, uses the dataset's own annotation manual as the prompt, reports precision/recall/F1, and releases code. The limitations section is unusually candid—it explicitly acknowledges that Qwen's training data is unknown and that the zero-shot comparison to supervised baselines may be contaminated. That kind of honesty is worth crediting.\n\nThe soft spots are the usual ones for a small zero-shot study, but they matter here because the claims are keyed to small differences. No error bars, no multiple seeds, no significance tests. The test set has only 259 instances, so a 0.002 F1 difference (Qwen2 vs. baseline on Pathos) is noise. The contamination issue is real and acknowledged, but it doesn't rescue the Pathos claim—even taking the numbers at face value, Qwen3 is not better than the baseline on Pathos. The comparison to supervised baselines is also confounded by unknown pretraining, which the authors admit.\n\nAnother minor point: the paper only tests one model family (Qwen), so we learn nothing about how other VLMs would do. That's fine for a workshop-style study, but it limits the generality of any positive conclusion.\n\nWho is this for? Anyone working on multimodal argumentation or persuasion detection who wants a quick sense of how off-the-shelf VLMs compare to the 2022 ImageArg baselines. It is not a methodological advance, but it is a legitimate evaluation.\n\nMy recommendation: send it to peer review, but require the authors to correct the abstract and conclusion to match Table 2, add at least a basic variance estimate (seeds or bootstrapping), and reframe the Pathos claim. With those changes, it becomes a modest but honest contribution. As written, the central narrative overstates the evidence.","headline":"Small but honest zero-shot VLM evaluation on ImageArg; the abstract overstates Qwen3's Pathos result, which is actually below baseline, but the paper is redeemable with corrections.","tokens_in":5670,"tokens_out":1291,"would_cite":false,"duration_ms":14990,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that zero-shot Qwen vision-language models beat the 2022 ImageArg supervised baselines on all three persuasion-mode tasks, with Qwen3 raising Ethos F1 from 0.272 to 0.365.","keywords":["vision-language models","persuasion modes","logos","pathos","ethos","ImageArg","zero-shot evaluation","argumentation mining"],"falsifier":"Annotate a fresh sample of about 259 gun-control tweet-plus-image pairs with the same ImageArg manual, run the same Qwen2/Qwen3 zero-shot prompts, and compare F1 against the paper's baselines ($0.575$ Logos, $0.712$ Pathos, $0.272$ Ethos); if the Qwen models do not beat those numbers on the fresh sample, the claimed zero-shot advantage is an artifact of test-set leakage.","tokens_in":4893,"feed_emoji":"💬","tokens_out":13805,"duration_ms":126102,"temperature":0.7,"pith_summary":"This paper tests whether general vision-language models can classify the three Aristotelian persuasion modes—logos (reasoning), pathos (emotion), and ethos (credibility)—in tweet-plus-image arguments. Running Qwen2-VL-7B-Instruct and Qwen3-VL-8B-Instruct zero-shot on the ImageArg gun-control dataset, with the original annotation manual as the prompt, it reports that the best Qwen model beats every 2022 supervised baseline: logos $F_1$ rises from 0.575 to 0.709, pathos from 0.712 to 0.714, and ethos from 0.272 to 0.365. The largest relative gain is on ethos, the mode with the lowest human annotator agreement and the weakest old text baseline. The paper concludes that newer Qwen generations have improved at reading images and applying annotation rules without task-specific training, making a reusable step toward automatic multimodal persuasion analysis.","feed_headline":"Zero-shot Qwen models beat 2022 baselines on persuasion tasks","feed_subtitle":"A two-model Qwen study finds modern VLMs match or beat supervised baselines on logos, pathos, and ethos recognition.","key_machinery":"The load-bearing mechanism is the ImageArg coding manual reused as a zero-shot prompt. The paper feeds the same annotation instructions that built the dataset directly to two Qwen models, sets the generation prompt to TRUE, and asks for a classification on each test tweet-image pair; the comparison point is the original corpus's published baselines (random bm and text-only T-M for Logos/Pathos and Ethos respectively). Aristotle's persuasion triangle—logos, pathos, ethos—is the classification scheme that gives the task its three overlapping labels, and the low annotator agreement around $\\alpha=0.51$–$0.58$ is what makes the apparent VLM improvement nontrivial.","core_discovery":"On its own terms, the paper's discovery is that zero-shot generative VLMs can outperform supervised classifiers on ImageArg's multimodal persuasion-mode task. With no gradient updates, Qwen2-VL-7B-Instruct and Qwen3-VL-8B-Instruct achieve the best $F_1$ on each mode: both reach 0.709 on logos against the 0.575 baseline, Qwen2 reaches 0.714 on pathos against 0.712, and Qwen3 reaches 0.365 on ethos against 0.272. The paper attributes Qwen3's ethos jump to better image processing and numerical-image understanding, and Qwen2's pathos edge to slightly better recognition of hidden emotional appeal. The paper leaves the image-to-text contribution to improvement open, noting that an analysis of how","pith_inferences":["A direct ablation the paper anticipates but does not run—text-only, image-only, and combined prompts—would tell whether Qwen3's Ethos advantage is genuinely multimodal; if image-only prompts match or exceed text-only prompts, the model is reading visual credibility cues rather than the tweet text.","Because the ImageArg coding manual is public, the manual-as-prompt protocol could transfer to other annotated multimodal argumentation tasks, such as fallacy detection or stance detection, giving future work a zero-shot baseline without new training.","A cross-topic replication on non-gun-control arguments would separate a general VLM capability for persuasion from topic-specific exposure to gun-control discourse during pretraining."],"forward_implications":["The annotation-manual-as-prompt setup can serve as a reusable zero-shot benchmark for tracking VLM progress on persuasion detection, removing the need to retrain a classifier for each new model.","Qwen3's Ethos gain points to a specific capability—reading credibility cues from images, such as a person portrayed as trustworthy—that older text-only supervised models lacked, making Ethos a useful diagnostic for multimodal reasoning.","If the zero-shot result reproduces on fresh data, ImageArg becomes a cheap capability probe for open VLM families, complementing visual question answering and image captioning benchmarks.","Pathos being nearly flat suggests emotional persuasion is the mode where raw model scale alone does not help; progress there may require modeling audience-specific bias rather than better image-text fusion."],"supporting_citations":[{"why":"Supplies the ImageArg dataset, the persuasion-mode test set and annotation manual used as the zero-shot prompt, and every baseline F1 number the Qwen results are compared against.","marker":"Liu et al. (2022)"},{"why":"Documents the Qwen3 model family evaluated in the Ethos experiment; Qwen3's 0.365 Ethos F1 is the paper's largest improvement.","marker":"Yang et al. (2025)"},{"why":"Defines BERT, the text-only T-M baseline whose Ethos F1 of 0.272 is the benchmark Qwen3 improves to 0.365.","marker":"Devlin et al. (2019)"},{"why":"Defines ResNet50, the image encoder behind the ImageArg I-M and M-M baselines that the Qwen models surpass on F1.","marker":"He et al. (2016)"}],"fun_headline_variants":["Qwen VLMs outscore supervised baselines on persuasion modes","Zero-shot Qwen3 beats baselines on Logos and Ethos detection","Qwen2 and Qwen3 top supervised F1 on persuasion tasks","Modern VLMs surpass 2022 baselines in multimodal persuasion","Qwen vision-language models win on Logos, Pathos, Ethos"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The zero-shot comparison collapses if the Qwen models have already seen ImageArg or very similar examples during pretraining, because model training data are not public and the paper states it cannot verify this.","fun_headline_variants_meta":{"raw":{"variants":["Qwen VLMs outscore supervised baselines on persuasion modes","Zero-shot Qwen3 beats baselines on Logos and Ethos detection","Qwen2 and Qwen3 top supervised F1 on persuasion tasks","Modern VLMs surpass 2022 baselines in multimodal persuasion","Qwen vision-language models win on Logos, Pathos, Ethos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000578,"raw_usage":{"total_tokens":2528,"prompt_tokens":680,"completion_tokens":1848,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":424,"completion_tokens_details":{"reasoning_tokens":1764}},"tokens_in":424,"tokens_out":1848,"duration_ms":12824,"temperature":1.0,"reasoning_tokens":1764,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:25:28.986792+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Annotate a fresh sample of about 259 gun-control tweet-plus-image pairs with the same ImageArg manual, run the same Qwen2/Qwen3 zero-shot prompts, and compare F1 against the paper's baselines ($0.575$ Logos, $0.712$ Pathos, $0.272$ Ethos); if the Qwen models do not beat those numbers on the fresh sample, the claimed zero-shot advantage is an artifact of test-set leakage.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ImageArg dataset, the persuasion-mode test set and annotation manual used as the zero-shot prompt, and every baseline F1 number the Qwen results are compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines BERT, the text-only T-M baseline whose Ethos F1 of 0.272 is the benchmark Qwen3 improves to 0.365."}],"review_version":1}