Pith. sign in

REVIEW 3 major objections 4 minor 80 references

PragMatch: Separating Pragmatic Incongruity from Cross-Modal Mismatch in Large Vision-Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Large vision-language models fail to distinguish sarcastic incongruity from ordinary image–text mismatch.

desk verdict Solid relation-level benchmark and a striking paired-accuracy result, but the shortcut claim rests on perturbations that may change the label rather than only surface form. read the letter →

arxiv 2608.09772 v1 pith:XMZ5LSEK submitted 2026-08-10 cs.CL

Zhanna Mukhametsharip (1) , Vera Demberg (1 , 2) , Varsha Suresh (2) ((1) Saarland University , Germany , (2) Max Planck Institute for Informatics , Germany) This is my paper · ORCID
classification cs.CL
keywords PragMatchmultimodalsarcasmdetectionlargevision-languagemodelsshortcutlearningpragmaticincongruitypairedaccuracysurfacecuesbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PragMatch is a controlled test of whether large vision-language models (LVLMs) detect sarcasm by understanding pragmatic incongruity or by latching onto surface cues. The benchmark collects 3,000 image–caption pairs in three conditions that share the same image: sarcastic captions (P), literal descriptions (L), and non-pragmatic mismatches (M) drawn from visually similar images. The paper argues that standard accuracy overestimates pragmatic understanding because no model scores above chance on both P and M, and paired accuracy measuring joint correctness falls below 25% chance for three of four models. Masking and injection experiments show that lexical, stylistic, and OCR-derived cues can flip predictions even though the image–caption relationship and gold label are unchanged. If correct, this means current LVLMs are not performing genuine multimodal pragmatic reasoning, and evaluation must move to relation-level consistency and shortcut robustness.

What carries the argument

The central object is PragMatch, a benchmark that fixes the image and varies only the communicative relation among three captions: Pragmatic Incongruity (P), Non-Pragmatic Mismatch (M), and Literal Congruity (L). The load-bearing evaluation is paired and grouped accuracy, which credits a model only when it classifies all matched conditions from the same image correctly, complemented by AUROC over the continuous sarcasm score derived from yes/no token log-probabilities. The diagnostic machinery is a set of masking and injection interventions over three cue families — lexical markers, stylistic formatting, and simulated OCR text — that are asserted to modify only surface form while preserving the image–caption relation and gold label. A robust model should be invariant to these edits; prediction changes are attributed to shortcut reliance.

What would settle it

Give human annotators the original and perturbed caption pairs and ask whether the image–caption relationship (and the sarcasm label) is preserved; if a substantial share of OCR-injected pairs are judged to change the relationship or become sarcastic, the shortcut attribution collapses for that cue family. Alternatively, a model that achieves high paired accuracy on P–M and stays invariant under all three perturbation families would directly refute the claim.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that LVLM predictions in multimodal sarcasm detection are sensitive to lexical, OCR-derived, and stylistic cues, with injected surface signals causing substantial changes in model predictions despite unchanged underlying image–text relationships. No model reaches above-chance accuracy on both pragmatic incongruity (P) and non-pragmatic mismatch (M); models either default to non-sarcastic for both (InternVL2.5-8B, Qwen2.5-VL-7B, Idefics2-8B) or treat cross-modal inconsistency as sarcasm regardless of intent (LLaVA-v1.6-7B). Paired accuracy that requires both P and M to be correct for the same image falls below the 25% chance level for three of four models under zero-shot prompting, even when single-condition accuracy exceeds 96%. Score-level AUROC shows that some models (Qwen2.5-VL-7B at 90.0) separate the two classes in their continuous scores while still making wrong decisions, pointing to decision bias rather than missing relational information. Chain-of-thought prompting improves sarcastic-pair recognition but increases false sarcasm on non-pragmatic pairs and can lower AUROC, showing that reasoning traces do not fix the underlying relation modeling.

Load-bearing premise

The diagnosis assumes that masking and injection change only surface form and leave the image–caption relationship and the gold label untouched; if an injection such as appended OCR text genuinely changes what the caption means, then prediction flips are not necessarily shortcuts.

Editorial extensions

If this is right

  • Standard item-level accuracy and F1 on sarcasm benchmarks overestimate relational understanding; paired and grouped metrics should be reported alongside them.
  • A model can separate sarcasm from literal content in its continuous scores yet fail at the decision boundary, so accuracy gains from prompting do not imply better representation.
  • OCR-derived surface cues are the strongest shortcut family in this testbed; models that treat injected visible-text phrases as speaker evidence will be fooled by adversarial captions.
  • Text-only inputs match or exceed multimodal inputs for most tested models, implying that apparent visual grounding in sarcasm detection can be an illusion.
  • Chain-of-thought prompting is not a cure: it raises sarcasm recall but increases false positives on non-pragmatic mismatches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the perturbation premise is relaxed — particularly for OCR injection, where "Visible text in the image says: Yeah, right." adds new semantic content — some verdict flips are arguably correct responses to a changed caption rather than proof of shortcut reliance; human rating of perturbed pairs would settle this.
  • The same paired-accuracy design transfers to other pragmatic phenomena such as irony, humour, and indirect requests, where incongruity is communicative rather than arbitrary.
  • The benchmark's own text-only classifier result (99.6% accuracy) suggests that caption provenance and register differ across conditions, so cross-condition comparisons may partly measure style, and matching literal captions for register is a direct next step the paper itself flags.
  • One testable extension is to train or fine-tune a model on PragMatch's P–M contrast and check whether paired accuracy, not just item accuracy, improves; current results imply it would not without explicit relational supervision.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces PragMatch, a 3,000-pair benchmark derived from MMSD2.0 in which each source image is paired with a sarcastic pragmatic-incongruity caption (P), a literal caption (L), and a non-pragmatic mismatch caption (M). It evaluates four open LVLMs under zero-shot and chain-of-thought prompting, reporting standard accuracy, per-condition accuracy, paired/grouped accuracy, and AUROC. It further proposes masking and injection perturbations over lexical, style, and OCR cues to diagnose shortcut reliance. The central claim is that models are sensitive to surface cues and that injected cues change predictions even though the underlying image-text relation and gold label are supposedly unchanged; on this basis the authors argue that current LVLMs do not genuinely perform multimodal pragmatic reasoning.

Significance. If the central claim is supported, PragMatch is a valuable diagnostic resource: it separates pragmatic incongruity from generic cross-modal mismatch while holding the image fixed, and its paired/grouped accuracy metrics expose a genuine gap between standard accuracy and relation-level consistency. The paper also deserves credit for reporting AUROC separately from discrete decisions, for human validation of the original labels, and for constructing hard negatives with CLIP and RelSim rather than relying on trivially unrelated captions. However, the paper's headline shortcut-reliance conclusion currently rests on two unverified or internally inconsistent pieces of evidence: the injection interventions are not validated to preserve the gold label, and the perturbation results in Table 3 compare baselines and paired accuracies on different subsets in ways that produce impossible values. The benchmark itself is useful even if the causal shortcut claim is subsequently softened to a claim about cue sensitivity.

major comments (3)
  1. [Section 3.3 and Table 5] The claim that all interventions 'modify only surface form without altering the image-caption relationship or the gold label' is asserted but not validated. The OCR template appends 'Visible text in the image says: "Yeah, right."' to the caption; this is not a purely surface edit because it introduces a new factual assertion about the image, and it can plausibly change the relation from non-sarcastic to sarcastic. Lexical and style templates such as 'lol', 'ironic', and '#nice' are conventional sarcasm markers that can shift the perceived intent of an otherwise literal caption. Section 3.4 reports human validation only for the original labels, not for the perturbed pairs. Therefore the prediction flips in Table 3 (e.g., LLaVA-v1.6-7B's M accuracy dropping from 85.9 to 9.4 after OCR injection) do not yet establish shortcut reliance; they could reflect a reasonable update to a changed image-text relation. I ask the authors to either provide human ratings confirming that the gold label is invariant under the injected perturbations, or reframe the injection results as evidence of sensitivity to cue presence rather than as evidence of shortcut reliance.
  2. [Table 3 and Section 4.1] The baseline comparisons in Table 3 are not computed on matched subsets. Section 3.5 states that masking is applied only when the relevant cue is present (n=19 for lexical, n=638 for style, n=505 for OCR), but the reported baseline P values are the full-set AccP from Table 1. Consequently, a paired accuracy such as LLaVA-v1.6-7B's P-P- of 100.0 cannot be compared with its full-set baseline P of 62.5, and the negative 'Drop' values cannot be interpreted as evidence of response bias. The same problem appears on the injection side: Qwen2.5-VL-7B CoT reports M-M+ of 79.1 for the lexical cue against a baseline M of 61.5, which is impossible if M-M+ requires the original M prediction to be correct on the same examples. The manuscript needs to recompute baselines on the exact subset used for each paired metric, report those subset-specific baselines, and then re-evaluate the 'Drop' analysis and the claims about masking and injection in Section 5.2.
  3. [Section 5.1 and Table 1] The sentence 'no model is above chance on both P and M' is directly contradicted by the paper's own Table 1: LLaVA-v1.6-7B under zero-shot prompting achieves P=62.5 and M=85.9, both above 50. The paper appears to have meant that no model is above chance on both P and L, or that paired accuracy is below chance for three of four models, but as written the statement is false and it weakens the argument that no model shows per-condition relational competence. This needs correction, and the surrounding discussion should be adjusted so that the claim about below-chance behavior refers specifically to paired/grouped accuracy rather than to per-condition accuracy.
minor comments (4)
  1. [Section 5.1] In the per-condition discussion, LLaVA-v1.6-7B is described as scoring 62.5% on P but only 43.1% on L; the table also shows M=85.9, so the sentence should clarify that the contrast is with L, not with M.
  2. [Notation] The notation 'P –P −', 'P −–M +', and 'P–P −' is visually confusing because the minus signs are sometimes rendered as separate dashes; please use a consistent notation such as Pmasked, Minjected, and paired metrics defined once in Section 4.1.
  3. [Author line] There are typographical spacing issues in the author line: 'V era Demberg' and 'LLaV A-v1.6-7B' should be corrected to 'Vera Demberg' and 'LLaVA-v1.6-7B'.
  4. [Limitations] The Limitations paragraph correctly acknowledges that the three conditions differ in caption provenance and style, but the abstract's global wording ('current LVLMs do not perform genuine multimodal pragmatic reasoning') is stronger than what the cross-condition comparisons can support; consider softening the abstract to reflect the acknowledged confound.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PragMatch's benchmark and evaluations are externally grounded, with no fitted parameter, equation, or load-bearing self-citation reducing the claims to their inputs.

full rationale

PragMatch's claim chain is self-contained with respect to benchmark construction and evaluation. The pragmatic (P) examples and labels come from MMSD2.0; the literal (L) captions are generated by Gemma-3-27B-IT; the non-pragmatic mismatch (M) captions are retrieved by CLIP and filtered by RelSim; and the four evaluated LVLMs (InternVL2.5-8B, Qwen2.5-VL-7B, LLaVA-v1.6-7B, Idefics2-8B) are neither used to construct the labels nor fitted to the benchmark. The reported accuracy, AUROC, and paired-accuracy metrics are computed directly from model verdict log-probabilities (Eq. 1), so no quantity is defined in terms of the outcome it is used to explain. The masking and injection diagnostics rest on the premise that perturbations preserve the underlying image-caption relation and gold label, but this is an empirical validity assumption about the intervention rather than a definitional identity: the paper does not define a 'surface cue' as 'whatever flips an LVLM,' and no result is forced by construction. The only author self-citation (Saha et al., 2025, MUS-TReason) appears as related-work context and is not load-bearing. Therefore no circular step is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the source labels (MMSD2.0), on the quality of the CLIP/RelSim/Gemma construction pipeline, and on the validity of the score-based AUROC analysis. No free parameters are fit to model outputs, but several design constants (RelSim threshold, neighbor count, token limit) are hand-chosen and affect benchmark difficulty.

free parameters (3)
  • RelSim relational similarity threshold range = 0.2 to 0.4
    Chosen by hand to retain hard negatives that are visually plausible but relationally mismatched; determines benchmark difficulty and is not derived from theory. Section 3.3.
  • Number of nearest visual neighbors = 5
    CLIP retrieval depth chosen as a design constant; affects the pool of candidate mismatch captions. Section 3.3.
  • Gemma-3 caption length limit = 40 tokens
    Generation constraint for literal captions; may introduce a register mismatch with social-media sarcasm captions. Section 3.3.
assumptions (4)
  • domain assumption MMSD2.0 gold labels y=1 correctly identify sarcastic intent, and the three constructed conditions are valid instances of pragmatic incongruity, literal congruity, and non-pragmatic mismatch.
    The entire benchmark validity rests on the source dataset labels and the construction filters. Section 3.3.
  • domain assumption CLIP and RelSim provide reliable visual and relational similarity for selecting hard negatives.
    The M captions are selected using CLIP nearest-neighbor retrieval and RelSim filtering (0.2-0.4); if these models misjudge relation, the M condition is not truly non-pragmatic mismatch. Section 3.3.
  • domain assumption The scoring function s (Eq. 1), derived from log-probabilities of Yes/No tokens, is a valid continuous sarcasm score for AUROC analysis.
    AUROC conclusions assume the score ranks sarcasm vs non-sarcasm monotonically. Section 3.1, Eq. 1.
  • domain assumption Gemma-3-27B-IT literal captions are accurate and style-neutral.
    The L condition is generated by Gemma-3 with a constrained prompt; the authors themselves note these captions differ in provenance and register from social-media sarcasm captions. Section 3.3 and Limitations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PragMatch: Separating Pragmatic Incongruity from Cross-Modal Mismatch in Large Vision-Language Models." pith.science (2026). https://pith.science/paper/XMZ5LSEK

@misc{pith2026260809772,
  author       = {Pith},
  title        = {Pith review of: PragMatch: Separating Pragmatic Incongruity from Cross-Modal Mismatch in Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XMZ5LSEK}},
  note         = {Machine review of arXiv:2608.09772}
}
read the original abstract

Large Vision-Language Models (LVLMs) have demonstrated strong performance on multimodal benchmarks, yet it remains unclear whether they genuinely reason about relationships between images and text or rely on superficial correlations, known as shortcut learning. This question is particularly important for multimodal sarcasm detection, where successful prediction depends on recognizing pragmatic incongruity rather than treating sarcasm as simple image-text mismatch. We introduce PragMatch, a controlled benchmark of 3,000 image-text pairs derived from MMSD2.0, including original sarcastic examples and constructed literal and hard-negative pairs. We identify influential shortcut cues through systematic masking and evaluate their impact through targeted injection experiments. Our results show that LVLM predictions are sensitive to lexical, OCR-derived and stylistic cues, with injected surface signals causing substantial changes in model predictions despite unchanged underlying image-text relationships. Our findings reveal limitations in current LVLMs while PragMatch provides a systematic testbed for evaluating multimodal pragmatic reasoning beyond surface-level image-text alignment.

Figures

Figures reproduced from arXiv: 2608.09772 by the authors.

Figure 1
Figure 1. PragMatch benchmark and shortcut perturbation pipeline. (a) The benchmark compares three types of [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Qualitative examples of shortcut-induced ver [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 51 canonical work pages

  1. [1]

    Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages=

    Schroedinger’s Threshold: When the AUC doesn’t predict Accuracy , author=. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages=

    work page 2024

  2. [2]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  3. [3]

    Publications Manual , year = "1983", publisher =

    1983

  4. [4]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    arXiv 1981

  5. [5]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  6. [6]

    Dan Gusfield , title =. 1997

    1997

  7. [7]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  8. [8]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

Show all 80 references
  1. [9]

    Information Fusion , pages=

    A comprehensive survey of Vision-Language Models: Pretrained models, fine-tuning, prompt engineering, adapters, and benchmark datasets , author=. Information Fusion , pages=. 2025 , publisher=

    2025

  2. [10]

    cortex , volume=

    Distinction between the literal and intended meanings of sentences: a functional magnetic resonance imaging study of metaphor and sarcasm , author=. cortex , volume=. 2012 , publisher=

    2012

  3. [11]

    arXiv preprint arXiv:2412.14613 , year=

    Multi-modal, Multi-task, Multi-criteria Automatic Evaluation with Vision Language Models , author=. arXiv preprint arXiv:2412.14613 , year=

  4. [12]

    Making Up Titles for Conference Papers äöü , author =. Ext. Abstracts CHI 2000 , year =

    2000

  5. [13]

    Be Consistent in Your Bibliography , author =. Ext. Abstracts CHI 1988 , year =

    1988

  6. [14]

    Proceedings of the 34th ACM International Conference on Information and Knowledge Management , pages=

    Can Large Vision-Language Models Understand Multimodal Sarcasm? , author=. Proceedings of the 34th ACM International Conference on Information and Knowledge Management , pages=

  7. [15]

    Sarcasm and the Space Structuring Model , year =

    Seana Coulson , booktitle =. Sarcasm and the Space Structuring Model , year =

  8. [16]

    Communications of the ACM , volume=

    Shortcut learning of large language models in natural language understanding , author=. Communications of the ACM , volume=. 2023 , publisher=

    2023

  9. [17]

    Proceedings of the 24th ACM international conference on Multimedia , pages=

    Detecting sarcasm in multimodal social platforms , author=. Proceedings of the 24th ACM international conference on Multimedia , pages=

  10. [18]

    Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

    Multi-modal sarcasm detection in twitter with hierarchical fusion model , author=. Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

  11. [19]

    0: Towards a reliable multi-modal sarcasm detection system , author=

    MMSD2. 0: Towards a reliable multi-modal sarcasm detection system , author=. Findings of the association for computational linguistics: ACL 2023 , pages=

    2023

  12. [20]

    Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

    Multi-modal sarcasm detection via cross-modal graph convolutional network , author=. Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

  13. [21]

    Proceedings of the 29th ACM international conference on multimedia , pages=

    Multi-modal sarcasm detection with interactive in-modal and cross-modal graphs , author=. Proceedings of the 29th ACM international conference on multimedia , pages=

  14. [22]

    2020 International Joint Conference on Neural Networks (IJCNN) , pages=

    I didn’t mean what I wrote! Exploring Multimodality for Sarcasm Detection , author=. 2020 International Joint Conference on Neural Networks (IJCNN) , pages=. 2020 , organization=

    2020

  15. [23]

    Proceedings of the 20th international conference on multimodal interaction: adjunct , pages=

    Sarcasm detection on facebook: A supervised learning approach , author=. Proceedings of the 20th international conference on multimodal interaction: adjunct , pages=

  16. [24]

    how long did you marinate in it? multimodal sarcasm explanation , author=

    Nice perfume. how long did you marinate in it? multimodal sarcasm explanation , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  17. [25]

    Findings of the Association for Computational Linguistics: EMNLP 2020 , pages=

    Modeling intra and inter-modality incongruity for multi-modal sarcasm detection , author=. Findings of the Association for Computational Linguistics: EMNLP 2020 , pages=

    2020

  18. [26]

    Proceedings of the 32nd ACM International Conference on Multimedia , pages=

    Moba: Mixture of bi-directional adapter for multi-modal sarcasm detection , author=. Proceedings of the 32nd ACM International Conference on Multimedia , pages=

  19. [27]

    International conference on machine learning , pages=

    Learning transferable visual models from natural language supervision , author=. International conference on machine learning , pages=. 2021 , organization=

    2021

  20. [28]

    Wei, Yiwei and Yuan, Shaozu and Zhou, Hengyang and Wang, Longbiao and Yan, Zhiling and Yang, Ruosong and Chen, Meng , booktitle=. G\^

  21. [29]

    IEEE Transactions on Affective Computing , volume=

    Fusion and discrimination: A multimodal graph contrastive learning framework for multimodal sarcasm detection , author=. IEEE Transactions on Affective Computing , volume=. 2024 , publisher=

    2024

  22. [30]

    arXiv preprint arXiv:2410.18882 , year=

    A survey of multimodal sarcasm detection , author=. arXiv preprint arXiv:2410.18882 , year=

    arXiv

  23. [31]

    arXiv preprint arXiv:2303.08774 , year=

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    arXiv

  24. [32]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Improved baselines with visual instruction tuning , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  25. [33]

    International conference on machine learning , pages=

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation , author=. International conference on machine learning , pages=. 2022 , organization=

    2022

  26. [34]

    2025 IEEE International Conference on Data Mining Workshops (ICDMW) , pages=

    Evaluating open-source vision-language models for multimodal sarcasm detection , author=. 2025 IEEE International Conference on Data Mining Workshops (ICDMW) , pages=. 2025 , organization=

    2025

  27. [35]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Do llms overcome shortcut learning? an evaluation of shortcut challenges in large language models , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  28. [36]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Is sarcasm detection a step-by-step reasoning process in large language models? , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  29. [37]

    arXiv preprint arXiv:2503.04550 , year=

    Benchmarking reasoning robustness in large language models , author=. arXiv preprint arXiv:2503.04550 , year=

    arXiv

  30. [38]

    Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    Towards interpreting and mitigating shortcut learning behavior of NLU models , author=. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    2021

  31. [39]

    ArXiv , year=

    Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning , author=. ArXiv , year=

  32. [40]

    arXiv preprint arXiv:2509.22437 , year=

    Chimera: Diagnosing shortcut learning in visual-language understanding , author=. arXiv preprint arXiv:2509.22437 , year=

  33. [41]

    Nature Machine Intelligence , volume=

    Shortcut learning in deep neural networks , author=. Nature Machine Intelligence , volume=. 2020 , publisher=

    2020

  34. [42]

    arXiv preprint arXiv:2310.16228 , year=

    On the foundations of shortcut learning , author=. arXiv preprint arXiv:2310.16228 , year=

    arXiv

  35. [43]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Overcoming shortcut problem in VLM for robust out-of-distribution detection , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  36. [44]

    arXiv preprint arXiv:2602.15183 , year=

    Seeing to Generalize: How Visual Data Corrects Binding Shortcuts , author=. arXiv preprint arXiv:2602.15183 , year=

  37. [45]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Conditional Information Bottleneck for Multimodal Fusion: Overcoming Shortcut Learning in Sarcasm Detection , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  38. [46]

    Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

    Towards multimodal sarcasm detection (an \_obviously\_ perfect paper) , author=. Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

  39. [47]

    Proceedings of the 31st International Conference on Computational Linguistics , pages=

    Multi-view incongruity learning for multimodal sarcasm detection , author=. Proceedings of the 31st International Conference on Computational Linguistics , pages=

  40. [48]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages =

    Vision-Language Models Do Not Understand Negation , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages =

  41. [49]

    arXiv preprint arXiv:2507.01790 , year =

    How Do Vision-Language Models Process Conflicting Information Across Modalities? , author =. arXiv preprint arXiv:2507.01790 , year =

    arXiv

  42. [50]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Relational Visual Similarity , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  43. [51]

    arXiv preprint arXiv:2409.12191 , year=

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution , author=. arXiv preprint arXiv:2409.12191 , year=

    arXiv

  44. [52]

    arXiv preprint arXiv:2304.08485 , year =

    Visual Instruction Tuning , author =. arXiv preprint arXiv:2304.08485 , year =

    arXiv

  45. [53]

    arXiv preprint arXiv:2507.18043 , year=

    GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs , author=. arXiv preprint arXiv:2507.18043 , year=

    arXiv

  46. [54]

    Findings of the Association for Computational Linguistics: EMNLP 2022 , pages=

    Language prior is not the only shortcut: A benchmark for shortcut learning in vqa , author=. Findings of the Association for Computational Linguistics: EMNLP 2022 , pages=

    2022

  47. [55]

    Advances in Neural Information Processing Systems , volume=

    Truth over tricks: Measuring and mitigating shortcut learning in misinformation detection , author=. Advances in Neural Information Processing Systems , volume=

  48. [56]

    arXiv preprint arXiv:2604.02486 , year=

    Vlms need words: Vision language models ignore visual detail in favor of semantic anchors , author=. arXiv preprint arXiv:2604.02486 , year=

    arXiv

  49. [57]

    arXiv preprint arXiv:2501.02189 , volume=

    Benchmark evaluations, applications, and challenges of large vision language models: A survey , author=. arXiv preprint arXiv:2501.02189 , volume=

    arXiv

  50. [58]

    IEEE MultiMedia , volume=

    Modeling incongruity between modalities for multimodal sarcasm detection , author=. IEEE MultiMedia , volume=. 2021 , publisher=

    2021

  51. [59]

    ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    Semantic-Aware Prompt Learning for Multimodal Sarcasm Detection , author=. ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2025 , organization=

    2025

  52. [60]

    Findings of the Association for Computational Linguistics: EMNLP 2025 , pages=

    Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routing , author=. Findings of the Association for Computational Linguistics: EMNLP 2025 , pages=

    2025

  53. [61]

    IEEE Transactions on Affective Computing , year=

    Spoken in jest, detected in earnest: A systematic review of sarcasm recognition-multimodal fusion, challenges, and future prospects , author=. IEEE Transactions on Affective Computing , year=

  54. [62]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Debiasing Multimodal Sarcasm Detection with Contrastive Learning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=. 2024 , doi=

    2024

  55. [63]

    Proceedings of the 31st International Conference on Computational Linguistics , pages=

    Multi-View Incongruity Learning for Multimodal Sarcasm Detection , author=. Proceedings of the 31st International Conference on Computational Linguistics , pages=

  56. [64]

    Wen, Changsong and Jia, Guoli and Yang, Jufeng , booktitle=

  57. [65]

    2025 , eprint=

    MUStReason: A Benchmark for Diagnosing Pragmatic Reasoning in Video-LMs for Multimodal Sarcasm Detection , author=. 2025 , eprint=

    2025

  58. [66]

    Anantha Ramakrishnan, Aashish and Anantha Ramakrishnan, Aadarsh and Lee, Dongwon , booktitle=

  59. [67]

    IEEE Transactions on Affective Computing , year=

    InterARM: interpretable affective reasoning model for multimodal sarcasm detection , author=. IEEE Transactions on Affective Computing , year=

  60. [68]

    2023 , eprint=

    When and why vision-language models behave like bags-of-words, and what to do about it? , author=. 2023 , eprint=

    2023

  61. [69]

    2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages=

    @ CREPE: Can vision-language foundation models reason compositionally? , author=. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages=. 2023 , organization=

    2023

  62. [70]

    Advances in neural information processing systems , volume=

    Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality , author=. Advances in neural information processing systems , volume=

  63. [71]

    Teaching Structured Vision & Language Concepts to Vision & Language Models , year=

    Doveh, Sivan and Arbelle, Assaf and Harary, Sivan and Schwartz, Eli and Herzig, Roei and Giryes, Raja and Feris, Rogerio and Panda, Rameswar and Ullman, Shimon and Karlinsky, Leonid , booktitle=. Teaching Structured Vision & Language Concepts to Vision & Language Models , year=

  64. [72]

    European Conference on Computer Vision , pages=

    The hard positive truth about vision-language compositionality , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  65. [73]

    Advances in Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks , year =

    NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples , author =. Advances in Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks , year =. 2410.14669 , archivePrefix =

    arXiv

  66. [74]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    PunchBench: Benchmarking MLLMs in multimodal punchline comprehension , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  67. [75]

    2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages=

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models , author=. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages=. 2024 , organization=

    2024

  68. [76]

    2025 , eprint=

    Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling , author=. 2025 , eprint=

    2025

  69. [77]

    2025 , eprint=

    Qwen2.5-VL Technical Report , author=. 2025 , eprint=

    2025

  70. [78]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , year=

    Improved Baselines with Visual Instruction Tuning , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , year=

  71. [79]

    Advances in Neural Information Processing Systems , volume=

    What matters when building vision-language models? , author=. Advances in Neural Information Processing Systems , volume=

  72. [80]

    2025 , eprint=

    Gemma 3 Technical Report , author=. 2025 , eprint=

    2025

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.