Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

A new benchmark, RIO-Bench, evaluates whether vision-language models can decide when to read scene text and when to ignore inserted distractor text, and shows that balanced fine-tuning on both object-attack and text-attack examples achieves

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 17:25 UTC pith:4GNA52HT

load-bearing objection A genuinely new same-scene benchmark for selective text use in LVLMs, but the object-side ground truth is machine-labeled without human checks and some appendix examples look misaligned, so treat Table 1 as provisional until the authors release and validate. the 3 major comments →

arxiv 2512.11899 v2 pith:4GNA52HT submitted 2025-12-10 cs.CV

Read or Ignore? A Unified Benchmark for Typographic-Attack Robustness and Text Recognition in Vision-Language Models

Futa Waseda , Shojiro Yamabe , Daiki Shiono , Kento Sasaki , Tsubasa Takahashi This is my paper
classification cs.CV
keywords typographic attackvision-language modelsvisual question answeringselective text userobustness benchmarkscene text readingcounterfactual evaluationsupervised fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that typographic-attack robustness has been evaluated only from an object-recognition perspective, which has pushed defenses toward simply ignoring all text. It introduces Read-or-Ignore VQA, a task that crosses question type (object vs. text) with text condition (clean vs. attacked) on the same scene, so a model must read text when the question asks about it and ignore it when the question is about objects. Using this benchmark, the paper shows that current vision-language models and existing defenses fail to balance robustness and text reading: strong models overtrust misleading text, while text-ignoring defenses lose reading ability. The paper then shows that standard supervised fine-tuning on a balanced mix of object-attack and text-attack examples restores robustness without sacrificing text reading, and that this selective text-use behavior is primarily learned in the language component, not the vision encoder.

Core claim

The central discovery is that typographic robustness and text-reading ability are not inherently opposed, but the way defenses are trained makes them look opposed. Prior robustness measures only rewarded ignoring misleading text, so models trained under that objective legitimately learned to suppress text sensitivity. On a same-scene counterfactual benchmark that also asks text questions, those defenses collapse on the text branch. A model fine-tuned on a balanced mixture of object-attack and text-attack examples learns to use text selectively: it attends to the correct scene text when asked to read, and disregards semantically related distractor words when asked about objects. The paper als

What carries the argument

The load-bearing mechanism is the same-scene counterfactual design: for each image the benchmark produces four variants (object/clean, object/attack, text/clean, text/attack) that differ only in question intent and in whether a typographic attack word is overlaid, so differences in behavior are attributable to selective text use rather than scene differences. On top of this, RIO-RT uses a balanced mixture of object-attack and text-attack examples for standard supervised fine-tuning, teaching the model when to read and when to ignore. A dedicated robustness metric for open-ended object questions rewards predictions close to the true object class and penalizes closeness to the attack word.

Load-bearing premise

The conclusions stand only if the automatically chosen 'correct' object labels and the injected misleading words are genuinely correct and genuinely misleading; otherwise the observed robustness–reading trade-off could be an artifact of benchmark construction rather than a property of models.

What would settle it

A human audit of a random sample of RIO-Bench items: if in a noticeable fraction the automatically chosen ground-truth object is absent from the image, or the injected attack word reads as a plausible part of the scene, the benchmark is not isolating selective text use. A sharper test: blur all scene text in the text-clean split; a model that ignores text yet still scores well would show the text branch can be passed without reading.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If RIO-RT's result holds, typographic robustness and text reading can be trained jointly, so defenses should be evaluated on both axes rather than on object accuracy alone.
  • Robustness training should mix object-attack and text-attack examples; one-sided ignore-text training will silently trade away reading ability.
  • The language model's reasoning, not the vision encoder, is the right place to implement selective text use, contradicting earlier vision-side suppression defenses.
  • The benchmark's construction can be applied to any dataset that has both object and text annotations for the same images, so the evaluation scheme can spread to other domains.
  • The 16K-sample training requirement suggests selective text use can be learned cheaply via lightweight fine-tuning, not a new architecture.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This suggests a direct extension to embodied or driving scenarios, where a model must read signs while ignoring planted adversarial text; the benchmark's same-scene design makes that natural to test.
  • This also suggests that dataset balancing, not architectural defenses, may be the primary lever for typographic robustness in vision-language models, shifting research toward curating dual-purpose counterfactual data.
  • A testable extension would measure whether RIO-RT also improves robustness to other perceptual manipulations, such as adversarial patches or stickers, since the mechanism is contextual decision-making rather than text suppression.
  • The finding that language reasoning, not vision features, drives selective text use implies that future defenses could be implemented at the instruction or prompt level with targeted data, perhaps without modifying the vision encoder.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Read-or-Ignore VQA (RIO-VQA), a task that requires a vision-language model (LVLM) to decide from context whether to read or ignore scene text, together with RIO-Bench, a counterfactual benchmark built from TextVQA and Open Images. Four scenarios are formed by crossing object- versus text-centric questions with clean versus typographically attacked images. The authors evaluate several LVLMs and existing defenses, report a trade-off between typographic robustness and text-reading ability, and propose RIO-RT, a simple balanced supervised fine-tuning method that mixes object-attack and text-attack samples to learn selective text use. Cross-dataset generalization experiments are reported in Appendix F.2.

Significance. If the benchmark is valid, it addresses a genuine gap: most typographic-attack evaluations measure object robustness only and ignore the cost to text reading. The same-scene counterfactual design and the controlled attack levels (semantic and spatial) are methodologically attractive. The paper also provides a concrete data-driven defense and tests cross-dataset generalization, which partly mitigates the in-domain circularity of training and evaluating on the same benchmark. However, two validity issues currently threaten the headline conclusions: the object-side ground-truth and negative labels can be semantically ambiguous, and some inherited TextVQA questions are mismatched with their images. These are fixable with additional auditing and metric adjustments, so the manuscript is promising but not yet acceptable in its current form.

major comments (3)
  1. [§4.2, Appendix A.2.1, §4.4] The object-side MC labels do not guarantee a unique correct answer. As described in Appendix A.2.1(ii), easy negative options are sampled as 'higher-level ancestor' labels. For a visible GT object (e.g., 'cat'), any ancestor (e.g., 'animal') is also present in the image, so the question 'Which object is present?' has multiple semantically correct options. The GT is selected by CLIP image–text similarity without human verification (A.2.1(i)), so the benchmark's one-correct-answer assumption is not met. The same negative labels are reused as object-attack words (§4.3); therefore, for easy attacks the inserted word can itself be a true object label. R-CLIP-M (§4.4) then penalizes similarity to the attack word even when the model outputs a correct hypernym. These issues contaminate the object-side columns of Table 1 and undermine the quantitative basis for the claimed robustness/reading trad
  2. [Appendix A.4, Figs. 9–10] The text-side benchmark inherits noisy TextVQA questions without an audit. In Fig. 9, the Text-Clean question 'what kind of beer is this?' is paired with an image whose Open Images labels are 'Sports uniform, Man, Sports equipment, Boy', and the gold answer is '22'. In Fig. 10, the same question is paired with a watch image (gold answer '5:41'). These are not coherent question–image pairs. The paper does not report how frequent such mismatches are, and the Text-Clean/Text-Attack accuracies in Table 1 are the basis for the 'reading' half of the trade-off. Please filter or audit the TextVQA questions and report the mismatch rate.
  3. [§5.1.2] The comparison between IT-RT and RIO-RT is confounded by training-set composition. IT-RT is trained on 16K Obj-Attack samples (all difficulty levels), whereas RIO-RT is trained on 8K Obj-Attack (Hard) + 8K Text-Attack (Hard). Thus the comparison varies both the object/text balance and the attack-difficulty distribution. To support the claim that balanced training — rather than the different object-attack mix — drives RIO-RT's behavior, hold the object-attack portion fixed (e.g., 8K Hard) and vary only the addition of Text-Attack samples. Without this control, RIO-RT's advantage could be an artifact of the sampling scheme.
minor comments (4)
  1. [§4.3, Appendix A.3.3] The 'near' distance bucket for text attacks is discarded as ill-posed even for humans, so 'hard' is actually mid-distance. Please justify or provide examples; near but non-overlapping distractors are plausible real-world attacks and would provide a stronger stress test.
  2. [Tables 1 and 4] The column headers 'A VG' are ambiguous — it is not clear which columns are averaged over difficulty levels and which are per-level. Please define the averaging procedure explicitly.
  3. [Figure 3] The flowchart is dense and the labels 'Obj-based Extension' / 'Text-based Extension' are easy to confuse with the four subtask names. Consider annotating the figure with the RIO-VQA taxonomy terms (Obj-Clean, Obj-Attack, Text-Clean, Text-Attack) directly.
  4. [References] Reference [9] contains stray page numbers ('1(2):3'), and the in-text citations for [9] include a stray page '3, 12'. Please clean up the bibliography.

Circularity Check

0 steps flagged

No significant circularity; benchmark construction and defense evaluation are self-contained.

full rationale

The paper's contribution is a benchmark and a training-based defense, not a derivation that reduces to its inputs. RIO-Bench is constructed from independent public data (TextVQA and Open Images) with CLIP used only as an automated label-selection tool; the R-CLIP-M metric is a defined evaluation measure, not a fitted predictor of the paper's conclusions. RIO-RT is trained on a training split and evaluated on a validation split, with additional cross-dataset generalization tests on Typo-D, AI2D, ChartQA, DocVQA, and InfoVQA, so the central robustness/reading trade-off is not equivalent to the training data by construction. There are no load-bearing self-citations or imported uniqueness claims. Potential concerns about CLIP-selected ground truth or hierarchical negatives producing ambiguous labels are correctness/validity risks, not circularity under the stated rubric.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 4 invented entities

The central result is an empirical benchmark and training recipe; the claimed trade-off and the success of RIO-RT rest on the paper's own design choices (attack construction, CLIP-based labeling, metric definition, and the specific 16k training mix) rather than on externally fixed anchors.

free parameters (7)
  • Attack difficulty levels = object: easy/medium/hard (semantic similarity); text: easy/hard (spatial distance)
    Chosen by authors; the headline robustness gains are reported on these levels, so the difficulty calibration determines the measured drop (Table 1).
  • CLIP-based ground-truth selection = highest CLIP image-text similarity among annotated Open Images classes
    The 'correct' object answer is a CLIP ranking, not a human judgment; errors here propagate to Obj-Clean and Obj-Attack numbers (Appendix A.2.1).
  • R-CLIP-M metric = k=5; R-CLIP-M = CLIP-M@5(GT) - CLIP-M@5(attack)
    Hand-defined metric without external validation; the OE robustness numbers in Table 1 depend directly on it.
  • RIO-RT training mix = 4k Obj-Attack (Hard) MC + 4k Obj-Attack (Hard) OE + 8k Text-Attack (Hard), 16k total
    No ablation over mix ratios; the reported balance may depend on this specific 1:1:2 selection (Sec. 5.1.2).
  • LoRA hyperparameters = r=16, alpha=16, lr=1e-4, 1 epoch
    Fixed across all defense runs; no sensitivity analysis is provided.
  • Text-attack placement = N=7 grid; easy=far, hard=mid; near bucket discarded as ill-posed
    Placing the distractor near the key text was excluded as ambiguous for humans; this exclusion shapes the Text-Attack difficulty distribution (Appendix A.3.3).
  • Key-text localization thresholds = alpha=0.5, similarity threshold 0.6
    Controls which OCR text is treated as 'key'; mis-identification changes which Text-Attack examples are hard (Appendix A.3.3).
axioms (6)
  • domain assumption Open Images class annotations are reliable ground-truth labels for the images.
    Object questions derive from these labels (§4.1, Appendix A.2.1).
  • domain assumption TextVQA questions and OCR boxes are valid text-understanding tasks for the collapsed image set.
    Text-Clean branch uses these directly; appendix examples raise doubt about pairing (§4.1, Appendix A.4).
  • domain assumption CLIP similarity is a valid proxy for object presence.
    Used to pick GT classes and to score OE predictions (§4.2, Appendix B.2).
  • domain assumption Synthetic typographic text overlays are effective and fair attacks.
    The benchmark's attack branches rest on this (§4.3); 'effective and fair' is asserted, not externally validated.
  • domain assumption Same-scene counterfactual isolation identifies selective text use.
    Core design assumption: varying only question type and text condition while fixing the scene is sufficient (§4.1, Fig. 2).
  • domain assumption Llama-3-generated contradictory words are plausible distractors for open-ended text VQA.
    Text-Attack words are LLM-generated (§4.3, Appendix A.3.3).
invented entities (4)
  • RIO-VQA task no independent evidence
    purpose: Task defining selective text use across object/text intent and clean/attack conditions
    New task definition; no external validation of its relation to real-world requirements.
  • RIO-Bench no independent evidence
    purpose: Benchmark implementing RIO-VQA with same-scene counterfactuals
    Dataset/code not yet released; construction choices are all in-paper.
  • R-CLIP-M metric no independent evidence
    purpose: Robustness metric for open-ended object VQA
    Ad hoc metric subtracting CLIP similarity to the attack word; no human-correlation study.
  • RIO-RT no independent evidence
    purpose: Balanced robust training defense
    Method evaluated in-paper; cross-dataset results are on a few external benchmarks.

pith-pipeline@v1.3.0-alltime-deepseek · 24712 in / 17624 out tokens · 161369 ms · 2026-08-03T17:25:44.382123+00:00 · methodology

0 comments
read the original abstract

Large vision-language models (LVLMs) are vulnerable to typographic attacks, where misleading text inserted into an image can override visual understanding. However, existing evaluation protocols and defenses are largely focused on object recognition and do not consider text-reading capability. This is a critical oversight: real-world scenarios often require both recognizing objects and reading scene text (e.g., recognizing pedestrians while reading traffic signs), where simply ignoring all text for robustness is unacceptable in practice. To address this gap, we introduce a novel task, Read-or-Ignore VQA (RIO-VQA), which jointly evaluates both requirements: models must decide, from context, when to read scene text and when to ignore inserted distractor text. To evaluate this capability, we present RIO-Bench, a same-scene counterfactual benchmark that holds the scene fixed while varying only question intent (object vs. text) and text condition (clean vs. attack), enabling direct comparisons of model behaviors with reduced confounding factors. Using RIO-Bench, we highlight a trade-off: representative defenses developed in object-centric settings can achieve robustness by suppressing text sensitivity, at the cost of text-reading performance (i.e., "ignoring" text). Motivated by this trade-off, we provide a data-driven defense baseline that improves both requirements on RIO-Bench, complementing prior text-ignoring baselines. Overall, this work highlights a fundamental misalignment between the current object-centric robustness scope and real-world multimodal requirements, providing a principled path toward reliable LVLMs.

Figures

Figures reproduced from arXiv: 2512.11899 by Daiki Shiono, Futa Waseda, Kento Sasaki, Shojiro Yamabe, Tsubasa Takahashi.

Figure 1
Figure 1. Figure 1: Desired behavior in our novel task, Read-or-Ignore VQA (RIO-VQA). Robust LVLMs must decide when to read or ignore text depending on the context: No defense (red): easily fooled by typographic attacks; Existing defenses (yellow): focus on object recognition and gain robustness, but lose reading ability; Ideal model (green): achieves selective text use within the same scene, reading or ignoring text appropri… view at source ↗
Figure 2
Figure 2. Figure 2: Taxonomy of the Read-or-Ignore VQA (RIO-VQA) task. Unlike prior typographic-attack benchmarks that focus only on object recognition robustness while neglecting reading capabil￾ity, RIO-VQA jointly evaluates both. It defines four scenarios by crossing Object vs. Text question types with Clean vs. Attacked inputs within the same scene, enabling counterfactual evaluation of when to read or ignore text. 3. Tas… view at source ↗
Figure 3
Figure 3. Figure 3: Data construction pipeline of Read-or-Ignore Benchmark (RIO-Bench). Starting from TextVQA [35], we add object￾centric questions so each image supports both Object- and Text-VQA within the same scene. Object-Attack samples are built by inserting semantically misleading words selected via a class hierarchy (Easy/Medium/Hard), avoiding text overlap using bounding boxes. Text-Attack samples are generated by an… view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of defense models on RIO-Bench. Obj-/Text-Attack results are averaged over multiple attack levels. CoT [8] improves Obj-Attack robustness but harms text-VQA due to its object-centric prompt design. IT-RT, trained on the Obj-Attack set, achieves strong robustness; however, it tends to simply ignore text, resulting in poor text-VQA accuracy. In contrast, RIO-RT, trained on both Obj￾and Text-Attack… view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of attention maps. We compare the pretrained LLaVA-1.5-7B, the model fine-tuned on Obj-Attack only (IT-RT), and the model fine-tuned on both Obj- and Text-Attack (RIO-RT). For Obj-Attack (left), the pretrained model is misled by “Remote Control,” while both IT-RT and RIO-RT correctly focus on the object (“Computer monitor”) with less distraction from the attack word. For Text-Attack (right), … view at source ↗
Figure 6
Figure 6. Figure 6: Examples of RIO-Bench dataset. Example ID: 34603. (best viewed in zoom). [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Examples of RIO-Bench dataset. Example ID: 34604. (best viewed in zoom). [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Examples of RIO-Bench dataset. Example ID: 34605. (best viewed in zoom). [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Examples of RIO-Bench dataset. Example ID: 34607. (best viewed in zoom). [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Examples of RIO-Bench dataset. Example ID: 34608. (best viewed in zoom). [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Examples of RIO-Bench dataset. Example ID: 34610. (best viewed in zoom). [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Qualitative comparison of LLaVA-1.5-7B on RIO-Bench. Example ID: 34613. (best viewed in zoom). Obj-Clean (MC) Obj-Attack (MC) --- hard Text-Clean (OE) Text-Attack (OE) --- hard Answer Which object is present in the image? (A) Vegetable (B) Book (C) Bench (D) Ladder. Answer with only the option letter (A, B, C, or D). Which object is present in the image? (A) Vegetable (B) Book (C) Bench (D) Ladder. Answer… view at source ↗
Figure 13
Figure 13. Figure 13: Qualitative comparison of LLaVA-1.5-7B on RIO-Bench. Example ID: 34749. (best viewed in zoom). 14 [PITH_FULL_IMAGE:figures/full_fig_p024_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Qualitative comparison of LLaVA-1.5-7B on RIO-Bench. Example ID: 35860. (best viewed in zoom). Obj-Clean (MC) Obj-Attack (MC) --- hard Text-Clean (OE) Text-Attack (OE) --- hard Answer Pretrained RIO-RT IT-RT Which object is present in the image? (A) Drill (B) Ratchet (C) Coffee (D) Wine. Answer with only the option letter (A, B, C, or D). Which object is present in the image? (A) Coffee (B) Drill (C) Wine… view at source ↗
Figure 15
Figure 15. Figure 15: Qualitative comparison of Qwen2.5-7B on RIO-Bench. Example ID: 34613. (best viewed in zoom). 15 [PITH_FULL_IMAGE:figures/full_fig_p025_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Qualitative comparison of Qwen2.5-7B on RIO-Bench. Example ID: 34825. (best viewed in zoom). Obj-Clean (MC) Obj-Attack (MC) --- hard Text-Clean (OE) Text-Attack (OE) --- hard Answer Which object is present in the image? (A) Humidifier (B) Building (C) Snake (D) Door handle. Answer with only the option letter (A, B, C, or D). Which object is present in the image? (A) Humidifier (B) Building (C) Door handle… view at source ↗
Figure 17
Figure 17. Figure 17: Qualitative comparison of Qwen2.5-7B on RIO-Bench. Example ID: 34962. (best viewed in zoom). 16 [PITH_FULL_IMAGE:figures/full_fig_p026_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Effect of training data size on robustness (LLaVA-1.5-13B). 100% represents the default settings throughout the paper, using [PITH_FULL_IMAGE:figures/full_fig_p028_18.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Towards Robustness against Typographic Attack with Training-free Concept Localization

    cs.CV 2026-07 unverdicted novelty 6.0

    Training-free mechanistic interpretability locates lexical-encoding attention heads in ViT and shows that targeted interventions on them improve robustness to typographic attacks in CLIP and downstream LVLMs.

  2. Token-Efficient Multimodal Reasoning via Image Prompt Packaging

    cs.CV 2026-04 unverdicted novelty 6.0

    IPPg embeds text into images to reduce multimodal model inference costs by 35.8-91% with competitive accuracy on many VQA and code benchmarks.

Reference graph

Works this paper leans on

42 extracted references · 8 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736,

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736,

  2. [2]

    Defense-prefix for pre- venting typographic attacks on clip

    Hiroki Azuma and Yusuke Matsui. Defense-prefix for pre- venting typographic attacks on clip. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 3644–3653, 2023. 1, 2, 5

    2023

  3. [3]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023. 2, 5

    Pith/arXiv arXiv 2023

  4. [4]

    Vizwiz: nearly real-time answers to visual questions

    Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Lit- tle, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. InProceedings of the 23nd annual ACM symposium on User interface soft- ware and technology, pages 333–342, 2010. 2, 3

    2010

  5. [5]

    Scene text visual question answering

    Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marc ¸al Rusinol, Ernest Valveny, CV Jawahar, and Dimos- thenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291–4301, 2019. 2

    2019

  6. [6]

    Adversarial patch.arXiv preprint arXiv:1712.09665, 2017

    Tom B Brown, Dandelion Man ´e, Aurko Roy, Mart´ın Abadi, and Justin Gilmer. Adversarial patch.arXiv preprint arXiv:1712.09665, 2017. 1

    Pith/arXiv arXiv 2017

  7. [7]

    Scenetap: Scene- coherent typographic adversarial planner against vision- language models in real-world environments

    Yue Cao, Yun Xing, Jie Zhang, Di Lin, Tianwei Zhang, Ivor Tsang, Yang Liu, and Qing Guo. Scenetap: Scene- coherent typographic adversarial planner against vision- language models in real-world environments. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 25050–25059, 2025. 1, 2

    2025

  8. [8]

    Un- veiling typographic deceptions: Insights of the typographic vulnerability in large vision-language models

    Hao Cheng, Erjia Xiao, Jindong Gu, Le Yang, Jinhao Duan, Jize Zhang, Jiahang Cao, Kaidi Xu, and Renjing Xu. Un- veiling typographic deceptions: Insights of the typographic vulnerability in large vision-language models. InEuropean Conference on Computer Vision, pages 179–196. Springer,

  9. [9]

    The llama 3 herd of models.arXiv e-prints, pages arXiv–2407,

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv e-prints, pages arXiv–2407,

  10. [10]

    Sari sandbox: A virtual retail store environment for embodied ai agents

    Janika Deborah Gajo, Gerarld Paul Merales, Jerome Escar- cha, Brenden Ashley Molina, Gian Nartea, Emmanuel G Maminta, Juan Carlos Roldan, and Rowel O Atienza. Sari sandbox: A virtual retail store environment for embodied ai agents. InProceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 2369–2378, 2025. 2

    2025

  11. [11]

    Open-ended vqa benchmarking of vision-language models by exploiting classification datasets and their semantic hierarchy.arXiv preprint arXiv:2402.07270, 2024

    Simon Ging, Mar ´ıa A Bravo, and Thomas Brox. Open-ended vqa benchmarking of vision-language models by exploiting classification datasets and their semantic hierarchy.arXiv preprint arXiv:2402.07270, 2024. 5, 11

    Pith/arXiv arXiv 2024

  12. [12]

    Multimodal neurons in artificial neural networks.Dis- till, 6(3):e30, 2021

    Gabriel Goh, Nick Cammarata, Chelsea V oss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks.Dis- till, 6(3):e30, 2021. 1, 2

    2021

  13. [13]

    Explaining and harnessing adversarial examples

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. InICLR,

  14. [14]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017. 2, 5, 11

    2017

  15. [15]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 6700–6709, 2019. 2

    2019

  16. [16]

    Towards mech- anistic defenses against typographic attacks in clip.arXiv preprint arXiv:2508.20570, 2025

    Lorenz Hufe, Constantin Venhoff, Maximilian Dreyer, Se- bastian Lapuschkin, and Wojciech Samek. Towards mech- anistic defenses against typographic attacks in clip.arXiv preprint arXiv:2508.20570, 2025. 2

    arXiv 2025

  17. [17]

    5: a vision-language-action model with open-world generaliza- tion, 2025.URL https://arxiv

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al.π0. 5: a vision-language-action model with open-world generaliza- tion, 2025.URL https://arxiv. org/abs/2504.16054, 1(2):3,

    Pith/arXiv arXiv 2025

  18. [18]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. InEuropean conference on computer vision, pages 235–251. Springer, 2016. 17

    2016

  19. [19]

    Ocr-free document understanding transformer

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sang- doo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. InEuropean Confer- ence on Computer Vision, pages 498–517. Springer, 2022. 1, 2

    2022

  20. [20]

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.Interna- tional journal of computer vision, 128(7):1956–1981, 2020. 3, 4, 1

    1956

  21. [21]

    Align before fuse: Vision and language representation learn- ing with momentum distillation.Advances in neural infor- mation processing systems, 34:9694–9705, 2021

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learn- ing with momentum distillation.Advances in neural infor- mation processing systems, 34:9694–9705, 2021. 1

    2021

  22. [22]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InIn- ternational conference on machine learning, pages 19730– 19742. PMLR, 2023

    2023

  23. [23]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023. 1, 2, 5 9

    2023

  24. [24]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. InThe 36th Conference on Neural Information Processing Systems (NeurIPS), 2022. 2

    2022

  25. [25]

    Towards deep learning models resistant to adversarial attacks

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. InICLR, 2018. 1

    2018

  26. [26]

    Smolvlm: Redefining small and efficient multimodal models.arXiv preprint arXiv:2504.05299, 2025

    Andr ´es Marafioti, Orr Zohar, Miquel Farr ´e, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, et al. Smolvlm: Redefining small and efficient multimodal models.arXiv preprint arXiv:2504.05299, 2025. 5

    Pith/arXiv arXiv 2025

  27. [27]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. InProceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. 2, 3

    2019

  28. [28]

    Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022. 2, 17

    Pith/arXiv arXiv 2022

  29. [29]

    Dis- entangling visual and written concepts in clip

    Joanna Materzy ´nska, Antonio Torralba, and David Bau. Dis- entangling visual and written concepts in clip. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16410–16419, 2022. 1, 2, 5, 7

    2022

  30. [30]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InProceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 2, 3, 17

    2021

  31. [31]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. InProceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pages 1697–1706, 2022. 17

    2022

  32. [32]

    Vision-llms can fool themselves with self-generated typographic attacks.arXiv preprint arXiv:2402.00626, 2024

    Maan Qraitem, Nazia Tasnim, Piotr Teterwak, Kate Saenko, and Bryan A Plummer. Vision-llms can fool themselves with self-generated typographic attacks.arXiv preprint arXiv:2402.00626, 2024. 1, 4

    Pith/arXiv arXiv 2024

  33. [33]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021. 1, 2

    2021

  34. [34]

    Roadtext- 1k: Text detection & recognition dataset for driving videos

    Sangeeth Reddy, Minesh Mathew, Lluis Gomez, Marc ¸al Rusinol, Dimosthenis Karatzas, and CV Jawahar. Roadtext- 1k: Text detection & recognition dataset for driving videos. In2020 IEEE International Conference on Robotics and Au- tomation (ICRA), pages 11074–11080. IEEE, 2020. 2

    2020

  35. [35]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 2, 3, 4, 1

    2019

  36. [36]

    Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text

    Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wo- jciech Galuba, and Tal Hassner. Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8802–8812, 2021. 2, 3

    2021

  37. [37]

    Mtvqa: Benchmarking multilingual text-centric visual question answering

    Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, An-Lan Wang, Chunhui Lin, Hao Feng, Zhen Zhao, Yanjie Wang, et al. Mtvqa: Benchmarking multilingual text-centric visual question answering. InFindings of the Association for Computational Linguistics: ACL 2025, pages 7748–7763,

    2025

  38. [38]

    Reading between the lanes: Text videoqa on the road

    George Tom, Minesh Mathew, Sergi Garcia-Bordils, Dimos- thenis Karatzas, and CV Jawahar. Reading between the lanes: Text videoqa on the road. InInternational Confer- ence on Document Analysis and Recognition, pages 137–

  39. [39]

    Clip in mirror: Disentangling text from visual images through re- flection.Advances in Neural Information Processing Sys- tems, 37:24523–24546, 2024

    Tiancheng Wang, Yuguang Yang, Linlin Yang, Shaohui Lin, Juan Zhang, Guodong Guo, and Baochang Zhang. Clip in mirror: Disentangling text from visual images through re- flection.Advances in Neural Information Processing Sys- tems, 37:24523–24546, 2024. 1, 2, 5, 7

    2024

  40. [40]

    Alpamayo-r1: Bridging reasoning and action prediction for generalizable autonomous driving in the long tail.arXiv preprint arXiv:2511.00088, 2025

    Yan Wang, Wenjie Luo, Junjie Bai, Yulong Cao, Tong Che, Ke Chen, Yuxiao Chen, Jenna Diamond, Yifan Ding, Wen- hao Ding, et al. Alpamayo-r1: Bridging reasoning and action prediction for generalizable autonomous driving in the long tail.arXiv preprint arXiv:2511.00088, 2025. 1, 2

    Pith/arXiv arXiv 2025

  41. [41]

    What word is written on the sign?

    Justus Westerhoff, Erblina Purelku, Jakob Hackstein, Jonas Loos, Leo Pinetzki, and Lorenz Hufe. Scam: A real-world typographic robustness evaluation for multimodal founda- tion models.arXiv preprint arXiv:2504.04893, 2025. 1, 2, 5 10 Read or Ignore? A Unified Benchmark for Typographic-Attack Robustness and Text Recognition in Vision-Language Models Supple...

    arXiv 2025

  42. [2024]

    1, 2, 4, 5, 6, 7, 12, 17