Pith. sign in

REVIEW 4 major objections 6 minor 56 references

Multimodal LLMs systematically underperform on image-interleaved inputs that are semantically equivalent to text, and training on TokenSwap-augmented data can close much of that gap.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

TokenSwap measures and mitigates the MLLM modality gap: swapping textual concepts for matched images lowers accuracy by 4-47% across 42 models, and training with such swaps reduces the gap.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection Solid benchmark + training study on the MLLM modality gap; the main caveat is that the benchmark's importance filtering likely inflates the gap beyond a random MMLU subset. the 4 major comments →

arxiv 2607.28640 v1 pith:HXMSTDT7 submitted 2026-05-20 cs.CL cs.CV

TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

Andong Hua , Colton Bishop , Igor Mordatch , Arian Hosseini , Jindong Gu , Aleksandra Faust , Rebecca Roelofs , Yao Qin This is my paper
classification cs.CL cs.CV
keywords modality gapmultimodal large language modelssemantic equivalenceimage-interleaved inputsTokenSwapbenchmarkreasoning modelscross-modal consistency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that multimodal large language models are systematically less capable when the same content is presented with images interleaved in text rather than as pure text, and that this 'modality gap' can be measured and reduced. It introduces TokenSwap, a procedure that replaces concrete words in a question with semantically matched images, and uses it to build a benchmark from MMLU. Across 42 models, accuracy on image-interleaved questions is lower than on text-only questions for every model, by 4.2% to 47.4% (average 19.6%). Reasoning-oriented models show a much smaller gap than non-reasoning ones, while prompting and added compute do not reliably close it. The paper also shows that training on TokenSwap-augmented data shrinks the gap without hurting text or vision-language performance.

Core claim

The central claim is that the modality gap—measured as the difference between a model's accuracy on text-only inputs and on semantically equivalent image-interleaved inputs—is a pervasive and trainable property of multimodal LLMs, not a quirk of one benchmark. The authors demonstrate that on TokenSwap-Bench, built by swapping visualizable concepts in MMLU questions for generated images, all 42 evaluated models score lower on the interleaved version, with gaps from 4.2% to 47.4%. They further show that reasoning-trained models have markedly smaller gaps (10.1% average vs 25.5%), that prompting strategies and scale alone barely help, and that augmenting training data with TokenSwap-style inter

What carries the argument

The central mechanism is the TokenSwap operation S(c) → I_c, which replaces a contiguous span of text tokens representing a concept with an image encoded as visual tokens, producing an interleaved sequence. To ensure the images are semantically recoverable, the benchmark pipeline applies three filters: validity filtering (the image represents the concept in context), importance filtering (removing the concepts flips the correct answer), and caption-guided validation (captioning the image and substituting the caption back preserves the correct answer). The same operation is then repurposed as a training augmentation: text-only instruction data is converted into image-interleaved data by repla

Load-bearing premise

The load-bearing premise is that the images generated by Gemini and validated with Gemini-2.0-Flash are semantically equivalent to the textual concepts they replace, so the observed accuracy drop is due to modality rather than to ambiguous or misrecognized images.

What would settle it

Build a benchmark version where every substituted image is independently verified by multiple human raters for strict semantic equivalence (not just recoverability), re-run the 42-model evaluation, and check whether the average 19.6% gap persists; if the gap shrinks dramatically, the 'modality gap' is partly an artifact of imperfect image substitutes rather than a genuine cross-modal deficiency.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the gap is pervasive, any MLLM deployment that accepts images alongside text should expect worse accuracy on multimodal inputs than on text alone, even when the images add no new information.
  • Reasoning-oriented training, not prompting, is the ingredient that appears to narrow the gap; future models may need training-time interleaving rather than test-time scaffolds.
  • Scaling compute alone cannot be relied on to close the gap: a 10x FLOPs increase buys only ~2.8% absolute improvement.
  • Training on TokenSwap-augmented data is a practical, data-centric mitigation that preserves text and vision-language performance, so it can be dropped into existing pipelines.
  • Because the gap grows with the number of images per input, applications with many interleaved images are at highest risk.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the modality gap is partly a training-distribution artifact, then any text-heavy pretraining corpus may be encoding a text bias; TokenSwap provides a cheap probe to measure that bias without new model training.
  • One testable extension: apply TokenSwap to other text benchmarks (mathematics, code) and predict that the gap will be smaller in formal, notation-dense domains and larger in knowledge- and entity-rich domains—a prediction the paper's per-subject results already hint at.
  • The caption-round-trip validation suggests a potential test-time intervention: caption interleaved images back to text and let the model compare its confidence under both presentations, using disagreement as an uncertainty signal.
  • Domain mismatch results imply TokenSwap-style training benefits are local to the visual domain of the training images; extending it to OCR worked only when training used OCR-style images, so gains may not transfer across image types.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces TokenSwap, a procedure that replaces selected textual concepts in a question with natural images, and applies it to MMLU to construct TokenSwap-Bench (1,516 samples, 6,946 images). Across 42 MLLMs, the authors report that text-only accuracy is consistently higher than image-interleaved accuracy, with gaps ranging from 4.2% to 47.4% and a mean of 19.6%. They further report that reasoning models have smaller gaps, that prompting and compute scaling give limited improvements, and that adding TokenSwap-style interleaved data during pre/post-training reduces the gap for Qwen2-VL-7B without hurting text-only or standard vision-language performance.

Significance. If validated, this is a useful empirical resource: it is broader than prior rendered-text/symbolic consistency benchmarks, covers many model families, uses paired text/image variants, includes a retrieval/generation comparison, and proposes a concrete training intervention with an OCR generalization check. The paper's strengths include the breadth of the 42-model evaluation, the matched construction of text-only and image-interleaved inputs, the human concept-recoverability study, and the inclusion of standard VLM benchmarks in the training experiments. However, the central construct of semantic equivalence is currently proxied by filters built with a single model family, and the human study tests only isolated concept naming with easy distractors. Because the headline gap is measured on a concept-critical, single-proxy-filtered subset, the general 'pervasive modality gap' claim is not yet established at the level claimed. The contributions are promising but conditional on additional construct-validity evidence.

major comments (4)
  1. [Section 3.2.3, Eq. (3), and Section 3.3] Semantic equivalence is the load-bearing assumption. The caption-guided validation only certifies that Gemini-2.0-Flash can regenerate a caption that preserves Gemini's answer; it does not certify that the image itself carries all compositional/contextual information needed by an arbitrary model or by a human for the actual question. The human study in Section 3.3 tests isolated concept naming with distractors that Appendix A.4 itself calls 'generally not highly confusable.' That makes the reported 89-92% accuracy an upper bound on full-question equivalence, not the 'conservative measure' claimed. Because the headline 19.6% gap is interpreted as a modality effect, the paper needs a validation of full image-interleaved questions (e.g., human answer accuracy on a sample, or caption round-trip with one or more independent vision models), or the claims should be explicitly scoped to 'semanti
  2. [Section 3.2.3, Eq. (2)] The importance filter keeps only samples where Eval(X_text)=1 and Eval(X^-C_text)=0 for Gemini-2.0-Flash. This is a strong selection: the benchmark is not a random MMLU subset but an enriched set of concept-critical items, and the proxy model comes from the same family used for image generation. The average gap of 19.6% is therefore an estimate on this filtered subset, and the abstract/conclusion phrasing ('transforming MMLU', 'pervasive modality gap') overstates generality. Please report the gap on a random or validity-only filtered subset of MMLU, or at least quantify how many MMLU samples are dropped by each filter and present the unfiltered gap.
  3. [Section 4.3, Fig. 3(a), Table 2] The reasoning vs. non-reasoning comparison is supported by some within-family pairs, but the main statistical claim uses a t-test over 42 models as independent observations. Models from the same family share data, recipes, and release dates; this pseudo-replication likely overstates significance. Please fit a model with family as a random effect, present paired within-family differences (e.g., Qwen3-VL-4B vs. 4B-Thinking and 8B vs. 8B-Thinking separately), and/or report bootstrap by family. This would make the 'reasoning models exhibit smaller gaps' claim more robust.
  4. [Section 5.2, Figure 6] The central mitigation result is based on a single training run per setting for one base model (Qwen2-VL-7B). No variance or seeds are reported, and the gap reduction (0.268 to 0.167) is accompanied by small text-accuracy shifts; without paired error bars it is hard to know whether the reduction is beyond run-to-run noise. Also, Appendix C.1 notes that the training data removes Importance Filtering and Caption-Guided Validation, so the training distribution differs from the benchmark distribution; please state whether this mismatch affects the conclusion and, ideally, ablate with the full filtering pipeline.
minor comments (6)
  1. [General] No code/data URL is provided for TokenSwap-Bench or the training pipeline. Release details are needed for reproducibility.
  2. [Appendix A.4] The phrase 'conservative measure of semantic alignment' is contradicted by the easy-distractor design; rephrase or add a hard-distractor human study.
  3. [Figure 1 caption] The caption contains awkward punctuation ('4.0%, and 15.0%') and does not define 'retention lines' in the main text. Please clarify.
  4. [Section 4.5, Figure 10] Clarify whether the '7.9%' relative gap reduction is percentage points of relative gap or a relative change; the current sentence is ambiguous.
  5. [Appendix A.2] Equations (5) and (6) duplicate Eqs. (2) and (3) exactly; refer back to the main text instead of repeating them.
  6. [Section 4.2] The confounds listed (visual token count, multi-image processing, image artifacts) are acknowledged, but no quantitative decomposition is attempted; a short analysis separating at least one confound would strengthen the 'modality gap is intrinsic' claim.

Circularity Check

0 steps flagged

No significant circularity: the modality gap is an empirical measurement, not a derived quantity, and the benchmark construction does not force the reported gap.

full rationale

TokenSwap-Bench is built by transforming MMLU text and filtering candidates with Gemini-2.0-Flash. The filter equations (Eq. 2 / Eq. 5) require Eval(X_text)=1 and Eval(X_text^{-C})=0, and the caption-guided validation (Eq. 3 / Eq. 6) requires Eval(X_text)=1 and Eval(tilde X_text)=1, where tilde X_text is the caption round-trip version. Critically, neither filter uses or optimizes Eval(X_interleaved), the quantity that defines the modality gap in Eq. (1). Image-interleaved accuracy is therefore a held-out measurement for every evaluated model, and the headline result that all 42 models lie below the diagonal is an observed statistic, not an algebraic consequence of the construction. The Section 4.5 scaling slope is a descriptive regression, not a fitted parameter later presented as a prediction. The paper is also anchored to external evidence: it reports positive correlations with SEAM for text and image accuracies (r=0.811 and r=0.914) and standard vision-language benchmarks in Table 4, indicating independent content. The main caveat is that Gemini generates, filters, and validates the images while Gemini models appear in the evaluation set, which is a potential home-court confound in benchmark design; and the human study's distractors are admittedly easy, which bears on whether the gap is inflated by image ambiguity or selection, but these are validity/correctness concerns rather than circularity. No load-bearing step reduces, by the paper's own equations or by self-citation, to its inputs, so no circular step meets the quoted-reduction bar.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claims depend on a chain of domain assumptions: semantic equivalence of generated images, validity of a single proxy model's filtering, attribution of accuracy drops to modality rather than confounds, and generalization from one 7B model in training experiments. These are not hidden, but they are load-bearing.

free parameters (4)
  • Absolute modality-gap scaling slope = -0.028 per decade
    Linear regression in Figure 4(a) used to claim a 10× FLOPs increase reduces the absolute gap by only 2.8%; R²=0.253.
  • Relative modality-gap scaling slope = -0.079 per decade
    Linear regression in Figure 10 used to claim a 10× FLOPs increase reduces the relative gap by 7.9%; R²=0.811.
  • Cosine similarity threshold for retrieval = 0.3
    Hand-chosen threshold in Section 4.6 to filter retrieved images; affects which images enter the retrieval-based benchmark and the measured gap magnitude.
  • Number of retrieved candidates = 5
    Top-5 nearest CLIP neighbors in Section 4.6; a hand-chosen count that affects retrieval quality and benchmark difficulty.
axioms (4)
  • domain assumption Gemini-2.0-Flash is a reliable proxy for semantic equivalence across all evaluated models
    Used for concept extraction, validity filtering, importance filtering, and caption-guided validation (Section 3.2, Appendix A.2). If the proxy's judgments do not transfer to other models, the benchmark selection is biased.
  • domain assumption Replacing a textual concept with a generated image preserves semantic equivalence in context
    Section 3.3 concedes perfect equivalence is impossible and substitutes 'semantic recoverability'; the human study covers 213 of 6,946 images, so the assumption is only partially verified.
  • domain assumption The accuracy drop is due to modality, not to longer sequences, multi-image processing, or image artifacts
    Section 4.2 lists these confounds and argues indirectly that the gap persists for models that handle multi-image inputs, but there is no controlled experiment equalizing sequence length or number of images.
  • domain assumption Training conclusions from Qwen2-VL-7B generalize to other MLLMs
    All mitigation experiments use one 7B base model (Section 5.1), while the abstract and conclusion phrase the result more broadly.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs." pith.science (2026). https://pith.science/paper/HXMSTDT7

@misc{pith2026260728640,
  author       = {Pith},
  title        = {Pith review of: TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HXMSTDT7}},
  note         = {Machine review of arXiv:2607.28640}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Multimodal large language models (MLLMs) should generate consistent responses given semantically equivalent inputs across modalities. However, we observe a systematic discrepancy in model predictions under such cross-modal variations. Specifically, we define the modality gap as the difference in model performance under semantically equivalent textual and multimodal inputs. We introduce TokenSwap, a method that constructs such inputs by replacing textual concepts with semantically aligned images, resulting in sequences where visual tokens are interleaved with text tokens. Based on TokenSwap, we transform existing text-based benchmarks such as MMLU into image-interleaved counterparts, resulting in TokenSwap-Bench. Across 42 MLLMs, we observe a pervasive modality gap, with performance decreasing by 4.2% to 47.4% when moving from text-only to image-interleaved inputs, averaging 19.6% +/- 3.3% across models. Notably, we observe that reasoning models exhibit consistently smaller gaps, achieving an average gap of 10.1% compared to 25.5% for non-reasoning models. In contrast, neither prompting strategies nor scaling training compute alone reliably reduces the modality gap. Finally, we demonstrate that incorporating TokenSwap during training effectively mitigates this gap while preserving strong text-only and vision-language performance.

Figures

Figures reproduced from arXiv: 2607.28640 by Aleksandra Faust, Andong Hua, Arian Hosseini, Colton Bishop, Igor Mordatch, Jindong Gu, Rebecca Roelofs, Yao Qin.

Figure 1
Figure 1. Figure 1: A pervasive modality gap exists: all models perform worse on image-interleaved inputs than on text-only inputs, with gaps ranging from 4.2% to 47.4%. Each point represents a model, colored by family, with triangles indicating reasoning models. The diagonal y = x denotes equal performance, and dashed lines indicate constant gap levels (4.0%, and 15.0%). All models lie below the diagonal, indicating consiste… view at source ↗
Figure 2
Figure 2. Figure 2: TokenSwap constructs semantically aligned image-interleaved questions, where Step 1 panel shows the original MMLU question and the Final Benchmark panel shows its image￾interleaved counterpart. We first extract visualizable concepts and generate or retrieve images representing them, followed by rigorous filtering to ensure validity, task relevance, and semantic consistency, resulting in semantically aligne… view at source ↗
Figure 3
Figure 3. Figure 3: Reasoning models exhibit a smaller modality gap, while prompting strategies fail to reduce it. (a) Distribution of modality gap for reasoning and non-reasoning models. (b) We compare different prompting strategies. Each pair shows text accuracy (hollow) and image-interleaved accuracy (filled), with the modality gap ∆ indicated. non-reasoning models (10.1% ± 2.2% vs. 25.5% ± 3.7%), a difference that is stat… view at source ↗
Figure 4
Figure 4. Figure 4: Scaling provides limited improvement in modality gap, and generation-based bench￾marks exhibit smaller gaps. (a) plots absolute modality gap versus training FLOPs. FLOPs are collected for all open-source non-reasoning models with publicly available training compute esti￾mates, based on original papers and Epoch AI1 . (b) Each blue/red pair corresponds to the same model evaluated with retrieved versus gener… view at source ↗
Figure 5
Figure 5. Figure 5: TOKENSWAP-BENCH is complementary to the existing benchmark. We plot 8 overlap￾ping models, with SEAM [36] results taken from the original paper; numerical values are provided in the Appendix B.5. We observe high Pearson correlation with SEAM in both text accuracy (r = 0.811) and image-interleaved accuracy (r = 0.914), but near-zero correlation in modality gap (r = −0.028). 4.7 Complementary to Existing Ben… view at source ↗
Figure 6
Figure 6. Figure 6: TokenSwap reduces the modality gap while preserving text performance. We compare Text training (blue) and TokenSwap training (red) across both pre-training and post-training, using either generated (Gen) or retrieved (Ret) images. Text training includes Gen and Ret variants, as it is paired with the TokenSwap variants constructed from the same samples. Each pair shows text accuracy (hollow) and image-inter… view at source ↗
Figure 7
Figure 7. Figure 7: Example of a multiple-choice question used in the human study. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Images in few-shot demonstrations. Images are generated by Gemini to represent corresponding textual concepts. In the image-interleaved version, these textual concepts are replaced with the corresponding images. "Argentina" "Amish" "Backgammon" "Beanie Babies" "Cheese" generation retrieval [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Examples of retrieved and generated images for the same concepts. Each column corresponds to a replaced word, with a generated image shown in the top row and a retrieved image shown in the bottom row. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Strong scaling fit but limited improvement in relative modality gap. While the linear fit achieves a high R2 = 0.811, indicating a consistent scaling trend, the slope remains small, suggesting limited gains from increasing training compute. Each point represents a model, plotting relative modality gap against estimated training FLOPs. abstract algebra anatomy astronomy business ethics clinical knowledge c… view at source ↗
Figure 11
Figure 11. Figure 11: Per-subject accuracy on MMLU comparing text-only and image-interleaved inputs. [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Distribution of relative modality gap for reasoning and non-reasoning models. [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: The modality gap increases with the number of image replacements. We average results across all evaluated models and group samples by the number of image replacements. Overall, the modality gap generally increases as more textual concepts are replaced with images. The decrease beyond seven replacements is likely due to the small sample size in these bins. One possible explanation is that, in quantitative … view at source ↗
Figure 14
Figure 14. Figure 14: Applying TokenSwap to the Magpie-Pro dataset. We construct image-interleaved instructions by replacing key textual concepts in the original queries with semantically aligned images. The red text highlights the concepts that are replaced by images. To formulate the task as a conversational instruction, we construct user prompts that query the word shown in the image. For example, given an image–text pair w… view at source ↗
Figure 15
Figure 15. Figure 15: Examples in TOKENSWAP-BENCH 25 [PITH_FULL_IMAGE:figures/full_fig_p025_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Applying TokenSwap to the IIIT 5K-Word dataset. We transform standard image–text pairs into image-interleaved instructions by replacing target words with images. The queries are generated from the ground-truth text using an LLM, where the OCR word is substituted with its image representation. C.5 Effect of Domain Mismatch on TokenSwap Training We study the impact of domain mismatch between TokenSwap train… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 25 linked inside Pith

  1. [1]

    Vision-language models struggle to align entities across modalities

    Iñigo Alonso, Gorka Azkune, Ander Salaberria, Jeremy Barnes, and Oier Lopez De Lacalle. Vision-language models struggle to align entities across modalities. InFindings of the Associa- tion for Computational Linguistics: ACL 2025, pages 18846–18862, 2025

    2025

  2. [2]

    Introducing Claude Haiku 4.5

    Anthropic. Introducing Claude Haiku 4.5. https://www.anthropic.com/news/ claude-haiku-4-5, 2025. Accessed: 2026-05-06

    2025

  3. [3]

    Introducing Claude Opus 4.6

    Anthropic. Introducing Claude Opus 4.6. https://www.anthropic.com/news/ claude-opus-4-6, 2026. Accessed: 2026-05-06

    2026

  4. [4]

    Introducing Claude Sonnet 4.6

    Anthropic. Introducing Claude Sonnet 4.6. https://www.anthropic.com/news/ claude-sonnet-4-6, 2026. Accessed: 2026-05-06

    2026

  5. [5]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Qwen2.5-VL Technical Report.arXiv e-prints, art

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL Technical Report.ar...

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.13923 2025

  7. [7]

    Omnixr: Evaluating omni-modality language models on reasoning across modalities.arXiv preprint arXiv:2410.12219, 2024

    Lichang Chen, Hexiang Hu, Mingda Zhang, Yiwen Chen, Zifeng Wang, Yandong Li, Pranav Shyam, Tianyi Zhou, Heng Huang, Ming-Hsuan Yang, et al. Omnixr: Evaluating omni-modality language models on reasoning across modalities.arXiv preprint arXiv:2410.12219, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2818–2829, 2023

    2023

  10. [10]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Mitigate the gap: Investigating approaches for improving cross-modal alignment in clip.arXiv preprint arXiv:2406.17639, 2024

    Sedigheh Eslami and Gerard de Melo. Mitigate the gap: Investigating approaches for improving cross-modal alignment in clip.arXiv preprint arXiv:2406.17639, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    It’s not a modality gap: Characterizing and addressing the contrastive gap.arXiv preprint arXiv:2405.18570, 2024

    Abrar Fahim, Alex Murphy, and Alona Fyshe. It’s not a modality gap: Characterizing and addressing the contrastive gap.arXiv preprint arXiv:2405.18570, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Datacomp: In search of the next generation of multimodal datasets.Advances in Neural Information Processing Systems, 36:27092–27112, 2023

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets.Advances in Neural Information Processing Systems, 36:27092–27112, 2023. 10

    2023

  15. [15]

    Gemini API Models

    Google. Gemini API Models. https://ai.google.dev/gemini-api/docs/models, 2026. Accessed: 2026-05-06

    2026

  16. [16]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Pith/arXiv arXiv 2009

  17. [17]

    Glm-4.5 v and glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning.arXiv preprint arXiv:2507.01006, 2025

    Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.5 v and glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning.arXiv preprint arXiv:2507.01006, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  19. [19]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. InInternational conference on machine learning, pages 4904–4916. PMLR, 2021

    2021

  20. [20]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning

    Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:17612–17625, 2022

    2022

  22. [22]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306, 2024

    2024

  23. [23]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233. Springer, 2024

    2024

  24. [24]

    Scene text recognition using higher order language priors

    Anand Mishra, Karteek Alahari, and CV Jawahar. Scene text recognition using higher order language priors. InBMVC-British machine vision conference. BMV A, 2012

    2012

  25. [25]

    GPT-4o System Card

    OpenAI. GPT-4o System Card. https://openai.com/index/gpt-4o-system-card/ ,

  26. [26]

    Introducing GPT-4.1 in the API

    OpenAI. Introducing GPT-4.1 in the API. https://openai.com/index/gpt-4-1/ , 2025. Accessed: 2026-05-06

    2025

  27. [27]

    Introducing GPT-5

    OpenAI. Introducing GPT-5. https://openai.com/index/introducing-gpt-5/ , 2025. Accessed: 2026-05-06

    2025

  28. [28]

    Introducing GPT-5.1 for developers

    OpenAI. Introducing GPT-5.1 for developers. https://openai.com/index/ gpt-5-1-for-developers/, 2025. Accessed: 2026-05-06

    2025

  29. [29]

    Introducing GPT-5.2

    OpenAI. Introducing GPT-5.2. https://openai.com/index/introducing-gpt-5-2/ ,

  30. [30]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  31. [31]

    Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language models.arXiv preprint arXiv:2404.07983, 2024

    Simon Schrodi, David T Hoffmann, Max Argus, V olker Fischer, and Thomas Brox. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language models.arXiv preprint arXiv:2404.07983, 2024. 11

    Pith/arXiv arXiv 2024

  32. [32]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    Pith/arXiv arXiv 2017

  33. [33]

    Understanding the modality gap in clip.ICLR, Stockholm, Sweden, pages 8–10, 2023

    PY Shi, M Welle, M Bjørkman, and D Kragic. Understanding the modality gap in clip.ICLR, Stockholm, Sweden, pages 8–10, 2023

    2023

  34. [34]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019

    2019

  35. [35]

    Eva-clip: Improved training techniques for clip at scale.arXiv preprint arXiv:2303.15389, 2023

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale.arXiv preprint arXiv:2303.15389, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    Seam: Semantically equivalent across modalities benchmark for vision-language models.arXiv preprint arXiv:2508.18179, 2025

    Zhenwei Tang, Difan Jiao, Blair Yang, and Ashton Anderson. Seam: Semantically equivalent across modalities benchmark for vision-language models.arXiv preprint arXiv:2508.18179, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    Gemini: A family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A family of highly capable multimodal models. arxiv 2023.arXiv preprint arXiv:2312.11805, 2024

    Pith/arXiv arXiv 2023

  38. [38]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    Same content, different answers: Cross-modal inconsistency in mllms.arXiv preprint arXiv:2512.08923, 2025

    Angela van Sprang, Laurens Samson, Ana Lucic, Erman Acar, Sennay Ghebreab, and Yuki M Asano. Same content, different answers: Cross-modal inconsistency in mllms.arXiv preprint arXiv:2512.08923, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Xmodbench: Benchmarking cross-modal capabilities and consistency in omni-language models.arXiv preprint arXiv:2510.15148, 2025

    Xingrui Wang, Jiang Liu, Chao Huang, Xiaodong Yu, Ze Wang, Ximeng Sun, Jialian Wu, Alan Yuille, Emad Barsoum, and Zicheng Liu. Xmodbench: Benchmarking cross-modal capabilities and consistency in omni-language models.arXiv preprint arXiv:2510.15148, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing.arXiv preprint arXiv:2406.08464, 2024

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing.arXiv preprint arXiv:2406.08464, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Explaining and mitigating the modality gap in contrastive multimodal learning.arXiv preprint arXiv:2412.07909, 2024

    Can Yaras, Siyi Chen, Peng Wang, and Qing Qu. Explaining and mitigating the modality gap in contrastive multimodal learning.arXiv preprint arXiv:2412.07909, 2024

    arXiv 2024

  44. [44]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9556–9567, 2024

    2024

  45. [45]

    Mmmu-pro: a more robust multi-discipline multimodal understanding benchmark (2025).URL https://arxiv.org/abs/2409.02813, 2025

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: a more robust multi-discipline multimodal understanding benchmark (2025).URL https://arxiv.org/abs/2409.02813, 2025

    Pith/arXiv arXiv 2025

  46. [46]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023

    2023

  47. [47]

    Lost in translation: When gpt-4v (ision) can’t see eye to eye with text

    Xiang Zhang, Senyu Li, Zijun Wu, and Ning Shi. Lost in translation: When gpt-4v (ision) can’t see eye to eye with text. a vision-language-consistency analysis of vllms and beyond.arXiv preprint arXiv:2310.12520, 2023. 12

    Pith/arXiv arXiv 2023

  48. [48]

    Cross-modal consistency in multimodal large language models.arXiv preprint arXiv:2411.09273, 2024

    Xiang Zhang, Senyu Li, Ning Shi, Bradley Hauer, Zijun Wu, Grzegorz Kondrak, Muhammad Abdul-Mageed, and Laks VS Lakshmanan. Cross-modal consistency in multimodal large language models.arXiv preprint arXiv:2411.09273, 2024

    Pith/arXiv arXiv 2024

  49. [49]

    apple,"

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. A Details for TOKENSWAP-BENCHConstruction A.1 Detailed Token-Level Formalization of TokenSwap In Se...

    Pith/arXiv arXiv 2025

  50. [52]

    - **RETURN FALSE IF**: The word is a verb but the image shows a noun (e.g., ’die’ as in death vs

    **Part-of-Speech (POS) Match**: - The image MUST represent the word in its specific grammatical role in THIS sentence. - **RETURN FALSE IF**: The word is a verb but the image shows a noun (e.g., ’die’ as in death vs. ’die’ as a dice)

  51. [53]

    human head

    **Semantic Sense Alignment (Primary Defense)**: - Words often have multiple meanings. The image MUST match the exact sense used in the context. - **RETURN FALSE IF**: - Text means "human head" but image shows a "guitar amplifier head". - Text means "human nature" but image shows "forest/greenery". - Text means "tea time/meal" but image shows "tea leaves/p...

  52. [54]

    hot desert

    **Environmental & Adjective Consistency**: - The image must not contradict the sentence’s setting or descriptors. - **RETURN FALSE IF**: Text describes a "hot desert" but image shows "wet sea sand"; text says "wrinkled" but image is "smooth"

  53. [55]

    **No Symbols, Icons, or Text**: - **RETURN FALSE IF**: The image is an icon, a diagram, a branded product, or contains significant printed text (like a book cover or a screenshot of words)

  54. [56]

    The hot <image> (sand) scorched his feet

    **Entity Specificity**: - **RETURN FALSE IF**: The image is a specific artifact (a toy, a statue, a specific brand) when the text refers to a generic living being or object. --- ### Calibration Examples: - "The hot <image> (sand) scorched his feet." | Image: Ocean waves on sand. | Result: {{"i": 0, " word": "sand", "ok": false}} - "The sun beat on his <im...

  55. [57]

    XModBench [41] further extends cross-modal consistency evaluation to tri-modal settings involving text, vision, and audio

    construct semantically equivalent comparisons in controlled domains with standardized textual and visual notations (e.g., chess, chemistry, music, and graphs), and observes notable cross-modal inconsistencies. XModBench [41] further extends cross-modal consistency evaluation to tri-modal settings involving text, vision, and audio. 28

  56. [2025]

    Accessed: 2026-05-06

    2026

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.