Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One transformer over shared image-text tokens beats late fusion.

desk verdict Promising single-encoder design, but the early-vs-late fusion claim is confounded by the frozen reconstruction tokenizer versus from-scratch baseline encoders. read the letter →

arxiv 2506.03096 v1 pith:7YZVGICQ submitted 2025-06-03 cs.CV cs.LG

Christian Schlarmann , Francesco Croce , Nicolas Flammarion , Matthias Hein This is my paper
classification cs.CVcs.LG
keywords multimodalembeddingearlyfusiondiscreteimagetokenizationcontrastivepre-trainingmaskedmodelingtext-guidedtransformationhardnegativessingle-encodertransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that CLIP-style contrastive pre-training does not need two separate encoders: a single transformer over a shared vocabulary of discrete image and text tokens can embed unimodal and multimodal inputs alike, with image and text interacting from the first layer. That early fusion, the authors argue, produces richer representations than late fusion, which merges independently computed unimodal features only at the end. FuseLIP implements the idea by converting images into 128 discrete tokens with a frozen TiTok tokenizer, appending text tokens, and training the one encoder with a sigmoid contrastive loss plus a masked multimodal modeling loss on the same forward pass. On new tasks that require fine visual structure, such as retrieving the correctly cropped, rotated, or flipped image or distinguishing the left versus right instance of an object, FuseLIP-B beats late-fusion baselines by large margins while remaining competitive on classification, VQA, retrieval, and grounding. A companion finding is that hard negatives in each batch are essential: removing them drops text-guided transformation accuracy from 94.3% to 13.6%.

What carries the argument

The central object is the multimodal token sequence: a frozen TiTok image tokenizer maps each image to 128 discrete visual tokens, the text tokenizer maps captions and prompts to text tokens, and a single transformer encoder with bidirectional attention processes the concatenated sequence (images first, then text with boundary tokens). The final embedding is the output at the end-of-text token, so both modalities contribute to one vector. The training signal is the sigmoid contrastive loss of SigLIP, applied across mixed unimodal and multimodal samples, plus a masked multimodal modeling loss in which 10% of non-special tokens are masked and a shared-weight prediction head reconstructs them. A batch-sampling scheme that guarantees hard negatives, such as all nine crops of the same image or three additional region descriptions for the same image, is what makes the multimodal tasks learnable.

What would settle it

Swap the image tokenizer for one trained to be rotation- or flip-invariant while keeping all other components fixed, then run the CC3M-TGIT rotation and flip subtasks: if accuracy drops toward chance while semantic VQA accuracy is unchanged, the early-fusion advantage is carried by the tokenizer's spatial fidelity rather than by early fusion per se. A second check is to scramble the image-token embedding matrix at inference, which should cause large drops on crop, rotate, and flip retrieval but smaller drops on classification if the paper's mechanism is right.

Watch

Extended reading notes

Core claim

The central discovery is that a single encoder can serve as a CLIP-style contrastive model if both modalities are first reduced to discrete tokens in one shared vocabulary. Images go through a frozen TiTok tokenizer into 128 visual tokens; text goes through the standard BPE tokenizer; the two sequences are concatenated with special boundary tokens and fed to a bidirectional transformer whose end-of-text output is the joint embedding. Because attention spans the whole sequence at every layer, the text query can steer the image representation as early as the first block, which the authors identify as the reason FuseLIP solves text-guided transformations: on CC3M-TGIT and CC12M-TGIT, FuseLIP-B reaches 94.3% and 94.2% accuracy versus 69.4% and 78.1% for the strongest late-fusion baseline. The same token-level design makes masked multimodal modeling cheap, since one prediction head on the same forward pass reconstructs masked tokens, and the MMM loss improves every task family. The paper also establishes that contrastive multimodal training requires hard negatives, i.e., batches containing several transformations of the same image or several descriptions of the same image.

Load-bearing premise

Everything rests on the frozen TiTok image tokenizer retaining enough spatial detail, such as layout, orientation, and local texture, for the model to solve text-guided transformations; if tokenization discarded that information, the early-fusion advantage would disappear.

Editorial extensions

If this is right

  • A single encoder trained this way can serve both unimodal and multimodal inputs, so image-only, text-only, and image-text embeddings live in one space and can be compared directly.
  • Masked modeling and contrastive objectives can be combined in one forward pass, simplifying setups that otherwise need separate heads and extra tokenizers.
  • Multimodal evaluation should include structure-sensitive tasks such as crop, rotation, flip, and position retrieval, where late fusion fails, rather than relying only on semantic benchmarks.
  • Hard-negative batch construction becomes a core ingredient of contrastive multimodal training rather than an optional trick.
  • The approach inherits advances in discrete image tokenization directly: better reconstruction tokenizers should translate into better embeddings without retraining the fusion architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the frozen reconstruction tokenizer is what preserves spatial structure, then any high-fidelity discrete image tokenizer, even one never exposed to text, may serve as a universal vision front-end, substantially lowering the cost of building new multimodal encoders.
  • The paper's observation that early fusion yields smaller modality gaps at initialization hints that a shared token vocabulary makes the representation space inherently more symmetric; a testable consequence is that linear probes on multimodal embeddings should behave differently from those on two-tower features.
  • Because inputs are merely concatenated token sequences, FuseLIP should extend to multiple images or interleaved image-text inputs without architectural change; a natural stress test is conversational retrieval where a query refers to two previous images.
  • The near-perfect text-guided transformation scores raise the question of whether the model learns transformations as compositional operations or exploits token-level statistical shortcuts; an extension would be to test on held-out transformation types or unseen object categories and see whether the large margin persists.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces FuseLIP, a multimodal embedding method based on early fusion of discrete image and text tokens. Images are encoded by a frozen TiTok tokenizer into 128 discrete tokens, and these are concatenated with text tokens and processed by a single transformer encoder trained with a sigmoid contrastive loss and a masked multimodal modeling (MMM) loss. The authors collect and generate new training data (CC3M/CC12M-TGIT, caption-derived VQA, VG-Crop, HQ-Edit) and introduce two new evaluation tasks (OI-Crop, OI-Pos). They compare FuseLIP against two late-fusion baselines (score fusion and MagicLens-style feature fusion) trained on the same data, reporting that FuseLIP-B achieves the best results on most benchmarks, with especially large margins on text-guided transformation tasks and OI-Pos, while remaining comparable on unimodal tasks such as classification and ImageNet.

Significance. If the central claim is supported, FuseLIP offers a conceptually simple and effective alternative to late fusion for multimodal embeddings, and its use of a single encoder with a frozen discrete tokenizer elegantly enables an MMM loss without extra modules. The paper provides code and model links, detailed hyperparameters, and evaluates on the external MMEB benchmark in addition to its own tasks, which are clearly described and will be useful to the community. The MMM-loss simplification and the hard-negative construction are well demonstrated. However, the main comparison between early and late fusion is not yet cleanly isolated from the visual backbone initialization, and the largest margins occur on author-designed tasks; the external MMEB results are more moderate, though still consistently in FuseLIP's favor.

major comments (3)
  1. [§5.3, Tables 3–4] The central claim that early fusion outperforms late fusion is confounded by the visual backbone initialization. FuseLIP uses the frozen TiTok tokenizer pretrained for reconstruction on ImageNet (described in Sec. 3.1), whereas the SigLIP-SF and SigLIP-MLF baselines train their CLIP vision encoders from scratch on CC3M/CC12M. Thus fusion depth, visual tokenization, and visual pretraining vary simultaneously across the compared models. The largest margins on TGIT and OI-Pos concern exactly spatial layout and orientation (Table 4, e.g., crop 99.4 vs. 42.7 for FuseLIP-B vs. SigLIP-BMLF on CC3M+MM), and a reconstruction-pretrained tokenizer is designed to preserve such information. The discussion in Sec. 5.3 attributes the gap to early fusion, but the experiments do not rule out that a late-fusion baseline with a reconstruction-pretrained vision encoder would close it. Please add an ablation that controls for visual pretraining, e.g., a late-fusion baseline initialized from a reconstruction-pretrained ViT or a late-fusion variant that consumes the same frozen TiTok features, or an early-fusion variant whose tokenizer is trained from scratch on the same data.
  2. [§5.1, first paragraph] The statement 'As we train all models from scratch, we do not compare against methods that fine-tune pre-trained models' is inaccurate because FuseLIP relies on the frozen TiTok tokenizer, which is a pre-trained component. This asymmetry is precisely the confound behind the main comparison. The sentence should be revised to clarify that the encoder is trained from scratch while the image tokenizer is a fixed pre-trained module, and the implications for the comparison should be discussed.
  3. [Table 5, §5.4] The claim that the MMM loss 'improves performance across all tasks' is not supported by Table 5. For FuseLIP-S trained on CC3M+MM, adding the MMM loss decreases TGIT from 83.8 to 79.0. The subsequent sentence that masking removal leads to 'significantly worse results across all tasks' is likewise contradicted by this row. Please qualify the claim to the settings where it holds, or run additional seeds to verify whether the direction is stable.
minor comments (6)
  1. [§5.1] The statement that the authors do not compare against methods that fine-tune pre-trained models is also contradicted by Table 11, where VLM2Vec (fine-tuned on MMEB) is included. The main tables may exclude such methods, but the text should say so explicitly.
  2. [Table 5] The formatting of the 'Hard Neg.' column uses '✗-' and '✓ ✓' inconsistently; the reader has to infer that '-' means the same as '✗' for the MMM column. Please use uniform symbols.
  3. [§3.1, Eq. (1)] The notation fθtok,θenc suggests that both the tokenizer and encoder are parameters of the model, but the tokenizer is frozen. It would be cleaner to write fθenc and state that θtok is fixed.
  4. [Throughout] No error bars or multiple seeds are reported, so it is unclear whether the smaller differences in Table 3 (e.g., Classification 31.2 vs. 30.3 for CC12M+MM) are significant.
  5. [Table 8] The asset table contains a typo: 'VLM2Vechttps://huggingface.co/...' is missing a space between the model name and the URL.
  6. [Abstract and §4] The abstract says the authors 'collect new datasets,' but the main new datasets are generated from existing corpora (e.g., CC3M/VG/OpenImages). 'Generate' would be more precise than 'collect'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FuseLIP's claims are empirical comparisons against external and held-out benchmarks, with no equation-level or citation-level reduction of outputs to inputs.

full rationale

FuseLIP's central claim—that early fusion of discrete image and text tokens improves multimodal embeddings—is supported by training runs evaluated on the external MMEB benchmark and on held-out validation splits of tasks the authors constructed (CC3M-TGIT validation split, OI-Crop, OI-Pos). The paper's own equations in Section 3.2 define the SigLIP and MMM losses as training objectives; neither is derived from the evaluation metrics, and no parameter is fitted to the reported benchmark numbers and then renamed a prediction. The MMM loss is ablated in Table 5 rather than assumed to work, and hard negatives are likewise ablated. References to TiTok and Chameleon are external prior work, not self-citations, and no uniqueness theorem is imported from the authors' own prior papers. The comparison is not perfectly controlled because FuseLIP uses a frozen reconstruction-pretrained TiTok vision side while the SigLIP baselines train their CLIP vision encoders from scratch, but this is an experimental confound or correctness risk, not a circular reduction of the kind where an equation equals its own input by construction. Designing new evaluation tasks is a legitimate empirical contribution and does not make the results circular, especially since the paper also reports scores on the pre-existing MMEB benchmark. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical, so the ledger lists hyperparameters and domain assumptions. No new physical or theoretical entities are introduced.

free parameters (3)
  • alpha (MMM loss weight) = 0.25
    Chosen by hand in all experiments; balances contrastive and masked modeling losses.
  • mask probability p = 0.1
    Chosen by hand for masked modeling; not tuned.
  • number of image tokens = 128
    Determined by the frozen TiTok tokenizer; impacts sequence length but is not fitted.
assumptions (4)
  • domain assumption The frozen TiTok tokenizer provides a discrete vocabulary that preserves sufficient visual information for multimodal alignment.
    Section 3.1 states TiTok is trained for image reconstruction without text-guided semantic alignment, so it does not introduce bias.
  • standard math The SigLIP sigmoid loss can be applied to a single encoder processing both modalities.
    Equation 2 in Section 3.2 reuses the loss with a single function f.
  • domain assumption Masked multimodal modeling is a useful auxiliary objective for learning multimodal embeddings.
    Section 5.4 shows it improves performance, but the benefit is empirical.
  • domain assumption Training on CC3M/CC12M plus generated multimodal data is sufficient to learn general multimodal representations.
    Section 4 chooses these datasets due to academic compute constraints.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens." pith.science (2026). https://pith.science/paper/7YZVGICQ

@misc{pith2026250603096,
  author       = {Pith},
  title        = {Pith review of: FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7YZVGICQ}},
  note         = {Machine review of arXiv:2506.03096}
}
read the original abstract

Contrastive language-image pre-training aligns the features of text-image pairs in a common latent space via distinct encoders for each modality. While this approach achieves impressive performance in several zero-shot tasks, it cannot natively handle multimodal inputs, i.e., encoding image and text into a single feature vector. As a remedy, it is common practice to use additional modules to merge the features extracted by the unimodal encoders. In this work, we present FuseLIP, an alternative architecture for multimodal embedding. Leveraging recent progress in discrete image tokenizers, we propose to use a single transformer model which operates on an extended vocabulary of text and image tokens. This early fusion approach allows the different modalities to interact at each depth of encoding and obtain richer representations compared to common late fusion. We collect new datasets for multimodal pre-training and evaluation, designing challenging tasks for multimodal encoder models. We show that FuseLIP outperforms other approaches in multimodal embedding tasks such as VQA and text-guided image transformation retrieval, while being comparable to baselines on unimodal tasks.

Figures

Figures reproduced from arXiv: 2506.03096 by the authors.

Figure 1
Figure 1. Comparison of architectures. To obtain multimodal embedding via contrastive learning, late fusion approaches first extract unimodal representations via unimodal encoders, then merge by addition [27] or a fusion module [48]. Conversely, our FuseLIP uses a frozen image tokenizer to tokenize inputs of any modality into tokens from a unified vocabulary, which are then processed by a single encoder model. This approach l… view at source ↗
Figure 2
Figure 2. OI-Pos, OI-Crop and VG-Crop tasks. We show examples of these tasks (described in Sec. 5.2). The retrieval pool of OI-Crop comprises crops from the same image, as well as crops of the target object from other images. In contrast, for OI-Pos and VG-Crop it contains only crops of the query image. We show the whole retrieval pool for OI-Pos, OI-Crop, and a sample for VG-Crop. For every sample the ground-truth answer is … view at source ↗
Figure 3
Figure 3. System prompt for generating question-answer pairs. We use Llama-3.1-8B-Instruct to generate VQA samples from image-text pairs of CC3M as described in Sec. 4.2. not trained on instruction-following data, we remove this part of the prompts (except when evaluating the VLM2Vec model in [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: CC3M-TGIT evaluation examples. We illustrate the tasks in CC3M-TGIT, together with the prediction of the embedding models. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]
Figure 5
Figure 5. Figure 5: OI-Crop and OI-Pos. We show images retrieved on the OI-Crop and OI-Pos tasks by models trained on CC12M+MM. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: VG-Crop. We show images retrieved on the VG-Crop task by models trained on CC12M+MM. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Multimodal pretraining transfers asymmetrically: language boosts vision, understanding boosts generation, generation is mostly neutral, and early unified training prevents vision laziness.

Reference graph

Works this paper leans on

55 extracted references · 44 canonical work pages · cited by 1 Pith paper

  1. [1]

    4M-21: An any-to-any vision model for tens of tasks and modalities

    Roman Bachmann, O˘guzhan Fatih Kar, David Mizrahi, Ali Garjani, Mingfei Gao, David Griffiths, Jiaming Hu, Afshin Dehghan, and Amir Zamir. 4M-21: An any-to-any vision model for tens of tasks and modalities. InNeurIPS, 2024. 1, 3

    work page 2024

  2. [2]

    Zero-shot composed image retrieval with textual inversion

    Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. Zero-shot composed image retrieval with textual inversion. InICCV, 2023. 2

    work page 2023

  3. [3]

    BEiT: BERT pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEiT: BERT pre-training of image transformers. In ICLR, 2022. 4

    work page 2022

  4. [4]

    Paligemma: A versatile 3b vlm for transfer.arXiv preprint arXiv:2407.07726, 2024

    Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer.arXiv preprint arXiv:2407.07726, 2024. 1

    arXiv 2024

  5. [5]

    All you may need for vqa are image captions

    Soravit Changpinyo, Doron Kukliansky, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. All you may need for vqa are image captions. InNAACL, 2022. 6

    work page 2022

  6. [6]

    Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. InCVPR, 2021. 5

    work page 2021

  7. [7]

    Understanding transferable representation learning and zero-shot transfer in CLIP

    Zixiang Chen, Yihe Deng, Yuanzhi Li, and Quanquan Gu. Understanding transferable representation learning and zero-shot transfer in CLIP. InICLR, 2024. 1

    work page 2024

  8. [8]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. InCVPR, 2023. 1, 6, 13

    work page 2023

Show all 55 references
  1. [9]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InCVPR, 2009. 8, 15

    2009

  2. [10]

    Bert: Pre-training of deep bidirec- tional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. InNAACL-HLT, 2019. 4

    2019

  3. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

  4. [12]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024. 6, 13

    2024 arXiv

  5. [13]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. InCVPR, 2021. 3

    2021

  6. [14]

    Dreamsim: Learning new dimensions of human visual similarity using synthetic data

    Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. InNeurIPS, 2023. 1 10

    2023

  7. [15]

    Language-only efficient training of zero-shot composed image retrieval

    Geonmo Gu, Sanghyuk Chun, Wonjae Kim, Yoohoon Kang, and Sangdoo Yun. Language-only efficient training of zero-shot composed image retrieval. InCVPR, 2024. 1, 2

    2024

  8. [16]

    Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. InNeurIPS, 2023. 15

    2023

  9. [17]

    Hq-edit: A high-quality dataset for instruction-based image editing

    Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang, Yuyin Zhou, and Cihang Xie. Hq-edit: A high-quality dataset for instruction-based image editing. InICLR, 2025. 6

    2025

  10. [18]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. InICML, 2021. 2

    2021

  11. [19]

    E5-v: Universal embeddings with multimodal large language models.arXiv preprint arXiv:2407.12580, 2024

    Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language models.arXiv preprint arXiv:2407.12580, 2024. 2

    2024 arXiv

  12. [20]

    Vlm2vec: Training vision-language models for massive multimodal embedding tasks

    Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. Vlm2vec: Training vision-language models for massive multimodal embedding tasks. InICLR, 2025. 2, 7, 13, 14, 16, 17

    2025

  13. [21]

    Hard negative mixing for contrastive learning

    Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. InNeurIPS, 2020. 6

    2020

  14. [22]

    Democratizing text-to-image masked generative models with compact text-aware one-dimensional tokens

    Dongwon Kim, Ju He, Qihang Yu Yu, Chenglin Yang, Xiaohui Shen, Suha Kwak, and Chen Liang-Chieh. Democratizing text-to-image masked generative models with compact text-aware one-dimensional tokens. arXiv preprint arXiv:2501.07730, 2025. 3

    2025 arXiv

  15. [23]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations.IJCV, 2017

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations.IJCV, 2017. 6

    2017

  16. [24]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visu...

    2020

  17. [25]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning

    Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. InNeurIPS, 2022. 16

    2022

  18. [26]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023. 1

    2023

  19. [27]

    Universal vision-language dense retrieval: Learning a unified representation space for multi-modal retrieval

    Zhenghao Liu, Chenyan Xiong, Yuanhuiyi Lv, Zhiyuan Liu, and Ge Yu. Universal vision-language dense retrieval: Learning a unified representation space for multi-modal retrieval. InICLR, 2023. 1, 2, 3

    2023

  20. [28]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InICLR, 2018. 13

    2018

  21. [29]

    Unified-io: A unified model for vision, language, and multi-modal tasks.arXiv preprint arXiv:2206.08916, 2022

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks.arXiv preprint arXiv:2206.08916, 2022. 1

    2022 arXiv

  22. [30]

    4M: Massively multimodal masked modeling

    David Mizrahi, Roman Bachmann, O˘guzhan Fatih Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4M: Massively multimodal masked modeling. InNeurIPS, 2023. 1, 3

    2023

  23. [31]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InICML, 2021. 1, 2, 4, 8, 15

    2021

  24. [32]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 2020. 1

    2020

  25. [33]

    Hierarchical text-conditional image generation with clip latents.arXiv:2204.06125, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv:2204.06125, 2022. 1

    2022 arXiv

  26. [34]

    Contrastive learning with hard negative samples

    Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. InICLR, 2021. 6

    2021

  27. [35]

    Pic2word: Mapping pictures to words for zero-shot composed image retrieval

    Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, and Tomas Pfister. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. InCVPR, 2023. 1, 2 11

    2023

  28. [36]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. InACL, 2018. 5

    2018

  29. [37]

    Towards understanding the modality gap in clip

    Peiyang Shi, Michael C Welle, Mårten Björkman, and Danica Kragic. Towards understanding the modality gap in clip. InICLR 2023 workshop on multimodal representation learning: perks and pitfalls, 2023. 16

    2023

  30. [38]

    FLA V A: A foundational language and vision alignment model

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. FLA V A: A foundational language and vision alignment model. InCVPR,

  31. [39]

    Eva-clip: Improved training techniques for clip at scale.arXiv preprint arXiv:2303.15389, 2023

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale.arXiv preprint arXiv:2303.15389, 2023. 1

    2023 arXiv

  32. [40]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024. 3

    2024 arXiv

  33. [41]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. InNeurIPS, 2017. 3

    2017

  34. [42]

    Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to- sequence learning framework

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to- sequence learning framework. InICML, 2022. 1

    2022

  35. [43]

    Uniir: Training and benchmarking universal multimodal information retrievers

    Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. InECCV, 2024. 1

    2024

  36. [44]

    Robust fine-tuning of zero-shot models

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. InCVPR, 2022. 1

    2022

  37. [45]

    Bridgetower: Building bridges between encoders in vision-language representation learning

    Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, Wanxiang Che, and Nan Duan. Bridgetower: Building bridges between encoders in vision-language representation learning. InAAAI, 2023. 1, 2

    2023

  38. [46]

    An image is worth 32 tokens for reconstruction and generation

    Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. InNeurIPS, 2024. 1, 3, 13

    2024

  39. [47]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InCVPR, 2023. 1, 2, 4, 13, 17

    2023

  40. [48]

    Magiclens: Self-supervised image retrieval with open-ended instructions

    Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming-Wei Chang. Magiclens: Self-supervised image retrieval with open-ended instructions. InICML, 2024. 1, 2, 3, 6, 7, 13

    2024

  41. [49]

    upper left, upper center,

    Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, and Yongping Xiong. Vista: visualized text embedding for universal multi-modal retrieval. InACL, 2024. 1, 2, 3 12 Appendix This appendix provides additional details and results to support the main text. In App. A we report details ...

    2024

  42. [50]

    For g e n e r a t i n g question / answer pairs , only use i n f o r m a t i o n that is evident from the caption

  43. [51]

    Do not mention the word ’ caption ’ in the question or answer

  44. [52]

    Answers should be at least a couple words long ( not single word )

  45. [53]

    Don ’ t start every question with " What "

  46. [54]

    The {object_name} on the left/right

    Respond in the format : Question : < question > Answer : < answer > Examples : Caption : A group of friends are having a barbecue in the backyard . Question : Where is the barbecue taking place ? Answer : In the backyard . Caption : A child is playing with a toy airplane on th...

  47. [55]

    Notably, this model is much larger in the amount of parameters (4.15B, i.e

    that is based on the Phi-3.5-V large vision-language model and fine-tuned on MMEB. Notably, this model is much larger in the amount of parameters (4.15B, i.e. over 25x that of FuseLIP-B) and has undergone much longer pre-training. Expectedly, this model outperforms our MMEB-fi...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.