Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Visual Lexicon: Rich Image Features in Language Space

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ViLex asks whether a single image representation can both reconstruct pixels and understand scenes, and answers by mapping images into the text-token space of a frozen text-to-image diffusion model.

desk verdict A useful continuous-token alternative to DeDiffusion with convincing reconstruction results, but the VLM gains need a matched training control before the understanding claim is fully established. read the letter →

arxiv 2412.06774 v1 pith:QDUTESFM submitted 2024-12-09 cs.CV cs.AIcs.LG

XuDong Wang , Xingyi Zhou , Alireza Fathi , Trevor Darrell , Cordelia Schmid This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords visuallanguagetext-to-imagediffusionimagereconstructionvisionencoderself-supervisedlearningtextembeddingspacezero-shotgenerationrepresentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ViLex, an image encoder whose outputs live directly in the text-embedding space of a frozen text-to-image diffusion model. ViLex is trained only to reconstruct the input image through that frozen decoder, so the same representation must carry high-level semantics and fine visual detail. The authors report that even a single ViLex token reconstructs images with lower FID than a text-based inversion baseline, and that using ViLex as a vision encoder improves a vision-language model across 15 benchmarks relative to a strong SigLIP baseline. The central wager is that a generative decoder can serve as a teacher for understanding features.

What carries the argument

The load-bearing object is the ViLex token sequence, produced by attention pooling over ViT patch features and placed into the text-embedding lookup space of a frozen text encoder, so a frozen diffusion U-Net treats the tokens exactly like text. The training loss is the standard diffusion denoising objective through that frozen decoder; TailDrop randomly drops the last k tokens during training to push semantic information into the earlier tokens; and Text-Free Guidance interpolates between ViLex-only and ViLex-plus-text conditioning at inference. The frozen decoder is a 64x64 Imagen model, which sets a hard resolution bottleneck on the visual information the representation must carry.

What would settle it

Train the same ViLex pipeline with a higher-resolution base T2I model, for example a 256x256 Imagen, and measure small-object recall and fine-detail reconstruction; if the representation quality stays flat despite the richer decoder, the encoder objective is the bottleneck, while a sharp increase would tie the current claims to the 64x64 teacher. Alternatively, probe a frozen single-token ViLex embedding on the Describable Textures dataset: if it cannot discriminate texture classes above chance, the claim that one ViLex token preserves fine visual detail is falsified.

Watch

Extended reading notes

Core claim

ViLex is a 'visual lexicon': a sequence of continuous tokens that behave like text tokens for a frozen text-to-image model but encode visual content that words cannot express. The encoder, a ViT initialized from SigLIP followed by five attention-pooling blocks, projects patch features into the index-to-embedding lookup space of the T2I model's text encoder, so the frozen diffusion U-Net can consume them as if they were text. Training minimizes a standard denoising objective, backpropagating through the frozen text encoder and U-Net into the vision encoder; a TailDrop schedule randomly drops trailing tokens, forcing the leading tokens to carry the semantic core. At inference the token count becomes a dial between coarse semantics (one token) and fine detail (75 tokens). The paper claims this setup beats text-based image inversion in reconstruction fidelity, enables zero-shot DreamBooth-style personalization by concatenating ViLex tokens with natural-language prompts, and improves a SigLIP-initialized vision encoder on captioning, visual question answering, and referring segmentation.

Load-bearing premise

The paper assumes that training only to reconstruct images through a frozen 64x64 text-to-image model forces the embeddings to carry both semantic and detailed visual information that transfers to understanding tasks; if the reconstruction loss instead biases the encoder toward low-level memorization, or if the low-resolution decoder drops information such as small objects, the claimed gains over SigLIP would be overstated.

Editorial extensions

If this is right

  • ViLex tokens can be typed directly into any text prompt, turning subject-driven generation into a zero-shot operation that needs no per-object fine-tuning.
  • Vision encoders trained with the ViLex objective improve in both reconstruction and understanding, so the same objective can upgrade existing encoders such as SigLIP.
  • The token count becomes a user-settable fidelity dial, letting systems trade semantic compactness for fine detail as the task requires.
  • Because ViLex tokens share a space with text, a single sequence can interleave visual and textual content for a frozen T2I model, enabling re-contextualization, art rendition, and accessorization without adapter modules.
  • Representation learning for understanding can draw its training signal from a frozen generative decoder alone, with image-only training barely hurting downstream vision-language performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If ViLex transfers to other frozen generative models, the apparent dichotomy between understanding features and generative features becomes a matter of training target rather than architecture, suggesting a general recipe for visual-language pretraining.
  • The paper's FID comparison uses different token counts for DeDiffusion and ViLex; a fairer bit-budget-equal test would hold the total token count constant, and the paper's one-token result predicts ViLex would still win.
  • The 64x64 base model is a likely ceiling: a higher-resolution frozen T2I decoder should improve small-object recall and fine-detail reconstruction, a testable extension the supplement itself gestures toward.
  • ViLex tokens could double as a retrieval index, since images become searchable in text space by cosine similarity to caption embeddings, yielding a text-to-image retrieval procedure that inherits the T2I model's semantic granularity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ViLex, a ViT-based image encoder with attention pooling that maps images into n continuous embeddings in the token-embedding space of a frozen text encoder (OpenCLIP ViT-H/14) that conditions a frozen 64x64 Imagen text-to-image model. Training uses the diffusion denoising reconstruction loss plus a TailDrop strategy that randomly drops the last k tokens to encourage earlier tokens to carry richer semantics. The authors report three main results: (i) image reconstruction with lower FID/IS than DeDiffusion and image-guided DALL-E 3, including single-token reconstructions; (ii) zero-shot DreamBooth-style editing by concatenating ViLex tokens with natural-language prompts; and (iii) improved vision-language understanding when replacing SigLIP in the PaliGemma VLM with ViLex features, reported across 15 benchmarks. The supplement provides human-study instructions, ablations over training steps, pooling layers, and encoder initializations, and explicitly acknowledges that small objects can be missed because the base T2I model operates at 64x64 resolution.

Significance. If the central claims hold, ViLex offers an appealing unification: a single continuous token sequence in language space that can serve both as a prompt for a frozen text-to-image model and as a vision representation for downstream understanding. The reconstruction comparison against DeDiffusion is well posed, since DeDiffusion uses the same encoder architecture and the same Imagen decoder; the TailDrop mechanism is simple and makes the token budget adaptive; and the supplement contains useful ablations. However, the paper's central 'simultaneous reconstruction and understanding' claim is not yet established: the understanding gains in Section 4.3 are confounded with additional fine-tuning, and the state-of-the-art claim in Table 5 rests on unmatched comparisons. The paper would still be significant as a representation-learning recipe even if the mechanism is not fully isolated, but the current evidence requires revision.

major comments (4)
  1. [§4.3, Table 3 and Supplement Table A1] The headline claim that ViLex improves a strong SigLIP vision encoder across 15 benchmarks is confounded by the amount of additional training. Table 3 compares an untouched official SigLIP checkpoint with a checkpoint that was further trained for 300K or 600K steps on WebLI through a frozen Imagen decoder; the main text in §4.1 says 300K steps, while Supplement Table A1 reports 150K and 600K checkpoints. Supplement Table A1 itself shows that extending the ViLex fine-tuning from 150K to 600K steps raises many downstream scores (e.g., SciQA from 86.1 to 89.3 and RefCOCOg-test from 60.2 to 62.6), so training duration alone is a known confound. A matched control is missing: the same initialization, data, and step budget should be trained with an alternative objective (e.g., continued SigLIP contrastive pretraining or a pixel-space/VAE reconstruction objective) and evaluated with identical hyperparameter sweeps. In addition, the features used for Table 3 are the pre-pooling ViT patch features, not the ViLex text-space tokens; the results therefore establish that ViLex-style fine-tuning does not destroy semantic usefulness, but they do not establish that the language-space/T2I-decoder mechanism is responsible for the reported gains.
  2. [§4.3, Table 5] The claim of 'SOTA performance across multiple visual question answering tasks' is not supported by the comparisons shown. ViLex is evaluated in a frozen-encoder PaliGemma-2B setup with 336 tokens at 224px, whereas the competing models use 7B-13B language models, higher resolutions (336–448px), 576–1280 tokens, and are fine-tuned on the target datasets, as the asterisks in Table 5 indicate. These differences in model scale, input resolution, token budget, and training data dominate the VQA score differences, so no conclusion about state-of-the-art status can be drawn without matched ablations at a fixed scale. The claim should be softened to 'competitive with much larger systems' or supported with controlled comparisons.
  3. [§4.2, Tables 1 and 2] The quantitative reconstruction results are reported without uncertainty estimates. Table 1 gives FID and IS for single runs, and Table 2 reports 91–98% human win rates without stating the number of participants, the number of image-prompt pairs, how stimuli were sampled, or inter-annotator agreement. Since the central 'higher fidelity' claim rests on these numbers, the paper should report standard errors or confidence intervals for FID/IS (e.g., across multiple random subsets or generation seeds) and full human-study statistics. Without these details, the magnitude of the reported margins is difficult to assess.
  4. [Supplement A3, Table A2 and main Tables 1 and 3] The FID values reported for ViLex are mutually inconsistent across tables. Table 3 reports FID 2.38 for ViLex SigLIP, Table 1 reports FID 2.07 for the 75-token ViLex configuration, and Supplement Table A2 reports FID 2.58 for the 5-layer attention-pooling configuration described in the main text. The reconstruction protocol (token count, input resolution, evaluation set, whether the pooling layer is used, and the number of generation seeds) is not specified consistently across these tables, so the reader cannot reconcile the numbers. Please state the exact protocol for each FID column or remove the redundant values.
minor comments (5)
  1. [§4.1 and Supplement A1] The image encoder is described in the main text as 'SigLIP@224 [88] ... ViT-base' but in the supplement as 'SigLIP-So400M@224'; these are different models, so the architecture actually used must be stated consistently.
  2. [§4.1 and Supplement A1] The main text says training runs for 300K steps, while the supplement says doubling from 300K to 600K improves performance and Table A1 evaluates 150K and 600K checkpoints; the main-text tables should identify which checkpoint underlies each reported number.
  3. [§3.1] The symbol k is used both for the number of patch tokens and for the number of tokens dropped by TailDrop; please use distinct notation for these two quantities.
  4. [§4.3, Table 3] The text states that ViLex improves the baseline 'by over 1 point margin on a variety' of tasks, but Table 3 shows several tasks with gains below 1 point (e.g., VQAv2 +0.2, OKVQA +1.3, MSRVTT +1.3); the wording should match the data.
  5. [§4.3, Table 4] The VAE baseline in Table 4 is not described: the table does not state which VAE tokenization is used, how many tokens are fed to the VLM, or whether the VAE encoder is frozen; without this information the '>8 times lower' comparison is hard to interpret.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ViLex's reconstruction objective and downstream evaluations are externally grounded.

full rationale

The paper's central derivation is the ViLex encoder, trained with a denoising reconstruction loss through a frozen Imagen text-to-image model (Eq. 3, Section 4.1). The representation is not defined in terms of the downstream metrics; reconstruction FID is measured on MS-COCO images not used to train the encoder, and the understanding evaluations in Tables 3-5 use frozen ViLex features with only the PaliGemma language model fine-tuned. The same frozen T2I model serves as both training supervisor and evaluation decoder, but this is the intended autoencoder design rather than a circular argument: the encoder must generalize to held-out images, and the text-embedding baseline (DeDiffusion) uses the same decoder for comparison. No load-bearing self-citation appears; the only self-citation is reference [75], which sits in a background list of feature-learning methods and does not justify any central premise. The design choices (diffusion-as-decoder, TailDrop, TFG) are introduced and ablated in the paper rather than justified by prior work of the same authors, so the ansatz-smuggling and uniqueness-import patterns do not apply. The supplement's limitation about missing small objects is an honest acknowledgment of a resolution-dependent failure mode, not circularity. A remaining concern is that Table 3 compares a SigLIP encoder fine-tuned by ViLex for 300K-600K steps against the untouched official SigLIP checkpoint, so the gains could partly reflect continued pretraining rather than the language-space mechanism; this is a correctness or interpretation risk, not a reduction of the result to its inputs by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claim rests on a small set of free parameters (token count, TailDrop k, TFG guidance) and on domain assumptions about the suitability of diffusion models as teachers and of text embedding spaces as a host for visual features. No fundamental physical constants or new entities are introduced. The main scientific risk is that the transferability of the learned representation to understanding tasks is an empirical assumption, not a guaranteed consequence of the reconstruction objective.

free parameters (3)
  • Number of ViLex tokens at inference = 1, 4, 16, 75 (user-selected)
    The token count is dynamically adjusted at inference to trade off semantic richness and detail; the paper reports results for multiple values, but the optimal number is not determined by the method.
  • TailDrop drop count k = not specified (randomly chosen during training)
    The last k visual tokens are randomly dropped during training to encourage earlier tokens to carry semantic information; k is not fixed and is not reported.
  • TFG guidance scale wtfg = not specified
    Controls the trade-off between ViLex tokens and text prompt in multimodal generation; the value is not reported in the main text, and it is a manually chosen hyperparameter.
assumptions (4)
  • domain assumption Frozen T2I diffusion models capture both semantic and visual information in their denoising process.
    The paper relies on Imagen's pretrained knowledge to supervise ViLex; this is argued from prior work (Gan et al., InstructCV; Li et al., zero-shot classifier) but not proven for this setting.
  • domain assumption The text encoder embedding space (CLIP token embeddings) can host continuous multi-token visual representations that are composable with natural language tokens.
    ViLex inserts continuous embeddings into the vocabulary lookup matrix space; the paper assumes these soft tokens will behave like real text tokens for generation and composition.
  • domain assumption The standard diffusion denoising objective is a suitable representation learning loss when the decoder is frozen.
    The method optimizes the reconstruction of clean images from noise conditioned on ViLex tokens; it assumes gradient signal from this loss produces useful features for downstream semantics.
  • ad hoc to paper TailDrop encourages earlier tokens to capture richer semantic information.
    This is a heuristic training strategy introduced by the authors; no formal proof is provided that it induces the intended semantic ordering.
invented entities (1)
  • ViLex token embeddings
    purpose: Continuous image representations placed in the text vocabulary embedding space, used as prompts for T2I models and as vision features for VLMs.
    The tokens are learned and evaluated only within the paper's pipeline; no external or independent validation of the token space is provided, and they are not a new physical entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Visual Lexicon: Rich Image Features in Language Space." pith.science (2026). https://pith.science/paper/QDUTESFM

@misc{pith2026241206774,
  author       = {Pith},
  title        = {Pith review of: Visual Lexicon: Rich Image Features in Language Space},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QDUTESFM}},
  note         = {Machine review of arXiv:2412.06774}
}
read the original abstract

We present Visual Lexicon, a novel visual language that encodes rich image information into the text space of vocabulary tokens while retaining intricate visual details that are often challenging to convey in natural language. Unlike traditional methods that prioritize either high-level semantics (e.g., CLIP) or pixel-level reconstruction (e.g., VAE), ViLex simultaneously captures rich semantic content and fine visual details, enabling high-quality image generation and comprehensive visual scene understanding. Through a self-supervised learning pipeline, ViLex generates tokens optimized for reconstructing input images using a frozen text-to-image (T2I) diffusion model, preserving the detailed information necessary for high-fidelity semantic-level reconstruction. As an image embedding in the language space, ViLex tokens leverage the compositionality of natural languages, allowing them to be used independently as "text tokens" or combined with natural language tokens to prompt pretrained T2I models with both visual and textual inputs, mirroring how we interact with vision-language models (VLMs). Experiments demonstrate that ViLex achieves higher fidelity in image reconstruction compared to text embeddings--even with a single ViLex token. Moreover, ViLex successfully performs various DreamBooth tasks in a zero-shot, unsupervised manner without fine-tuning T2I models. Additionally, ViLex serves as a powerful vision encoder, consistently improving vision-language model performance across 15 benchmarks relative to a strong SigLIP baseline.

Figures

Figures reproduced from arXiv: 2412.06774 by the authors.

Figure 1
Figure 1. Given the cute corgi painting in the top left corner, how can we extract a visual representation that captures semantic-level information – such as object categories and layouts – while preserving rich visual details like image styles, textures and colors? We introduce ViLex model that generates image representations in the text vocabulary space, acting as a new visual “language”, while retaining intricate visual de… view at source ↗
Figure 2
Figure 2. top) ViLex empowers linguistic space to capture vi [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The pipeline of ViLex: We learn a Visual Lexicon from a frozen diffusion model using an image reconstruction loss. After training, ViLex can be directly used as the “text-prompt” to a frozen text encoder, e.g., CLIP or T5, enabling the re-creation of semantically similar images without the need for actual text prompts. In addition, during training, we implement the TailDrop strategy, where the last k tokens are rand… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: ViLex retains more visual details in image-to-image generation compared to DALL·E 3 [5] and DeDiffusion [76], accurately capturing elements such as image style (e.g., the oil painting style in row 1), layout (e.g., the relative position of the corgi and the lighthouse)…
Figure 5
Figure 5. Figure 5: ViLex can be seamlessly integrated with natural language prompts for zero-shot unsupervised image re-contextualization using a frozen text-to-image (T2I) diffusion model. Unlike DreamBooth [58], ViLex requires no fine-tuning of the T2I model on a set of input images fr…
Figure 6
Figure 6. Figure 6: ViLex can also support zero-shot unsupervised art ren￾dition via prompting T2I models with ViLex and text prompts. and an input resolution of 64×64. The text encoder of Ima￾gen is OpenCLIP ViT-H/14 [13, 32] with a vocabulary size of 49408. The U-Net conditions on text …
Figure 7
Figure 7. Figure 7: Qualitative results of semantic-level image recon [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Vision-Language-Vision Auto-Encoder: Scalable Knowledge Distillation from Diffusion Models

    cs.CV 2025-07 conditional novelty 5.0 of 10

    An image autoencoder compresses pictures into caption-like embeddings via a frozen diffusion decoder, and a fine-tuned LLM reads those embeddings into captions claimed to rival GPT-4o at under $1,000 training cost.

Reference graph

Works this paper leans on

89 extracted references · 47 canonical work pages · cited by 1 Pith paper

  1. [1]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv:2308.12966, 2023. 8, 9

    arXiv 2023

  2. [2]

    BEit: BERT pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In ICLR, 2022. 3

    2022

  3. [3]

    Label-efficient se- mantic segmentation with diffusion models

    Dmitry Baranchuk, Andrey V oynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient se- mantic segmentation with diffusion models. In ICLR, 2022. 2, 3

    2022

  4. [4]

    Generalized denoising auto-encoders as generative models

    Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vin- cent. Generalized denoising auto-encoders as generative models. Advances in neural information processing systems, 26, 2013. 2, 3

    2013

  5. [5]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023. 2, 5, 7

    2023

  6. [6]

    Paligemma: A versatile 3b vlm for trans- fer

    Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexan- der Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for trans- fer. arXiv:2407.07726, 2024. 2, 3, 8, 13, 14

    arXiv 2024

  7. [7]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 4

    arXiv 2005

  8. [8]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In CVPR, 2021. 1, 3

    2021

Show all 89 references
  1. [9]

    Generative pre- training from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Hee- woo Jun, David Luan, and Ilya Sutskever. Generative pre- training from pixels. In International conference on machine learning, pages 1691–1703. PMLR, 2020. 3

    2020

  2. [10]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv:1504.00325, 2015. 2

    2015 arXiv

  3. [11]

    Pali: A jointly-scaled multilingual language-image model

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergio- vanni, Piotr Padlewski, Daniel Salz, Sebastian Good- man, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv:2209.06794, 2022. 6, 13

    2022 arXiv

  4. [12]

    De- constructing denoising diffusion models for self-supervised learning

    Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. De- constructing denoising diffusion models for self-supervised learning. arXiv:2401.14404, 2024. 3

    2024 arXiv

  5. [13]

    Reproducible scal- ing laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- ing laws for contrastive language-image learning. In CVPR,

  6. [14]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 1, 3

    2009

  7. [15]

    Large scale adversarial representation learning

    Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. NeurIPS, 32, 2019. 3

    2019

  8. [16]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  9. [17]

    A new algorithm for data compression

    Philip Gage. A new algorithm for data compression. The C Users Journal, 12(2):23–38, 1994. 4

    1994

  10. [18]

    An image is worth one word: Personalizing text-to-image gen- eration using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image gen- eration using textual inversion. ICLR, 2023. 3, 16 9

    2023

  11. [19]

    Instructcv: Instruction- tuned text-to-image diffusion models as vision generalists

    Yulu Gan, Sungwoo Park, Alexander Marcel Schubert, An- thony Philippakis, and Ahmed Alaa. Instructcv: Instruction- tuned text-to-image diffusion models as vision generalists. In The Twelfth International Conference on Learning Repre- sentations, 2024. 2, 3

    2024

  12. [20]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023. 3

    2023

  13. [21]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In CVPR, 2017. 2, 13

    2017

  14. [22]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  15. [23]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 1, 2, 3

    2022

  16. [24]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 7

    2017

  17. [25]

    Autoencoders, mini- mum description length and helmholtz free energy.Advances in neural information processing systems, 6, 1993

    Geoffrey E Hinton and Richard Zemel. Autoencoders, mini- mum description length and helmholtz free energy.Advances in neural information processing systems, 6, 1993. 2, 3, 8

    1993

  18. [26]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. 4

    2021

  19. [27]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 2, 5

    2020

  20. [28]

    Scicap: Generating captions for scientific figures

    Ting-Yao Hsu, C Lee Giles, and Ting-Hao’Kenneth’ Huang. Scicap: Generating captions for scientific figures. arXiv preprint arXiv:2110.11624, 2021. 13

    2021 arXiv

  21. [29]

    Lora: Low-rank adaptation of large language mod- els

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language mod- els. arXiv:2106.09685, 2021. 5, 6, 7, 16

    2021 arXiv

  22. [30]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 14

    2019

  23. [31]

    Soda: Bottleneck diffusion models for representation learning

    Drew A Hudson, Daniel Zoran, Mateusz Malinowski, An- drew K Lampinen, Andrew Jaegle, James L McClelland, Loic Matthey, Felix Hill, and Alexander Lerchner. Soda: Bottleneck diffusion models for representation learning. In CVPR, 2024. 3

    2024

  24. [32]

    Open- clip, 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Open- clip, 2021. 6, 13

    2021

  25. [33]

    Referitgame: Referring to objects in pho- tographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in pho- tographs of natural scenes. In Proceedings of the 2014 con- ference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014. 14

    2014

  26. [34]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 1, 2, 3, 8

    2013 arXiv

  27. [35]

    Your diffusion model is secretly a zero-shot classifier

    Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 2206–2217,

  28. [36]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv:2408.03326, 2024. 3

    2024 arXiv

  29. [37]

    Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. In ICLR. PMLR, 2022. 3

    2022

  30. [38]

    Imagefolder: Autoregressive im- age generation with folded tokens

    Xiang Li, Hao Chen, Kai Qiu, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregressive im- age generation with folded tokens. arXiv:2410.01756, 2024. 3

    2024 arXiv

  31. [39]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR, 2024. 9

    2024

  32. [40]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 13, 15

    2014

  33. [41]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR,

  34. [42]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 8, 9

    2024

  35. [43]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507–2521,

  36. [44]

    Prompting hard or hardly prompting: Prompt inversion for text-to-image diffusion models

    Shweta Mahajan, Tanzila Rahman, Kwang Moo Yi, and Leonid Sigal. Prompting hard or hardly prompting: Prompt inversion for text-to-image diffusion models. In CVPR,

  37. [45]

    Generation and comprehension of unambiguous object descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016. 14

    2016

  38. [46]

    Ok-vqa: A visual question answering 10 benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering 10 benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. 13

    2019

  39. [47]

    Finite scalar quantization: Vq-vae made simple

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. ICLR, 2024. 3

    2024

  40. [48]

    Null-text inversion for editing real im- ages using guided diffusion models

    Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real im- ages using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038–6047, 2023. 3

    2023

  41. [49]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML. PMLR,

  42. [50]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv:2304.07193, 2023. 1, 3

    2023 arXiv

  43. [51]

    Styleclip: Text-driven manipulation of stylegan imagery

    Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 2085–2094,

  44. [52]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth Inter- national Conference on Learning Representations, 2023. 3, 5

    2023

  45. [53]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICLR, 2021. 1, 2, 3, 4, 6, 13

    2021

  46. [54]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 2020. 2, 4

    2020

  47. [55]

    Hierarchical text-conditional image gen- eration with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gen- eration with clip latents. arXiv preprint arXiv:2204.06125,

    arXiv

  48. [56]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, 2022. 2, 3, 8

    2022

  49. [57]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015. 5, 13

    2015

  50. [58]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023. 2, 3, 6, 7, 16

    2023

  51. [59]

    Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models. InCVPR, 2024. 3, 16

    2024

  52. [60]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  53. [61]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2021. 13

    2021

  54. [62]

    Neural machine translation of rare words with subword units

    Rico Sennrich. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015. 4

    2015 arXiv

  55. [63]

    Adafactor: Adaptive learning rates with sublinear memory cost

    Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In ICML. PMLR,

  56. [64]

    Textcaps: a dataset for image caption- ing with reading comprehension

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image caption- ing with reading comprehension. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 742–758. Springer,

    2020

  57. [65]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 13

    2019

  58. [66]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML. PMLR, 2015. 5

    2015

  59. [67]

    Generative modeling by esti- mating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by esti- mating gradients of the data distribution. Advances in neural information processing systems, 32, 2019. 5

    2019

  60. [68]

    Rethinking the inception archi- tecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception archi- tecture for computer vision. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 2818–2826, 2016. 7

    2016

  61. [69]

    Gemma: Open models based on gemini research and tech- nology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi `ere, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and tech- nology. arXiv preprint arXiv:2403.08295, 2024. 8

    2024 arXiv

  62. [70]

    Crossmodal-3600: A massively multilingual multi- modal evaluation dataset

    Ashish V Thapliyal, Jordi Pont-Tuset, Xi Chen, and Radu Soricut. Crossmodal-3600: A massively multilingual multi- modal evaluation dataset. arXiv preprint arXiv:2205.12522,

    arXiv

  63. [71]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Li- wei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. NeurIPS, 2024. 3

    2024

  64. [72]

    Con- trastive multiview coding

    Yonglong Tian, Dilip Krishnan, and Phillip Isola. Con- trastive multiview coding. In ECCV, 2020. 3

    2020

  65. [73]

    Extracting and composing robust features with denoising autoencoders

    Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 11 25th international conference on Machine learning , pages 1096–1103, 2008. 2, 3

    2008

  66. [74]

    Diffusion feedback helps clip see better

    Wenxuan Wang, Quan Sun, Fan Zhang, Yepeng Tang, Jing Liu, and Xinlong Wang. Diffusion feedback helps clip see better. arXiv:2407.20171, 2024. 3

    2024 arXiv

  67. [75]

    Unsupervised feature learning by cross-level instance-group discrimina- tion

    Xudong Wang, Ziwei Liu, and Stella X Yu. Unsupervised feature learning by cross-level instance-group discrimina- tion. In CVPR, 2021. 3

    2021

  68. [76]

    De-diffusion makes text a strong cross- modal interface

    Chen Wei, Chenxi Liu, Siyuan Qiao, Zhishuai Zhang, Alan Yuille, and Jiahui Yu. De-diffusion makes text a strong cross- modal interface. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 13492–13503, 2024. 2, 3, 5, 6, 7, 13

    2024

  69. [77]

    Unsupervised feature learning via non-parametric instance discrimination

    Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018. 3

    2018

  70. [78]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016. 2, 14, 16

    2016

  71. [79]

    Open-vocabulary panop- tic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiao- long Wang, and Shalini De Mello. Open-vocabulary panop- tic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition , pages 2955–2966, 2023. 2, 3

    2023

  72. [80]

    Diffusion model as repre- sentation learner

    Xingyi Yang and Xinchao Wang. Diffusion model as repre- sentation learner. In ICCV, 2023. 3

    2023

  73. [81]

    Vector-quantized image modeling with improved vqgan

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. ICLR, 2022. 3

    2022

  74. [82]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022. 4, 15, 16

    2022

  75. [83]

    Modeling context in referring expres- sions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. In ECCV, 2016. 2, 14

    2016

  76. [84]

    Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms

    Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolf- gang Macherey, Yanping Huang, David Ross, Irfan Essa, Yonatan Bisk, Ming-Hsuan Yang, et al. Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms. NeurIPS, 2024. 3

    2024

  77. [85]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, Jos ´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. ICLR,

  78. [86]

    An image is worth 32 tokens for reconstruction and generation

    Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. NeurIPS, 2024. 1, 3

    2024

  79. [87]

    Soundstream: An end- to-end neural audio codec

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end- to-end neural audio codec. IEEE/ACM Transactions on Au- dio, Speech, and Language Processing, 2021. 5

    2021

  80. [88]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In CVPR, 2023. 2, 3, 6, 8, 13, 15, 16

    2023

  81. [89]

    Method A

    Yue Zhao, Yuanjun Xiong, and Philipp Kr ¨ahenb¨uhl. Image and video tokenization with binary spherical quantization. arXiv:2406.07548, 2024. 3 12 Visual Lexicon: Rich Image Features in Language Space Supplementary Material A1. Technical Details We introduced the main technical...

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.