Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractor

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pre-trained text-to-image diffusion models can serve as question-aware visual encoders that supply the fine-grained detail CLIP misses.

desk verdict Interesting empirical study of diffusion features for MLLMs, but the load-bearing question-conditioned pipeline is never tested for prompt leakage, so the headline gains are provisional. read the letter →

arxiv 2507.07106 v1 pith:SKN4OV52 submitted 2025-07-09 cs.CV cs.LG

Vatsal Agarwal , Matthew Gwilliam , Gefen Kohavi , Eshan Verma , Daniel Ulbricht , Abhinav Shrivastava This is my paper
classification cs.CVcs.LG
keywords multimodallargelanguagemodelsdiffusionasvisualencoderstext-to-imageinstruction-awarefeaturescross-attentionmapspromptleakagequestionansweringvision-centricreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a pre-trained text-to-image diffusion model, used as a frozen visual encoder, can give a multimodal large language model (MLLM) the fine-grained, compositional visual information that CLIP-based encoders miss. The authors show that conditioning the diffusion model on the input question focuses its internal features on task-relevant regions, and that fusing these conditional features with CLIP improves vision-centric question answering: the paper reports +7 points on MMVP and +0.9 points on NaturalBench over the LLaVA-v1.5 baseline. The work also identifies a failure mode called prompt leakage, where the language model learns to decode the text prompt from the diffusion features, and proposes caption dropout during training as a mitigation. If correct, this opens a simple route to stronger MLLMs by reusing generative models as task-aware feature extractors.

What carries the argument

The mechanism that carries the argument is the text-conditioned cross-attention module inside the diffusion U-Net, specifically the pixel-wise query features (Cross-Q) extracted at up-stage blocks such as U-L1-R1-B0-Cross-Q at a 16x16 resolution and timestep around 50. These features directly interact with the text embedding, and the paper shows that amplifying the text guidance via $X_{\text{amp}} = X_{\text{uncond}} + s(X_{\text{cond}} - X_{\text{uncond}})$ increasingly focuses the representation on prompt-relevant image regions, measured by CKA similarity and PCA visualizations. For fusion, CLIP features act as queries over diffusion features as keys and values in a cross-attention layer, letting the LLM pull task-relevant spatial details from the diffusion encoder while keeping CLIP's global semantic alignment. The leakage analysis uses mismatched image-text pairs and CIDEr/SPICE scores to quantify how much of the prompt the LLM can decode from the features, and caption dropout is the proposed mitigation.

What would settle it

Ask the final CLIP + question-conditioned diffusion model the same MMVP question paired with a randomly chosen, unrelated image. If accuracy stays near the reported 30.7 rather than collapsing to chance, the answer is being recovered from the question prompt embedded in the diffusion features, refuting the claim that the gain comes from improved visual understanding.

Watch

Extended reading notes

Core claim

The paper's central claim is that the cross-attention features of a text-to-image diffusion model (Stable Diffusion v2.1-base) are semantically rich, encode fine-grained image-text alignment, and can be steered by the text prompt so that the visual representation highlights regions relevant to the user's question. Treating the diffusion model as a frozen, instruction-aware encoder, the authors align its features to an LLM through the LLaVA training pipeline and evaluate the resulting MLLM on general and vision-centric benchmarks. They report that the best configuration—CLIP features as queries in a cross-attention layer over conditional diffusion features—improves MMVP accuracy by 7 points and NaturalBench accuracy by 0.9 points over the LLaVA-v1.5 baseline. A key part of the discovery is the leakage phenomenon: when trained with captions as the text condition, the LLM can recover the original prompt from the diffusion features (up to a CIDEr of 12.97 on mismatched image-text pairs), and this leakage is mitigated by randomly dropping the text condition during pretraining. The paper concludes that diffusion models are promising as complementary, query-aware visual encoders, especially for spatial and compositional reasoning.

Load-bearing premise

The claim's VQA gains assume that the prompt leakage demonstrated with caption prompts does not also let the language model answer from question text hidden inside the diffusion features.

Editorial extensions

If this is right

  • Frozen diffusion encoders can be dropped into existing MLLM pipelines to improve spatial, compositional, and part-level visual reasoning without changing the LLM.
  • Text conditioning at guidance scale s=4 yields better task-aware features than unconditional features, suggesting that prompt-aware encoding, not just richer features, drives the gains.
  • Prompt leakage is a real failure mode when conditioning an encoder on text, and caption dropout is a simple, effective mitigation during pretraining.
  • MLLMs built on CLIP plus conditional diffusion features beat the CLIP-only baseline on MMVP and NaturalBench while staying competitive on general VQA benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An open question the paper leaves implicit: the leakage probe is run with caption prompts only, so the reported VQA gains could partly reflect the LLM decoding the question from question-conditioned features; rerunning the mismatched image-text protocol with questions would settle this.
  • The guidance-scale interpolation suggests ensembling features across multiple values of s (e.g., 0, 1.5, 4) could push performance further, an idea the authors mention only as future work.
  • Because the paper shows SDXL exhibits even stronger text-dependence than SD2.1, the leakage-vs-task-awareness trade-off is likely a general property of text-to-image models, so any instruction-conditioned diffusion encoder should be screened for leakage.
  • The 'register' tokens observed in output features resemble global descriptors found in self-supervised ViTs, hinting that diffusion output features could serve a dual role: global image descriptor plus fine-grained local features, but the paper does not exploit this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes using a frozen text-to-image diffusion model (Stable Diffusion v2.1-base) as an instruction-aware visual encoder for multimodal LLMs. The authors analyze diffusion features across blocks and timesteps, quantify cross-attention alignment on MMVP-VLM and Winoground, identify a prompt-leakage phenomenon in caption-conditioned features, propose caption dropout as a mitigation, and finally fuse CLIP with question-conditioned diffusion features in a LLaVA-1.5 framework, reporting improvements on MMVP and NaturalBench in Table 2. The paper is primarily empirical and includes substantial appendix material on block configurations, training hyperparameters, per-timestep results, and SDXL comparisons.

Significance. If the results hold, the paper makes a useful empirical case that diffusion representations carry complementary fine-grained, compositional, and question-relevant visual information that CLIP lacks, and it documents an important failure mode (prompt leakage) that any such approach must address. The strengths are the breadth of the analysis across blocks, timesteps, guidance scales, and cross-attention maps; the honest documentation of leakage and a concrete mitigation; and the reproducible training setup described in the appendix. The significance is conditional on closing the question-conditioned leakage gap and on showing that the reported gains are not artifacts of benchmark-driven configuration selection.

major comments (3)
  1. [§5.2, Table 2 and §4.3] The final pipeline conditions the diffusion model on the user's question at s=4 during SFT (Sec 5.2), but the leakage analysis in Sec 4.3 is restricted to caption prompts. The mismatched-image experiment shows that with caption conditioning at s=4 the LLM can recover the prompt text from diffusion features (CIDEr 12.97 for SD2.1 and 63.48 for SDXL in Table 9), and caption dropout only mitigates this. Because MMVP and NaturalBench questions contain the very concepts that must be verified visually, a question-conditioned model that leaks the prompt could answer from language alone, making the reported MMVP and NaturalBench gains in Table 2 reflect a shortcut rather than improved visual encoding. The authors need to run the mismatched-image test with question prompts (e.g., random image paired with a target question) and report whether the model can recover the question or answer it without the correct image.
  2. [§3.2, Fig 3 and §5.2, Table 2] The main quantitative claims rely on configurations selected on the evaluation benchmarks themselves. The block/timestep choice U-L1-R1-B0-Cross-Q at T=50 is chosen after comparing performance on MMVP, BLINK-val, and NaturalBench in Fig 3, and the guidance scale s=4 is chosen after inspecting Table 2 on the same benchmarks. This selection-dependent reporting can inflate the reported gains. The paper should either fix the configuration on a held-out validation set before evaluating on the test benchmarks, or present the full configuration grid with a multiple-comparison-corrected significance statement.
  3. [Table 2] No error bars or significance tests are reported for the central fusion results. Several of the reported improvements are small (NaturalBench G-Acc 15.26 vs 14.32, Q-Acc 40.71 vs 37.70, with GQA actually lower at 62.5 vs 62.7), so a single run does not establish that these gains are reliable. Please report multiple seeds or variance estimates for Table 2, and correct the headline '+7 points' to '+6.0 points' to match the table (30.7 vs 24.7).
minor comments (6)
  1. [Abstract and §5.2] The text states that MMVP improves by '+7 points' and the abstract says '+6%', but Table 2 shows 30.7 vs 24.7, which is +6.0 points; please reconcile these numbers.
  2. [§3.2] The sentence 'We evaluate using a two sets of benchmarks' contains a typo and should read 'two sets'.
  3. [Table 5] The entry '31..48' appears to be a typo for '31.48'.
  4. [§4.1] The image-text matching score is computed directly from the same cross-attention maps being evaluated, via LogSumExp pooling; this is a legitimate proxy but should be explicitly flagged as less independent than a separate scoring model.
  5. [Table 1] The comparison in Table 1 is not fully apples-to-apples: the SD cross-attention method uses 512×512 images and an ensemble of five timesteps, while the CLIP models use a single forward pass at 224 or 336 resolution; please state this clearly and avoid the claim of outperforming 'CLIP-based models across all benchmarks' when DFN-CLIP achieves a higher MMVP-VLM average (39.3 vs 34.6).
  6. [Table 2] The column header 'Config LLaVA-B MMVP GQA Natural-Bench All Acc Acc Q-Acc I-Acc G-Acc' is difficult to parse; please clarify which columns correspond to which benchmark and metric.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the central claim is validated on external benchmarks and no load-bearing step reduces to its own inputs by construction.

full rationale

The paper's central claim, that fusing CLIP with question-conditioned Stable Diffusion features improves MLLM accuracy, is tested against the LLaVA-v1.5 baseline on external benchmarks (MMVP, NaturalBench, BLINK, Winoground); none of these benchmarks is defined in terms of the method's own outputs, so the result is not circular. The Sec 4.1 image-text matching evaluation uses cross-attention maps as a scoring function, but correctness is judged by MMVP-VLM and Winoground, making the comparison falsifiable rather than definitional. The layer and timestep choices are supported by the paper's own PCA/CKA/benchmark sweeps; this is analysis-driven design, not a fitted parameter renamed as a prediction. Citations to the authors' prior work ([26], [51]) are background statements about repurposing diffusion features and are not load-bearing for the reported gains. The remaining concern flagged in the reader's take and in Sec 4.3/6, namely that the final question-conditioned pipeline is not explicitly tested for prompt leakage, is a validity threat, not a circularity: it would require an empirical demonstration that the LLM recovers question text from diffusion features, which the paper does not provide. No step in the derivation chain is equivalent to its inputs by construction.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central experiments rest on hand-selected configuration values (guidance scale, timestep, block, and ensemble), none of which are derived from first principles, and on the assumption that question conditioning does not leak answer-relevant text into the LLM. No invented entities are introduced.

free parameters (5)
  • Guidance scale s = 4
    Selected by comparing s=0, 1.5, 4 on the same benchmarks (Table 2, Figs. 5-6); higher s increases text modulation but also leakage.
  • Timestep T = 50
    Chosen after scanning T in {10,50,100,250,500} on LLaVA-Bench, MMVP, NaturalBench, BLINK (Fig. 3).
  • Block configuration = U-L1-R1-B0-Cross-Q
    Selected because it 'exhibits good unconditional performance and also experiences reasonable text-modulation' (Sec 4.3); chosen after evaluating six block configs (Table 3, Fig. 3).
  • Timestep ensemble for cross-attention matching = t in {189,389,589,789,989}
    Ensemble of five timesteps chosen from per-timestep MMVP/Winoground results (Tables 6-7); used for Table 1 matching scores.
  • Caption dropout rate = not specified
    Dropout is introduced in Sec 4.3 as leakage mitigation, but the probability of masking the caption is not given, making the mitigation unreproducible.
assumptions (3)
  • domain assumption Stable Diffusion v2.1-base provides sufficiently rich intermediate features for downstream LLM alignment.
    The entire method relies on SD2.1 features as the visual representation; this is assumed rather than derived.
  • domain assumption The 16x16 spatial resolution of extracted features preserves task-relevant detail.
    All features are resized to 16x16 (256 tokens) for the LLM; the paper does not show that higher resolutions improve these results, beyond one qualitative SDXL comparison.
  • domain assumption The question text passed to the diffusion model at inference does not leak answer information into the features in the VQA setting.
    Leakage is measured only with captions (Sec 4.3); the final pipeline conditions on questions (Sec 5.2) without a corresponding leak test.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractor." pith.science (2026). https://pith.science/paper/SKN4OV52

@misc{pith2026250707106,
  author       = {Pith},
  title        = {Pith review of: Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractor},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SKN4OV52}},
  note         = {Machine review of arXiv:2507.07106}
}
read the original abstract

Recent advances in multimodal large language models (MLLMs) have enabled image-based question-answering capabilities. However, a key limitation is the use of CLIP as the visual encoder; while it can capture coarse global information, it often can miss fine-grained details that are relevant to the input query. To address these shortcomings, this work studies whether pre-trained text-to-image diffusion models can serve as instruction-aware visual encoders. Through an analysis of their internal representations, we find diffusion features are both rich in semantics and can encode strong image-text alignment. Moreover, we find that we can leverage text conditioning to focus the model on regions relevant to the input question. We then investigate how to align these features with large language models and uncover a leakage phenomenon, where the LLM can inadvertently recover information from the original diffusion prompt. We analyze the causes of this leakage and propose a mitigation strategy. Based on these insights, we explore a simple fusion strategy that utilizes both CLIP and conditional diffusion features. We evaluate our approach on both general VQA and specialized MLLM benchmarks, demonstrating the promise of diffusion models for visual understanding, particularly in vision-centric tasks that require spatial and compositional reasoning. Our project page can be found https://vatsalag99.github.io/mustafar/.

Figures

Figures reproduced from arXiv: 2507.07106 by the authors.

Figure 1
Figure 1. Overview. (Left) We present our full multimodal pipeline. Following LLaVA, we first extract visual features from the frozen diffusion model and pass the question as text-prompt. The LLM then uses these features to generate its answer. Cross-attention maps show that the model can use the question to focus on relevant regions (Right) We show examples on MMVP [1] where diffusion features outperform CLIP. We first analy… view at source ↗
Figure 2
Figure 2. Inspecting Diffusion Features. (a, b) We visualize spatial features for three image pairs from MMVP-VLM using PCA across blocks and timesteps. For (a), we use T=50; for (b), we fix U-L1-R1-B0-Cross-Q. We observe: (1) different blocks capture either shared semantics or image-specific details; (2) higher timesteps encode coarse layout, while lower timesteps emphasize fine-grained structure; and (3) features like out a… view at source ↗
Figure 3
Figure 3. General Model Performance (Left): We evaluate multimodal reasoning using the LLaVA framework with diffusion features at different layers and timesteps. The table reports accuracy on LLaVA-Bench, MMVP, and NaturalBench under varying feature extraction points. BLINK-val Performance (Right): The plot shows BLINK-val benchmark performance across different timesteps. SD-based models consistently outperform CLIP (in black… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Visualizing Cross-Attention Maps. We show a sample from COCO-captions and averaged cross-attention maps at low and high timesteps for a few key words, representing the attention between pixel features and a specific word. We observe that cross-attention maps at higher …
Figure 5
Figure 5. Figure 5: Visualizing Text Conditioned Diffusion Features. (a) A few images are sampled from the MMVP-VLM dataset and visualize PCA maps of spatial features extracted from different intermediate layers under both unconditional and text-conditioned settings (t = 50). While the ov…
Figure 6
Figure 6. Figure 6: COCO-Captions Performance (Left): We compare model performance on COCO￾Captions across different pretraining text-guidance settings. Models trained with stronger ground￾truth conditioning outperform the CLIP baseline when given ground-truth captions at inference but de…
Figure 7
Figure 7. Figure 7: Visualizing Question-Conditioned Features. We sample an image with two ques￾tions from NaturalBench and visualize spatial features conditioned on questions via PCA. We see that the difference between conditional and unconditional features finds the relevant regions. Co…
Figure 8
Figure 8. Figure 8: More Diffusion Feature Visualizations. We sample six pairs of images from the Natural￾Bench benchmark [57] and view the joint PCA maps across different blocks and layers. Please zoom in to see more details. C.2 CKA Analysis Here, we aim to understand the relationship b…
Figure 9
Figure 9. Figure 9: CKA Block-Wise Similarity. We compute block-wise CKA similarity using the COCO￾Captions test set (5000 images) across various guidance scales (s = 0, 1, 4) ing on text, we observe minimal changes in feature similarities. However, increasing text guidance (s = 4), we se…
Figure 10
Figure 10. Figure 10: More Cross-Attention Maps. We display more examples of cross-attention maps from images in the COCO-Captions test set. For the image of the skier, we can see that object-attribute binding such as“black” and “coat” are better aligned at later timesteps compared to earl…
Figure 11
Figure 11. Figure 11: Cross-Attention Maps Across Layers. We display cross-attention maps at timestep 50 across various layers. We see that cross-attention maps are not uniform and that maps at the up-stage encode more robust image-text alignment. one set of maps is not necessarily better …
Figure 12
Figure 12. Figure 12: More Visualizations of Question-Conditioned Features. Please zoom in to see smaller regions of focus (e.g., pencil in second row, rightmost column of U-L2-R1-B0-Cross-Q) F Investigating SDXL Architecture [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LAP: Fast LAtent Diffusion Planner for Autonomous Driving

    cs.RO 2025-11 conditional novelty 6.0 of 10

    LAP plans trajectories in a VAE-learned latent space with one- or two-step latent diffusion, beating prior learning-based planners on nuPlan hard scenarios with up to ~10x lower inference latency.

Reference graph

Works this paper leans on

70 extracted references · 41 canonical work pages · cited by 1 Pith paper

  1. [1]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms,

    S. Tong, Z. Liu, Y . Zhai, Y . Ma, Y . LeCun, and S. Xie, “Eyes wide shut? exploring the visual shortcomings of multimodal llms,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568–9578, 2024

    work page 2024

  2. [2]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark,et al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning, pp. 8748–8763, PMLR, 2021

    work page 2021

  3. [3]

    Detailclip: Detail-oriented clip for fine-grained tasks,

    A. K. Monsefi, K. P. Sailaja, A. Alilooee, S.-N. Lim, and R. Ramnath, “Detailclip: Detail-oriented clip for fine-grained tasks,”arXiv preprint arXiv:2409.06809, 2024

    arXiv 2024

  4. [4]

    Is clip the main roadblock for fine-grained open-world perception?,

    L. Bianchi, F. Carrara, N. Messina, and F. Falchi, “Is clip the main roadblock for fine-grained open-world perception?,” 2024

    work page 2024

  5. [5]

    Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learners

    X. He, W. Feng, T.-J. Fu, V . Jampani, A. Akula, P. Narayana, S. Basu, W. Y . Wang, and X. E. Wang, “Discffusion: Discriminative diffusion models as few-shot vision and language learners,”arXiv preprint arXiv:2305.10722, 2023

    work page Pith review arXiv 2023

  6. [6]

    Brave: Broadening the visual encoding of vision-language models,

    O. F. Kar, A. Tonioni, P. Poklukar, A. Kulshrestha, A. Zamir, and F. Tombari, “Brave: Broadening the visual encoding of vision-language models,”arXiv preprint arXiv:2404.07204, 2024

    arXiv 2024

  7. [7]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms,

    S. Tong, E. Brown, P. Wu, S. Woo, M. Middepogu, S. C. Akula, J. Yang, S. Yang, A. Iyer, X. Pan, et al., “Cambrian-1: A fully open, vision-centric exploration of multimodal llms,”arXiv preprint arXiv:2406.16860, 2024

    arXiv 2024

  8. [8]

    From clip to dino: Visual encoders shout in multi-modal large language models,

    D. Jiang, Y . Liu, S. Liu, X. Zhang, J. Li, H. Xiong, and Q. Tian, “From clip to dino: Visual encoders shout in multi-modal large language models,”arXiv preprint arXiv:2310.08825, 2023

    arXiv 2023

Show all 70 references
  1. [9]

    Mini-gemini: Mining the potential of multi-modality vision language models,

    Y . Li, Y . Zhang, C. Wang, Z. Zhong, Y . Chen, R. Chu, S. Liu, and J. Jia, “Mini-gemini: Mining the potential of multi-modality vision language models,” 2024

    2024

  2. [10]

    Prismer: A vision-language model with multi-task experts,

    S. Liu, L. Fan, E. Johns, Z. Yu, C. Xiao, and A. Anandkumar, “Prismer: A vision-language model with multi-task experts,”arXiv preprint arXiv:2303.02506, 2023

    2023 arXiv

  3. [11]

    Vcoder: Versatile vision encoders for multimodal large language models,

    J. Jain, J. Yang, and H. Shi, “Vcoder: Versatile vision encoders for multimodal large language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27992–28002, 2024

    2024

  4. [12]

    Question aware vision transformer for multimodal reasoning,

    R. Ganz, Y . Kittenplon, A. Aberdam, E. B. Avraham, O. Nuriel, S. Mazor, and R. Litman, “Question aware vision transformer for multimodal reasoning,” 2024

    2024

  5. [13]

    Api: Attention prompting on image for large vision-language models,

    R. Yu, W. Yu, and X. Wang, “Api: Attention prompting on image for large vision-language models,” 2024

    2024

  6. [14]

    Instructblip: Towards general-purpose vision-language models with instruction tuning,

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision-language models with instruction tuning,” 2023

    2023

  7. [15]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” inInternational conference on machine learning, pp. 19730–19742, PMLR, 2023

    2023

  8. [16]

    High-resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684–10695, 2022

    2022

  9. [17]

    Photorealistic text-to-image diffusion models with deep language understanding,

    C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi, “Photorealistic text-to-image diffusion models with deep language understanding,” inAdvances in Neural Information...

    2022

  10. [18]

    Hierarchical text-conditional image generation with clip latents,

    A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” 2022

    2022

  11. [19]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis,

    D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach, “Sdxl: Improving latent diffusion models for high-resolution image synthesis,” 2023

    2023

  12. [20]

    Prompt-to-prompt image editing with cross attention control,

    A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y . Pritch, and D. Cohen-Or, “Prompt-to-prompt image editing with cross attention control,” 2022. 11

    2022

  13. [21]

    What the daam: Interpreting stable diffusion using cross attention,

    R. Tang, L. Liu, A. Pandey, Z. Jiang, G. Yang, K. Kumar, P. Stenetorp, J. Lin, and F. Ture, “What the daam: Interpreting stable diffusion using cross attention,” 2022

    2022

  14. [22]

    Towards understanding cross and self-attention in stable diffusion for text-guided image editing,

    B. Liu, C. Wang, T. Cao, K. Jia, and J. Huang, “Towards understanding cross and self-attention in stable diffusion for text-guided image editing,” 2024

    2024

  15. [23]

    Plug-and-play diffusion features for text-driven image- to-image translation,

    N. Tumanyan, M. Geyer, S. Bagon, and T. Dekel, “Plug-and-play diffusion features for text-driven image- to-image translation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1921–1930, 2023

    1921

  16. [24]

    Diffusion model is secretly a training-free open vocabulary semantic segmenter,

    J. Wang, X. Li, J. Zhang, Q. Xu, Q. Zhou, Q. Yu, L. Sheng, and D. Xu, “Diffusion model is secretly a training-free open vocabulary semantic segmenter,”arXiv preprint arXiv:2309.02773, 2023

    2023 arXiv

  17. [25]

    Repurposing diffusion-based image generators for monocular depth estimation,

    B. Ke, A. Obukhov, S. Huang, N. Metzger, R. C. Daudt, and K. Schindler, “Repurposing diffusion-based image generators for monocular depth estimation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  18. [26]

    Do text-free diffusion models learn discriminative visual representations?,

    S. Mukhopadhyay, M. Gwilliam, Y . Yamaguchi, V . Agarwal, N. Padmanabhan, A. Swaminathan, T. Zhou, J. Ohya, and A. Shrivastava, “Do text-free diffusion models learn discriminative visual representations?,” arXiv preprint arXiv:2311.17921, 2023

    2023 arXiv

  19. [27]

    Deconstructing denoising diffusion models for self-supervised learning,

    X. Chen, Z. Liu, S. Xie, and K. He, “Deconstructing denoising diffusion models for self-supervised learning,” 2024

    2024

  20. [28]

    Coca: Contrastive captioners are image-text foundation models,

    J. Yu, Z. Wang, V . Vasudevan, L. Yeung, M. Seyedhosseini, and Y . Wu, “Coca: Contrastive captioners are image-text foundation models,”Trans. Mach. Learn. Res., vol. 2022, 2022

    2022

  21. [29]

    Multimodal few-shot learning with frozen language models,

    M. Tsimpoukelli, J. L. Menick, S. Cabi, S. M. A. Eslami, O. Vinyals, and F. Hill, “Multimodal few-shot learning with frozen language models,” inAdvances in Neural Information Processing Systems(M. Ranzato, A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, eds.), vol. 3...

    2021

  22. [30]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. L. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. a. Bi´nkowski, R...

    2022

  23. [31]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” inNeurIPS, 2023

    2023

  24. [32]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models,

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,”arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  25. [33]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning,

    J. Chen, D. Zhu, X. Shen, X. Li, Z. Liu, P. Zhang, R. Krishnamoorthi, V . Chandra, Y . Xiong, and M. Elhoseiny, “Minigpt-v2: large language model as a unified interface for vision-language multi-task learning,”arXiv preprint arXiv:2310.09478, 2023

    arXiv

  26. [34]

    Openflamingo: An open-source framework for training large autoregressive vision-language models,

    A. Awadalla, I. Gao, J. Gardner, J. Hessel, Y . Hanafy, W. Zhu, K. Marathe, Y . Bitton, S. Gadre, S. Sagawa, J. Jitsev, S. Kornblith, P. W. Koh, G. Ilharco, M. Wortsman, and L. Schmidt, “Openflamingo: An open-source framework for training large autoregressive vision-language m...

    2023 arXiv

  27. [35]

    Evaluating object hallucination in large vision-language models,

    Y . Li, Y . Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen, “Evaluating object hallucination in large vision-language models,”arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  28. [36]

    Multi-modal hallucination control by visual information grounding,

    A. Favero, L. Zancato, M. Trager, S. Choudhary, P. Perera, A. Achille, A. Swaminathan, and S. Soatto, “Multi-modal hallucination control by visual information grounding,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14303–14312, 2024

    2024

  29. [37]

    Detecting and preventing hallucinations in large vision language models,

    A. Gunjal, J. Yin, and E. Bas, “Detecting and preventing hallucinations in large vision language models,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 18135–18143, 2024

    2024

  30. [38]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,

    L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin,et al., “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” arXiv preprint arXiv:2311.05232, 2023

    2023 arXiv

  31. [39]

    Halle-switch: Rethinking and controlling object existence hallucinations in large vision language models for detailed caption,

    B. Zhai, S. Yang, X. Zhao, C. Xu, S. Shen, D. Zhao, K. Keutzer, M. Li, T. Yan, and X. Fan, “Halle-switch: Rethinking and controlling object existence hallucinations in large vision language models for detailed caption,”arXiv preprint arXiv:2310.01779, 2023. 12

    arXiv

  32. [40]

    Aligning large multimodal models with factually augmented rlhf,

    Z. Sun, S. Shen, S. Cao, H. Liu, C. Li, Y . Shen, C. Gan, L.-Y . Gui, Y .-X. Wang, Y . Yang,et al., “Aligning large multimodal models with factually augmented rlhf,”arXiv preprint arXiv:2309.14525, 2023

    2023 arXiv

  33. [41]

    Blink: Multimodal large language models can see but not perceive,

    X. Fu, Y . Hu, B. Li, Y . Feng, H. Wang, X. Lin, D. Roth, N. A. Smith, W.-C. Ma, and R. Krishna, “Blink: Multimodal large language models can see but not perceive,”arXiv preprint arXiv:2404.12390, 2024

    2024 arXiv

  34. [42]

    Llava-next: Improved reasoning, ocr, and world knowledge,

    H. Liu, C. Li, Y . Li, B. Li, Y . Zhang, S. Shen, and Y . J. Lee, “Llava-next: Improved reasoning, ocr, and world knowledge,” January 2024

    2024

  35. [43]

    Mm1: Methods, analysis & insights from multimodal llm pre-training,

    B. McKinzie, Z. Gan, J.-P. Fauconnier, S. Dodge, B. Zhang, P. Dufter, D. Shah, X. Du, F. Peng, F. Weers,et al., “Mm1: Methods, analysis & insights from multimodal llm pre-training,”arXiv preprint arXiv:2403.09611, 2024

    2024 arXiv

  36. [44]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” 2023

    2023

  37. [45]

    4m: Massively multimodal masked modeling,

    D. Mizrahi, R. Bachmann, O. Kar, T. Yeo, M. Gao, A. Dehghan, and A. Zamir, “4m: Massively multimodal masked modeling,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  38. [46]

    4m-21: An any-to-any vision model for tens of tasks and modalities,

    R. Bachmann, O. F. Kar, D. Mizrahi, A. Garjani, M. Gao, D. Griffiths, J. Hu, A. Dehghan, and A. Zamir, “4m-21: An any-to-any vision model for tens of tasks and modalities,”arXiv preprint arXiv:2406.09406, 2024

    2024 arXiv

  39. [47]

    Llava-plus: Learning to use tools for creating multimodal agents,

    S. Liu, H. Cheng, H. Liu, H. Zhang, F. Li, T. Ren, X. Zou, J. Yang, H. Su, J. Zhu, L. Zhang, J. Gao, and C. Li, “Llava-plus: Learning to use tools for creating multimodal agents,” 2023

    2023

  40. [48]

    Visual programming: Compositional visual reasoning without training,

    T. Gupta and A. Kembhavi, “Visual programming: Compositional visual reasoning without training,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14953–14962, 2023

    2023

  41. [49]

    Spatialbot: Precise spatial understanding with vision language models,

    W. Cai, I. Ponomarenko, J. Yuan, X. Li, W. Yang, H. Dong, and B. Zhao, “Spatialbot: Precise spatial understanding with vision language models,” 2024

    2024

  42. [50]

    Your diffusion model is secretly a zero- shot classifier,

    A. C. Li, M. Prabhudesai, S. Duggal, E. Brown, and D. Pathak, “Your diffusion model is secretly a zero- shot classifier,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2206–2217, October 2023

    2023

  43. [51]

    Diffusion models beat gans on image classification,

    S. Mukhopadhyay, M. Gwilliam, V . Agarwal, N. Padmanabhan, A. Swaminathan, S. Hegde, T. Zhou, and A. Shrivastava, “Diffusion models beat gans on image classification,”arXiv preprint arXiv:2307.08702, 2023

    2023 arXiv

  44. [52]

    Open-V ocabulary Panoptic Segmentation with Text-to-Image Diffusion Models,

    J. Xu, S. Liu, A. Vahdat, W. Byeon, X. Wang, and S. De Mello, “Open-V ocabulary Panoptic Segmentation with Text-to-Image Diffusion Models,”arXiv preprint arXiv:2303.04803, 2023

    2023 arXiv

  45. [53]

    Diffusion models for zero-shot open-vocabulary segmentation,

    L. Karazija, I. Laina, A. Vedaldi, and C. Rupprecht, “Diffusion models for zero-shot open-vocabulary segmentation,”arXiv preprint arXiv:2306.09316, 2023

    2023 arXiv

  46. [54]

    Not all diffusion model activations have been evaluated as discriminative features,

    B. Meng, Q. Xu, Z. Wang, X. Cao, and Q. Huang, “Not all diffusion model activations have been evaluated as discriminative features,” 2024

    2024

  47. [55]

    Dinov2: Learning robust visual features without supervision,

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y . Huang, S.-W. Li, I. Misra, M. Rabbat, V . Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. ...

    2024

  48. [56]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez, et al., “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,”See https://vicuna. lmsys. org (accessed 14 April 2023), vol. 2, no. 3, p. 6, 2023

    2023

  49. [57]

    Naturalbench: Evaluating vision-language models on natural adversarial samples,

    B. Li, Z. Lin, W. Peng, J. d. D. Nyandwi, D. Jiang, Z. Ma, S. Khanuja, R. Krishna, G. Neubig, and D. Ramanan, “Naturalbench: Evaluating vision-language models on natural adversarial samples,” inThe Thirty-eight Conference on Neural Information Processing Systems Datasets and B...

    2024

  50. [58]

    Openclip,

    G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V . Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt, “Openclip,” July 2021. If you use this software, please cite it as below

    2021

  51. [59]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” 2023

    2023

  52. [60]

    Data filtering networks,

    A. Fang, A. M. Jose, A. Jain, L. Schmidt, A. Toshev, and V . Shankar, “Data filtering networks,” 2023. 13

    2023

  53. [61]

    Demystifying clip data,

    H. Xu, S. Xie, X. E. Tan, P.-Y . Huang, R. Howes, V . Sharma, S.-W. Li, G. Ghosh, L. Zettlemoyer, and C. Feichtenhofer, “Demystifying clip data,” 2024

    2024

  54. [62]

    Eva-clip: Improved training techniques for clip at scale,

    Q. Sun, Y . Fang, L. Wu, X. Wang, and Y . Cao, “Eva-clip: Improved training techniques for clip at scale,” 2023

    2023

  55. [63]

    Microsoft COCO captions: Data collection and evaluation server,

    X. Chen, H. Fang, T. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick, “Microsoft COCO captions: Data collection and evaluation server,”CoRR, vol. abs/1504.00325, 2015

    2015 arXiv

  56. [64]

    Accurate computation of the log-sum-exp and softmax functions,

    P. Blanchard, D. J. Higham, and N. J. Higham, “Accurate computation of the log-sum-exp and softmax functions,” 2019

    2019

  57. [65]

    Winoground: Probing vision and language models for visio-linguistic compositionality,

    T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross, “Winoground: Probing vision and language models for visio-linguistic compositionality,” inCVPR, 2022

    2022

  58. [66]

    Similarity of neural network representations revisited,

    S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, “Similarity of neural network representations revisited,” inInternational conference on machine learning, pp. 3519–3529, PMLR, 2019

    2019

  59. [67]

    Cider: Consensus-based image description evaluation,

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” inProceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566–4575, 2015

    2015

  60. [68]

    Spice: Semantic propositional image caption evaluation,

    P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” inComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pp. 382–398, Springer, 2016

    2016

  61. [69]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” 2019. 14 Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractor Supplementary Material A Experimental Settings A.1 Block Configurations We first describe our block-selection ...

    2019

  62. [70]

    Frisbees

    For the second image of the otter, we see that the “Frisbees” map precisely localizes the frisbees 3 Table 7: Comparison of SD2.1 model across varying timesteps for Winoground Benchmark, using 512×512 images. For ‘Ensemble’ we use timesteps t∈ {189,389,589,789,989} , and avera...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.