Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

LLMs can see and hear without any training

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A scoring loop gives LLMs sight and hearing with zero training.

desk verdict Simple, broad test-time search method with real ablations, but the abstract's SOTA claim is contradicted by its own Table 1 and the video/audio evidence is thin. read the letter →

arxiv 2501.18096 v1 pith:VCHEE3M7 submitted 2025-01-30 cs.CV cs.AIcs.CLcs.LG

Kumar Ashutosh , Yossi Gandelsman , Xinlei Chen , Ishan Misra , Rohit Girdhar This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LG
keywords multimodalLLMzero-shotcaptioningtest-timeoptimizationiterativerefinementtraining-freetext-to-imagegenerationcross-modalarithmetic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents MILS (Multimodal Iterative LLM Solver), a training-free procedure that lets an otherwise text-only LLM solve multimodal tasks by generating candidate answers, having an off-the-shelf multimodal model score them, and feeding the scores back for the next round of candidates. Its central claim is emergent zero-shot generalization: MILS can caption images, videos, and audio without ever having been trained on captioning data, and it reports state-of-the-art or competitive results against methods specialized for those tasks. It also applies the same loop to text-to-image generation, style transfer, and cross-modal arithmetic by inverting multimodal embeddings into text. A sympathetic reader would care because the paper proposes that task-specific training and data curation can be replaced by test-time search over text candidates, provided a good scorer exists.

What carries the argument

The load-bearing object is the iterative GENERATOR–SCORER loop. For each test sample, an LLM serves as GENERATOR, producing a candidate set of text outputs such as captions, rewritten prompts, or editing instructions; an off-the-shelf scorer, such as SigLIP for images, ViCLIP for video, ImageBind for audio, PickScore for generated images, or Gram-matrix distance for style, assigns each candidate a scalar score against the test sample; the top-K scored candidates are rendered as text and returned to the LLM as feedback, and the loop repeats until convergence or N steps. For captioning tasks the loop starts from a large bootstrap pool of candidate captions generated from class labels before the sample is seen. The loop converts any scalar evaluator into a reward signal for a text-only LLM, which is why the same machinery can cross tasks and modalities.

What would settle it

Construct a test set whose ground-truth captions exclusively describe concepts absent from the class labels used to seed the initial candidate pool; if MILS cannot produce a correct caption on any such sample after full iteration, the capacity is bounded by the seed vocabulary, not emergent perception.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a blind and deaf LLM can acquire multimodal perception and generation abilities purely through an inference-time optimization loop. The same GENERATOR–SCORER architecture, with different off-the-shelf models plugged in, yields faithful image, video, and audio captions without any captioning-specific training; improves human-rated quality and text faithfulness of text-to-image models by discovering prompt rewrites; performs style transfer by minimizing Gram-matrix distances; and supports cross-modal arithmetic by turning embeddings into text. The paper frames this as a new instance of emergent zero-shot behavior, where the model generalizes to new tasks and modalities, not merely to new data.

Load-bearing premise

MILS only works if the initial candidate pool already contains plausible outputs for the test sample and the scorer ranks them truthfully; if either fails, the feedback loop can reshuffle but cannot invent the right answer.

Editorial extensions

If this is right

  • Captioning images, video, and audio requires no captioning data, only a frozen LLM and a suitable similarity scorer, so new languages or modalities can be added by swapping components.
  • Text-to-image generation improves at test time through LLM prompt rewriting selected by a human-preference scorer, which could automate a step that usually requires hand-tuned rewrite prompts.
  • Because the optimization is gradient-free, multimodal embeddings can be inverted into discrete text and then recombined, enabling operations like audio-plus-image arithmetic that plug into any text-to-image model.
  • Downstream metrics and the scorer's own signal rise together over optimization steps and converge in roughly 10 to 20 steps, suggesting the procedure is usable with modest inference-time compute.
  • Performance scales with the size of the LLM and the scorer, so future improvements in either component would transfer directly to MILS without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If MILS is right, the real bottleneck shifts from supervised task training to the coverage of the seed candidate pool and the accuracy of the scorer; the paper's own ablations show final performance rises with initial set size.
  • The phrase 'without any training' applies to the MILS loop itself; the underlying LLM and scorers carry pretrained knowledge, so the result is better interpreted as replacing task-specific optimization with generic pretrained components, not as perception from a blank model.
  • A testable extension is to make the bootstrap pool sample-dependent, for example by retrieving candidate captions from a text corpus or generating candidates conditioned on a coarse content description; this would separate seed coverage from iterative reasoning and likely raise the ceiling on hard examples.
  • Any domain with a cheap scalar evaluator but little paired data, such as robot control with physics simulators, program synthesis with unit tests, or molecule design with docking scores, could inherit the same generate-score-refine strategy without fine-tuning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MILS (Multimodal Iterative LLM Solver), a training-free test-time optimization framework in which an LLM acts as a GENERATOR to propose candidate outputs, an off-the-shelf multimodal model acts as a SCORER to evaluate them, and scores are fed back to the LLM to refine the candidates over several iterations. The method is applied to image, video, and audio captioning; text-to-image prompt rewriting; style transfer; and cross-modal arithmetic. The main claims are that MILS exhibits emergent zero-shot generalization to these tasks without task-specific training and that it establishes a new state of the art in emergent zero-shot image, video, and audio captioning.

Significance. If the central claims were substantiated, the work would be significant: it would show that a generic LLM combined with off-the-shelf multimodal scorers can solve a range of multimodal understanding and generation tasks through a simple test-time loop, without any task-specific training or data curation. The paper has welcome strengths: the framework is conceptually simple, it is evaluated across three modalities and several tasks, ablations analyze the effect of optimization steps, initial candidate set size, and choice of GENERATOR/SCORER, and code is released. However, the headline 'new state-of-the-art' claim is directly contradicted by the paper's own Table 1, and the absence of a zero-iteration retrieval control leaves open the possibility that the reported captioning quality is attributable to the pretrained SCORER plus a task-specific text prior rather than to iterative LLM reasoning. These issues are load-bearing for the paper's core claims.

major comments (3)
  1. [Abstract and Section 4.1, Table 1] The abstract's claim that MILS 'establish[es] a new state-of-the-art on emergent zero-shot image, video and audio captioning' is not supported by the paper's own results. In Table 1, MeaCap (Zeng et al., 2024) achieves higher CIDEr (42.5 vs 33.3), METEOR (16.6 vs 15.0), and SPICE (11.8 vs 9.6) than MILS, with MILS ahead only on BLEU-4 (8.0 vs 7.1). Since the paper itself argues that METEOR and SPICE are the more meaningful metrics for zero-shot captioning, describing MILS as state-of-the-art is inaccurate. The text and abstract should be revised to 'competitive' or the comparison should be extended to include a stronger baseline.
  2. [Section 4.1, Appendix B.1, Figure 10, Section 5] The central attribution of MILS's captioning performance to the iterative LLM loop is not established because the paper does not report a zero-iteration control. The initial candidate pool is generated from ImageNet or AudioSet class labels before the test sample is seen (Appendix B.1), and the SCORER is a pretrained text-to-modality similarity model. Ranking this pool once with the SCORER and outputting the top candidate would give the lower bound that the iterative loop must beat. Figure 10 shows that final performance is strongly correlated with the initial pool size, and Section 5 concedes that performance is bounded by generator diversity and scorer accuracy. Without this control, the reported captioning quality may be explained by the pretrained SCORER plus a task-specific text prior, rather than by the LLM's multi-step reasoning.
  3. [Section 4.5] The style transfer experiment is evaluated only qualitatively, with no benchmark or human study, and the SCORER's objective (Gram-matrix distance between VGG features of the generated image and the style image) is the same measure used to select the final output. Because the LLM's edit prompts are optimized to minimize exactly this score, the examples demonstrate convergence of the loss rather than perceptual style transfer quality. An external evaluation, such as a user study or an established style-transfer metric, is needed to support the claim that MILS 'produces accurately stylized images'.
minor comments (5)
  1. [Section 4.2, Table 2] The text says MILS is 'competitive' with (Nagrani et al., 2022) trained on VideoCC3M, but the CIDEr gap is large (2.3 vs 8.2), so the claim should be qualified or the comparison should be placed in proper context.
  2. [Figure 9] The axes in Figure 9 are difficult to read, especially the left plot's CLIP similarity scale and the right plot's PickScore scale; larger fonts or labeled ranges would improve clarity.
  3. [Conclusion] The word 'exhitbits' on page 8 should be corrected to 'exhibits'.
  4. [Title and Section 1] The phrase 'without any training' in the title and abstract is imprecise, since the system depends on pretrained multimodal scorers (SigLIP, ViCLIP, ImageBind, PickScore) that were trained on large paired datasets; the paper should consistently state that no task-specific training is performed, rather than implying no training at all.
  5. [Section 4.4] The human evaluation uses majority vote over three annotators; reporting inter-annotator agreement would strengthen the reliability of the reported win rates.

Circularity Check

1 steps flagged · score 4.0 of 10

Central captioning and generation results are externally grounded; only the style-transfer demonstration is circular, since the optimized Gram-matrix objective is also the stated quality measure.

  1. self definitional [Section 4.5 (Style Transfer), SCORER paragraph and Results paragraph; see also Section 5 limitation.]
    "SCORER. To measure the quality of the style transfer, we use a simple approach to estimate the similarity of colors and textures in the generated image compared to the style image. We use the distance between Gram matrices of the image features... We use MILS to minimize both the style and content losses. Results. Figure 6 shows some sample style transfer results. MILS generalizes to this novel task completely zero-shot and produces accurately stylized images."

    For style transfer, the paper's quality measure is the Gram-matrix distance, which is exactly the loss that MILS is run to minimize. The SCORER paragraph says the Gram distance is used 'to measure the quality of style transfer' and then says MILS is used 'to minimize both the style and content losses.' Since the optimization loop is driven by the same function that defines success, any output it selects improves on that measure by construction. No independent metric or human evaluation is reported for this demonstration, so the claimed 'accurately stylized' result does not provide evidence beyond the optimizer's own objective. This is a circumscribed component of the paper, not the captioning or generation core.

full rationale

The captioning results are not circular: the optimization objective (SigLIP, ViCLIP, or ImageBind similarity) differs from the reported evaluation metrics (BLEU, METEOR, CIDEr, SPICE), and MILS is compared against prior zero-shot systems on standard benchmarks. The central 'LLMs can see and hear' claim therefore has independent content, though it is weakened by the absence of a zero-iteration retrieval control: the paper does not report what a single top-1 ranking of the pre-generated candidate pool by the SCORER would achieve, so the specific contribution of the iterative LLM loop over the pretrained scorer and broad text prior is not isolated. That missing baseline is a completeness/attack-surface issue, not circularity. The T2I improvements are externally checked by human raters against the base model, so they are not circular either. There is a minor self-citation of Gandelsman et al. for the initial candidate set and of Girdhar et al. for the term 'emergent zero-shot' and for ImageBind; these are not load-bearing in the sense of forbidding alternatives or supplying the result. The abstract's 'new state-of-the-art' claim is not supported by Table 1, where MeaCap TF exceeds MILS on CIDEr, METEOR, and SPICE, but that is an accuracy/overclaim issue rather than circularity. The only true circular step is the style-transfer demonstration, where the quality measure equals the optimized objective. Score 4 reflects one peripheral self-referential demonstration while the core captioning and generation derivations remain externally grounded.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a hand-chosen initial candidate set generated from class labels, a scorer that must rank candidates correctly, and the transfer of that candidate set across datasets. No fundamentally new entities or forces are introduced. The free parameters are all hyperparameters of the search loop.

free parameters (4)
  • initial_candidate_set_size = 30,000 (image/video); 50,000 (audio)
    Hand-chosen; larger sets improve final accuracy (Figure 10).
  • top_k_candidates = 50
    Hand-chosen; greedy top-K selection returns the 50 highest-scoring candidates to the LLM each step.
  • optimization_steps = 10 (captioning); 10-20 (generation)
    Hand-chosen stopping criterion; metrics converge by step 10-20 (Figure 9).
  • candidate_prompt_design = LLM prompt with class labels, 40-50 captions per class
    Hand-designed to induce diversity; derived from Gandelsman et al. (2024), a self-citation.
assumptions (4)
  • domain assumption LLM can produce diverse, relevant candidate captions from class labels alone
    Appendix B.1 generates the 30K image/video and 50K audio candidates using only ImageNet/AudioSet class names; no test signal is used.
  • domain assumption Scoring model similarity is a reliable proxy for caption or prompt quality
    Sections 4.1-4.4 use SigLIP, ViCLIP, ImageBind, or PickScore as the sole objective guiding the search; if scores mis-rank candidates, the loop converges to wrong outputs.
  • domain assumption The initial candidate set transfers across datasets (ImageNet/COCO, AudioSet/Clotho)
    Candidates are generated from ImageNet classes but tested on MSCOCO and MSR-VTT; the paper provides no analysis of coverage.
  • standard math Automated captioning metrics (BLEU/CIDEr/METEOR/SPICE) are valid for zero-shot evaluation
    These are standard benchmarks; their validity for emergent zero-shot captions is assumed, especially when the candidate vocabulary is not benchmark-specific.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMs can see and hear without any training." pith.science (2026). https://pith.science/paper/VCHEE3M7

@misc{pith2026250118096,
  author       = {Pith},
  title        = {Pith review of: LLMs can see and hear without any training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VCHEE3M7}},
  note         = {Machine review of arXiv:2501.18096}
}
read the original abstract

We present MILS: Multimodal Iterative LLM Solver, a surprisingly simple, training-free approach, to imbue multimodal capabilities into your favorite LLM. Leveraging their innate ability to perform multi-step reasoning, MILS prompts the LLM to generate candidate outputs, each of which are scored and fed back iteratively, eventually generating a solution to the task. This enables various applications that typically require training specialized models on task-specific data. In particular, we establish a new state-of-the-art on emergent zero-shot image, video and audio captioning. MILS seamlessly applies to media generation as well, discovering prompt rewrites to improve text-to-image generation, and even edit prompts for style transfer! Finally, being a gradient-free optimization approach, MILS can invert multimodal embeddings into text, enabling applications like cross-modal arithmetic.

Figures

Figures reproduced from arXiv: 2501.18096 by the authors.

Figure 1
Figure 1. Our proposed approach, MILS, enables various applications, from captioning images, video, or audio; improving text-to-image generation; image editing such as style transfer; as well as arithmetic across different modalities by inverting them all into text. It accomplishes all this using a purely test-time optimization approach without any task specific training or data curation! Abstract We present MILS: Multimodal … view at source ↗
Figure 2
Figure 2. MILS leverages two key modules, GENERATOR and SCORER, to solve multimodal tasks. The GENERATOR will generate a number of text candidates, e.g. captions for image captioning and prompts for T2I, each of which will be scored by the SCORER, and passed back into the GENERATOR as feedback to generate the next batch of text candidates, eventually producing the final output for the input test sample. capacity (context leng… view at source ↗
Figure 3
Figure 3. Image Captioning using MILS, compared to existing state-of-the-art zero-shot approach, MeaCap (Zeng et al., 2024). MILS, while being a much simpler approach, produces more accurate and syntactically correct captions to the image. Method BLEU4 CIDEr METEOR SPICE ZeroCap (Tewel et al., 2022) 2.6 14.6 11.5 5.5 ConZIC (Zeng et al., 2023) 1.3 13.3 11.2 5.0 CLIPRe (Li et al., 2023c) 4.6 25.6 13.3 9.2 MeaCapTF (Zeng et al.… view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Improving image generation using MILS. Ap￾plying MILS to a GENERATOR using the same base model, a Latent Diffusion Model (LDM) in this case, leads to much higher quality images. We show the original input prompt, the generation from the base model, and from MILS. 4.3. …
Figure 7
Figure 7. Figure 7: MILS enables cross-modal arithmetic by in￾verting modalities into text, combining them, and mapping them back to an image. thetic quality using MILS is clearly apparent. We find that MILS is able to simplify complex prompts and add aesthetic details, that improve the o…
Figure 8
Figure 8. Figure 8: Image captioning over steps. This shows the captions generated by MILS at different steps (#) of opti￾mization process, getting progressively more accurate. SCORER. To measure the quality of the style transfer, we use a simple approach to estimate the similarity of col…
Figure 9
Figure 9. Figure 9: Performance with number of optimization steps for captioning (left) and generation (right). We show the optimization metric, i.e. SCORER’s output (CLIP Similarity and PickScore), and the downstream metric (SPICE (Anderson et al., 2016) and human evaluation win% for qua…
Figure 11
Figure 11. Figure 11: Improved image generation over optimization steps. The quality of the output improves over the optimization steps (#). We also show the prompt being produced by the LLM in the GENERATOR, which is passed to the T2I model. 1B 3B 8B 8 8.5 9 9.5 GENERATOR Parameters SPICE…
Figure 12
Figure 12. Figure 12: Impact of GENERATOR (left) and SCORER (right) size. We report the downstream metric SPICE for both these ablations since CLIP similarity may not be comparable across model sizes. For the GENERATOR and SCORER we use different sized Llama or MetaCLIP models. As evident …
Figure 13
Figure 13. Figure 13: Human evaluation question form for faithfulness (left) and quality (right). We ask two questions. The first question asks which of the two images the rater prefers. The next question elaborates on the first response by asking the rater to list the reason for their pre…
Figure 14
Figure 14. Figure 14: High-quality image generation faithfulness examples. We provide these examples to the annotators to judge the faithfulness. The generated image should have the correct object, color, texture or scene layout. descriptions to form new descriptions, rephrase in￾dividual …
Figure 15
Figure 15. Figure 15: High-quality image generation quality exam￾ples. We provide these examples to the annotators to judge the quality. The resulting image should have high pixel sharpness, object and scene consistency, and the concepts should be recognizable. Model Training data SPICE CL…
Figure 16
Figure 16. Figure 16: Performance comparison with various LLMs as GENERATOR– Gemma2 9B, Mistral 8B, and Llama 3.1 8B, on CLIP similarity (left) and SPICE (right). This trend shows that the performance improves over optimization steps, regardless of the choice of the LLM. In particular, Lla…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models

    cs.LG 2025-08 conditional novelty 6.0 of 10

    A noise hypernetwork learns a reward-tilted initial noise distribution for frozen distilled diffusion generators, recovering roughly half of test-time noise-optimization gains at a fraction of the compute.

  2. SmartAvatar: Text- and Image-Guided Human Avatar Generation with VLM AI Agents

    cs.CV 2025-06 reject novelty 6.0 of 10

    A VLM-agent pipeline generates rigged 3D avatars from image or text by iteratively refining Blender/HumGen3D parameters against a similarity-based auto-verification loop, yet its reported evaluation does not support t...

Reference graph

Works this paper leans on

64 extracted references · 38 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    B., Chaplot, D., Chudnovsky, J., Garg, S., Gervet, T., Ghosh, S., H \'e liou, A., Jacob, P., et al

    Agrawal, P., Antoniak, S., Hanna, E. B., Chaplot, D., Chudnovsky, J., Garg, S., Gervet, T., Ghosh, S., H \'e liou, A., Jacob, P., et al. Pixtral 12b. arXiv preprint arXiv:2410.07073, 2024

    arXiv 2024

  3. [3]

    SPICE : S emantic propositional image caption evaluation

    Anderson, P., Fernando, B., Johnson, M., and Gould, S. SPICE : S emantic propositional image caption evaluation. In ECCV, 2016

    work page 2016

  4. [4]

    and Lavie, A

    Banerjee, S. and Lavie, A. METEOR : A n automatic metric for mt evaluation with improved correlation with human judgments. In ACL, 2005

    work page 2005

  5. [5]

    Improving image generation with better captions

    Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., Manassra, W., Dhariwal, P., Chu, C., and Jiao, Y. Improving image generation with better captions. 2023

    work page 2023

  6. [6]

    W., Fidler, S., and Kreis, K

    Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023

    2023

  7. [7]

    Emu: Enhancing image generation models using photogenic needles in a haystack

    Dai, X., Hou, J., Ma, C.-Y., Tsai, S., Wang, J., Wang, R., Zhang, P., Vandenhende, S., Wang, X., Dubey, A., et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807, 2023

    arXiv 2023

  8. [8]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, 2009

    2009

Show all 64 references
  1. [9]

    Clotho: An audio captioning dataset

    Drossos, K., Lipping, S., and Virtanen, T. Clotho: An audio captioning dataset. In ICASSP, 2020

    2020

  2. [10]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  3. [11]

    M., Jain, A., Schmidt, L., Toshev, A., and Shankar, V

    Fang, A., Jose, A. M., Jain, A., Schmidt, L., Toshev, A., and Shankar, V. Data filtering networks. In ICLR, 2024

    2024

  4. [12]

    A., and Steinhardt, J

    Gandelsman, Y., Efros, A. A., and Steinhardt, J. Interpreting the second-order effects of neurons in clip. arXiv preprint arXiv:2406.04341, 2024

    2024 arXiv

  5. [13]

    Gatys, L. A. A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576, 2015

    2015 arXiv

  6. [14]

    On the content bias in fr \'e chet video distance

    Ge, S., Mahapatra, A., Parmar, G., Zhu, J.-Y., and Huang, J.-B. On the content bias in fr \'e chet video distance. In CVPR, 2024

    2024

  7. [15]

    F., Ellis, D

    Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, 2017

    2017

  8. [16]

    V., Joulin, A., and Misra, I

    Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Misra, I. ImageBind : O ne embedding space to bind them all. In CVPR, 2023

    2023

  9. [17]

    S., Shah, A., Yin, X., Parikh, D., and Misra, I

    Girdhar, R., Singh, M., Brown, A., Duval, Q., Azadi, S., Rambhatla, S. S., Shah, A., Yin, X., Parikh, D., and Misra, I. Emu video: Factorizing text-to-video generation by explicit image conditioning. In ECCV, 2024

    2024

  10. [18]

    Mmg-ego4d: Multimodal generalization in egocentric action recognition

    Gong, X., Mohan, S., Dhingra, N., Bazin, J.-C., Li, Y., Wang, Z., and Ranjan, R. Mmg-ego4d: Multimodal generalization in egocentric action recognition. In CVPR, 2023

    2023

  11. [19]

    Audioclip: Extending clip to image, text and audio

    Guzhov, A., Raue, F., Hees, J., and Dengel, A. Audioclip: Extending clip to image, text and audio. In ICASSP, 2022

    2022

  12. [20]

    P., Poole, B., Norouzi, M., Fleet, D

    Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022

    arXiv

  13. [21]

    Openclip, 2021

    Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L. Openclip, 2021

    2021

  14. [22]

    Rethinking fid: Towards a better evaluation metric for image generation

    Jayasumana, S., Ramalingam, S., Veit, A., Glasner, D., Chakrabarti, A., and Kumar, S. Rethinking fid: Towards a better evaluation metric for image generation. In CVPR, 2024

    2024

  15. [23]

    Mistral 7b

    Jiang, A., Sablayrolles, A., Mensch, A., Bamford, C., Singh Chaplot, D., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arxiv. arXiv preprint arXiv.2310.06825, 2023

    2023 arXiv

  16. [24]

    and Fei-Fei, L

    Karpathy, A. and Fei-Fei, L. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015

    2015

  17. [25]

    What do we learn from inverting clip models? arXiv preprint arXiv:2403.02580, 2024

    Kazemi, H., Chegini, A., Geiping, J., Feizi, S., and Goldstein, T. What do we learn from inverting clip models? arXiv preprint arXiv:2403.02580, 2024

    2024 arXiv

  18. [26]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation

    Kirstain, Y., Polyak, A., Singer, U., Matiana, S., Penna, J., and Levy, O. Pick-a-pic: An open dataset of user preferences for text-to-image generation. NeurIPS, 2023

    2023

  19. [27]

    S., Reid, M., Matsuo, Y., and Iwasawa, Y

    Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. NeurIPS, 2022

    2022

  20. [28]

    Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities

    Kong, Z., Goel, A., Badlani, R., Ping, W., Valle, R., and Catanzaro, B. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. In ICML, 2024

    2024

  21. [29]

    Labs, B. F. Flux.1-schnell. https://huggingface.co/black-forest-labs/FLUX.1-schnell. Accessed: 2024-10-01

    2024

  22. [30]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023 a

    2023

  23. [31]

    A large-scale dataset towards multi-modal multilingual instruction tuning

    Li, L., Yin, Y., Li, S., Chen, L., Wang, P., Ren, S., Li, M., Yang, Y., Xu, J., Sun, X., et al. A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv preprint arXiv:2306.04387, 2023 b

    2023 arXiv

  24. [32]

    DeCap : D ecoding CLIP latents for zero-shot captioning via text-only training

    Li, W., Zhu, L., Wen, L., and Yang, Y. DeCap : D ecoding CLIP latents for zero-shot captioning via text-only training. In ICLR, 2023 c

    2023

  25. [33]

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In ECCV, 2014

    2014

  26. [34]

    T., Ben-Hamu, H., Nickel, M., and Le, M

    Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  27. [35]

    Improving text-to-image consistency via automatic prompt optimization

    Ma \ n as, O., Astolfi, P., Hall, M., Ross, C., Urbanek, J., Williams, A., Agrawal, A., Romero-Soriano, A., and Drozdzal, M. Improving text-to-image consistency via automatic prompt optimization. arXiv preprint arXiv:2403.17804, 2024

    2024 arXiv

  28. [36]

    Whiteboard-of-thought: Thinking step-by-step across modalities

    Menon, S., Zemel, R., and Vondrick, C. Whiteboard-of-thought: Thinking step-by-step across modalities. arXiv preprint arXiv:2406.14562, 2024

    2024 arXiv

  29. [37]

    How T o100 M : L earning a T ext- V ideo E mbedding by W atching H undred M illion N arrated V ideo C lips

    Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., and Sivic, J. How T o100 M : L earning a T ext- V ideo E mbedding by W atching H undred M illion N arrated V ideo C lips. In ICCV, 2019

    2019

  30. [38]

    Leave no context behind: Efficient infinite context transformers with infini-attention

    Munkhdalai, T., Faruqui, M., and Gopal, S. Leave no context behind: Efficient infinite context transformers with infini-attention. arXiv preprint arXiv:2404.07143, 2024

    2024 arXiv

  31. [39]

    H., Seybold, B., Hauth, A., Manen, S., Sun, C., and Schmid, C

    Nagrani, A., Seo, P. H., Seybold, B., Hauth, A., Manen, S., Sun, C., and Schmid, C. Learning audio-video modalities from image captions. In ECCV, 2022

    2022

  32. [40]

    Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In ICML, 2021

    2021

  33. [41]

    Introducing openai o1-preview

    OpenAI. Introducing openai o1-preview. https://openai.com/index/introducing-openai-o1-preview/. Accessed: 2024-10-01

    2024

  34. [42]

    BLEU : A method for automatic evaluation of machine translation

    Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. BLEU : A method for automatic evaluation of machine translation. In ACL, 2002

    2002

  35. [43]

    Movie gen: A cast of media foundation models

    Polyak, A., Zohar, A., Brown, A., Tjandra, A., Sinha, A., Lee, A., Vyas, A., Shi, B., Ma, C.-Y., Chuang, C.-Y., et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720, 2024

    2024 arXiv

  36. [44]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In ICML, 2021

    2021

  37. [45]

    Hierarchical text-conditional image generation with clip latents

    Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022

    2022 arXiv

  38. [46]

    High-resolution image synthesis with latent diffusion models

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, 2022

    2022

  39. [47]

    L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., Ho, J., Fleet, D

    Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 2022

    2022

  40. [48]

    Zero-shot audio captioning with audio-language model guidance and audio context keywords

    Salewski, L., Fauth, S., Koepke, A., and Akata, Z. Zero-shot audio captioning with audio-language model guidance and audio context keywords. In NeurIPS Workshops, 2023

    2023

  41. [49]

    Zero-shot audio captioning via audibility guidance

    Shaharabany, T., Shaulov, A., and Wolf, L. Zero-shot audio captioning via audibility guidance. arXiv preprint arXiv:2309.03884, 2023

    2023 arXiv

  42. [50]

    H., Tan, H., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., and Keutzer, K

    Shen, S., Li, L. H., Tan, H., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., and Keutzer, K. How much can clip benefit vision-and-language tasks? In ICLR, 2022

    2022

  43. [51]

    Emu edit: Precise image editing via recognition and generation tasks

    Sheynin, S., Polyak, A., Singer, U., Kirstain, Y., Zohar, A., Ashual, O., Parikh, D., and Taigman, Y. Emu edit: Precise image editing via recognition and generation tasks. In CVPR, 2024

    2024

  44. [52]

    and Zisserman, A

    Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015

    2015

  45. [53]

    S., Love, J., et al

    Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi \`e re, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  46. [54]

    Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic

    Tewel, Y., Shalev, Y., Schwartz, I., and Wolf, L. Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic. In CVPR, 2022

    2022

  47. [55]

    CIDEr : C onsensus-based image description evaluation

    Vedantam, R., Lawrence Zitnick, C., and Parikh, D. CIDEr : C onsensus-based image description evaluation. In CVPR, 2015

    2015

  48. [56]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation

    Wang, Y., He, Y., Li, Y., Li, K., Yu, J., Ma, X., Li, X., Chen, G., Chen, X., Wang, Y., et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. In ICLR, 2023

    2023

  49. [57]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022

    2022

  50. [58]

    E., Huang, P.-Y., Howes, R., Sharma, V., Li, S.-W., Ghosh, G., Zettlemoyer, L., and Feichtenhofer, C

    Xu, H., Xie, S., Tan, X. E., Huang, P.-Y., Howes, R., Sharma, V., Li, S.-W., Ghosh, G., Zettlemoyer, L., and Feichtenhofer, C. Demystifying clip data. In ICLR, 2024

    2024

  51. [59]

    MSR-VTT : A large video description dataset for bridging video and language

    Xu, J., Mei, T., Yao, T., and Rui, Y. MSR-VTT : A large video description dataset for bridging video and language. In CVPR, 2016

    2016

  52. [60]

    V., Zhou, D., and Chen, X

    Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models as optimizers. In ICLR, 2023

    2023

  53. [61]

    ConZIC : Controllable zero-shot image captioning by sampling-based polishing

    Zeng, Z., Zhang, H., Lu, R., Wang, D., Chen, B., and Wang, Z. ConZIC : Controllable zero-shot image captioning by sampling-based polishing. In CVPR, 2023

    2023

  54. [62]

    Meacap: Memory-augmented zero-shot image captioning

    Zeng, Z., Xie, Y., Zhang, H., Chen, C., Chen, B., and Wang, Z. Meacap: Memory-augmented zero-shot image captioning. In CVPR, 2024

    2024

  55. [63]

    Sigmoid loss for language image pre-training

    Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sigmoid loss for language image pre-training. In ICCV, 2023

    2023

  56. [64]

    a henb \

    Zhou, X., Girdhar, R., Joulin, A., Kr \"a henb \"u hl, P., and Misra, I. Detecting twenty-thousand classes using image-level supervision. In ECCV, 2022

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.