Pith. sign in

REVIEW 5 major objections 5 minor 50 references

Seeing Sound: Assembling Sounds from Visuals for Audio-to-Image Generation

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training audio-to-image generative models on artificially sonified pairs is competitive with training on real video pairs.

desk verdict A credible and useful data-recipe paper for audio-to-image generation, with an honest limitations section; the central claim is plausible but the comparisons lack the same-backbone control and pair-quality checks needed to fully justify it. read the letter →

arxiv 2501.05413 v1 pith:6RSYEFYJ submitted 2025-01-09 cs.SD cs.CVcs.GReess.AS

Darius Petermann , Mahdi M. Kalayeh This is my paper · ORCID
classification cs.SDcs.CVcs.GReess.AS
keywords audio-to-imagegenerationimagesonificationvision-languagemodelscross-modalretrievaldiffusionaudio-visualcorrespondenceacousticscenemodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper challenges the field's default assumption: to train an audio-to-image generative model you need real audio-visual pairs, usually cut from videos. It argues that this assumption needlessly limits scale, quality, and diversity, and instead proposes a scalable image sonification pipeline that fabricates pairs from large, disjoint image-only and audio-only datasets. A vision-language model is prompted to list the sounds an image would make, and a contrastive audio-text retrieval step fetches matching audio clips, which are then mixed into a single soundtrack per image. Using about one million such sonified images, the authors adapt a pre-trained text-to-image diffusion model to audio conditioning and report competitive or better results against state-of-the-art baselines across five benchmarks, including on out-of-domain data. The paper further shows the model spontaneously develops audio-driven controls such as semantic mixing, loudness-based emphasis, and reverberation as a cue to physical space.

What carries the argument

The load-bearing mechanism is a modular retrieval pipeline for sonifying images. First, a vision-language model prompted with a hand-tuned instruction returns a numbered list of one to three sounds associated with objects visibly present in the image, yielding sounding concepts as text. Second, a contrastively trained joint audio-text encoder embeds both each concept and each candidate audio clip in a shared space; cosine similarity scores are transformed with a signed square root to suppress outlier matches, a dynamic threshold based on the mean of the top-k scores governs which clips are eligible, and one clip is randomly sampled per concept. Third, the selected clips are loudness-normalized and linearly summed in the time domain to form the image's audio counterpart, which is then encoded with an audio-spectrogram transformer into a vector time series that conditions a pre-trained text-to-image diffusion transformer through a trainable audio projector. This machinery converts the data bottleneck into a retrieval problem: any image dataset can be paired with any audio pool.

What would settle it

Take a random sample of the released sonified pairs, have human annotators judge whether each audio clip matches the visible sounding objects, and compute retrieval precision; if precision is near chance while model performance stays high, the claim that semantic pair correspondence drives the results would be refuted.

Watch

Extended reading notes

Core claim

The central claim is that ground-truth audio-visual correspondence is not a prerequisite for audio-to-image generation. The authors construct synthetic pairs by using a vision-language model to convert an image into a small set of sounding concepts, such as the objects in the scene and the noises they make, then retrieving audio excerpts from a large clip pool via a shared audio-text embedding space. The retrieved clips are loudness-normalized and summed in the time domain, producing an audio track for each of roughly one million images from an image segmentation dataset. Fine-tuning a pre-trained text-to-image diffusion transformer on these pairs yields a model whose FID, image-image similarity, and audio-image similarity scores are competitive with or better than prior audio-to-image models, even though those prior models were evaluated in-sample on their own training data while the proposed model is out-of-sample. The authors interpret this as evidence that the insistence on real synchronized video data is overstated.

Load-bearing premise

The central assumption is that the retrieval step returns audio that truly matches the image's content; the paper does not directly measure retrieval precision or pair quality with human judgments, and it documents failure cases such as hallucinated seagulls and statues treated as sounding objects.

Editorial extensions

If this is right

  • Audio-to-image training data can scale with image-only and audio-only datasets, so domain coverage is no longer limited by the availability of synchronized video.
  • Out-of-domain evaluation on five benchmarks shows competitive FID and top audio-image alignment on most benchmarks, so models trained on sonified data generalize to unseen distributions.
  • The model learns to mix multiple sound sources into a coherent image, to weight object prominence by loudness, and to render acoustic spaces such as church or office reverberation, without explicit supervision for these properties.
  • Contrastive pre-training of the audio projector is not necessary: training it from scratch alongside the diffusion backbone gives comparable quality and convergence, contrary to prior practice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same sonification pipeline could be inverted: generated images from text-to-image models could be sonified on the fly, enabling audio-conditioned editing of arbitrary image collections for which no audio was ever recorded.
  • If retrieval precision were gated by a human or automated quality filter, the approach should improve further; a testable prediction is that model audio-image similarity scores track retrieval precision dataset-wide.
  • The emergent reverberation control hints that joint audio-text encoders can serve as a proxy for acoustic scene labels; one could systematically probe this by training on audio prompts whose reverberation is parametrically controlled and measuring how well generated images are classified by scene type.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. Petermann and Kalayeh propose a scalable pipeline that replaces in-the-wild video-derived audio-visual pairs with synthetic pairs: a vision-language model (LLaVA and CogVLM) extracts sounding concepts from images in the SAM dataset, and CLAP-based text-to-audio retrieval with a signed square root transform and a dynamic threshold selects audio clips from a pool built from AudioSet, FSD50K, and BBC SFX; the retrieved clips are loudness-normalized and linearly mixed. The resulting roughly one million synthetic pairs are used to adapt a PixArt-alpha diffusion transformer to audio-to-image generation. The paper reports competitive FID, IIS, and AIS scores on five benchmarks and presents qualitative ablations on semantic mixing, loudness interpolation, and reverberation. It concludes that ground-truth audio-visual correspondence is not necessary for training audio-to-image models.

Significance. If the central claim holds, the work is significant: it offers a scalable route to large-scale audio-visual data from disjoint uni-modal sources and challenges a widely held data assumption in the field. The paper deserves credit for planning to release code, model weights, and the dataset; for evaluating on five out-of-domain benchmarks; for documenting failure cases in Appendix A.4; and for a careful analysis of loudness retention in AST embeddings. The hypothesis is not circular in the fitting sense: no parameters are tuned to benchmark outputs, and evaluation uses independent datasets. However, the evidence is not yet conclusive. The comparison lacks a same-backbone control trained on real video-derived pairs, the retrieval pipeline's precision is never quantified, and the AIS metric is partially aligned with the training-time data construction. These gaps are addressable and do not invalidate the idea, but they currently leave the main claim under-supported.

major comments (5)
  1. [Section 4.4, Table 1] The central claim that sonified images are sufficient is not tested against the appropriate control. All baselines use different backbones, training data, and exposure to test sets; the proposed model initializes from PixArt-alpha at 512x512, which is a substantially stronger base model than those used by SonicDiffusion or AudioToken. Without a model trained with the same backbone, data budget, and training procedure on real video-derived pairs, the reported FID and AIS gains cannot be attributed to the sonification pipeline. Please add such a same-backbone control, or substantially qualify the claim that ground-truth correspondence is unnecessary.
  2. [Section 4.2, AIS metric] AIS is computed with Wav2CLIP, a model trained on audio-CLIP image-text alignment. Since the proposed training pairs are constructed via CLAP text-audio retrieval, the evaluation metric is partially aligned with the data construction process. This raises the risk that high AIS reflects generic audio-visual association rather than semantic alignment. Please report AIS with additional audio encoders (e.g., the CLAP audio tower or an AST-based similarity) and, ideally, include a small human evaluation of semantic agreement between generated images and audio prompts.
  3. [Section 3.3, Algorithm 1] The data-quality assumption is load-bearing and is not directly verified. The signed square root transform, the top-k threshold, and the random choice among candidates are heuristics, and retrieval precision or recall is never measured on a labeled subset. Appendix A.4 documents systematic failures (seagulls for beaches without visible birds, sounds for statues), so the claim of strong cross-modal correspondence needs quantitative support. Please provide a human-evaluated or labeled-subset measurement of retrieval precision for a random sample of images.
  4. [Section 4.4, Table 1, Greatest Hits] On Greatest Hits the proposed model has FID 228.4 versus SonicDiffusion's 87.3, and this is the row where the paper's in-sample/out-of-sample distinction most strongly favors the competitor. The paper should discuss what this large gap implies for the central claim: when domain-specific real pairs are available, they appear to provide a substantial advantage that synthetic sonification does not match. This does not refute the hypothesis, but it needs to be addressed explicitly rather than set aside as a single exception.
  5. [Section 4.5, Figures 5 and 7] The ablations on semantic mixing, loudness interpolation, and reverberation are presented qualitatively, and the loudness interpolation is described as 'rather abrupt' because the model saw very small variance in source loudness during training. Since these emergent capabilities are a key contribution, please quantify them (for example, by measuring whether the generated image contains the expected sources using a classifier, or by running a human rating of interpolation smoothness), and report the actual LUFS range used for gamma_m. As written, the abrupt transition weakens the claim of loudness calibration.
minor comments (5)
  1. [Section 4.5] The phrase 'the model successfully incubates multi-label audio mixtures' appears to be a typo for 'incorporates'; please correct it.
  2. [Appendix A.7] This section says the three audio sources 'total in 70k audio clips,' while Section 4.1 reports roughly 500k five-second excerpts after standardization; please clarify the relationship between these two numbers.
  3. [Table 1] The table does not report per-benchmark sample counts; please include the number of evaluation instances for each dataset so that FID and AIS comparisons are interpretable.
  4. [Appendix A.2] The filtering step removes images whose LLaVA captions contain a fixed keyword list; please report how many images were removed and how the one-million-image subset was sampled after filtering.
  5. [Appendix A.10] In the text, 'Hubert' should be 'HuBERT', and 'along side' should be 'alongside'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the derivation chain is data construction plus standard training, evaluated on independent benchmarks; no parameter is fitted to the evaluation targets, and no load-bearing self-citation appears.

full rationale

The paper's central claim rests on a pipeline: VLM-extracted sounding concepts, CLAP-based text-to-audio retrieval, AST audio embeddings, and fine-tuning a pre-trained PixArt-α diffusion backbone. Nothing in this chain is defined in terms of the evaluation quantities it is used to predict. The model is trained once on roughly one million synthetic image-audio pairs and then evaluated on Greatest Hits, Landscapes+ITW, VEGAS, and VGGSound, with FID, IIS, and AIS reported as external metrics. No benchmark statistic enters as a training loss, threshold, or fitted parameter, so there is no fitted-input-called-prediction pattern. The AIS metric uses Wav2CLIP rather than the CLAP model used for retrieval, so the evaluation function is not identical to the data-construction objective; any semantic correlation between the two embedding spaces is a measurement-concern, not a definitional reduction. The reference list contains no self-citations by the authors, and the load-bearing external components (PixArt-α, LDM, AST, CLAP, Wav2CLIP) are pre-trained independently of this paper's outputs. The qualitative ablations of semantic mixing, loudness interpolation, and reverberation are presented as observed behaviors rather than as predictions derived from the method's own assumptions; the paper even documents failure cases such as hallucinated seagulls and statue-generated sounds, which are candid negative evidence rather than circular support. The main caveats, including unquantified retrieval precision and the absence of a same-backbone control against video-derived pairs, are rigor and correctness concerns, not circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several unverified domain assumptions about VLM and CLAP fidelity, plus untested filtering and mixing choices. None of these are machine-checked or externally validated in the paper.

free parameters (3)
  • Retrieval top-k threshold parameter k
    Algorithm 1 computes the minimum eligible similarity score as the mean of the top-k scores; the value of k is not reported, and it controls the diversity/quality trade-off of retrieved audio.
  • LUFS mixing range for gamma_m
    Section 3.3 samples loudness normalization values from a uniformly-distributed decibel-LUFS range with very small variance; the actual range is not specified.
  • Number of sounding concepts per image = one to three (prompt-limited)
    The VLM prompt requests one to three sounds; this design choice limits the complexity of audio mixtures the model sees.
assumptions (5)
  • domain assumption VLMs, when prompted, can extract sounding concepts that correspond to objects visible in the image.
    Section 3.2 relies on LLaVA and CogVLM descriptions as ground truth for what is audible in an image; Appendix A.4 documents hallucinated sounds (e.g., seagulls at beaches, statues treated as sounding objects).
  • domain assumption CLAP's joint embedding space makes text-to-audio cosine similarity a valid proxy for semantic match.
    Section 3.3 uses CLAP to retrieve audio for a sounding concept; the quality of the synthetic pairs depends entirely on this assumption, with no direct validation of retrieval precision.
  • domain assumption AST embeddings retain loudness and multi-source information sufficiently for audio-driven image control.
    Section 3.4 and Appendix A.8 justify this with a KNN loudness-retention study, but the study only shows a gradual accuracy decrease, not that the embedding encodes loudness in a controllable way.
  • domain assumption In-the-wild audio-visual pairs have low enough semantic correspondence that synthetic pairs are not worse.
    This motivates the method; the paper provides illustrative examples but no quantitative comparison of pair quality between real and synthetic data.
  • ad hoc to paper Mixing audio sources in the time-domain with random LUFS gains produces plausible and useful audio for training.
    Section 3.3 mixes retrieved clips by linear summation; no perceptual validation of the mixtures is given, though the model's performance suggests it is workable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Seeing Sound: Assembling Sounds from Visuals for Audio-to-Image Generation." pith.science (2026). https://pith.science/paper/6RSYEFYJ

@misc{pith2026250105413,
  author       = {Pith},
  title        = {Pith review of: Seeing Sound: Assembling Sounds from Visuals for Audio-to-Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6RSYEFYJ}},
  note         = {Machine review of arXiv:2501.05413}
}
read the original abstract

Training audio-to-image generative models requires an abundance of diverse audio-visual pairs that are semantically aligned. Such data is almost always curated from in-the-wild videos, given the cross-modal semantic correspondence that is inherent to them. In this work, we hypothesize that insisting on the absolute need for ground truth audio-visual correspondence, is not only unnecessary, but also leads to severe restrictions in scale, quality, and diversity of the data, ultimately impairing its use in the modern generative models. That is, we propose a scalable image sonification framework where instances from a variety of high-quality yet disjoint uni-modal origins can be artificially paired through a retrieval process that is empowered by reasoning capabilities of modern vision-language models. To demonstrate the efficacy of this approach, we use our sonified images to train an audio-to-image generative model that performs competitively against state-of-the-art. Finally, through a series of ablation studies, we exhibit several intriguing auditory capabilities like semantic mixing and interpolation, loudness calibration and acoustic space modeling through reverberation that our model has implicitly developed to guide the image generation process.

Figures

Figures reproduced from arXiv: 2501.05413 by the authors.

Figure 1
Figure 1. Our audio-visual data modeling demonstrates versatile control for image generation through audio manipulations, including [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Visually-aligned (up) vs. sonically-aligned (down) image descriptions using [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Qualitative comparison of state-of-the-art audio-to-image generative models on four different datasets; Greatest Hits (top left), [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Examples of various semantics mixed together in the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 7
Figure 7. Figure 7: Chatter recorded in various acoustical places and their [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 6
Figure 6. Figure 6: Impact of semantic source separation on generation. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Examples of inherently silent images that are excluded [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Examples of sounding concepts inferred from the SAM dataset [ [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Additional examples of sounding concepts inferred from the SAM dataset [ [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: More generated images by our model trained on our proposed audio-visual dataset. Each of the examples comes with a file ID [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Limitations of our audio-to-image generation model [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Examples of visual limitations induced by current [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Some generated examples of SonicDiffusion [ [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: We perform KNN on AST embeddings and demon [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: Comparative output examples over seven epochs [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 33 canonical work pages

  1. [1]

    Sonicdiffusion: Audio-driven image generation and editing with pretrained diffusion models, 2024

    Burak Can Biner, Farrin Marouf Sofian, Umur Berkay Karakas ¸, Duygu Ceylan, Erkut Erdem, and Aykut Erdem. Sonicdiffusion: Audio-driven image generation and editing with pretrained diffusion models, 2024. 2, 3, 5, 7, 16, 17

    work page 2024

  2. [2]

    Large scale gan training for high fidelity natural image synthesis

    Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. ArXiv, abs/1809.11096, 2018. 2

    arXiv 2018

  3. [3]

    VGGSound: a large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zis- serman. VGGSound: a large-scale audio-visual dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 721–725. IEEE, 2020. 3, 6, 15

    work page 2020

  4. [4]

    Pixart-α: Fast training of dif- fusion transformer for photorealistic text-to-image synthesis,

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-α: Fast training of dif- fusion transformer for photorealistic text-to-image synthesis,

  5. [5]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 2

    arXiv 2018

  6. [6]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural informa- tion processing systems, 34:8780–8794, 2021. 2

    2021

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representa- tions, 2021. 2

    2021

  8. [8]

    FSD50K: an open dataset of human- labeled sound events

    Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. FSD50K: an open dataset of human- labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829–852, 2022. 5, 16

    work page 2022

Show all 50 references
  1. [9]

    Gemmeke, Daniel P

    Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human- labeled dataset for audio events. In 2017 IEEE Interna- tional Conference on Acoustics, Speech and Signal Proc...

    2017

  2. [10]

    ImageBind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. ImageBind: One embedding space to bind them all. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15180–15190, 2023. 3

    2023

  3. [11]

    AST: Audio Spectrogram Transformer

    Yuan Gong, Yu-An Chung, and James Glass. AST: Audio Spectrogram Transformer. In Proc. Interspeech 2021, pages 571–575, 2021. 5, 6, 7, 16

    2021

  4. [12]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), 2014. 2

    2014

  5. [13]

    Grimm and M

    E. Grimm and M. Sch ¨opping. Toward a recommendation for a european standard of peak and lkfs loudness levels.SMPTE Motion Imaging Journal, 119, 2010. 5, 7, 11, 16, 17

    2010

  6. [14]

    AudioCLIP: Extending clip to image, text and au- dio

    Andrey Guzhov, Federico Raue, J ¨orn Hees, and Andreas Dengel. AudioCLIP: Extending clip to image, text and au- dio. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 976–980. IEEE,

  7. [15]

    CLIPScore: a reference-free evaluation met- ric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: a reference-free evaluation met- ric for image captioning. In EMNLP, 2021. 5

    2021

  8. [16]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. In Neural Information Processing Systems, 2017. 5

    2017

  9. [17]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. arXiv preprint arxiv:2006.11239,

    2006 arXiv

  10. [18]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 29:3451–3460, 2021. 16

    2021

  11. [19]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 4401–4410, 2019. 2

    2019

  12. [20]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick. Segment anything. arXiv:2304.02643, 2023. 5, 11, 12, 13

    2023 arXiv

  13. [21]

    Sound-guided se- mantic video generation

    Seung Hyun Lee, Gyeongrok Oh, Wonmin Byeon, Chany- oung Kim, Won Jeong Ryoo, Sang Ho Yoon, Hyunjun Cho, Jihyun Bae, Jinkyu Kim, and Sangpil Kim. Sound-guided se- mantic video generation. In European Conference on Com- puter Vision (ECCV), pages 34–50. Springer, 2022. 2, 3, 6

    2022

  14. [22]

    Sound-guided semantic image manipulation

    Seung Hyun Lee, Wonseok Roh, Wonmin Byeon, Sang Ho Yoon, Chanyoung Kim, Jinkyu Kim, and Sangpil Kim. Sound-guided semantic image manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3377–3386, 2022. 2, 3

    2022

  15. [23]

    Learning visual styles from audio-visual associations

    Tingle Li, Yichen Liu, Andrew Owens, and Hang Zhao. Learning visual styles from audio-visual associations. InEu- ropean Conference on Computer Vision (ECCV), pages 235–

  16. [24]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 2, 3, 4, 11, 12, 13

    2023

  17. [25]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2017. 6

    2017

  18. [26]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022. 2

    2022

  19. [27]

    Visually indicated sounds

    Andrew Owens, Phillip Isola, Josh McDermott, Antonio Tor- ralba, Edward H Adelson, and William T Freeman. Visually indicated sounds. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2405–2413,

  20. [28]

    Jour- neydb: A benchmark for generative image understanding,

    Junting Pan, Keqiang Sun, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Jour- neydb: A benchmark for generative image understanding,

  21. [29]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022. 2

    2022 arXiv

  22. [30]

    Glue- gen: Plug and play multi-modal encoders for x-to-image generation

    Can Qin, Ning Yu, Chen Xing, Shu Zhang, Zeyuan Chen, Stefano Ermon, Yun Fu, Caiming Xiong, and Ran Xu. Glue- gen: Plug and play multi-modal encoders for x-to-image generation. In IEEE/CVF International Conference on Com- puter Vision (ICCV), pages 23085–23096, 2023. 2, 3, 7

    2023

  23. [31]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International Conference on Machine Learning...

    2021

  24. [32]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 2

    2020

  25. [33]

    Hierarchical text-conditional image gener- ation with CLIP latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with CLIP latents. arXiv preprint arXiv:2204.06125, 1 (2):3, 2022. 2

    2022 arXiv

  26. [34]

    Generative ad- versarial text to image synthesis

    Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Lo- geswaran, Bernt Schiele, and Honglak Lee. Generative ad- versarial text to image synthesis. In Proceedings of The 33rd International Conference on Machine Learning, pages 1060–1069, New York, New York, USA, 2016. PMLR. 2

    2016

  27. [35]

    High-resolution image syn- thesis with latent diffusion models, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models, 2021. 2, 3, 6, 17

    2021

  28. [36]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Informat...

    2022

  29. [37]

    Sound to visual scene genera- tion by audio-to-visual latent alignment, 2023

    Kim Sung-Bin, Arda Senocak, Hyunwoo Ha, Andrew Owens, and Tae-Hyun Oh. Sound to visual scene genera- tion by audio-to-visual latent alignment, 2023. 2, 3, 6, 7

    2023

  30. [38]

    Llama: Open and efficient foundation lan- guage models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aure- lien Rodriguez, Armand Joulin, Edouard Grave, and Guil- laume Lample. Llama: Open and efficient foundation la...

    2023

  31. [39]

    Cogvlm: Visual expert for pretrained language models, 2023

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2023. 2, 4, 12, 13

    2023

  32. [40]

    Wav2clip: Learning robust audio repre- sentations from clip

    Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio repre- sentations from clip. In ICASSP 2022-2022 IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 4563–4567. IEEE, 2022. 5

    2022

  33. [41]

    Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and ...

    2023

  34. [42]

    Attngan: Fine- grained text to image generation with attentional generative adversarial networks

    Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine- grained text to image generation with attentional generative adversarial networks. 2018 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 1316–1324,

    2018

  35. [43]

    Audiotoken: Adaptation of text-conditioned dif- fusion models for audio-to-image generation

    Guy Yariv, Itai Gat, Lior Wolf, Yossi Adi, and Idan Schwartz. Audiotoken: Adaptation of text-conditioned dif- fusion models for audio-to-image generation. arXiv preprint arXiv:2305.13050, 2023. 2, 3, 5, 7, 17

    2023 arXiv

  36. [44]

    A survey on segment anything model (sam): Vision foundation model meets prompt engineering, 2023

    Chaoning Zhang, Fachrina Puspitasari, Sheng Zheng, Chenghao Li, Yu Qiao, Taegoo Kang, Xinru Shan, Chen- shuang Zhang, Caiyan Qin, Franc ¸ois Rameau, Lik-Hang Lee, Sung-Ho Bae, and Choong Seon Hong. A survey on segment anything model (sam): Vision foundation model meets prompt ...

    2023

  37. [45]

    Stack- gan: Text to photo-realistic image synthesis with stacked generative adversarial networks

    Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiao- gang Wang, Xiaolei Huang, and Dimitris Metaxas. Stack- gan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 5908–5916,

    2017

  38. [46]

    Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiao- gang Wang, Xiaolei Huang, and Dimitris N. Metaxas. Stack- gan++: Realistic image synthesis with stacked generative ad- versarial networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8):1947–1962, 2019. 2

    1947

  39. [47]

    Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L. Berg. Visual to sound: Generating natural sound for videos in the wild. 2018 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 3550–3558,

    2018

  40. [48]

    , Building - The sound of footsteps on the pavement

    Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. DM- GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis , 2019. 2 10 A. Appendix This Appendix section is organized as follows. We first present some image sonification examples to better show- case th...

    2019

  41. [50]

    Pre-trained

    as our base architecture. We point out that this model in- stance was originally pre-trained on visual representations beyond realistic photographs [28]. All of its weights are used towards the initialization of our audio-to-image model with the exception of the initial audio ...

  42. [252]

    Springer, 2022. 2, 3, 6

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.