Pith. sign in

REVIEW 4 major objections 5 minor 46 references

SoundBrush: Sound as a Brush for Visual Scene Editing

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SoundBrush claims that audio alone can serve as a control signal for editing existing visual scenes, both 2D images and 3D scenes, without a text prompt.

desk verdict SoundBrush is a credible sound-guided editor with a substantial new dataset, but the 'sound as brush' claim is undercut by evaluation metrics that overlap with training and no test of same-category audio variation. read the letter →

arxiv 2501.00645 v1 pith:PTAQZQ7N submitted 2024-12-31 cs.CV cs.LGcs.SDeess.AS

Kim Sung-Bin , Kim Jun-Seong , Junseok Ko , Yewon Kim , Tae-Hyun Oh This is my paper · ORCID
classification cs.CVcs.LGcs.SDeess.AS
keywords sound-guidedimageeditinglatentdiffusionmodelaudio-to-tokenmappingmultimodalscene3Daudio-visualdatasetobjectinsertionalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SoundBrush claims that audio alone can serve as a control signal for editing existing visual scenes, both 2D images and 3D scenes, without a text prompt. The paper builds a large paired dataset of audio and before/after image pairs using off-the-shelf tools, then trains a mapping network that turns sound features into tokens in the textual embedding space of a latent diffusion model. If this works, sound becomes a brush for adding or changing visible objects and scenery while leaving the original image structure intact, something previous sound-guided methods did not achieve. The paper also reports that the model captures sound intensity and that it extends to 3D editing through a NeRF-based pipeline.

What carries the argument

The central object is the audio-token mapping network, a Transformer that takes CLAP audio features and a set of learnable query tokens and outputs five continuous tokens in the textual vocabulary of the CLIP text encoder. These tokens are passed through the text encoder to form the conditioning input for InstructPix2Pix's diffusion denoiser. The work it does is to turn an audio waveform into a text-aligned conditioning vector that the diffusion model already knows how to use, with the InfoNCE loss aligning the token-derived features to the target image's CLIP features and an ℓ1 penalty keeping the tokens well-distributed.

What would settle it

Take a large set of real images paired with sounds whose visible sources are human-annotated, run SoundBrush to insert the sounding object, then measure the spatial overlap between the inserted object and the annotated source location; if the model places the object at the true location no more often than chance, the claim that it accurately inserts sounding objects while preserving content is refuted.

Watch

Extended reading notes

Core claim

The central claim is that by learning to translate audio features into the text-token space of a latent diffusion model, a network can edit images so that the visual outcome matches the sound: inserting the sounding object (e.g., a dog or train) or altering the whole scene (e.g., rain) while preserving the original content and structure. The authors construct a dataset of 83,614 audio-paired editing samples, half synthetic and half real, and train a mapping network with a denoising objective plus an InfoNCE loss that aligns the audio-derived tokens with the visual features of the ground-truth edited image. They also adapt the diffusion model with low-rank adaptation to strengthen object insertion. The result is a model that, given an image and an audio clip, produces an edited image without any text instruction.

Load-bearing premise

The automatically generated before/after image pairs are assumed to be faithful ground truth for what the audio implies; if the synthetic prompt pairs or real-data inpainting introduce spurious changes, the model learns the wrong audio-to-edit mapping.

Editorial extensions

If this is right

  • Editing with sound becomes a supervised, training-based task rather than a generation-only one, so any pre-trained text-edit diffusion model can be re-targeted to audio.
  • Users can edit images by example sounds, including intensity changes that text prompts cannot capture, such as light versus heavy rain.
  • The same audio-conditioned editor can be plugged into NeRF-based pipelines to edit 3D scenes in a sound-consistent way.
  • The dataset construction pipeline provides a way to create audio-visual editing supervision without manual annotation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mapping network transfers across domains, the same audio-to-token module could be attached to other conditional diffusion models, such as video or multi-view generators, without retraining the whole backbone.
  • The volume-sensitivity finding suggests that audio-conditioned editors could be used to control not only what appears but how strongly it appears, offering a dial-like interface for scene attributes.
  • A stiffer test of the core claim would be to measure object insertion location against known sound source positions; the failure cases shown (train in the sky) suggest spatial grounding is the main unresolved weakness.
  • One could test whether the model generalizes to sounds outside the training categories; if the token space remains semantic, it should edit with unseen sounds, but if it overfits to categories, it will fail.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. SoundBrush proposes a sound-conditioned image editing model that maps CLAP audio features through a learned mapping network into CLIP text-token space, feeding them into an InstructPix2Pix latent diffusion model jointly fine-tuned with LoRA. Training data are produced by an automated pipeline: a fully synthetic subset generated from GPT-4 prompt pairs with Prompt-to-Prompt, and a real subset built from VGGSound by removing the sound source with sound source localization and LaMa inpainting. The paper reports quantitative comparisons (AVS, IIS, TVS, FID), a user study, ablations, and a 3D extension via InstructNeRF2NeRF, concluding that audio alone can insert sounding objects and manipulate scene appearance while preserving original content.

Significance. Sound-guided editing is a timely and underexplored problem, and the paper's dataset-construction pipeline and token-level audio-to-text mapping are practical contributions. If the central claim is correct, the method would go beyond style-only sound editors and demonstrate a useful form of cross-modal control. The paper also includes a user study and ablations that broadly support the qualitative behavior. However, the evidence is currently not strong enough to establish the distinctive claim that the method uses sound as a brush rather than as a category label: no experiment varies audio within a category, and the headline metrics overlap with the training objective and data filtering criteria.

major comments (4)
  1. [Experiments] The central claim that "sound is used as a brush" is not distinguished from a category-label shortcut. All evaluation and training material is organized around VGGSound categories: the synthetic target prompts are generated from sound category keywords, the real subset is labeled with the same categories, and the AVS/TVS metrics reward category-level agreement between audio and edited image. No experiment tests whether two different audio clips from the same category (e.g., two different dog barking sounds) produce different edits. The volume experiments in Fig. 4 and Fig. S3 are the only evidence against the shortcut, but they are qualitative and not statistically tested. I request a controlled same-category experiment: multiple audio clips per category, with a quantitative measure of edit diversity (e.g., pairwise LPIPS/CLIP distance between edited images, or a human study where raters judge whether the edited images plausibly match each specific audio clip). Without such a test, the method may reduce to a text-prompt editor whose prompt is the category name.
  2. [Dataset construction] The headline AVS metric is partly circular. The fully synthetic subset is filtered by requiring ImageBind audio-visual similarity between the target image and the sound, and the real subset is filtered using the same ImageBind similarity to discard inpainted images that still contain the sound source; evaluation then measures AVS in ImageBind space between the input audio and the edited image. Because the training target images were selected to score highly on this exact metric, high AVS values are partially by construction. Please report AVS on evaluation data that was not filtered with ImageBind, or use a different held-out metric (e.g., human ratings of audio-image correspondence). In addition, the InfoNCE loss in Eq. (2) explicitly pulls the audio-token-derived CLIP features toward the ground-truth image's CLIP visual features, while IIS measures CLIP similarity between edited and ground-truth images; the paper should explain why IIS is not redundantly inflated by this training objective.
  3. [3D experiments] The 3D editing claim is supported only by qualitative examples (Fig. 6 and Fig. S5). Unlike the 2D experiments, no quantitative metric or user evaluation is provided for 3D consistency or content preservation. Because the paper's abstract and introduction explicitly claim that the framework extends to 3D scene editing, this is a load-bearing part of the contribution. Please add at least a quantitative multi-view consistency measure (e.g., per-view IIS/AVS averages, or a user study on 3D samples), or clearly reframe the 3D results as a preliminary demonstration.
  4. [Discussion] The real-data subset depends on the correctness of sound source localization and LaMa inpainting, and the paper itself acknowledges that human-related sound edits show low quality due to dataset bias (Discussion, Fig. 7). This limitation is not quantified. I ask for a per-category analysis of training-data quality (e.g., fraction of pairs passing each filtering threshold, and failure rates of SSL/inpainting per category) and a discussion of how these failures affect the final editing behavior. This is important because the supervised training signal is only as good as the automatically constructed before/after pairs.
minor comments (5)
  1. [Method] There is a typo "soley" in the paragraph before Eq. (2); also Eq. (3) writes the L1 regularization term as "λℓ1 |V A|1", which should be parenthesized for clarity.
  2. [Experiments] The text says "further increasing to ten (C) begins to degrade performance," but in Table 1 the ten-token configuration is labeled (D), while (C) is the five-token configuration with LNCE. Please correct the label references.
  3. [Experiments] The user study reports means without error bars or statistical significance testing. Given the strong claim that SoundBrush "significantly outperforms" other methods in Q3, please provide confidence intervals and a paired significance test.
  4. [Appendix] The filtering description in the Appendix states that pairs are discarded if "the feature similarity between the audio and the inpainted image exceeds that between the audio and the before-inpainted image"; this wording is ambiguous because the inpainted image is the before-editing image in the real subset. Please clarify which two images are being compared.
  5. [Experiments] The caption of Fig. 5 contains a typo, "comaprison"; also, the table in Fig. 5(a) is not formatted as a proper LaTeX table in the text, which makes the numeric comparison harder to read.

Circularity Check

2 steps flagged · score 5.0 of 10

Evaluation metrics overlap with the training-data filters and training objectives, making the quantitative support for the central claim partially circular; the method itself retains independent architectural content.

  1. fitted input called prediction [Method (Constructing the Training Dataset, Fully synthetic subset) and Experiments (Evaluation metrics)]
    "Additionally, we incorporate ImageBind (Girdhar et al. 2023) to validate whether the target image accurately represents the sound event by measuring the audio-visual feature similarity in the ImageBind space and excluding those that do not meet the required threshold. ... For objective metrics, we measure audio-visual similarity (A VS) using ImageBind (Girdhar et al. 2023) space by computing the feature similarity between the input audio and the edited image."

    The paper's headline quantitative evidence that edited images 'best match' the input audio is the AVS score, but AVS is the same ImageBind audio-visual similarity used as an inclusion threshold (0.2, per the Appendix) when constructing the training targets. The model is trained with an LDM reconstruction loss to output these pre-filtered targets, so on a test set built by the identical pipeline, the AVS score partly re-measures the data filter rather than an independent audio-to-image mapping. The result is not a full tautology because the model must still generalize, but the central evaluation metric is not independent of the supervision signal.

  2. other [Method (Learning objectives, Eq. 2) and Experiments (Evaluation metrics)]
    "We employ the InfoNCE loss (Oord, Li, and Vinyals 2018), treating pairs of qV and qI as positive and those from different pairs as negative in the batch N. ... Image-image similarity (IIS) is assessed in the CLIP space by comparing the feature similarity between edited and ground-truth images."

    The InfoNCE objective explicitly trains audio-token features to be CLIP-similar to the ground-truth target image (qV versus qI), while the LDM loss trains the edited image to approximate that same target. The IIS metric then measures CLIP similarity between the edited image and the ground-truth target. Consequently, the reported content-preservation score is the same family of CLIP visual similarity that the training objective directly maximizes, making the IIS advantage over baselines at least partly a consequence of the training loss rather than an independent property of audio-conditioned editing.

full rationale

The derivation chain in SoundBrush is a supervised pipeline: GPT-4 and Prompt-to-Prompt construct training pairs, CLIP and ImageBind filters remove noisy pairs, a mapping network translates CLAP audio features into LDM text-token space, and the LDM is fine-tuned with LoRA. No load-bearing self-citation chain appears; the authors' prior generation papers are cited only as related work, and no uniqueness theorem is imported. The main circularity concern is that the evaluation metrics are not independent of the training-data construction and training losses. The AVS metric is the same ImageBind audio-visual similarity used to filter training pairs, the IIS metric is the same CLIP visual-similarity family that the InfoNCE loss directly optimizes against the ground-truth target, and the TVS metric uses the category names that generated the target prompts. These overlaps mean the quantitative results partly re-measure the selection criteria and training objectives rather than independently confirming that sound, rather than category labels, drives the edit. The paper does acknowledge real limitations, including poor spatial insertion and low-quality human edits, which further qualify the central claim. However, the model must still generalize to held-out test splits, and the qualitative volume-change experiments and human study provide some independent evidence, so the circularity is partial rather than total. Score 5 reflects that the evaluation of the central claim is significantly contaminated by training-set and training-loss overlap, while the method itself has independent architectural content.

Assumptions & free parameters 6 free parameters · 7 assumptions · 1 invented entities

The method leans on a large stack of pretrained models and automatically generated supervision. The most consequential unstated load is the assumption that synthetic and inpainted image pairs are faithful audio-conditioned ground truth, and that CLIP and ImageBind similarity are valid proxies for edit quality. Several hyperparameters, including the two loss weights, are undisclosed.

free parameters (6)
  • Number of audio tokens = 5
    Selected via ablation in Table 1; 1 token underperforms and 10 tokens degrade performance, so 5 is chosen by hand from the validation results.
  • Contrastive loss weight (lambda_NCE) = not reported
    Hyperparameter in Eq. (3) that controls the CLIP-space alignment driving the reported IIS gains, but its value is never stated in the paper.
  • L1 regularization weight (lambda_L1) = not reported
    Hyperparameter in Eq. (3) applied to audio tokens; the value is not disclosed.
  • Dataset filtering thresholds = 0.2 CLIP directional, 0.7 CLIP feature, 0.2 ImageBind AVS, 0.7 IIS
    Ad hoc thresholds in the dataset construction pipeline determine which training samples survive; they also use the same CLIP and ImageBind spaces as the evaluation metrics.
  • LoRA rank and alpha = rank 2, alpha 2
    Chosen architectural hyperparameters for the InstructPix2Pix fine-tune, with no sensitivity analysis reported.
  • Prompt-to-Prompt p-value and number of noises = p=0.5, five noises
    Fixed during synthetic dataset generation after the authors observed little variation with different p-values.
assumptions (7)
  • standard math The latent diffusion and denoising objective of InstructPix2Pix, Eq. (1), is a valid and sufficient base for sound-conditioned editing.
    The method builds directly on the LDM and InstructPix2Pix formulation without re-deriving or questioning it.
  • domain assumption Pretrained CLAP, CLIP, and ImageBind embeddings meaningfully align audio, text, and images in shared spaces.
    CLAP features are the input to the mapping network, CLIP text and image spaces supervise and evaluate the model, and ImageBind is used for dataset filtering and evaluation.
  • domain assumption Prompt-to-Prompt with GPT-4-generated prompt pairs produces before/after images whose only meaningful difference is the audio-related edit.
    The fully synthetic subset is built on this assumption; if the generated pairs differ in unrelated ways, the model learns spurious correlations.
  • domain assumption Sound source localization followed by LaMa inpainting on VGGSound frames removes the sounding object cleanly and leaves a valid before image.
    The real-data subset treats the inpainted image as ground truth before the audio-driven edit; imperfect inpainting would corrupt training targets.
  • domain assumption GPT-4-generated source-target prompt pairs faithfully describe audio-driven visual edits.
    The synthetic prompts are generated by GPT-4 with few-shot examples and are not manually verified beyond automatic filtering.
  • domain assumption The frozen CLIP text encoder can interpret learned continuous audio tokens as meaningful textual conditions.
    The mapping network outputs tokens that are fed directly into CLIP's text encoder; this is a central architectural premise with no external validation.
  • domain assumption VGGSound categories and test splits are representative of in-the-wild sounds and provide a fair evaluation distribution.
    Both training and evaluation draw from VGGSound category names, and only 20 categories are used for evaluation, limiting scope.
invented entities (1)
  • Audio tokens in the CLIP textual space
    purpose: A learned sequence of tokens that carries audio information into the frozen CLIP text encoder, replacing text instructions for InstructPix2Pix.
    The tokens are trainable parameters learned on the constructed dataset. The only evidence of their validity comes from the paper's own metrics, which overlap with training and filtering spaces.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SoundBrush: Sound as a Brush for Visual Scene Editing." pith.science (2026). https://pith.science/paper/PTAQZQ7N

@misc{pith2026250100645,
  author       = {Pith},
  title        = {Pith review of: SoundBrush: Sound as a Brush for Visual Scene Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PTAQZQ7N}},
  note         = {Machine review of arXiv:2501.00645}
}
read the original abstract

We propose SoundBrush, a model that uses sound as a brush to edit and manipulate visual scenes. We extend the generative capabilities of the Latent Diffusion Model (LDM) to incorporate audio information for editing visual scenes. Inspired by existing image-editing works, we frame this task as a supervised learning problem and leverage various off-the-shelf models to construct a sound-paired visual scene dataset for training. This richly generated dataset enables SoundBrush to learn to map audio features into the textual space of the LDM, allowing for visual scene editing guided by diverse in-the-wild sound. Unlike existing methods, SoundBrush can accurately manipulate the overall scenery or even insert sounding objects to best match the audio inputs while preserving the original content. Furthermore, by integrating with novel view synthesis techniques, our framework can be extended to edit 3D scenes, facilitating sound-driven 3D scene manipulation. Demos are available at https://soundbrush.github.io/.

Figures

Figures reproduced from arXiv: 2501.00645 by the authors.

Figure 1
Figure 1. Sound-guided visual scene editing. We propose SoundBrush, a model that can modify visual scenes by adding sounding objects and adjusting the scene to align with the input sound (left). Furthermore, it can be extended to edit 3D visual scenes using the input sound (right). Abstract We propose SoundBrush, a model that uses sound as a brush to edit and manipulate visual scenes. We extend the generative capabilities of … view at source ↗
Figure 2
Figure 2. Our proposed approach. We start by designing an automatic dataset-construction pipeline as in (a). The dataset is constructed with a fully synthetic subset, involving synthetically generated image pairs paired with audio, and real data involved subset, involving real audio and images. Using this dataset, we train SoundBrush to effectively translate audio features into audio tokens, which can then be used as control … view at source ↗
Figure 3
Figure 3. Qualitative comparison. We compare our model with existing sound-guided visual scene editing methods and demonstrate that our model can edit visual scenes using diverse in-the-wild audio while preserving the rest of the content unchanged. Experiments We validate the editing power of our proposed SoundBrush both qualitatively and quantitatively. We begin by outlining the experimental setup, which includes the dataset… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Quantitative comparison and user study. We compare SoundBrush with existing methods and demonstrate that it outper￾forms them in overall metrics (a). We also provide human evaluation results that align with the quantitative comparisons (b). Num. of tokens LNCE AVS (↑) …
Figure 6
Figure 6. Figure 6: 3D visual scene editing results. We compare SoundBrush with InstructAny2Pix in 3D scene editing. SoundBrush effectively edits the input image while preserving the original geometric properties, achieving successful editing in 3D scenes. In contrast, InstructAny2Pix str…
Figure 7
Figure 7. Figure 7: Examples of failure cases. We observe that our model occasionally inserts sounding objects into the scene without proper spatial understanding (a), or it tends to insert humans into the scene in incomplete forms (b). from multichannel audio could address these issues, …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 38 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; and et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; and et al. 2024. GPT-4 Technical Report. arXiv:2303.08774

    arXiv 2024

  4. [4]

    C.; Sofian, F

    Biner, B. C.; Sofian, F. M.; Karakaş, U. B.; Ceylan, D.; Erdem, E.; and Erdem, A. 2024. SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models. arXiv preprint arXiv:2405.00878

    arXiv 2024

  5. [5]

    Brooks, T.; Holynski, A.; and Efros, A. A. 2023. Instructpix2pix: Learning to follow image editing instructions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2023

  6. [6]

    Chen, H.; Xie, W.; Vedaldi, A.; and Zisserman, A. 2020. VGGSound: A Large-scale Audio-Visual Dataset. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

    work page 2020

  7. [7]

    Chen, S.; Wu, Y.; Wang, C.; Liu, S.; Tompkins, D.; Chen, Z.; and Wei, F. 2022. Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:2212.09058

    arXiv 2022

  8. [8]

    H.; Chechik, G.; and Cohen-Or, D

    Gal, R.; Patashnik, O.; Maron, H.; Bermano, A. H.; Chechik, G.; and Cohen-Or, D. 2022. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics (SIGGRAPH)

    work page 2022

Show all 46 references
  1. [9]

    A.; Ecker, A

    Gatys, L. A.; Ecker, A. S.; and Bethge, M. 2016. Image style transfer using convolutional neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2016

  2. [10]

    V.; Joulin, A.; and Misra, I

    Girdhar, R.; El-Nouby, A.; Liu, Z.; Singh, M.; Alwala, K. V.; Joulin, A.; and Misra, I. 2023. Imagebind: One embedding space to bind them all. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2023

  3. [11]

    Hao, W.; Zhang, Z.; and Guan, H. 2018. Cmcgan: A uniform framework for cross-modal visual-audio mutual generation. In AAAI Conference on Artificial Intelligence (AAAI)

    2018

  4. [12]

    A.; Holynski, A.; and Kanazawa, A

    Haque, A.; Tancik, M.; Efros, A. A.; Holynski, A.; and Kanazawa, A. 2023. Instruct-nerf2nerf: Editing 3d scenes with instructions. In IEEE International Conference on Computer Vision (ICCV)

    2023

  5. [13]

    Hertz, A.; Mokady, R.; Tenenbaum, J.; Aberman, K.; Pritch, Y.; and Cohen-Or, D. 2023. Prompt-to-prompt image editing with cross attention control. In International Conference on Learning Representations (ICLR)

    2023

  6. [14]

    J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

    Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR)

    2022

  7. [15]

    Huang, X.; Liu, M.-Y.; Belongie, S.; and Kautz, J. 2018. Multimodal unsupervised image-to-image translation. In European Conference on Computer Vision (ECCV)

    2018

  8. [16]

    Huberman-Spiegelglas, I.; Kulikov, V.; and Michaeli, T. 2024. An edit friendly ddpm noise space: Inversion and manipulations. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2024

  9. [17]

    Isola, P.; Zhu, J.-Y.; Zhou, T.; and Efros, A. A. 2017. Image-to-image translation with conditional adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2017

  10. [18]

    Karras, T.; Aittala, M.; Aila, T.; and Laine, S. 2022. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems (NeurIPS)

    2022

  11. [19]

    Kawar, B.; Zada, S.; Lang, O.; Tov, O.; Chang, H.; Dekel, T.; Mosseri, I.; and Irani, M. 2023. Imagic: Text-based real image editing with diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2023

  12. [20]

    Kim, G.; Kwon, T.; and Ye, J. C. 2022. Diffusionclip: Text-guided diffusion models for robust image manipulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2022

  13. [21]

    H.; Roh, W.; Byeon, W.; Yoon, S

    Lee, S. H.; Roh, W.; Byeon, W.; Yoon, S. H.; Kim, C.; Kim, J.; and Kim, S. 2022. Sound-guided semantic image manipulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2022

  14. [22]

    Li, S.; Singh, H.; and Grover, A. 2023. InstructAny2Pix: Flexible Visual Editing via Multimodal Instruction Following. arXiv preprint arXiv:2312.06738

    2023 arXiv

  15. [23]

    Li, T.; Liu, Y.; Owens, A.; and Zhao, H. 2022. Learning visual styles from audio-visual associations. In European Conference on Computer Vision (ECCV)

    2022

  16. [24]

    Liu, M.-Y.; Breuel, T.; and Kautz, J. 2017. Unsupervised image-to-image translation networks. Advances in neural information processing systems

    2017

  17. [25]

    L \"u ddecke, T.; and Ecker, A. 2022. Image segmentation using text and image prompts. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2022

  18. [26]

    Mildenhall, B.; Srinivasan, P.; Tancik, M.; Barron, J.; Ramamoorthi, R.; and Ng, R. 2020. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV)

    2020

  19. [27]

    A.; and Darrell, T

    Narasimhan, M.; Ginosar, S.; Owens, A.; Efros, A. A.; and Darrell, T. 2022. Strumming to the Beat: Audio-Conditioned Contrastive Video Textures. In IEEE Winter Conference on Applications of Computer Vision (WACV)

    2022

  20. [28]

    T.; Rubinstein, M.; and Matusik, W

    Oh, T.-H.; Dekel, T.; Kim, C.; Mosseri, I.; Freeman, W. T.; Rubinstein, M.; and Matusik, W. 2019. Speech2face: Learning the face behind a voice. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2019

  21. [29]

    Oord, A. v. d.; Li, Y.; and Vinyals, O. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748

    2018 arXiv

  22. [30]

    Park, S.; Senocak, A.; and Chung, J. S. 2024. Can CLIP Help Sound Source Localization? In IEEE Winter Conference on Applications of Computer Vision (WACV)

    2024

  23. [31]

    Qin, C.; Yu, N.; Xing, C.; Zhang, S.; Chen, Z.; Ermon, S.; Fu, Y.; Xiong, C.; and Xu, R. 2023. Gluegen: Plug and play multi-modal encoders for x-to-image generation. In IEEE International Conference on Computer Vision (ICCV)

    2023

  24. [32]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML)

    2021

  25. [33]

    Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022. High-resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2022

  26. [34]

    L.; Ghasemipour, K.; Gontijo Lopes, R.; Karagol Ayan, B.; Salimans, T.; et al

    Saharia, C.; Chan, W.; Saxena, S.; Li, L.; Whang, J.; Denton, E. L.; Ghasemipour, K.; Gontijo Lopes, R.; Karagol Ayan, B.; Salimans, T.; et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Syste...

    2022

  27. [35]

    Salamon, J.; Jacoby, C.; and Bello, J. P. 2014. A dataset and taxonomy for urban sound research. In ACM International Conference on Multimedia (MM)

    2014

  28. [36]

    H.; Pan, J.; Yan, H.; Zhang, W.; Tan, V

    Shi, Y.; Xue, C.; Liew, J. H.; Pan, J.; Yan, H.; Zhang, W.; Tan, V. Y.; and Bai, S. 2024. Dragdiffusion: Harnessing diffusion models for interactive point-based image editing. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2024

  29. [37]

    Y.; Kim, J.; and Kim, J.-K

    Shim, J. Y.; Kim, J.; and Kim, J.-K. 2021. S2I-Bird: Sound-to-Image Generation of Bird Species using Generative Adversarial Networks. In International Conference on Pattern Recognition (ICPR)

    2021

  30. [38]

    Sung-Bin, K.; Senocak, A.; Ha, H.; and Oh, T.-H. 2024. Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignment. arXiv preprint arXiv:2412.06209

    2024 arXiv

  31. [39]

    Sung-Bin, K.; Senocak, A.; Ha, H.; Owens, A.; and Oh, T.-H. 2023. Sound to visual scene generation by audio-to-visual latent alignment. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2023

  32. [40]

    Suvorov, R.; Logacheva, E.; Mashikhin, A.; Remizova, A.; Ashukha, A.; Silvestrov, A.; Kong, N.; Goka, H.; Park, K.; and Lempitsky, V. 2022. Resolution-robust large mask inpainting with fourier convolutions. In IEEE Winter Conference on Applications of Computer Vision (WACV)

    2022

  33. [41]

    Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; and Wojna, Z. 2016. Rethinking the inception architecture for computer vision. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2016

  34. [42]

    Tumanyan, N.; Geyer, M.; Bagon, S.; and Dekel, T. 2023. Plug-and-play diffusion features for text-driven image-to-image translation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2023

  35. [43]

    Wu, Y.; Chen, K.; Zhang, T.; Hui, Y.; Berg-Kirkpatrick, T.; and Dubnov, S. 2023. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

    2023

  36. [44]

    Yang, B.; Gu, S.; Zhang, B.; Zhang, T.; Chen, X.; Sun, X.; Chen, D.; and Wen, F. 2023. Paint by example: Exemplar-based image editing with diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2023

  37. [45]

    Yariv, G.; Gat, I.; Wolf, L.; Adi, Y.; and Schwartz, I. 2023. Audiotoken: Adaptation of text-conditioned diffusion models for audio-to-image generation. In Conference of the International Speech Communication Association (INTERSPEECH)

    2023

  38. [46]

    Zhu, J.-Y.; Park, T.; Isola, P.; and Efros, A. A. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In IEEE International Conference on Computer Vision (ICCV)

    2017

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.