Pith. sign in

REVIEW 3 major objections 6 minor 57 references

Audio-Guided Visual Editing with Complex Multi-Modal Prompts

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Audio can be converted into a diffusion-model text-prompt vector with a single matrix inversion, enabling zero-shot image and video editing from sound, alone or combined with text.

desk verdict The patch-selection fusion is a real win, but the unstated learned mapping M makes the audio integration unverifiable. read the letter →

arxiv 2508.20379 v1 pith:XUUTGXMM submitted 2025-08-28 cs.CV

Hyeonyu Kim , Seokhoon Jeong , Seonghee Han , Chanhyuk Choi , Taehwan Kim This is my paper
classification cs.CV
keywords audio-guidededitingdiffusionmodelsmulti-modalpromptstraining-freeCLIPspacemappingnoisebranchingadaptivepatchselectionimageandvideo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to make audio a first-class editing prompt for diffusion models, alongside text, with no training. It claims that a pre-trained aligned multi-modal encoder's audio embedding can be moved into Stable Diffusion's prompt space by a learned linear map and a matrix inversion, so a sound such as splashing water or thunder behaves like an invisible caption. For tasks that require text and audio together, it replaces naive noise averaging with separate noise branching per prompt and per-patch selection of the strongest residual relative to the inversion prompt. On audio-augmented versions of two existing benchmarks, the method is reported to beat text-only and caption-based baselines on audio-alignment and integrated-prompt metrics. If true, this gives editors a zero-shot way to specify effects through sound that text alone cannot capture.

What carries the argument

The work's load-bearing objects are two training-free bridges. The first is the linear audio-to-text bridge: a learned bias-free map from Stable Diffusion's pooled text embedding to an aligned multi-modal encoder's shared space, inverted by a Tikhonov pseudo-inverse so a sound becomes a pseudo text prompt. The second is the fusion rule: separate noise branching computes the noise prediction for each prompt, and adaptive patch-wise selection adds to the inversion noise, at each spatial patch, the residual with the largest channel-wise L2 norm, preserving detail that averaging destroys.

What would settle it

Run the audio-to-prompt mapping on a set of sounds, use the recovered pseudo-prompt alone to drive Stable Diffusion, and measure whether images generated from the mapped vector are more CLIP-similar to each sound's true text caption than to randomly matched captions; if the mapped embeddings are no closer to their true captions than to unrelated ones, the linear-invertibility premise fails and the audio channel is not actually carrying semantics.

Watch

Extended reading notes

Core claim

The central claim is that audio can be made a first-class prompt for Stable-Diffusion-based editing by algebra rather than by training. The authors find that a pretrained aligned multi-modal encoder's audio embedding can be transported into Stable Diffusion's CLIP text space with a no-bias linear map M and then recovered as a pseudo prompt by solving a Tikhonov-regularized least-squares problem (their Eq. 4). For multiple simultaneous prompts, they reject naive averaging of the denoiser's noise estimates, which collapses high-frequency detail; instead they branch the frozen denoiser once per prompt, subtract the inversion prompt's noise, and at each spatial patch keep the largest-magnitude r

Load-bearing premise

The load-bearing premise is that a pretrained audio encoder's embeddings live in a space so tightly linearly related to the diffusion model's pooled text-embedding space that the pseudo-inverse of a learned no-bias map returns a semantically faithful text-prompt vector; if that linear relation is weak or the audio-text alignment is inaccurate, the audio channel fails no matter how the noise predictions are fused.

Editorial extensions

If this is right

  • Any Stable-Diffusion editor built on DDIM inversion with feature and self-attention injection can accept audio prompts by inserting this mapping, with no paired training data.
  • Sound properties beyond semantic labels, such as volume, texture, and dynamics, can steer edits; the paper shows louder audio produces stronger effects.
  • Text and audio prompts can be combined in a single sampling pass, so an editor can specify a subject in text and an ambient sound in audio without training a condition adapter.
  • The same patch-wise maximum-residual rule is a generic parameter-free way to fuse any set of conditioning signals beyond text and audio, such as depth or edge maps.
  • The new audio-augmented benchmarks give future audio-guided editing methods a common testbed with 300 image-audio pairs and 267 video-audio pairs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the linear-map assumption holds broadly, the learned matrix M is itself a compressed audio-text alignment; learning M deliberately on paired audio-text data would be a natural way to test how much semantic weight the pseudo-inverse carries, since the paper does not specify how M was obtained.
  • The per-patch maximum rule implicitly assumes different prompts dominate spatially disjoint patches; when two prompts target the same region, the stronger residual wins outright, so the fused result may inherit one prompt rather than blending both.
  • The approach is bounded by the audio encoder's pretraining: sounds that co-occur with text in the encoder's training data will map cleanly, while rare or novel sounds may degrade to noise regardless of the matrix inversion.
  • A direct extension would be to use the recovered pseudo text prompt for text-to-audio or audio-to-video generation tasks, isolating whether the mapped vector is semantically meaningful outside the editing setup.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes an audio-guided visual editing framework that integrates CoDi audio embeddings into Stable Diffusion without additional training, using a learned linear mapping (Eq. 3) and its Tikhonov-regularized pseudo-inverse (Eq. 4). It also introduces separate noise branching with adaptive patch-wise selection (Eq. 6) to combine multiple text and audio prompts. The authors contribute two new benchmarks, PIEBench-multi and DAVIS-multi, and report quantitative and user-study results showing improvement over text-only and audio-captioning baselines across image and video editing tasks.

Significance. If the central mapping is sound, the paper addresses a realistic gap: audio carries information that text prompts often cannot convey, and doing so without fine-tuning a diffusion model is practically attractive. The separate noise branching idea is simple and plausibly general, and the authors provide new benchmarks that could support future research. The paper also includes an ablation study and a user study, which strengthens the empirical contribution. However, the load-bearing component—the learned linear map M that transfers audio semantics into Stable Diffusion's prompt space—is not described, implemented, or validated. This makes the core claim unverifiable and the 'training-free' assertion ambiguous. The significance of the paper depends critically on resolving this gap.

major comments (3)
  1. [Sec. 3.2, Eqs. (3) and (4)] The paper states 'M : C_SD -> C_CLIP is a learned linear mapping with no bias' but never specifies how M was obtained: what data was used, what loss was optimized, or whether M is a pretrained component from another model. If M was fit for this paper, the framework is not training-free; if it is taken from an existing model, that model must be cited. Furthermore, Eq. (3) includes L2 normalization, making the forward map non-injective, and Eq. (4) arbitrarily scales c_A by ||c_inv|| before applying the pseudo-inverse. The manuscript provides no derivation for this scaling, no condition-number or rank information for M, and no evidence that the inverse maps audio embeddings to semantically corresponding text embeddings. This is the linchpin of the method: if M is arbitrary or poorly conditioned, the audio signal does not actually enter the diffusion process, and the reported benefits over
  2. [Sec. 4.4, CLIP_Audio metric] The description states that for audio editing prompts, CLIP_Audio is computed 'using category names for audio editing prompts.' This means the score is CLIP similarity between the edited image and a text label (e.g., 'thunder'), not between the output and the actual audio signal. CLIP_Audio therefore measures class-label alignment, not fidelity to the specific audio input. To support the claim that audio content is being transferred, the paper needs an audio-based metric (e.g., CLAP score between the original audio and an audio-like representation of the output) or at least a user study where participants actually listen to the audio. The current user study setup is not described in enough detail to determine whether audio was presented to the raters; please clarify the protocol.
  3. [Sec. 3.3, Eq. (6)] The proposed adaptive patch selection uses the per-pixel maximum of residual noise magnitudes. The paper claims this 'preserves high-frequency detail and prevents destructive interference' (Sec. 3.3), but no theoretical justification is given, and the only ablation compares against simple averaging. Max-norm selection is a strong heuristic; it could favor one prompt's effect at each spatial location or amplify outlier noise. The authors should compare against alternative fusion strategies (e.g., attention-weighted averaging, channel-wise pooling, or learned weights) and provide an analysis of why per-pixel max is the right choice. This is not as load-bearing as the M issue, but it affects the validity of the multi-prompt claim.
minor comments (6)
  1. [Sec. 3.2] The phrase 'training-free integration' is used despite M being a 'learned linear mapping.' Please clarify whether 'training-free' means 'no fine-tuning of the diffusion model' or 'no learning whatsoever.' If M comes from a pretrained model, say so explicitly.
  2. [Sec. 4.1] The implementation uses Stable Diffusion v1.5 and A100 GPUs with seed 1. Providing the exact code or a link to a public repository would aid reproducibility; at minimum, list the hyperparameters for the inversion and sampling steps.
  3. [Sec. 4.5] In Tables 1 and 2, the 'Ours' rows sometimes lack user-study columns (e.g., Table 1, A2I-edit row). This makes the comparison of user-study results incomplete. Please either report the numbers or clearly indicate why they are omitted.
  4. [Sec. 4.2] The benchmark construction selects '28 classes of audio suitable for editing by filtering out low-quality content from VGGSound.' The filtering criteria are not described. Please specify the filtering procedure to make the benchmark creation transparent and reproducible.
  5. [Sec. 4.7] The limitation statement acknowledges that the mapping assumes close alignment between audio and textual counterparts, but this is precisely the assumption that needs to be tested. The reference to [25] is appropriate; please elaborate on how the observed modality gap might affect the proposed method in practice.
  6. [General] Some captions in Figures 2 and 3 contain text that is difficult to read (e.g., overlapping labels). Please enlarge or reposition labels for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; the audio mapping and noise fusion are stated as algebraic definitions/inversions, and the evaluation is empirical.

full rationale

The paper's derivation chain is largely definitional rather than circular. Eq. (3) defines a linear map M from SD text-pooled embeddings to CoDi's aligned CLIP space, and Eq. (4) is the Tikhonov-regularized algebraic inverse of that map. The resulting text-space embedding is obtained by applying the inverse map to an audio embedding; this is a construction, not a prediction derived from the benchmarks. No target result is shown to be an input in disguised form: the benchmarks (PIEBench-multi, DAVIS-multi) are newly assembled, and the reported CLIP/user-study numbers are empirical outcomes not implied by Eqs. (3)-(6) alone. The sole self-citation ([23], by two of the present authors) appears in related work on audio-to-image generation and is not load-bearing for the method. The stated limitation in Sec. 4.7 ('our mapping function assumes that audio features align closely with textual counterparts, which may not always be accurate') is a validity caveat, not a circularity. The main genuine concern is that M is described only as 'a learned linear mapping with no bias' with no provenance or training data; this is an unspecified/possibly non-training-free component and a reproducibility gap, but the paper does not claim Eq. (4) is derived from anything other than Eq. (3), so it does not satisfy the standard for an exhibited reduction. Score 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The framework introduces no new physical entities. The central contribution rests on the learned map M, the assumption that audio and text embeddings are linearly relatable, and the inherited structural injection of PnP/TokenFlow. The paper provides no details for M, so these axioms are the main load-bearing unverified components.

free parameters (3)
  • M (linear map from SD pooled text space to CoDi aligned CLIP space) = not disclosed
    Needed in Equation (3); Equation (4) is its pseudo-inverse. Without knowing how M was learned, the audio integration cannot be reproduced and the 'training-free' claim is unverifiable.
  • Lambda (Tikhonov regularization) = 1e-5
    Chosen by hand in Equation (4); small effect unless M^T M is near singular.
  • Audio token repetition count = unspecified ('certain times')
    The inverted feature is replicated and concatenated with cinv special tokens; the count affects prompt length and conditioning strength, but is not specified.
assumptions (5)
  • domain assumption CoDi's audio encoder and CLIP text encoder share a linear correspondence M
    Equation (3) assumes cCLIP = M cSD,pooled / norm; no evidence is given beyond pretraining claims.
  • ad hoc to paper Pseudo-inverse of M transfers semantic content from audio to SD text space
    Equation (4) assumes minimizing the linear residual yields a usable SD conditioning vector; the paper says 'minor precision errors' but gives no supporting validation.
  • domain assumption Stable Diffusion's prompt encoder accepts concatenated special-token sequences
    The inverted feature is concatenated with the first and last special tokens of cinv, with no validation of length or placement.
  • domain assumption PnP-Diffusion and TokenFlow structural injection preserves source content during multi-prompt editing
    Inherited from prior editing methods; not re-validated in this paper.
  • ad hoc to paper Audio features align closely with textual counterparts
    Stated as a limitation in Section 4.7, citing [25]; if false, Equation (4) cannot map audio semantics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Audio-Guided Visual Editing with Complex Multi-Modal Prompts." pith.science (2026). https://pith.science/paper/XUUTGXMM

@misc{pith2026250820379,
  author       = {Pith},
  title        = {Pith review of: Audio-Guided Visual Editing with Complex Multi-Modal Prompts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XUUTGXMM}},
  note         = {Machine review of arXiv:2508.20379}
}
read the original abstract

Visual editing with diffusion models has made significant progress but often struggles with complex scenarios that textual guidance alone could not adequately describe, highlighting the need for additional non-text editing prompts. In this work, we introduce a novel audio-guided visual editing framework that can handle complex editing tasks with multiple text and audio prompts without requiring additional training. Existing audio-guided visual editing methods often necessitate training on specific datasets to align audio with text, limiting their generalization to real-world situations. We leverage a pre-trained multi-modal encoder with strong zero-shot capabilities and integrate diverse audio into visual editing tasks, by alleviating the discrepancy between the audio encoder space and the diffusion model's prompt encoder space. Additionally, we propose a novel approach to handle complex scenarios with multiple and multi-modal editing prompts through our separate noise branching and adaptive patch selection. Our comprehensive experiments on diverse editing tasks demonstrate that our framework excels in handling complicated editing scenarios by incorporating rich information from audio, where text-only approaches fail.

Figures

Figures reproduced from arXiv: 2508.20379 by the authors.

Figure 1
Figure 1. Left: To address the limited generalization of previous methods that require addi￾tional training to align audio data with text, our framework leverages an aligned multimodal encoder pretrained on large-scale datasets without extra training. Right: Instead of enhancing diffusion models to handle multiple editing prompts through additional training, we propose a novel method, separate noise branching and adaptive pat… view at source ↗
Figure 2
Figure 2. Editing results for Audio-guided Image Editing ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Editing results for Audio-guided Video Editing ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of different noise integration methods. Directly averaging each noise [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Audio-guided visual editing examples with complex settings—(a) magnitude guided [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 29 canonical work pages

  1. [27]

    Zero-shot audio-visual editing via cross-modal delta denoising

    Yan-Bo Lin, Kevin Lin, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Chung-Ching Lin, Xiaofei Wang, Gedas Bertasius, and Lijuan Wang. Zero-shot audio-visual editing via cross-modal delta denoising. arXiv preprint arXiv:2503.20782, 2025

    arXiv 2025

  2. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [2]

    Text2live: Text-driven layered image and video editing

    Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In European conference on computer vision, pages 707–723. Springer, 2022

    work page 2022

  4. [3]

    SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models

    Burak Can Biner, Farrin Marouf Sofian, Umur Berkay Karaka¸ s, Duygu Ceylan, Erkut Erdem, and Aykut Erdem. Sonicdiffusion: Audio-driven image generation and editing with pretrained diffusion models. arXiv preprint arXiv:2405.00878, 2024

    work page Pith review arXiv 2024

  5. [4]

    Align your latents: High-resolution video synthesis with latent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023

    2023

  6. [5]

    Ledits++: Limitless image editing using text-to-image models

    Manuel Brack, Felix Friedrich, Katharia Kornmeier, Linoy Tsaban, Patrick Schramowski, Kristian Kersting, and Apolinário Passos. Ledits++: Limitless image editing using text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8861–8870, 2024

    work page 2024

  7. [6]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 721–725. IEEE, 2020

    work page 2020

  8. [7]

    Diffedit: Diffusion-based semantic image editing with mask guidance

    Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based semantic image editing with mask guidance. arXiv preprint arXiv:2210.11427, 2022

    arXiv 2022

Show all 57 references
  1. [8]

    Con- ditional generation of audio from video via foley analogies

    Yuexi Du, Ziyang Chen, Justin Salamon, Bryan Russell, and Andrew Owens. Con- ditional generation of audio from video via foley analogies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2426–2436, 2023

    2023

  2. [9]

    Structure and content-guided video synthesis with diffusion models

    Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7346–7356, 2023

    2023

  3. [10]

    Tokenflow: Consistent diffusion features for consistent video editing

    Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.10373, 2023

    2023 arXiv

  4. [11]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15180–15190, 2023

    2023

  5. [12]

    Atomovideo: High fidelity image-to-video generation

    Litong Gong, Yiran Zhu, Weijie Li, Xiaoyang Kang, Biao Wang, Tiezheng Ge, and Bo Zheng. Atomovideo: High fidelity image-to-video generation. arXiv preprint arXiv:2403.01800, 2024. 12 KIM ET AL : AUDIO-GUIDED VISUAL EDITING WITH COMPLEX MULTI-MODAL PROMPTS

    2024 arXiv

  6. [13]

    Dif- fuseq: Sequence to sequence text generation with diffusion models

    Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. Dif- fuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.08933, 2022

    2022 arXiv

  7. [14]

    Flexecontrol: Flexible and efficient multimodal control for text-to-image generation

    Xuehai He, Jian Zheng, Jacob Zhiyuan Fang, Robinson Piramuthu, Mohit Bansal, Vicente Ordonez, Gunnar A Sigurdsson, Nanyun Peng, and Xin Eric Wang. Flexecontrol: Flexible and efficient multimodal control for text-to-image generation. arXiv preprint arXiv:2405.04834, 2024

    2024 arXiv

  8. [15]

    Prompt-to-prompt image editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022

    2022 arXiv

  9. [16]

    Clip- score: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clip- score: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021

    2021 arXiv

  10. [17]

    Direct inversion: Boosting diffusion-based editing with 3 lines of code

    Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Direct inversion: Boosting diffusion-based editing with 3 lines of code. arXiv preprint arXiv:2310.01506, 2023

    2023 arXiv

  11. [18]

    Text2video-zero: Text-to- image diffusion models are zero-shot video generators

    Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to- image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer...

    2023

  12. [19]

    Enclap: Combining neural audio codec and audio-text joint embedding for automated audio captioning

    Jaeyeon Kim, Jaeyoon Jung, Jinjoo Lee, and Sang Hoon Woo. Enclap: Combining neural audio codec and audio-text joint embedding for automated audio captioning. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6735–673...

    2024

  13. [20]

    Multi-concept customization of text-to-image diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931–1941, 2023

    1931

  14. [21]

    Sound-guided semantic image manipulation

    Seung Hyun Lee, Wonseok Roh, Wonmin Byeon, Sang Ho Yoon, Chanyoung Kim, Jinkyu Kim, and Sangpil Kim. Sound-guided semantic image manipulation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3377–3386, 2022

    2022

  15. [22]

    Soundini: Sound-guided diffusion for natural video editing

    Seung Hyun Lee, Sieun Kim, Innfarn Yoo, Feng Yang, Donghyeon Cho, Youngseo Kim, Huiwen Chang, Jinkyu Kim, and Sangpil Kim. Soundini: Sound-guided diffusion for natural video editing. arXiv preprint arXiv:2304.06818, 2023

    2023 arXiv

  16. [23]

    Generating real- istic images from in-the-wild sounds

    Taegyeong Lee, Jeonghun Kang, Hyeonyu Kim, and Taehwan Kim. Generating real- istic images from in-the-wild sounds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7160–7170, 2023

    2023

  17. [24]

    Learning visual styles from audio-visual associations

    Tingle Li, Yichen Liu, Andrew Owens, and Hang Zhao. Learning visual styles from audio-visual associations. In European Conference on Computer Vision, pages 235–252. Springer, 2022. KIM ET AL : AUDIO-GUIDED VISUAL EDITING WITH COMPLEX MULTI-MODAL PROMPTS 13

    2022

  18. [25]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning

    Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems , 35:17612–17625, 2022

    2022

  19. [26]

    Ctrl-adapter: An efficient and versatile framework for adapting diverse controls to any diffusion model

    Han Lin, Jaemin Cho, Abhay Zala, and Mohit Bansal. Ctrl-adapter: An efficient and versatile framework for adapting diverse controls to any diffusion model. arXiv preprint arXiv:2404.09967, 2024

    2024 arXiv

  20. [28]

    Audioldm: Text-to-audio generation with latent diffusion models

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503, 2023

    2023 arXiv

  21. [29]

    Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models

    Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  22. [30]

    Videofusion: Decomposed diffusion models for high-quality video generation

    Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2023

  23. [31]

    Zero-shot unsupervised and text-based audio editing using ddpm inversion

    Hila Manor and Tomer Michaeli. Zero-shot unsupervised and text-based audio editing using ddpm inversion. arXiv preprint arXiv:2402.10009, 2024

    2024 arXiv

  24. [32]

    Sdedit: Guided image synthesis and editing with stochastic differential equations

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021

    2021 arXiv

  25. [33]

    Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition

    Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, and Bolei Zhou. Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pag...

    2024

  26. [34]

    Null-text inversion for editing real images using guided diffusion models

    Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6038–6047, 2023

    2023

  27. [35]

    Conditional image-to-video generation with latent flow diffusion models

    Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image-to-video generation with latent flow diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18444– 18455, 2023

    2023

  28. [36]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 14 KIM ET AL : AUDIO-GUIDED VISUAL EDITING ...

    2023 arXiv

  29. [37]

    The 2017 davis challenge on video object segmentation

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine- Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675, 2017

    2017 arXiv

  30. [38]

    Grad-tts: A diffusion probabilistic model for text-to-speech

    Vadim Popov, Ivan V ovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pages 8599–8608. PMLR, 2021

    2021

  31. [39]

    Fatezero: Fusing attentions for zero-shot text-based video editing

    Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15932–15942, 2023

    2023

  32. [40]

    Hierarchi- cal text-conditional image generation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchi- cal text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    2022 arXiv

  33. [41]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684–10695, 2022

    2022

  34. [42]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  35. [43]

    Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers

    Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2304.09116, 2023

    2023 arXiv

  36. [44]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  37. [45]

    Codi-2: In-context, interleaved, and interactive any-to-any generation

    Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal. Codi-2: In-context, interleaved, and interactive any-to-any generation. arXiv preprint arXiv:2311.18775, 2023

    2023 arXiv

  38. [46]

    Any-to-any generation via composable diffusion

    Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. arXiv preprint arXiv:2305.11846, 2023

    2023 arXiv

  39. [47]

    Splicing vit features for semantic appearance transfer

    Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10748–10757, 2022

    2022

  40. [48]

    Plug-and-play diffusion features for text-driven image-to-image translation

    Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1921–1930, 2023

    1921

  41. [49]

    Audit: Audio editing by following instructions with latent diffusion models

    Yuancheng Wang, Zeqian Ju, Xu Tan, Lei He, Zhizheng Wu, Jiang Bian, et al. Audit: Audio editing by following instructions with latent diffusion models. Advances in Neural Information Processing Systems, 36, 2024. KIM ET AL : AUDIO-GUIDED VISUAL EDITING WITH COMPLEX MULTI-MODAL...

    2024

  42. [50]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference...

    2023

  43. [51]

    Cvpr 2023 text guided video editing competition

    Jay Zhangjie Wu, Xiuyu Li, Difei Gao, Zhen Dong, Jinbin Bai, Aishani Singh, Xiaoyu Xiang, Youzeng Li, Zuwei Huang, Yuanxi Sun, et al. Cvpr 2023 text guided video editing competition. arXiv preprint arXiv:2310.16003, 2023

    2023 arXiv

  44. [52]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023

    2023 arXiv

  45. [53]

    Ar-diffusion: Auto-regressive diffusion model for text generation

    Tong Wu, Zhihao Fan, Xiao Liu, Hai-Tao Zheng, Yeyun Gong, Jian Jiao, Juntao Li, Jian Guo, Nan Duan, Weizhu Chen, et al. Ar-diffusion: Auto-regressive diffusion model for text generation. Advances in Neural Information Processing Systems , 36, 2024

    2024

  46. [54]

    Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners

    Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners. arXiv preprint arXiv:2402.17723, 2024

    2024 arXiv

  47. [55]

    Align, adapt and inject: Sound-guided unified image generation

    Yue Yang, Kaipeng Zhang, Yuying Ge, Wenqi Shao, Zeyue Xue, Yu Qiao, and Ping Luo. Align, adapt and inject: Sound-guided unified image generation. arXiv preprint arXiv:2306.11504, 2023

    2023 arXiv

  48. [56]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586–595, 2018

    2018

  49. [57]

    Uni-controlnet: All-in-one control to text-to-image diffusion models

    Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems , 36, 2024

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.