Pith. sign in

REVIEW 5 major objections 4 minor 41 references

Seeing Soundscapes: Audio-Visual Generation and Separation from Soundscapes Using Audio-Visual Separator

T0 review · 5 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mixed audio containing two sound classes can be split into two class-aligned embeddings, letting a pretrained image generator draw either a combined scene or a separate image for each class.

desk verdict New task framing for mixed-audio image generation, but the method is a thin extension and the evaluation is too weak to support the state-of-the-art claim. read the letter →

arxiv 2504.18283 v1 pith:E3RERAZ7 submitted 2025-04-25 cs.CV cs.AIcs.MMcs.SDeess.AS

Minjae Kang , Martim Brandão This is my paper · ORCID
classification cs.CVcs.AIcs.MMcs.SDeess.AS
keywords audio-visualgenerationmixedaudiosoundscapeseparationcontrastivelearningClassRepresentationScoreVGGSoundimagefrom
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces AV-GAS, a model that generates images from soundscapes—audio containing several sound classes at once—rather than only from single-class audio. It claims to be the first to produce both a single image containing all classes and one separate image per class from the same mixed audio. The paper also proposes new evaluation metrics, Class Representation Score (CRS) and a modified R@K, and reports that its method outperforms the state of the art by 7% in CRS and 4% in R@2* on the VGGSound dataset. The central idea is that a contrastively trained separator can split a mixed-audio embedding into two halves aligned with the two source classes, and that a weighted sum of those halves yields a combined image.

What carries the argument

The load-bearing object is the audio-visual separator $f_{AV}$, a ResNet-18 that maps mixed audio to a single 4096-dimensional embedding $z^{AV}_{mix}$, with the first 2048 dimensions trained to match one class and the second 2048 to match the other. Training uses InfoNCE contrastive losses that pull each half toward the corresponding unmixed audio embedding (A2A alignment) and optionally toward the corresponding image embedding (A2V alignment), without class labels. At inference, the pretrained image generator $G$ produces separated images from each half and a combined image from $\lambda z^{AV}_1 + (1-\lambda) z^{AV}_2$, where $\lambda$ controls the blend. The whole argument rests on this linear split-and-recombine operation being valid in the pretrained latent space.

What would settle it

Take a held-out pair of classes outside the 20 training combinations, such as 'dog barking' plus 'playing harp', feed the mixed audio at $\lambda = 0.5$, and score the resulting image with an open-vocabulary detector for both prompts. If both classes are not detected at a rate significantly above the single-class baseline, the claimed linear composition does not generalise beyond the trained pairs.

Watch

Extended reading notes

Core claim

The paper's central claim is that mixed-audio generation reduces to an embedding problem rather than a generator problem. Given a pretrained image generator, a separator network can map a mixed audio signal to a 4096-dimensional vector whose first and second halves are contrastively aligned to the two source classes; feeding either half to the generator produces a class-separated image, and feeding a weighted sum produces a single image containing both classes. The authors report that audio-to-audio alignment alone gives the best results, with CRS 7 points higher and R@2* 4 points higher than the state-of-the-art baseline on mixed-audio generation, and that separated foreground and background images reach CRS values of 47% and 57% respectively. They also define the new audio-visual separation task of generating one image per class from a mixed input, which prior single-class audio-to-image models cannot do.

Load-bearing premise

The paper assumes that the latent space of the pretrained audio and image encoders and the image generator is linearly compositional: splitting a mixed-audio embedding into two halves, and later recombining them with a scalar weight, preserves each class's identity in the generated image.

Editorial extensions

If this is right

  • From a mixed audio input, the model can generate a single image containing all classes present, a capability single-class audio-to-image models lack.
  • The same mixed audio also yields one separate image per class, defining the new audio-visual separation task.
  • The proposed Class Representation Score (CRS) measures directly whether all audio classes appear in a generated image and can be applied to any multi-class generation model.
  • Reported results on VGGSound are 7% higher CRS and 4% higher R@2* than the state of the art for mixed-audio image generation.
  • The model also produces plausible images for unrealistic mixtures (e.g., an airplane over underwater bubbling), suggesting the separated embeddings carry class semantics rather than memorised training scenes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the latent space is truly linearly compositional, the same split-and-recombine trick could be transferred to other pretrained generators, including diffusion-based text-to-image models, provided their conditioning latents are approximately linear; the paper does not make this extension.
  • A direct next test is whether the separator can be trained on more than two sources by splitting the embedding into more segments; the paper lists this as future work, and the architecture does not obviously forbid it.
  • Because the separator is trained without class labels and the half-ordering is fixed by training tuple order, test-time inference has no mechanism to know which half is foreground and which is background; downstream uses would need a permutation-invariant step.
  • The CRS metric could be reused to evaluate multi-object text-to-image generation, where the same 'are all requested objects present?' question arises in a neighbouring task.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes AV-GAS, an audio-to-image system that takes a mixed audio input containing two sound sources and generates either a single composite image containing both classes or two class-separated images. The method trains an 'audio-visual separator' (a ResNet-18 in the Sound2Scene pipeline) whose 4096-dimensional output is split into two 2048-dimensional embeddings. The first and second halves are aligned, via InfoNCE losses, to the audio and image embeddings of the two ground-truth classes (A2A and A2V alignments). At inference, each half is fed into a pretrained BigGAN/ICGAN generator to produce separated images, and a convex combination of the two halves is fed to the same generator to produce a mixed image (Eq. 4). The paper also introduces two tasks—mixed-audio generation and audio-visual separation—and a new metric, Class Representation Score (CRS), plus a modified R@K. Experiments are conducted on VGGSound, using 20 hand-crafted class combinations of one foreground and one background sound, and the method is compared only against Sound2Scene.

Significance. If the central claims held, the paper would be a useful first step toward a genuinely harder audio-visual generation setting: generating scenes from soundscapes rather than single-class audio. The proposal of a distinct audio-visual separation task and a detector-based class-representation metric are constructive contributions, and the reuse of Sound2Scene's pretrained encoders and generator makes the ablation clean. The paper is also honest about several limitations, including failure cases and the restriction to background/foreground distinctions. However, the evidence is not yet sufficient: the main mechanism relies on an unvalidated linear-compositionality assumption for the generator's latent space; the quantitative gains are small relative to their reported variance; the new metrics depend on manually tuned prompts; and the comparison rests on a single baseline. The core idea is defensible and the missing experiments are well defined, so the paper is repairable within its scope.

major comments (5)
  1. [§3.2 and Eq. (4)] The mixed-image generation claim depends on the assumption that the pretrained generator's conditioning space is linearly compositional: the model feeds λz_AV1 + (1−λ)z_AV2 directly to G. Nothing in the InfoNCE training (Eqs. 1–3) constrains this convex combination to be a valid conditioning input for a generator trained only on single-class embeddings. This is load-bearing for the first task. I recommend adding an oracle experiment: feed λz_A1_GT + (1−λ)z_A2_GT (or its visual counterpart) using Sound2Scene's own encoders into G and report CRS and R@2*. If the oracle also produces images containing both classes, the linear-compositionality premise is supported; if not, the reported mixed-image successes are not explained by the proposed separation mechanism. The paper should also report how λ was chosen, since λ is a free parameter that directly controls the trade-off.
  2. [§4.2.2, Table 1] The claimed quantitative gains are small relative to the reported variability. For mixed-image generation, CRS is 0.11±0.12 for the proposed A2A method versus 0.04±0.05 for Sound2Scene, and R@2* is 0.06±0.06 versus 0.02±0.00. The '7% higher CRS and 4% higher R@2*' figures are absolute percentage-point differences that lie within one standard deviation of the measurements. No significance test, confidence interval, or per-combination breakdown is provided, so the statement in §4.2.2 that the method 'significantly outperforms' the baseline is not supported by the presented evidence.
  3. [§4.2.1 and Tables 1–3] The state-of-the-art comparison is limited to Sound2Scene. The paper justifies this by noting that other single-class audio-to-image methods are either not designed for audio-visual generation or are not publicly available, but the abstract and introduction nevertheless claim state-of-the-art performance and 'first' success on mixed audio. To support that claim, at least one additional baseline that can generate multiple objects or a recent open-source audio-to-image model should be evaluated, or the claim should be explicitly scoped to the Sound2Scene architecture family.
  4. [Supplementary B, Eq. (5)] CRS and the modified R@K depend on prompts that are manually tuned to the detector and to CLIP. The supplementary rules allow replacing prompts when detection fails (e.g., 'mountain' instead of 'volcano', 'aquarium' instead of 'underwater bubbling'), and the same prompt list is used for both CRS and R@2*/R@1. Because the proposed metrics are a core contribution, they need to be robust to evaluation choices. I ask for a sensitivity analysis: report CRS and R@K with a fixed, pre-registered prompt set, with no substitution rules, and with an alternative detector, to show that the reported ranking is not an artifact of prompt selection.
  5. [§3.3.1 and Conclusion] The assignment of the first and second halves of the split embedding to foreground and background classes is fixed by the training tuple order ('We aligned the first half of the embedding vector to foreground classes and the second half to background classes'), and no permutation inference is performed at test time. The conclusion itself acknowledges that the method is 'limited to background/foreground distinctions' and lists automated assignment as future work. This means the claim that the model can 'generate separate images for each class present in a mixed audio input' is overstated for arbitrary soundscapes: for two foreground classes or an unseen ordering of the same classes, the model has no mechanism to know which half corresponds to which class. In addition, the statement in §4.1 that 'class labels were not used for training' is in tension with the use of foreground/background ordering to construct training tuples. Please reconcile this and either restrict the claims to the 20 pre-defined (background, foreground) combinations or add a permutation/assignment module.
minor comments (4)
  1. [Table 1] The column header 'A2A† A2A†' appears to be a typo; one of the two columns should presumably be 'A2V†'.
  2. [Figure 3 and Eq. (4)] The control parameter λ is used in Figure 3 but is not defined until Eq. (4); please define it where it first appears or add a forward reference.
  3. [Figure 1 caption] Figure 1 labels 'Task 1: Audio-Visual Generation' and 'Task 2: Audio-Visual Separation', but the caption text and Figure 3 use different numbering conventions that may confuse readers; please standardize the task labels.
  4. [§3.1] The phrase 'two ground truth separated images that cannot be mixed like audio' is vague; clarify that the two images are kept separate because there is no natural 'mixed image' ground truth for the combined scene.

Circularity Check

1 steps flagged · score 6.0 of 10

Separated-image 'predictions' decode embeddings fitted to the ground-truth classes; mixed-image claim is assumption-heavy but not circular.

  1. fitted input called prediction [Section 3.2-3.3, Eqs. (1)-(4); Tables 2-3]
    "The first and the second half of the embedding correspond to each of the classes present in mixed audio, respectively. Each is compared with a ground truth audio embedding and aligned with ground truth image embeddings. ... First, unmixed separated images can be generated, where each image contains the class corresponding to one of the separated audio embeddings."

    The two halves of the 4096-dim separator output are trained, via InfoNCE (Eqs. 2-3), to match the ground-truth audio and image embeddings of class 1 and class 2, respectively. The 'separated image' for each class is then produced by feeding the corresponding fitted half into the pretrained generator (Eq. 4). Success on CRS/R@1 in Tables 2-3 therefore largely reflects that each half was explicitly aligned to that class's embeddings during training; the separation result is a direct decode of the training targets rather than an independently discovered ability. The paper presents this as a new task and result, but the output is forced by the training objective.

full rationale

The combined-image result (Eq. 4) is not circular: the separator is trained against Sound2Scene's pretrained encoders, and the resulting images are scored by external YOLO-World and CLIP, so the mixed-audio generation claim has independent empirical content even though the linear-compositionality premise is an unverified assumption. The separated-image contribution, however, is a fitted-input-called-prediction: each half of the 4096-d embedding is explicitly optimized to match one class's ground-truth audio/image embedding, and the 'separated image' is then decoded from that fitted half by the pretrained generator. The paper's CRS/R@1 results in Tables 2-3 therefore largely re-report the training alignment rather than an independently discovered ability to separate mixed audio. No self-citation chain is load-bearing; the reliance on Sound2Scene is external. The fixed foreground/background assignment (acknowledged in the Conclusion) also means no permutation inference is performed, reinforcing that the separation is constructed rather than inferred. The mixed-image generation, by contrast, rests on an untested linear-compositionality assumption, which is a correctness risk but not a circular step.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central claims rest on assumptions about the compositionality of a pretrained latent space, a fixed embedding ordering, and a hand-tuned evaluation protocol. There are no fitted scalar constants beyond the evaluation prompt choices and lambda, but those choices materially affect reported scores.

free parameters (2)
  • CRS/R@K prompt set = e.g., 'mountain' for 'volcano', 'aquarium' for 'underwater bubbling', 'machine' for 'slot machine'
    Supplementary Table 5 lists hand-selected YOLO-World prompts; prompts were changed when the detector struggled, so the reported CRS, R@2*, and R@1 scores are partly determined by these choices.
  • lambda (embedding mixing weight) = 0.5 for figures, no ablation
    Eq. 4 combines the two separated embeddings with a scalar lambda; the paper states figures use lambda=0.5 but does not study its effect on CRS/R@K.
assumptions (4)
  • domain assumption Sound2Scene's pretrained audio and image encoders define a shared latent space in which linearly interpolating or splitting embeddings preserves class-specific semantic content.
    Used throughout Section 3.2 and Eq. 4; no evidence for linear compositionality is provided.
  • ad hoc to paper A fixed split of the mixed embedding into first and second halves can represent foreground and background classes, with assignment decided by training tuple order rather than inferred from audio content.
    Section 3.2 says the first half corresponds to one class and the second to the other; this imposes an ordering that may not generalize beyond two sound sources.
  • domain assumption Mixing two audio signals by simple addition retains enough class-discriminative information for a ResNet-18 encoder to separate them, and VGGSound single-class audio and sampled frames are reliable class indicators.
    Section 3.1 constructs training tuples by adding audio and using Sound2Scene's high-correlation frame numbers; label noise in VGGSound is not addressed.
  • ad hoc to paper YOLO-World detections with the manually chosen prompt list are a valid measure of whether generated images contain the audio classes.
    Section 4.1 and Supplementary B define CRS using YOLO-World; prompts are tuned for detection, so the metric is not an independent benchmark.
invented entities (1)
  • audio-visual features (split 2048-d halves of a 4096-d mixed-audio embedding)
    purpose: Represent each sound class in a shared audio-visual latent space so that each half can drive the image generator
    No external probe, ablation, or released code demonstrates that each half encodes the intended class; the only evidence is the paper's own CRS/R@K numbers under hand-tuned prompts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Seeing Soundscapes: Audio-Visual Generation and Separation from Soundscapes Using Audio-Visual Separator." pith.science (2026). https://pith.science/paper/E3RERAZ7

@misc{pith2026250418283,
  author       = {Pith},
  title        = {Pith review of: Seeing Soundscapes: Audio-Visual Generation and Separation from Soundscapes Using Audio-Visual Separator},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E3RERAZ7}},
  note         = {Machine review of arXiv:2504.18283}
}
read the original abstract

Recent audio-visual generative models have made substantial progress in generating images from audio. However, existing approaches focus on generating images from single-class audio and fail to generate images from mixed audio. To address this, we propose an Audio-Visual Generation and Separation model (AV-GAS) for generating images from soundscapes (mixed audio containing multiple classes). Our contribution is threefold: First, we propose a new challenge in the audio-visual generation task, which is to generate an image given a multi-class audio input, and we propose a method that solves this task using an audio-visual separator. Second, we introduce a new audio-visual separation task, which involves generating separate images for each class present in a mixed audio input. Lastly, we propose new evaluation metrics for the audio-visual generation task: Class Representation Score (CRS) and a modified R@K. Our model is trained and evaluated on the VGGSound dataset. We show that our method outperforms the state-of-the-art, achieving 7% higher CRS and 4% higher R@2* in generating plausible images with mixed audio.

Figures

Figures reproduced from arXiv: 2504.18283 by the authors.

Figure 1
Figure 1. Comparison between our approach and existing methods. Our approach processes a mixed audio input to generate images, whereas existing methods generate images given single-class audio and fail to generate plausible images given mixed audio. Our method can be used for two tasks: first, to generate an image containing all classes present in the audio (task1: audio-visual generation (coloured in red)); second, to genera… view at source ↗
Figure 2
Figure 2. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of our approach. Our approach includes two modules: the separation module (for training) and the generation module (for inference). In the separation module, audio-visual training input tuples extracted from video samples are used for training. The audio-visual separator learns to distinguish objects in mixed audio using contrastive learning loss without class labels. The audio and/or image embeddings extra… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Results of audio-visual generation (generating images given mixed audio) Existing methods [13, 20, 26] focus on generating images from a single-class audio input, whereas our method focuses on generating images from a mixed audio input. We show the results from the sta…
Figure 5
Figure 5. Figure 5: For comparison, the results of Sound2Scene [20] are shown when mixed sound is given as input. Each column represents results from Sound2Scene [20] and our method generating mixed images (task: audio-visual generation), and generating separated images (task: audio-visua…
Figure 6
Figure 6. Figure 6: Examples of Failure Cases. The two images show representative failure cases of generated images from our model. The first image corresponds to a generated image from ‘people crowd + waterfall burbling’ mixed audio where a GAN-based image generator struggles to generate…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 30 canonical work pages

  1. [1]

    Sonicdiffusion: Audio-driven image generation and editing with pretrained diffusion models

    Burak Can Biner, Farrin Marouf Sofian, Umur Berkay Karakas ¸, Duygu Ceylan, Erkut Erdem, and Aykut Er- dem. Sonicdiffusion: Audio-driven image generation and editing with pretrained diffusion models. arXiv preprint arXiv:2405.00878, 2024. 2

    arXiv 2024

  2. [2]

    Large scale gan training for high fidelity natural image synthesis

    Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018. 3, 5

    arXiv 2018

  3. [3]

    Instance- conditioned gan

    Arantxa Casanova, Marl `ene Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero-Soriano. Instance- conditioned gan. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2021. 3, 5

    work page 2021

  4. [4]

    Chatterjee and A

    M. Chatterjee and A. Cherian. Sound2sight: Generating vi- sual dynamics from sound and context. In European Confer- ence on Computer Vision (ECCV), 2020. 2

    work page 2020

  5. [5]

    Learning audio-visual dynamics using scene graphs for au- dio source separation

    Moitreya Chatterjee, Narendra Ahuja, and Anoop Cherian. Learning audio-visual dynamics using scene graphs for au- dio source separation. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 2

    work page 2024

  6. [6]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zis- serman. Vggsound: A large-scale audio-visual dataset. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020. 2, 3, 5, 1

    work page 2020

  7. [7]

    iquery: Instruments as queries for audio-visual sound separation

    Jiaben Chen, Renrui Zhang, Dongze Lian, Jiaqi Yang, Ziyao Zeng, and Jianbo Shi. iquery: Instruments as queries for audio-visual sound separation. In IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2023. 2, 8 8

    work page 2023

  8. [8]

    Deep cross-modal audio-visual generation

    Lele Chen, Sudhanshu Srivastava, Zhiyao Duan, and Chen- liang Xu. Deep cross-modal audio-visual generation. InPro- ceedings of the on Thematic Workshops of ACM Multimedia,

    work page

Show all 41 references
  1. [9]

    Filter-recovery network for multi-speaker audio- visual speech separation

    Haoyue Cheng, Zhaoyang Liu, Wayne Wu, and Limin Wang. Filter-recovery network for multi-speaker audio- visual speech separation. In International Conference on Learning Representations (ICLR), 2023. 2

    2023

  2. [10]

    Yolo-world: Real-time open- vocabulary object detection

    Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xing- gang Wang, and Ying Shan. Yolo-world: Real-time open- vocabulary object detection. In IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR) , 2024. 6, 1, 2

    2024

  3. [11]

    Jha, Teddy Koker, Luca Di Liello, Daniel Stancl, Changsheng Quan, Maxim Grechkin, and William Falcon

    Nicki Skafte Detlefsen, Jiri Borovec, Justus Schock, A. Jha, Teddy Koker, Luca Di Liello, Daniel Stancl, Changsheng Quan, Maxim Grechkin, and William Falcon. Torchmetrics - measuring reproducibility in pytorch. Journal of Open Source Software, 2022. 6

    2022

  4. [12]

    B. Duan, W. Wang, H. Tang, H. Latapie, and Y . Yan. Cascade attention guided residue learning gan for cross-modal trans- lation. In International Conference on Pattern Recognition (ICPR), 2021. 2

    2021

  5. [13]

    Fanzeres and Climent Nadeu

    Leonardo A. Fanzeres and Climent Nadeu. Sound-to- imagination: An exploratory study on unsupervised cross- modal translation using diverse audiovisual data. arXiv preprint arXiv:2106.01266, 2022. 1, 2, 6, 7

    2022 arXiv

  6. [14]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), 2014. 2

    2014

  7. [15]

    Cmcgan: A uniform framework for cross-modal visual-audio mutual generation

    Wangli Hao, Zhaoxiang Zhang, and He Guan. Cmcgan: A uniform framework for cross-modal visual-audio mutual generation. arXiv preprint arXiv:1711.08102, 2017. 2

    2017 arXiv

  8. [16]

    Zhang, Shaoqing Ren, and Jian Sun

    Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. 3

    2015

  9. [17]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. In Advances in Neural Information Processing Systems (NeurIPS), 2017. 5

    2017

  10. [18]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. In Advances in Neural Informa- tion Processing Systems(NeurIPS), 2020. 2

    2020

  11. [19]

    Ultralyt- ics yolo

    Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralyt- ics yolo. https://github.com/ultralytics/ ultralytics, 2023. Accessed: 2024-06-15. 6

    2023

  12. [20]

    Sound to visual scene generation by audio-to-visual latent alignment

    Sung-Bin Kim, Arda Senocak, Hyunwoo Ha, Andrew Owens, and Tae-Hyun Oh. Sound to visual scene generation by audio-to-visual latent alignment. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 1, 2, 3, 4, 5, 6, 7

    2023

  13. [21]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. 6

    2015

  14. [22]

    King’s computational research, en- gineering and technology environment (create), 2022

    King’s College London. King’s computational research, en- gineering and technology environment (create), 2022. Ac- cessed: 2024-02-02. 6

    2022

  15. [23]

    Montesinos, Venkatesh S

    Juan F. Montesinos, Venkatesh S. Kadandale, and Gloria Haro. V ovit: Low latency graph-based audio-visual voice separation transformer. In European Conference on Com- puter Vision (ECCV), 2022. 2

    2022

  16. [24]

    Narasimhan, S

    M. Narasimhan, S. Ginosar, A. Owens, A. Efros, and T. Dar- rell. Strumming to the beat: Audio-conditioned contrastive video textures. In IEEE Winter Conference on Applications of Computer Vision (WACV), 2022. 2

    2022

  17. [25]

    Freeman, Michael Rubinstein, and Wojciech Ma- tusik

    Tae-Hyun Oh, Tali Dekel, Changil Kim, Inbar Mosseri, William T. Freeman, Michael Rubinstein, and Wojciech Ma- tusik. Speech2face: Learning the face behind a voice. In IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2019

    2019

  18. [26]

    Estimat- ing visual information from audio through manifold learn- ing

    Fabrizio Pedersoli, Dryden Wiebe, Amin Banitalebi, Yong Zhang, George Tzanetakis, and Kwang Moo Yi. Estimat- ing visual information from audio through manifold learn- ing. arXiv preprint arXiv:2208.02337, 2022. 1, 2, 6, 7

    2022 arXiv

  19. [27]

    Image generation associ- ated with music data

    Yue Qiu and Hirokatsu Kataoka. Image generation associ- ated with music data. In IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), 2018. 2

    2018

  20. [28]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International conference on machine learning...

    2021

  21. [29]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Confer- ence on Machine Learning (ICML), 2021. 2

    2021

  22. [30]

    Hierarchical text-conditional image gen- eration with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gen- eration with clip latents. arXiv preprint arXiv:2204.06125, 2022

    2022 arXiv

  23. [31]

    High-resolution im- age synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution im- age synthesis with latent diffusion models. arXiv preprint arXiv:2112.10752, 2021. 2

    2021 arXiv

  24. [32]

    Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen

    Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. arXiv preprint arXiv:1606.03498, 2016. 5

    2016 arXiv

  25. [33]

    S2i- bird: Sound-to-image generation of bird species using gen- erative adversarial networks

    Joo Yong Shim, Joongheon Kim, and Jong-Kook Kim. S2i- bird: Sound-to-image generation of bird species using gen- erative adversarial networks. In International Conference on Pattern Recognition (ICPR), 2021. 2

    2021

  26. [34]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Confer- ence on Machine Learning (ICML), 2015. 2

    2015

  27. [35]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In International Conference on Learning Representations (ICLR), 2021. 2 9

    2021

  28. [36]

    Rethinking the in- ception architecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the in- ception architecture for computer vision. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 5, 6, 8

    2016

  29. [37]

    R. Tan, A. Ray, A. Burns, B. A. Plummer, J. Salamon, O. Nieto, B. Russell, and K. Saenko. Language-guided audio-visual source separation via trimodal consistency. In 2023 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), 2023. 2

    2023

  30. [38]

    Efthymios Tzinis, Scott Wisdom, Tal Remez, and John R. Hershey. Audioscopev2: Audio-visual attention architec- tures for calibrated open-domain on-screen sound separation. In European Conference on Computer Vision (ECCV), 2022. 2

    2022

  31. [39]

    To- wards audio to scene image synthesis using generative adver- sarial network

    Chia-Hung Wan, Shun-Po Chuang, and Hung yi Lee. To- wards audio to scene image synthesis using generative adver- sarial network. In IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), 2018. 2

    2018

  32. [40]

    Visual echoes: A simple unified transformer for audio-visual gener- ation

    Shiqi Yang, Zhi Zhong, Mengjie Zhao, Shusuke Takahashi, Masato Ishii, Takashi Shibuya, and Yuki Mitsufuji. Visual echoes: A simple unified transformer for audio-visual gener- ation. arXiv preprint arXiv:2405.14598, 2024. 2

    2024 arXiv

  33. [41]

    Audiotoken: Adaptation of text-conditioned dif- fusion models for audio-to-image generation

    Guy Yariv, Itai Gat, Lior Wolf, Yossi Adi, and Idan Schwartz. Audiotoken: Adaptation of text-conditioned dif- fusion models for audio-to-image generation. arXiv preprint arXiv:2305.13050, 2023. 2 10 Seeing Soundscapes: Audio-Visual Generation and Separation from Soundscapes Us...

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.