Pith. sign in

REVIEW 6 cited by

SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Sound

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.06612 v2 pith:OWDILHDJ submitted 2024-06-06 cs.CV cs.LGcs.SDeess.AS

Rishit Dagli , Shivesh Prakash , Robert Wu , Houman Khosravani This is my paper · ORCID
classification cs.CVcs.LGcs.SDeess.AS
keywords audiospatialgeneratingcontentimagesvisualexperiencesgenerated
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Generating combined visual and auditory sensory experiences is critical for the consumption of immersive content. Recent advances in neural generative models have enabled the creation of high-resolution content across multiple modalities such as images, text, speech, and videos. Despite these successes, there remains a significant gap in the generation of high-quality spatial audio that complements generated visual content. Furthermore, current audio generation models excel in either generating natural audio or speech or music but fall short in integrating spatial audio cues necessary for immersive experiences. In this work, we introduce SEE-2-SOUND, a zero-shot approach that decomposes the task into (1) identifying visual regions of interest; (2) locating these elements in 3D space; (3) generating mono-audio for each; and (4) integrating them into spatial audio. Using our framework, we demonstrate compelling results for generating spatial audio for high-quality videos, images, and dynamic images from the internet, as well as media generated by learned approaches.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FoleySpace: Vision-Aligned Binaural Spatial Audio Generation

    cs.SD 2025-08 conditional novelty 7.0 of 10

    FoleySpace generates binaural audio from silent video by estimating a 3D sound-source trajectory from object detection and depth and conditioning a diffusion model on that trajectory plus monaural audio.

  2. In-the-wild Audio Spatialization with Flexible Text-guided Localization

    cs.SD 2025-06 conditional novelty 6.0 of 10

    A text-guided latent diffusion model converts monaural audio into binaural audio whose perceived directions and distances follow user-specified text prompts.

  3. OmniAudio: Generating Spatial Audio from 360-Degree Video

    eess.AS 2025-04 conditional novelty 6.0 of 10

    OmniAudio generates First-order Ambisonics audio directly from 360-degree video using dual-branch video encoding and flow-matching pre-training, and it introduces the Sphere360 dataset and benchmark.

  4. Listen to Your Map: An Online Representation for Spatial Sonification

    cs.RO 2024-12 conditional novelty 5.0 of 10

    A sensor-centric 360-degree circular and cylindrical projection of an online Gaussian process distance field provides more accurate and more complete spatial information for sonification than raw depth images.

  5. Deep Learning for Personalized Binaural Audio Reproduction

    eess.AS 2025-08 accept novelty 4.0 of 10

    A structured survey of deep learning for personalized binaural audio, covering explicit HRTF prediction and end-to-end synthesis, datasets, metrics, and open challenges.

  6. ASAudio: A Survey of Advanced Spatial Audio Research

    eess.AS 2025-08 unverdicted novelty 3.0 of 10

    A comprehensive survey that systematically categorizes spatial audio research by representation, task, dataset, and evaluation.

Pith tools